Host-aware request-throttle settings for a request-policy

Configures per-host request pacing for a request-policy. Pass the result as request_policy(throttle = ). Requests are keyed by canonical host:port, so different origins pace independently and a redirect onto another host pays that host's pace, not the origin's. Supply EITHER a minimum interval between consecutive requests to one host (min_interval, seconds) OR a request budget per time window (requests per window seconds, evenly paced at window / requests). Pacing is shared across every hop of one logical operation (roots, redirects, robots, discovery candidates, and index children), so an index traversal's many children pace against one set of host buckets. NULL (the default policy field) means no pacing.

Usage

request_throttle(min_interval = NULL, requests = NULL, window = NULL)

Arguments

min_interval

Minimum seconds between consecutive requests to one host.

requests, window

A request budget: at most requests requests per window seconds to one host, evenly paced (interval window / requests). Both must be supplied together and take precedence over min_interval.

Value

A sitemapr_request_throttle for request_policy(throttle = ).

See also

Other request-policy: request_auth_basic(), request_auth_bearer(), request_policy(), request_proxy(), request_retry()

Examples

# At most one request per host every 2 seconds.
request_throttle(min_interval = 2)
#> $min_interval
#> [1] 2
#> 
#> attr(,"class")
#> [1] "sitemapr_request_throttle"

# A budget of 10 requests per 60 seconds per host (evenly paced).
request_throttle(requests = 10, window = 60)
#> $min_interval
#> [1] 6
#> 
#> attr(,"class")
#> [1] "sitemapr_request_throttle"