Default limits for sitemapindex expansion

Returns the configurable bounds the index-expansion engine applies while recursively expanding a sitemapindex. Both are safety bounds, not protocol rules: they keep a hostile or accidentally huge index tree from triggering an unbounded burst of requests or unbounded recursion.

Usage

index_limits(
  max_depth = getOption("sitemapr.max_index_depth", 3L),
  max_children = getOption("sitemapr.max_index_children", 50000L),
  max_total_sitemaps = getOption("sitemapr.max_total_sitemaps", 50000L),
  max_total_urls = getOption("sitemapr.max_total_urls", 2.5e+07)
)

Arguments

max_depth

Maximum recursion depth below the root index (integer). Resolves from the argument, then getOption("sitemapr.max_index_depth"), then the default of 3.

max_children

Maximum number of distinct child entries expanded per index (integer). Resolves from the argument, then getOption("sitemapr.max_index_children"), then the default of 50 000 (the sitemap-protocol per-index entry limit).

max_total_sitemaps

Maximum number of child sitemaps fetched across the entire traversal (numeric). Resolves from the argument, then getOption("sitemapr.max_total_sitemaps"), then the default of 50 000 (the sitemap-protocol per-index entry limit, reused as the traversal-wide ceiling). Kept numeric so Inf is representable.

max_total_urls

Maximum number of URL rows gathered across the entire traversal (numeric). Resolves from the argument, then getOption("sitemapr.max_total_urls"), then the default of 25 000 000. Kept numeric so Inf is representable.

Value

A named list of limits with coerced types.

Details

max_depth counts levels below the root index: the root index is depth 0, its children are depth 1, and an index whose children would land beyond max_depth is not descended (an INDEX_DEPTH_EXCEEDED event). max_children caps how many child entries one index contributes after dedup; entries beyond the cap are dropped (an INDEX_CHILD_COUNT_EXCEEDED event). max_total_sitemaps and max_total_urls are TRAVERSAL-WIDE aggregate budgets spanning the whole recursive expansion: the first caps how many child sitemaps are fetched in total (an INDEX_TOTAL_SITEMAPS_EXCEEDED event), the second caps how many URL rows are gathered in total (an INDEX_TOTAL_URLS_EXCEEDED event). Both are FINITE by default — 50 000 child sitemaps and 25 000 000 URL rows, the ceilings a traversal of protocol-legal sitemaps cannot exceed in practice — so an embedding caller is never exposed to an unbounded traversal over a pathological index graph. Pass Inf explicitly to opt back out. When a budget is reached the traversal stops and returns the accumulated PARTIAL result.

See also

fetch_limits() and discovery_limits() for the other bound constructors, and read_sitemap() which accepts index_limits.

Examples

index_limits()
#> $max_depth
#> [1] 3
#> 
#> $max_children
#> [1] 50000
#> 
#> $max_total_sitemaps
#> [1] 50000
#> 
#> $max_total_urls
#> [1] 2.5e+07
#> 
index_limits(max_depth = 2, max_children = 1000)
#> $max_depth
#> [1] 2
#> 
#> $max_children
#> [1] 1000
#> 
#> $max_total_sitemaps
#> [1] 50000
#> 
#> $max_total_urls
#> [1] 2.5e+07
#> 
index_limits(max_total_sitemaps = 500, max_total_urls = 1e6)
#> $max_depth
#> [1] 3
#> 
#> $max_children
#> [1] 50000
#> 
#> $max_total_sitemaps
#> [1] 500
#> 
#> $max_total_urls
#> [1] 1e+06
#>