Validate a sitemap under an explicit robots context (sitemap-spec §13.0)

The robots-aware entry point parallel to validate_sitemap(). It runs the identical validation pipeline with the robots allow/disallow layer on, evaluated under a caller-supplied robots_context() rather than the bare matcher user-agent validate_sitemap() accepts, and appends the context to the result as a robots_context list-column.

Usage

validate_sitemap_robots(
  x,
  context = robots_context(),
  mode = c("strict", "non-strict"),
  user_agent = default_user_agent(),
  limits = fetch_limits(),
  index_limits = NULL,
  policy = request_policy(),
  inspect_pages = FALSE,
  page_sample = 50L,
  page_mode = c("sample", "full"),
  page_budget = page_inspection_budget(),
  page_user_agent = default_user_agent()
)

validate_sitemaps_robots(
  x,
  context = robots_context(),
  mode = c("strict", "non-strict"),
  user_agent = default_user_agent(),
  limits = fetch_limits(),
  index_limits = NULL,
  policy = request_policy(),
  inspect_pages = FALSE,
  page_sample = 50L,
  page_mode = c("sample", "full"),
  page_budget = page_inspection_budget(),
  page_user_agent = default_user_agent()
)

Arguments

x: One or more sitemap URLs or paths to local sitemap files (.xml, .txt, .gz, or .tar.gz).
context: A robots evaluation context from robots_context() or robots_context_preset() (the three ADR-009 §1 robots axes). Defaults to robots_context(), the Google-default widening validate_sitemap() has always applied. Carried into the robots_context list-column of the result.
mode: "strict" (the default) or "non-strict". In non-strict, strict-only findings are dropped and schema violations are downgraded to warning; in strict, the documented info-to-warning codes are elevated.
user_agent: The User-Agent header for HTTP fetches. Defaults to the package User-Agent.
limits: Network limits for HTTP fetches, as from fetch_limits().
index_limits: Sitemapindex-expansion bounds (recursion depth and per-index child-count cap), as from index_limits(). Defaults to index_limits().
policy: A request_policy() applied to every HTTP hop (root, robots.txt, discovery, redirects, and index children) — configure custom headers, authentication, a proxy, TLS options, retry/backoff, and per-host throttling there. Defaults to the no-op policy.
inspect_pages: Logical; the master opt-in for per-URL page inspection (Layer E). When FALSE (the default) no page is fetched and the result is byte-identical to a call without it: the pinned ten-column findings surface and no page_coverage attribute. When TRUE, a budgeted, deduplicated, deterministically-sampled set of the advertised page URLs is fetched and each fetch's transport outcome maps to at most one page-layer finding (PAGE_STATUS_ERROR, PAGE_STATUS_REDIRECT, PAGE_REDIRECT_CHAIN, PAGE_FETCH_FAILED, PAGE_SSRF_BLOCKED); the run's coverage rides the page_coverage attribute (see Value). Network expansion is never implicit. Page inspection is batch-wide: one budget over the union of the call's deduped page URLs.
page_sample: Integer sample size for page_mode = "sample": how many of the deduplicated page URLs to inspect, chosen by a deterministic stable hash order so re-runs pick the same set. Ignored when page_mode = "full".
page_mode: "sample" (inspect page_sample deduplicated URLs, the default) or "full" (inspect every deduplicated URL, subject to the budget caps).
page_budget: A page-inspection budget list: the aggregate caps (max pages, max requests/hops, max aggregate bytes, per-page body cap, max wall time), each caller-overridable with a safe default. Applies only when inspect_pages = TRUE.
page_user_agent: The HTTP request User-Agent sent when fetching pages (recorded for the "what did the inspector see" caveat; distinct from a robots product token). Defaults to sitemapr's inspector UA.

Value

The findings tibble of validate_sitemap() — the pinned ten columns — plus a robots_context list-column of the context as a named list, appended last. The same source, mode and context yield a row-for-row identical tibble across calls.

Details

Use it when the robots question is "would this engine fetch these URLs". validate_sitemap() carries only robots_user_agent, a matcher user-agent string that widens onto the Google policy and matcher; this entry point carries all three axes, so the ROBOTS_DISALLOWED / ROBOTS_INDETERMINATE / ROBOTS_SITEMAP_DISALLOWED findings are decided by the selected engine's own status policy and matcher. Under the default robots_context() the result is exactly validate_sitemap(check_robots = TRUE), plus the added column.

The robots axes are independent of the sitemap_ruleset axis (ADR-009 §1): this entry point selects an engine's robots semantics and does not select a sitemap ruleset. It returns the baseline schema-v1 result, so the additive per-engine ruleset columns of validate_sitemap_ruleset() are not present. To select both axes, call validate_sitemap_ruleset() with its robots_context argument; independence means the axes are chosen separately, not that they cannot be chosen together. This entry point remains the shorthand for the robots axis alone, and is exactly validate_sitemap_ruleset(x, "sitemaps.org", robots_context = context).

A backend the installed robotstxtr reports as capability_unavailable decides nothing rather than guessing: every advertised URL comes back as ROBOTS_INDETERMINATE. The same is true of a product token a bounded matcher backend does not accept — prefer robots_context_preset(), whose tokens are known-good for their backend.

Examples

xml <- paste0(
  '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">',
  '<url><loc>https://example.com/</loc></url>',
  '</urlset>'
)
path <- tempfile(fileext = ".xml")
writeLines(xml, path)

# A local file advertises a remote URL, so this reaches the network only
# when `robotstxtr` is installed; without it the layer warns and skips.
# validate_sitemap_robots(path, robots_context_preset("yandex"))