Validate a sitemap source against the schema, protocol, and classification rules

The public validation entry point. Reads one or more sitemap sources — sitemap URLs or local sitemap files — runs every finding-producer over them (the XSD schema layer, the protocol/semantic layer, the byte-level classification layer, and, for a sitemap index, the bounded index-expansion layer), and assembles the results into the stable findings contract.

Usage

validate_sitemap(
  x,
  mode = c("strict", "non-strict"),
  user_agent = default_user_agent(),
  limits = fetch_limits(),
  index_limits = NULL,
  policy = request_policy(),
  check_robots = FALSE,
  robots_user_agent = "*",
  inspect_pages = FALSE,
  page_sample = 50L,
  page_mode = c("sample", "full"),
  page_budget = page_inspection_budget(),
  page_user_agent = default_user_agent()
)

validate_sitemaps(
  x,
  mode = c("strict", "non-strict"),
  user_agent = default_user_agent(),
  limits = fetch_limits(),
  index_limits = NULL,
  policy = request_policy(),
  check_robots = FALSE,
  robots_user_agent = "*",
  inspect_pages = FALSE,
  page_sample = 50L,
  page_mode = c("sample", "full"),
  page_budget = page_inspection_budget(),
  page_user_agent = default_user_agent()
)

Arguments

x: One or more sitemap URLs or paths to local sitemap files (.xml, .txt, .gz, or .tar.gz).
mode: "strict" (the default) or "non-strict". In non-strict, strict-only findings are dropped and schema violations are downgraded to warning; in strict, the documented info-to-warning codes are elevated.
user_agent: The User-Agent header for HTTP fetches. Defaults to the package User-Agent.
limits: Network limits for HTTP fetches, as from fetch_limits().
index_limits: Sitemapindex-expansion bounds (recursion depth and per-index child-count cap), as from index_limits(). Defaults to index_limits().
policy: A request_policy() applied to every HTTP hop (root, robots.txt, discovery, redirects, and index children) — configure custom headers, authentication, a proxy, TLS options, retry/backoff, and per-host throttling there. Defaults to the no-op policy.
check_robots: Logical; when TRUE, run the robots.txt allow/disallow check over the advertised URLs and the sitemap document itself (requires the optional robotstxtr package). Defaults to FALSE.
robots_user_agent: The robots.txt group to match against when check_robots = TRUE, e.g. "*" (the catch-all group, the default) or a specific crawler token such as "Googlebot".
inspect_pages: Logical; the master opt-in for per-URL page inspection (Layer E). When FALSE (the default) no page is fetched and the result is byte-identical to a call without it: the pinned ten-column findings surface and no page_coverage attribute. When TRUE, a budgeted, deduplicated, deterministically-sampled set of the advertised page URLs is fetched and each fetch's transport outcome maps to at most one page-layer finding (PAGE_STATUS_ERROR, PAGE_STATUS_REDIRECT, PAGE_REDIRECT_CHAIN, PAGE_FETCH_FAILED, PAGE_SSRF_BLOCKED); the run's coverage rides the page_coverage attribute (see Value). Network expansion is never implicit. Page inspection is batch-wide: one budget over the union of the call's deduped page URLs.
page_sample: Integer sample size for page_mode = "sample": how many of the deduplicated page URLs to inspect, chosen by a deterministic stable hash order so re-runs pick the same set. Ignored when page_mode = "full".
page_mode: "sample" (inspect page_sample deduplicated URLs, the default) or "full" (inspect every deduplicated URL, subject to the budget caps).
page_budget: A page-inspection budget list: the aggregate caps (max pages, max requests/hops, max aggregate bytes, per-page body cap, max wall time), each caller-overridable with a safe default. Applies only when inspect_pages = TRUE.
page_user_agent: The HTTP request User-Agent sent when fetching pages (recorded for the "what did the inspector see" caveat; distinct from a robots product token). Defaults to sitemapr's inspector UA.

Value

The findings tibble described in docs/findings-contract.md: the columns code, severity, layer, subject_type, subject_ref, message, evidence, mode, is_strict_only, and remediation_hint, in the contract's stable order. The same source and mode yield a row-for-row identical tibble across calls. A genuine transport, SSRF, or HTTP failure raises a classed error condition. When inspect_pages = TRUE, the tibble additionally carries a page_coverage attribute (attr(x, "page_coverage")) — a versioned, batch-wide named list reporting what the run covered (eligible, deduplicated, selected, attempted, completed, partial, and which caps bit) so a sampled or capped run is never misread as clean; it is absent when inspect_pages = FALSE. The tibble may also carry a layers_run attribute (attr(x, "layers_run")) — a character vector naming validation layers this run exercised whose execution the result cannot otherwise evidence, so that report_sitemap() can tell "ran and found nothing" from "never ran". It records "robots" for check_robots = TRUE (a clean robots run emits no findings) and "schema" when XSD validation ran on the parsed document. It is a run manifest and not part of the row contract: treat it as advisory, present only when there was something to record.

Details

The source is read once and branched on its sniffed format: an HTML document served where a sitemap was expected yields an UNSUPPORTED_HTML_MASQUERADE classification finding; a plain-text sitemap is checked line-by-line; an RSS 2.0 or Atom 0.3/1.0 feed is parsed into rows and protocol-validated; an XML document is dispatched on its root element. An XML root that is neither urlset nor sitemapindex yields an UNSUPPORTED_ROOT finding rather than an error. A urlset is schema- and protocol-validated; a sitemapindex is schema-validated and recursively expanded (cycle-, depth-, and count-capped), with the traversal events surfaced as INDEX_* findings.

When x contains more than one source, inputs are normalized, deduplicated, and capped using the submitted-list source-record policy. Per-source failures are returned as fetch-layer findings and successful sources still contribute their findings. Scalar calls keep the stricter historical behavior: genuine transport, SSRF, or HTTP failures raise classed conditions.

When check_robots = TRUE, each sitemap-advertised URL is additionally tested against its governing robots.txt (via the optional robotstxtr engine), emitting robots-layer findings for URLs that are disallowed (ROBOTS_DISALLOWED, warning) or that cannot be decided because robots.txt would not fetch (ROBOTS_INDETERMINATE, info). Each distinct origin's robots.txt is fetched once under the SSRF-guarded fetch policy; matching is offline, so every advertised URL is checked with no sampling. The sitemap document's own URL is tested too: a sitemap published at a path its own robots.txt disallows yields ROBOTS_SITEMAP_DISALLOWED (warning, scoped to the source). When robotstxtr is not installed, a classed warning naming the install command is signalled and the check is skipped; every other layer is unaffected.

Examples

xml <- paste0(
  '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">',
  '<url><loc>https://example.com/</loc>',
  '<priority>2.0</priority></url>',
  '</urlset>'
)
path <- tempfile(fileext = ".xml")
writeLines(xml, path)
validate_sitemap(path, mode = "non-strict")
#> # A tibble: 2 × 10
#>   code        severity layer subject_type subject_ref message evidence     mode 
#>   <chr>       <chr>    <chr> <chr>        <chr>       <chr>   <list>       <chr>
#> 1 SCHEMA_INV… warning  sche… field        /tmp/RtmpK… Elemen… <named list> non-…
#> 2 PROTOCOL_P… error    prot… entry        /tmp/RtmpK… <prior… <named list> non-…
#> # ℹ 2 more variables: is_strict_only <lgl>, remediation_hint <chr>

# Validate directly from a sitemap URL.
# validate_sitemap("https://example.com/sitemap.xml")