Validate a sitemap under an engine-aware ruleset (ADR-009)

The versioned, engine-aware entry point parallel to validate_sitemap(). It runs the identical validation pipeline (the XSD schema, protocol/semantic, byte-level classification, and bounded index-expansion layers), then, when an engine overlay is selected, augments the findings tibble with the additive schema-v2 columns (docs/decisions/ADR-009-per-engine-validation-profiles.md §5/§6, docs/findings-contract.md "Per-engine ruleset extension").

Usage

validate_sitemap_ruleset(
  x,
  sitemap_ruleset = sitemap_rulesets(),
  context = ruleset_context(),
  mode = c("strict", "non-strict"),
  user_agent = default_user_agent(),
  limits = fetch_limits(),
  index_limits = NULL,
  policy = request_policy(),
  check_robots = FALSE,
  robots_user_agent = "*",
  robots_context = NULL,
  inspect_pages = FALSE,
  page_sample = 50L,
  page_mode = c("sample", "full"),
  page_budget = page_inspection_budget(),
  page_user_agent = default_user_agent()
)

validate_sitemaps_ruleset(
  x,
  sitemap_ruleset = sitemap_rulesets(),
  context = ruleset_context(),
  mode = c("strict", "non-strict"),
  user_agent = default_user_agent(),
  limits = fetch_limits(),
  index_limits = NULL,
  policy = request_policy(),
  check_robots = FALSE,
  robots_user_agent = "*",
  robots_context = NULL,
  inspect_pages = FALSE,
  page_sample = 50L,
  page_mode = c("sample", "full"),
  page_budget = page_inspection_budget(),
  page_user_agent = default_user_agent()
)

Arguments

x: One or more sitemap URLs or paths to local sitemap files (.xml, .txt, .gz, or .tar.gz).
sitemap_ruleset: The engine ruleset to validate under; one of sitemap_rulesets() (baseline "sitemaps.org" first, the default). The baseline emits the schema-v1 result; an engine overlay adds the additive columns.
context: A per-source validation context from ruleset_context() (the four independent ADR-009 §1 axes). Carried into the context list-column of the additive result. Ignored on the baseline path (which emits no additive columns). This is the RULESET context; the robots context is the separate robots_context argument, and passing one where the other belongs is rejected rather than silently accepted.
mode: "strict" (the default) or "non-strict". In non-strict, strict-only findings are dropped and schema violations are downgraded to warning; in strict, the documented info-to-warning codes are elevated.
user_agent: The User-Agent header for HTTP fetches. Defaults to the package User-Agent.
limits: Network limits for HTTP fetches, as from fetch_limits().
index_limits: Sitemapindex-expansion bounds (recursion depth and per-index child-count cap), as from index_limits(). Defaults to index_limits().
policy: A request_policy() applied to every HTTP hop (root, robots.txt, discovery, redirects, and index children) — configure custom headers, authentication, a proxy, TLS options, retry/backoff, and per-host throttling there. Defaults to the no-op policy.
check_robots: Logical; when TRUE, run the robots.txt allow/disallow check over the advertised URLs and the sitemap document itself (requires the optional robotstxtr package). Defaults to FALSE.
robots_user_agent: The robots.txt group to match against when check_robots = TRUE, e.g. "*" (the catch-all group, the default) or a specific crawler token such as "Googlebot".
robots_context: A robots evaluation context from robots_context() or robots_context_preset(), or NULL (the default) to leave the robots axis as validate_sitemap() treats it. Supplying one runs the robots layer by construction — as in validate_sitemap_robots(), the context IS the request, so check_robots need not also be set — and appends the robots_context list-column. It cannot be combined with a non-default robots_user_agent: that argument is the string shorthand for the same axis, and honouring both would make it ambiguous which one decided the findings.
inspect_pages: Logical; the master opt-in for per-URL page inspection (Layer E). When FALSE (the default) no page is fetched and the result is byte-identical to a call without it: the pinned ten-column findings surface and no page_coverage attribute. When TRUE, a budgeted, deduplicated, deterministically-sampled set of the advertised page URLs is fetched and each fetch's transport outcome maps to at most one page-layer finding (PAGE_STATUS_ERROR, PAGE_STATUS_REDIRECT, PAGE_REDIRECT_CHAIN, PAGE_FETCH_FAILED, PAGE_SSRF_BLOCKED); the run's coverage rides the page_coverage attribute (see Value). Network expansion is never implicit. Page inspection is batch-wide: one budget over the union of the call's deduped page URLs.
page_sample: Integer sample size for page_mode = "sample": how many of the deduplicated page URLs to inspect, chosen by a deterministic stable hash order so re-runs pick the same set. Ignored when page_mode = "full".
page_mode: "sample" (inspect page_sample deduplicated URLs, the default) or "full" (inspect every deduplicated URL, subject to the budget caps).
page_budget: A page-inspection budget list: the aggregate caps (max pages, max requests/hops, max aggregate bytes, per-page body cap, max wall time), each caller-overridable with a safe default. Applies only when inspect_pages = TRUE.
page_user_agent: The HTTP request User-Agent sent when fetching pages (recorded for the "what did the inspector see" caveat; distinct from a robots product token). Defaults to sitemapr's inspector UA.

Value

The findings tibble of validate_sitemap(). Under the baseline sitemap_ruleset it is exactly the pinned ten columns; under an engine overlay it additionally carries ruleset (character), ruleset_revision (character), context (a list-column of the context object as a named list), and provenance (character, per finding), appended in that order after the ten pinned columns. The same source, mode, ruleset, and context yield a row-for-row identical tibble across calls. An engine overlay also stamps a ruleset_run attribute (attr(x, "ruleset_run")) naming the selected ruleset, so report_sitemap() can tell an engine-gated check that ran and passed from one this call could never reach — the ruleset column answers that per row, and a clean run has no rows. Like layers_run it is a run manifest and not part of the row contract; the baseline path stamps nothing. Supplying robots_context appends one further list-column, robots_context, last — after the additive ruleset columns when both axes are selected. Because each axis governs its own columns, a baseline call carrying a robots context returns exactly validate_sitemap_robots()'s result for the same context.

Details

Backwards compatibility is preserved by construction (ADR-009 §5): a baseline call (sitemap_ruleset = "sitemaps.org", the default) returns exactly the pinned ten-column schema-v1 result, byte-identical to validate_sitemap(). The four additive columns appear only for an explicit engine overlay ("google" / "bing" / "yandex"); there is deliberately no profile= argument on validate_sitemap() and no silent default switch to an engine.

This slice fixes the engine-aware carrier and the additive schema; no per-engine evaluators exist yet, so every finding produced under an overlay is a reused baseline code and carries provenance = "inherited_protocol" (an ADR-009 §0 executable class). Later slices override the provenance per code.

Per-URL page inspection (inspect_pages = TRUE) works exactly as in validate_sitemap(), except the transport / canonical / hreflang page findings are assembled under the selected sitemap_ruleset too: under an engine overlay they carry the same additive schema-v2 columns as the base findings, so the per-engine provenance / context (ADR-009 §5.2/§5.3) engages over the page layer rather than emitting as generic baseline diagnostics. As in validate_sitemap(), inspect_pages = FALSE is byte-identical to a call without the argument.

This is the engine-aware entry point, and it carries both engine-aware axes: pass robots_context = to select an engine's robots semantics in the same call (SITE-otfmeyqx). The two axes stay independent as ADR-009 §1 requires — neither is derived from the other, so a Bing sitemap ruleset with a Yandex robots context is a legal and honoured pair — but they are no longer mutually exclusive, and a combined call runs the pipeline once rather than fetching everything twice. Each axis governs its own columns: the additive ruleset columns appear only under an engine overlay, and robots_context appears only when an explicit robots context is supplied. Use validate_sitemap_robots() when you want the robots axis alone.

Examples

xml <- paste0(
  '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">',
  '<url><loc>https://example.com/</loc></url>',
  '</urlset>'
)
path <- tempfile(fileext = ".xml")
writeLines(xml, path)

# Baseline: identical to validate_sitemap().
validate_sitemap_ruleset(path, "sitemaps.org")
#> # A tibble: 0 × 10
#> # ℹ 10 variables: code <chr>, severity <chr>, layer <chr>, subject_type <chr>,
#> #   subject_ref <chr>, message <chr>, evidence <list>, mode <chr>,
#> #   is_strict_only <lgl>, remediation_hint <chr>

# Engine overlay: adds the additive schema-v2 columns.
validate_sitemap_ruleset(path, "google")
#> # A tibble: 0 × 14
#> # ℹ 14 variables: code <chr>, severity <chr>, layer <chr>, subject_type <chr>,
#> #   subject_ref <chr>, message <chr>, evidence <list>, mode <chr>,
#> #   is_strict_only <lgl>, remediation_hint <chr>, ruleset <chr>,
#> #   ruleset_revision <chr>, context <list>, provenance <chr>

# Both engine-aware axes in one call, independently chosen.
# validate_sitemap_ruleset(
#   path, "bing", robots_context = robots_context_preset("yandex")
# )

Usage

Arguments

Value

Details

See also

Examples