Skip to contents

Construct a robots evaluation context (ADR-009 §1, sitemap-spec §13.0)

Source: R/robots-context.R
robots_context.Rd

Bundles the three independent robots axes one allow/disallow evaluation runs under. It is the carrier validate_sitemap_robots() accepts, and it is deliberately separate from ruleset_context(): selecting a sitemap ruleset does not select a robots policy (ADR-009 §1). A Bing sitemap ruleset does not imply a Bing robots policy — the two are chosen independently.

Usage

robots_context(
  product_token = "*",
  policy_ruleset = "google",
  matcher_backend = "google"
)

Arguments

product_token

The robots.txt group to match against; defaults to the catch-all "*".

policy_ruleset

The robots status-policy ruleset; defaults to "google", the historical behavior of validate_sitemap().

matcher_backend

The robots matcher backend; defaults to "google".

Value

An object of class sitemapr_robots_context: a named list of the three axes.

Details

  • product_token — the robots.txt group used for matching: "*" for the catch-all group, or a crawler token such as "Googlebot". This is not the HTTP request user-agent sitemapr fetches with.

  • policy_ruleset — whose HTTP-status → policy semantics govern (what a 404, a 403 or a 5xx robots.txt means).

  • matcher_backend — which matcher decides a rule against a URL.

The vocabulary for the last two belongs to the sibling robotstxtr package, not to sitemapr, and each value is validated against the set the installed build publishes on its engine contract — so an axis this build cannot honour is rejected here rather than deep inside the engine. robotstxtr is an optional dependency; when it is absent only the shape of each axis is checked.

A backend whose published token_policy is bounded_profiles (Bing, Yandex) accepts only its own vendor profile tokens, and the accepted set is not published, so it cannot be validated here. An unsupported token is not an error: every URL comes back undecided and surfaces as ROBOTS_INDETERMINATE. Prefer robots_context_preset(), whose tokens are known-good for their backend.

See also

robots_context_preset() for the per-engine presets, validate_sitemap_robots() for the entry point that accepts one, and ruleset_context() for the independent sitemap-source axes.

Examples

robots_context()
#> $product_token
#> [1] "*"
#> 
#> $policy_ruleset
#> [1] "google"
#> 
#> $matcher_backend
#> [1] "google"
#> 
#> attr(,"class")
#> [1] "sitemapr_robots_context"
robots_context(product_token = "Googlebot")
#> $product_token
#> [1] "Googlebot"
#> 
#> $policy_ruleset
#> [1] "google"
#> 
#> $matcher_backend
#> [1] "google"
#> 
#> attr(,"class")
#> [1] "sitemapr_robots_context"