Probe a URL to inspect what it is, without resolving it

probe_url() is a lightweight, non-resolving inspection: it fetches exactly one URL (or reads one local file) and reports what it is — a sitemap, a sitemap index, a feed, robots.txt, an HTML page, or an error state — without expanding indexes or following children. It is the diagnostic counterpart to read_sitemap(), which resolves a sitemap to its final URL rows. Use probe_url() to diagnose first and read_sitemap() to resolve.

Usage

probe_url(
  url,
  limits = fetch_limits(),
  user_agent = default_user_agent(),
  ssrf_guard = TRUE,
  policy = request_policy()
)

Arguments

url

A single sitemap URL or local file path to inspect.

limits

Network limits for the fetch, as from fetch_limits().

user_agent

The User-Agent header for the fetch. Defaults to the package User-Agent.

ssrf_guard

Logical; when TRUE (default) the structural SSRF guard runs on every hop.

policy

A request policy applied to the fetch, as from request_policy(). Defaults to the no-op policy.

Value

A sitemapr_probe object: a classed list with the fields url, final_url, status_code, content_type, detected_type, xml_root, is_compressed, child_count, sample, problems, and suggested_next. detected_type is one of "sitemap", "sitemap_index", "feed", "robots_txt", "html", "xml_other", "not_found", "fetch_error", or "parse_error".

Details

The fetch goes through the same bounded, SSRF-safe engine as the rest of the package, so probe_url() inherits the network-safety policy (per-hop SSRF guard, redirect and body-size ceilings) and the limits / policy overrides. It fetches only the given URL: for a sitemap index it counts the direct <sitemap> children by parsing the index locally, but it never fetches them.

A fetch or parse failure is represented in the returned record (detected_type becomes "fetch_error", "not_found", or "parse_error", with an explanatory problems row), not thrown — probe_url() is diagnostic. Only invalid input (a non-string url) raises an error.

See also

read_sitemap() to resolve a sitemap to its URL rows, and sitemap_tree() to discover a site's sitemaps from its root.

Examples

# Probe a local sitemap file: detected as a urlset, with a child count.
xml <- paste0(
  '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">',
  "<url><loc>https://example.com/</loc></url>",
  "<url><loc>https://example.com/about</loc></url>",
  "</urlset>"
)
path <- tempfile(fileext = ".xml")
writeLines(xml, path)
probe_url(path)
#> <sitemapr_probe>
#>   url:           /tmp/Rtmpd4DRti/file78a4b0148f6.xml
#>   detected_type: sitemap
#>   xml_root:      urlset
#>   is_compressed: FALSE
#>   child_count:   2
#>   suggested_next: read_sitemap() to extract the URL rows
#>   problems:      0

# A sitemap index: children are COUNTED, never fetched.
index <- paste0(
  '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">',
  "<sitemap><loc>https://example.com/s1.xml</loc></sitemap>",
  "<sitemap><loc>https://example.com/s2.xml</loc></sitemap>",
  "</sitemapindex>"
)
index_path <- tempfile(fileext = ".xml")
writeLines(index, index_path)
probe_url(index_path)
#> <sitemapr_probe>
#>   url:           /tmp/Rtmpd4DRti/file78a1688878a.xml
#>   detected_type: sitemap_index
#>   xml_root:      sitemapindex
#>   is_compressed: FALSE
#>   child_count:   2
#>   suggested_next: read_sitemap() to expand the index and extract URLs
#>   problems:      0

# Probe a live URL (requires network):
# probe_url("https://example.com/sitemap.xml")