Skip to contents

Read a sitemap into a tidy tibble of URLs

Source: R/read-sitemap.R
read_sitemap.Rd

Parses one or more sitemap sources — sitemap URLs or local sitemap files — into the tidy row tibble: one row per URL with loc, lastmod, changefreq, priority, the images/video/news/alternates extension list-columns, and source_sitemap provenance. Supported formats are XML urlset and sitemapindex, RSS 2.0 and Atom 0.3/1.0 feeds, the plain-text format, transparent gzip (.xml.gz/.txt.gz), and — local files only — bounded .tar.gz archives. A feed is parsed into the same row schema, one row per item/entry.

Usage

read_sitemap(
  x,
  user_agent = default_user_agent(),
  limits = fetch_limits(),
  index_limits = NULL,
  policy = request_policy(),
  max_active = NULL
)

read_sitemaps(
  x,
  user_agent = default_user_agent(),
  limits = fetch_limits(),
  index_limits = NULL,
  policy = request_policy(),
  max_active = NULL
)

Arguments

x

One or more sitemap URLs or paths to local sitemap files (.xml, .txt, .gz, or .tar.gz).

user_agent

The User-Agent header for HTTP fetches. Defaults to the package User-Agent.

limits

Network limits for HTTP fetches, as from fetch_limits().

index_limits

Sitemapindex-expansion bounds (recursion depth and per-index child-count cap), as from index_limits(). Defaults to index_limits().

policy

A request_policy() applied to every HTTP hop (root, robots.txt, discovery, redirects, and index children) — configure custom headers, authentication, a proxy, TLS options, retry/backoff, and per-host throttling there. Defaults to the no-op policy.

max_active

Optional worker cap for opt-in bounded-concurrency expansion of a sitemap index (ADR-008). NULL (the default) keeps child fetches sequential; a larger value (a small, polite bound such as 4–6 is typical) fetches up to that many independent child sitemaps at once. Concurrency changes only when a child's bytes arrive, never the row order or budget-truncation point — the result is byte-identical to sequential mode.

Value

A tibble of URL rows with sources and problems attributes. An entry-point fetch failure or unsupported content raises a classed error condition.

Details

The result carries a sources attribute (the per-source fetch-metadata records) and a problems attribute (a tibble of non-fatal issues such as skipped archive members, unfetchable index children, or failed members of a submitted vector). read_sitemap() never returns a validation findings tibble; use validate_sitemap() for that.

A top-level sitemap index is expanded recursively (cycle-safe, depth- and count-capped) so every reachable child sitemap's rows carry per-child provenance; the bounds are configurable via index_limits.

When x contains more than one source, inputs are normalized, deduplicated, and capped using the submitted-list source-record policy. Per-source failures are recorded in the problems attribute and successful sources still contribute rows. Scalar calls keep the stricter historical behavior: an entry-point fetch failure or unsupported content raises a classed condition.

Examples

# Read a local sitemap file into a tidy tibble of URLs.
xml <- paste0(
  '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">',
  '<url><loc>https://example.com/</loc>',
  '<lastmod>2024-01-01</lastmod></url>',
  '<url><loc>https://example.com/about</loc></url>',
  '</urlset>'
)
path <- tempfile(fileext = ".xml")
writeLines(xml, path)
read_sitemap(path)
#> # A tibble: 2 × 9
#>   loc    lastmod             changefreq priority images video  news   alternates
#>   <chr>  <dttm>              <chr>         <dbl> <list> <list> <list> <list>    
#> 1 https… 2024-01-01 00:00:00 NA               NA <NULL> <NULL> <NULL> <NULL>    
#> 2 https… NA                  NA               NA <NULL> <NULL> <NULL> <NULL>    
#> # ℹ 1 more variable: source_sitemap <chr>

# Read directly from a sitemap URL; a top-level index expands recursively.
# read_sitemap("https://example.com/sitemap.xml")