Extract
Map Website URLs
Give a site and get back its page URLs in one synchronous call: gathered from its sitemaps (including sitemap indexes) and the links on its start page, deduplicated, filtered by path or keyword and limited. Pages disallowed by the site's robots.txt are left out. 1 credit on the normal tier. Use it to plan a crawl, pick the pages worth extracting, or turn a sitemap into a URL list.
Authorizations
AuthorizationstringheaderrequiredBearer authentication header of the form Bearer <token>, where <token> is your Scavio API key (e.g. Bearer sk_live_your_key).
Body
application/jsonurlstringrequiredThe site to map, 1-2048 characters. A bare host like example.com is upgraded to https. Private, loopback and metadata hosts are rejected with a 400 before anything runs.
Example: https://scavio.dev
searchstringOnly return URLs that contain every word of this text, case-insensitive, as a substring of the URL ("api" also matches "apify"). Max 200 characters.
limitintegerdefault:1000Maximum links to return, 1-5,000. Defaults to 1,000 or your plan's limit if lower. Plan limits: Free 500, PAYG 1,000, every paid plan 5,000. A limit above your plan returns a 400 naming the cap. total_found tells you how many matched before the limit.
sitemapenum<string>default:includeWhere the URLs come from.
include— The site's sitemaps plus the links on the start page. The default.only— Sitemaps alone. Returns a 404 when the site has none.skip— Links on the start page alone.
include_subdomainsbooleandefault:falseAlso return URLs on subdomains (docs.example.com for example.com). www and the bare domain always count as one site.
include_pathsstring[]Path globs to keep, max 50. * matches within one path segment and ** across segments. A pattern with no wildcard matches that path and everything under it, so ["/docs"] keeps /docs and /docs/anything; ["/docs/**"] does not keep /docs itself.
Example: ["/blog"]
exclude_pathsstring[]Path globs to drop, same syntax as include_paths. Excludes win over includes.
Example: ["/tag/*", "/page/**"]
ignore_query_parametersbooleandefault:trueTreat URLs that differ only by query string as one URL.
modeenum<string>default:normalFetch tier, as on Extract. Map is 1 credit on normal; a higher tier costs up to 2 (advanced) or 10 (ultra) depending on how many fetches the map needed. Stay on normal unless the site refuses it.
normal— The standard fetch. 1 credit. The default.advanced— For sites whose links are built by JavaScript. Up to 2 credits.ultra— For the hardest sites. Up to 10 credits.
Notes
Sitemap first, then the start page. Map reads the sitemaps the site lists in robots.txt and its sitemap.xml, follows sitemap indexes and gzipped sitemaps, then adds the links on the start page. Each link says where it came from in source, and sitemap links carry the site's lastmod when it publishes one. A single call can map a site with thousands of pages: in the capture behind the playground sample, scavio.dev returned 1,009 URLs from one sitemap.
Robots.txt is always obeyed. URLs the site disallows for crawlers are left out, and there is no override.
From URL list to content. Pass the URLs you keep to Batch Extract, or let Crawl discover and extract the pages for you in one job.
Request
from scavio import ScavioClient
client = ScavioClient(api_key="sk_live_your_key")
# 1 credit: every page URL on the site, from its sitemaps and start page
site = client.extract.map("https://scavio.dev", include_paths=["/alternatives"], limit=100)
print(site["data"]["total_found"])
for link in site["data"]["links"]:
print(link["url"], link["source"], link.get("lastmod"))Response