Website Crawler API

A website crawler API crawls a site for you and returns its pages as data. With Scavio you POST a start URL to /api/v1/extract/crawl; the crawl discovers same-site pages from the sitemap and from links, breadth-first within your depth, page and path limits, and returns each page as Markdown, plain text or HTML. It runs as a job: a 202 with an id, then results from a status URL or a signed webhook. Map lists a site's URLs for 1 credit, and batch extracts up to 1,000 URLs you already have.

Yes, always. Crawl and map read the site's robots.txt and skip disallowed pages, which appear in crawl results as skipped with skip_reason robots and cost nothing. A Crawl-delay is honoured up to 5 seconds. There is no override. Batch behaves like single-page Extract: you named each URL, so it is fetched as you asked.

Up to 10,000 pages per crawl, depending on your plan: Free 10, PAYG 100, Project 500, Bootstrap 2,000, Startup 5,000 and Growth 10,000. Active crawl and batch jobs at once are 1, 1, 2, 3, 5 and 10, and URLs per batch are 10, 50, 100, 500, 1,000 and 1,000. Map returns up to 500 links on Free, 1,000 on PAYG and 5,000 on paid plans. A request above your plan's limit returns a 400 that names the cap.

Yes, set mode to advanced. The default normal mode covers static pages, docs and most blogs; advanced renders JavaScript for pages whose content is built in the browser, at the same 1 credit per page; ultra handles the hardest pages at 2 credits per page. The mode applies to every page in the crawl or batch.

You pay only for pages that come back: 1 credit per successful page on normal and advanced, 2 on ultra. Failed pages, 404s, robots-skipped pages, the robots.txt and sitemap reads, polling and cancelling are free, and map costs 1 credit per call. On the Growth plan billed yearly ($2 per 1,000 credits) that is $0.002 per page. New accounts get 50 free credits with no card, and paid plans start at $30 per month for 7,000 credits.

Credits are taken page by page as the crawl runs, with nothing reserved up front. If your balance runs out, the crawl stops cleanly with stopped_reason insufficient_credits, keeps every page it already fetched, and those pages stay readable from the status URL. You need one page's cost to start a job.

Set webhook_url to a public https URL. When the job completes, fails or is cancelled, Scavio sends one POST with event, the job summary and sent_at, and a Scavio-Signature header of the form t=<unix>,v1=<hex>, where v1 is the HMAC-SHA256 of the timestamp, a dot and the raw body, keyed with the webhook_secret returned in the 202. Verify it in constant time and reject old timestamps. A delivery without a 2xx is retried 30 seconds and 5 minutes later. Webhook verification code in Python and JavaScript

Yes. Markdown is the default format for crawl and batch, with links kept as absolute URLs, so each page is ready to chunk and embed. Use text for plain text or html for the raw page. Each result also carries its URL, depth, HTTP status and content_length, which make good metadata for a vector store.

Sitemap first. Map reads the sitemaps listed in robots.txt and the site's sitemap.xml, follows sitemap indexes and gzipped sitemaps, then adds the links on the start page. Each link says whether it came from the sitemap or the page, with the site's lastmod date when it publishes one. Filter with include_paths, exclude_paths and search, and set sitemap to only or skip to use one source. One call returns up to 5,000 links for 1 credit.

Call map with sitemap set to only to get the URLs the sitemap lists, keep the ones you want, then send them to batch extract, up to 1,000 per job depending on plan. Batch returns each URL as Markdown in input order and bills only the pages that come back. Or start a crawl with sitemap set to only to fetch the sitemap's pages without following links.

Yes. DELETE /api/v1/extract/crawl/{id} stops a queued or running crawl straight away. No further pages are fetched or billed, the pages already fetched stay readable for 3 days, and a cancelled event goes to your webhook if you set one. Cancelling also frees an active-job slot.

3 days after the job finishes; the status response gives the exact expires_at. Each page stores up to 2 MB of content (marked truncated beyond that), each job up to 250 MB, and a job runs for at most 6 hours. Copy the pages you need into your own storage.

For crawl, map and batch scraping to Markdown, yes, with differences worth knowing. Both charge about 1 credit per page. Firecrawl's pricing page (October 2026) says a page that responds with an error status such as 403 or 404 still costs 1 credit; Scavio bills only successful pages. Firecrawl's paid plans include monthly credits starting at Hobby, $16 per month billed yearly for 5,000; Scavio has 50 free credits once, then plans from $30 per month for 7,000 credits, and the same key also covers search and 50+ platform APIs. Firecrawl offers features Scavio does not, such as JSON extraction formats, an interact API and a recurring free tier.

Yes. A crawl only fetches pages on the start URL's site, where www and the bare domain count as one site. Set allow_subdomains to include subdomains such as docs.example.com. Pages under the start URL's path are crawled first, query-string variants count as one page by default, and no URL is fetched twice.