Crawl a Website

How a crawl runs. The crawl starts at url, reads the site's robots.txt and (unless sitemap is skip) its sitemap, then fetches same-site pages breadth-first, following the links on each page up to max_depth. Pages under the start URL's path go first, so a crawl started at /docs spends its pages on the docs before the rest of the site. It never leaves the site, never fetches the same URL twice and stops at max_pages (stopped_reason: "max_pages").

Robots.txt is always obeyed. Disallowed pages are not fetched; they appear in the results as skipped with skip_reason: "robots" and cost nothing. A Crawl-delay in robots.txt is honoured up to 5 seconds. There is no override.

Pricing: successful pages only. Each page that comes back is charged once, at Extract's price for the tier: normal 1 credit, advanced 1, ultra 2 (credits_per_page in the 202). Failed pages, 404s, robots-skipped pages and the robots.txt and sitemap reads cost nothing, and starting, polling and cancelling are free. Credits are taken page by page as the crawl runs; you need one page's cost to start, and if your balance runs out the crawl stops cleanly with stopped_reason: "insufficient_credits", keeping every page it already fetched.

Plan limits. Pages per crawl, crawl and batch jobs running at once, and URLs per batch:

Starting a job while your plan's active jobs are still running returns a 429 with running_jobs and a Retry-After header. Per-request concurrency limits apply to the POST itself, not to the pages a crawl fetches.

Retention and size. Results are kept for 3 days after the job finishes (expires_at), then deleted; copy what you need. Each page stores up to 2 MB of content (truncated: true beyond that) and each job up to 250 MB (stopped_reason: "storage_limit"). A job runs for at most 6 hours (stopped_reason: "time_limit").

Results. Read them from Crawl Status, paged by cursor. Stop a crawl early with Cancel Crawl. To list a site's URLs without fetching them, use Map; to extract a list of URLs you already have, use Batch Extract.

Webhooks. Set webhook_url to a public https URL and Scavio POSTs one signed JSON event when the crawl ends: event is completed, failed or cancelled, job is the same summary the status endpoint returns, and sent_at is the send time. Results are not in the body; fetch them from the status URL. Headers: Scavio-Signature, Scavio-Event and Scavio-Job-Id. A delivery that does not get a 2xx within 10 seconds is retried twice, 30 seconds and 5 minutes after the first attempt. Redirects are not followed, and the URL must use a public host name (no IP addresses or private hosts).

Verify every delivery. The 202 response returns webhook_secret (whsec_...) once; store it. Scavio-Signature is t=<unix seconds>,v1=<hex>, where v1 is the HMAC-SHA256 of <t>.<raw request body> keyed with that secret. Recompute it over the raw bytes, compare in constant time, and reject timestamps more than 5 minutes old:

import hashlib, hmac, time

def verify_scavio_signature(secret: str, header: str, raw_body: bytes, tolerance: int = 300) -> bool:
    """header is the Scavio-Signature value: t=<unix>,v1=<hex>."""
    parts = dict(p.strip().split("=", 1) for p in header.split(",") if "=" in p)
    t, v1 = parts.get("t", ""), parts.get("v1", "")
    if not t.isdigit() or not v1 or abs(time.time() - int(t)) > tolerance:
        return False
    signed = t.encode() + b"." + raw_body
    expected = hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, v1)

# Flask example
from flask import Flask, request, abort
app = Flask(__name__)
WEBHOOK_SECRET = "whsec_from_the_202_response"

@app.post("/scavio/webhook")
def scavio_webhook():
    if not verify_scavio_signature(WEBHOOK_SECRET, request.headers.get("Scavio-Signature", ""), request.get_data()):
        abort(400)
    event = request.get_json()
    print(event["event"], event["job"]["id"], event["job"]["pages"]["succeeded"])
    return "", 204

A delivery body looks like this:

{
  "event": "completed",
  "job": {
    "id": "crawl_01a12d2e-f813-73c2-8c07-4c1f0ae402df",
    "type": "crawl",
    "status": "completed",
    "stopped_reason": null,
    "error": null,
    "url": "https://example.com/",
    "format": "markdown",
    "mode": "normal",
    "pages": { "discovered": 1, "queued": 0, "succeeded": 1, "failed": 0, "skipped": 0 },
    "credits_used": 1,
    "created_at": "2026-10-11T22:56:55.312Z",
    "started_at": "2026-10-11T22:57:00.424Z",
    "completed_at": "2026-10-11T22:57:02.665Z",
    "expires_at": "2026-10-14T22:57:02.665Z",
    "webhook": { "status": "pending", "attempts": 1 },
    "status_url": "https://api.scavio.dev/api/v1/extract/crawl/crawl_01a12d2e-f813-73c2-8c07-4c1f0ae402df"
  },
  "sent_at": "2026-10-11T22:57:02.701Z"
}