Batch Extract URLs

Batch or crawl? Use batch when you already know the URLs: from Map, a sitemap, a search result or your own list. It can span any number of sites. Use Crawl when you want one site's pages discovered for you.

Pricing: successful pages only. Each URL that comes back is charged once at the tier price (credits_per_page in the 202). URLs refused up front (in invalid_urls), pages that fail or 404, and starting, polling and cancelling cost nothing. Credits are taken page by page; if your balance runs out, the batch stops with stopped_reason: "insufficient_credits" and keeps the pages it already fetched. If every URL is invalid the POST returns 400 with invalid_urls and no job is created.

Robots.txt. Batch behaves like Extract: you named each URL, so it is fetched as you asked. Crawl and Map, which discover pages for you, always obey robots.txt.

Plan limits. URLs per batch, and crawl and batch jobs running at once:

Retention. Results are kept 3 days after the batch finishes, up to 2 MB of content per page and 250 MB per job; a job runs for at most 6 hours. Read them from Batch Status and stop a batch early with Cancel Batch.

Webhooks. Set webhook_url to a public https URL and Scavio POSTs one signed JSON event when the batch ends: event is completed, failed or cancelled, job is the same summary the status endpoint returns, and sent_at is the send time. Results are not in the body; fetch them from the status URL. Headers: Scavio-Signature, Scavio-Event and Scavio-Job-Id. A delivery that does not get a 2xx within 10 seconds is retried twice, 30 seconds and 5 minutes after the first attempt. Redirects are not followed, and the URL must use a public host name (no IP addresses or private hosts).

Verify every delivery. The 202 response returns webhook_secret (whsec_...) once; store it. Scavio-Signature is t=<unix seconds>,v1=<hex>, where v1 is the HMAC-SHA256 of <t>.<raw request body> keyed with that secret. Recompute it over the raw bytes, compare in constant time, and reject timestamps more than 5 minutes old:

import hashlib, hmac, time

def verify_scavio_signature(secret: str, header: str, raw_body: bytes, tolerance: int = 300) -> bool:
    """header is the Scavio-Signature value: t=<unix>,v1=<hex>."""
    parts = dict(p.strip().split("=", 1) for p in header.split(",") if "=" in p)
    t, v1 = parts.get("t", ""), parts.get("v1", "")
    if not t.isdigit() or not v1 or abs(time.time() - int(t)) > tolerance:
        return False
    signed = t.encode() + b"." + raw_body
    expected = hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, v1)

# Flask example
from flask import Flask, request, abort
app = Flask(__name__)
WEBHOOK_SECRET = "whsec_from_the_202_response"

@app.post("/scavio/webhook")
def scavio_webhook():
    if not verify_scavio_signature(WEBHOOK_SECRET, request.headers.get("Scavio-Signature", ""), request.get_data()):
        abort(400)
    event = request.get_json()
    print(event["event"], event["job"]["id"], event["job"]["pages"]["succeeded"])
    return "", 204

A delivery body looks like this:

{
  "event": "completed",
  "job": {
    "id": "crawl_01a12d2e-f813-73c2-8c07-4c1f0ae402df",
    "type": "crawl",
    "status": "completed",
    "stopped_reason": null,
    "error": null,
    "url": "https://example.com/",
    "format": "markdown",
    "mode": "normal",
    "pages": { "discovered": 1, "queued": 0, "succeeded": 1, "failed": 0, "skipped": 0 },
    "credits_used": 1,
    "created_at": "2026-10-11T22:56:55.312Z",
    "started_at": "2026-10-11T22:57:00.424Z",
    "completed_at": "2026-10-11T22:57:02.665Z",
    "expires_at": "2026-10-14T22:57:02.665Z",
    "webhook": { "status": "pending", "attempts": 1 },
    "status_url": "https://api.scavio.dev/api/v1/extract/crawl/crawl_01a12d2e-f813-73c2-8c07-4c1f0ae402df"
  },
  "sent_at": "2026-10-11T22:57:02.701Z"
}