Extract
Crawl Status and Results
Read a crawl's progress and its pages. Each response carries the job's status, page counts (discovered, queued, succeeded, failed, skipped) and credits used so far, plus up to 100 results paged by cursor, each with its URL, depth, HTTP status, content and the credits it cost. Free: polling never uses credits and works even at a zero balance. Results are kept for 3 days after the crawl finishes.
Authorizations
AuthorizationstringheaderrequiredBearer authentication header of the form Bearer <token>, where <token> is your Scavio API key (e.g. Bearer sk_live_your_key).
Parameters
path + queryidstringpathrequiredThe crawl id from the 202 response (crawl_...). Only jobs started with your API key's account are visible; any other id returns 404.
Example: crawl_01a12d21-fb42-7000-88cf-6887f653ee5d
cursorstringquerynext_cursor from the previous response. Omit it for the first page of results.
limitintegerquerydefault:25Results per response, 1-100. A response is also cut at about 5 MB of page content, so it can hold fewer than limit.
statusenum<string>queryOnly results with this status.
succeeded— Pages that came back, with content. Each cost the tier price.failed— Pages that could not be fetched, with error and status_code. Free.skipped— Pages not fetched, with skip_reason (for example robots). Free.
include_contentbooleanquerydefault:trueSet false to page through results without their content: a light way to poll progress or list URLs.
Notes
Job status. status moves from queued to running and ends as completed, failed (with error) or cancelled. A completed crawl that stopped before running out of pages says why in stopped_reason: max_pages, insufficient_credits, time_limit or storage_limit. Poll every few seconds with include_content=false, then page through the content once the job is done, or set a webhook_url on Crawl and skip polling.
Each result. status is succeeded, failed (with error and the page's status_code) or skipped (with skip_reason, for example robots). credits is what that page cost: the tier price for a success, 0 for anything else. truncated is true when a page passed the 2 MB stored-content cap.
Paging. Results come back in the order pages were discovered, 25 per response by default and up to 100, and a response is also cut at about 5 MB of content. Pass next_cursor as cursor until it is null. Results can be read while the crawl is still running.
Retention. Results are deleted at expires_at, 3 days after the crawl finishes. After that the id returns 404.
Request
from scavio import ScavioClient
client = ScavioClient(api_key="sk_live_your_key")
job_id = "crawl_01a12d21-fb42-7000-88cf-6887f653ee5d"
# Progress only, no page content: cheap to call every few seconds
s = client.extract.crawl_status(job_id, include_content=False)
print(s["data"]["status"], s["data"]["pages"], s["data"]["credits_used"])
# Every successful page, 100 per response
cursor = None
while True:
page = client.extract.crawl_status(job_id, status="succeeded", limit=100, cursor=cursor)
for r in page["data"]["results"]:
print(r["url"], r["content_length"])
cursor = page["data"]["next_cursor"]
if not cursor:
breakResponse