Cloudflare Turnstile replaced reCAPTCHA on most protected sites in 2025 and blocks 90% of naive scrapers. This tutorial shows how to get what you need from a Turnstile-protected site through Scavio instead of fighting the challenge yourself. Scavio's Extract endpoint (POST /api/v1/extract) reads a URL and returns the page as HTML, Markdown or plain text. Start in normal mode and step up only when a site refuses it: advanced renders JavaScript, and ultra is built for the hardest pages. Only a successful extraction is billed, so an attempt that fails costs nothing.
API Docs: Extract API
Prerequisites
- Python 3.10+
- A Scavio API key
- A target URL behind Turnstile
Walkthrough
Step 1: Detect the Turnstile block
A baseline fetch returns a challenge page, not your content.
import requests
html = requests.get('https://turnstile-protected.com').text
if 'Just a moment' in html or 'challenge-platform' in html:
print('Blocked by Turnstile')Step 2: Route through Scavio extract
You get the page HTML back without solving the challenge yourself.
import os
# Scavio Extract reads one URL and returns the page as markdown, html or text.
# mode: "normal" (1 credit), "advanced" renders JavaScript (1 credit),
# "ultra" for the hardest pages (2 credits). Only a successful call is billed.
# Docs: https://scavio.dev/docs/extract
def scavio_extract(url, headers, mode="normal", fmt="markdown"):
r = requests.post("https://api.scavio.dev/api/v1/extract", headers=headers,
json={"url": url, "format": fmt, "mode": mode}, timeout=90)
r.raise_for_status()
return r.json()["data"]["content"]
API_KEY = os.environ['SCAVIO_API_KEY']
def fetch(url, mode='normal'):
return scavio_extract(url, {'Authorization': f'Bearer {API_KEY}'}, mode=mode, fmt='html')Step 3: Validate the response
No Turnstile markers in the returned HTML.
def passed(html):
return 'challenge-platform' not in html and len(html) > 1000Step 4: Step up the mode
If normal mode comes back blocked, retry with advanced, then ultra (2 credits).
def fetch_with_fallback(url):
# Step up one tier at a time. A failed attempt is not billed.
for mode in ('normal', 'advanced', 'ultra'):
try:
html = fetch(url, mode=mode)
except requests.HTTPError:
continue
if passed(html):
return html
return NoneStep 5: Cache to avoid rework
Keep successful fetches cached for 24h.
import time, hashlib, os
def cache_key(url):
return 'cache/' + hashlib.md5(url.encode()).hexdigest() + '.html'
def cached_fetch(url):
k = cache_key(url)
if os.path.exists(k) and time.time() - os.path.getmtime(k) < 86400:
return open(k).read()
html = fetch(url)
os.makedirs('cache', exist_ok=True); open(k, 'w').write(html)
return htmlPython Example
import os, requests
# Scavio Extract reads one URL and returns the page as markdown, html or text.
# mode: "normal" (1 credit), "advanced" renders JavaScript (1 credit),
# "ultra" for the hardest pages (2 credits). Only a successful call is billed.
# Docs: https://scavio.dev/docs/extract
def scavio_extract(url, headers, mode="normal", fmt="markdown"):
r = requests.post("https://api.scavio.dev/api/v1/extract", headers=headers,
json={"url": url, "format": fmt, "mode": mode}, timeout=90)
r.raise_for_status()
return r.json()["data"]["content"]
API_KEY = os.environ['SCAVIO_API_KEY']
def fetch(url, mode='advanced'):
return scavio_extract(url, {'Authorization': f'Bearer {API_KEY}'}, mode=mode, fmt='html')
html = fetch('https://turnstile-protected.com')
print('clean' if 'challenge-platform' not in html else 'still blocked')JavaScript Example
const API_KEY = process.env.SCAVIO_API_KEY;
// mode: 'normal' | 'advanced' (renders JavaScript) | 'ultra' (hardest pages)
export async function fetchPage(url, mode = 'advanced') {
const r = await fetch('https://api.scavio.dev/api/v1/extract', {
method: 'POST',
headers: { 'Authorization': `Bearer ${API_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({ url, format: 'html', mode })
});
if (!r.ok) throw new Error(`extract failed: ${r.status}`);
return (await r.json()).data.content;
}Expected Output
The page HTML in data.content, with its length in data.content_length. normal and advanced cost 1 credit, ultra costs 2, and a failed extraction costs nothing. Some protected pages still refuse every tier; those come back as an error, not as billed challenge HTML you have to detect.