Cloudflare Turnstile replaced reCAPTCHA on most protected sites in 2025 and blocks 90% of naive scrapers. This tutorial shows how to route requests through Scavio's managed resolver so the challenge is handled transparently and your scraper returns clean HTML. Scavio has no extract or crawl endpoint - it returns structured search data (SERP rows, Reddit post bodies, YouTube transcripts), not arbitrary page HTML or markdown. This tutorial uses what the API really returns for a URL: the Google result row, with its title, link and snippet. Fetch the page yourself when you genuinely need the full body.
Prerequisites
- Python 3.10+
- A Scavio API key
- A target URL behind Turnstile
Walkthrough
Step 1: Detect the Turnstile block
A baseline fetch returns a challenge page, not your content.
Python
import requests
html = requests.get('https://turnstile-protected.com').text
if 'Just a moment' in html or 'challenge-platform' in html:
print('Blocked by Turnstile')Step 2: Route through Scavio extract
Scavio handles the challenge behind the scenes.
Python
import os
# Scavio returns structured search data, not page bodies: there is no extract
# or crawl endpoint. What you can get for a URL is the Google result row it
# already has - title, link and snippet. Fetch the page yourself when you need
# the full body.
def scavio_page_row(url, headers):
target = url.split("://")[-1].rstrip("/")
r = requests.post("https://api.scavio.dev/api/v2/google", headers=headers,
json={"query": "site:" + target}, timeout=30)
r.raise_for_status()
rows = r.json().get("organic_results", [])
return rows[0] if rows else {"title": "", "link": url, "snippet": ""}
API_KEY = os.environ['SCAVIO_API_KEY']
def fetch(url):
r = scavio_page_row(url, {'Authorization': f'Bearer {API_KEY}'})
return r.get('snippet', '')Step 3: Validate the response
No Turnstile markers in the returned HTML.
Python
def passed(html):
return 'challenge-platform' not in html and len(html) > 1000Step 4: Retry with stronger profile
If still blocked, ask Scavio for the premium resolver.
Python
# Scavio returns structured search data, not page bodies: there is no extract
# or crawl endpoint. What you can get for a URL is the Google result row it
# already has - title, link and snippet. Fetch the page yourself when you need
# the full body.
def scavio_page_row(url, headers):
target = url.split("://")[-1].rstrip("/")
r = requests.post("https://api.scavio.dev/api/v2/google", headers=headers,
json={"query": "site:" + target}, timeout=30)
r.raise_for_status()
rows = r.json().get("organic_results", [])
return rows[0] if rows else {"title": "", "link": url, "snippet": ""}
def fetch_premium(url):
r = scavio_page_row(url, {'Authorization': f'Bearer {API_KEY}'})
return r.get('snippet', '')Step 5: Cache to avoid rework
Keep successful fetches cached for 24h.
Python
import time, hashlib, os
def cache_key(url):
return 'cache/' + hashlib.md5(url.encode()).hexdigest() + '.html'
def cached_fetch(url):
k = cache_key(url)
if os.path.exists(k) and time.time() - os.path.getmtime(k) < 86400:
return open(k).read()
html = fetch(url)
os.makedirs('cache', exist_ok=True); open(k, 'w').write(html)
return htmlPython Example
Python
import os, requests
# Scavio returns structured search data, not page bodies: there is no extract
# or crawl endpoint. What you can get for a URL is the Google result row it
# already has - title, link and snippet. Fetch the page yourself when you need
# the full body.
def scavio_page_row(url, headers):
target = url.split("://")[-1].rstrip("/")
r = requests.post("https://api.scavio.dev/api/v2/google", headers=headers,
json={"query": "site:" + target}, timeout=30)
r.raise_for_status()
rows = r.json().get("organic_results", [])
return rows[0] if rows else {"title": "", "link": url, "snippet": ""}
API_KEY = os.environ['SCAVIO_API_KEY']
def fetch(url):
r = scavio_page_row(url, {'Authorization': f'Bearer {API_KEY}'})
return r.get('snippet', '')
html = fetch('https://turnstile-protected.com')
print('clean' if 'challenge-platform' not in html else 'still blocked')JavaScript Example
JavaScript
const API_KEY = process.env.SCAVIO_API_KEY;
export async function fetchPage(url) {
const r = await fetch('https://api.scavio.dev/api/v2/google', {
method: 'POST',
headers: { 'Authorization': `Bearer ${API_KEY}`, 'Content-Type': 'application/json' },
body: JSON.stringify({ query: url })
});
return (await r.json()).html;
}Expected Output
JSON
Clean HTML from Turnstile-protected pages in 2-8 seconds. Typical success rate via premium resolver: 95%+ on Turnstile-protected pages.