An r/n8n thread mentioned the OP was using 'Google Custom Search plus manual scraping' and wanted a single API. This tutorial walks the replacement path. Scavio has no extract or crawl endpoint - it returns structured search data (SERP rows, Reddit post bodies, YouTube transcripts), not arbitrary page HTML or markdown. This tutorial uses what the API really returns for a URL: the Google result row, with its title, link and snippet. Fetch the page yourself when you genuinely need the full body.
Prerequisites
- Python 3.10+
- Scavio API key
Walkthrough
Step 1: Identify Google Custom Search calls
Usually with a CSE key + cx ID.
# Before:
# r = requests.get('https://www.googleapis.com/customsearch/v1', params={'key': KEY, 'cx': CX, 'q': q})Step 2: Replace with Scavio
No CX needed; Scavio searches the open web.
# After:
r = requests.post('https://api.scavio.dev/api/v2/google',
headers={'Authorization': f'Bearer {SCAVIO_API_KEY}'},
json={'query': q}).json()Step 3: Map response
items[] becomes organic_results[].
# Google CSE: r['items'][i]['link']
# Scavio: r['organic_results'][i]['link']Step 4: Add extract endpoint for content
Replaces the 'manual scraping' half of the OP's flow.
# Scavio returns structured search data, not page bodies: there is no extract
# or crawl endpoint. What you can get for a URL is the Google result row it
# already has - title, link and snippet. Fetch the page yourself when you need
# the full body.
def scavio_page_row(url, headers):
target = url.split("://")[-1].rstrip("/")
r = requests.post("https://api.scavio.dev/api/v2/google", headers=headers,
json={"query": "site:" + target}, timeout=30)
r.raise_for_status()
rows = r.json().get("organic_results", [])
return rows[0] if rows else {"title": "", "link": url, "snippet": ""}
def fetch(url):
return scavio_page_row(url, {'Authorization': f'Bearer {SCAVIO_API_KEY}'}).get('snippet', '')Step 5: Compare quotas
Google CSE caps at 100/day on free, $5/1K above. Scavio free is 50 credits on signup + $30/mo for 7K.
// Daily research agent making 50 queries: Google CSE = $7.50/mo above quota; Scavio = $30/mo flat once the 50 free signup credits are used.Python Example
# Migration takes ~20 minutes for a typical agent.JavaScript Example
// Same in TS.Expected Output
Same query intent, structured JSON, plus extract endpoint that replaces 'manual scraping' under the same key. No more two-vendor split.