An r/n8n thread complained that search APIs return raw HTML breaking token limits or strip too much context. This tutorial walks the middle path: structured snippets via Scavio, full-page extracts only for the top 1-2 hits.
Prerequisites
- Python 3.10+
- Scavio API key
Walkthrough
Step 1: Search returns 10 typed snippets
Each snippet fits in ~100 tokens.
Python
import requests, os
API_KEY = os.environ['SCAVIO_API_KEY']
def snippets(q):
r = requests.post('https://api.scavio.dev/api/v1/search',
headers={'x-api-key': API_KEY}, json={'query': q}).json()
return r.get('organic_results', [])[:10]Step 2: LLM picks top 1-2 to read fully
Cheaper than fetching all 10.
Python
import anthropic
client = anthropic.Anthropic()
def pick(q, snips):
msg = client.messages.create(model='claude-sonnet-4-6', max_tokens=200,
messages=[{'role':'user','content':f'Q: {q}. SNIPPETS: {snips}. Return indices of the top 2 to read fully.'}])
return msg.content[0].textStep 3: Extract those pages as markdown
Markdown is cheaper tokens than HTML.
Python
def fetch(url):
r = requests.post('https://api.scavio.dev/api/v1/extract',
headers={'x-api-key': API_KEY}, json={'url': url, 'format': 'markdown'}).json()
return r.get('markdown', '')[:5000] # token-budget the pageStep 4: Compose final answer
Snippets give breadth, full pages give depth.
Python
def answer(q):
snips = snippets(q)
picks = [int(i) for i in pick(q, snips).split(',') if i.strip().isdigit()]
deep = [fetch(snips[i]['link']) for i in picks[:2]]
return {'snippets': snips, 'deep_reads': deep}Step 5: Token math
10 snippets ≈ 1K tokens; 2 trimmed pages ≈ 8K tokens; total context ≈ 9K tokens — fits in any 200K-context model.
Text
// Token budget: well under 16K even for a 32K-context model.Python Example
Python
# Per question: 1 search + 2 extracts = 3 credits = $0.013. Plus LLM token cost.JavaScript Example
JavaScript
// Same pattern in TS.Expected Output
JSON
Per question, the agent has 10 snippets and 2 full reads in its context. No raw HTML, no manual cleaning, no token overflow.