Resumen
Search-as-source flujo de trabajo for building un 10M-token RAG corpus de indexed public contenido. Avoids mas scraper pain.
Desencadenador
Per topic construir (one-shot o quarterly refresh)
Programación
Per topic (one-shot o quarterly)
Pasos del flujo de trabajo
1
Define 200-500 seed consultas covering el topic
Topical breadth > depth on individual consultas.
2
Scavio Google SERP per seed
Recopilar organic_results URLs.
3
Deduplicate URL establecer
Many seeds surface el same authoritative paginas.
4
Scavio /extraer on top-2K URLs
Returns limpiar Markdown text.
5
Token-budget trim
Stop at 10M tokens; prefer URLs con higher autoridad de dominio.
6
Embed y ship to vector almacenar
Per your existing RAG embedding pipeline.
Implementacion en Python
Python
import requests, os
H = {'x-api-key': os.environ['SCAVIO_API_KEY']}
def build_corpus(seeds, per_query=10):
urls = set()
for q in seeds:
r = requests.post('https://api.scavio.dev/api/v1/search', headers=H, json={'query': q}).json()
for o in (r.get('organic_results') or [])[:per_query]:
urls.add(o['link'])
docs = []
for u in list(urls)[:2000]:
d = requests.post('https://api.scavio.dev/api/v1/extract', headers=H, json={'url': u}).json()
if d.get('text'): docs.append(d['text'])
return docsImplementacion en JavaScript
JavaScript
// Same shape in TS — search per seed, dedupe, extract top-N.Plataformas utilizadas
Búsqueda web con grafo de conocimiento, PAA y resúmenes de IA