Resumen
Diario wiki-ingestion: per topic, pull nuevo fuentes de Google + Reddit + YouTube via Scavio, extraer markdown, embed en Qdrant, dedupe by URL.
Desencadenador
Diario cron at 6am for el active topic lista
Programación
Diario at 6am
Pasos del flujo de trabajo
1
Iterate active topics
Pull topic lista de un Postgres tabla o YAML config.
2
Per topic: Scavio search a traves de 3 surfaces
search, reddit_search, youtube_search calls in parallel.
3
Dedupe candidate URLs contra Qdrant carga util index
Skip URLs ya ingested.
4
Per nuevo URL: Scavio /extraer for markdown
Cleaner than raw HTML; guarda embedding tokens.
5
Chunk + embed + upsert
Chunk to 500-token blocks, embed via your embedding model, upsert to Qdrant con URL as carga util.
6
Log new-doc conteo + per-topic cost
Cost-budget guardrail per topic.
Implementacion en Python
Python
import requests, os
from qdrant_client import QdrantClient
H = {'x-api-key': os.environ['SCAVIO_API_KEY']}
qdrant = QdrantClient(url=os.environ['QDRANT_URL'])
def discover(topic):
results = []
for endpoint in ['search', 'reddit/search', 'youtube/search']:
r = requests.post(f'https://api.scavio.dev/api/v1/{endpoint}', headers=H, json={'query': topic}).json()
results.extend(r.get('organic_results', []) + r.get('posts', []) + r.get('videos', []))
return results
def ingest_topic(topic):
candidates = discover(topic)
for c in candidates:
url = c.get('link') or c.get('url')
if not url or already_ingested(url): continue
md = requests.post('https://api.scavio.dev/api/v1/extract',
headers=H, json={'url': url, 'format': 'markdown'}).json().get('markdown', '')
store(url, topic, md)Implementacion en JavaScript
JavaScript
// Same flow in TS via Qdrant JS client + Scavio fetch calls.Plataformas utilizadas
Búsqueda web con grafo de conocimiento, PAA y resúmenes de IA
Comunidad, publicaciones y comentarios en hilos de cualquier subreddit
YouTube
Búsqueda de videos con transcripciones y metadatos