Resumen
Asynchronous bid-discovery pipeline. Cron at dawn ejecuta Google Dorks a traves de government dominios, extrae PDFs, converts to typed JSON, caches in SQLite. CrewAI agent lee de cache.
Desencadenador
Diario 4 AM
Programación
Diario 4 AM
Pasos del flujo de trabajo
1
Cron at 4 AM
Ejecutar pipeline antes de business horas.
2
Google Dorks a traves de objetivo dominios
sitio:gov.br filetype:pdf 2026, sitio:europa.eu filetype:pdf, etc.
3
Filtrar for fresh PDFs
Date filtrar o LLM screening.
4
Scavio extraer per PDF
PDF-aware extraer returns markdown.
5
LLM converts to typed JSON
Strict-esquema prompt.
6
Cache in SQLite
Sub-50ms repeat lookups.
7
CrewAI agent consultas cache
Fresh bids disponible sin latencia.
Implementacion en Python
Python
import os, requests, sqlite3, json, time
API_KEY = os.environ['SCAVIO_API_KEY']
H = {'x-api-key': API_KEY}
DORKS = ['site:gov.br filetype:pdf 2026 contratos', 'site:europa.eu filetype:pdf AI act']
conn = sqlite3.connect('bids.db')
conn.execute('CREATE TABLE IF NOT EXISTS bids(url TEXT PRIMARY KEY, payload TEXT, ts REAL)')
def crawl():
for q in DORKS:
r = requests.post('https://api.scavio.dev/api/v1/search', headers=H, json={'query': q}).json()
for o in r.get('organic_results', []):
if o.get('link', '').endswith('.pdf'):
e = requests.post('https://api.scavio.dev/api/v1/extract', headers=H, json={'url': o['link'], 'format': 'markdown'}).json()
conn.execute('INSERT OR REPLACE INTO bids VALUES (?, ?, ?)', (o['link'], json.dumps(e), time.time()))
conn.commit()Implementacion en JavaScript
JavaScript
const H = { 'x-api-key': process.env.SCAVIO_API_KEY, 'Content-Type': 'application/json' };
const DORKS = ['site:gov.br filetype:pdf 2026 contratos'];
async function crawl() {
for (const q of DORKS) {
const r = await fetch('https://api.scavio.dev/api/v1/search', { method: 'POST', headers: H, body: JSON.stringify({ query: q }) }).then(r => r.json());
// Per result, fetch extract and write to cache.
}
}Plataformas utilizadas
Búsqueda web con grafo de conocimiento, PAA y resúmenes de IA