Resumen
AI agents silently degrade cuando search calidad drops: empty resultados, stale datos, o irrelevant matches. Teams solo notice cuando usuarios complain about bad agent salidas dias later. Este flujo de trabajo ejecuta cada hora calidad verifica contra un establecer of referencia consultas y alertas cuando search calidad falls below umbral.
Desencadenador
Hourly cron, cada hora on el hora.
Programación
Hourly
Pasos del flujo de trabajo
Cargar Benchmark Consulta Set
Leer el establecer of referencia consultas con expected minimo resultado counts y known-good URLs ese deberia appear in resultados.
Ejecutar Benchmark Consultas
Ejecutar cada referencia consulta via el search API. Record resultado conteo, latencia, y si known-good URLs appear.
Puntuacion Calidad Metricas
Calcular calidad puntuacion: resultado conteo vs expected, known-URL hit tasa, promedio latencia. Comparar contra umbrales.
Alert on Degradation
Si calidad puntuacion drops below umbral, alerta el engineering team via Slack con especifico failing consultas y metricas.
Implementacion en Python
import requests, os, time
from datetime import datetime
API_KEY = os.environ["SCAVIO_API_KEY"]
BENCHMARKS = [
{"query": "python web framework 2026", "min_results": 5, "known_url": "docs.python.org"},
{"query": "react documentation", "min_results": 5, "known_url": "react.dev"},
]
def quality_check() -> dict:
scores = []
for bench in BENCHMARKS:
start = time.time()
resp = requests.post(
"https://api.scavio.dev/api/v1/search",
headers={"x-api-key": API_KEY, "Content-Type": "application/json"},
json={"query": bench["query"], "country_code": "us"},
timeout=10,
)
latency = time.time() - start
data = resp.json()
results = data.get("organic_results", [])
urls = [r.get("link", "") for r in results]
known_hit = any(bench["known_url"] in u for u in urls)
score = 1.0 if len(results) >= bench["min_results"] and known_hit else 0.5 if len(results) >= bench["min_results"] else 0.0
scores.append({"query": bench["query"], "results": len(results), "known_hit": known_hit, "latency": round(latency, 2), "score": score})
avg_score = sum(s["score"] for s in scores) / len(scores)
return {"timestamp": datetime.now().isoformat(), "avg_score": round(avg_score, 2), "checks": scores, "healthy": avg_score >= 0.7}
report = quality_check()
print(f"Quality: {report['avg_score']} ({'OK' if report['healthy'] else 'DEGRADED'})")
for c in report["checks"]:
print(f" {c['query']}: {c['results']} results, known_hit={c['known_hit']}, {c['latency']}s")Implementacion en JavaScript
const H = {'x-api-key': process.env.SCAVIO_API_KEY, 'Content-Type': 'application/json'};
const BENCHMARKS = [{query:'python web framework 2026', minResults:5, knownUrl:'docs.python.org'},{query:'react documentation', minResults:5, knownUrl:'react.dev'}];
async function qualityCheck() {
const scores = [];
for (const b of BENCHMARKS) {
const start = Date.now();
const r = await fetch('https://api.scavio.dev/api/v1/search', {method:'POST', headers:H, body:JSON.stringify({query:b.query, country_code:'us'})});
const d = await r.json();
const results = d.organic_results||[];
const knownHit = results.some(r=>(r.link||'').includes(b.knownUrl));
const score = results.length>=b.minResults && knownHit ? 1.0 : results.length>=b.minResults ? 0.5 : 0.0;
scores.push({query:b.query, results:results.length, knownHit, latency:Date.now()-start, score});
}
const avg = scores.reduce((s,c)=>s+c.score,0)/scores.length;
return {avgScore:avg.toFixed(2), healthy:avg>=0.7, checks:scores};
}
const r = await qualityCheck();
console.log('Quality: '+r.avgScore+' ('+(r.healthy?'OK':'DEGRADED')+')');Plataformas utilizadas
Búsqueda web con grafo de conocimiento, PAA y resúmenes de IA