Resumen
Ejecuta un semanal success-rate referencia contra N representative objetivo sitios. Cada sitio obtiene un Scavio solicitud y un solicitud via your own scraper; ambos attempts obtener logged con estado, block indicators, y latencia. Produces un informe semanal showing your scraper's tasa de exito vs Scavio's linea base so you know cuando to outsource un objetivo.
Desencadenador
Cron programar (semanal on Sunday at 2 AM UTC)
Programación
Semanal on Sundays at 2 AM UTC
Pasos del flujo de trabajo
Cargar objetivo sitio lista
10 to 100 representative sitios your team extrae regularly.
Scavio linea base solicitud
Call Scavio sitio:dominio for cada objetivo y registro success/fail.
Own-scraper probar solicitud
Ejecutar el same consulta via your internal scraper y registro el outcome.
Compute delta
Per sitio, compute (scavio_success - own_success) y marcar negative valores atipicos.
Persist to warehouse
Escribir semanal fila to BigQuery con {sitio, scavio_ok, own_ok, latency_ms, date}.
Email informe semanal
Enviar CSV resumen to engineering lead cada Sunday noche.
Implementacion en Python
import os, requests, time
API_KEY = os.environ["SCAVIO_API_KEY"]
H = {"x-api-key": API_KEY}
SITES = ["cnn.com", "walmart.com", "reddit.com"]
def scavio_probe(site):
r = requests.post("https://api.scavio.dev/api/v1/search",
headers=H, json={"query": f"site:{site}"}, timeout=15)
return r.ok and len(r.json().get("organic_results", [])) > 0
def own_probe(site):
try:
return requests.get(f"https://{site}", timeout=10).ok
except: return False
for s in SITES:
print(s, scavio_probe(s), own_probe(s))Implementacion en JavaScript
const API_KEY = process.env.SCAVIO_API_KEY;
const H = { "x-api-key": API_KEY, "content-type": "application/json" };
const SITES = ["cnn.com", "walmart.com", "reddit.com"];
async function scavioProbe(site) {
const r = await fetch("https://api.scavio.dev/api/v1/search", {
method: "POST", headers: H, body: JSON.stringify({ query: "site:" + site })
});
if (!r.ok) return false;
return ((await r.json()).organic_results || []).length > 0;
}
async function ownProbe(site) {
try { const r = await fetch("https://" + site, { signal: AbortSignal.timeout(10000) }); return r.ok; }
catch { return false; }
}
for (const s of SITES) console.log(s, await scavioProbe(s), await ownProbe(s));Plataformas utilizadas
Búsqueda web con grafo de conocimiento, PAA y resúmenes de IA