Resumen
Datos scientists lose horas busqueda for conjuntos de datos a traves de fragmented plataformas. Este flujo de trabajo usa Scavio's MCP servidor to search a traves de Kaggle, HuggingFace, datos.gov, y Google Dataset Search de un single agent connection. Resultados son deduplicated, scored by relevance y freshness, y formatted for notebook importar.
Desencadenador
On-demand cuando un datos scientist necesita conjuntos de datos for un nuevo project.
Programación
On-demand
Pasos del flujo de trabajo
Define Dataset Requirements
Specify el topic, requerido columnas, minimo tamano, preferred formato, y freshness requisitos.
Search Across Plataformas via MCP
Call Scavio MCP herramientas/call con site-specific consultas for Kaggle, HuggingFace, datos.gov, y Google Dataset Search.
Analizar y Normalize Resultados
Extraer conjunto de datos titles, URLs, descriptions, y fuente plataformas de resultados de busqueda.
Deduplicate y Puntuacion
Eliminar duplicate conjuntos de datos ese appear on multiples plataformas. Puntuacion by relevance to requisitos.
Salida Discovery Informe
Format resultados as un markdown informe o JSON file con descargar enlaces y metadata.
Implementacion en Python
import requests, os, json
API_KEY = os.environ["SCAVIO_API_KEY"]
H = {"x-api-key": API_KEY, "Content-Type": "application/json"}
PLATFORMS = {
"kaggle": "site:kaggle.com/datasets",
"huggingface": "site:huggingface.co/datasets",
"data_gov": "site:data.gov",
"google_datasets": "site:datasetsearch.research.google.com",
}
def discover_datasets(topic: str, platforms: list = None) -> list:
targets = {k: v for k, v in PLATFORMS.items() if not platforms or k in platforms}
all_datasets = []
for source, site_filter in targets.items():
resp = requests.post(
"https://api.scavio.dev/api/v1/search",
headers=H,
json={"query": f"{topic} dataset {site_filter}", "country_code": "us"},
timeout=10,
)
for r in resp.json().get("organic_results", []):
all_datasets.append({
"title": r.get("title", ""),
"url": r.get("link", ""),
"source": source,
"snippet": r.get("snippet", ""),
"date": r.get("date", ""),
})
# Deduplicate
seen = set()
unique = []
for d in all_datasets:
if d["url"] not in seen:
seen.add(d["url"])
unique.append(d)
return unique
results = discover_datasets("air quality monitoring sensor data")
print(f"Found {len(results)} datasets across {len(set(d['source'] for d in results))} platforms")
for d in results[:8]:
print(f" [{d['source']}] {d['title']}: {d['url']}")Implementacion en JavaScript
const H = {'x-api-key': process.env.SCAVIO_API_KEY, 'Content-Type': 'application/json'};
const PLATFORMS = {kaggle:'site:kaggle.com/datasets', huggingface:'site:huggingface.co/datasets', data_gov:'site:data.gov', google_datasets:'site:datasetsearch.research.google.com'};
async function discoverDatasets(topic, platforms) {
const targets = platforms ? Object.fromEntries(Object.entries(PLATFORMS).filter(([k])=>platforms.includes(k))) : PLATFORMS;
const all = [];
for (const [source, siteFilter] of Object.entries(targets)) {
const r = await fetch('https://api.scavio.dev/api/v1/search', {method:'POST', headers:H, body:JSON.stringify({query:topic+' dataset '+siteFilter, country_code:'us'})});
for (const o of (await r.json()).organic_results||[]) {
all.push({title:o.title||'', url:o.link||'', source, snippet:o.snippet||'', date:o.date||''});
}
}
const seen = new Set();
return all.filter(d=>{ if (seen.has(d.url)) return false; seen.add(d.url); return true; });
}
const results = await discoverDatasets('air quality monitoring sensor data');
console.log('Found '+results.length+' datasets across '+new Set(results.map(d=>d.source)).size+' platforms');
for (const d of results.slice(0,8)) console.log(' ['+d.source+'] '+d.title+': '+d.url);Plataformas utilizadas
Búsqueda web con grafo de conocimiento, PAA y resúmenes de IA