Overview
Staying current on 10+ topics means checking dozens of sources daily. This workflow automates it: define your topics, run daily searches, deduplicate against your existing knowledge base, and store only genuinely new information. Pairs with a local LLM for summarization so your private research stays local. Cost: under $1/month for 20 topics.
Trigger
Daily at 7 AM via cron.
Schedule
Daily 7 AM
Workflow Steps
Load Topic Watchlist
Read topics from a local config file. Each topic has a search query and a category.
Search for Each Topic
Run Scavio search for each topic. Extract top results with titles, snippets, and URLs.
Deduplicate Against Knowledge Base
Compare new result URLs against existing knowledge base entries. Keep only genuinely new items.
Summarize New Entries
For each new entry, create a structured summary with source, date, and key facts.
Append to Knowledge Base
Add new entries to the local knowledge base file with timestamps and categories.
Python Implementation
import requests, os, json
from pathlib import Path
from datetime import date
API_KEY = os.environ["SCAVIO_API_KEY"]
H = {"x-api-key": API_KEY, "Content-Type": "application/json"}
KB_FILE = Path("knowledge_base.json")
TOPICS = [
{"query": "ai agent framework news 2026", "category": "ai"},
{"query": "search api updates 2026", "category": "search"},
{"query": "python 3.14 release notes", "category": "python"},
]
def search_topic(query: str) -> list:
resp = requests.post(
"https://api.scavio.dev/api/v1/search",
headers=H,
json={"query": query, "country_code": "us"},
timeout=15,
)
results = resp.json().get("organic_results", [])[:5]
return [{"title": r.get("title", ""), "url": r.get("link", ""), "snippet": r.get("snippet", "")} for r in results]
def update_kb():
kb = json.loads(KB_FILE.read_text()) if KB_FILE.exists() else {"entries": [], "seen_urls": []}
seen = set(kb["seen_urls"])
new_entries = []
for topic in TOPICS:
results = search_topic(topic["query"])
for r in results:
if r["url"] not in seen:
entry = {
"date": str(date.today()),
"category": topic["category"],
"title": r["title"],
"url": r["url"],
"summary": r["snippet"],
}
new_entries.append(entry)
seen.add(r["url"])
kb["entries"].extend(new_entries)
kb["seen_urls"] = list(seen)
KB_FILE.write_text(json.dumps(kb, indent=2))
return len(new_entries)
added = update_kb()
print(f"Added {added} new entries to knowledge base")JavaScript Implementation
const H = {'x-api-key': process.env.SCAVIO_API_KEY, 'Content-Type': 'application/json'};
const fs = await import('fs');
const KB_FILE = 'knowledge_base.json';
const TOPICS = [
{query:'ai agent framework news 2026', category:'ai'},
{query:'search api updates 2026', category:'search'},
{query:'python 3.14 release notes', category:'python'},
];
async function searchTopic(query) {
const r = await fetch('https://api.scavio.dev/api/v1/search', {method:'POST', headers:H, body:JSON.stringify({query, country_code:'us'})});
return ((await r.json()).organic_results || []).slice(0,5).map(r=>({title:r.title||'', url:r.link||'', snippet:r.snippet||''}));
}
async function updateKb() {
let kb = {entries:[], seenUrls:[]};
try { kb = JSON.parse(fs.readFileSync(KB_FILE, 'utf8')); } catch {}
const seen = new Set(kb.seenUrls);
const newEntries = [];
for (const topic of TOPICS) {
const results = await searchTopic(topic.query);
for (const r of results) {
if (!seen.has(r.url)) {
newEntries.push({date:new Date().toISOString().split('T')[0], category:topic.category, title:r.title, url:r.url, summary:r.snippet});
seen.add(r.url);
}
}
}
kb.entries.push(...newEntries);
kb.seenUrls = [...seen];
fs.writeFileSync(KB_FILE, JSON.stringify(kb, null, 2));
return newEntries.length;
}
const added = await updateKb();
console.log('Added '+added+' new entries to knowledge base');Platforms Used
Web search with knowledge graph, PAA, and AI overviews