Historical SERP Competitive Analysis
Track how competitor rankings change over weeks and months. Daily SERP snapshots stored in PostgreSQL reveal content strategies and algorithm impact.
Weekly SERP snapshots stored as JSON create a competitive intelligence archive that no subscription tool provides out of the box. By comparing snapshots over time, you can detect when competitors gain or lose SERP features, when AI Overviews appear for your keywords, and which content changes correlate with ranking improvements.
Why Historical Data Matters
Current SERP data tells you where you rank today. Historical data tells you trends: is a competitor steadily climbing? Did an AI Overview citation change after a content update? Are featured snippets shifting between domains? These patterns are invisible in point-in-time snapshots and require time-series data to detect.
Enterprise tools like Semrush and Ahrefs track rank history, but their data is proprietary and expensive. Building your own archive from raw SERP data gives you full control over what you track, how long you retain it, and what custom metrics you compute.
Building a SERP Archive
import os, json, requests
from datetime import date
API_KEY = os.environ["SCAVIO_API_KEY"]
H = {"x-api-key": API_KEY, "Content-Type": "application/json"}
def snapshot_keywords(keywords, output_dir="serp_archive"):
os.makedirs(output_dir, exist_ok=True)
today = str(date.today())
snapshot = {"date": today, "keywords": {}}
for kw in keywords:
res = requests.post("https://api.scavio.dev/api/v1/search",
headers=H, json={"query": kw, "country_code": "us",
"include_ai_overview": True})
data = res.json()
snapshot["keywords"][kw] = {
"organic": [{
"position": i + 1,
"domain": r.get("link", "").split("/")[2] if r.get("link") else "",
"title": r.get("title", "")[:100],
} for i, r in enumerate(data.get("organic_results", [])[:10])],
"ai_overview": bool(data.get("ai_overview")),
"ai_overview_domains": [
c.get("url", "").split("/")[2]
for c in data.get("ai_overview", {}).get("citations", [])
if c.get("url")
],
"paa_count": len(data.get("people_also_ask", [])),
"local_pack": len(data.get("local_results", [])) > 0,
}
filepath = os.path.join(output_dir, f"serp_{today}.json")
with open(filepath, "w") as f:
json.dump(snapshot, f, indent=2)
print(f"Saved {len(keywords)} keywords to {filepath}")
print(f"Cost: ${len(keywords) * 0.005:.2f}")
return snapshot
keywords = [
"best SERP API 2026",
"web scraping vs API",
"TikTok analytics tools",
"n8n enrichment workflow",
"AI Overview optimization",
]
snapshot_keywords(keywords)Comparing Snapshots Over Time
import json, glob
def load_snapshots(archive_dir="serp_archive"):
files = sorted(glob.glob(f"{archive_dir}/serp_*.json"))
return [json.load(open(f)) for f in files]
def detect_changes(snapshots, keyword, my_domain):
changes = []
for i in range(1, len(snapshots)):
prev = snapshots[i-1]["keywords"].get(keyword, {})
curr = snapshots[i]["keywords"].get(keyword, {})
prev_rank = next((o["position"] for o in prev.get("organic", [])
if my_domain in o["domain"]), None)
curr_rank = next((o["position"] for o in curr.get("organic", [])
if my_domain in o["domain"]), None)
prev_aio = my_domain in [d for d in prev.get("ai_overview_domains", [])]
curr_aio = my_domain in [d for d in curr.get("ai_overview_domains", [])]
if prev_rank != curr_rank or prev_aio != curr_aio:
changes.append({
"date": snapshots[i]["date"],
"rank_change": f"#{prev_rank} -> #{curr_rank}" if prev_rank and curr_rank
else f"{'Entered' if curr_rank else 'Dropped'} at #{curr_rank or prev_rank}",
"aio_change": f"{'Gained' if curr_aio and not prev_aio else 'Lost' if prev_aio and not curr_aio else 'No change'} AIO citation",
})
return changes
snapshots = load_snapshots()
if len(snapshots) >= 2:
changes = detect_changes(snapshots, "best SERP API 2026", "scavio.dev")
for c in changes:
print(f" {c['date']}: {c['rank_change']} | {c['aio_change']}")Storage and Cost
A weekly snapshot of 100 keywords costs 100 credits = $0.50/week = $2/month. Each snapshot is roughly 50-100KB of JSON. A year of weekly snapshots for 100 keywords takes about 5MB of storage. SQLite or even flat JSON files work fine at this scale.
For higher volume (1,000+ keywords), DataForSEO's standard queue at $0.0006/query reduces the cost to $0.60/week. The queue mode adds a 5-minute delay, which is irrelevant for weekly archival workloads. DataForSEO also offers a dedicated "SERP Archive" product with pre-stored historical data, though its coverage varies by keyword.
Actionable Patterns to Watch
- Competitor enters top 5 for the first time: review their content to understand what changed
- AI Overview appears on a keyword you rank well for: optimize for citation (answer-first structure)
- Your AI Overview citation disappears: competitor published better answer-first content
- PAA count increases on your keyword: Google sees ambiguity, create content addressing each PAA question
- Local pack appears on a previously non-local keyword: Google is localizing the query, adjust strategy