ScrapingRobot broke against anti-bot defenses. Sonar API credits got wiped. SerpAPI has an active lawsuit. No single API is infallible. This tutorial builds automatic failover across multiple vendors.
Prerequisites
- API keys for 2-3 SERP APIs
- Python 3.8+
Walkthrough
Step 1: Define the vendor stack
Primary, fallback, and emergency vendors.
Python
vendors = [
{'name': 'scavio', 'url': 'https://api.scavio.dev/api/v1/search',
'headers': {'x-api-key': os.environ['SCAVIO_API_KEY']},
'payload_fn': lambda q: {'platform': 'google', 'query': q}},
{'name': 'serper', 'url': 'https://google.serper.dev/search',
'headers': {'X-API-KEY': os.environ['SERPER_API_KEY']},
'payload_fn': lambda q: {'q': q}},
]Step 2: Build the failover function
Try vendors in order, fail to next on error.
Python
import requests
def search_with_failover(query):
for vendor in vendors:
try:
r = requests.post(vendor['url'], headers=vendor['headers'],
json=vendor['payload_fn'](query), timeout=15)
if r.status_code == 200:
return {'vendor': vendor['name'], 'data': r.json()}
except Exception:
continue
raise Exception('All vendors failed')Step 3: Normalize responses
Each vendor returns different JSON shapes. Normalize.
Python
def normalize(vendor_name, data):
if vendor_name == 'scavio':
return [{'title': r['title'], 'url': r['link'], 'snippet': r['snippet']}
for r in data.get('organic_results', [])]
elif vendor_name == 'serper':
return [{'title': r['title'], 'url': r['link'], 'snippet': r['snippet']}
for r in data.get('organic', [])]
return dataStep 4: Log vendor performance
Track which vendors succeed and fail over time.
Python
import json, datetime
def log_result(vendor, success, latency):
with open('failover_log.jsonl', 'a') as f:
f.write(json.dumps({'ts': datetime.datetime.now().isoformat(),
'vendor': vendor, 'success': success, 'latency': latency}) + '\n')Python Example
Python
# Production failover: Scavio (primary) → Serper (fallback)
# Normalize output shape across vendors
# Log performance for quarterly vendor reviewJavaScript Example
JavaScript
// Same failover pattern in JS with try/catch per vendor.Expected Output
JSON
Multi-vendor SERP failover stack: primary → fallback → emergency. Normalized output, performance logging, vendor-neutral application code.