SEO API Pipeline Reliability: The Semrush Problem

Semrush API pipelines break in production because the Business plan ($499.95/mo minimum) gates API access, API unit quotas are opaque, and rate limits change without notice. Teams running nightly rank tracking or content audits at scale hit reliability walls that cheaper structured SERP APIs avoid entirely.

Where Semrush pipelines fail

Three failure modes dominate: API unit exhaustion mid-cycle (you get a 429 with no retry-after header), authentication token expiry on long-running jobs, and response schema changes that silently break parsing. The API unit system charges differently per endpoint -- a domain overview costs 10 units while a keyword overview costs 10-40 units depending on database. At ~$50 per million API units on top of the $499.95 base, a 50K-keyword daily pipeline can burn through budget in a week.

Building a reliable alternative pipeline

A structured SERP API returns the same JSON schema regardless of query volume. No unit accounting, no tiered endpoint pricing. Scavio charges $0.005/credit flat across all search types including Google, Maps, Shopping, and News.

import os, requests, time, json
from datetime import datetime

API_KEY = os.environ["SCAVIO_API_KEY"]
HEADERS = {"x-api-key": API_KEY}
BASE = "https://api.scavio.dev/api/v1/search"

def reliable_serp_fetch(keywords, max_retries=3):
    results = []
    for kw in keywords:
        for attempt in range(max_retries):
            try:
                resp = requests.post(BASE, headers=HEADERS, json={
                    "query": kw,
                    "num_results": 10,
                    "include_ai_overview": True,
                }, timeout=30)
                resp.raise_for_status()
                data = resp.json()
                results.append({
                    "keyword": kw,
                    "timestamp": datetime.utcnow().isoformat(),
                    "organic": data.get("organic_results", []),
                    "ai_overview": data.get("ai_overview"),
                })
                break
            except requests.exceptions.RequestException as e:
                if attempt == max_retries - 1:
                    results.append({"keyword": kw, "error": str(e)})
                time.sleep(2 ** attempt)
    return results

# 1,000 keywords/day = 1,000 credits = $5/day
keywords = ["saas seo tools", "rank tracking api", "serp monitoring"]
data = reliable_serp_fetch(keywords)
print(json.dumps(data[0], indent=2))

Cost comparison at scale

For a 10K keyword daily pipeline (300K queries/month):

• Semrush API: $499.95/mo base + ~$150 in API units = ~$650/mo
• SerpAPI: $275/mo for 30K searches (not enough -- need custom plan)
• Scavio: 300K credits at $0.005 = $1,500/mo on pay-as-you-go, or $250/mo plan (85K credits) + overage
• DataForSEO live: 300K at $0.002 = $600/mo

Pipeline reliability checklist

Idempotent retries with exponential backoff. Store raw API responses before parsing so schema changes do not lose data. Monitor credit balance via dashboard, not by counting locally. Run keyword batches in chunks of 100 with 1-second gaps to avoid burst throttling on any provider.

import time

def chunked_fetch(keywords, chunk_size=100, delay=1.0):
    all_results = []
    for i in range(0, len(keywords), chunk_size):
        chunk = keywords[i:i + chunk_size]
        results = reliable_serp_fetch(chunk)
        all_results.extend(results)
        if i + chunk_size < len(keywords):
            time.sleep(delay)
    return all_results

# 10K keywords in ~100 chunks, ~2 min total
daily_data = chunked_fetch(large_keyword_list)

When Semrush API still makes sense

If you need Semrush-proprietary metrics (keyword difficulty score, traffic estimates, backlink data), no SERP API replaces that. The reliability problem is specifically about using Semrush as a SERP data source -- for that use case, a dedicated SERP API is more predictable in pricing, uptime, and schema stability.

Key takeaway

Separate your SERP data pipeline from your SEO analytics platform. Use Semrush for proprietary metrics and a structured SERP API for raw ranking data. This eliminates the single-vendor reliability risk that breaks overnight pipelines.