The Problem
Your agent's web-search step hits a 429 rate limit or a hung timeout and the whole run stalls. This is the failure mode in the r/ClaudeWorkflows thread on debugging web search and content extraction in a Claude Code agent: the search-then-extract step times out or gets throttled, and there's no recovery path, so the agent dies mid-task. The two root causes are almost always the same. You ignore the Retry-After header on a 429, and you fetch raw HTML then parse it live instead of asking for structured JSON.
The Scavio Solution
Three fixes, in order of payoff. First, honor the Retry-After header on every 429. Most provider docs bury it, so engineers find out the hard way that the server already told them exactly how long to wait. Read the header, sleep that many seconds, and if it's absent fall back to exponential backoff with jitter so a fleet of agents doesn't retry in lockstep. Second, stop fetching HTML and parsing it at request time. Live-page parsing is where most extraction timeouts come from: a slow render, a redirect chain, a bot wall, and your headless browser hangs past the deadline. Scavio's POST https://api.scavio.dev/api/v1/google returns typed JSON in one call, with organic, people_also_ask, knowledge_graph, and related_searches fields already parsed server-side, no headless browser on your end. Third, set a hard per-request timeout and a fallback provider, so a single slow call can't block the run, and cache identical queries so a retry doesn't re-bill you. Honest limit: a structured SERP API doesn't help when your target is a JS-heavy single-page app or content behind a login. For those you still need a real browser or an authenticated session.
Before
Agent fetches raw HTML and parses it live with no backoff. One 429 or one slow render hangs the headless browser past the deadline, and the run dies with no recovery.
After
Agent calls a structured JSON SERP endpoint, honors Retry-After with exponential backoff plus jitter, enforces a hard timeout with a fallback provider, and caches identical queries so retries don't re-bill.
Who It Is For
Engineers building Claude Code, LangChain, or CrewAI agents whose web-search or extraction step keeps timing out or hitting 429s, and who want a structured JSON SERP call with proper backoff instead of a live HTML parse that hangs.
Key Benefits
- Retry-After plus jittered backoff turns a 429 into a short pause instead of a dead run
- Structured JSON from /api/v1/google removes the live HTML parse that causes most extraction timeouts
- No headless browser to babysit: organic, people_also_ask, and knowledge_graph come parsed in one response
- A hard per-request timeout plus a fallback provider stops one slow call from blocking the agent
- Query caching means retries hit the cache, not your credit balance
Python Example
import time, random, requests
API = "https://api.scavio.dev/api/v1/google"
HEADERS = {"Authorization": "Bearer YOUR_API_KEY"}
_cache = {}
def search(query, light_request=False, timeout=15, max_retries=5):
if query in _cache:
return _cache[query]
body = {"query": query, "light_request": light_request}
for attempt in range(max_retries):
try:
r = requests.post(API, json=body, headers=HEADERS, timeout=timeout)
except requests.Timeout:
time.sleep(min(2 ** attempt + random.random(), 30))
continue
if r.status_code == 429:
retry_after = r.headers.get("Retry-After")
wait = float(retry_after) if retry_after else min(2 ** attempt + random.random(), 30)
time.sleep(wait)
continue
r.raise_for_status()
data = r.json()
_cache[query] = data
return data
raise RuntimeError("search failed after retries")
result = search("agent web search timeout fix")
for item in result.get("organic", []):
print(item["title"], item["link"])JavaScript Example
const API = "https://api.scavio.dev/api/v1/google";
const HEADERS = { Authorization: "Bearer YOUR_API_KEY", "Content-Type": "application/json" };
const cache = new Map();
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
async function search(query, { lightRequest = false, timeout = 15000, maxRetries = 5 } = {}) {
if (cache.has(query)) return cache.get(query);
const body = JSON.stringify({ query, light_request: lightRequest });
for (let attempt = 0; attempt < maxRetries; attempt++) {
const ctrl = new AbortController();
const t = setTimeout(() => ctrl.abort(), timeout);
try {
const r = await fetch(API, { method: "POST", headers: HEADERS, body, signal: ctrl.signal });
clearTimeout(t);
if (r.status === 429) {
const retryAfter = r.headers.get("Retry-After");
const wait = retryAfter ? parseFloat(retryAfter) * 1000 : Math.min(2 ** attempt * 1000 + Math.random() * 1000, 30000);
await sleep(wait);
continue;
}
if (!r.ok) throw new Error(`HTTP ${r.status}`);
const data = await r.json();
cache.set(query, data);
return data;
} catch (e) {
clearTimeout(t);
await sleep(Math.min(2 ** attempt * 1000 + Math.random() * 1000, 30000));
}
}
throw new Error("search failed after retries");
}
const result = await search("agent web search timeout fix");
for (const item of result.organic ?? []) console.log(item.title, item.link);