To give a self-hosted LLM agent web search, connect it to an MCP server that returns structured results, then let the model call it as a tool. You have two real options. Self-host SearXNG plus a headless browser and own the whole anti-bot stack yourself: TLS fingerprints, Cloudflare managed challenges, CAPTCHA solving, and HTML-to-text extraction. Or point the agent at a hosted search API over MCP and get parsed JSON back without running a proxy pool. This tutorial wires the second path in about ten minutes using Scavio's remote MCP server at https://mcp.scavio.dev/mcp, and shows a plain-HTTP fallback for harnesses that don't speak MCP yet. SearXNG is genuinely fine when your targets are cooperative and you don't mind maintenance; the moment Google rate-limits your datacenter IP or Cloudflare throws a Turnstile challenge, you're maintaining infrastructure instead of building your agent. A managed endpoint moves that failure surface off your box: Google SERP, Reddit, YouTube, Amazon, Walmart, and TikTok all come back as typed JSON from one key. No Puppeteer, no markitdown pass, no CAPTCHA queue.
Prerequisites
- A Scavio API key (50 free credits on signup, no card) from scavio.dev
- An MCP-capable agent host: Claude Code, Cursor, VS Code, Windsurf, opencode, or a custom harness like Pi.dev
- Python 3.10+ or Node 18+ for the HTTP fallback path
- Basic familiarity with your agent's tool-calling config
Walkthrough
Step 1: Decide: self-hosted SearXNG or a hosted search MCP
SearXNG gives you a private meta-search you fully control, but you inherit every anti-bot problem: datacenter IPs get 403s, Cloudflare challenges need a full browser render, and you'll write your own HTML-to-text extraction. A hosted search MCP returns clean JSON and absorbs the fingerprinting and rate-limit fight. If you need fully local with no cloud calls, run SearXNG. If you want the agent working today, use the hosted MCP and spend your time on the agent logic instead of proxy rotation.
Step 2: Register the Scavio MCP server with your agent host
The remote MCP server lives at https://mcp.scavio.dev/mcp and authenticates with the x-api-key header (note: the REST API uses Authorization: Bearer, but the MCP transport uses x-api-key). Add it to your MCP client config. In Claude Code, Cursor, or any client that reads a .mcp.json, the block below registers every Scavio tool, including Google SERP, Reddit search, and the TikTok endpoints.
{
"mcpServers": {
"scavio": {
"type": "http",
"url": "https://mcp.scavio.dev/mcp",
"headers": {
"x-api-key": "${SCAVIO_API_KEY}"
}
}
}
}Step 3: Confirm the tools are exposed to the model
Restart the agent host so it handshakes with the MCP server. The model should now see search tools it can call on its own; a Google search returns organic results, and setting light_request to false additionally returns the knowledge graph, people-also-ask, and related searches in the same call. That richer call costs 2 credits instead of 1, which matters if your agent fans one question into dozens of searches.
Step 4: Add a plain-HTTP fallback for harnesses without MCP
If your harness (a custom Pi.dev loop, an OpenAI-function-calling script, a bare Ollama setup) doesn't speak MCP, register the endpoint as a normal tool. Every REST endpoint uses Authorization: Bearer. The Google endpoint replaces the SearXNG + browser + extraction chain with one POST that returns parsed JSON, so there's no HTML to clean and nothing to feed through markitdown.
Step 5: Handle the failure modes you were dreading
Cloudflare challenges, CAPTCHAs, and search-engine rate limits are handled server-side, so your agent never sees a Turnstile page. Public, indexed targets (Google SERP, Reddit threads, YouTube listings, Amazon and Walmart products) come back structured. What a search API cannot do: pages behind a login, or a specific site that only renders after heavy JavaScript and isn't in any index. For those you still need a real browser. Route the public/indexed subset through the API and reserve your headless browser for the handful of targets that genuinely require it.
Python Example
import os, requests
HEADERS = {
"Authorization": f"Bearer {os.environ['SCAVIO_API_KEY']}",
"Content-Type": "application/json",
}
def web_search(query: str, rich: bool = False) -> dict:
"""Tool the agent can call. rich=True also returns
knowledge_graph + people_also_ask (2 credits vs 1)."""
r = requests.post(
"https://api.scavio.dev/api/v1/google",
headers=HEADERS,
json={"query": query, "light_request": not rich},
timeout=30,
)
r.raise_for_status()
data = r.json()
# Feed only the useful text to the LLM, no HTML cleanup needed
return {
"organic": [
{"title": o["title"], "link": o["link"], "snippet": o.get("snippet", "")}
for o in data.get("organic", [])[:8]
],
"answer_box": data.get("knowledge_graph") if rich else None,
}
if __name__ == "__main__":
print(web_search("self-hosted vector database comparison", rich=True))JavaScript Example
const HEADERS = {
Authorization: `Bearer ${process.env.SCAVIO_API_KEY}`,
"Content-Type": "application/json",
};
async function webSearch(query, rich = false) {
const res = await fetch("https://api.scavio.dev/api/v1/google", {
method: "POST",
headers: HEADERS,
body: JSON.stringify({ query, light_request: !rich }),
});
if (!res.ok) throw new Error(`Scavio ${res.status}`);
const data = await res.json();
return {
organic: (data.organic || []).slice(0, 8).map((o) => ({
title: o.title,
link: o.link,
snippet: o.snippet ?? "",
})),
answerBox: rich ? data.knowledge_graph ?? null : null,
};
}
webSearch("self-hosted vector database comparison", true).then(console.log);Expected Output
A JSON object your agent can drop straight into its context window: an `organic` array of up to 8 results, each with `title`, `link`, and `snippet`, plus an optional `answer_box` populated from the knowledge graph when you pass rich=True. No HTML, no boilerplate stripping, no CAPTCHA page. A basic Google call costs 1 credit ($0.005 on pay-as-you-go); the rich call with knowledge graph and people-also-ask costs 2 credits. The 50 free signup credits are enough to wire this in and run about 25 rich searches before you decide whether to keep SearXNG around for the fully-local cases.