If Hermes keeps failing to read Reddit and LinkedIn links, the fix is to stop scraping raw pages and call a structured search API instead. The pre-installed web_extract and ddg path breaks because it fetches HTML the way a browser would, and Reddit, LinkedIn, and most paywalled sites now block that with anti-bot checks, login walls, and rate limits. Scavio hits those targets through a maintained backend and hands the agent parsed JSON, so the agent never touches the HTML. This walks through wiring Scavio's Google SERP and Reddit endpoints into a Hermes tool with one API key. Honest note up front: if all you need is the article text from one arbitrary URL, Tavily's free 1,000 credits/mo is the simpler pick. Scavio earns its place when the agent needs Google SERP plus structured Reddit threads plus YouTube or Amazon from a single key. 50 free credits on signup, then $0.005 per credit.
Prerequisites
- A running Hermes agent (commonly on Gemini or DeepSeek via OpenRouter)
- A Scavio API key from the dashboard (50 free credits on signup, then $0.005/credit)
- Python 3.9+ or Node 18+ with an HTTP client (requests / fetch)
- Ability to register a custom tool in your Hermes config
Walkthrough
Step 1: Understand why the pre-installed scraper breaks
web_extract and ddg fetch raw HTML and try to parse it client-side. Reddit and LinkedIn return a login wall or a bot challenge to that kind of request, ddg throttles automated fetches, and paywalled news pages serve a stub. A structured search API does not hit that wall for public, indexed targets: it reaches the source through a maintained backend, handles the anti-bot layer, and returns parsed fields. The agent gets a clean object instead of broken HTML it has to scrape.
# The failing path (do not use):
# Hermes -> web_extract(url) -> raw HTML -> regex/parse -> often empty or blocked
#
# The reliable path:
# Hermes -> scavio_tool(query) -> https://api.scavio.dev -> parsed JSONStep 2: Store the API key as an environment variable
Never hardcode the key in the tool file. Export it once so both the Google and Reddit calls read it. All Scavio REST endpoints authenticate with the header Authorization: Bearer {API_KEY}.
export SCAVIO_API_KEY="sk_your_key_here"Step 3: Call the Google SERP endpoint
POST to /api/v1/google. Default light_request:true costs 1 credit. Set light_request:false (2 credits) to get organic results, people-also-ask, knowledge graph, related searches, and news in one response. This replaces the ddg fetch that stopped returning results.
curl -X POST https://api.scavio.dev/api/v1/google \
-H "Authorization: Bearer $SCAVIO_API_KEY" \
-H "Content-Type: application/json" \
-d '{"query": "hermes agent web scraping", "light_request": false}'Step 4: Call the Reddit search endpoint
POST to /api/v1/reddit/search (1 credit) to find threads. When the agent needs the full discussion, POST to /api/v1/reddit/post (2 credits) to get the post plus its threaded comment tree as JSON. This is the link access Hermes could not get through web_extract.
curl -X POST https://api.scavio.dev/api/v1/reddit/search \
-H "Authorization: Bearer $SCAVIO_API_KEY" \
-H "Content-Type: application/json" \
-d '{"query": "hermes agent setup"}'Step 5: Register it as a Hermes tool
Wrap the two calls in one function and expose it to the agent. The function takes a query, calls Google for web context and Reddit for community discussion, and returns a single clean dict. Use the Python or JS example below verbatim. Scavio also ships an official Hermes integration if you prefer not to hand-roll the tool.
# See the full Python example below — paste it as your tool implementationPython Example
import os
import requests
SCAVIO_API_KEY = os.environ["SCAVIO_API_KEY"]
BASE = "https://api.scavio.dev/api/v1"
HEADERS = {
"Authorization": f"Bearer {SCAVIO_API_KEY}",
"Content-Type": "application/json",
}
def google_search(query: str) -> dict:
r = requests.post(
f"{BASE}/google",
headers=HEADERS,
json={"query": query, "light_request": False},
timeout=30,
)
r.raise_for_status()
return r.json()
def reddit_search(query: str) -> dict:
r = requests.post(
f"{BASE}/reddit/search",
headers=HEADERS,
json={"query": query},
timeout=30,
)
r.raise_for_status()
return r.json()
def web_research(query: str) -> dict:
"""Hermes tool: reliable web + Reddit data as clean JSON."""
return {
"web": google_search(query),
"reddit": reddit_search(query),
}
if __name__ == "__main__":
data = web_research("hermes agent web scraping")
print("organic results:", len(data["web"].get("results", [])))
print("reddit threads:", len(data["reddit"].get("results", [])))
JavaScript Example
const SCAVIO_API_KEY = process.env.SCAVIO_API_KEY;
const BASE = "https://api.scavio.dev/api/v1";
const HEADERS = {
Authorization: `Bearer ${SCAVIO_API_KEY}`,
"Content-Type": "application/json",
};
async function googleSearch(query) {
const r = await fetch(`${BASE}/google`, {
method: "POST",
headers: HEADERS,
body: JSON.stringify({ query, light_request: false }),
});
if (!r.ok) throw new Error(`Google ${r.status}`);
return r.json();
}
async function redditSearch(query) {
const r = await fetch(`${BASE}/reddit/search`, {
method: "POST",
headers: HEADERS,
body: JSON.stringify({ query }),
});
if (!r.ok) throw new Error(`Reddit ${r.status}`);
return r.json();
}
// Hermes tool: reliable web + Reddit data as clean JSON
export async function webResearch(query) {
const [web, reddit] = await Promise.all([
googleSearch(query),
redditSearch(query),
]);
return { web, reddit };
}
webResearch("hermes agent web scraping").then((data) => {
console.log("organic results:", (data.web.results || []).length);
console.log("reddit threads:", (data.reddit.results || []).length);
});
Expected Output
organic results: 9
reddit threads: 8
# The Google call with light_request:false returns organic results plus related searches, people-also-ask (questions), knowledge_graph and news_results in one response. The Reddit call returns parsed threads the agent can read directly, no HTML scraping.