Hermes Search Tool Calls Fail Silently: The ChatML Fix

Your agent decides to search, then answers from memory anyway. No exception, no failed HTTP request, no "search failed" message -- just a confident answer built on nothing. This is the most common web-search failure in Hermes setups, and it survives version bumps. It is the symptom this guide was originally written about several minor releases ago; as of Hermes 0.20.x it is still the first thing to check. The reason it never gets fixed upstream is that it is not a Hermes bug. The break is in the integration layer, where your handler expects one tool-call format and the model emits another.

The failure looks like a hallucination, not an error

Nothing in the stack raises. The model writes a well-formed tool call into its output stream, your handler looks for a field that is not there, gets an empty list, returns None, and the loop continues as if the model had decided not to search. The model then does what any model does with no tool result: it answers from training data. You only notice when a date or a price is wrong.

Two tool-call formats, one agent loop

There are two conventions in circulation and they are not interchangeable.

• ChatML: the call is text. The model writes <tool_call>{"name": ..., "arguments": {...}}</tool_call> into the assistant message body. Hermes-family models and most ChatML-tuned local models do this.
• OpenAI function calling: the call is structure. It arrives on the message object as tool_calls[0].function.arguments, already parsed, never in the text.

Which one you get depends on the backend, not on the Hermes version. Point Hermes Agent at a hosted OpenAI-compatible provider and you get structured calls. Point it at a local GGUF through llama-cpp-python or Ollama's raw completion endpoint and you get tags in text. Same agent config, same prompt, different parse path -- which is exactly why this breaks the day someone swaps the backend.

The broken flow

# What a ChatML-tuned model emits for a search:
# <tool_call>
# {"name": "search", "arguments": {"query": "latest python version"}}
# </tool_call>

# The mistake: the handler only understands OpenAI format
def handle_tool_call(response):
    # Looks for response["tool_calls"][0]["function"]["arguments"]
    tool_calls = response.get("tool_calls", [])  # empty, the call is in the text
    return None  # search silently skipped, model answers unaided

Thirty-second diagnosis

Before writing any parser, print the raw model output for a turn where you know a search should have happened:

print(repr(response["choices"][0]["text"]))

If you see the literal string <tool_call> in there, your model is emitting ChatML and your handler is reading the wrong place. If the text is clean prose and response["choices"][0]["message"].get("tool_calls") is populated, you are on the structured path and the bug is elsewhere.

The fix: parse the tags

import re, json, os, requests

SCAVIO_KEY = os.environ["SCAVIO_API_KEY"]

TOOL_CALL_RE = re.compile(r"<tool_call>\s*(\{.*?\})\s*</tool_call>", re.DOTALL)

def parse_hermes_tool_calls(text: str) -> list:
    """Extract ChatML tool calls from an assistant message body."""
    calls = []
    for match in TOOL_CALL_RE.findall(text):
        try:
            calls.append(json.loads(match))
        except json.JSONDecodeError:
            # Truncated or malformed emission. Skip it rather than
            # killing the turn -- see the pitfalls section.
            continue
    return calls

def execute_search(query: str) -> str:
    """Run the query against a search API and format it for the model."""
    resp = requests.post(
        "https://api.scavio.dev/api/v2/google",
        headers={"Authorization": f"Bearer {SCAVIO_KEY}"},
        json={"query": query, "gl": "us", "hl": "en"},
        timeout=30,
    )
    resp.raise_for_status()
    results = resp.json().get("organic_results", [])[:5]
    return "\n".join(
        f"[{r.get('title', '')}]({r.get('link', '')}): {r.get('snippet', '')}"
        for r in results
    )

def hermes_search_handler(model_output: str) -> str:
    """Run every search tool call found in a model turn."""
    results = []
    for call in parse_hermes_tool_calls(model_output):
        if call.get("name") != "search":
            continue
        query = call.get("arguments", {}).get("query", "")
        if query:
            results.append(execute_search(query))
    return "\n\n".join(results)

sample_output = """I need to look this up.
<tool_call>
{"name": "search", "arguments": {"query": "python 3.15 release date"}}
</tool_call>"""

print(hermes_search_handler(sample_output))

/api/v2/google is a POST endpoint that takes {"query": ...} and returns the SERP as JSON: organic_results with title, link and snippet, plus ai_overview, related_questions and knowledge_graph when Google returns them. Auth is Authorization: Bearer <key>. Every Scavio endpoint follows that shape, which matters below when you want more than Google.

Handle both formats and stop caring which backend you are on

The version-proof move is to not branch on the model at all. Try the structured field, fall back to the tags:

def extract_tool_calls(choice: dict) -> list:
    """Normalise OpenAI-style and ChatML-style tool calls to one list."""
    message = choice.get("message") or {}

    structured = message.get("tool_calls") or []
    if structured:
        out = []
        for call in structured:
            fn = call.get("function", {})
            args = fn.get("arguments", "{}")
            if isinstance(args, str):
                try:
                    args = json.loads(args)
                except json.JSONDecodeError:
                    args = {}
            out.append({"name": fn.get("name"), "arguments": args})
        return out

    text = message.get("content") or choice.get("text") or ""
    return parse_hermes_tool_calls(text)

Roughly twenty lines, and it survives a backend swap, a model upgrade and the next six Hermes releases.

Full loop with llama-cpp-python

from llama_cpp import Llama

llm = Llama(model_path="./your-hermes-model.gguf", n_ctx=8192)

SEARCH_TOOL_PROMPT = """You have access to the following tool:
- search: Search the web for current information. Input: {"query": "your search query"}

When you need current information, use the tool like this:
<tool_call>
{"name": "search", "arguments": {"query": "your query"}}
</tool_call>

Stop after the tool call and wait for the result before continuing."""

def chat_with_search(user_message: str) -> str:
    prompt = f"{SEARCH_TOOL_PROMPT}\n\nUser: {user_message}\nAssistant:"
    first = llm(prompt, max_tokens=512, stop=["</tool_call>"])
    output = first["choices"][0]["text"]

    # The stop sequence eats the closing tag, so put it back before parsing.
    if "<tool_call>" in output and "</tool_call>" not in output:
        output += "</tool_call>"

    search_results = hermes_search_handler(output)
    if not search_results:
        return output

    augmented = (
        f"{prompt}{output}\n\n"
        f"<tool_response>{search_results}</tool_response>\n\n"
        f"Answer using the tool response above:"
    )
    return llm(augmented, max_tokens=1024)["choices"][0]["text"]

print(chat_with_search("What is the current stable Python release?"))

Pitfalls that bite after the parser works

• Truncation produces unclosed tags. If max_tokens cuts the model off mid-call, the regex never matches and you are back to a silent skip. Use a stop sequence on </tool_call> and re-append it, as above.
• Streaming splits the tags across chunks. Never run the regex on a delta. Buffer the full message, then parse.
• Multiple calls per turn are normal. Iterate, do not take [0].
• Malformed JSON inside a valid tag happens on smaller models. Skip the call and let the loop retry rather than raising.
• Aggressive quantization degrades tool calling before it degrades prose. If calls are frequently malformed at Q4 and below, that is the first thing to test, not the prompt.
• Search results eat the context window. Cap at three to five results and truncate snippets; a full SERP dump will push your system prompt out of the window on an 8k model.

Or skip the parser entirely

The parser is worth writing if you are driving a local model directly. If you are running Hermes Agent, its MCP support already does the format translation for you, and the tool schemas arrive from the server instead of from a hand-written prompt block.

{
  "mcpServers": {
    "scavio": {
      "command": "npx",
      "args": ["-y", "@scavio/mcp-server"],
      "env": { "SCAVIO_API_KEY": "YOUR_SCAVIO_API_KEY" }
    }
  }
}

That is the stdio form, which every MCP client understands. If your client speaks streamable HTTP, point it at https://mcp.scavio.dev/mcp instead and pass the key in an x-api-key header -- no local process at all.

The server registers 106 tools across 11 platforms by default. It is deliberately not everything: all 191 tools is a large payload in every single session, and small local models get measurably worse at picking a tool as the list grows. Widen it only when you need to: SCAVIO_PLATFORMS on the local server, or the x-scavio-platforms header on the hosted one. Both are additive, so default,zillow,sec adds to the default set rather than replacing it.

There is also a skill path, which is lighter than an MCP server if you only want one platform:

hermes skills install @scavio-ai/scavio-amazon

50 skills cover the 50 platforms, one per platform surface, and they are plain SKILL.md files -- no server process to keep alive.

Either way the free tier is 50 credits on signup with no card (dashboard.scavio.dev/sign-up), which is enough to confirm your parser works before you decide anything. Credit cost varies by platform: a Google SERP call is 1 credit, Yelp or Tripadvisor is 2, G2 is 5. The docs list the cost per endpoint.

Key takeaway

The model is not broken and downgrading will not help. It generates a valid tool call; your code reads for it in the wrong place. Normalise both formats in one twenty-line function and the failure stops recurring every time you change backends or the version number moves.