LangChain ShadowAudit: Runtime Tool Call Enforcement
LangChain agents silently skip tool calls. ShadowAudit logs every expected call and flags ungrounded answers that bypassed search.
LangChain agents can silently skip tool calls and answer from the LLM's training data instead. ShadowAudit is a runtime enforcement pattern that logs every expected tool call, checks whether it actually executed, and flags hallucinated answers that bypassed the tool layer.
The Silent Skip Problem
When a LangChain agent decides a tool call is unnecessary, it skips it and generates an answer directly. This is by design -- the LLM chooses when to use tools. But for search-grounded pipelines, skipping the search tool means the agent answers from stale training data without telling you.
In production, this manifests as answers that look correct but contain outdated information. The agent did not search; it guessed. Without runtime enforcement, you cannot tell the difference.
The ShadowAudit Pattern
Wrap every tool in a logging layer that records invocations. After the agent generates its response, compare the audit log against expected tool calls. If a search-dependent query produced no search tool invocations, flag the response as ungrounded.
from langchain.tools import Tool
from langchain.agents import AgentExecutor
import requests, os
H = {"x-api-key": os.environ["SCAVIO_API_KEY"]}
# Audit log
tool_audit = []
def audited_search(query: str) -> str:
"""Search with audit logging."""
tool_audit.append({
"tool": "search",
"query": query,
"invoked": True,
})
r = requests.post("https://api.scavio.dev/api/v1/search",
headers=H,
json={"platform": "google", "query": query},
timeout=10
).json()
results = r.get("organic", [])[:5]
return "\n".join([
f"- {item['title']}: {item.get('snippet', '')}"
for item in results
])
search_tool = Tool(
name="web_search",
description="Search the web for current information",
func=audited_search,
)
def shadow_audit(query, response):
"""Check if search was actually called for search queries."""
search_indicators = [
"latest", "current", "2026", "today",
"price", "cost", "how much",
]
needs_search = any(
indicator in query.lower()
for indicator in search_indicators
)
was_searched = any(
log["tool"] == "search" for log in tool_audit
)
if needs_search and not was_searched:
return {
"grounded": False,
"warning": "Query needed search but agent skipped it",
"response": response,
}
return {"grounded": True, "response": response}
# After agent execution:
# result = shadow_audit(user_query, agent_response)
# if not result["grounded"]:
# print(f"WARNING: {result['warning']}")Forcing Tool Calls
For critical pipelines, do not let the LLM decide whether to search. Call the search API before the LLM and inject results into the prompt. This eliminates the skip problem entirely.
def forced_grounded_answer(query):
"""Always search, then answer. No LLM choice."""
# Step 1: Always search
r = requests.post("https://api.scavio.dev/api/v1/search",
headers=H,
json={"platform": "google", "query": query},
timeout=10
).json()
context = "\n".join([
f"- {item['title']}: {item.get('snippet', '')}"
for item in r.get("organic", [])[:5]
])
# Step 2: LLM answers with forced context
prompt = f"""Answer based ONLY on these search results:
{context}
Question: {query}
If the search results don't contain the answer, say so."""
# Send prompt to your LLM
return prompt # placeholder for LLM call
# The LLM never gets a chance to skip search
answer = forced_grounded_answer("scavio api pricing 2026")Audit Metrics Worth Tracking
Track: tool call skip rate (how often the LLM bypasses tools), false grounding rate (answers that cite search but used stale data), and latency delta between grounded and ungrounded answers. A skip rate above 5% on search-dependent queries means your agent prompt needs stronger tool-use instructions.
MCP Server Advantage
MCP servers like https://mcp.scavio.dev/mcp make tool enforcement easier because the tool schema is explicit. The MCP protocol logs tool invocations at the transport layer, giving you an audit trail without wrapping each tool function manually.