Qwen hallucinates on web search because it is being handed raw scraped HTML, and a small local model cannot reliably separate the answer from navigation, cookie banners and footer boilerplate. Feed it typed JSON fields instead of a page dump and most of the problem goes away. Cloud models hide this failure because their search tooling cleans and structures results before the model ever sees them.
Why local LLMs hallucinate worse than cloud LLMs
Two structural reasons:
- Tighter context windows. Qwen 27B at 32K context. Llama-3 8B at 8K-32K. Cloud Claude/GPT routinely run 200K+. When the agent feeds raw scraped HTML (25-40K tokens for a 60KB page), the local LLM has no room for instructions, citation rules, or multi-source comparison.
- Smaller training, less robustness to noisy input. Frontier models smooth over noisy retrieval; smaller models propagate the noise into the answer.
The fix has two parts
First, the input shape. Raw HTML is the wrong input. Typed JSON sources (Scavio's organic_results) compress 10x: ~1.5K tokens for 10 sources vs 25-40K for raw HTML.
Second, the prompt structure. Local LLMs respond to explicit, numbered citation requirements better than soft instructions.
The minimum-viable grounding stack
import requests, os
H = {'x-api-key': os.environ['SCAVIO_API_KEY']}
PROMPT = '''Answer using ONLY the sources below.
Every claim must be followed by [N] where N is the source number.
If the sources do not answer the question, say "I don't know based on the provided sources."
Sources:
{sources}
Question: {question}'''
def search(q):
r = requests.post('https://api.scavio.dev/api/v1/search',
headers=H, json={'query': q}).json()
return r.get('organic_results', [])[:10]
def fmt_sources(results):
return '\\n'.join(
f'[{i+1}] {r["title"]} ({r["link"]}): {r["snippet"]}'
for i, r in enumerate(results)
)
def ask_qwen(q):
sources = search(q)
prompt = PROMPT.format(sources=fmt_sources(sources), question=q)
r = requests.post('http://localhost:11434/api/generate',
json={'model': 'qwen2.5:32b', 'prompt': prompt, 'stream': False}).json()
return r['response']The cross-check pattern
Add a second call to Scavio with include_ai_overview: true. Compare the local LLM's claims against Google's AI Overview citation set. Disagreement is a hallucination flag — surface to the user instead of presenting as fact.
Why this works
Three reinforcing effects:
- Context budget is preserved for actual reasoning, not parsing
- The numbered source list gives the LLM unambiguous reference points
- The "say I don't know" clause gives an out for unsupported questions; without it, the LLM fabricates rather than admit gaps
Reported impact in practice
In the r/LocalLLaMA post, Qwen 27B + raw HTML grounding produced ~18% factual hallucination rate. The same model with typed JSON + the strict citation prompt dropped to under 3% on the same benchmark.
The fix isn't a bigger model. It's the input shape and prompt discipline.
The honest constraints
Local LLMs still trail cloud LLMs on truly open-ended synthesis. Grounded retrieval narrows the gap; it doesn't close it. For high-stakes factual questions on broad topics, frontier cloud models still win.
For tight, focused queries (single domain, narrow topic, well-formed question) with grounded sources, local Qwen/Llama/DeepSeek with this pattern is production-viable in 2026.
Stack cost
Scavio Project tier $30/mo + Ollama (free, local) + a GPU you already own = the entire grounding stack. Compared to cloud-LLM-only at typical Claude/GPT API rates, this is the privacy-friendly, cost-controlled option for teams that already have local inference running.
Common questions
Why does my local LLM hallucinate on web search? Because it receives raw page text. A scraped page is mostly navigation, ads and boilerplate, and a 9B to 35B model has limited capacity to decide which span is the answer. It fills the gap by generating something plausible. Cloud assistants avoid this by cleaning and structuring results before the model sees them.
How do I stop an AI from hallucinating on web results? Three changes do most of the work: return typed JSON fields rather than page HTML, cap how many results enter context, and require the model to cite which result each claim came from so an uncited claim is visibly unsupported. Scavio returns structured fields per result, which removes the parsing step from the model entirely.
Does a bigger model fix it? Only partly. Scaling improves the filtering, but the underlying problem is input quality rather than capacity. A well-structured input makes a 9B model behave more reliably on grounded answers than a raw HTML dump makes a 70B one.
How can I spot hallucinations in web-grounded answers? Require a source URL per claim and check that the claim appears in that source. Answers assembled from a single vague passage, or citing a URL that does not contain the fact, are the common failure signature.