Best Ways to Get Web Data with Claude Code

Claude Code can browse the web, but "browsing" covers a wide range of approaches with very different trade-offs. You can scrape HTML directly, use an MCP tool, or call a structured API. Each method has different reliability, speed, and cost characteristics.

This post compares the three main ways to get web data inside Claude Code and helps you pick the right one for your use case.

Option 1: Direct Scraping with Fetch

Claude Code can run shell commands, which means it can usecurl, wget, or a Node script to fetch raw HTML. This works for simple pages but breaks quickly.

# Claude Code can run this directly
curl -s https://example.com/pricing | head -100

The problems are predictable: JavaScript-rendered pages return empty shells, anti-bot protections block requests, and parsing unstructured HTML is fragile. Claude can extract data from raw HTML, but it burns tokens on markup that has nothing to do with the actual content.

• Works for static, public pages with simple HTML
• Fails on SPAs, bot-protected sites, and dynamic content
• No structured output -- Claude has to parse the HTML itself
• Token-inefficient due to large HTML payloads

Option 2: MCP Tools for Structured Search

MCP tools give Claude Code access to external APIs through a standardized protocol. Instead of scraping, Claude calls a tool that returns clean, structured JSON.

# Add Scavio MCP to Claude Code
claude mcp add --transport http scavio https://mcp.scavio.dev/mcp \
  --header "x-api-key: YOUR_API_KEY"

Once connected, Claude can call tools like search_google orsearch_amazon and get back structured results -- titles, URLs, prices, ratings -- without touching any HTML.

{
  "results": [
    {
      "title": "Sony WH-1000XM6",
      "price": "$348.00",
      "rating": 4.7,
      "url": "https://amazon.com/dp/B0DEXAMPLE"
    }
  ]
}

• Structured JSON output -- no parsing needed
• Works across Google, Amazon, YouTube, Walmart, Reddit
• Token-efficient since only relevant data is returned
• Requires an API key and credits

Option 3: Direct API Calls

Claude Code can also call REST APIs directly using curl or a script. This gives you the same structured data as MCP tools but without the MCP protocol layer.

curl -X POST https://api.scavio.dev/api/v1/search \
  -H "Content-Type: application/json" \
  -H "x-api-key: YOUR_API_KEY" \
  -d '{"query": "best laptops 2026", "platform": "google"}'

Direct API calls are useful when you want full control over the request or when you are building a script that Claude Code will run. The downside is that Claude has to be told about the API format -- it does not discover it automatically like it does with MCP tools.

When to Use Each Approach

The right choice depends on what you are building:

• Direct scraping: Quick one-off extractions from simple, static pages. Do not use for anything that needs to work reliably.
• MCP tools: Interactive sessions where Claude needs to search and reason over results. The tool discovery mechanism means Claude knows exactly what is available without prompting.
• Direct API calls: Scripts and automations where you want deterministic behavior and full control over error handling.

Practical Recommendations

For most Claude Code workflows, MCP tools are the best default. They combine the reliability of a structured API with the convenience of automatic tool discovery. Claude knows the parameters, handles the calling convention, and gets clean data back.

Use direct API calls when you are writing scripts that will run outside of Claude Code, or when you need to batch requests. Use direct scraping only as a last resort for pages that no API covers.

Whatever approach you choose, avoid sending raw HTML to Claude when structured data is available. The token savings alone make structured APIs worth the setup cost.