n8n: Migrate Scraping Nodes to API Calls

n8n HTTP Request nodes that scrape websites break when sites change their HTML layout, add CAPTCHAs, or block the IP. Replace scraping nodes with structured API calls that return typed JSON. The API response schema is stable across updates, eliminating the most common cause of n8n workflow failures.

Why scraping nodes break

• Site layout changes: CSS selectors that worked yesterday fail today
• Anti-bot protection: Cloudflare, reCAPTCHA, rate limiting
• IP blocking: repeated requests from the same n8n server IP
• Silent failures: the node returns 200 OK with empty or wrong data

Before: n8n HTTP scraping node

{
  "nodes": [{
    "name": "Scrape Google Results",
    "type": "n8n-nodes-base.httpRequest",
    "parameters": {
      "url": "https://www.google.com/search?q={{$json.keyword}}",
      "options": {
        "response": { "response": { "fullResponse": true } }
      }
    }
  }, {
    "name": "Parse HTML",
    "type": "n8n-nodes-base.html",
    "parameters": {
      "operation": "extractHtmlContent",
      "extractionValues": {
        "values": [{
          "key": "titles",
          "cssSelector": "h3.LC20lb",
          "returnArray": true
        }]
      }
    }
  }]
}

After: n8n API node

{
  "nodes": [{
    "name": "Search API",
    "type": "n8n-nodes-base.httpRequest",
    "parameters": {
      "method": "POST",
      "url": "https://api.scavio.dev/api/v1/search",
      "sendHeaders": true,
      "headerParameters": {
        "parameters": [{
          "name": "x-api-key",
          "value": "={{$env.SCAVIO_API_KEY}}"
        }, {
          "name": "Content-Type",
          "value": "application/json"
        }]
      },
      "sendBody": true,
      "bodyParameters": {
        "parameters": [{
          "name": "query",
          "value": "={{$json.keyword}}"
        }, {
          "name": "country_code",
          "value": "us"
        }]
      }
    }
  }]
}

What changes

• No HTML parsing node needed (API returns structured JSON)
• No proxy configuration (API handles that)
• No CAPTCHA solving (structured API bypasses anti-bot)
• Stable response schema (fields do not change with site redesigns)
• Cost becomes predictable: $0.005/query instead of variable proxy/solver costs

Migration checklist

1. List all HTTP Request nodes that scrape external sites
2. Identify which target Google, Amazon, Reddit, YouTube, or Walmart (API-replaceable)
3. Replace the scraping node with a POST to the search API
4. Remove downstream HTML parsing nodes
5. Update Set/Function nodes to use JSON field paths instead of CSS selectors
6. Add Error Trigger nodes for API failures (rate limit, auth errors)