n8n web scraping workflows break when target sites change their HTML. The HTTP Request node fetches raw HTML, the HTML Extract node parses it with CSS selectors, and one class name change breaks the entire flow. Replacing the scraping nodes with a search API call returns structured JSON that never changes format regardless of how the target site updates its frontend.
Why n8n Scraping Breaks
n8n's built-in scraping approach chains HTTP Request, HTML Extract, and Function nodes. Each CSS selector is hardcoded to the site's current DOM structure. Google changes its SERP layout every few weeks. Amazon rotates product page templates. YouTube updates its video card format. Each change requires manual selector updates, and the workflow silently returns empty data until someone notices.
The Search API Replacement
// n8n Function node — replaces HTTP Request + HTML Extract chain
const API_KEY = $env.SCAVIO_API_KEY;
const response = await fetch("https://api.scavio.dev/api/v1/search", {
method: "POST",
headers: {
"x-api-key": API_KEY,
"Content-Type": "application/json",
},
body: JSON.stringify({
platform: "google", // or youtube, amazon, walmart, reddit
query: $input.first().json.searchQuery,
}),
});
const data = await response.json();
// Structured JSON — no CSS selectors, no DOM parsing
return data.organic.slice(0, 10).map(r => ({
json: {
title: r.title,
url: r.link,
snippet: r.snippet,
position: r.position,
}
}));Multi-Platform in One Node
The same API endpoint handles Google, YouTube, Amazon, Walmart, and Reddit by changing the platform parameter. An n8n workflow that previously needed 5 different scraping chains (each with custom selectors) now uses one Function node with a platform variable. When YouTube changes its layout, your workflow keeps working.
Cost Comparison
n8n Cloud starts at $24/month for 2,500 executions. Adding a search API at $0.005/query means 500 daily searches cost $75/month. Self-hosted n8n is free but requires server maintenance. For most automation workflows running 50-200 searches per day, the total cost is $7.50-$30/month for search plus whatever n8n hosting costs. This replaces both the scraping infrastructure and the maintenance time spent fixing broken selectors.
When Scraping Still Wins
If you need data from pages that are not search results (internal dashboards, authenticated pages, specific product detail pages), direct scraping is still necessary. Search APIs return search result data, not arbitrary page content. For page extraction, tools like Firecrawl ($16/month for 3,000 credits) or Apify ($49/month) are better suited.
Decide Whether You Have a UI Problem or a Data Problem
Before choosing any tool, separate the two things people mean by "browser automation" in n8n, because they have different answers and the wrong pick is expensive.
UI automation means the workflow has to act on a page: log in, click through a multi-step flow, fill and submit a form, dismiss a modal. There is no API substitute. You need a real browser, and Browserless or a self-hosted Playwright container behind an HTTP Request node is the standard shape.
Data extraction means you only want values off a page, and the browser is incidental. Here a browser is the expensive answer. You inherit captcha handling, fingerprint and stealth patches, proxy rotation, and a permanent maintenance tax, all to obtain data that a structured endpoint returns as JSON.
The tell is simple: if you could get the same values from a JSON response, you have a data problem and you should not be running a browser at all.
The common n8n failure is building a full browser-automation stack, complete with captcha solving and residential proxies, for what was fundamentally a data pull, then wondering why the workflow breaks every week. Each of those layers is a component that can fail independently, and they all fail on someone else's schedule.
Rule of thumb for the mixed case: use a structured API for everything it covers, and reserve the browser for the specific steps that genuinely require clicking. Most workflows that start out "we need browser automation" end up with one browser node and five API calls.