Most teams now run a split stack: self-hosted scrapers with residential proxies and stealth headless browsers for anything behind auth or heavy JavaScript, and hosted search/SERP APIs for anything public and already indexed. There is no single tool that does everything well in 2026, and a recent r/dataengineering thread ("Out of the scraping game a few years, what does everyone run now?", 60 upvotes) confirms it. One reply: "Scraping Google Search is really difficult... you need SERP or beat Google engineers." Another: "Too brittle to run myself, I use SERP services at work." A third named the real reason: "huge increase in Cloudflare anti-bot thanks to AI, nobody wants their data taken for free anymore."
That last point is the whole story. The cost of self-hosted scraping went up. Cloudflare, DataDome, and PerimeterX got better at fingerprinting headless browsers because the AI training-data gold rush made everyone defensive. So the question stopped being "which scraper library" and became "which layer of the stack does this target belong to."
Layer 1: self-hosted scrapers + residential proxies
You still need this for targets behind a login or rendered entirely client-side. Think a logged-in dashboard, an internal SaaS report, a React app that ships an empty <div id="root">. Here you run Playwright or a stealth fork, rotate residential proxies, and accept that you will babysit it. This layer is the most flexible and the most fragile. Every Cloudflare challenge rev costs you an afternoon. SearXNG sits near here too: free and self-hosted, but it breaks when upstream engines change their HTML and needs your own proxies at volume.
Layer 2: hosted scraping / crawl APIs
When you need page content from arbitrary sites but do not want to run the browsers, a crawl API earns its fee. Firecrawl is the common pick: free 1,000 credits/month, Hobby $16/mo for ~3,000 credits, with AI extraction billed at 5 credits per call and credits that do not roll over. Jina AI's r.jina.ai reader returns clean text and gives 10M free tokens per key for non-commercial use. These convert messy HTML into LLM-ready text. They do not give you typed fields like "price" or "rating" unless you pay for the AI extract pass.
Layer 3: structured search / SERP APIs
This is the layer most people underuse. If the target is public and indexed — Google results, Amazon listings, Reddit threads, YouTube — a search API hands you structured JSON and never fights Cloudflare, because you are not crawling the site, you are querying an API that already did. Serper runs $1.00 per 1,000 credits down to $0.30/1k at scale, with 2,500 free credits valid six months. SerpApi gives 250 free searches/month, then $25/mo for 1,000. Scavio is credit-based at $0.005/credit; a full-feature Google SERP costs 2 credits, a light request 1, and one API key covers Google, Reddit, YouTube, Amazon, Walmart, and TikTok.
The decision rule
Is the target public and indexed? Use a search API and get typed JSON with no anti-bot fight. Is it behind auth or JS-rendered? You are back in Layer 1, no API saves you there. Be clear about this: a SERP API does not replace scraping for authenticated or client-rendered pages. It replaces the specific, painful job of scraping public SERPs and marketplace listings — the exact thing the Reddit thread called "really difficult."
Here is a full-feature Google query against Scavio returning structured JSON:
import requests
resp = requests.post(
"https://api.scavio.dev/api/v1/google",
headers={"Authorization": "Bearer YOUR_API_KEY"},
json={"query": "best web scraping stack 2026", "light_request": False},
)
data = resp.json()
for result in data["organic"]:
print(result["position"], result["title"], result["link"])
# full_request also returns people_also_ask, knowledge_graph, related_searches
for q in data.get("people_also_ask", []):
print("PAA:", q["question"])No headless browser, no proxy pool, no Cloudflare. For Google, Reddit, Amazon, and YouTube, that is the 2026 answer most teams landed on. For the gated and JS-heavy stuff, keep your Playwright rig warm.
Where each loses
No tool wins everywhere. If you only hit Google a few hundred times a month, SerpApi's free 250 or Serper's 2,500 free credits may cost you nothing and beat any paid plan. If you need clean article text from random blogs, Firecrawl or Jina reads pages a SERP API was never built to fetch. And if your targets are all behind a login, none of Layer 2 or 3 helps — self-host and proxy up. Match the layer to the target, not the hype.