Large RAG Corpus Build Stack (10M Tokens)

The Problem

An r/Rag post: ~10M tokens of tech articles + docs + blogs + PDFs needed for a RAG pipeline. Naive choice is to scrape everything; in 2026 the cheaper, more reliable shape is search-as-source for indexed public content.

The Scavio Solution

Search-as-source pipeline: 200-500 seed queries → Scavio Google SERP → URL deduplication → Scavio /extract for top URLs → token-budgeted Markdown export. Reserve actual scraping for behind-auth or JS-heavy targets only.

Who It Is For

AI engineers building RAG, RAG SaaS founders, research labs constructing domain corpora at the 1-10M token scale.

Key Benefits

• Avoids most scraper pain on indexed public content
• Typed JSON in and out
• Predictable per-topic cost
• 10M tokens at $20-90 in Scavio + extract
• Scraping reserved only for behind-auth / JS-heavy

Python Example

import requests, os
H = {'x-api-key': os.environ['SCAVIO_API_KEY']}

def build_corpus(seeds, per_query=10):
    urls = set()
    for q in seeds:
        r = requests.post('https://api.scavio.dev/api/v1/search', headers=H, json={'query': q}).json()
        for o in (r.get('organic_results') or [])[:per_query]:
            urls.add(o['link'])
    docs = []
    for u in list(urls)[:2000]:
        d = requests.post('https://api.scavio.dev/api/v1/extract', headers=H, json={'url': u}).json()
        if d.get('text'): docs.append(d['text'])
    return docs

JavaScript Example

// Same shape in TS — search per seed, dedupe, extract top-N.

Platforms Used