To build a job board without scraping thousands of career sites, use a Google SERP API as a discovery layer: query Google for fresh postings on the big applicant-tracking domains (Greenhouse, Lever, Ashby, Workday), parse the organic results for title, link, and snippet, dedupe by URL, and store what you keep. You never run or maintain a single site-specific scraper. One developer on r/developersIndia scraped over 2 million job postings across 100,000+ company career sites into a unified dataset, and a commenter pointed out the obvious cost sink: he was "building my own google search API (which costs a lot)." That is the trap. Maintaining 100,000 parsers breaks every time a career page changes its markup, and a homegrown SERP layer eats proxies and engineering time. Google already crawls every public job page; an indexed query surfaces new listings the day they appear. This tutorial walks through POST /api/v1/google with targeted site: operators and time-fresh queries, parsing organic_results, deduping, and storing. Scavio charges 2 credits ($0.01) for a full SERP call and 1 credit ($0.005) for a light one, on true pay-as-you-go with no monthly floor and no minimum deposit. Honest scope: SERP discovery gives you breadth and freshness cheaply, but for fully structured fields (salary, normalized location, apply URL) you still parse the destination page or use an ATS API where one exists. SERP is the discovery layer, not the whole ETL pipeline.
Prerequisites
- A Scavio API key from scavio.dev (free tier includes 50 one-time credits)
- Python 3.9+ with the requests library, or Node.js 18+ for the JS version
- Basic familiarity with Google search operators (site:, intitle:, after:)
- A datastore for listings (SQLite, Postgres, or even a JSON file to start)
Walkthrough
Step 1: Set your API key and pick discovery queries
Export your key as an environment variable, then design queries that target the applicant-tracking domains where most companies post. Each query combines a board with a role and optionally a location. Run several narrow queries instead of one broad one; SERP coverage per query is finite, so segmenting by board and role surfaces more distinct listings.
export SCAVIO_API_KEY="your_key_here"
# Discovery queries: board + role (+ optional location)
QUERIES = [
'site:boards.greenhouse.io OR site:jobs.lever.co "backend engineer" remote',
'site:jobs.ashbyhq.com "product designer"',
'site:boards.greenhouse.io "data engineer" New York',
]Step 2: Call POST /api/v1/google and read organic_results
Send each query to the Google endpoint with light_request=false so you get the full SERP. The response includes organic_results; each row carries position, title, link, and snippet. The link is the live job posting on Greenhouse or Lever, and the snippet usually contains the role and location. Use Bearer auth in the Authorization header.
import os, requests
H = {"Authorization": f"Bearer {os.environ['SCAVIO_API_KEY']}", "Content-Type": "application/json"}
def search(query):
r = requests.post("https://api.scavio.dev/api/v1/google", headers=H,
json={"query": query, "light_request": False})
r.raise_for_status()
return r.json().get("organic_results", [])
for row in search('site:jobs.lever.co "backend engineer" remote'):
print(row["position"], row["title"], row["link"])Step 3: Dedupe by URL and normalize into listing records
The same posting shows up across multiple queries, so dedupe by URL before you store anything. Strip query strings and tracking params so two URLs for the same job collapse into one. Map each organic result into a flat listing record you control, keeping the source query so you can audit coverage later.
from urllib.parse import urlsplit, urlunsplit
def canonical(url):
p = urlsplit(url)
return urlunsplit((p.scheme, p.netloc, p.path.rstrip("/"), "", ""))
def to_listing(row, source_query):
return {
"url": canonical(row["link"]),
"title": row["title"],
"snippet": row.get("snippet", ""),
"source_query": source_query,
}
seen = set()
listings = []
for q in QUERIES:
for row in search(q):
rec = to_listing(row, q)
if rec["url"] not in seen:
seen.add(rec["url"])
listings.append(rec)Step 4: Keep it fresh and store to your datastore
Add a date-fresh query so you only pull recent postings, then upsert by URL on a schedule (a daily cron is enough for most boards). Google indexes new job pages within hours to a day, so a daily sweep keeps the board current without re-scraping anything. Here we persist to SQLite; swap in Postgres for production.
import sqlite3
db = sqlite3.connect("jobs.db")
db.execute("""CREATE TABLE IF NOT EXISTS listings (
url TEXT PRIMARY KEY, title TEXT, snippet TEXT, source_query TEXT,
first_seen TEXT DEFAULT CURRENT_TIMESTAMP)""")
# Fresh-only discovery query (last 7 days)
fresh = 'site:boards.greenhouse.io "software engineer" after:2026-06-13'
listings += [to_listing(r, fresh) for r in search(fresh)
if canonical(r["link"]) not in seen]
for rec in listings:
db.execute("""INSERT OR IGNORE INTO listings (url, title, snippet, source_query)
VALUES (:url, :title, :snippet, :source_query)""", rec)
db.commit()
print(f"stored {len(listings)} listings")Python Example
import os, requests, sqlite3
from urllib.parse import urlsplit, urlunsplit
H = {"Authorization": f"Bearer {os.environ['SCAVIO_API_KEY']}", "Content-Type": "application/json"}
QUERIES = [
'site:boards.greenhouse.io OR site:jobs.lever.co "backend engineer" remote',
'site:jobs.ashbyhq.com "product designer"',
'site:boards.greenhouse.io "data engineer" New York',
]
def search(query):
r = requests.post("https://api.scavio.dev/api/v1/google", headers=H,
json={"query": query, "light_request": False})
r.raise_for_status()
return r.json().get("organic_results", [])
def canonical(url):
p = urlsplit(url)
return urlunsplit((p.scheme, p.netloc, p.path.rstrip("/"), "", ""))
db = sqlite3.connect("jobs.db")
db.execute("""CREATE TABLE IF NOT EXISTS listings (
url TEXT PRIMARY KEY, title TEXT, snippet TEXT, source_query TEXT,
first_seen TEXT DEFAULT CURRENT_TIMESTAMP)""")
seen, stored = set(), 0
for q in QUERIES:
for row in search(q):
url = canonical(row["link"])
if url in seen:
continue
seen.add(url)
db.execute("INSERT OR IGNORE INTO listings (url, title, snippet, source_query) VALUES (?, ?, ?, ?)",
(url, row["title"], row.get("snippet", ""), q))
stored += 1
db.commit()
print(f"stored {stored} unique listings")JavaScript Example
import Database from "better-sqlite3";
const H = {
Authorization: `Bearer ${process.env.SCAVIO_API_KEY}`,
"Content-Type": "application/json",
};
const QUERIES = [
'site:boards.greenhouse.io OR site:jobs.lever.co "backend engineer" remote',
'site:jobs.ashbyhq.com "product designer"',
'site:boards.greenhouse.io "data engineer" New York',
];
async function search(query) {
const r = await fetch("https://api.scavio.dev/api/v1/google", {
method: "POST",
headers: H,
body: JSON.stringify({ query, light_request: false }),
});
const data = await r.json();
return data.organic_results ?? [];
}
function canonical(url) {
const u = new URL(url);
return `${u.protocol}//${u.host}${u.pathname.replace(/\/$/, "")}`;
}
const db = new Database("jobs.db");
db.exec(`CREATE TABLE IF NOT EXISTS listings (
url TEXT PRIMARY KEY, title TEXT, snippet TEXT, source_query TEXT)`);
const insert = db.prepare("INSERT OR IGNORE INTO listings (url, title, snippet, source_query) VALUES (?, ?, ?, ?)");
const seen = new Set();
let stored = 0;
for (const q of QUERIES) {
for (const row of await search(q)) {
const url = canonical(row.link);
if (seen.has(url)) continue;
seen.add(url);
insert.run(url, row.title, row.snippet ?? "", q);
stored++;
}
}
console.log(`stored ${stored} unique listings`);Expected Output
1 Senior Backend Engineer - Acme - jobs.lever.co https://jobs.lever.co/acme/8f2c...
2 Backend Engineer (Remote) - Globex https://jobs.lever.co/globex/1a9d...
3 Staff Backend Engineer - Initech https://jobs.lever.co/initech/4b7e...
stored 27 unique listings