Composing a Search Engine from APIs

Building a search engine used to mean crawling the web, building an index, and ranking documents. In 2026, you can skip all of that. Search APIs from Google, Amazon, YouTube, and others give you structured results on demand. The challenge is not getting results -- it is combining results from multiple APIs into a unified experience that feels like a single search engine.

The Architecture

A composed search engine has three layers: the query router, the API fanout layer, and the result merger. The query router decides which APIs to call based on the user's intent. The fanout layer calls those APIs in parallel. The merger normalizes the responses into a single ranked list.

interface SearchResult {
  title: string;
  url: string;
  snippet: string;
  source: "google" | "amazon" | "youtube" | "reddit";
  metadata: Record<string, unknown>;
}

async function composedSearch(query: string): Promise<SearchResult[]> {
  const sources = classifyIntent(query);
  const results = await Promise.all(
    sources.map((source) => fetchFromApi(source, query))
  );
  return mergeAndRank(results.flat());
}

The key insight is that each API returns different fields. Google gives you snippets and links. Amazon gives you prices and ratings. YouTube gives you view counts and durations. Your merger needs to normalize these into a common schema while preserving the source-specific metadata that makes each result useful.

Query Routing

Not every query should hit every API. Searching Amazon for "latest news on trade policy" wastes credits and returns irrelevant products. A simple classifier can route queries to the right APIs:

• Product-related queries go to Amazon and Walmart
• How-to and tutorial queries go to YouTube and Google
• Opinion and discussion queries go to Reddit and Google
• Factual queries go to Google only

You can build this classifier with an LLM, a keyword heuristic, or a small fine-tuned model. For most applications, a keyword-based approach is fast and accurate enough. LLM-based classification adds latency but handles ambiguous queries better.

Parallel API Fanout

Latency is the main enemy of composed search. If you call three APIs sequentially, each taking 500ms, your total response time is 1.5 seconds. Call them in parallel and you are bound by the slowest one -- typically under 700ms.

async function fetchFromApi(
  source: string,
  query: string
): Promise<SearchResult[]> {
  const response = await fetch("https://api.scavio.dev/api/v1/search", {
    method: "POST",
    headers: {
      "x-api-key": process.env.SCAVIO_API_KEY!,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ platform: source, query }),
  });
  const data = await response.json();
  return normalizeResults(data, source);
}

Using a single API that supports multiple platforms -- like Scavio -- simplifies the fanout layer. You use the same endpoint, the same authentication, and the same response format for every platform. The only variable is the platform field in the request body.

Result Merging and Ranking

Merging results from different APIs is harder than it looks. A Google result ranked #1 is not necessarily more relevant than an Amazon product ranked #3. Each API has its own ranking signal, and those signals are not comparable.

A practical approach is reciprocal rank fusion (RRF). Assign each result a score based on its position in its source list, then combine scores across sources. Results that appear high in multiple sources rise to the top.

function reciprocalRankFusion(
  resultSets: SearchResult[][],
  k = 60
): SearchResult[] {
  const scores = new Map<string, number>();
  const items = new Map<string, SearchResult>();

  for (const results of resultSets) {
    results.forEach((result, rank) => {
      const key = result.url;
      const prev = scores.get(key) || 0;
      scores.set(key, prev + 1 / (k + rank + 1));
      items.set(key, result);
    });
  }

  return [...scores.entries()]
    .sort((a, b) => b[1] - a[1])
    .map(([key]) => items.get(key)!);
}

Handling Failures Gracefully

Individual APIs will fail. Your fanout layer should treat each call as optional -- use Promise.allSettled instead of Promise.all, set aggressive timeouts (2-3 seconds), cache recent results as fallbacks, and log failure rates per source.

When to Build vs Buy

Building a composed search engine makes sense when you need custom ranking or domain-specific query routing. If you just need multi-platform results without custom logic, a unified API like Scavio handles the fanout and normalization for you. The composed approach shines when your application has unique ranking requirements that no generic API can satisfy.