Tutorial

How to Optimize D2C Site for AI Agent Discovery

Make your D2C brand visible to AI shopping agents. Structured data, llms.txt, and product feed optimization for agent traffic.

AI shopping agents from ChatGPT, Perplexity, and Claude now recommend products directly. If your D2C site is not optimized for agent crawling, you lose these referrals to Amazon. This tutorial shows how to audit your current agent visibility, add structured data, create llms.txt, and verify agents can find your products.

Prerequisites

  • Python 3.8+
  • requests library
  • A Scavio API key from scavio.dev
  • D2C site URL with product pages

Walkthrough

Step 1: Audit current AI agent visibility

Check if AI agents can find and recommend your products today.

Python
import os, requests

API_KEY = os.environ['SCAVIO_API_KEY']
SH = {'x-api-key': API_KEY, 'Content-Type': 'application/json'}

def check_visibility(brand, product):
    queries = [
        f'best {product} 2026',
        f'{product} recommendations',
        f'buy {product} online direct',
        f'{brand} {product} review',
    ]
    results = {'found': 0, 'total': 0, 'positions': []}
    for q in queries:
        data = requests.post('https://api.scavio.dev/api/v1/search',
            headers=SH, json={'query': q, 'country_code': 'us'}).json()
        organic = data.get('organic_results', [])
        results['total'] += 1
        for i, r in enumerate(organic):
            if brand.lower() in r.get('link', '').lower() or brand.lower() in r.get('title', '').lower():
                results['found'] += 1
                results['positions'].append(i + 1)
                break
    visibility = results['found'] / results['total'] * 100 if results['total'] else 0
    avg_pos = sum(results['positions']) / len(results['positions']) if results['positions'] else 0
    print(f'Brand: {brand}')
    print(f'Visibility: {visibility:.0f}% ({results["found"]}/{results["total"]} queries)')
    print(f'Avg position when found: {avg_pos:.1f}')
    print(f'Cost: ${results["total"] * 0.005:.3f}')
    return results

check_visibility('YourBrand', 'organic face cream')

Step 2: Check structured data and llms.txt

Verify that your site has the right signals for AI agent crawling.

Python
def audit_agent_readiness(domain):
    """Check if a D2C site is ready for AI agent discovery."""
    checks = []
    # Check llms.txt
    data = requests.post('https://api.scavio.dev/api/v1/search',
        headers=SH, json={'query': f'site:{domain} llms.txt', 'country_code': 'us'}).json()
    has_llms = len(data.get('organic_results', [])) > 0
    checks.append(('llms.txt exists', has_llms))
    # Check structured data
    data = requests.post('https://api.scavio.dev/api/v1/search',
        headers=SH, json={'query': f'site:{domain} product', 'country_code': 'us'}).json()
    results = data.get('organic_results', [])
    has_rich = any(r.get('rich_snippet') for r in results)
    checks.append(('Rich snippets on product pages', has_rich))
    # Check sitemap
    data = requests.post('https://api.scavio.dev/api/v1/search',
        headers=SH, json={'query': f'site:{domain} sitemap.xml', 'country_code': 'us'}).json()
    has_sitemap = len(data.get('organic_results', [])) > 0
    checks.append(('Sitemap indexed', has_sitemap))
    print(f'\nAgent Readiness Audit: {domain}')
    for check, passed in checks:
        status = 'PASS' if passed else 'FAIL'
        print(f'  [{status}] {check}')
    score = sum(1 for _, p in checks if p)
    print(f'\nScore: {score}/{len(checks)}')
    return checks

audit_agent_readiness('example-d2c-brand.com')

Step 3: Generate llms.txt for your D2C site

Create an llms.txt file that helps AI agents understand your product catalog.

Python
def generate_llms_txt(brand, domain, products):
    """Generate llms.txt for a D2C brand."""
    lines = [
        f'# {brand}',
        f'',
        f'> {brand} sells directly to consumers at {domain}.',
        f'> All products ship from our warehouse. Free returns within 30 days.',
        f'',
        f'## Products',
    ]
    for p in products:
        lines.append(f'- [{p["name"]}](https://{domain}/products/{p["slug"]}): {p["description"]}')
    lines.extend([
        f'',
        f'## Ordering',
        f'- [Shop All](https://{domain}/shop)',
        f'- [FAQ](https://{domain}/faq)',
        f'- [Shipping Policy](https://{domain}/shipping)',
        f'- [Returns](https://{domain}/returns)',
    ])
    content = '\n'.join(lines)
    print(content)
    return content

# Example product catalog
products = [
    {'name': 'Organic Face Cream', 'slug': 'organic-face-cream', 'description': '50ml, $28, for sensitive skin'},
    {'name': 'Vitamin C Serum', 'slug': 'vitamin-c-serum', 'description': '30ml, $22, brightening formula'},
]

llms_txt = generate_llms_txt('GlowCo', 'glowco.com', products)
print(f'\nSave this as /llms.txt on your domain root')

Step 4: Monitor agent discovery over time

Track whether AI agents start finding your products after optimization.

Python
import json

def track_agent_visibility(brand, domain, products, history_file='visibility_log.json'):
    try:
        with open(history_file) as f:
            history = json.load(f)
    except FileNotFoundError:
        history = []
    today = {'date': '2026-05-20', 'queries': []}
    for product in products:
        query = f'best {product} 2026'
        data = requests.post('https://api.scavio.dev/api/v1/search',
            headers=SH, json={'query': query, 'country_code': 'us'}).json()
        organic = data.get('organic_results', [])
        position = None
        for i, r in enumerate(organic):
            if domain in r.get('link', ''):
                position = i + 1
                break
        today['queries'].append({'query': query, 'position': position})
    history.append(today)
    found = sum(1 for q in today['queries'] if q['position'])
    print(f'Visibility check: {found}/{len(products)} products found in SERP')
    for q in today['queries']:
        pos = f'#{q["position"]}' if q['position'] else 'Not found'
        print(f'  {q["query"][:40]:40} | {pos}')
    if len(history) >= 2:
        prev = history[-2]
        prev_found = sum(1 for q in prev['queries'] if q['position'])
        delta = found - prev_found
        print(f'\nChange vs last check: {delta:+d} products visible')
    print(f'Cost: ${len(products) * 0.005:.3f}')

track_agent_visibility('GlowCo', 'glowco.com', ['organic face cream', 'vitamin c serum'])

Python Example

Python
import os, requests
SH = {'x-api-key': os.environ['SCAVIO_API_KEY'], 'Content-Type': 'application/json'}

def check_d2c(brand, product):
    data = requests.post('https://api.scavio.dev/api/v1/search',
        headers=SH, json={'query': f'best {product} 2026', 'country_code': 'us'}).json()
    for i, r in enumerate(data.get('organic_results', [])):
        if brand.lower() in r.get('title', '').lower():
            print(f'{brand} found at position {i+1}')
            return
    print(f'{brand} not found in top results')

check_d2c('GlowCo', 'organic face cream')
print('Cost: $0.005')

JavaScript Example

JavaScript
const SH = { 'x-api-key': process.env.SCAVIO_API_KEY, 'Content-Type': 'application/json' };
async function checkD2C(brand, product) {
  const data = await fetch('https://api.scavio.dev/api/v1/search', {
    method: 'POST', headers: SH,
    body: JSON.stringify({ query: `best ${product} 2026`, country_code: 'us' })
  }).then(r => r.json());
  const pos = (data.organic_results || []).findIndex(r => r.title.toLowerCase().includes(brand.toLowerCase()));
  console.log(pos >= 0 ? `${brand} at #${pos+1}` : `${brand} not found`);
}
await checkD2C('GlowCo', 'organic face cream');

Expected Output

JSON
Brand: GlowCo
Visibility: 50% (2/4 queries)
Avg position when found: 6.5
Cost: $0.020

Agent Readiness Audit: glowco.com
  [FAIL] llms.txt exists
  [PASS] Rich snippets on product pages
  [PASS] Sitemap indexed

Score: 2/3

Visibility check: 2/2 products found in SERP
  best organic face cream 2026             | #5
  best vitamin c serum 2026                | #8
Cost: $0.010

Related Tutorials

Frequently Asked Questions

Most developers complete this tutorial in 15 to 30 minutes. You will need a Scavio API key (free tier works) and a working Python or JavaScript environment.

Python 3.8+. requests library. A Scavio API key from scavio.dev. D2C site URL with product pages. A Scavio API key gives you 250 free credits per month.

Yes. The free tier includes 250 credits per month, which is more than enough to complete this tutorial and prototype a working solution.

Scavio has a native LangChain package (langchain-scavio), an MCP server, and a plain REST API that works with any HTTP client. This tutorial uses the raw REST API, but you can adapt to your framework of choice.

Start Building

Make your D2C brand visible to AI shopping agents. Structured data, llms.txt, and product feed optimization for agent traffic.