Is llms.txt Still Worth It After Google GEO Guide?

Google published its GEO (Generative Engine Optimization) guide in April 2026 and explicitly stated that llms.txt is not required for Google Search or AI Overviews. The r/aeo community immediately split into two camps: those who stopped maintaining llms.txt and those who kept going. The second group is right, but for reasons that have nothing to do with Google.

What Google Actually Said

The relevant section of Google's GEO guide says: "There is no special file or protocol required for Google's AI features to access and understand your content. Standard SEO best practices, including well-structured HTML, schema markup, and sitemap.xml, are sufficient for Google's systems."

This is accurate for Google specifically. Googlebot has its own rendering pipeline, its own JavaScript execution engine, and its own understanding of page structure. It does not need a separate text file to understand your site. For Google, llms.txt adds zero ranking benefit.

But Google Is Not the Only LLM Provider

Here is what Google's guide does not mention: other LLM providers actively use llms.txt. Anthropic maintains llms.txt files for its own documentation. Perplexity has indicated it checks for llms.txt when crawling. Mistral's documentation references llms.txt as a supported format. The emerging convention is real, just not for Google.

The non-Google LLM ecosystem is significant and growing. When a user asks Perplexity a question about your product, Perplexity crawls your site. When a developer asks an AI coding assistant about your API, the assistant may check your llms.txt for a concise description. These interactions happen millions of times daily and drive real traffic and conversions.

What llms.txt Contains

The llms.txt specification is simple. It is a plain text file at your domain root that provides a structured summary of your site optimized for LLM consumption. There are two variants:

• llms.txt: A concise overview with links to key pages, suitable for limited context windows.
• llms-full.txt: A comprehensive version with more detail about each page, for LLMs that can handle larger context.

# Example llms.txt

# Scavio
> Unified search API for AI agents. Google, Amazon, YouTube, TikTok, Walmart, Reddit data as structured JSON.

## Docs
- [API Reference](https://scavio.dev/docs/api): REST API endpoints, authentication, response formats
- [MCP Integration](https://scavio.dev/docs/mcp): Model Context Protocol setup for Claude, Cursor, VS Code
- [Python SDK](https://scavio.dev/docs/python): pip install scavio, quickstart, async usage
- [Pricing](https://scavio.dev/pricing): $0.005/credit, 250 free/mo, $30/mo for 7K credits

## Blog
- [LangChain Web Search](https://scavio.dev/blog/langchain-web-search): Tutorial for adding search to LangChain agents
- [MCP Setup Guide](https://scavio.dev/blog/mcp-integration): Connect to any MCP-compatible client

The Cost of Maintaining llms.txt

This is where the argument gets practical. The cost of maintaining llms.txt is negligible. If your site has stable core pages (docs, pricing, product pages), the file changes maybe once a month. The initial creation takes 30-60 minutes. Monthly updates take 5-10 minutes.

The question is not "is llms.txt worth a huge investment?" It is "is 10 minutes per month worth potential visibility in non-Google LLM products?" For any business that gets traffic from AI assistants, the answer is obviously yes.

Automating llms.txt Generation

You can automate llms.txt generation as part of your build pipeline. Scan your sitemap, extract page titles and descriptions, and generate the file automatically:

import xml.etree.ElementTree as ET
import requests

def generate_llms_txt(sitemap_url, site_name, site_description):
    """Generate llms.txt from sitemap."""
    resp = requests.get(sitemap_url)
    root = ET.fromstring(resp.content)

    ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
    urls = [loc.text for loc in root.findall(".//sm:loc", ns)]

    lines = [
        f"# {site_name}",
        f"> {site_description}",
        "",
    ]

    # Group by section
    sections = {}
    for url in urls:
        path = url.split("/", 3)[-1] if "/" in url else ""
        section = path.split("/")[0] if "/" in path else "pages"
        if section not in sections:
            sections[section] = []
        sections[section].append(url)

    for section, section_urls in sections.items():
        lines.append(f"## {section.title()}")
        for url in section_urls[:20]:  # Limit per section
            lines.append(f"- [{url.split('/')[-1]}]({url})")
        lines.append("")

    return "\n".join(lines)

What the r/aeo Discussion Revealed

The r/aeo thread about llms.txt after the Google guide had over 80 comments. The most useful data point came from a SaaS founder who tracked referral sources from AI products. Before adding llms.txt, they got approximately 200 visits/month from non-Google AI products (Perplexity, various coding assistants). After adding llms.txt with a clear product description, that number rose to approximately 340 visits/month within 6 weeks.

This is not a controlled experiment and correlation does not imply causation. But a 70% increase in non-Google AI referrals after a 30-minute investment is a reasonable signal. The key insight from the thread: llms.txt works best when it clearly describes what your product does, not just links to pages.

The Strategic View

Search is fragmenting. Google still dominates traditional search, but AI-powered answer engines are capturing an increasing share of informational queries. Optimizing only for Google means ignoring a growing channel.

llms.txt is a low-cost hedge. It takes minimal effort to maintain, has zero downside (Google ignores it; it does not hurt rankings), and has measurable upside for non-Google AI visibility. The teams that dropped llms.txt after the Google guide are optimizing for one platform while ignoring the rest of the market.

Practical Recommendation

Maintain both llms.txt and llms-full.txt. Keep them updated when you ship new features or change pricing. Focus the content on clear, factual descriptions of what your product does, not marketing copy. LLMs parse facts better than superlatives.

Do not expect Google ranking benefits. Do expect better representation in Perplexity, AI coding assistants, and other LLM-powered products that are growing rapidly. The effort is trivial and the downside is zero.