Generate an llms.txt
for any website
Enter a domain and get a valid llms.txt file: the site's key pages, grouped into sections, ready to publish.
A free llms.txt file generator
An llms.txt file gives language models and AI agents a short, curated map of your site: what it is, and which pages matter. Writing one by hand means auditing your sitemap and deciding what to keep. This tool does that pass for you from the site's sitemaps and homepage, and returns a file in the llmstxt.org format.
Use it to publish an llms.txt for your own site, to see how a competitor's site is structured, or as a starting point for an AI agent that needs a site's key pages. To crawl a whole site into Markdown, see the website crawler API.
How to generate an llms.txt file
- 1
Enter the domain
Type the site's domain, like example.com, or paste its homepage URL. Any public site works.
- 2
Generate the file
We read the site's sitemaps and homepage links, then group the pages into sections and pull the site name and description from the homepage.
- 3
Publish it at /llms.txt
Copy or download the file, review the sections, and upload it to the root of your site so it is served at yourdomain.com/llms.txt.
Frequently asked questions
What is an llms.txt file?+
llms.txt is a proposed standard (llmstxt.org) for a Markdown file at the root of a website that tells language models what the site is and where its most useful pages are. It starts with an H1 site name, an optional blockquote summary, and H2 sections that list links in the form - [name](url).
Does the generated file follow the llmstxt.org format?+
Yes. The output has the required H1 with the site name, a blockquote summary taken from the homepage meta description, H2 sections of Markdown links, and an Optional section for secondary pages like careers, legal and contact, which the format reserves for links an agent can skip.
How are the pages chosen?+
The tool maps the site from its sitemaps and the links on its homepage, removes duplicates, assets and translated copies, and groups pages by their first path segment. Key pages such as pricing, docs and features are listed first, and large sections like a blog are balanced so they do not crowd out the rest. The free file is capped at 50 links.
Where do the link titles come from?+
Titles are built from each page's URL slug, so /blog/how-we-ship becomes How we ship. That keeps generation to two API calls. Review the titles before publishing and rename any that read awkwardly.
What is llms-full.txt?+
llms-full.txt is a common companion file that holds the full text of the pages listed in llms.txt, so a model can read the content without visiting each link. With a free Scavio account you can build it with the crawl API, which extracts every page to Markdown in one job.
Is the llms.txt generator free?+
Yes. You get 5 free llms.txt files a day with no account. Sign up free to keep generating on your own Scavio credits.
Where do I put llms.txt on my site?+
Serve it at the root, for example https://example.com/llms.txt. The format also allows files at subpaths such as /docs/llms.txt, which cover the pages under that path.
Does llms.txt replace robots.txt or sitemap.xml?+
No. robots.txt controls crawler access and sitemap.xml lists every URL for search engines. llms.txt is a short, curated, human-readable guide for language models and agents. Most sites keep all three.
Can I generate llms.txt from code?+
Yes. POST /api/v1/extract/map returns a site's URLs from its sitemaps and start page in one call, and POST /api/v1/extract reads any page as Markdown. The same calls power this tool.
What if the site has no sitemap?+
The tool also collects links from the homepage, so most sites still produce a usable file. If a site blocks automated access entirely, you will see an error and nothing counts against your daily free runs.
Build with the Extract API
This tool runs on Scavio's map and extract endpoints. Map any site's URLs, read pages as Markdown, and crawl whole sites from your own pipelines and agents.