llms.txt: Getting Found by AI Search Engines

llms.txt is a simple Markdown file you place at the root of your website (at /llms.txt) that gives large language models a clean, condensed, machine-readable map of your most important content. The idea: instead of an AI system having to wade through your full HTML, menus, cookie banners and scripts, it gets a tidy summary with links to your key pages and a short description of each one.

That sounds sensible. But before you spend a morning on it, it is worth being honest about what llms.txt does and does not do in 2026. It is not an official standard, and the parties you probably want to reach do not all read the file today. In this article you will learn what llms.txt actually is, how it differs from robots.txt and sitemap.xml, how widely it is really used, and whether it is worth doing for your business.

Many businesses arrive at this topic through the bigger picture: getting found by AI. If you are not across that yet, start with our guide on getting found in AI search. This article adds to that foundation, it does not replace it.

How is it different from robots.txt and sitemap.xml?

You probably already know two files that sit at the root of a website: robots.txt and sitemap.xml. llms.txt looks similar, but it has a different audience and a different purpose. Putting the three side by side helps.

robots.txt has existed since the 1990s and is an instruction to crawlers: which parts of your site they may and may not visit. It says nothing about content. sitemap.xml is a list of all the URLs on your site, meant to help search engines miss nothing. It too says nothing about the meaning of a page. llms.txt fills that gap: it describes, in plain language, what your most important pages are and why they matter, specifically for language models.

So the difference is not only the audience but also the form. robots.txt and sitemap.xml are instruction files for machines. llms.txt is meant to be read: it is written in Markdown, with headings, links and short descriptions, so a language model quickly grasps what your site is about. Alongside /llms.txt there is often a fuller variant, /llms-full.txt, which contains the complete text of your key pages instead of just links and summaries.

One thing to remember: llms.txt does not replace either of the other two files. It sits alongside them. You still need a good sitemap and a correctly configured robots.txt. llms.txt is an extra layer, not an alternative.

Does llms.txt actually work in 2026?

This is where the honest account matters, because there is a lot of noise here. The short answer: llms.txt is used, but nowhere near as widely or as officially as some articles suggest.

Start with the facts on adoption. A study by SE Ranking across roughly 300,000 domains found that about 10.13% of sites had an llms.txt file. Ten percent is more than nothing, but it is far from a norm. There is a second wave underway: SaaS companies, publishers and marketing sites in particular are putting the file in place. For WordPress users, it helps that the popular SEO plugin Yoast can now auto-generate an llms.txt, which lowers the barrier.

But adoption says nothing about whether it gets read. And that is the crucial point. The major AI search crawlers you probably want to reach do not fetch /llms.txt in practice today. Think of GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot and Google-Extended: these crawl your HTML directly, just like an ordinary search engine. They are not built to look for a separate Markdown file first.

On top of that, llms.txt is not an official standard. It is a community proposal, not ratified by the IETF or W3C, and as of early 2026 there is no formal endorsement from Google, OpenAI, Anthropic, Mistral or Meta. That does not make it worthless, but it does mean nobody guarantees the file will be used or that it will keep working this way in the future.

So where is it routinely used? In developer tools and documentation tools. Coding assistants and IDE agents such as Cursor, Claude Code and GitHub Copilot do fetch /llms.txt and /llms-full.txt to quickly understand what a product or its documentation is about. That makes llms.txt today more of a business-to-agent (B2A) play than a guaranteed ranking factor for AI search results. If your business offers software, an API or extensive documentation, the odds that the file pays off are higher than for a local service provider.

The honest summary: llms.txt is promising but unproven as an SEO tool. It is a bet on a future that may arrive, not a button that improves your discoverability today. Anyone who promises you that an llms.txt file will raise your rankings in ChatGPT or Perplexity is going further than the evidence currently supports.

How do you set up an llms.txt?

Suppose you want to do it anyway, which most sites can reasonably justify. Setting it up is deliberately simple. You do not need to be a developer to put the basics in place.

Step 1: place the file at the root. The file must be reachable at yourdomain.com/llms.txt, so right next to your robots.txt and sitemap.xml. Not in a subfolder, not under a page. The root, otherwise tools will not find it.

Step 2: write it in Markdown. Start with an H1 holding your company name, followed by one short line explaining what you do. Below that, add sections listing your most important pages as links, each with a one-sentence description. Think of your service pages, your key blog articles, your contact and about pages. Keep it concise: the goal is a clean map, not a copy of your whole site.

Step 3: use descriptions that are accurate. Write one clear sentence per link explaining what the page covers. This is literally the context a language model receives, so be concrete. Not "our services" but "web development for SMBs, including pricing and process".

Step 4: consider an llms-full.txt. For the fuller variant you paste the complete text of your key pages into a single file. That is most useful if you have documentation or in-depth guides. For a small service site, /llms.txt alone is often enough.

Step 5: keep it current. An outdated llms.txt is worse than none, because it sends tools to pages that no longer exist. Update it whenever you add or remove important pages. If you run WordPress with Yoast, the plugin can generate and update the file automatically, which takes this maintenance off your hands.

One last practical point: a clean, machine-readable file starts with a clean, fast site. If your HTML is messy, your load times are slow or your structure is illogical, you will not fix that by adding a text file. A solid technical foundation is the real basis, and you can read more about that in our articles on website development costs in 2026 and Core Web Vitals and site performance.

Is it worth it for your business?

Time for a sober verdict. Setting up llms.txt costs little time, carries almost no risk, and has a possible upside that may grow in the future. That is a favorable ratio. But the emphasis is on possible: do not expect miracles, and certainly not an immediate jump in your discoverability.

Who is it useful for? If you offer software, an API, a SaaS product or extensive documentation, the odds are highest that the file already does something today, because coding and documentation tools are exactly what read it. If you offer a more traditional service, llms.txt is mainly cheap insurance: you are ready for the moment AI search crawlers do start taking the file seriously, should that happen.

Who is it less urgent for? If you do not yet have a clean site structure, structured data and FAQ sections, spend your energy there first. Those are the things AI search engines do read today. llms.txt is the finishing touch on good GEO, not the starting point. Think of it the same way you think about context you give an AI system: the cleaner and clearer it is, the better the result. We cover that principle more broadly in context engineering explained for business.

The middle road for most SMBs: set up an llms.txt as part of a broader GEO approach, not as a standalone trick. Combine it with structured data, clear content, consistent business information and a technically healthy site. If you serve multiple language markets, make sure your content is accurate per language, because the same logic applies there, as we explain in AI translation for multilingual content. The combination makes the difference, not any single file.

Staying honest pays off. An llms.txt is cheap, harmless and possibly useful. That is precisely what it is, no more and no less. Treat it that way and you take a sensible step without fooling yourself.

Want your site ready for both Google and the new generation of AI tools, with a clean structure, fast load times and the right technical foundation? Take a look at our web development for SMBs and we will work out together where the quickest wins are, from llms.txt to structured data.