llms.txt and making your site AI-readable

By Marek Fodor4 min read

Language models now read websites to answer people’s questions. The trouble is that an ordinary web page is messy for a machine: navigation, ads, cookie banners, footers, and scripts make up most of the HTML, while the real content gets lost in the noise. The model has to guess what matters.

The llms.txt proposal answers this simply. It is a single Markdown file at the root of your domain that offers a clean, curated map of your most important content. No clutter, no layout, just links and short descriptions. In this post we explain what it is, how it differs from robots.txt and sitemap.xml, how to write one, and why staying level-headed about it is the healthy stance.

What llms.txt is

llms.txt is a community proposal for a file format, not an official standard. The idea is that you place the file at https://yourdomain.com/llms.txt and write it in plain Markdown, which a model reads directly without parsing HTML.

  • A curated map, not a full list. You pick the pages you actually want the model to know: products, documentation, key articles.
  • Short descriptions. Each link gets a single sentence that tells machines and people alike what they will find on the page.
  • Readable by human and model. Because it is Markdown, you can open it in a browser and immediately see what it contains.

The goal is not to replace your website. It is a shortcut that saves the model work and lowers the chance it invents something inaccurate.

How it differs from robots.txt and sitemap.xml

These three files are easy to confuse, yet each solves a different problem. They are not rivals, they complement each other.

  • robots.txt tells crawlers where they may and may not go. It is a list of access rules, not content.
  • sitemap.xml lists every URL for search engines. It is complete, machine-readable, and unselective: it aims to cover the whole site.
  • llms.txt is the opposite of a sitemap in spirit. It is deliberately small and curated. It does not try to list everything, it tries to show what matters and give it context in natural language.

Put another way: robots.txt is the gatekeeper, sitemap.xml is the catalog, and llms.txt is a short guide written for a reader in a hurry.

How to write one

The format is deliberately loose. Start with a heading for the project name, add a short paragraph about what you do, then sections of links. A small example:

# Vyxos

> A Slovak software studio that builds simple custom software for businesses.

## Start here
- [Services](https://vyxos.eu/services): what we build and for whom.
- [About](https://vyxos.eu/about): who we are and how we work.

## Blog
- [Why we build simple software](https://vyxos.eu/blog/why-we-build-simple-software): our take on complexity.

## Contact
- [Get in touch](https://vyxos.eu/contact): how to start working with us.

Follow a few principles. Link to the pages you want the model to cite. Write descriptions plainly, without marketing phrases. Keep the file short and current, or the advantage turns into another source of inaccuracy. If you have extensive documentation, the proposal also allows a variant where individual pages are offered as clean Markdown.

The current state: a proposal, not a guarantee

Honesty belongs here. llms.txt is a proposal from late 2024 that gained attention, but it is not yet a universal standard. There is no guarantee the major model operators will read it, and its future is uncertain.

  • Adoption is early. Part of the community has embraced the file, but broad support across models is not confirmed.
  • It is not a substitute for a good website. If a model ignores the file, it falls back to your normal content. That is why your normal content has to be sound.
  • The risk is low. Writing one Markdown file takes a few minutes and breaks nothing. If the format catches on, you are ready. If it does not, you have lost little.

Our position: llms.txt is a cheap experiment with limited downside, not an obligation. Add it if it makes sense for you, but do not build a strategy on it.

The broader principle: a site that can be read

llms.txt is just one tool within a larger idea. The sites that models and search engines handle well are the ones that are readable by nature. This was true long before AI arrived, and it still holds.

  • Clean HTML. Semantic tags like article, nav, and h1 through h3 tell a machine what is what. A tangle of div blocks says nothing.
  • Content in the HTML, not hidden behind JavaScript. If text only appears after a script runs, some crawlers may never see it. Important content belongs in the page directly.
  • Accessibility helps machines too. Image descriptions, a clear heading structure, and meaningful links help screen readers and language models at the same time.

If you want to go deeper, this connects to how AI assistants cite you and how structured data and schema work in the AI era.

Our approach here is the same as everywhere. We favor simple, standard, and robust solutions over hype. llms.txt fits precisely because it is small, readable, and easy to maintain. If it catches on, you will be ready. And even if it does not, a site that is clean and clear serves people and machines all the same.