Llms.txt vs Robots.txt: What Each File Does and Doesn't Do

How llms.txt and robots.txt Differ at a Glance#

Robots.txt tells crawlers which paths they may request, while llms.txt offers language models a curated overview of useful pages. Neither file, on its own, determines whether an AI answer recommends a SaaS product.

That distinction challenges the common sequence of publishing llms.txt first and treating robots.txt as legacy SEO housekeeping. A blocked pricing page can't be fetched by a compliant answer bot, however polished an llms.txt summary sounds. Conversely, open crawl access doesn't tell a model which comparison page best explains a product category, pricing tier, or integration list.

The history of the llms.txt standard places RFC 9309 for robots.txt in September 2022 and Jeremy Howard's llms.txt proposal on September 3, 2024, which helps explain their different status. One is a formal Internet standard with defined crawler behavior. The other remains a community proposal, useful as documentation but not a protocol crawlers must follow.

Two practical consequences#

• No robots.txt means no stated crawl rules. Compliant crawlers receive no path policy from that file, so important SaaS pages and unwanted staging paths aren't separated by robots directives.
• No llms.txt means no documented requirement is missing. AI crawlers can still fetch accessible pages, sitemaps, and linked resources. The absent file removes an optional curation layer, not the underlying ability to crawl.

Publication isn't consumption, and crawl permission isn't recommendation.

The useful question isn't which filename sounds more modern. It's whether crawlers can fetch the right public evidence, whether that evidence is readable and current, and whether AI systems use it when buyers ask category questions. That's the same measurement discipline behind generative engine optimization, where access, source selection, and answer outcomes are tracked separately.

For a SaaS site, the priority order is therefore clear. Get robots.txt right first, treat llms.txt as a testable guide second, and judge both by server logs and AI answers rather than by whether the files exist.

How robots.txt Controls Crawler Access and Whether AI Bots Obey It#

Robots.txt sits at the domain root, conventionally at /robots.txt, and a compliant crawler fetches it before requesting site pages. RFC 9309 defines robots.txt as the Robots Exclusion Protocol, in which a site publishes rules that tell automated clients which URI paths they may access.

The mechanism is simple but easy to misread. A crawler identifies itself through a user agent, selects the group matching its name, and compares the requested path with that group's Allow and Disallow rules. A general group can set default policy, while a named group can give one search bot, answer bot, or training bot different instructions.

What the directives actually do#

• User-agent names the audience. The value must match the bot name stated in the vendor's own documentation, not a guessed label copied from a forum post.
• Disallow names paths not to request. A rule for a private app area, internal search results, or staging path asks compliant crawlers to skip those URLs.
• Allow carves out permitted paths. It's useful when a broader disallowed directory contains one public help article or product resource that should remain crawlable.

Google states that its crawlers fetch and parse robots.txt before crawling to determine which URL paths may be accessed, a point summarized alongside the protocol distinction in the source material for RFC 9309. That sequence matters because robots.txt operates before page fetching. Meta tags and on-page instructions arrive too late if the goal is to prevent a compliant crawler from requesting the page at all.

The limit is equally important. RFC 9309 doesn't create server-side access control, so a crawler that ignores the rules can still request a blocked resource. Robots.txt also doesn't authenticate people, hide a URL from someone who already knows it, or remove copies collected before a rule changed.

Practical rule: Treat robots.txt as a request to compliant crawlers, never as a lock on private SaaS content.

Three checks catch most operational failures. First, confirm /robots.txt returns a 200 status and readable plain text rather than an HTML error page. Second, test representative pricing, product, comparison, documentation, login, and app URLs against every named group, because one broad path rule can block assets or pages by accident. Third, compare each user-agent token with current vendor documentation, since AI companies may run separate crawlers for training, search indexing, and user-initiated retrieval.

An AI crawler checker can speed up the first pass, but server logs provide the compliance evidence. Extract AI bot user agents from the last 30 days, match each name to vendor documentation, and list every request for a disallowed path after the relevant rule went live. Record each bot and path combination as observed or not observed, rather than labeling an absent log line as proof of perfect compliance.

That test separates three situations that dashboards often blur. A bot may obey the rule and never request the path. It may not have visited during the window, leaving the result unobserved. Or it may request the path despite the rule, which calls for server controls if the material must stay private.

A public-page policy might allow a named search bot across product and pricing paths while disallowing account areas. A separate training-bot group might disallow the full site if the owner doesn't want public pages used for model training. Those are policy choices, not universal settings, and bot names should come directly from each vendor's documentation before deployment.

Content that must remain private needs authentication, authorization checks, paywalls, or server rules that return an appropriate denial. Robots.txt can reduce compliant crawling, but it can't protect customer data, unreleased pricing, or gated research by itself.

What llms.txt Is and How It Is Structured#

Llms.txt is a proposed Markdown file placed at /llms.txt or at a subpath such as /docs/llms.txt for documentation. The llms.txt specification states that a file applies to URLs under its own path and, when multiple files apply, agents should use the most specific one.

Jeremy Howard proposed the format in September 2024 as a concise guide for language-model systems. Its job is orientation. It can say what a SaaS product does, which pages explain pricing, and where the authoritative API reference lives, without forcing a model to infer site structure only from navigation and page HTML.

Required and optional parts#

The specification makes one element mandatory. Everything else supports that heading with context and links.

• One H1 is required. It contains the project or site name, such as the SaaS product name used consistently across the homepage and product pages.
• A blockquote summary is optional. One or two sentences can state the category, target buyer, and primary use case in plain language.
• Explanatory paragraphs add context. Short text can clarify product boundaries, supported platforms, or documentation scope before the links begin.
• H2 sections organize link lists. Useful SaaS groupings include Product, Pricing, Comparisons, Documentation, and Company, provided each linked page is public.

Each list item must contain a Markdown hyperlink in the form name, with optional notes after a colon. A concrete item could read: Pricing: Plan limits, billing terms, and feature differences. The note helps a reader or model understand why that destination matters, but it shouldn't introduce claims that the destination page doesn't support.

A minimal file therefore needs little more than an H1, a brief summary, and a small set of canonical links. A documentation subpath file can be narrower, pointing to the quickstart, authentication guide, API reference, and changelog for URLs under that docs path. That specificity is the format's strength when a site serves buyers, administrators, and developers with different information needs.

The limit is built into the format. Llms.txt defines no Disallow, Allow, or crawl-delay directives, so it can't restrict fetching, grant permission, or override robots.txt. Listing a page doesn't guarantee retrieval, citation, or recommendation, and omitting a page doesn't hide it from crawlers that discover it elsewhere.

Before publication, open the file in a browser and test it like any other public resource. Confirm it returns readable Markdown rather than a styled error page, every linked URL is public and canonical, and the summary uses the same product name and category as the rest of the site. Stale comparison links, redirected pricing pages, or a category label the sales site no longer uses create ambiguity instead of useful guidance.

How Often AI Systems Request llms.txt#

Adoption figures show a wide gap between the two files before request behavior even enters the picture. Valid llms.txt files appeared on 2.13% of desktop sites and 2.10% of mobile sites in 2025, according to an analysis citing HTTP Archive's Web Almanac.

Read row by row, the table supports narrower conclusions than launch announcements usually draw. Robots.txt is the established baseline, while llms.txt remains uncommon as a valid published file. GPTBot's growing presence in robots.txt shows that site owners increasingly express AI policy through the older protocol, not that every named rule is correctly implemented.

The llmscentral row also needs careful scope. It describes 2,147 websites across 15 industries and reports that 14% had no llms.txt file, which is a sample-specific adoption snapshot rather than a whole-web census. Different samples and validation rules can produce very different publication rates, especially when some responses return a success status without valid Markdown.

A published llms.txt file proves availability. It doesn't prove an AI system fetched, understood, or used it.

The Ahrefs result is the strongest reason for measurement. In that analysis, AI retrieval bots generated only 1.1% of the requests that did occur, even though many files existed. SEO tools, uptime checks, researchers, and generic crawlers can request /llms.txt without affecting a single buyer-facing AI answer.

Site owners can test consumption directly. Pull server logs for /llms.txt and /robots.txt over the same period, separate documented AI bot user agents from generic tools, and compare counts by bot and date. Record periods with no confirmed AI request as not observed, then compare any later citation changes against that baseline instead of crediting publication day by default.

Which File to Act On for Your Situation#

Start with robots.txt because it can change what compliant crawlers fetch next. Add llms.txt only after the pages it would promote are public, crawlable, and worth citing for SaaS buying questions.

Caveat: robots.txt governs future fetches by compliant crawlers and doesn't remove content already collected, while llms.txt shouldn't be credited with raising citations until log and answer tests support that claim.

The table's pattern is consistent. Robots.txt handles permission, including separate decisions for search, answer, and training bots. Llms.txt handles prioritization among public material, such as choosing the current pricing page over an older feature announcement or the main comparison page over a campaign variant.

For gated SaaS content, that boundary prevents a costly mistake. An llms.txt link to a public overview is reasonable, but listing gated URLs won't make them accessible or protect them. Server authentication protects the resource, robots.txt states crawler policy, and llms.txt stays silent about material that shouldn't be promoted.

Teams that need a starting draft can use a free AI visibility files check that writes an llms.txt and Organization, WebSite and FAQ structured data from the live site, plus robots.txt lines when a change is needed. The draft still needs human review against current pricing, product names, and access rules before anything goes live.

Citation work also extends beyond root files. Clear product pages, current comparison pages, third-party listings, and direct answers to buyer questions give AI systems evidence they can retrieve and attribute, consistent with practical guidance on how to get cited in Google AI Overviews. Root files support that evidence, but they don't substitute for it.

If there's a need to see how a brand is named across ChatGPT, Google AI Overviews, Google AI Mode, Gemini and Copilot, Kliqo tracks those engines at kliqo.ai.

How to Set Up Both Files This Week#

Work in access order, not filename order. Confirm what crawlers can fetch, fix permission mistakes, publish a narrow llms.txt only when its destinations pass the same access test, and save a baseline before claiming impact.

1. Run a crawler access check first, because changes without a baseline can't be evaluated. Kliqo AI's free AI crawler access check shows whether ChatGPT, Claude, Perplexity, Google and other AI crawlers can read a site, covering robots.txt health, the sitemap and the homepage without JavaScript, with the exact fix. The linked AI crawler access check gives a starting list of access problems to verify against live URLs.
2. Review every Disallow rule, because broad paths often block intended SaaS evidence. Test pricing, product, comparison, documentation, stylesheet, and script paths under each named user-agent group. Keep private app and account paths blocked for crawlers, but protect them with authentication as well.
3. Set policy per bot from vendor documentation, because training and search are different decisions. Allow search and answer bots on public buying pages when AI visibility is a goal. Decide separately whether training bots should be disallowed, and record the business reason beside each rule.
4. Confirm priority pages work without JavaScript, because permission has little value if fetched text is empty. Each important page should return 200, expose readable product and pricing text in the initial HTML, and avoid sending crawlers through unnecessary redirects. Fix blocked resource paths that prevent rendering or interpretation.
5. Draft llms.txt in Markdown, because a short canonical guide is easier to maintain than a site dump. Use one H1, a one-sentence blockquote summary, and H2 sections for product, pricing, comparisons, and documentation. Add one-line notes that match the destination pages exactly.
6. Cross-check both files, because llms.txt shouldn't promote URLs robots.txt asks crawlers to skip. Open every listed link, confirm it returns the intended public page rather than a redirect chain or login screen, and remove stale campaign pages. Repeat the crawl test after deployment rather than trusting the staging copy.
7. Record a 30-day baseline, because recommendation changes need a comparison point. Save log counts for both files by AI user agent and run five SaaS buying questions across the tracked AI engines. Repeat after 30 days and report fetches, citations, and recommendations as observed or not observed.

Change one layer at a time when testing matters, so a later citation shift isn't credited to the wrong fix.

A clean result at the end of the week is modest but defensible. Robots.txt states the intended policy, key pages are fetchable and readable, llms.txt points only to approved public evidence, and logs show whether documented AI bots requested either file.

FAQ#

Does llms.txt replace robots.txt#

No. Robots.txt states which paths a crawler may request. llms.txt is a Markdown guide that points an AI client toward selected pages. One states access policy, while the other organizes supporting evidence. A site can use both, but neither file guarantees that an AI system will read it.

Does publishing llms.txt improve AI recommendations#

Publication alone is weak evidence. The file can sit unrequested during a measurement window, and a fetch does not prove a model used it in an answer. The meaningful test is whether server logs show requests by documented AI user agents and whether tracked buying prompts record a later citation or recommendation change.

Is robots.txt enforced#

No. Compliance is voluntary, so a crawler can ignore a Disallow rule. Treat the file as policy rather than security. Authentication and server-side authorization create the reliable boundary for private SaaS content, while robots.txt rules and request logs show whether declared AI bots respect that policy.

Where should llms.txt go#

Use the site root for a site-wide guide. A subpath can hold a scoped guide for an area such as documentation, provided its entries stay relevant to that area. Keep the descriptions consistent with the destination pages so crawlers do not encounter conflicting versions of the same evidence.

Should GPTBot be blocked#

Decide whether the goal is to restrict training access, AI search retrieval, or both. GPTBot is one OpenAI user agent, and blocking it does not set policy for other vendors' bots. Check OpenAI's current crawler documentation, write separate rules where appropriate, and compare crawler access and tracked answers after the change.

---

The practical order is access first, measurement second. Kliqo AI shows whether major AI crawlers can read a site, tracks visibility, recommendations and cited sources across tracked engines, and connects suggested fixes to recorded answers. Its free AI crawler access check covers ChatGPT, Claude, Perplexity, Google and other AI crawlers, including robots.txt health, the sitemap and the homepage without JavaScript, then gives the exact fix. Visit Kliqo AI to check crawler access before judging whether llms.txt changed what AI tells buyers.