Crawlspace

All topics

Llms.txt

41 quoted answers, each with its source

The short answer

Google added a note to its AI optimization guide on June 15, 2026 clarifying Google Search's usage of llms.txt files, and as of July 10, 2026, Google Search Central states that Google Search ignores llms.txt files and that creating or maintaining them neither harms nor helps visibility or rankings in Google Search, while Chrome for Developers Lighthouse documentation instructs site owners to create an llms.txt file in the site root and describes the file as an optional emerging convention. Ahrefs log analysis reported on June 16, 2026 across 137,000 domains found 97% of llms.txt files received zero requests, only about 1,100 of roughly 38,000 valid files received any traffic, and SE Ranking's analysis of 300,000 domains showed no connection between having an llms.txt file and AI citation frequency. On August 7, 2026, Search Engine Journal published a statement attributed to Google's John Mueller saying no AI system currently uses llms.txt and that server logs make this obvious. As of August 10, 2026, llmstxt.org states llms.txt is used most heavily for software documentation, where coding agents follow the files to find API references and tutorials, and the version 2 proposal removed the special mechanical meaning of the Optional section and allows replacing the file extension.

Every part of this comes from the quotes below. How we write these.

What the rules actually say14

Quoted straight from the people who make or enforce the rules.

#

Version 1 of the llms.txt proposal specified only one URL form for markdown versions of pages, with .md appended to the full page URL.

first-party
1 quote from 1 source
v1 specified one URL form for markdown versions, .md appended to the full page URL (page.html.md). Some publishing tools instead replace the extension (page.md), so v2 allows both.
#

The llms.txt proposal at llmstxt.org states that…

  • the expectation was that llms.txt would mainly be useful for inference rather than training.
  • llms.txt has mainly been used for inference rather than training.
first-party
1 quote from 1 source
Our expectation was that llms.txt would mainly be useful for inference rather than training, and that is how it has been used, though training runs could take advantage of the information too.
#

The llms.txt proposal at llmstxt.org states that…

  • llms.txt files are used most heavily for software documentation.
  • coding agents follow llms.txt files to find API references and tutorials.
first-party
1 quote from 1 source
llms.txt files are used most heavily for software documentation, where coding agents follow them to find API references and tutorials.
#

Google Search Central's documentation states, under a heading naming LLMS.txt files, that Google Search itself does not use machine readable files, AI text files, markup or Markdown created to appear in Google Search.

first-party
1 quote from 1 source
LLMS.txt files and other "special" markup: You don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them.
#

Google Search Central's documentation states that…

  • creating and maintaining LLMS.txt files will neither harm nor help a site's visibility or rankings in Google Search.
  • Google Search ignores LLMS.txt files.
first-party
1 quote from 1 source
It's completely fine if you decide to create and maintain LLMS.txt files (or other similar files) for other services or systems that use these files. Doing so will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them.
#

Google's Chrome for Developers Lighthouse documentation describes the llms.txt file as…

  • an emerging convention.
  • used to provide a machine-readable summary of a website's content.
  • designed for LLMs and AI agents.
first-party
1 quote from 1 source
The llms.txt file is an emerging convention used to provide a machine-readable summary of a website's content, specifically designed for LLMs and AI agents.
#

Google's Chrome for Developers Lighthouse documentation…

  • states that providing an llms.txt file is optional at the moment.
  • marks the audit Not Applicable when the llms.txt file returns a 404.
first-party
1 quote from 1 source
If the file is not provided by the server (resulting in a 404), the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment.
#

Google's Chrome for Developers Lighthouse documentation states that without an llms.txt file, agents may spend more time crawling a site to understand its high-level structure and primary content.

first-party
1 quote from 1 source
Without this file, agents may spend more time crawling the site to understand its high-level structure and primary content.

Measured finding15

Something someone actually measured, with the method attached.

#

Search Engine Journal reports that Ahrefs…

  • analyzed logs from 137,000 domains.
  • found that 97% of llms.txt files got zero requests.
medium confidencetrade press
1 quote from 1 source
Ahrefs analyzed logs from 137,000 domains and found 97% of llms.txt files got zero requests. No bots, no humans.
#

Search Engine Journal reports that…

  • around 28% of the 137,000 domains in the Ahrefs llms.txt analysis publish an llms.txt file.
  • llms.txt adoption on the broader web is likely lower than the 28% rate in the Ahrefs llms.txt analysis because Ahrefs' customers are more technical.
medium confidencetrade press
1 quote from 1 source
Around 28% of 137,000 domains publish an llms.txt file, but since Ahrefs' customers are more technical, the actual adoption on the broader web is likely lower.
#

Search Engine Journal reports that in the Ahrefs llms.txt analysis…

  • roughly 38,000 domains had valid llms.txt files.
  • only about 1,100 of the roughly 38,000 domains with valid llms.txt files received any traffic.
medium confidencetrade press
1 quote from 1 source
Of roughly 38,000 domains with valid files, only about 1,100 received any traffic.
#

In the Ahrefs llms.txt analysis, AI retrieval bots linked to ChatGPT and Perplexity made up 1% of requests to llms.txt files.

medium confidencetrade press
1 quote from 1 source
Of files with requests, 96% came from bots, mostly non-AI. AI retrieval bots linked to ChatGPT and Perplexity made up 1%.
#

Search Engine Journal reports that in the Ahrefs llms.txt analysis, 12% of requests to llms.txt files came from tools that audit, scan or study the files rather than use them.

medium confidencetrade press
1 quote from 1 source
The report found 12% of requests from tools that audit, scan, or study llms.txt files rather than use them.
#

Search Engine Journal reports that in the Ahrefs llms.txt analysis, Slackbot alone fetched llms.txt files more often than PerplexityBot did.

medium confidencetrade press
1 quote from 1 source
Slackbot alone fetched llms.txt files more often than PerplexityBot did.
#
  • The Ahrefs llms.txt analysis measured requests rather than whether bots acted on what they fetched.
  • Every figure in the Ahrefs llms.txt analysis report is a ceiling.
medium confidencetrade press
1 quote from 1 source
Every figure in this report is a ceiling. Ahrefs measured requests, not whether bots acted on what they fetched.
#

Requests for /llms.txt paths returning 404 errors drew no AI traffic in the Ahrefs analysis.

medium confidencetrade press
1 quote from 1 source
Requests for /llms.txt paths with 404 errors drew no AI traffic.
#

The Chrome Lighthouse llms.txt audit generated…

  • about 22 requests across the Ahrefs dataset.
  • roughly 1 in 1,000 requests across the Ahrefs dataset.
medium confidencetrade press
1 quote from 1 source
The Chrome Lighthouse llms.txt audit, which reignited the llms.txt debate in May, generated about 22 requests across the dataset, roughly 1 in 1,000.
#

SE Ranking's analysis of 300,000 domains showed no connection between having an llms.txt file and AI citation frequency.

low confidencetrade press
1 quote from 1 source
SE Ranking's earlier analysis of 300,000 domains showed no connection between having llms.txt and AI citation frequency.

Practical effect3

What is observed to happen in practice, as distinct from what is documented.

#
  • Search Engine Journal published a statement attributed to Google's John Mueller about llms.txt.
  • The statement attributed to Google's John Mueller says that no AI system currently uses llms.txt.
  • The statement attributed to Google's John Mueller says that server logs make it obvious that no AI system currently uses llms.txt.
medium confidencetrade press
2 quotes from 1 source
"FWIW no AI system currently uses llms.txt, [..] It's super-obvious if you look at your server logs. The consumer LLMs / chatbots (the ones that SEOs want traffic from) will fetch your pages – for training and grounding, but none of them fetch the llms.txt file. Maybe they will tomorrow? Maybe I'll win in the lottery tomorrow?"
John Mueller, Google

Changed on4

A dated change. This is where advice quietly stops being true.

#

The llms.txt proposal published at llmstxt.org was revised to version 2 in August 2026.

first-party
2 quotes from 1 source
v2 (August 2026)
The original llms.txt proposal was published in September 2024, when the idea that language models would routinely read websites was still speculative.
#

Version 2 of the llms.txt proposal removed the special mechanical meaning of the Optional section.

first-party
1 quote from 1 source
The context-expansion tooling is no longer part of the proposal, and with it goes the special meaning of the Optional section, which told those tools what to omit. Optional sections are still allowed, and remain a useful convention for secondary links, but they no longer carry mechanical semantics.
#

Version 2 of the llms.txt proposal allows replacing the extension.

first-party
1 quote from 1 source
v1 specified one URL form for markdown versions, .md appended to the full page URL (page.html.md). Some publishing tools instead replace the extension (page.md), so v2 allows both.
#

Google added a note to its AI optimization guide clarifying Google Search's usage of llms.txt files on June 15, 2026.

first-party
4 quotes from 1 source
June 2026
June 15
Clarifying guidance on llms.txt files
What: Added a note to the AI optimization guide clarifying Google Search's usage of llms.txt files.

Disputed between sources2

Sources that do not agree, presented without picking a winner.

#
  • Google's Chrome for Developers Lighthouse documentation instructs site owners to create an llms.txt file in their website's root directory.
  • Google Search Central's guide to optimizing for generative AI features on Google Search tells site owners they can ignore creating AI text files like llms.txt.
first-party
3 quotes from 2 sources
Create an llms.txt file and place it in the root directory of your website (for example, https://example.com/llms.txt). The file should follow the llms.txt specification and provide a concise Markdown summary of your site's purpose and key links.
Optimizing your website for generative AI features on Google Search
Prioritize effective SEO strategies over "AEO/GEO hacks": For Google Search, you can ignore tactics like "chunking" content, creating unnecessary AI text files (like llms.txt), or pursuing inauthentic mentions.

Often said, but we could not find it in the rules3

Lots of people say this. We went looking for an official source that backs it up and could not find one, so treat it as something people repeat rather than something settled.

#

Search Engine Journal asserts that large studies show no measurable citation advantage for sites that add an llms.txt file.

medium confidencetrade press
1 quote from 1 source
a finding since echoed by other large studies showing no measurable citation advantage for sites that add one
#

Search Engine Journal's account reported that cats.txt passed all four proofs commonly cited for llms.txt.

trade press
1 quote from 1 source
Then, I checked it against the exact four "proofs" people cite for llms.txt. It passed all four.
#

Search Engine Journal published the argument that a crawler fetching llms.txt tells you nothing about whether the contents of llms.txt are read, weighted, trusted or acted upon.

trade press
1 quote from 1 source
A crawler fetching a file tells you nothing about whether the contents are read, weighted, trusted, or acted upon. Fetching things is the entire job description of a crawler.