robots.txt
28 established claims - 9 refused by the grounding gate
Verdict
Robots.txt is a crawl directive, not an enforcement or hiding mechanism: Google states that crawlers may choose whether to obey the instructions, that a robots.txt block prevents Google from crawling a URL but the URL can still be indexed and appear without a description if linked from elsewhere, and that robots.txt should not be used for canonicalization. In September 2022, RFC 9309 made the Robots Exclusion Protocol an IETF Standards Track document, extending the method originally defined by Martijn Koster in 1994, and in September 2023 Google added the Google-Extended robots.txt control for Bard and Vertex AI generative APIs. As of July 1, 2025, Cloudflare changed its default to block AI crawlers unless they pay creators; by August 2026, Search Engine Journal reported that BuzzStream measured 75% of top U.S. and UK publishers blocking training crawlers, and roughly 95% of the 492,000 robots.txt mentions of CCBot existed to block it.
Written from the 28 established claims below, then checked against them. How that works.
Documented position11
What the operator of the system says in its own documentation.
Google's common crawlers documentation states that crawling preferences addressed to the GoogleOther user agent in robots.txt do not affect any specific product.
1 quote from 1 source
Crawling preferences addressed to the GoogleOther user agent don't affect any specific product. GoogleOther is the generic crawler that may be used by various
Google's canonicalization documentation states that…
- the robots.txt file should not be used for canonicalization purposes.
- Google may still index URLs that are disallowed in robots.txt without their content.
1 quote from 1 source
Don't use the robots.txt file for canonicalization purposes. Google may still index URLs that are disallowed in robots.txt without their content.
Google Search Central's robots.txt introduction states that a robots.txt file is not a mechanism for keeping a web page out of Google.
1 quote from 1 source
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.
Google Search Central's robots.txt introduction states that…
- the instructions in robots.txt files cannot enforce crawler behavior.
- it is up to the crawler to obey the instructions in robots.txt files.
1 quote from 1 source
The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them. While Googlebot and other respectable web crawlers obey the instructions in a robots.txt file, other crawlers might not.
Google Search Central's robots.txt introduction warns against using a robots.txt file as a means to hide web pages from Google Search results.
1 quote from 1 source
Warning: Don't use a robots.txt file as a means to hide your web pages (including PDFs and other text-based formats supported by Google) from Google Search results.
Google's AI features documentation states that robots.txt directives for Googlebot are the control site owners use to manage how their sites are crawled for Search.
1 quote from 1 source
AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search.
RFC 9309 states that it specifies and extends the Robots Exclusion Protocol method originally defined by Martijn Koster in 1994, adding definition language for the protocol, instructions for handling errors, and instructions for caching.
1 quote from 1 source
This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.
Google's Search Central blog states that the proposed Robots Exclusion Protocol draft…
- does not change the rules created in 1994 for robots.txt.
- defines essentially all undefined scenarios for robots.txt parsing and matching.
1 quote from 1 source
It doesn't change the rules created in 1994, but rather defines essentially all undefined scenarios for robots.txt parsing and matching, and extends it for the modern web.
Practical effect9
What is observed to happen in practice, as distinct from what is documented.
In 2026, a CCBot block in a site's robots.txt is as likely to be a platform default as a decision.
1 quote from 1 source
Its managed robots.txt reached 3.8 million more domains that never wrote one. Ad plugins add blocklists. In 2026, a CCBot block is as likely to be a platform default as a decision.
Search Engine Journal reports that Common Crawl's crawler CCBot never runs JavaScript.
1 quote from 1 source
Both sides are read before JavaScript runs, which is the point, because CCBot never runs it.
Google's crawl budget documentation states that Google will not shift crawl budget freed by robots.txt blocking to other pages unless Google is already hitting the site's crawl capacity limit.
1 quote from 1 source
Don't use robots.txt to temporarily reallocate crawl budget for other pages; use robots.txt to block pages or resources that you don't want Google to crawl at all. Google won't shift this newly available crawl budget to other pages unless Google is already hitting your site's crawl capacity limit.
Google's crawl budget documentation states that URLs blocked by robots.txt stay part of the crawl queue much longer than URLs returning a 404.
1 quote from 1 source
Google won't forget a URL that it knows about, but a 404 status code is a strong signal not to crawl that URL again. Blocked URLs, however, will stay part of your crawl queue much longer, and will be recrawled when the block is removed.
Google's crawl budget documentation states that blocking URLs with robots.txt…
- prevents Google from crawling them.
- significantly decreases the chance the URLs will be processed by other Google systems such as getting indexed by Google Search.
1 quote from 1 source
Blocking URLs with robots.txt prevents Google from crawling them, and significantly decreases the chance the URLs will be processed by other Google systems (such as getting indexed by Google Search).
Google Search Central's robots.txt introduction states that Google may still find and index a URL disallowed in robots.txt if it is linked from other places on the web.
1 quote from 1 source
While Google won't crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google Search results.
Google Search Central's robots.txt introduction states that if a web page is blocked with a robots.txt file, the web page's…
- URL can still appear in search results.
- search result will not have a description.
1 quote from 1 source
If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result won't have a description.
Changed on5
A dated change. This is where advice quietly stops being true.
Cloudflare called the day on which Cloudflare announced the robots.txt default change 'Content Independence Day'.
1 quote from 1 source
That changes today, July 1, what we’re calling Content Independence Day. Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content.
- Google announced Google-Extended, made available through robots.txt.
- Google-Extended, made available through robots.txt, is a new control.
- Web publishers can use Google-Extended, made available through robots.txt, to manage whether their sites help improve Bard and Vertex AI generative APIs.
2 quotes from 1 source
Today we’re announcing Google-Extended, a new control that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products.
Making simple and scalable controls, like Google-Extended, available through robots.txt is an important step in providing transparency and control that we believe all providers of AI models should make available.
The Robots Exclusion Protocol was published as IETF Standards Track document RFC 9309 in September 2022, having previously never been turned into an official Internet standard.
2 quotes from 2 sources
Internet Engineering Task Force (IETF) M. Koster Request for Comments: 9309 Category: Standards Track G. Illyes ISSN: 2070-1721 H. Zeller L. Sassman Google LLC September 2022 Robots Exclusion Protocol
However, the REP was never turned into an official Internet standard, which means that developers have interpreted the protocol somewhat differently over the years.
Echoed without a primary source3
Widely repeated in the field, but traceable only to other repetitions. Listed because knowing something is merely echoed is worth as much as knowing it is documented.
- Roughly 492,000 sites name CCBot in robots.txt.
- About 95% of the mentions of CCBot in robots.txt exist to block CCBot.
1 quote from 1 source
Chris Green posted HTTP Archive numbers on LinkedIn recently. Roughly 492,000 sites name CCBot in robots.txt, and about 95% of those mentions exist to block it.
Search Engine Journal reports that BuzzStream measured the practice of blocking training crawlers using robots.txt at 75% of the top U.S. and UK publishers.
1 quote from 1 source
The BBC robots.txt blocks 13 of the 14 AI crawlers the checker tracks, and other top news sites have been blocking training crawlers on purpose since 2023. BuzzStream measured it at 75% of the top U.S. and UK publishers.
Refused9
Claims the research produced that the grounding gate would not accept, with its reason. They are here because a site that only shows what passed gives you no way to judge whether the gate does anything.
Show what was refused
Cloudflare announced that the default of robots.txt will change to block AI crawlers unless those AI crawlers pay creators for their content.
Refused: The quote states changing the default to block AI crawlers, but does not mention robots.txt specifically.
Unsupported: robots.txt
What it rested on
That changes today, July 1, what we’re calling Content Independence Day. Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content.
Google's Search Central blog states that the Robots Exclusion Protocol draft introduced a requirement that developers must parse at least the first 500 kibibytes of a robots.txt file.
Refused: The quote states the parsing requirement but does not say the Robots Exclusion Protocol draft introduced it.
Unsupported: the Robots Exclusion Protocol draft introduced
What it rested on
Developers must parse at least the first 500 kibibytes of a robots.txt. Defining a maximum file size ensures that connections are not open for too long, alleviating unnecessary strain on servers.
The Robots Exclusion Protocol draft introduced a new maximum caching time for robots.txt of 24 hours or the cache directive value if available.
Refused: The quote states the new maximum caching time but does not attribute its introduction to the Robots Exclusion Protocol draft.
Unsupported: The Robots Exclusion Protocol draft introduced
What it rested on
A new maximum caching time of 24 hours or cache directive value if available, gives website owners the flexibility to update their robots.txt whenever they want, and crawlers aren't overloading websites with robots.txt requests.
robots.txt's documented position is that Google's crawling infrastructure defines a site as a unique hostname.
Refused: The quote states Google's crawling infrastructure defines a site as a unique hostname, but it does not attribute this as a documented position of robots.txt.
Unsupported: robots.txt's documented position
What it rested on
In this context, Google's crawling infrastructure defines a site as a unique hostname. For example, https://www.example.com/ and https://code.example.com/ are treated as separate sites and have separate crawl budgets.
robots.txt's documented position is that subdomains are treated as separate sites with separate crawl budgets.
Refused: The quote states Google's crawl-budget documentation treats subdomains as separate sites, but it is not a robots.txt documented position.
Unsupported: robots.txt's documented position
What it rested on
In this context, Google's crawling infrastructure defines a site as a unique hostname. For example, https://www.example.com/ and https://code.example.com/ are treated as separate sites and have separate crawl budgets.
The robots.txt finding reported by Search Engine Journal states that Mozilla audited the LLMs released between 2019 and 2023.
Refused: The cited quote states Mozilla audited those LLMs but does not mention a robots.txt finding.
Unsupported: robots.txt finding
What it rested on
When Mozilla audited the LLMs released between 2019 and 2023, 64% had trained on Common Crawl data in some form, and GPT-3 was roughly 60% filtered Common Crawl by training weight.
The robots.txt finding reported by Search Engine Journal states that 64% of the audited LLMs had trained on Common Crawl data in some form.
Refused: The quote supports the 64% statistic from Search Engine Journal but does not describe it as a robots.txt finding.
Unsupported: robots.txt finding
What it rested on
When Mozilla audited the LLMs released between 2019 and 2023, 64% had trained on Common Crawl data in some form, and GPT-3 was roughly 60% filtered Common Crawl by training weight.
Cloudflare's managed robots.txt reached 3.8 million domains that never wrote one.
Refused: The quote says 'Its' managed robots.txt and does not name Cloudflare, so the entity attribution is not established.
Unsupported: Cloudflare's
What it rested on
Its managed robots.txt reached 3.8 million more domains that never wrote one. Ad plugins add blocklists. In 2026, a CCBot block is as likely to be a platform default as a decision.
Google's crawl budget documentation states that URLs blocked by robots.txt are recrawled when the block is removed.
Refused: The quote says blocked URLs will be recrawled when the block is removed, but does not state the block is by robots.txt.
Unsupported: by robots.txt
What it rested on
Google won't forget a URL that it knows about, but a 404 status code is a strong signal not to crawl that URL again. Blocked URLs, however, will stay part of your crawl queue much longer, and will be recrawled when the block is removed.