Robots.txt
28 quoted answers, each with its source
The short answer
Robots.txt is a crawl directive, not an enforcement or hiding mechanism: Google states that crawlers may choose whether to obey the instructions, that a robots.txt block prevents Google from crawling a URL but the URL can still be indexed and appear without a description if linked from elsewhere, and that robots.txt should not be used for canonicalization. In September 2022, RFC 9309 made the Robots Exclusion Protocol an IETF Standards Track document, extending the method originally defined by Martijn Koster in 1994, and in September 2023 Google added the Google-Extended robots.txt control for Bard and Vertex AI generative APIs. As of July 1, 2025, Cloudflare changed its default to block AI crawlers unless they pay creators; by August 2026, Search Engine Journal reported that BuzzStream measured 75% of top U.S. and UK publishers blocking training crawlers, and roughly 95% of the 492,000 robots.txt mentions of CCBot existed to block it.
Every part of this comes from the quotes below. How we write these.
What the rules actually say18
Quoted straight from the people who make or enforce the rules.
Google's crawl budget documentation states that Google will not shift crawl budget freed by robots.txt blocking to other pages unless Google is already hitting the site's crawl capacity limit.
1 quote from 1 source
Don't use robots.txt to temporarily reallocate crawl budget for other pages; use robots.txt to block pages or resources that you don't want Google to crawl at all. Google won't shift this newly available crawl budget to other pages unless Google is already hitting your site's crawl capacity limit.
Google's crawl budget documentation states that URLs blocked by robots.txt stay part of the crawl queue much longer than URLs returning a 404.
1 quote from 1 source
Google won't forget a URL that it knows about, but a 404 status code is a strong signal not to crawl that URL again. Blocked URLs, however, will stay part of your crawl queue much longer, and will be recrawled when the block is removed.
Google's crawl budget documentation states that blocking URLs with robots.txt…
- prevents Google from crawling them.
- significantly decreases the chance the URLs will be processed by other Google systems such as getting indexed by Google Search.
1 quote from 1 source
Blocking URLs with robots.txt prevents Google from crawling them, and significantly decreases the chance the URLs will be processed by other Google systems (such as getting indexed by Google Search).
Google's common crawlers documentation states that crawling preferences addressed to the GoogleOther user agent in robots.txt do not affect any specific product.
1 quote from 1 source
Crawling preferences addressed to the GoogleOther user agent don't affect any specific product. GoogleOther is the generic crawler that may be used by various
Google's canonicalization documentation states that…
- the robots.txt file should not be used for canonicalization purposes.
- Google may still index URLs that are disallowed in robots.txt without their content.
1 quote from 1 source
Don't use the robots.txt file for canonicalization purposes. Google may still index URLs that are disallowed in robots.txt without their content.
Google Search Central's robots.txt introduction states that a robots.txt file is not a mechanism for keeping a web page out of Google.
1 quote from 1 source
A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.
Google Search Central's robots.txt introduction states that Google may still find and index a URL disallowed in robots.txt if it is linked from other places on the web.
1 quote from 1 source
While Google won't crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web. As a result, the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google Search results.
Google Search Central's robots.txt introduction states that…
- the instructions in robots.txt files cannot enforce crawler behavior.
- it is up to the crawler to obey the instructions in robots.txt files.
1 quote from 1 source
The instructions in robots.txt files cannot enforce crawler behavior to your site; it's up to the crawler to obey them. While Googlebot and other respectable web crawlers obey the instructions in a robots.txt file, other crawlers might not.
Google Search Central's robots.txt introduction warns against using a robots.txt file as a means to hide web pages from Google Search results.
1 quote from 1 source
Warning: Don't use a robots.txt file as a means to hide your web pages (including PDFs and other text-based formats supported by Google) from Google Search results.
Google Search Central's robots.txt introduction states that if a web page is blocked with a robots.txt file, the web page's…
- URL can still appear in search results.
- search result will not have a description.
1 quote from 1 source
If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result won't have a description.
Google's AI features documentation states that robots.txt directives for Googlebot are the control site owners use to manage how their sites are crawled for Search.
1 quote from 1 source
AI is built into Search and integral to how Search functions, which is why robots.txt directives for Googlebot is the control for site owners to manage access to how their sites are crawled for Search.
RFC 9309 states that it specifies and extends the Robots Exclusion Protocol method originally defined by Martijn Koster in 1994, adding definition language for the protocol, instructions for handling errors, and instructions for caching.
1 quote from 1 source
This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.
Google's Search Central blog states that the proposed Robots Exclusion Protocol draft…
- does not change the rules created in 1994 for robots.txt.
- defines essentially all undefined scenarios for robots.txt parsing and matching.
1 quote from 1 source
It doesn't change the rules created in 1994, but rather defines essentially all undefined scenarios for robots.txt parsing and matching, and extends it for the modern web.
Measured finding1
Something someone actually measured, with the method attached.
In 2026, a CCBot block in a site's robots.txt is as likely to be a platform default as a decision.
1 quote from 1 source
Its managed robots.txt reached 3.8 million more domains that never wrote one. Ad plugins add blocklists. In 2026, a CCBot block is as likely to be a platform default as a decision.
Practical effect1
What is observed to happen in practice, as distinct from what is documented.
Search Engine Journal reports that Common Crawl's crawler CCBot never runs JavaScript.
1 quote from 1 source
Both sides are read before JavaScript runs, which is the point, because CCBot never runs it.
Changed on5
A dated change. This is where advice quietly stops being true.
- Google announced Google-Extended, made available through robots.txt.
- Google-Extended, made available through robots.txt, is a new control.
- Web publishers can use Google-Extended, made available through robots.txt, to manage whether their sites help improve Bard and Vertex AI generative APIs.
2 quotes from 1 source
Today we’re announcing Google-Extended, a new control that web publishers can use to manage whether their sites help improve Bard and Vertex AI generative APIs, including future generations of models that power those products.
Making simple and scalable controls, like Google-Extended, available through robots.txt is an important step in providing transparency and control that we believe all providers of AI models should make available.
The Robots Exclusion Protocol was published as IETF Standards Track document RFC 9309 in September 2022, having previously never been turned into an official Internet standard.
2 quotes from 2 sources
Internet Engineering Task Force (IETF) M. Koster Request for Comments: 9309 Category: Standards Track G. Illyes ISSN: 2070-1721 H. Zeller L. Sassman Google LLC September 2022 Robots Exclusion Protocol
However, the REP was never turned into an official Internet standard, which means that developers have interpreted the protocol somewhat differently over the years.
Cloudflare called the day on which Cloudflare announced the robots.txt default change 'Content Independence Day'.
1 quote from 1 source
That changes today, July 1, what we’re calling Content Independence Day. Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content.
Often said, but we could not find it in the rules3
Lots of people say this. We went looking for an official source that backs it up and could not find one, so treat it as something people repeat rather than something settled.
- Roughly 492,000 sites name CCBot in robots.txt.
- About 95% of the mentions of CCBot in robots.txt exist to block CCBot.
1 quote from 1 source
Chris Green posted HTTP Archive numbers on LinkedIn recently. Roughly 492,000 sites name CCBot in robots.txt, and about 95% of those mentions exist to block it.
Search Engine Journal reports that BuzzStream measured the practice of blocking training crawlers using robots.txt at 75% of the top U.S. and UK publishers.
1 quote from 1 source
The BBC robots.txt blocks 13 of the 14 AI crawlers the checker tracks, and other top news sites have been blocking training crawlers on purpose since 2023. BuzzStream measured it at 75% of the top U.S. and UK publishers.