Common Crawl
35 established claims - 11 refused by the grounding gate
Verdict
CCBot checks robots.txt first, honors nofollow and Crawl-delay, supports sitemaps, fetches via HTTP GET, and does not use cookies; sites can block it by naming the CCBot user-agent in robots.txt. Common Crawl documents itself as of 2026 as a 501(c)(3) non-profit providing a free sample of the web, not the entire or a representative web. Its July 2026 crawl was crawled July 7 to July 25 and contained 2.14 billion pages from 40.5 million hosts or 33.2 million registered domains, with 603 million URLs not visited in any prior Common Crawl crawl. GPT-3 used Common Crawl from 41 monthly shards covering 2016 to 2019, 570GB after filtering, at a 60% training-mix weight; a February 2024 review found at least 64% of 47 text-generation LLMs published 2019 to October 2023 used at least one filtered Common Crawl version for pre-training. Common Crawl's June 1, 2026 AI Visibility Audit asserts that if you are not in the crawl, you are not in the model and puts the latest crawl's English share at roughly 41 percent; an August 10, 2026 Search Engine Journal article counted roughly 492,000 sites naming CCBot in robots.txt, about 95 percent of them to block it.
Written from the 35 established claims below, then checked against them. How that works.
Documented position18
What the operator of the system says in its own documentation.
Common Crawl's announcement of The AI Visibility Audit says the guide explains how to run…
- a repeatable audit.
- a five-check audit.
- the audit using only free tools.
- the audit in about 90 minutes.
1 quote from 1 source
It explains how AI systems actually discover content, why training-data inclusion behaves like a ranking factor, and how to run a repeatable, five-check audit using only free tools in about 90 minutes.
Common Crawl's announcement of The AI Visibility Audit puts the English share of Common Crawl's latest crawl at roughly 41 percent.
1 quote from 1 source
The guide walks through how CCBot crawls the open web and publishes the archive that helps train modern LLMs, how harmonic centrality in the Common Crawl Web Graph sets crawl priority, why CDN and WAF defaults now silently block AI crawlers and training data crawlers, and why AI still leans toward English, with the English share of the latest crawl at roughly 41 percent.
Common Crawl's announcement of The AI Visibility Audit asserts that if you are not in the crawl, you are not in the model.
1 quote from 1 source
The old world was index and rank. The new world is train and retrieve. If you are not in the crawl, you are not in the model.
Common Crawl's FAQ page describes Common Crawl as…
- a 501(c)(3) non-profit organization.
- dedicated to providing a copy of the Internet to Internet researchers, companies and individuals at no cost for the purpose of research and analysis.
1 quote from 1 source
Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the Internet to Internet researchers, companies and individuals at no cost for the purpose of research and analysis.
Common Crawl's FAQ states that…
- CCBot checks robots.txt first.
- if crawling a page is allowed, CCBot fetches pages using HTTP GET requests.
1 quote from 1 source
CCBot is an automated crawler, checking first the robots.txt, and if crawling a page is allowed, fetches pages using HTTP GET requests.
Common Crawl's FAQ states that CCBot currently does not use cookies.
- Common Crawl's dataset is a sample of the web.
- Common Crawl does not generally archive any entire website, but rather a randomly selected subset of each website.
1 quote from 1 source
Common Crawl's dataset is a sample of the web, and we do not generally archive any entire website but a randomly selected subset of it.
Common Crawl's FAQ states that a site can block Common Crawl's crawler by adding a robots.txt rule for the exclusion UserAgent string CCBot.
2 quotes from 1 source
You configure your robots.txt file which uses the Robots Exclusion Protocol to block the crawler. Our bot's exclusion UserAgent string is: CCBot.
User-agent: CCBot Disallow: /
Common Crawl's FAQ states that Common Crawl obeys the Crawl-delay parameter in robots.txt.
Common Crawl's FAQ states that Common Crawl currently honors the nofollow attribute as it applies to links embedded on a site.
1 quote from 1 source
We currently honor the nofollow attribute as it applies to links embedded on your site.
Common Crawl's FAQ states that Common Crawl's crawler…
- supports the Sitemap Protocol.
- utilizes any Sitemap announced in the robots.txt file.
1 quote from 1 source
The crawler supports the Sitemap Protocol and utilizes any Sitemap announced in the robots.txt file.
Measured finding14
Something someone actually measured, with the method attached.
Common Crawl's July 2026 crawl archive…
- contains 2.14 billion web pages.
- was crawled between July 7th and July 25th.
1 quote from 1 source
The data was crawled between July 7th and July 25th, and contains 2.14 billion web pages (or 364.01 TiB of uncompressed content).
Common Crawl reports that page captures in its July 2026 crawl…
- come from 40.5 million hosts or 33.2 million registered domains.
- include 603 million new URLs not visited in any prior Common Crawl crawl.
1 quote from 1 source
Page captures are from 40.5 million hosts or 33.2 million registered domains and include 603 million new URLs, not visited in any of our prior crawls.
- A February 2024 Mozilla Foundation report by Stefan Baack reviewed 47 text-generation LLMs published between 2019 and October 2023 for their pre-training use of Common Crawl.
- The February 2024 Mozilla Foundation report by Stefan Baack found that at least 64% (30 models) of the 47 text-generation LLMs reviewed in the February 2024 Mozilla Foundation report by Stefan Baack used at least one filtered version of Common Crawl for pre-training.
2 quotes from 1 source
Training Data for the Price of a Sandwich1 Common Crawl’s Impact on Generative AI February 2024 By Stefan Baack Mozilla Insights
A review of 47 LLMs for text generation published between 2019 and October 2023 shows that at least 64% of these models (30) used at least one filtered version of Common Crawl for their pre-training (Figure 1).
A February 2024 Mozilla Foundation report states that Common Crawl made up more than 80% of the tokens in OpenAI's GPT-3.
1 quote from 1 source
For example, Common Crawl made up more than 80% of the tokens in OpenAI’s GPT-3 (Brown et al. 2020).
Common Crawl does not contain…
- the entire web.
- a representative sample of the web.
1 quote from 1 source
Common Crawl does not contain the “entire web,” nor a representative sample of it. Despite its size, there are important limitations on how much of the web is covered.
The GPT-3 paper states that the CommonCrawl data used to train GPT-3…
- was downloaded from 41 shards of monthly CommonCrawl covering 2016 to 2019.
- amounted to 45TB of compressed plaintext before filtering.
- amounted to 570GB after filtering.
1 quote from 1 source
The CommonCrawl data was downloaded from 41 shards of monthly CommonCrawl covering 2016 to 2019, constituting 45TB of compressed plaintext before filtering and 570GB after filtering, roughly equivalent to 400 billion byte-pair-encoded tokens.
- The GPT-3 paper's table of datasets used to train GPT-3 gives filtered Common Crawl a weight in the training mix of 60%.
- For filtered Common Crawl, weight in training mix means the fraction of training examples drawn from Common Crawl rather than Common Crawl's share of the corpus.
2 quotes from 1 source
Common Crawl (filtered) 410 billion 60% 0.44
Table 2.2: Datasets used to train GPT-3. “Weight in training mix” refers to the fraction of examples during training that are drawn from a given dataset, which we intentionally do not make proportional to the size of the dataset.
Changed on1
A dated change. This is where advice quietly stops being true.
Common Crawl published a document called The AI Visibility Audit.
2 quotes from 2 sources
Introducing the AI Visibility Audit June 1, 2026
Today Common Crawl is publishing The AI Visibility Audit, a free field guide built for the SEOs and GEOs who are already doing this work and want a concrete framework rather than theory.
Echoed without a primary source2
Widely repeated in the field, but traceable only to other repetitions. Listed because knowing something is merely echoed is worth as much as knowing it is documented.
A Search Engine Journal article dated August 10, 2026 states that…
- roughly 492,000 sites name Common Crawl's CCBot in robots.txt.
- about 95% of the mentions of Common Crawl's CCBot in robots.txt exist to block it.
2 quotes from 1 source
August 10, 2026
Chris Green posted HTTP Archive numbers on LinkedIn recently. Roughly 492,000 sites name CCBot in robots.txt, and about 95% of those mentions exist to block it.
Refused11
Claims the research produced that the grounding gate would not accept, with its reason. They are here because a site that only shows what passed gives you no way to judge whether the gate does anything.
Show what was refused
Common Crawl announced in a blog post that the Common Crawl blog index dates June 1, 2026.
Refused: The quotes show a blog post titled "Introducing the AI Visibility Audit" dated June 1, 2026, but do not mention a "Common Crawl blog index" or state that such an index dates June 1, 2026.
Unsupported: Common Crawl blog index
What it rested on
Introducing the AI Visibility Audit June 1, 2026
Today Common Crawl is publishing The AI Visibility Audit, a free field guide built for the SEOs and GEOs who are already doing this work and want a concrete framework rather than theory.
Common Crawl's published erratum states that the truncation threshold for archived content was 1 MiB prior to the March 2025 crawl (CC-MAIN-2025-13).
Refused: The quote gives the threshold fact but is cited from a Common Crawl blog post introducing the AI Visibility Audit, not from a published erratum.
Unsupported: published erratum
What it rested on
Prior to March 2025 (CC-MAIN-2025-13), the truncation threshold was 1 MiB. From the March 2025 crawl onwards, this limit has been increased to 5 MiB.
Common Crawl's published erratum states that the truncation threshold for archived content was increased to 5 MiB from the March 2025 crawl onwards.
Refused: The quote states the threshold change but does not identify it as a published erratum.
Unsupported: published erratum
What it rested on
Prior to March 2025 (CC-MAIN-2025-13), the truncation threshold was 1 MiB. From the March 2025 crawl onwards, this limit has been increased to 5 MiB.
Common Crawl's FAQ states that CCBot currently does not execute JavaScript.
Refused: The quote says JavaScript is not executed but does not name CCBot.
Unsupported: CCBot
Common Crawl's documented position is that Cloudflare is changing the default to block AI crawlers unless those AI crawlers pay creators for their content.
Refused: Quotes are from a Cloudflare blog, not from Common Crawl.
Unsupported: Common Crawl's documented position
What it rested on
July 1, 2025 Content Independence Day: no AI crawl without compensation!
That changes today, July 1, what we’re calling Content Independence Day. Cloudflare, along with a majority of the world's leading publishers and AI companies, is changing the default to block AI crawlers unless they pay creators for their content.
A Search Engine Journal article dated August 10, 2026 states that BuzzStream measured deliberate blocking of Common Crawl at 75% of the top U.S. and UK publishers.
Refused: The quote says BuzzStream measured blocking of training crawlers generally at 75%, not Common Crawl specifically.
Unsupported: deliberate blocking of Common Crawl
What it rested on
The BBC robots.txt blocks 13 of the 14 AI crawlers the checker tracks, and other top news sites have been blocking training crawlers on purpose since 2023. BuzzStream measured it at 75% of the top U.S. and UK publishers.
Common Crawl echoed the claim that a Search Engine Journal article dated August 10, 2026 states that since July 1 2025, Cloudflare blocks AI crawlers by default on every new domain it serves.
Refused: The quote supports the Cloudflare statement from a Search Engine Journal article but does not show Common Crawl echoing it or the August 10, 2026 date.
Unsupported: Common Crawl echoed the claim that a Search Engine Journal article dated August 10, 2026 states
What it rested on
Since July 1 2025, Cloudflare blocks AI crawlers by default on every new domain it serves.
A Search Engine Journal article dated August 10, 2026 restates the Mozilla figure as 64% of LLMs released between 2019 and 2023 having trained on Common Crawl data in some form.
Refused: The quote restates the Mozilla figure in a Search Engine Journal article but does not establish the article's publication date.
Unsupported: dated August 10, 2026
What it rested on
When Mozilla audited the LLMs released between 2019 and 2023, 64% had trained on Common Crawl data in some form, and GPT-3 was roughly 60% filtered Common Crawl by training weight.
A Search Engine Journal article dated August 10, 2026 states that GPT-3 was roughly 60% filtered Common Crawl by training weight.
Refused: The quote supports the GPT-3 statistic but does not establish the article's date.
Unsupported: dated August 10, 2026
What it rested on
When Mozilla audited the LLMs released between 2019 and 2023, 64% had trained on Common Crawl data in some form, and GPT-3 was roughly 60% filtered Common Crawl by training weight.
A Search Engine Journal article dated August 10, 2026 states that being in Common Crawl does not put a site in a model.
Refused: The cited quote states the core sentence, but no cited quote or source label provides the August 10, 2026 date.
Unsupported: dated August 10, 2026
What it rested on
Being in Common Crawl doesn’t put you in a model.
Every training set filters the crawl, and the frontier labs stopped disclosing their mixes around 2023.
A Search Engine Journal article dated August 10, 2026 states that every training set filters the crawl.
Refused: The quote supports the statement and Search Engine Journal source, but no quote provides the article date.
Unsupported: dated August 10, 2026
What it rested on
Being in Common Crawl doesn’t put you in a model.
Every training set filters the crawl, and the frontier labs stopped disclosing their mixes around 2023.