crawl budget

Google's 2017 blog post on crawl budget carries a notice…

• stating that some information in the post may be outdated.
• directing readers to updated documentation on optimizing crawling efficiency.

Google's crawling infrastructure documentation defines a site's crawl budget as the set of URLs that Google can and wants to crawl.

Google's crawling infrastructure documentation states that site owners whose pages seem to be crawled the same day they are published do not need to read the crawl budget guide.

Google's crawling infrastructure documentation states that the page-count thresholds it gives for crawl budget are…

• a rough estimate to help classify a site.
• not exact thresholds.

• Google's crawling infrastructure documentation states that https://www.example.com/ and https://code.example.com/ are treated as separate sites with separate crawl budgets.
• Google's crawling infrastructure defines a site as a unique hostname.

Google's crawling infrastructure documentation states that for Google Search, not every page that is crawled will necessarily be indexed.

Google's crawling infrastructure documentation states that…

• every site starts with the same default, conservative crawl capacity limit.
• Google's systems adjust the crawl capacity limit of a site over time if there is demand to crawl more and the site remains healthy.

• Google's crawling infrastructure documentation states that the crawl capacity limit is shared across all of Google's crawlers.
• High demand from one of Google's crawlers can reduce the capacity available for other Google's crawlers.

Google Search Central's guide to optimizing for generative AI features on Google Search directs owners of very large and frequently updated sites to Google's guide to optimizing crawl budget.

Google's crawling infrastructure documentation states that…

• crawling is not a ranking signal in Google Search.
• improving crawl rate will not necessarily lead to better positions in Google Search results.

• Google's crawling infrastructure documentation states that compressing sitemaps does not increase crawl budget.
• Compressing sitemaps does not increase crawl budget because zipped sitemaps still have to be fetched from the server.

Google's crawling infrastructure documentation states that pages serving 4xx HTTP status codes, except 429, don't waste crawl budget.

Google's crawling infrastructure documentation states that the non-standard crawl-delay robots.txt rule is not processed by Google's crawlers.

Google's crawling infrastructure documentation states that…

• any URL Googlebot crawls generally counts towards a site's crawl budget.
• alternate URLs such as AMP or hreflang generally count towards a site's crawl budget.
• embedded content such as CSS and JavaScript generally counts towards a site's crawl budget.

Google's crawling infrastructure documentation states that…

• for Google Search, content is rated by quality regardless of age.
• there is no additional value in making pages artificially appear fresh through trivial changes.

Google's crawling infrastructure documentation states that time spent rendering a page counts as much as time spent requesting the page when Google crawls a site.

Google's 2017 blog post on crawl budget states that URLs disallowed through robots.txt do not affect a site's crawl budget.

Google's crawling infrastructure documentation states that Google will not shift crawl budget freed by robots.txt blocking to other pages unless Google is already hitting the site's crawl capacity limit.

• Google's crawling infrastructure documentation advises against using noindex to manage crawl budget.
• Google will still request the page and then drop it when it sees the noindex meta tag or header, wasting crawling time.

• TwoSquares states that optimising crawl budget on a site that does not need it is usually wasted effort.
• TwoSquares states that, regarding crawl budget, unindexed pages are almost always caused by weak internal linking, poor content quality, duplication, mixed signals or lack of demand.

Google's crawling infrastructure documentation states that removing URLs from Google's index with noindex can indirectly free up some crawl budget for a site in the long run.

Google's crawling infrastructure documentation states that it is probably more important to make a site faster for users than to make it faster to increase crawl coverage.

Google's crawling documentation changelog records that on July 22, 2026 the Optimize your crawl budget page was polished and clarified to improve clarity, terminology consistency and flow.

• Google's 2017 blog post defined crawl budget as the number of URLs Googlebot can and wants to crawl
• Google's current crawling infrastructure documentation defines crawl budget as the set of URLs that Google can and wants to crawl

• Google's 2017 blog post said that a site with fewer than a few thousand URLs would most of the time be crawled efficiently.
• Google's current crawl budget guide is aimed at large sites with 1 million or more unique pages whose content changes about once a week.

• On December 2, 2022, Google's Search Central changelog recorded that two new myths were added to the crawl budget documentation.
• The myth that noindex isn't a good way to control crawl budget was added to Google's crawl budget documentation on December 2, 2022.
• The myth that pages serving 4xx status codes except 429 don't waste crawl budget was added to Google's crawl budget documentation on December 2, 2022.

Google's Search Central changelog records that the…

• Large site owner's guide to managing your crawl budget was migrated from the Search Console Help Center on December 4, 2020.
• migration of the Large site owner's guide to managing your crawl budget involved no change to its content.

Backlinko defines crawl budget as the number of pages Googlebot crawls and indexes on a website within a given timeframe.

TwoSquares states that crawl budget is not a limiting factor for the majority of websites.