Crawl Budget
Crawl budget is the number of URLs Googlebot will crawl on your site within a given time period, determined by your site's crawl capacity (how fast Google can crawl without overloading you) and your crawl demand (how much content Google considers worth re-checking). For most small sites crawl budget is not the constraint; for sites over 10,000 URLs it can become one.
Also called: Googlebot crawl budget, Crawl rate · Last updated: May 27, 2026 · By Joseph W. Anady
Why it matters.
Crawl budget anxiety is one of the most over-discussed SEO topics for small sites where it doesn't matter. Google's John Mueller has stated repeatedly that crawl budget is only a meaningful concern for sites with 1M+ URLs or for sites with daily updates across 10K+ URLs. Most small business sites under 1,000 URLs have effectively unlimited crawl budget — Google can crawl every page weekly without issue.
How it works.
Crawl budget combines crawl capacity (max requests/second Googlebot can make without overloading your server) and crawl demand (how often Google wants to re-check each URL based on freshness, importance, change frequency). Sites with fast TTFB, strong internal linking, accurate sitemaps, and frequently-updated content get higher crawl demand. Sites with slow servers or duplicate content get lower crawl demand.
2026 reality check.
Crawl budget concerns have shifted in 2026 with AI Mode and Gemini crawler activity. Google's various crawler user-agents (Googlebot Desktop, Googlebot Smartphone, Googlebot Image, Googlebot News, GoogleOther) have different crawl budgets. AI Mode + Gemini training crawlers (Google-Extended user-agent) consume separate budget. For most small sites this is invisible; for high-traffic sites with thin pages it can become noticeable.
Data points
- Crawl budget = crawl capacity × crawl demand (Google internal definition)
- Meaningful constraint mainly for sites with 1M+ URLs or 10K+ URLs with daily updates (John Mueller guidance)
- Google Search Console > Settings > Crawl stats shows actual crawl activity per site
- Different Google crawler user-agents have different budgets (Googlebot, Google-Extended, etc.)
- TTFB directly affects crawl capacity (fast servers = more crawl requests per second)
First-hand insight from ThatDeveloperGuy.
ThatDeveloperGuy operates 130+ sites in the hosting substrate, ranging from 10-page small business sites to 300+ page enterprise sites. None of them are crawl-budget constrained. Most TDG client sites under 200 URLs see Googlebot recrawl every key page weekly. The largest TDG sites (~300 URLs) see Googlebot recrawl every page at least monthly. Crawl budget is not the bottleneck for small business SEO.
How TDG approaches it
TDG ignores crawl budget for sites under 1,000 URLs. For larger sites: accurate XML sitemaps with proper lastmod, fast TTFB (under 200ms), clean internal linking architecture, regular content updates on important pages. The cumulative effect is that Googlebot crawls important pages frequently without us having to optimize for crawl budget explicitly.
Common mistakes.
- Obsessing about crawl budget on a 50-page site (not the bottleneck)
- Using noindex to 'save crawl budget' (noindex'd pages still consume crawl budget)
- Blocking JavaScript/CSS files to save crawl budget (breaks rendering, hurts ranking)
- Adding nofollow to internal links to save crawl budget (counterproductive)
- Massive sitemap with low-quality URLs (signals low-quality site to Google)
FAQ.
Do I need to worry about crawl budget?
Probably not. Sites under 1,000 URLs with reasonable server speed almost never hit crawl budget limits. Above 10,000 URLs with daily updates, it can become a consideration.
How do I check my crawl budget?
Search Console > Settings > Crawl stats shows requests/day, average response time, and total kilobytes downloaded. Compare requests/day to total URL count to estimate days between crawls.
Does noindex save crawl budget?
No. Noindexed pages still get crawled (Googlebot has to fetch the page to read the noindex directive). To exclude from crawl entirely, use robots.txt Disallow.
Will adding more pages reduce crawl frequency on existing pages?
Not for small sites. For sites near crawl-budget capacity, adding many new pages can spread the budget thinner. Audit content quality before adding hundreds of new pages.
Does CDN affect crawl budget?
Indirectly — faster TTFB allows Googlebot to make more requests per second within capacity limits. CDN with fast cache hit + low cache miss latency improves crawl capacity.
Maintained by Joseph W. Anady at ThatDeveloperGuy. Back to glossary · Suggest a term