Picture this.
Google sends a crawler to your website. It has a finite amount of time and resources to spend. It starts crawling.
Page 1: your homepage. Great.
Page 2: your best blog post. Excellent.
Page 3: /shop?color=red&size=9&sort=price-asc&page=2 — a filtered product listing identical to seventeen other filtered URLs.
Page 4: /shop?color=red&size=9&sort=price-desc&page=2 — the same page, sorted the other way.
Page 5: a session ID URL that expires in four hours.
Page 6: your internal search results page for the query “shoes.”
Page 7: another faceted navigation URL.
By the time Googlebot gets to your actual product pages — the ones with unique content, real search value, and backlinks — it’s running low on budget. It crawls a few, queues the rest for later, and leaves.
Your important pages get crawled less frequently. New content takes longer to appear in search. Updates take longer to reflect. And you have no idea this is happening because everything looks fine in your browser.
This is a crawl budget problem. And it’s almost entirely self-inflicted.
Table of Contents
What Is Crawl Budget?
Crawl budget is the number of pages Googlebot will crawl on your website within a given time period.
Google defines crawl budget as the combination of two things:
Crawl rate limit — how fast Googlebot can crawl your site without overwhelming your server. Googlebot is polite. It monitors server response times and backs off if your server is struggling. Faster, more reliable servers get crawled more aggressively.
Crawl demand — how much Google actually wants to crawl your site, based on signals like your site’s authority, how frequently your content is updated, and how many external links point to your pages. High-authority, frequently updated sites get more crawl demand — Google wants to check them more often.
Together, these determine roughly how many pages Google will visit and how often. That budget is finite. What you do with it determines which pages get indexed quickly and which pages sit waiting.
Who Actually Needs to Worry About This?
Honest answer: not everyone.
If your website has a few hundred pages, is well-structured, and doesn’t generate massive amounts of low-value URLs — crawl budget is not your primary concern. Google will crawl you fine.
Crawl budget becomes a real issue when:
- Your site has thousands or tens of thousands of pages — e-commerce stores, news sites, large directories, aggregator platforms
- Your site generates large volumes of low-value URLs — faceted navigation, session IDs, parameter-based filtering, internal search result pages
- You’ve noticed new content taking weeks to appear in search results despite being internally linked
- You’re on a relatively new or lower-authority domain where Google allocates less crawl budget by default
- You’re running a large migration and need Google to discover and process thousands of new URLs quickly
For everyone else: fix your content quality and your technical fundamentals. Crawl budget will follow.
What Eats Your Crawl Budget
Faceted navigation and filter parameters
The single biggest crawl budget drain on e-commerce sites. Every combination of filters — colour, size, brand, price range, sort order — creates a unique URL. A modest product catalogue with five filter types can generate hundreds of thousands of URLs, the vast majority of which are near-identical.
/shoes?color=red and /shoes?color=red&size=9 and /shoes?color=red&size=9&brand=nike and /shoes?color=red&size=9&brand=nike&sort=price-asc — each a separate crawlable URL. Each returning nearly identical content. Each consuming crawl budget and contributing nothing to indexing.
Session IDs in URLs
Some platforms append a unique session ID to every URL for tracking purposes. Every session ID is a unique URL. One page becomes thousands of variations, all returning identical content, all consuming crawl budget individually.
Internal search result pages
/search?q=shoes, /search?q=red+shoes, /search?q=nike+shoes — each a crawlable URL. Search result pages typically have no unique content worth indexing and no external links pointing to them. They are pure crawl budget consumption with zero return.
Soft 404 pages returning 200 status codes
Pages that serve error content while telling Google they’re perfectly healthy, indexable pages. Google crawls them obediently, repeatedly, because they look like live content. We wrote an entire deep-dive on this one — read it here — because it’s that specific, that common, and that damaging when it accumulates at scale.
Redirect chains
Every hop in a redirect chain is an additional server request. Long chains slow Googlebot down and consume budget on the overhead of following them rather than on reading content.
Duplicate content at multiple URLs
HTTP/HTTPS, WWW/non-WWW, trailing slash variations — multiple accessible URLs serving identical content. Each one is a separate crawl. Each one is budget spent on a page Google ultimately doesn’t need to index independently.
Orphan pages with no internal links
Pages that can only be reached through the sitemap or direct URL — never through internal link navigation. Googlebot has to spend budget discovering and crawling them through the sitemap alone, rather than naturally encountering them through link-following.
What Crawl Budget Looks Like When It’s Going Wrong
Signs that crawl budget is being wasted on your site:
- New blog posts or product pages taking two to four weeks to appear in search results after publication
- Google Search Console’s Coverage report showing large numbers of “Discovered — currently not indexed” pages — Google knows they exist but hasn’t gotten around to crawling them
- The crawl stats report in Search Console showing high volumes of crawled pages with very low average response size (thin, low-value URLs are being crawled)
- A large gap between the number of pages you’ve published and the number of pages indexed
- Important pages being crawled infrequently according to the URL Inspection tool’s last crawl date
How to Optimise Your Crawl Budget
Block low-value URLs from crawling via robots.txt
Internal search result pages, filter parameter URLs, session ID URLs — add Disallow rules to robots.txt to prevent Googlebot from spending time on these. Be surgical. Block the specific patterns generating junk URLs, not entire sections.
textUser-agent: *
Disallow: /search?
Disallow: /*?sessionid=
Handle faceted navigation with canonical tags and parameter configuration
For e-commerce filter pages: implement canonical tags on all parameter-generated URLs pointing to the base category URL (e.g., all /shoes?color=red&size=9 variants canonical to /shoes/). Additionally, use Google Search Console’s URL Parameter handling settings to indicate which parameters don’t generate unique content.
Fix soft 404s
Configure pages that return error content to return the correct HTTP status code — 404 or 410. Remove them from your sitemap. Stop paying crawl budget for ghost pages.
Collapse redirect chains to single hops
Covered in the redirects guide — worth repeating here because redirect chains are both a page speed problem and a crawl budget problem simultaneously.
Improve site speed and server response times
Googlebot adjusts its crawl rate based on your server’s response speed. Fast servers get crawled more aggressively. Slow servers trigger Googlebot to back off. Improving TTFB and server response directly increases the crawl rate Google applies to your site.
Improve internal linking to important pages
Pages that are heavily internally linked get crawled more frequently and allocated more crawl budget. If a page is important — a key product category, a cornerstone blog post, a high-value service page — make sure it’s linked from multiple places in your site architecture. Don’t bury it five clicks deep.
Keep your sitemap clean
Your sitemap should only list indexable, canonical, live URLs. A sitemap full of redirected URLs, noindexed pages, and filter parameter variations tells Google to spend crawl budget on junk. Clean sitemaps direct budget to valuable pages.
How to Monitor Crawl Activity
Google Search Console → Settings → Crawl Stats
The Crawl Stats report shows the total number of pages Googlebot crawled per day over the past 90 days, along with average response time and response codes. Look for:
- Sudden spikes in crawl activity — often indicates a new source of crawlable junk URLs
- Consistently high 404 response rates — broken internal links or deleted pages Googlebot keeps trying to reach
- Low average response size across a high crawl volume — suggests Googlebot is spending time on thin, low-value pages
Google Search Console → Coverage Report
“Discovered — currently not indexed” is your backlog. These are pages Google knows exist but hasn’t crawled yet. A large, growing backlog on a site that’s publishing regularly suggests Google is either prioritising other pages first or encountering crawl budget constraints.
Screaming Frog — Crawl Analysis
Crawl your site and export the full URL list. Sort by page type, response code, and content type. Look for volumes of low-content, parameter-generated, or redirecting URLs that are eating into the total crawlable URL pool.
Log File Analysis
For large sites, analysing your server access logs is the most accurate way to see exactly which URLs Googlebot is visiting, how frequently, and what response codes it’s receiving. Tools like Screaming Frog Log Analyser or custom log analysis in Python can process this. More technical — but more precise than any other method.
The TL;DR
- Crawl budget is the number of pages Googlebot will crawl on your site in a given period — it’s finite and determined by your server speed and site authority
- Crawl budget matters most for large sites, sites with parameter-generated URLs, and sites where new content is slow to appear in search
- The biggest budget drains: faceted navigation, session IDs in URLs, internal search result pages, soft 404s, redirect chains, and duplicate URL variants
- Fix by blocking low-value URLs via robots.txt, implementing canonicals on parameter variants, fixing soft 404s, collapsing redirect chains, and improving server speed
- Monitor via Search Console’s Crawl Stats report and Coverage report — the “Discovered — currently not indexed” backlog is your early warning system
Google’s time on your site is limited. Every page it spends budget on that contributes nothing is time taken away from the pages that do.
Running a large e-commerce site and watching your crawl budget disappear into filter URL hell? Drop the specifics in the comments. This is solvable.
— Subu, SEO by Subu
