Google's official line is that sites under a million pages shouldn't worry about crawl budget. In practice, a new 50,000-page programmatic site gets a probationary trickle — often 500–2,000 Googlebot fetches a day — and at that rate full coverage takes months, with re-crawls of changed pages lagging weeks behind. The official guidance assumes crawl capacity is the constraint; on template-generated sites the constraint is crawl demand, which is Google's estimate of whether your pages are worth fetching. You influence capacity with server speed and demand with everything else.
Read the logs, not the crystal ball
Search Console's Crawl Stats report is a decent summary, but it's sampled, delayed, and won't show you per-section waste. Your access logs will:
# Googlebot fetches per day
zcat access.log*.gz | grep 'Googlebot' \
| awk '{print $4}' | cut -d: -f1 | tr -d '[' | sort | uniq -c
# where the budget actually goes, by section
grep 'Googlebot' access.log | awk '{print $7}' \
| cut -d/ -f2 | sort | uniq -c | sort -rn | head
# verify a claimed Googlebot IP before trusting any of this
host 66.249.66.1 # → crawl-66-249-66-1.googlebot.comFake Googlebots are common enough to skew the numbers; verify by reverse DNS or against Google's published IP ranges. The three numbers to extract: fetches per day, the share going to each page archetype, and the share wasted — parameters, redirects chains, 404s, and pages you don't even want indexed. On the first log review of this site, 22% of Googlebot hits were going to paginated category URLs beyond page 3, which nobody searches for. That's a fifth of the budget bought back with one template change.
The funnel: Discovered → Crawled → Indexed
Search Console's Pages report describes a funnel, and the stage where URLs pile up tells you which problem you have:
| GSC status | Diagnosis | Fix lives in |
|---|---|---|
| Discovered – currently not indexed | Google knows the URL, doesn't think it's worth fetching | Internal linking, sitemap quality, site reputation |
| Crawled – currently not indexed | Fetched, judged not worth indexing | The content — this is a quality verdict on the template |
| Duplicate states | Signal conflicts | Canonicals and URL hygiene |
"Discovered not indexed" stacking up is the classic new-programmatic-site state: a sitemap full of URLs with no internal-link reputation behind them. The fix is never "submit the sitemap again" — it's giving those pages real inbound paths, which is what a proper internal linking architecture does. "Crawled not indexed" at scale is worse news: Google spent the fetch and declined to index, and when 40% of a pattern sits there, the pattern itself reads as thin content.
The levers, ranked by what they've moved
- Server speed. Googlebot adjusts its request rate to what your server tolerates. Keep HTML TTFB under ~600ms and error rates near zero and crawl rate climbs on its own; sustained 5xx or multi-second responses and it backs off within days. This is the only lever that raises capacity — a static site behind a CDN mostly wins it for free.
- Kill crawl traps. Faceted parameters, calendar pages, sort orders, session IDs. Every URL pattern that multiplies pages without adding content is budget subtracted from pages that matter. Don't generate the links; block in robots.txt only what you can't stop linking to.
- Internal links. Crawl frequency tracks link depth. Pages within three clicks of the homepage get re-crawled; orphans reachable only via the sitemap get fetched once a quarter if that.
- Honest sitemaps and 304s. Real
lastmodvalues teach Google to trust your change signals, and serving304 Not Modifiedon unchanged pages makes each recrawl nearly free, so the same budget covers more URLs — the mechanics are in sitemaps past 50k. - Fast, decisive 410s. Removed pages should return 410, not soft-404 or redirect-to-homepage. Googlebot re-checks 404s for a long time; 410s get dropped faster and stop absorbing fetches.
Pruning is a crawl strategy, not just a quality one
The counterintuitive move that works: deleting or noindexing your worst pages speeds up crawling of your best ones. Crawl demand is set at the site and pattern level, so 20,000 zero-impression pages don't just sit there neutrally — they drag down the average that determines how eagerly everything else is fetched.
The mechanical version: pull 12 months of GSC data, find pages with zero impressions (not zero clicks — zero impressions), and for each decide: improve the data behind it, fold it into a parent page, or remove it with a 410. On directory-scale sites I've watched sections go from crawled-monthly to crawled-weekly within six weeks of a 25–30% prune, with the surviving pages' indexation rate climbing. Google notices when the average page it fetches from you stops being a waste of its time.
What I'd do
Monthly: one log review — fetches/day trend, waste percentage, top crawled sections versus top earning sections; the mismatch is your to-do list. Quarterly: a prune pass on zero-impression pages. Continuously: TTFB under 600ms, real lastmod, 410s for removals, and no new URL patterns without asking who's going to link to them. Crawl budget follows page quality — treat the crawl stats as a lagging indicator of whether your pages deserve the fetches, because that's exactly how Google treats them.