Crawl budget on large content sites: where to start
On a site with a few hundred pages, crawl budget is not worth worrying about. On a catalogue with hundreds of thousands of URLs, such as a video, news or e-commerce site, it becomes one of the main limits on growth. Search engines only crawl a site so fast, and every request spent on a duplicate, an expired page or an endless filter combination is a request not spent on new content.
This is the order we usually work in.
1. Find out where crawling actually goes
Start with data, not assumptions. The Crawl Stats report in Google Search Console shows total requests, average response time and a breakdown by response code and file type. It is a sample, but it shows the shape of the problem.
Server access logs are better. Filter them to genuine Googlebot and Bingbot requests (confirm the IP address with a reverse DNS lookup, because many scrapers copy the user agent) and group the URLs by page template. On most large sites a handful of URL patterns account for most of the crawling, and they are often not the pages that earn traffic.
2. Remove URL patterns that should not exist
The usual causes are parameters that create new addresses for the same content: sort orders, tracking codes, session IDs and filter combinations. Each one multiplies the number of URLs a crawler can discover.
Fix them at the source where possible. Link to the clean URL internally, strip tracking parameters before they reach the page, and do not generate links to filter combinations that have no search value. Canonical tags help search engines choose the right version, but they do not stop the crawling, so treat them as a backstop rather than the fix.
3. Answer removed content with the right status code
Large catalogues lose pages all the time: products sell out, videos are taken down, articles are withdrawn. If those URLs redirect to the home page, or return a normal page that says “not found”, crawlers keep coming back to them.
Return 404 or 410 for content that is gone for good, and use a 301 redirect only when there is a genuine replacement page. A 410 tells search engines the removal is deliberate, and in our experience those URLs drop out of crawling sooner.
4. Make important pages easy to reach
Crawlers follow links. New content that can only be reached through deep pagination, page 40 of a category, can wait weeks to be found. Link new items from pages that are crawled often, such as the home page, category pages and related-content blocks, and keep pagination shallow by splitting very large categories. Internal search result pages should normally be kept out of the index altogether.
5. Use sitemaps to signal freshness
XML sitemaps do not force crawling, but they help search engines discover URLs and decide what to recheck. Split them by content type and age, for example one file for items added in the last seven days and separate files for the archive, and keep the lastmod date honest. A lastmod that changes on every request teaches search engines to ignore it.
6. Keep the server fast
Crawl rate is tied to how quickly and reliably your server responds. If response times climb or the server returns 5xx errors, Googlebot slows down. Caching pages for anonymous visitors, serving static files from a CDN and making sure the origin does not do expensive work for bot traffic all raise the ceiling on how much can be crawled each day.
Measuring the effect
Change one thing at a time where you can, and watch three numbers over the following weeks: crawl requests to the URL patterns you care about, the count of indexed pages in the Page indexing report, and the time between publishing a page and its first appearance in search results. When those move in the right direction, organic traffic usually follows.