Crawl budget

Crawl budget is the number of pages that a search engine such as Google is willing and able to crawl on your website within a given period. Each site is allocated its own crawl budget, based on factors such as authority, speed, error messages and how often the content changes. For an SME website with 200 pages, this is hardly an issue, but for online shops with thousands of product pages or news sites with daily updates, the crawl budget determines whether all your valuable content ends up in the index at all.

How crawl budget works in practice

Google sends automated bots (crawlers) to your site to scan and index pages. These bots have a limit: they do not want to cause server load and prefer to spend their time on relevant pages. The crawl budget consists of two components. Firstly, crawl capacity: how much load your server can handle without slowing down. Secondly, crawl demand: how much interest Google has in your content. A slow site with lots of 404 errors and duplicate content wastes budget on useless pages. A fast site with unique, valuable content is visited more frequently and more thoroughly. In practice, this means that an online shop with 5,000 products will only see 800 pages crawled per week if its technical infrastructure is disorganised.

Why crawl budget matters more now than it did five years ago

In the past, Google used to crawl almost everything, no matter how large the site was. Since 2016, Google has been communicating more explicitly about crawl budget, mainly because the web is growing exponentially and search engines are becoming more selective. For small sites (under 1,000 pages), this is rarely a problem. However, as soon as you expand an online shop with variants, filters and categories, or manage a knowledge base with hundreds of articles, you will come up against limitations. Pages that are not crawled do not appear in search results. According to Google Developers, efficient crawl budget management is particularly crucial for sites with more than 10,000 URLs or sites that add a large number of new pages every day.

What crawl budget delivers for Dutch online shops and content platforms

If your crawl budget is wisely managed, Google will index your new products, blog articles or landing pages more quickly. This means a shorter time between publication and discoverability. For an online shop with seasonal offerings, this can make the difference between appearing in the search results during a peak period or not. Technical optimisation, such as cleaning up 404 pages, blocking irrelevant URL parameters and improving server response times, boosts efficiency. As part of a well-thought-out SEO strategy, you ensure that crawlers spend their time on pages that generate conversions, rather than on filtered sort orders or empty search results. Monkey Vision helps small and medium-sized companies with technical audits to identify crawl wastage and measurably improve indexing speed.

Optimising your crawl budget is not an end in itself, but a means of ensuring that Google finds and indexes your valuable pages quickly. Particularly for sites with many pages or frequent updates, efficient budget management makes the difference between visibility and invisibility. Below are four specific situations in which crawl budget optimisation has a measurable impact.

Online shops with thousands of product variants and filters

An online shop with 2,000 products can easily generate 15,000 URLs through filters, sorting options and pagination. Every combination of colour, size and price creates a unique URL. If you do nothing, Google will waste its crawl budget on pages such as ‘/products?filter=red&sort=price-low-high&page=3’ whilst your new collection remains invisible for weeks. The solution: block filtered URLs via robots.txt or use canonical tags to merge duplicate content. This way, you direct crawlers to product pages and categories that drive conversions. An online shop that implements this often sees new products appear in the index within two weeks, rather than after a month.

News sites and blogs with daily content updates

A news platform or corporate blog publishes new articles daily. Google needs to index these quickly to remain relevant in search results. However, if your site also hosts old archive pages, tag overviews and author pages that offer little value, you’ll be spreading your crawl budget across hundreds of pages. Prioritise by submitting an XML sitemap containing only recent articles and evergreen content. Use the ‘lastmod’ tag to tell Google which pages have been updated. Block empty tag pages or author pages with no content. This way, Google will crawl your new articles first, improving your visibility for current search terms more quickly.

Migrations and rebrands involving URL changes

A website migration or rebranding often involves changing hundreds of URLs. Google has to recrawl all these new URLs and remove the old ones from the index. If your crawl budget is low due to technical issues or a slow server, this process takes months rather than weeks. The result: visitors end up on 404 pages, rankings drop and turnover declines. Prevent this by cleaning up your backlinks, setting up redirects correctly and increasing server speed before the migration. After the migration, submit an updated sitemap and use Google Search Console to monitor how many pages are being crawled each day. Monkey Vision supports migrations with technical audits to prevent crawl budget wastage and double the indexing speed.

International websites with multiple language versions

An SME operating internationally often has the same content in the United Kingdom, English and German. Without correct hreflang tags and canonical settings, Google crawls all versions separately, which spreads the budget across duplicate content. Ensure that each language version is correctly tagged and that Google understands which version is intended for which country. Block any automatically generated translations or test subdomains. This way, Google will allocate its budget to the versions that actually generate visitors.

When is crawl budget optimisation the right choice, and when is it not?

Optimising the crawl budget only really pays off for sites with more than 1,000 indexed pages or with frequently changing content. Do you have a website with 50 pages that are updated once a month? Then the crawl budget is not a priority. Google usually crawls small sites in full within a week. In that case, focus first on on-page SEO, speed and content quality. Crawl budget only becomes a bottleneck if you see in Google Search Console that important pages haven’t been crawled for weeks, or if you publish new content daily that isn’t appearing in the index quickly enough. In such cases, a technical audit is the first step.