robots.txt

Robots.txt is a text file that you place in the root directory of your website to instruct search engines which parts of your site they may or may not index. The file acts as an access control mechanism: every search engine bot reads this file before it starts crawling pages. For small and medium-sized companies, robots.txt is particularly relevant for managing crawl budgets efficiently, shielding sensitive pages and preventing technical SEO issues.

How robots.txt works in practice

When a search engine such as Google visits your website, the bot first checks whether a robots.txt file exists at jouwdomein.nl/robots.txt. In that file are rules with 'User-agent' (which bot), 'Disallow' (do not crawl) and 'Allow' (do crawl). For example, a typical rule is 'User-agent: * / Disallow: /wp-admin/', meaning all bots should skip the WordPress admin pages. The file is publicly readable, so anyone can see which parts you are shielding. That makes it not a security measure, but an efficiency tool. Google Developers publishes the official specification and best practices on their robots.txt documentation.

Why robots.txt came into being and why it still matters today

The Robots Exclusion Protocol originated in 1994 as an informal agreement between website owners and search engines to save bandwidth and respect privacy. Back then, servers were slow and crawlers aggressive. Today, it is still relevant because Google allocates a limited crawl budget per site. If you have 500 product pages but Google crawls mostly your filter and sort pages, you're wasting that budget. With robots.txt, you direct the bot to what really counts. You also prevent internal search results, test environments or thank-you pages from ending up in the index, which reduces your duplicate content risk.

What robots.txt brings to technical SEO

In practice, we often see with SME clients that a well-configured robots.txt file offers three direct benefits. Firstly: faster indexing of new content, as Google spends its crawl budget on relevant pages rather than on pagination or filters. Secondly: less noise in Search Console, as you won’t receive notifications about blocked CSS or irrelevant URLs. Thirdly: greater control over what is and isn’t discoverable, for example by keeping a staging environment or client portal out of the index. A well-thought-out SEO strategy often begins with cleaning up the robots.txt file, so that crawlers can efficiently explore your site and prioritise the right pages.

Now that you know what robots.txt is and how it works, let's look at concrete situations in which SMEs deploy this file. The power is not in blocking everything, but in strategically steering search engines to the pages that bring conversion and avoiding pages that only cause noise.

Send crawl budget to commercial pages

An online shop with 800 products and 50 filter combinations per category can quickly generate thousands of URLs. Google does not crawl everything in one go. By blocking filter URLs, sorting options and pagination in robots.txt, you force the bot to index product pages and categories first. In practice, we’ve found that this enables online shops to get new products into the index more quickly within two weeks. An example: a garden centre in Greater London blocked all URLs containing ‘?orderby=’ and ‘?filter_kleur=’, causing Google to shift its crawl budget towards seasonal products. Result: 30% faster indexing of new items during the spring season. Please note: never block your main categories or important landing pages, as this will cause you to disappear from Google.

Shielding internal search results and thank-you pages

Many companies have a search function on their site that generates URLs such as '/search/?q=garden chairs'. If Google indexes those pages, you are competing with your own product pages and delivering a poor user experience. The same goes for thank you pages after a form or order: those are meant for a single visitor, not organic traffic. By including 'Disallow: /search/' and 'Disallow: /thank you/' in your robots.txt, you keep those pages out of the index. A B2B service provider in London saw its bounce rate drop by 12% after it blocked internal search results because visitors no longer landed on empty search pages via Google. Don't forget to check your XML sitemap too: blocked URLs don't belong there.

Staging environments and test pages off-index

Developers often work on a subdomain such as staging.yourcompany.co.uk or in a directory such as /test/. If that environment is not password-protected, Google may index it. This leads to duplicate content and confusion amongst visitors who accidentally land on the test version. By placing a separate robots.txt file on your staging environment with ‘User-agent: * / Disallow: /’, you block all bots. Another approach is to block the ‘/dev/’ or ‘/staging/’ directory in your live robots.txt file. For web projects, Monkey Vision we standardise this: every staging environment is given a full ‘Disallow’, so that clients never run into problems with duplicate indexing during the development phase.

When robots.txt is the right choice and when it is not

Robots.txt is suitable for pages you do not want crawled but do not contain sensitive information. The file is publicly readable, so anyone can see which folders you are blocking. Do you really want to secure a page? Then use password protection or a noindex meta tag in combination with access control. Robots.txt is also not suitable if you want to completely remove a page from the index that is already indexed: then you need a noindex tag or a 410 status code. In practice, you often see companies using robots.txt for privacy-sensitive URLs, but this is a misunderstanding. Google respects the block, but the URL can still appear in the index if external links point to it. In short: robots.txt directs crawlers, but does not guarantee privacy or complete deindexation.