What robots.txt is for
A robots.txt file sits at the root of your domain and tells crawlers which parts of your site they may request. It is the first thing most well-behaved bots fetch when they arrive, and it must live at exactly one location — the root of the domain, not in a subfolder.
Every subdomain needs its own file. A robots.txt at your main domain has no authority over a subdomain, which is worth knowing if you run tools or a blog on separate subdomains.
What it does not do
This is the most important thing to understand. Robots.txt is a request, not an enforcement mechanism. Well-behaved crawlers honour it; malicious scrapers ignore it entirely. It provides no security whatsoever, and listing a sensitive directory in it actively advertises that directory's existence to anyone who reads the file — which is public by definition.
Never use robots.txt to protect private content. Use authentication for that.
The second common misunderstanding is that blocking a page prevents it appearing in search results. It does not, reliably. If other sites link to a blocked page, Google may still index the URL, showing it without a description. To genuinely keep a page out of search results, allow crawling and use a noindex meta tag — because a crawler must be able to fetch the page to see that tag. Blocking it in robots.txt prevents that, which is a contradiction people fall into regularly.
What to block
Sensible candidates are administrative areas, shopping cart and checkout pages, internal search result pages, and faceted navigation that generates near-infinite URL combinations. That last case is the strongest practical reason for a robots.txt: preventing a crawler from wasting your server capacity on millions of filter permutations.
Be careful not to block resources the page needs. Blocking your CSS and JavaScript directories prevents Google from rendering the page as a visitor sees it, which can harm how the page is assessed.
The sitemap directive
Including a sitemap line points crawlers directly at your URL list. It costs nothing and is the one directive that actively helps discovery rather than restricting it. Use the full absolute URL.
Crawl delay
Crawl-delay asks bots to wait between requests. Google ignores it entirely, preferring its own adaptive rate limiting configured through Search Console. Bing and several others do honour it. Only set it if your server is genuinely struggling under crawler load.
Testing before you deploy
A syntax error in robots.txt can deindex an entire site, and this happens more often than you would think — most commonly a stray "Disallow: /" left over from a staging environment. Check the file with Google Search Console's robots.txt tester after uploading.
Frequently Asked Questions
Where must robots.txt be placed?
At the root of the domain, reachable at yourdomain.com/robots.txt. Every subdomain needs its own separate file.
Does robots.txt keep pages out of search results?
Not reliably. Blocked URLs can still be indexed if other sites link to them. To keep a page out of search, allow crawling and use a noindex meta tag instead.
Can I use it to protect private content?
No. It is a public file that offers no security, and listing a sensitive path advertises its existence. Use authentication instead.
Does Google honour crawl-delay?
No. Google ignores it and uses its own adaptive rate limiting. Bing and some other crawlers do respect it.
Should I block AI training crawlers?
That is a business judgement. Blocking them may reduce how often your content appears in AI-generated answers, which some sites want and others do not.