robots.txt is one of the smallest files on your entire site — often under 200 bytes — and also one of the most widely misunderstood. It doesn't hide pages from Google, it can't stop a determined bot, and one specific directive doesn't even work on the crawler most people write it for. Here's what it actually does.
robots.txt is a plain text file placed at the root of a website that tells crawlers which parts of the site they're permitted or asked not to access. It follows the Robots Exclusion Protocol, a standard dating back to 1994, respected voluntarily by well-behaved crawlers — search engines, AI companies, and most legitimate automated tools.
A robots.txt file is built from groups, each beginning with a User-agent line that names the bot the following rules apply to, followed by one or more Disallow or Allow lines.
User-agent: * Disallow: /admin/ Disallow: /cart/ Allow: / Sitemap: https://example.in/sitemap.xml
| Directive | Meaning |
|---|---|
User-agent: * | Applies the following rules to all crawlers |
Disallow: /path/ | Requests the bot not crawl anything under that path |
Allow: /path/ | Explicitly permits a path, often used to carve out an exception inside a disallowed folder |
Sitemap: [url] | Points crawlers to your sitemap.xml — can appear anywhere in the file |
This is the single most common misconception about robots.txt: disallowing a page prevents crawlers from fetching its content, but it does not prevent that URL from appearing in search results if other pages link to it. Google can still index and display the bare URL — typically without a proper title or description snippet, since it never actually crawled the page to generate one.
To reliably keep a page out of search results, use a noindex meta tag on the page itself instead. Counter-intuitively, this requires the page to remain crawlable — if robots.txt blocks the page, Google can't ever see the noindex instruction to honour it.
Since generative AI systems began training on web content at scale, many site owners want the option to opt specific pages or their entire site out of that training data collection. This is done the same way as any other bot-specific rule — a dedicated User-agent block per crawler.
| Bot Name | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data collection for GPT models |
| CCBot | Common Crawl | Open web archive used by many AI labs for training |
| ClaudeBot | Anthropic | Training data collection for Claude models |
| Google-Extended | Controls use in Gemini/Bard training, separate from regular Googlebot search crawling |
User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: /
Blocking an AI training bot's user-agent doesn't affect your regular Googlebot search crawling or SEO — each bot is addressed independently, so your rules for User-agent: * or Googlebot specifically remain untouched by AI-bot-specific blocks.
Crawl-delay requests a minimum number of seconds a bot should wait between requests, intended to reduce server load from aggressive crawling. Here's the catch: Google has publicly stated it does not support this directive and ignores it entirely — relying instead on its own crawl-rate algorithm and the manual crawl-rate controls available in Search Console.
| Crawler | Respects Crawl-delay? |
|---|---|
| Googlebot | No |
| Bingbot | Yes |
| Most other major crawlers | Varies |
A realistic file for a small e-commerce site, controlling search crawling, opting out of AI training, and referencing the sitemap:
User-agent: * Disallow: /cart/ Disallow: /checkout/ Disallow: /admin/ Allow: / User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / Sitemap: https://example.in/sitemap.xml
This blocks the cart, checkout, and admin paths from general crawling (which don't need to be indexed anyway), opts out of two major AI training crawlers entirely, and still points every well-behaved bot to the sitemap for efficient discovery of everything else. Build your own with ToolLoom's Robots.txt Generator — select paths and bots to block with a few clicks, no manual syntax needed.
robots.txt carries no technical enforcement mechanism whatsoever — it's a request, honoured by convention and reputation, not a lock. Reputable search engines and most major AI companies generally respect it because ignoring it would damage their standing and invite regulatory or legal scrutiny, but nothing in the protocol itself stops a bot from simply disregarding the file entirely.
Never rely on robots.txt to protect genuinely sensitive or private content — use real access control (authentication, server-level restrictions) for anything that actually needs to stay confidential.
yoursite.com/robots.txt, never in a subfolder/admin and /admin/ can match different sets of URLs depending on the crawler's interpretation