📋 In This Guide
Why robots.txt Matters
robots.txt is one of the oldest and simplest standards on the web — a plain text file at your site's root that tells crawlers which parts of your site they may access. It's the first thing well-behaved bots check before crawling anything else, making it your site's front-door instructions for automated visitors.
Beyond controlling search engine crawling, robots.txt has taken on new relevance in 2026 as a way to signal — though not enforce — whether AI companies' crawlers may scrape your content for model training.
Crawl Control
Keep bots away from admin, cart or duplicate-content paths
AI Bot Signaling
Opt out of AI training crawlers that respect the standard
Sitemap Discovery
Points crawlers straight to your sitemap.xml location
Staging Protection
Keep unfinished sites out of search results
robots.txt Syntax Explained
Each rule block starts with a User-agent line specifying which bot the following rules apply to — * means "all bots." Disallow blocks a path; Allow explicitly permits a path, useful for carving out an exception inside a broader disallowed folder. Paths support a wildcard (*) and an end-of-string anchor ($), so /*.pdf$ blocks every URL ending in .pdf.
Blocking AI Training Crawlers
Since 2023, most major AI companies have introduced dedicated crawlers for gathering training data, and most publish documentation on how to block them via robots.txt — a voluntary opt-out, not a technical block.
| User-agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Training data collection |
| ChatGPT-User | OpenAI | Live browsing during ChatGPT conversations |
| CCBot | Common Crawl | Public dataset widely used to train many models |
| Google-Extended | Gemini / AI training (separate from Googlebot search) | |
| ClaudeBot | Anthropic | Training data collection |
| Claude-Web | Anthropic | Live browsing during Claude conversations |
| Applebot-Extended | Apple | Apple Intelligence training |
| Bytespider | ByteDance | Training data collection |
| Amazonbot | Amazon | Alexa / AI training and product data |
Need to generate a sitemap too?
Create a valid sitemap.xml from your URL list to reference in robots.txt.
robots.txt vs noindex — An Important Distinction
This trips up a lot of people: disallowing a page in robots.txt does not reliably keep it out of Google's index. If other sites link to a disallowed URL, Google can still show that URL in search results — just without a description, since it was never allowed to crawl the page's content.
To reliably prevent a page from appearing in search results, use a <meta name="robots" content="noindex"> tag directly on the page — and critically, that page must not be disallowed in robots.txt, or Google can never crawl it to see the noindex instruction in the first place.
5 Common Robots.txt Mistakes to Avoid
Disallow: / under User-agent: * blocks your whole site from every crawler — always double-check the generated file before uploading it live.Frequently Asked Questions
A plain text file at the root of a website telling crawlers which parts of the site they're allowed or not allowed to access, following the long-standing Robots Exclusion Protocol.
Not reliably on its own — a disallowed URL can still appear in results without a snippet if other pages link to it. Use a noindex meta tag on a crawlable page for reliable exclusion.
Yes, for crawlers that respect robots.txt, including most major AI companies' bots. Adding a Disallow rule for that bot's user-agent blocks it, though compliance is voluntary.
No. It's a voluntary standard, not a technical or legal access control. Well-behaved crawlers generally respect it, but nothing stops a bot from ignoring it entirely.
To the root directory of your website, so it's accessible at yoursite.com/robots.txt exactly — not in a subfolder.
It suggests a minimum wait time between bot requests. Google ignores it and uses Search Console's crawl-rate settings instead; Bing and some others do respect it.
Yes, if a staging site is publicly accessible — a blanket disallow rule prevents premature indexing, but remember to remove it before going live.