Docs
Allow or block AI crawlers?
There is no universal answer — there is a per-category answer. Decide by what each bot gives you back.
The decision framework
| Category | You give | You get | Default advice |
|---|---|---|---|
| Training (GPTBot, CCBot, ClaudeBot) | Content for model training | Nothing direct | Your call — blocking costs no visibility |
| AI search (OAI-SearchBot, PerplexityBot) | Indexable pages | Citations + referral traffic | Allow — this is AI-era SEO |
| Assistants (ChatGPT-User, Claude-User) | On-demand page reads | Your links work inside AI products | Allow |
| Aggressive scrapers (Bytespider) | Bandwidth + content | Nothing | Block — at server level, it ignores robots.txt |
Policy: “Visible in AI, no training” (most sites)
# Block training
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Allow AI search & assistants (default allow covers them)
User-agent: *
Allow: /Policy: “Maximum AI visibility” (marketing sites, docs)
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xmlPair it with llms.txt (generate one) and JSON-LD schema for best results.
Policy: “Block everything AI” (paywalled/licensed content)
User-agent: GPTBot
User-agent: CCBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: / robots.txt is not enforcement. Bots like Bytespider are reported to ignore it. Real blocking needs WAF/CDN rules (e.g. Cloudflare's AI-crawler toggles) or User-Agent filters at the server. And enforcement cuts the other way too: your WAF may already be blocking bots you want — verify with a check.
Per-bot snippets
Every bot page in the directory has copy-paste allow/block snippets and the consequence of each choice.
Audit your current policy
Reading robots.txt policy…