Learn

How to block AI bots — without killing your AI visibility

The single most common mistake: blocking all AI bots and disappearing from ChatGPT, Perplexity and Google AI answers. Here's how to block precisely.

1. Know the three kinds of AI bots

• Training crawlers (GPTBot, ClaudeBot, CCBot, Meta-ExternalAgent, Bytespider): harvest content for future model training. Blocking them does not remove you from AI answers.
• Search index bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot): build the indexes AI assistants search. Block these and you stop appearing in AI search results.
• On-demand fetchers (ChatGPT-User, Claude-User, Perplexity-User): fetch a page live when a user asks. Blocking these breaks citations and link previews in AI chats.

2. The “protect content, keep visibility” robots.txt

# Block model training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Bytespider
Disallow: /

# Keep AI search & assistants (default allow — no rules needed)
# OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot: allowed

Note: Google-Extended is a robots.txt-only control token — it never crawls, it tells Google not to use your content for Gemini training. Blocking it does not affect Googlebot or your normal SEO rankings.

3. When robots.txt isn't enough

Some scrapers ignore robots.txt (Bytespider is the most notorious). For those, you need enforcement: User-Agent rules in your CDN/WAF, or IP verification for bots that publish IP ranges. Check each bot's page in our directory — we list whether it respects robots.txt and how to verify it.

4. Verify what's actually happening

Policy and reality drift apart: firewalls silently block bots you meant to allow, and scrapers ignore what you disallowed. Run a free check to see the real access status of 222 bots on your site.