Docs

What is an AI crawler?

An AI crawler is any automated client that fetches web content for an AI system. The critical insight: they are not one thing. Three very different jobs hide behind the label, and each deserves a different robots.txt decision.

1. Training crawlers

Examples: GPTBot, CCBot, ClaudeBot, Bytespider. They harvest content to train foundation models. Your content improves the model, but you get no traffic in return. Blocking them is a legitimate copyright/licensing stance and costs you nothing in visibility.

2. AI search / retrieval crawlers

Examples: OAI-SearchBot, PerplexityBot, Claude-SearchBot. They index your pages so AI answer engines can cite you. Blocking them removes you from AI search results — the fastest-growing referral channel. Most sites should allow these.

3. Assistant / agent fetchers

Examples: ChatGPT-User, Claude-User, Perplexity-User. They fetch a single page in real time because a human asked. Blocking them means AI assistants literally cannot open your links for users. Some (like Perplexity-User) ignore robots.txt by design because the request is user-initiated.

Control tokens ≠ crawlers

Tokens like Google-Extended and Applebot-Extended never fetch anything. The regular crawler (Googlebot, Applebot) fetches; the token only controls whether that content may be used for AI training. You manage them purely in robots.txt.

Why robots.txt alone can't answer “am I visible to AI?”

robots.txt is a request; WAFs and CDN bot management are enforcement. They routinely block AI User-Agents that robots.txt allows — silently. That's the gap our dual-engine check exposes.