How to block Claude-SearchBot: Anthropic's Search Index Crawler

AI Summary: Claude-SearchBot is operated by Anthropic to navigate the web and build an index that improves search result quality for Claude users. To prevent your site from being indexed for Claude's search features, add User-agent: Claude-SearchBot to your robots.txt or block its User-Agent string at your edge firewall.

Role and policy boundary

Anthropic utilizes a suite of distinct crawlers to manage how Claude interacts with the web. While anthropic-ai collects data for training foundation models, and Claude-User fetches pages in real-time at a user's request, Claude-SearchBot serves a specific function: it crawls the web to build and refine Claude's internal search index.

This places Claude-SearchBot firmly in the category of AI search crawlers. The policy boundary here concerns discoverability. Allowing Claude-SearchBot enables your content to be surfaced when users ask Claude questions that require broad web searches. Blocking it removes your site from Claude's search index, which protects your content but may reduce your brand's visibility and citation rate within Anthropic's ecosystem.

Layered verification

To effectively manage Claude-SearchBot's access to your content, a layered verification approach is necessary.

The standard and most cooperative method is using the robots.txt file. Anthropic explicitly respects the Claude-SearchBot directive, allowing webmasters to opt out of search indexing without necessarily blocking live user fetches (Claude-User).

For strict enforcement, network-level blocking is recommended. Claude-SearchBot identifies itself with a specific User-Agent string (e.g., Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com)). By configuring your WAF or web server to drop requests matching this string, you create a hard technical barrier that enforces your policy regardless of robots.txt parsing.

WAF and Nginx remediation examples

To enforce your policy at the server level, you can implement User-Agent matching rules to block Claude-SearchBot.

For Nginx servers, you can use the following configuration snippet to return a 403 Forbidden status when the search bot attempts to access your site:

if ($http_user_agent ~* (Claude-SearchBot)) {
    return 403;
}

If your infrastructure relies on Cloudflare WAF, you can deploy a custom firewall rule using the following JSON expression to block the bot at the network edge:

{
  "action": "block",
  "expression": "(http.user_agent contains \"Claude-SearchBot\")",
  "description": "Block Anthropic Claude-SearchBot indexing"
}

Review checklist

To ensure your policy regarding Anthropic's search indexing is correctly configured, review these steps:

1. Determine your visibility strategy: Do you want to block AI training (anthropic-ai) but allow search indexing (Claude-SearchBot)?
2. Verify that your robots.txt includes the correct User-agent: Claude-SearchBot directives based on your decision.
3. If blocking, confirm that your edge firewall (WAF) or Nginx configuration is actively rejecting the Claude-SearchBot string.
4. Monitor your server logs to ensure that the distinct Anthropic bots are being handled according to your granular policies.

Need to optimize your entire site for AI search visibility? Run a comprehensive audit with Geolify.ai.