How to block Claude-User: Anthropic's Real-Time Browsing Bot

AI Summary: Claude-User is the bot Anthropic uses when a user explicitly asks the Claude assistant to browse a specific web page for real-time information or summarization. To block Claude from reading your site's content during user chats, add User-agent: Claude-User to your robots.txt or block its User-Agent string at the network edge.

Role and policy boundary

Anthropic's crawler ecosystem is highly granular. While anthropic-ai is used for bulk training data ingestion and Claude-SearchBot builds search indexes, Claude-User serves a distinct, user-driven purpose. It acts as an AI assistant browser, fetching web pages in real-time only when a user explicitly prompts Claude to look up current information, summarize an article, or analyze a specific URL.

The policy boundary here separates automated bulk scraping from live, user-initiated assistance. Many publishers choose to block training crawlers to protect their intellectual property, while explicitly allowing Claude-User. Allowing this bot ensures that when a human user asks Claude about your brand or content, the assistant can access the live page and provide accurate, cited information. Blocking Claude-User means Claude will inform the user that it is unable to access your website.

Layered verification

To manage access for Claude-User, a layered verification approach ensures your specific policies are respected.

The primary and officially supported method is the robots.txt file. Anthropic respects the Claude-User directive, allowing webmasters to control real-time access independently of training or search indexing.

For strict enforcement, network-level blocking is required. The bot identifies itself with a specific User-Agent string (e.g., Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)). Implementing rules at your WAF or web server to drop requests from this User-Agent provides a hard technical barrier, ensuring the bot cannot access your site even if the robots.txt is misconfigured or bypassed.

WAF and Nginx remediation examples

To enforce your policy at the server level, you can implement User-Agent matching rules to block Claude-User.

For Nginx servers, you can use the following configuration snippet to return a 403 Forbidden status when the bot attempts to access your site:

if ($http_user_agent ~* (Claude-User)) {
    return 403;
}

If your infrastructure uses Cloudflare WAF, you can deploy a custom firewall rule using the following JSON expression to block the bot at the network edge:

{
  "action": "block",
  "expression": "(http.user_agent contains \"Claude-User\")",
  "description": "Block Anthropic Claude-User live browsing"
}

Review checklist

To ensure your policy regarding Anthropic's real-time browsing is correctly configured, review these steps:

1. Decide on your business policy: Do you want to block AI training (anthropic-ai) but allow live user summaries (Claude-User)?
2. Verify that your robots.txt includes the correct User-agent: Claude-User directives based on your decision.
3. If blocking, confirm that your edge firewall (WAF) or Nginx configuration is actively rejecting the Claude-User string.
4. Test the configuration by asking Claude to summarize a specific URL on your site to confirm the block is effective.

Need to optimize your entire site for AI search visibility? Run a comprehensive audit with Geolify.ai.