How to block Google-Extended: Google's AI Training Crawler

AI Summary: Google-Extended is the specific robots.txt product token used by Google to control whether your site's content helps improve their generative AI models, including Gemini. To opt out of Google's AI training without affecting your Google Search rankings, add User-agent: Google-Extended and Disallow: / to your robots.txt file.

Role and policy boundary

Google-Extended was introduced by Google to give webmasters granular control over how their content is used in the AI era. Crucially, Google-Extended is not a separate web crawler with its own distinct User-Agent string; rather, it is a policy token. The actual crawling is still performed by standard Googlebot instances.

The policy boundary here is strictly between search indexing and AI model training. Allowing standard Googlebot ensures your site remains visible in Google Search and Google Discover. Blocking Google-Extended specifically instructs Google not to use the crawled data to train generative AI foundation models like Gemini or Vertex AI APIs. This separation allows site owners to protect their intellectual property from model ingestion while maintaining their organic search presence.

Layered verification

Because Google-Extended is a policy token rather than a standalone crawler, layered verification requires a nuanced approach.

The primary and officially supported method for opting out is the robots.txt file. Google's systems are designed to parse the Google-Extended directive during standard Googlebot crawls and apply the restriction downstream before the data enters the training pipeline.

However, because the network requests arrive via the standard Googlebot User-Agent (e.g., Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)), you cannot block Google-Extended using WAF or Nginx User-Agent rules without simultaneously blocking Google Search. Therefore, edge-level blocking is not a viable strategy for this specific token. Verification relies entirely on ensuring your robots.txt is correctly formatted and accessible to Googlebot.

WAF and Nginx remediation examples

As noted, you should not attempt to block Google-Extended using network-level User-Agent rules, as this will block all Googlebot traffic and remove your site from Google Search.

Instead, the remediation must happen strictly within the robots.txt file. Ensure the following snippet is present and correctly placed in your robots.txt:

User-agent: Google-Extended
Disallow: /

(Note: While some systems use JSON-based WAF rules like the below for other bots, doing so for Google-Extended is strongly discouraged as it shares the standard Googlebot User-Agent string.)

{
  "action": "block",
  "expression": "(http.user_agent contains \"Google-Extended\")",
  "description": "DO NOT USE: This will block Google Search"
}

Do not place this inside a broader User-agent: Googlebot block, as the specific Google-Extended directive must be parsed independently.

map $http_user_agent $block_google_extended_private {
    default 0;
    ~*Google-Extended 1;
}

server {
    location ~ ^/(admin|account|private|licensed|internal|api)/ {
        if ($block_google_extended_private) { return 403; }
        try_files $uri $uri/ =404;
    }
}

Review checklist

To ensure your Google-Extended policy is correctly implemented without harming your search visibility, follow this checklist:

1. Verify that User-agent: Google-Extended is explicitly declared in your robots.txt file.
2. Ensure that the standard Googlebot is still allowed to crawl the necessary pages for search indexing.
3. Check your edge firewall (WAF) to confirm there are no rules inadvertently blocking standard Googlebot IPs or User-Agents.
4. Monitor Google Search Console for any unexpected crawl errors or drops in indexed pages after applying the policy.

Need to optimize your entire site for AI search visibility? Run a comprehensive audit with Geolify.ai.