Directory / Google-Extended

Google-Extended

AI Training Crawler Google robots.txt behaviour unknown robots.txt token only — no crawler UA

Google extended web crawler bot

Allow Google-Extended in robots.txt

User-agent: Google-Extended
Allow: /

Block Google-Extended in robots.txt

User-agent: Google-Extended
Disallow: /

Blocking stops your content being used for model training. It does not remove you from AI search results — that's governed by separate search/assistant bots.

Official documentation: https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers

Is Google-Extended blocked on your site — even by accident?

Free dual-engine check: robots.txt verdicts for all 222 bots plus live User-Agent probes. Finds WAF blocks your robots.txt doesn't show.