Learn

Silent blocks: when your firewall hides you from AI

The most damaging AI visibility problem is the one nobody configured on purpose.

What a silent block looks like

Your robots.txt allows OAI-SearchBot. But when OpenAI's crawler actually requests a page, your CDN's bot-management layer sees a non-browser User-Agent, scores it as a threat, and returns a 403 or a JavaScript challenge the bot can't solve. Result: to every static checker you look “open to AI”, while in reality no AI engine can read a single page.

Where silent blocks come from

• CDN bot management defaults — “block AI scrapers” toggles (Cloudflare, Akamai, Fastly) often block search & assistant bots too.
• WAF rules — generic “block bad bots” rulesets that match any UA containing “bot” or “GPT”.
• Rate limiters & challenges — CAPTCHA or JS challenges served to all non-browser traffic.
• Geo/IP filters — blanket blocks on cloud provider IP ranges where AI crawlers run.

How we detect them

aibotcheck sends real HTTP requests to your homepage using each bot's official User-Agent string and compares the outcome against your robots.txt policy. If robots.txt says allowed but the request comes back 403/challenge, that's flagged as a silent block. (Honest caveat: our requests come from our IPs, not the bot operator's — so we detect UA-level blocking; IP-level allowlists can still differ. Full IP-verified evidence comes from analyzing your real traffic — that's Geolify.)

How to fix a silent block

1. Identify which bots are affected — run the free check.
2. In your CDN/WAF, find the rule matching those User-Agents (bot-management category “AI crawler” is the usual culprit).
3. Create explicit allow exceptions for the bots you want (e.g. OAI-SearchBot, PerplexityBot), ideally combined with the operator's published IP ranges.
4. Re-run the check to confirm the fix, and keep the report link for your records.