Docs
Frequently asked questions
How long does a check take?
Usually 10–30 seconds. We fetch your homepage, robots.txt, llms.txt files and the Web Bot Auth directory, then probe your site live with 19 core bot User-Agents at bounded concurrency.
Will the check hurt my site or analytics?
No. A full check is roughly 30 lightweight requests — comparable to a single page view with assets. Probes are HEAD requests where possible.
Why does my report show 'robots.txt allows but blocked'?
Your robots.txt permits the bot, but a WAF, CDN bot-management rule, or security plugin returned 403/challenge to its User-Agent. This is the most common cause of accidental AI invisibility — check your WAF's bot rules and any 'block AI crawlers' toggles.
My robots.txt blocks a bot but the live probe says 'reachable'. Is that a bug?
No. robots.txt is voluntary — our probe measures what your server actually serves. Compliant bots will honour the disallow; non-compliant ones won't. If enforcement matters, add a server-level block.
How accurate is UA emulation vs the real crawler?
It measures how your server treats each User-Agent string from Cloudflare's network. It cannot reproduce vendor IP ranges or TLS fingerprints, so an IP-allowlisting WAF may treat real crawlers differently. We say this on every report — for IP-verified evidence from server logs, use Geolify.
What's a good AIVI score?
65+ is good; 85+ is excellent. Most unoptimized sites score 40–65: reachable but missing llms.txt, schema, or markdown alternates. The report links a fix for every gap.
Do you store my report? Can I delete it?
Reports are stored so your link keeps working, and are visible to anyone with the link (IDs are unguessable). Email reports@aibotcheck.tech to delete one.
How does weekly monitoring work?
A scheduled job re-runs your check weekly and emails you a digest with score changes and newly blocked/unblocked bots. One click unsubscribes.