Docs

Methodology

Every number in an aibotcheck report is reproducible. Here is exactly what we test, how we score it, and — just as important — what we cannot test.

Engine 1 · Static analysis (222 bots)

  • We fetch /robots.txt and parse it per RFC 9309 with Google extensions: group merging, longest-path-match, * wildcards and $ anchors, Allow-wins-on-tie.
  • Each bot's product token (e.g. GPTBot, Google-Extended) is evaluated for the root path. Most-specific User-agent group wins; * is the fallback; no robots.txt means allowed.
  • An HTML page served at /robots.txt with status 200 is treated as “no robots.txt” — a common misconfiguration.

Engine 2 · Live emulation (19 core bots)

  • We send a HEAD request (GET on 405/501) to your homepage with each core bot's documented User-Agent string, plus one control request with a normal Chrome UA.
  • Responses are classified: allowed (2xx/3xx), blocked (401/403/451 or challenge page — detected via cf-mitigated headers and challenge-body markers), rate-limited (429), server error (5xx), unreachable (timeout).
  • Static verdict × live outcome yields the two mismatch classes: robots allows but server blocks (silent AI invisibility) and robots blocks but server serves (unenforced policy).
  • The core subset is capped (~19) by Cloudflare Workers' subrequest budget; all other bots still get static verdicts.

Infrastructure & GEO signals

  • llms.txt (H1 required per spec), llms-full.txt, AGENTS.md — validated for existence, content type, and HTML-masquerade.
  • Homepage HTML: JSON-LD blocks (parsed, typed, error-counted), link rel="alternate" type="text/markdown", and six E-E-A-T markers (meta author, article meta, time elements, heading structure, content schema, entity schema).
  • Web Bot Auth: /.well-known/http-message-signatures-directory parsed for a JWKS keys set.

The AIVI formula

AIVI = 0.40·S_crawl + 0.25·S_geo + 0.20·S_infra + 0.15·S_fingerprint

S_crawl       = mean over core bots of (robots_allowed × live_reachability) × 100
S_geo         = 0.4·markdown_alternate + 0.3·schema_quality + 0.3·eeat_signals
S_infra       = 0.5·llms_txt + 0.3·llms_full_txt + 0.2·agents_md
S_fingerprint = 0.6·bot_defense_detected + 0.4·web_bot_auth

Grades: ≥ 85 Excellent · ≥ 65 Good · ≥ 40 Needs work · < 40 Poor. AIVI is one pillar of the Geolify GEO Score.

What we cannot test (and won't pretend to)

  • Vendor source IPs: probes originate from Cloudflare's network, not OpenAI's or Anthropic's IP ranges. A WAF rule keyed on verified IPs may treat real crawlers differently.
  • TLS fingerprints (JA4): we cannot reproduce each vendor's TLS stack.
  • Actual crawl behaviour: we test access, not whether a vendor chooses to crawl you.

For IP-verified, log-level evidence of real AI crawler visits, that's Geolify's job.

Run the check