Documentation

Make an evidence-based crawler decision.

Use these guides in order: understand the actors, set the policy, then verify what your infrastructure actually does.

Understand

Start with the crawler categories, the limits of robots.txt, and the difference between policy and server enforcement.

Decide

Make a per-category decision and use copy-paste rules without accidentally removing your site from AI search.

Operate

Interpret your report, understand accuracy boundaries, and monitor for infrastructure drift.