IP Allowlist
A network security control that restricts incoming traffic strictly to explicitly approved IP addresses and CIDR prefixes, rejecting all unlisted connections.
AI Summary: An IP allowlist is an access control list that permits traffic only from verified IP addresses and CIDR subnets. Because User-Agent headers are easily faked, publishers maintain IP allowlists based on officially published JSON feeds from vendors like OpenAI, Google, and Bing.
Technical Definition
An IP Allowlist is a deterministic perimeter control enforced at the network layer (L3/L4) or edge WAF (L7) that grants access exclusively to connections originating from a curated set of IP addresses or Classless Inter-Domain Routing (CIDR) blocks.
In crawler governance, IP allowlists serve as the only reliable defense against User-Agent spoofing.
Official Crawler IP Feeds
Legitimate search and AI vendors maintain cryptographically signed or publicly hosted JSON endpoints listing their active IP prefixes:
- OpenAI (ChatGPT-User, GPTBot): Published via
https://openai.com/gptbot.json - Google (Googlebot): Published via
https://developers.google.com/search/apis/ipranges/googlebot.json - Bing / Microsoft: Published via Microsoft Download Center JSON feeds.
Automating Edge Allowlisting in Cloudflare WAF
{
"description": "Allow verified OpenAI IP range only",
"action": "allow",
"expression": "(ip.src in {20.171.207.0/24 20.171.206.0/24} and http.user_agent contains "GPTBot")"
}
Ensure your edge WAF properly verifies genuine AI bot IPs without accidental false positives. Audit with Geolify.ai.