Crawl-delay Directive
A non-standard robots.txt directive originally introduced to throttle the request frequency of search crawlers and protect server performance.
AI Summary: The Crawl-delay directive specifies the number of seconds a crawler should wait between successive requests to a server. While historically honored by Bingbot and Yandex, major AI crawlers and Googlebot ignore Crawl-delay in favor of HTTP 429 status codes and edge rate limiting.
Technical Definition
The Crawl-delay directive is a parameter within robots.txt that instructs compliant crawlers to introduce a mandatory pause (measured in seconds) between consecutive HTTP requests to a domain:
User-agent: Bingbot
Crawl-delay: 5
Vendor Support & RFC 9309 Status
The official Internet Standard for robots exclusion (RFC 9309) does not specify Crawl-delay. As a consequence, vendor support is inconsistent:
- Googlebot: Completely ignores
Crawl-delay. Uses automatic adaptive throttling based on server latency. - Bingbot: Honors integer values.
- AI Crawlers (OpenAI, Anthropic, Perplexity): Do not guarantee compliance with
Crawl-delay. Instead, they rely on server response codes (429 Too Many Requests) to back off.
Recommended Alternative: HTTP 429 & Retry-After
Rather than relying on unstandardized robots directives to protect server capacity, implement explicit rate limiting at the edge with RFC-compliant headers:
HTTP/1.1 429 Too Many Requests
Content-Type: text/plain
Retry-After: 30
Rate limit exceeded for automated crawler. Please retry after 30 seconds.
Learn how to optimize crawler cadence without damaging indexation. Analyze your setup with Geolify.ai.