Rate Limiting

AI Summary: Rate limiting is a defensive traffic control mechanism that restricts the request frequency of clients based on IP, user-agent, or token. It protects origin servers from aggressive AI scrapers by returning HTTP 429 Too Many Requests when thresholds are exceeded.

Technical Definition

Rate Limiting is a defensive capacity-management strategy that caps the volume of incoming requests an API or web server processes from a particular identity (IP address, session cookie, or API key) over a specific time window (e.g., 60 requests per minute).

When an automated crawler exceeds the allowable ceiling, the server short-circuits the connection with an HTTP 429 (Too Many Requests) status.

Nginx Implementation Example

# Define rate limit zone by client IP (10 requests per second)
limit_req_zone $binary_remote_addr zone=crawler_limit:10m rate=10r/s;

server {
    location / {
        limit_req zone=crawler_limit burst=20 nodelay;
        limit_req_status 429;
    }
}

Balancing Rate Limits with AI Crawlers

• Aggressive scrapers that ignore advisory robots.txt rules must be halted with HTTP 429.
• Verified search engines (Googlebot, Bingbot) should have higher threshold allowances to prevent incomplete crawl passes and delayed index updates.

Optimize your edge rate-limiting rules without blocking beneficial AI search bots. Audit your architecture with Geolify.ai.