curl

Scraper / HTTP Client curl project Ignores robots.txt

Command-line HTTP client commonly used by scripts and automation.

User-Agent

curl/8.7.1

Allow curl in robots.txt

User-agent: curl
Allow: /

Block curl in robots.txt

User-agent: curl
Disallow: /

Blocking affects this tool's ability to access your site for its stated purpose.

Heads up: curl is reported to ignore robots.txt. To block it reliably you need server-level enforcement (WAF rule, User-Agent filter, or bot management) — robots.txt alone is only a request.

Official documentation: https://curl.se/

Check your site's bot access

Free dual-engine check: robots.txt verdicts for all 222 bots plus live User-Agent probes. Finds WAF blocks your robots.txt doesn't show.