Glossary Term
User-Agent Header
An HTTP request header that identifies the client application, operating system, vendor, and version submitting the network request to a web server.
AI Summary: The User-Agent is an HTTP request header sent by clients to identify their software, version, and operator. Webmasters inspect this string to match crawlers against robots.txt directives and apply custom routing, caching, or rate limits.
Technical Definition
The User-Agent header is a standard HTTP request header (defined in RFC 9110) transmitted by a client to identify its software application, hardware architecture, operating system, and build version to the receiving web server:
configuration / code
User-Agent: Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)
Anatomical Breakdown of an AI Bot User-Agent
- Mozilla Compatibility Token:
Mozilla/5.0 (compatible; ...)(historical legacy token). - Bot Identifier Token:
OAI-SearchBot/1.0(the primary string parsed byrobots.txtparsers). - Documentation URL:
+https://openai.com/searchbot(canonical reference explaining the bot's identity, IP ranges, and purpose).
Inspecting User-Agents in Server Logs
configuration / code
# Filter access logs for AI crawler activity
tail -n 1000 /var/log/nginx/access.log | grep -Ei "gptbot|claudebot|oai-searchbot|perplexitybot"
Ensure your web server parses and categorizes modern AI user-agents accurately. Audit with Geolify.ai.
Related terms
AI Search CrawlerA specialized web crawler operated by an AI search engine to retrieve real-time web content for live retrieval-augmented generation (RAG) and citations.AI Training CrawlerA web crawler deployed by AI laboratories to extract large text and multimodal datasets from the open web for pre-training and fine-tuning foundational models.Assistant CrawlerAn on-demand fetcher triggered directly by an end-user interacting with an AI chat interface or custom agent to browse a specific URL in real time.