Assistant Crawler
An on-demand fetcher triggered directly by an end-user interacting with an AI chat interface or custom agent to browse a specific URL in real time.
AI Summary: An assistant crawler (such as ChatGPT-User) is an on-demand HTTP fetcher dispatched when a human user asks an AI assistant to read or summarize a specific URL. Because it acts on immediate user request, blocking it prevents end-users from interacting with your content inside AI tools.
Technical Definition
An Assistant Crawler (also referred to as an on-demand fetcher or user-triggered agent) is an HTTP client dispatched directly when a person submits a prompt such as "Summarize the article at https://example.com/spec" into a conversational interface like ChatGPT, Claude, or Microsoft Copilot.
The defining client in this class is ChatGPT-User. Unlike autonomous background crawlers, assistant fetchers make point-to-point requests initiated by explicit human action.
Distinguishing Characteristics
- Trigger: Synchronous user request in an active conversational session.
- Traffic Pattern: Low volume, targeted URL fetches with negligible server load.
- Impact of Blocking: If blocked, users receive an error stating "I am unable to browse that link due to site restrictions", which degrades user experience.
Policy Configuration
# Allow user-requested browsing while declining autonomous bulk scraping
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /
Ensure your content remains accessible to legitimate user assistants while blocking unauthorized harvesting. Audit your crawler configuration with Geolify.ai.