Directory · 222 bots
The AI bot & crawler directory
What each bot does, who runs it, whether it honours robots.txt — and copy-paste snippets to allow or block it. Run a check to see which ones can reach your site.
Showing all 222 bots
No bots match that search
Try a company name, robots.txt token, or a broader category.
AI Training Crawlers · 71
| Bot | Company | Respects robots.txt | Purpose |
|---|---|---|---|
| Bytespider | ByteDance | Unknown | ByteDance Bytespider web crawler bot |
| CCBot | Common Crawl | Unknown | Common Crawl web crawler for indexing |
| claudebot | Anthropic | Unknown | Anthropic Claude web crawler bot |
| Cloudflare Crawler | Cloudflare | Unknown | Cloudflare crawler or user-triggered agent. |
| FacebookBot | Meta | Unknown | Meta's AI speech recognition training crawler |
| Google-CloudVertexBot | Unknown | Google Vertex AI crawler indexing content for enterprise search solutions | |
| Google-Extended | Unknown | Google extended web crawler bot | |
| GPTBot | OpenAI | Unknown | OpenAI GPT web crawler bot |
| meta-externalagent | Meta | Unknown | Meta external agent web crawler bot |
| Novellum AI Crawl | Novellum | Unknown | Novellum crawler or user-triggered agent. |
| ProRataInc | ProRata.ai | Unknown | ProRata.ai crawler or user-triggered agent. |
| AI2Bot | Unknown / community maintained | Unknown | Academic research crawler for NLP dataset creation |
| AISearchBot | Unknown / community maintained | Unknown | AI-powered search engine web crawler bot |
| Amzn-SearchBot | Unknown / community maintained | Unknown | Amazon's AI search indexer for Alexa |
| Amzn-User | Unknown / community maintained | Unknown | Amazon AI assistant for Alexa queries |
| Anomura | Unknown / community maintained | Unknown | Direqt's AI search indexer |
| anthropic-ai | Unknown / community maintained | Unknown | Anthropic AI web crawler bot |
| ApifyBot | Unknown / community maintained | Unknown | Apify's web scraping and data extraction |
| ApifyWebsiteContentCrawler | Unknown / community maintained | Unknown | Apify's full website content extractor |
| Aranet-SearchBot | Unknown / community maintained | Unknown | AI-powered search engine and content indexing crawler bot |
| atlassian-bot | Unknown / community maintained | Unknown | Atlassian Rovo's AI search crawler |
| AzureAI-SearchBot | Unknown / community maintained | Unknown | Microsoft's Azure AI search indexer |
| bigsur.ai | Unknown / community maintained | Unknown | Big Sur AI's web agent crawler |
| Brightbot | Unknown / community maintained | Unknown | Bright Data's AI-ready data collector |
| Channel3Bot | Unknown / community maintained | Unknown | Channel3's universal product catalog indexer |
| ChatGLM-Spider | Unknown / community maintained | Unknown | AI model training data collection crawler |
| Claude-Web | Unknown / community maintained | Unknown | Anthropic Claude web crawler bot |
| Cloudflare-AutoRAG | Unknown / community maintained | Unknown | Cloudflare's RAG service indexer |
| cohere-ai | Unknown / community maintained | Unknown | Cohere AI web crawler bot |
| cohere-training-data-crawler | Unknown / community maintained | Unknown | LLM training data collection for AI models |
| crawl4ai | Unknown / community maintained | Unknown | Open-source LLM-friendly web scraping and extraction library |
| DeepSeekBot | Unknown / community maintained | Unknown | DeepSeek AI model training and data collection crawler |
| Devin | Unknown / community maintained | Unknown | AI-driven browser automation and content extraction |
| ExteContextCrawl | Unknown / community maintained | Unknown | AI context extraction and content understanding web crawler |
| FirecrawlAgent | Unknown / community maintained | Unknown | Firecrawl's LLM data extraction crawler |
| Flyriverbot | Unknown / community maintained | Unknown | AI content source verification and attribution checking crawler |
| Gemini-Deep-Research | Unknown / community maintained | Unknown | Google Gemini deep research web crawler |
| Google-NotebookLM | Unknown / community maintained | Unknown | Google's AI research and note-taking assistant |
| HenkBot | Unknown / community maintained | Unknown | Valyu AI content analysis and data collection crawler |
| iAskBot | Unknown / community maintained | Unknown | AI search engine crawler indexing web content for question answering |
| iaskspider | Unknown / community maintained | Unknown | Chinese AI search engine and question answering crawler |
| ImageMind | Unknown / community maintained | Unknown | Image analysis crawler collecting visual data for AI training purposes |
| imageSpider | Unknown / community maintained | Unknown | ByteDance's image collection crawler |
| img2dataset | Unknown / community maintained | Unknown | Open source tool downloading images for machine learning dataset creation |
| kagi-fetcher | Unknown / community maintained | Unknown | Kagi AI assistant for web content |
| Kangaroo Bot | Unknown / community maintained | Unknown | Australian AI model training crawler |
| KendraBot | Unknown / community maintained | Unknown | AI-powered enterprise search crawler for documents |
| KunatoCrawler | Unknown / community maintained | Unknown | AI training data collection and web scraping crawler |
| laion-huggingface-processor | Unknown / community maintained | Unknown | LAION's image dataset builder for AI |
| LinerBot | Unknown / community maintained | Unknown | AI workspace crawler indexing web content for search and discovery |
| linkReader | Unknown / community maintained | Unknown | AI assistant crawler for content indexing |
| LinkupBot | Unknown / community maintained | Unknown | Linkup's enterprise AI search crawler |
| newsai | Unknown / community maintained | Unknown | NewsAI crawler for news content aggregation and indexing |
| Novellum | Unknown / community maintained | Unknown | AI data collection crawler for model training and enrichment |
| PerplexityUser | Unknown / community maintained | Unknown | Perplexity AI web crawler bot |
| PhindBot | Unknown / community maintained | Unknown | AI-powered developer answer engine crawler |
| Poggio-Citations | Unknown / community maintained | Unknown | Poggio's AI sales enablement data collector |
| SBIntuitionsBot | Unknown / community maintained | Unknown | SB Intuitions web crawler bot |
| semantic-visions | Unknown / community maintained | Unknown | Semantic content analysis and AI training data crawler |
| ShapBot | Unknown / community maintained | Unknown | Parallel's AI agent web context provider |
| Spawning-AI | Unknown / community maintained | Unknown | Web crawler indexing content for AI models |
| TaraGroup Intelligent Bot | Unknown / community maintained | Unknown | TaraGroup AI-powered web intelligence and analysis crawler |
| TavilyBot | Unknown / community maintained | Unknown | Tavily's real-time AI agent data crawler |
| TerraCotta | Unknown / community maintained | Unknown | Ceramic Network decentralized data indexing and crawling bot |
| The Knowledge AI | Unknown / community maintained | Unknown | AI data collection crawler indexing content and enhancing machine learning models |
| Thinkbot | Unknown / community maintained | Unknown | Experimental AI thinking and reasoning web crawler bot |
| TikTokSpider | Unknown / community maintained | Unknown | TikTok web crawler for content discovery |
| TSM-turingos | Unknown / community maintained | Unknown | TuringOS AI-powered content analysis and monitoring crawler |
| www.spider | Unknown / community maintained | Unknown | AI project web data crawler |
| ZanistaBot | Unknown / community maintained | Unknown | Zanista's AI search crawler |
| Applebot-Extended | Apple | Unknown | Apple robots.txt control token for generative AI model training. |
AI Search / Retrieval · 4
| Bot | Company | Respects robots.txt | Purpose |
|---|---|---|---|
| Claude-SearchBot | Anthropic | Unknown | Anthropic Claude search web crawler bot |
| OAI-SearchBot | OpenAI | Unknown | OpenAI search web crawler bot |
| PerplexityBot | Perplexity | Unknown | Perplexity AI web crawler for search |
| Terracotta Bot | Ceramic | Unknown | Ceramic crawler or user-triggered agent. |
AI Assistants / Agents · 9
| Bot | Company | Respects robots.txt | Purpose |
|---|---|---|---|
| Anchor Browser | Anchor | Unknown | Anchor crawler or user-triggered agent. |
| ChatGPT-User | OpenAI | Unknown | ChatGPT user web crawler bot |
| Claude-User | Anthropic | Unknown | Anthropic Claude user web crawler bot |
| DuckAssistBot | DuckDuckGo | Unknown | DuckDuckGo assistant bot for web crawling and search |
| Manus Bot | Manus | Unknown | Manus crawler or user-triggered agent. |
| Manus-User | Manus | Unknown | Manus crawler or user-triggered agent. |
| Meta-ExternalFetcher | Meta | Unknown | Meta crawler or user-triggered agent. |
| MistralAI-User | Mistral AI | Unknown | Mistral AI web crawler bot for content indexing |
| Perplexity-User | Perplexity | Unknown | Perplexity user web crawler bot |
Scrapers / HTTP Clients · 138
| Bot | Company | Respects robots.txt | Purpose |
|---|---|---|---|
| Amazonbot | Amazon | Unknown | Amazon web crawler for product discovery |
| Applebot | Apple | Unknown | Apple's web crawler for Siri and search |
| archive.org_bot | Internet Archive | Unknown | Internet Archive crawler or user-triggered agent. |
| bingbot | Microsoft | Unknown | Microsoft's web crawling bot for Bing search indexing |
| Googlebot | Unknown | Google's main web crawling bot for search indexing | |
| PetalBot | Huawei | Unknown | Petal search engine web crawler bot |
| Timpibot | Timpi | Unknown | Timpi web crawler for content discovery |
| 360Spider | Unknown / community maintained | Unknown | 360 search engine web crawler bot |
| 7Siters | Unknown / community maintained | Unknown | Russian web crawler for site indexing and analysis |
| A6-Indexer | Unknown / community maintained | Unknown | A6 Corporation web crawler for indexing |
| Aboundex | Unknown / community maintained | Unknown | Aboundex web crawler for content discovery |
| acoonbot | Unknown / community maintained | Unknown | Acoon web crawler for content discovery |
| AddSearchBot | Unknown / community maintained | Unknown | AddSearch web crawler for site search |
| Adidxbot | Unknown / community maintained | Unknown | Bing's advertising index web crawler bot |
| aiHitBot | Unknown / community maintained | Unknown | AiHit web crawler for data collection |
| AlexandriaOrgBot | Unknown / community maintained | Unknown | Alexandria web crawler for discovery |
| AmazonProductDiscovery | Unknown / community maintained | Unknown | Amazon product discovery web crawler |
| AmazonSellerInitiatedListing | Unknown / community maintained | Unknown | Amazon seller listing web crawler |
| APIs-Google | Unknown / community maintained | Unknown | Google's APIs bot for crawling API documentation and services |
| AppEngine-Google | Unknown / community maintained | Unknown | Google App Engine web crawler bot |
| arabot | Unknown / community maintained | Unknown | Arabot web crawler for content discovery |
| AspiegelBot | Unknown / community maintained | Unknown | Aspiegel web crawler for content discovery |
| Baiduspider | Unknown / community maintained | Unknown | Baidu search engine web crawler bot |
| BIGLOTRON | Unknown / community maintained | Unknown | Biglotron search engine web crawler bot |
| Bot.AraTurka.com | Unknown / community maintained | Unknown | AraTurka web crawler for content discovery |
| Cliqzbot | Unknown / community maintained | Unknown | Cliqz search engine web crawler bot |
| coccoc | Unknown / community maintained | Unknown | Coccoc search engine web crawler bot |
| convera | Unknown / community maintained | Unknown | Convera search engine web crawler bot |
| CyberFindCrawler | Unknown / community maintained | Unknown | Web search engine crawler for content discovery indexing |
| Daum | Unknown / community maintained | Unknown | Daum search engine web crawler bot |
| DeuSu | Unknown / community maintained | Unknown | DeuSu web crawler for search indexing |
| discobot | Unknown / community maintained | Unknown | Discovery Engine web crawler for content discovery |
| DuckDuckBot | Unknown / community maintained | Unknown | DuckDuckGo search engine web crawler bot |
| DuckDuckGo-Favicons-Bot | Unknown / community maintained | Unknown | DuckDuckGo favicon web crawler bot |
| edisterbot | Unknown / community maintained | Unknown | Edister web crawler for content discovery |
| exabot | Unknown / community maintained | Unknown | Exabot search engine web crawler for indexing |
| FAST Enterprise Crawler | Unknown / community maintained | Unknown | FAST enterprise-grade web crawler for search |
| FAST-WebCrawler | Unknown / community maintained | Unknown | FAST search engine web crawler for indexing |
| FediIndex | Unknown / community maintained | Unknown | Fediverse server indexing and discovery service web crawler |
| FemtosearchBot | Unknown / community maintained | Unknown | Femtosearch web crawler for content discovery |
| FindFiles.net | Unknown / community maintained | Unknown | FindFiles.net search engine crawler |
| findthatfile | Unknown / community maintained | Unknown | FindThatFile web crawler for file discovery |
| findxbot | Unknown / community maintained | Unknown | FindX web crawler for content discovery |
| fiperbot | Unknown / community maintained | Unknown | Web content indexing and search engine crawler bot |
| Flamingo_SearchEngine | Unknown / community maintained | Unknown | Flamingo search engine web crawler bot |
| FleebsBot | Unknown / community maintained | Unknown | Web content aggregation and discovery service crawler bot |
| fluffy | Unknown / community maintained | Unknown | Fluffy search engine web crawler bot |
| fr-crawler | Unknown / community maintained | Unknown | FR Crawler web crawler for content discovery |
| FyndSearchEngine-Crawler | Unknown / community maintained | Unknown | Fynd search engine indexing crawler |
| FyndSearchEngine-ReCrawler | Unknown / community maintained | Unknown | Fynd search engine re-indexing crawler |
| GeedoBot | Unknown / community maintained | Unknown | Geedo web crawler for content discovery |
| GeedoProductSearch | Unknown / community maintained | Unknown | Geedo product search web crawler bot |
| Gigablast | Unknown / community maintained | Unknown | Gigablast open-source search engine crawler |
| Gigabot | Unknown / community maintained | Unknown | Gigablast search engine web crawler bot |
| Gluten Free Crawler | Unknown / community maintained | Unknown | Gluten Free Pleasure web crawler bot |
| Google Favicon | Unknown / community maintained | Unknown | Google favicon web crawler bot |
| Google-Certificates-Bridge | Unknown / community maintained | Unknown | Google certificates bridge web crawler |
| Google-InspectionTool | Unknown / community maintained | Unknown | Google's inspection tool bot for testing and debugging search indexing |
| Google-PhysicalWeb | Unknown / community maintained | Unknown | Google physical web web crawler bot |
| Google-Read-Aloud | Unknown / community maintained | Unknown | Google read aloud web crawler bot |
| Google-Safety | Unknown / community maintained | Unknown | Google safety web crawler bot |
| Google-Site-Verification | Unknown / community maintained | Unknown | Google site verification web crawler bot |
| Google-Structured-Data-Testing-Tool | Unknown / community maintained | Unknown | Google structured data testing web crawler |
| google-xrawler | Unknown / community maintained | Unknown | Google xrawler web crawler bot |
| Googlebot-Image | Unknown / community maintained | Unknown | Google's image-specific web crawling bot for image search indexing |
| Googlebot-Mobile | Unknown / community maintained | Unknown | Google's legacy mobile crawler for Google Search indexing |
| Googlebot-News | Unknown / community maintained | Unknown | Google's news-specific web crawling bot for Google News indexing |
| Googlebot-Video | Unknown / community maintained | Unknown | Google's video crawler for video-related Google Search features |
| GoogleOther | Unknown / community maintained | Unknown | Google's other bots and services for various Google search features |
| Greppr Web Crawler | Unknown / community maintained | Unknown | Greppr unfiltered search engine crawler |
| grub.org | Unknown / community maintained | Unknown | Grub search engine web crawler for indexing |
| IbouBot | Unknown / community maintained | Unknown | Ibou web graph search crawler |
| ichiro | Unknown / community maintained | Unknown | Goo ichiro search engine web crawler bot |
| ImagesiftBot | Unknown / community maintained | Unknown | Imagesift bot for image search and indexing |
| IndeedBot | Unknown / community maintained | Unknown | Indeed job search web crawler bot |
| INETDEX-BOT | Unknown / community maintained | Unknown | INETDEX web crawler for indexing |
| IstellaBot | Unknown / community maintained | Unknown | Istella web crawler for search indexing |
| JobboerseBot | Unknown / community maintained | Unknown | Jobboerse web crawler for job discovery |
| Jooblebot | Unknown / community maintained | Unknown | Jooble job search web crawler bot |
| jyxobot | Unknown / community maintained | Unknown | Jyxo search engine bot for web crawling |
| lb-spider | Unknown / community maintained | Unknown | LB spider web crawler for content discovery |
| Linguee Bot | Unknown / community maintained | Unknown | Linguee translation web crawler bot |
| lyonl-asset-proxy | Unknown / community maintained | Unknown | Lyonl search engine asset proxy |
| lyonl-crawler | Unknown / community maintained | Unknown | Lyonl open web search crawler |
| MagiBot | Unknown / community maintained | Unknown | Peak Labs information extraction crawler |
| Mail.RU_Bot | Unknown / community maintained | Unknown | Mail.RU search engine web crawler bot |
| MatchorySearch | Unknown / community maintained | Unknown | Search engine and content discovery indexing crawler bot |
| MojeekBot | Unknown / community maintained | Unknown | Mojeek search engine web crawler bot |
| msnbot | Unknown / community maintained | Unknown | Microsoft's search engine bot for web indexing |
| msrbot | Unknown / community maintained | Unknown | Microsoft Research web crawler bot |
| NapBot | Unknown / community maintained | Unknown | Web content indexing and search engine crawler bot |
| Neevabot | Unknown / community maintained | Unknown | Neeva search engine web crawler bot |
| NestDaddybot | Unknown / community maintained | Unknown | NestDaddy search engine crawler |
| Nicecrawler | Unknown / community maintained | Unknown | Nicecrawler web crawler for discovery |
| NTENTbot | Unknown / community maintained | Unknown | NTENT web crawler for search results |
| Nutch | Unknown / community maintained | Unknown | Apache Nutch open-source web crawler framework |
| officestorebot | Unknown / community maintained | Unknown | Microsoft Office Store web crawler bot |
| OICrawler | Unknown / community maintained | Unknown | OpenIndex web crawler for indexing |
| OpenHoseBot | Unknown / community maintained | Unknown | OpenHose web crawler for content analysis |
| openindexspider | Unknown / community maintained | Unknown | OpenIndex web crawler for indexing |
| OrangeBot | Unknown / community maintained | Unknown | Orange search engine web crawler bot |
| PhxBot | Unknown / community maintained | Unknown | Phoenix web crawler for content discovery |
| psbot | Unknown / community maintained | Unknown | PicSearch web crawler for image discovery |
| purebot | Unknown / community maintained | Unknown | Pure web crawler for content discovery |
| Qwantbot | Unknown / community maintained | Unknown | Qwant privacy-focused search crawler |
| Qwantify | Unknown / community maintained | Unknown | Qwant search engine web crawler bot |
| SafeSearch microdata crawler | Unknown / community maintained | Unknown | Avira SafeSearch web crawler for safety |
| scribdbot | Unknown / community maintained | Unknown | Scribd document web crawler bot |
| Scrubby | Unknown / community maintained | Unknown | Scrub the Web business directory crawler |
| search.marginalia.nu | Unknown / community maintained | Unknown | Marginalia search web crawler bot |
| Seekbot | Unknown / community maintained | Unknown | Seekbot search engine web crawler for indexing |
| Seekport | Unknown / community maintained | Unknown | Seekport web crawler for content discovery |
| SeznamBot | Unknown / community maintained | Unknown | Seznam search engine web crawler bot |
| Slurp | Unknown / community maintained | Unknown | Yahoo's web crawling bot for Yahoo search indexing |
| SMTnetPMBot | Unknown / community maintained | Unknown | SMTnet electronics manufacturing search crawler |
| Sogou | Unknown / community maintained | Unknown | Sogou search engine web crawler bot |
| Sosospider | Unknown / community maintained | Unknown | Chinese search engine web indexing and crawling bot |
| speedy | Unknown / community maintained | Unknown | Entireweb Speedy Spider web crawler bot |
| stepstoneCrawlBot | Unknown / community maintained | Unknown | Stepstone Group job search crawler |
| Storebot-Google | Unknown / community maintained | Unknown | Google's Storebot for crawling product and e-commerce pages |
| StractBot | Unknown / community maintained | Unknown | Stract search engine web crawler bot |
| tagoobot | Unknown / community maintained | Unknown | Tagoo search engine web crawler bot |
| Teoma | Unknown / community maintained | Unknown | Ask Jeeves Teoma search engine web crawler |
| ToutiaoSpider | Unknown / community maintained | Unknown | Toutiao news platform web crawler bot |
| trovitBot | Unknown / community maintained | Unknown | Trovit web crawler for content discovery |
| UsineNouvelleCrawler | Unknown / community maintained | Unknown | Usine Nouvelle news site web crawler |
| voilabot | Unknown / community maintained | Unknown | Voila search engine web crawler bot |
| WadooBot | Unknown / community maintained | Unknown | Web content indexing and search engine crawler bot |
| wbsearchbot | Unknown / community maintained | Unknown | Warebay search web crawler bot |
| WepchSearchEngine | Unknown / community maintained | Unknown | Wepch search engine development crawler |
| WeSEE:Search | Unknown / community maintained | Unknown | WeSEE search engine web crawler bot |
| WikiDo | Unknown / community maintained | Unknown | Wiki content aggregation and search engine crawler bot |
| wocbot | Unknown / community maintained | Unknown | Wocodi web crawler for content discovery |
| WordupInfoSearch | Unknown / community maintained | Unknown | Wordup info search web crawler bot |
| woriobot | Unknown / community maintained | Unknown | Worio search engine web crawler bot |
| wotbox | Unknown / community maintained | Unknown | Wotbox web crawler for content discovery |
| Y!J | Unknown / community maintained | Unknown | Yahoo Japan search engine web crawler bot |
| curl | curl project | No | Command-line HTTP client commonly used by scripts and automation. |