AI crawlers, search bots and training agents
A reference of the bots that visit websites for AI assistants, search engines and model training: what each one does, how it identifies itself, whether it follows robots.txt and how to allow or block it.
AI answers
Fetch a page when someone asks an assistant about it.
- Amzn-User Amazon Fetches live information from the web on a user's behalf, for example to answer Alexa questions that need up-to-date facts. Amazon says it does not collect content for generative AI training.
- Claude-User Anthropic Visits a page when someone asks Claude a question that needs it. Blocking it stops Claude from reading your pages for user questions, which can reduce your visibility in answers.
- DuckAssistBot DuckDuckGo Fetches pages in real time for the AI-assisted answers in DuckDuckGo Search, which cite their sources. DuckDuckGo says the data is not used to train AI models, and blocking it does not affect your search ranking.
- Gemini Notebook Google Fetches the individual URLs that Gemini Notebook users add as sources to their projects. It replaces the former Google-NotebookLM agent, supported until August 2026.
- Google-Agent Google Used by AI agents hosted on Google infrastructure to navigate websites and carry out actions at a user's request.
- Meta-ExternalFetcher Meta Fetches individual links at a user's request and supports Meta's agentic AI features, such as helping AI navigate websites to complete tasks.
- MistralAI-User Mistral AI Visits a page when someone asks Mistral's assistant Vibe a question, and may link to it as a source. Mistral says it does not crawl automatically and is not used to collect training data.
- Kimi-User Moonshot AI Handles user-initiated actions in Kimi, such as summarising an article or answering a question that needs live web content. Blocking it may reduce your visibility in searches users make through Kimi.
- ChatGPT-User OpenAI Visits a page when a ChatGPT or custom GPT user asks something that needs it, or through GPT Actions. OpenAI says it does not crawl automatically and does not decide whether content appears in ChatGPT search.
- Perplexity-User Perplexity Visits a page when someone asks Perplexity a question, and may link to it in the answer. Perplexity says it is not used for crawling or to collect training data.
- SBIntuitions-SearchBot SB Intuitions Visits websites on a user's behalf when they ask a question in Sarashina, the assistant from SoftBank's SB Intuitions, and may link to them in the answer. The company says this content is not used for AI development.
Search indexes
Crawl the web to build the index of a search engine or AI search.
- Amzn-SearchBot Amazon Crawls pages for search experiences in Amazon products such as Alexa; blocking it makes your content ineligible for them. Amazon says it does not crawl for generative AI training.
- AndiBot Andi Search crawler and proxy of Andi, an AI search engine. It crawls the web to build Andi's search results and fetches pages anonymously for its reader and preview modes.
- Claude-SearchBot Anthropic Crawls the web to improve the relevance and accuracy of Claude's search results. Blocking it keeps your content out of that index, which may reduce your visibility in search answers.
- Applebot Apple Powers search in Spotlight, Siri and Safari, and its data may also be used to train Apple's foundation models. Blocking it removes your site from those results; to opt out of training only, block Applebot-Extended.
- Baiduspider Baidu Baidu's search crawler, which builds the Baidu index. Blocking it removes your pages from Baidu search and the other search results Baidu provides.
- Diffbot Diffbot Crawls the web to build the Diffbot Knowledge Graph and its web search services, where sites can be found and cited. Diffbot says it does not crawl pages to train generative AI foundation models.
- DuckDuckBot DuckDuckGo DuckDuckGo's web crawler, used to improve its search results. AI-assisted answers use a separate bot, DuckAssistBot.
- ExaSearchBot Exa Crawler of the Exa search engine, which indexes public pages so people and applications can find, retrieve and cite them. Pages Exa fetches on a user's request go out under a different name, ExaUser.
- Googlebot Google Builds the Google Search index. Rules for Googlebot also apply to Discover, Google Images, Video, Google News and other products, so blocking it removes your pages from Google Search.
- PetalBot Huawei Crawler of Huawei's Petal search engine, which also powers content recommendations in Huawei Assistant and AI Search. Blocking it removes your pages from Petal search.
- LinkupBot Linkup Builds the Linkup web search index, which provides search and web grounding for AI applications.
- Meta-WebIndexer Meta Crawls the web to improve the quality of Meta AI search results. Meta says allowing it helps Meta AI cite and link to your content in its answers.
- Bingbot Microsoft Bing's standard crawler, which handles most of its crawling and builds the Bing search index. Blocking it removes your pages from Bing.
- MistralAI-Index Mistral AI Crawls the web to index content for Mistral search, which helps answer questions in its assistant Vibe. Mistral says the content is not used for generative AI training.
- Kimi-SearchBot Moonshot AI Powers the search features of Kimi, the assistant from Moonshot AI, by analysing pages and building its search index. Blocking it keeps your site out of Kimi search results.
- OAI-SearchBot OpenAI Surfaces websites in ChatGPT's search features. Sites that block it are not shown in ChatGPT search answers, though they can still appear as navigational links.
- PerplexityBot Perplexity Surfaces and links websites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models.
- YandexBot Yandex Yandex's main indexing robot, which builds the Yandex search index. Blocking it removes your pages from Yandex search.
- YouBot You.com Crawler that powers the You.com search engine, discovering and indexing pages for its search results.
Training crawlers
Collect public pages that may be used to train AI models.
- AI2Bot Ai2 Crawler of Ai2 (Allen Institute for AI), a non-profit research institute. The content it collects is used to train open language models.
- Amazonbot Amazon Used to improve Amazon's products and services, and its data may be used to train Amazon AI models. Search in Alexa uses a separate bot, Amzn-SearchBot.
- ClaudeBot Anthropic Collects public web content that may be used to train Anthropic's Claude models. Blocking it signals that your future content should be left out of training data; Claude search and user fetches use other bots.
- Applebot-Extended Apple A robots.txt token, not a crawler: it decides whether data crawled by Applebot may be used to train Apple's foundation models. Blocking it does not remove your site from Apple search results.
- Bytespider ByteDance Crawler of ByteDance, the owner of TikTok. Bot directories report that it downloads training data for ByteDance's language models, including those behind its Doubao assistant.
- cohere-training-data-crawler Cohere Crawler that bot directories attribute to Cohere, used to download training data for the language models behind its enterprise AI products.
- CCBot Common Crawl Crawler of Common Crawl, a non-profit that maintains an open, freely downloadable archive of the web. The archive is widely used as a source of training data for language models, so blocking CCBot keeps future pages out of it.
- Google-Extended Google A robots.txt token, not a separate crawler: it controls whether content Google crawls may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. It does not affect inclusion or ranking in Google Search.
- PanguBot Huawei Crawler that bot directories attribute to Huawei, used to download training data for its PanGu multimodal language model.
- Meta-ExternalAgent Meta Crawls the web for uses such as training Meta's foundation AI models or improving products by indexing content directly. Blocking it keeps your content out of that crawl.
- MistralAI-Training Mistral AI Crawls web content to build datasets for training Mistral's generative AI models. Mistral says it is not used for search indexing or live answers in Vibe.
- KimiBot Moonshot AI Crawls content that may be used to train the foundation models behind Kimi, the assistant from Moonshot AI. Blocking it signals that your content should not be used for training.
- ICC-Crawler NICT Crawler of Japan's National Institute of Information and Communications Technology (NICT). The pages it collects are used for research on translation, information analysis and AI, and may be shared with third parties for joint research.
- GPTBot OpenAI Collects public web pages that may be used to train OpenAI's generative AI models. Blocking it keeps your content out of future training data; it does not affect ChatGPT search.
- SBIntuitionsBot SB Intuitions Crawler of SB Intuitions, a SoftBank company building Japanese language models, used for AI development and information analysis.
- Timpibot Timpi Crawler run by the independent node operators of Timpi's decentralised network. Bot directories note that the index it builds can be used to train language models.
- Webzio-Extended Webz.io Marks the data Webz.io collects as usable or not usable for AI and machine-learning training. Webz.io sells web data to other companies; blocking this agent signals that your content should not be offered for AI training.
Other AI bots
Other automated agents related to AI services.
- TerraCotta Ceramic AI Web crawler of Ceramic AI (ceramic.ai). The company says it identifies itself as TerraCotta and follows robots.txt.
- FirecrawlAgent Firecrawl Firecrawl is a scraping and crawling API that developers use to turn web pages into data for AI applications. It reads robots.txt before crawling a site and applies the rules for FirecrawlAgent and *.
- Google-CloudVertexBot Google Crawls sites at their owners' request to build Vertex AI Agents on Google Cloud. It has no effect on Google Search or other products.
- GoogleOther Google Generic Google crawler that product teams use to fetch public content, for example for one-off crawls for internal research and development. Blocking it does not affect any specific Google product.
- ImagesiftBot Hive Crawler of ImageSift by Hive, which collects publicly available images with their page text and alt text to build an index for reverse image search and other web intelligence products.
- Cotoyogi ROIS-DS Crawler of the Center for Research and Development on Data Lake at ROIS-DS, a Japanese research organisation, which collects Japanese-language data resources.
- omgili Webz.io Older crawler of Webz.io, built for the discontinued Omgili search engine. It indexes news sites, blogs, reviews and forums for the Webz.io data API, which other companies buy.
- YandexAdditional Yandex Lets you keep pages already indexed by YandexBot out of the AI responses in Yandex search. It makes no indexing requests of its own.
See which AI crawlers read your site
Statsdot keeps bots out of your visitors and, with AI crawler tracking, shows which crawlers read your pages and which pages they request. No cookies, no personal data.