Every AI crawler, what it does, and how to control it

Crawls content that may be used to train OpenAI's generative AI foundation models.

“Disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models.”

UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Surfaces websites in ChatGPT search results. Blocking it removes you from ChatGPT search answers.

“Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links.”

UA: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

OpenAI says robots.txt changes can take about 24 hours to apply to search.

Visits a page when a ChatGPT or Custom GPT user's request needs it. Not an automatic crawler.

“Because these actions are initiated by a user, robots.txt rules may not apply.”

UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

OpenAI: ChatGPT-User is not used to decide whether content appears in Search; use OAI-SearchBot for that.

Checks landing pages submitted as ads on ChatGPT. Only visits submitted ad pages.

“OAI-AdsBot only visits pages submitted as ads, and the data collected by OAI-AdsBot is not used to train generative AI foundation models.”

UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-AdsBot/1.0; +https://openai.com/adsbot

Collects public web content that could contribute to training Anthropic's models.

“When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets.”

Anthropic publishes the token, not a full UA string. It also supports the non-standard Crawl-delay directive.

Indexes content to improve the quality of Claude's search results.

“Disabling Claude-SearchBot on your site prevents our system from indexing your content for search optimization, which may reduce your site's visibility and accuracy in user search results.”

Fetches a page when a Claude user asks a question that needs it.

“Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search.”

Unlike most user-triggered fetchers, Anthropic says its bots, including Claude-User, honor robots.txt.

Surfaces and links websites in Perplexity search results. Not used for foundation-model training.

“PerplexityBot is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models.”

UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Visits a page to answer a Perplexity user's question, and may link it in the answer.

“Since a user requested the fetch, this fetcher generally ignores robots.txt rules.”

UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

Google Search's crawler. Google says Googlebot's robots.txt rules are the control for AI Overviews and AI Mode, because AI is built into Search.

“Crawling preferences addressed to the Googlebot user agent affect Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Video mode, Google News, and Discover.”

UA: Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google: "robots.txt directives for Googlebot is the control" for AI features in Search. To limit what is shown, it points to nosnippet, data-nosnippet, max-snippet or noindex. Blocking Googlebot removes you from Google Search entirely.

A robots.txt control token, not a crawler. Disallowing it opts content out of Gemini model training and grounding. It does not affect Google Search.

“Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.”

Google: "Google-Extended doesn't have a separate HTTP request user agent string." You will never see it in your logs. It covers training future Gemini models and grounding in Gemini Apps and Vertex AI.

Crawls only when a site owner asks Google to, to build Vertex AI Agents. No effect on Google Search.

“Crawling preferences addressed to the Google-CloudVertexBot user agent affect crawls requested by the site owners' for building Vertex AI Agents. It has no effect on Google Search or other products.”

Google lists the HTTP UA substring "Google-CloudVertexBot" and both Google-CloudVertexBot and Googlebot as robots.txt tokens.

Used by agents running on Google infrastructure to browse and act on the web when a user asks.

“Google-Agent is used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request.”

UA: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36

Google says user-triggered fetchers "generally ignore robots.txt rules", so robots.txt is not a reliable control here.

Fetches URLs that Gemini Notebook (formerly NotebookLM) users add as sources.

“The Gemini Notebook fetcher requests individual URLs that Gemini Notebook users have provided as sources for their projects.”

UA: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 (compatible; Google-GeminiNotebook; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-gemininotebook)

The former UA, Google-NotebookLM, is supported until August 2026.

Powers search in Spotlight, Siri and Safari. Its data may also train Apple models unless you disallow Applebot-Extended.

“Enabling Applebot in robots.txt allows website content to appear in search results for Apple users around the world in these products.”

UA: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

Apple: "If robots instructions don't mention Applebot but mention Googlebot, the Apple robot will follow Googlebot instructions." Applebot does not follow Crawl-delay.

A robots.txt control token. Disallowing it opts content out of training Apple's foundation models. Search is unaffected.

“Applebot-Extended does not crawl webpages. Webpages that disallow Applebot-Extended can still be included in search results.”

To keep content out of Siri and Search AI answers, Apple points to the nosnippet meta tag, not Applebot-Extended.

Improves Amazon products and services, and its data may be used to train Amazon AI models.

“Amazonbot is used to improve our products and services. This helps us provide more accurate information to customers and may be used to train Amazon AI models.”

UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36

Amazon also honors the noarchive robots meta tag as "do not use the page for model training".

Improves search in Amazon products such as Alexa. Not used for generative AI training.

“By permitting Amzn-SearchBot access to your website, your content is eligible to appear in search experiences such as Alexa.”

UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/W.X.Y.Z Safari/537.36

Amazon: if robots.txt doesn't mention Amzn-SearchBot but allows other search bots, it follows those rules. The checker evaluates its own group, then *.

Fetches live information on a user's behalf, for example to answer an Alexa question.

“Because actions taken by Amzn-User can be initiated by a user, it may not follow all robots.txt directives.”

UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-User/0.1) Chrome/W.X.Y.Z Safari/537.36

Crawls for uses such as training foundation AI models or improving products by indexing content directly.

“The Meta-ExternalAgent crawler crawls the web for use cases such as training foundation AI models or improving products by indexing content directly.”

UA: meta-externalagent/1.1

Meta's crawler page publishes no IP list.

Improves Meta AI search results. Allowing it helps Meta AI cite and link your content.

“Allowing Meta-WebIndexer in your robots.txt file helps us cite and link to your content in Meta AI's responses.”

UA: meta-webindexer/1.1

Fetches individual links at a user's request, including for agentic AI tasks.

“Accordingly, this crawler may bypass robots.txt rules.”

UA: meta-externalfetcher/1.1

Builds link previews (title, description, image) when a URL is shared on Facebook, Instagram or Messenger.

“Note that the FacebookExternalHit crawler might bypass robots.txt when performing security or integrity checks, such as checking for malware or malicious content.”

UA: facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

Crawls content to build datasets for training Mistral's generative models.

“MistralAI-Training crawls web content to help build datasets for training Mistral generative AI models. Webmasters can disallow this user agent in their robots.txt file.”

UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Training/1.0; +https://docs.mistral.ai/robots)

Mistral publishes no IP list for this crawler.

Indexes the web for Mistral search, which answers questions in Vibe. Not used for training.

“Content crawled by MistralAI-Index is not used for generative AI training of any kind.”

UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)

Mistral lists the token for robots.txt management but does not explicitly state compliance for this crawler.

Visits a page when a Vibe user asks a question, and may link it as a source.

“MistralAI-User governs which sites these user requests can be made to. It is not used for crawling the web in any automatic fashion, nor to crawl content for generative AI training.”

UA: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)

Crawls pages in real time for DuckDuckGo's AI-assisted answers, which cite sources. Not used for training.

“This data is not used in any way to train AI models.”

UA: DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)

DuckDuckGo: a disallow takes effect after 72 hours and does not affect organic rankings.

DuckDuckGo's regular search crawler.

“It respects WWW::RobotRules”

UA: DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)

Bing's crawler, which also feeds Copilot answers. Bing has no separate AI token. It uses the noarchive and nocache meta tags instead.

“Content tagged NOARCHIVE will not be included in Bing Chat answers, not be linked to in the answers.”

Bing's crawler list page renders client-side and could not be re-read on 2026-10-16, so no UA string is shown. The AI-answer controls come from Bing's official webmaster blog (2023, when Copilot was called Bing Chat): nocache = URL, title and snippet only; noarchive = excluded.

Builds Common Crawl's open web archive. The archive is a widely used source of AI training data.

“To prevent Common Crawl from crawling your website, include the following in your robots.txt”

UA: CCBot/2.0 (https://commoncrawl.org/faq/)

Common Crawl is a non-profit, not an AI lab. "Training" here reflects how its archive is commonly used.

ByteDance's crawler, widely reported as collecting AI training data. ByteDance publishes no bot documentation.

Unverified: no official ByteDance page describes this crawler. Purpose is based on third-party reports.

Token associated with Cohere. No reachable Cohere page documents it.

Unverified: purpose and robots behavior are not documented by Cohere.

Token associated with You.com search. No reachable You.com page documents it.

Unverified: purpose and robots behavior are not documented by You.com.