How do I block AI crawlers with robots.txt?

Add a User-agent line for each AI bot followed by "Disallow: /". For example: "User-agent: GPTBot" then "Disallow: /" blocks OpenAI's training crawler. This tool generates the full block for every major AI crawler — select the bots and copy the output into your robots.txt.

How do I block GPTBot specifically?

Add: User-agent: GPTBot Disallow: / to your robots.txt. Note that GPTBot is only OpenAI's training crawler — blocking it does not stop ChatGPT search (OAI-SearchBot) or user-triggered fetches (ChatGPT-User), which use separate user-agents you can block too.

Does blocking an AI crawler actually stop it?

robots.txt is advisory, not enforcement. Well-behaved crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot's indexer) honor it. Some, like ByteDance's Bytespider and Perplexity's user fetcher, are documented to ignore it — the reference table flags which respect robots.txt. For hard enforcement, block by user-agent or verified IP at your CDN or firewall.

Will blocking AI crawlers hurt my SEO?

Blocking AI-specific tokens like GPTBot, ClaudeBot, and Google-Extended does not affect traditional search ranking — Google-Extended is separate from Googlebot. But blocking a general search crawler such as Bingbot or Applebot also removes you from that search engine, so those are off by default here.

What's the difference between training, search, and user-fetch crawlers?

Training crawlers (GPTBot, ClaudeBot, CCBot) collect data to train models. Search/indexing crawlers (OAI-SearchBot, PerplexityBot) build an index used to cite your site in AI answers. User-fetch agents (ChatGPT-User, Perplexity-User) retrieve a page in real time because a user asked. You can allow or block each category independently.

AI Crawlers

AI Crawler Control Generator

Generate a ready-to-paste robots.txt block that allows or disallows GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and other AI crawlers.

Updated Jul 5, 2026FreeNo signup

robots.txt · 23 selected

# AI crawler rules — generated 2026-07-05 by Agent Native
# https://agentnative.dev/tools/ai-crawler-control
# robots.txt is advisory: well-behaved crawlers honor it, some ignore it.
User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: Claude-User
Disallow: /

User-agent: Claude-SearchBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Perplexity-User
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Meta-ExternalAgent
Disallow: /

User-agent: Meta-ExternalFetcher
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: Omgilibot
Disallow: /

User-agent: MistralAI-User
Disallow: /

User-agent: DuckAssistBot
Disallow: /

Add these lines to the robots.txt at your site root. robots.txt is advisory — some crawlers ignore it (marked above). For hard enforcement, block at the network/CDN layer.

TL;DR

Pick the AI crawlers to allow or block and copy the generated robots.txt lines. There are 28 major AI-related user-agents today across OpenAI, Anthropic, Google, Perplexity, Meta, Amazon, Apple and others; 18 are documented to honor robots.txt. Blocking training tokens like GPTBot and ClaudeBot does not affect traditional search ranking. The full user-agent reference is below.

AI crawler user-agent reference

Updated Jul 5, 2026

AI crawler user-agents, operators, and what each controls
User-agent	Operator	Purpose	Honors robots.txt	What blocking it controls
GPTBot	OpenAI	Model training	Yes	OpenAI's foundation-model training crawler. Blocking it does not affect ChatGPT search or user fetches.
OAI-SearchBot	OpenAI	AI search / RAG	Yes	Indexes/cites your site inside ChatGPT search. Blocking removes you from ChatGPT search results.
ChatGPT-User	OpenAI	User-triggered fetch	Unclear	Real-time fetch when a ChatGPT user asks it to visit a URL.
OAI-AdsBot	OpenAI	Other	Yes	Validates advertiser landing pages; not used for training.
ClaudeBot	Anthropic	Model training	Yes	Anthropic's primary training crawler for Claude.
anthropic-ai	Anthropic	Model training	Unclear	Legacy Anthropic training token (superseded by ClaudeBot); still worth including defensively.
Claude-Web	Anthropic	Other	Unclear	Legacy Anthropic token; included defensively.
Claude-User	Anthropic	User-triggered fetch	Yes	Real-time fetch when a Claude user's prompt requires a specific page.
Claude-SearchBot	Anthropic	AI search / RAG	Yes	Anthropic's search-indexing crawler for Claude search quality/citations.
Google-Extended	Google	Model training	Yes	robots.txt token that opts you out of Gemini/Vertex training. Does NOT affect Google Search ranking.
Google-CloudVertexBot	Google	Other	Yes	Crawls on a site owner's request to build Vertex AI grounding data.
PerplexityBot	Perplexity	AI search / RAG	Yes	Perplexity's search-indexing crawler. Blocking removes you from Perplexity answers.
Perplexity-User	Perplexity	User-triggered fetch	No	Live fetch to answer a specific user question. Perplexity states this generally ignores robots.txt.
CCBot	Common Crawl	Model training	Yes	Common Crawl's archive crawler; its dataset is widely reused for LLM training.
Bytespider	ByteDance	Model training	Unclear	ByteDance training crawler; widely reported to disregard robots.txt.
Amazonbot	Amazon	Search indexing	Yes	Gathers content to power answers/citations in Amazon services such as Alexa.
Applebot	Apple	Search indexing	Yes	Apple's base crawler for Siri/Spotlight/Safari suggestions. Blocking removes you from those surfaces.
Applebot-Extended	Apple	Model training	Yes	robots.txt token that opts content out of Apple Foundation Models training. Does not affect Siri/Spotlight search.
Meta-ExternalAgent	Meta	Model training	Yes	Meta's crawler for Meta AI / Llama training data and indexing.
Meta-ExternalFetcher	Meta	User-triggered fetch	Unclear	Real-time fetch by Meta AI on behalf of a user.
FacebookBot	Meta	Model training	Yes	Older Meta crawler for language tech and link previews; may feed Meta AI.
Bingbot	Microsoft	Search indexing	Yes	Bing's search crawler; the same index grounds Copilot. There's no separate AI opt-out token, so blocking it also drops you from Bing Search.
cohere-ai	Cohere	User-triggered fetch	Unclear	Retrieves web data for user prompts to Cohere's enterprise AI. (cohere-training-data-crawler handles training.)
Diffbot	Diffbot	Other	Unclear	AI extraction crawler that structures site content into knowledge-graph data. Compliance not guaranteed.
Timpibot	Timpi	Model training	Yes	Collects public web content for a decentralized index and LLM training data.
Omgilibot	Webz.io	Model training	Unclear	Webz.io crawler whose data is sold, including for LLM training.
MistralAI-User	Mistral	User-triggered fetch	Unclear	Real-time fetch by Mistral's Le Chat on behalf of a user.
DuckAssistBot	DuckDuckGo	User-triggered fetch	Yes	Fetches content for DuckDuckGo's DuckAssist AI answers.

How it works

The generator emits one User-agent group per selected crawler with a Disallow: / (block) or Allow: / (allow) rule. "Recommended" pre-selects AI-specific tokens while leaving general search crawlers (Bingbot, Applebot) unselected, since blocking those also drops you from that search engine.

robots.txt is advisory. The reference table records whether each operator documents that its crawler honors robots.txt — some don't. New crawlers appear regularly; this list is verified against operator docs and Cloudflare's AI crawler reference and updated as the landscape changes.

Sources

FAQ

How do I block AI crawlers with robots.txt?: Add a User-agent line for each AI bot followed by "Disallow: /". For example: "User-agent: GPTBot" then "Disallow: /" blocks OpenAI's training crawler. This tool generates the full block for every major AI crawler — select the bots and copy the output into your robots.txt.
How do I block GPTBot specifically?: Add: User-agent: GPTBot Disallow: / to your robots.txt. Note that GPTBot is only OpenAI's training crawler — blocking it does not stop ChatGPT search (OAI-SearchBot) or user-triggered fetches (ChatGPT-User), which use separate user-agents you can block too.
Does blocking an AI crawler actually stop it?: robots.txt is advisory, not enforcement. Well-behaved crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot's indexer) honor it. Some, like ByteDance's Bytespider and Perplexity's user fetcher, are documented to ignore it — the reference table flags which respect robots.txt. For hard enforcement, block by user-agent or verified IP at your CDN or firewall.
Will blocking AI crawlers hurt my SEO?: Blocking AI-specific tokens like GPTBot, ClaudeBot, and Google-Extended does not affect traditional search ranking — Google-Extended is separate from Googlebot. But blocking a general search crawler such as Bingbot or Applebot also removes you from that search engine, so those are off by default here.
What's the difference between training, search, and user-fetch crawlers?: Training crawlers (GPTBot, ClaudeBot, CCBot) collect data to train models. Search/indexing crawlers (OAI-SearchBot, PerplexityBot) build an index used to cite your site in AI answers. User-fetch agents (ChatGPT-User, Perplexity-User) retrieve a page in real time because a user asked. You can allow or block each category independently.