OAI-SearchBot vs GPTBot vs ChatGPT-User: which should you allow?
OpenAI runs three crawlers. Allow OAI-SearchBot to appear in ChatGPT search, block GPTBot to opt out of training. Exact robots.txt lines for each bot.
, 6 min read
If you want your site to show up in ChatGPT search but not be used to train OpenAI's models, allow OAI-SearchBot and disallow GPTBot in robots.txt. ChatGPT-User is a third agent that fetches pages when a person asks ChatGPT to, and OpenAI says robots.txt rules may not apply to it at all.
The three bots do different jobs, so a single "block OpenAI" rule usually costs more than intended. This guide covers what each one does, the exact robots.txt lines, and the matching bots from Anthropic, Perplexity and Google.
What each OpenAI crawler does
All three are described on OpenAI's crawler documentation page (checked November 18, 2026).
OAI-SearchBot: ChatGPT search
OAI-SearchBot surfaces websites in search results in ChatGPT's search features. This is the one that controls whether your pages can appear in ChatGPT search answers at all. OpenAI's wording is direct: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
- Robots.txt token:
OAI-SearchBot - User-agent string contains:
compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot(the version number may change) - Published IP ranges:
https://openai.com/searchbot.json
GPTBot: model training
GPTBot crawls content that may be used to train OpenAI's generative AI foundation models. According to OpenAI, disallowing GPTBot "indicates a site's content should not be used in training generative AI foundation models." Blocking it does not, by OpenAI's description, remove you from ChatGPT search. That is OAI-SearchBot's job.
- Robots.txt token:
GPTBot - User-agent string contains:
compatible; GPTBot/1.4; +https://openai.com/gptbot - Published IP ranges:
https://openai.com/gptbot.json
ChatGPT-User: fetches a person asked for
ChatGPT-User handles actions a user starts in ChatGPT or a Custom GPT, such as opening a page to answer a question. OpenAI says it is not used for automatic crawling and does not decide whether content appears in search. It also says: "Because these actions are initiated by a user, robots.txt rules may not apply."
- Robots.txt token:
ChatGPT-User - User-agent string contains:
compatible; ChatGPT-User/1.0; +https://openai.com/bot - Published IP ranges:
https://openai.com/chatgpt-user.json
A fourth agent, OAI-AdsBot, only visits landing pages submitted as ChatGPT ads. OpenAI says its data is not used for model training.
OpenAI notes that it can take about 24 hours after a robots.txt change for its search systems to adjust.
The robots.txt you probably want
For most small software and online-service companies, this is the common setup: visible in ChatGPT search, out of training.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /
If you are fine with training too, skip the GPTBot group, or set it to Allow: /. If you want out of OpenAI entirely, disallow OAI-SearchBot as well, knowing that it removes you from ChatGPT search answers.
Watch out for a catch-all block
Under the robots.txt standard (RFC 9309), a crawler follows the group that names it and ignores the User-agent: * group. That cuts both ways:
- If your file has
User-agent: *withDisallow: /(common on staging sites that got copied to production), OAI-SearchBot is blocked unless it has its own group. - If you give OAI-SearchBot its own group, it no longer inherits the
Disallowlines you wrote under*. Copy any paths you still want to keep private, such as/admin/, into its group.
Robots.txt is not the only gate
A clean robots.txt does nothing if your CDN or firewall returns a 403 to the bot. This is common with bot-protection products. OpenAI publishes the IP lists above so you can allow verified requests instead of trusting the user-agent string, which anyone can fake.
Cloudflare deserves a specific check. Its September 15, 2026 post describes per-domain controls under Security Settings with three categories, Search, Training and Agent, and the options Allow, Disallow AI Training, Block on pages with ads and Block. Two details matter:
- Setting Training to Block now also stops mixed-use crawlers, and Cloudflare says it will "stop Applebot, Bingbot, and Googlebot from reaching your site," search included. To stay out of training but keep search, Cloudflare points to Disallow AI Training instead.
- Since September 15, Cloudflare offers new ad-supported domains a preset during onboarding: Search Allow, Training Disallow AI Training, and Agent Block on pages with ads. The post gives "chat fetch bots and browser-use agents" as examples of the Agent category. It does not name ChatGPT-User, so check how your own dashboard classifies it.
Cloudflare's managed robots.txt documentation shows where the setting lives: Security Settings, filtered by Bot traffic, then AI Crawler and Agent access policies. When Bot Preference Sync is on, Cloudflare adds lines to your robots.txt, including a line such as Content-signal: search=yes, ai-train=no, use=reference. Open yourdomain.com/robots.txt after any change to see what is actually served.
Equivalent bots at Anthropic, Perplexity and Google
| Company | Search / index bot | Training bot | User-initiated fetcher | Follows robots.txt? |
|---|---|---|---|---|
| OpenAI | OAI-SearchBot | GPTBot | ChatGPT-User | Search and training bots yes; ChatGPT-User "may not" |
| Anthropic | Claude-SearchBot | ClaudeBot | Claude-User | Anthropic says its bots honor robots.txt |
| Perplexity | PerplexityBot | None listed (PerplexityBot is not used for training) | Perplexity-User | PerplexityBot yes; Perplexity-User "generally ignores" it |
| Googlebot (Search, AI Overviews, AI Mode) | Google-Extended (a token, not a separate bot) | Not covered here | Yes |
Anthropic
Anthropic's help article (checked November 18, 2026) says blocking ClaudeBot signals that future content should be excluded from training. Blocking Claude-SearchBot prevents indexing for search and blocking Claude-User prevents retrieval when someone asks Claude a question; Anthropic says both may reduce your visibility in user search. It also supports the non-standard Crawl-delay directive.
Perplexity
Perplexity's bot documentation says PerplexityBot surfaces and links sites in Perplexity results and "is not used to crawl content for AI foundation models." Perplexity-User fetches pages when a user asks, and "generally ignores robots.txt rules." IP lists are at https://www.perplexity.ai/perplexitybot.json and https://www.perplexity.ai/perplexity-user.json.
Google's crawler documentation says Google-Extended controls whether crawled content may be used to train future Gemini models (for Gemini Apps and the Vertex AI API for Gemini) and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. It has no user-agent string of its own; it is a robots.txt token read by Google's existing crawlers.
What Google-Extended does not control:
- Google Search: Google says it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
- AI Overviews and AI Mode: Google's AI features page says robots.txt rules for Googlebot are the control for these. Blocking Googlebot removes you from Search itself.
Because Google-Extended also covers grounding in Gemini Apps, disallowing it is not purely a training opt-out. Decide whether Gemini app answers matter to you before adding it.
A combined example
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
Disallow: /
# Optional: also opts out of grounding in Gemini Apps
# User-agent: Google-Extended
# Disallow: /
User-agent: *
Allow: /
What allowing these bots will and will not do
Allowing search bots removes a documented barrier. It does not promise citations. OpenAI's statement about OAI-SearchBot is one of the few first-party, cause-and-effect facts in this field. Most other AEO findings are correlational studies, many published by companies that sell AEO tools.
One more technical point: a Vercel and MERJ study (December 2024) found that none of the major AI crawlers it tracked, including GPTBot, ClaudeBot and PerplexityBot, render JavaScript. OpenAI's and Anthropic's crawlers fetched JavaScript files but did not execute them. The data is now nearly two years old, but if your key pages only render in the browser, these bots may see an empty page even when robots.txt allows them.
Check your site
The free scan at domyaeo.com reads your robots.txt rules for 14 AI and search crawlers, including all the bots above, and reports any Cloudflare Content-Signal line. It also tests how your CDN or firewall responds to AI crawler user agents (a hint, not proof), and flags JavaScript-only rendering, noindex, and pages that do not open with a direct answer.