All guides

Can ChatGPT crawl my website? How to check OpenAI's 3 bots

Check if ChatGPT can crawl and cite your site: OAI-SearchBot vs GPTBot vs ChatGPT-User, robots.txt rules, Cloudflare blocks and a curl test you can run.

, 6 min read

ChatGPT can crawl your website if your robots.txt, CDN and firewall let OpenAI's search crawler, OAI-SearchBot, fetch your pages, and if the text is in the HTML rather than built by JavaScript. To check, read the OAI-SearchBot rules in your robots.txt, look at your CDN's AI crawler settings, and request a page with OpenAI's user agent to see what comes back.

Being crawlable is the entry ticket, not a result. Nothing in this guide guarantees that ChatGPT will cite you. It only removes the reasons it can't.

OpenAI runs three bots that matter here

OpenAI's crawler documentation (checked November 18, 2026) lists separate user agents with separate jobs. The one you block or allow decides what happens.

User-agent token What OpenAI says it does Managed by robots.txt? IP list
OAI-SearchBot Surfaces websites in ChatGPT's search features. Yes openai.com/searchbot.json
GPTBot Crawls content that may be used to train OpenAI's foundation models. Yes openai.com/gptbot.json
ChatGPT-User Visits a page when a user asks ChatGPT or a custom GPT something. Not automatic crawling. "robots.txt rules may not apply" openai.com/chatgpt-user.json

There is also OAI-AdsBot, which only checks landing pages submitted as ChatGPT ads, so most sites can ignore it. OpenAI also notes that if you allow both OAI-SearchBot and GPTBot, it may use one crawl for both purposes.

The full user-agent strings look like this one for GPTBot, taken from the same page:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

OAI-SearchBot's string starts with a Chrome-style browser prefix and ends in compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot. Your robots.txt only needs the short token, such as OAI-SearchBot.

What happens if you block OAI-SearchBot

OpenAI's wording is direct: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." In practice your pages lose the chance to be quoted or cited as a source inside an answer. ChatGPT may still show your site as a navigational link.

What happens if you block GPTBot

Blocking GPTBot tells OpenAI not to use your content to train its models. OpenAI documents it as a separate control from search, so you can block training and still allow search. Whether training data affects how often a model mentions your brand later is not something OpenAI documents, and we have not seen good evidence either way.

How to read your robots.txt

Open https://yourdomain.com/robots.txt in a browser. Three rules from the robots.txt standard, RFC 9309, explain most surprises:

  1. A crawler obeys the group that names it, and ignores User-agent: * when such a group exists. If you have a User-agent: OAI-SearchBot group, the rules under * no longer apply to it.
  2. No named group means the * group applies. A Disallow: / under User-agent: * blocks OAI-SearchBot too, unless OAI-SearchBot has its own group.
  3. The longest matching path wins. Allow: /blog/ beats Disallow: / for a URL under /blog/.

If you want ChatGPT search to see your site but prefer to keep your content out of training, these lines do that:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

If you are happy to allow both, you can drop the GPTBot group, as long as nothing under User-agent: * blocks it. OpenAI says it can take "~24 hours" after a robots.txt update for its search systems to adjust.

Common mistakes we see in robots.txt files:

  • A blanket Disallow: / under User-agent: *, left over from a staging site.
  • An "AI bot block list" pasted from a blog post that includes OAI-SearchBot alongside GPTBot.
  • A robots.txt that returns an error or a login page instead of plain text.

CDN and firewall blocks

Robots.txt is a request. A CDN or firewall is enforcement, and it can block OpenAI's bots even when robots.txt allows them.

Cloudflare. Since September 15, 2026, Cloudflare sorts crawlers into Search, Training and Agent controls. New domains on sites that earn money from ads are offered a preset with Search set to Allow, Training set to Disallow AI Training and Agent set to Block on pages with ads. Disallow AI Training publishes a no-training preference in robots.txt and blocks training-only crawlers, including OpenAI's, while mixed-use crawlers keep access for search. Cloudflare also warns that choosing Block for Training now stops Googlebot, Bingbot and Applebot as well, which removes you from regular search. By Cloudflare's own count, fewer than 1% of its sites block search bots and 17% block training.

To check: in the Cloudflare dashboard, select your domain and go to AI Crawl Control, then open the Crawlers tab (Cloudflare docs). The Action column shows whether each crawler is allowed or blocked. Cloudflare's post does not say which category each OpenAI user agent falls into, so look for OAI-SearchBot, GPTBot and ChatGPT-User by name rather than assuming.

Other CDNs and WAFs. Check for bot-management rules, rate limits and custom rules that match "bot" or unknown user agents. A response of 403, 429, 503 or a JavaScript challenge page means the crawler did not get your content.

How to test it yourself

1. Request a page as OAI-SearchBot

Run this from a terminal, replacing the URL:

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \
  https://yourdomain.com/pricing

Then run it again with a normal browser user agent and compare. A 200 for both is a good sign. A 403 or 503 for the bot only points to a CDN or firewall rule.

Treat this as a hint, not proof. Your request comes from your own IP address, not OpenAI's. Some CDNs verify bots by IP and may challenge your fake request while letting the real crawler through, or the reverse.

2. Check the HTML has your words in it

Drop -o /dev/null and pipe the output through grep for a sentence from the page. If the sentence is missing, your content is probably rendered by JavaScript. Vercel's December 2024 study found GPTBot fetched JavaScript files but did not execute them. That is one dataset and it is two years old, and OpenAI does not say how OAI-SearchBot renders, so server-rendered HTML is the safer bet.

3. Look at your server logs

Search your access logs for OAI-SearchBot and GPTBot. Note the status codes. To confirm a request really came from OpenAI, compare its IP with the ranges in https://openai.com/searchbot.json. No visits at all can mean a block upstream, or simply that OpenAI has not crawled you yet.

4. Check OpenAI's Business Hub, if you qualify

OpenAI opened "Businesses in ChatGPT" in beta on November 10, 2026. According to PPC Land (November 11, 2026), verified businesses see mentions, website clicks and top discovery topics over 7 and 28 days in the Business Hub. Verification uses a text or phone call to the main business number, or DNS, and a government ID is needed before editing business information. Data only appears above undisclosed thresholds, so a small site may see little. It does not show prompts or list cited pages. A "Website access" panel showed "Access confirmed" for one test site, but OpenAI has not explained what that check covers. It tells you whether ChatGPT mentions you, not why.

A quick checklist

  • robots.txt loads as plain text and has no Disallow: / that applies to OAI-SearchBot
  • OAI-SearchBot is not in any pasted "block AI" list
  • Cloudflare AI Crawl Control (or your CDN) shows OAI-SearchBot allowed, and Training is not set to Block
  • A curl request with the OAI-SearchBot user agent returns 200
  • Key pages contain their main text in the raw HTML
  • No noindex on pages you want found
  • Logs show OAI-SearchBot visits from IPs in OpenAI's published ranges

Access is the one lever here that OpenAI itself documents. Most other AEO advice rests on correlation studies, so fix access first and be sceptical of anyone promising citations after that.

Check your site

The free scan at domyaeo.com reads your robots.txt rules for 14 AI and search crawlers, including OAI-SearchBot, GPTBot and ChatGPT-User, and checks Cloudflare Content-Signal, noindex and JavaScript-only rendering. It also tests how your CDN or firewall responds to AI crawler user agents (a hint, not proof) and whether key pages open with a direct answer.