How to block AI training on Cloudflare and stay in AI search
Set Cloudflare's Training control to Disallow AI Training, not Block. Block now also stops Googlebot, Bingbot and Applebot. A setup for B2B sites.
, 6 min read
To block AI training on Cloudflare and keep showing up in AI search, set Search to Allow and Training to Disallow AI Training, not Block. Since September 15, 2026, Cloudflare's "Block" option for training also stops Googlebot, Bingbot and Applebot, which takes your site out of Google, Bing and Apple search along with the AI answers built on them.
The rest of this page explains what changed, what each Content-Signal value asks for, and a setup we suggest for a B2B software site.
What changed on Cloudflare in September 2026
In its September 15, 2026 post, Cloudflare split its AI crawler controls into three categories, each set per domain under Security Settings:
- Search: Allow, Block on pages with ads, Block.
- Training: Allow, Disallow AI Training, Block on pages with ads, Block.
- Agent: Allow, Block on pages with ads, Block.
The same post says the older "Block AI Bots" toggle is deprecated, and Managed robots.txt is replaced by Bot Preference Sync, which writes your choices into robots.txt. Legacy Training settings of "Block" or "Block on pages with ads" were migrated to "Disallow AI Training".
New domains now start from one of two presets:
| Setting | Does not monetize with ads | Monetizes with ads |
|---|---|---|
| Preference Sync | Enabled | Enabled |
| Search | Allow | Allow |
| Training | Allow | Disallow AI Training |
| Agent | Allow | Block on pages with ads |
If you onboarded a domain before the change, look at your live robots.txt anyway. PPC Land (October 8, 2026) reported HasData's finding that 118 Cloudflare sites whose archived robots.txt carried Cloudflare's managed ai-train=no block no longer had it by September 16.
The trap: "Block" for Training blocks Googlebot
Googlebot, Bingbot and Applebot are what Cloudflare calls mixed-use crawlers: one crawler collects pages for both search and AI. Cloudflare's post is direct about it: choosing Block "will stop Applebot, Bingbot, and Googlebot from reaching your site," search included. The same now applies to "Block on pages with ads", which blocks those crawlers on your ad pages.
"Disallow AI Training" is the setting that separates the two. It publishes a no-training preference in robots.txt (for Google and Apple, through their Google-Extended and Applebot-Extended tokens) and lets those crawlers keep crawling for search.
This matters for AI visibility because Google says its AI features are "rooted in our core Search ranking and quality systems" (Google Search Central). If Googlebot cannot reach you, AI Overviews and AI Mode have nothing to work from.
Few sites make this mistake on purpose. Cloudflare's same post says under 1% of its sites block search bots, while 17% use some mechanism to block training. The risk is someone choosing "Block" because it sounds stronger.
What Content-Signal asks for
Content-Signal is a robots.txt line from Cloudflare's Content Signals Policy (September 24, 2025). It has three values:
| Signal | What it covers (Cloudflare's definition) | For AI search, no training |
|---|---|---|
search |
"building a search index and providing search results" (links and short excerpts). It "does not include providing AI-generated search summaries." | yes |
ai-input |
"inputting content into one or more AI models", such as retrieval augmented generation, grounding, or real-time use "for generative AI search answers" | yes |
ai-train |
"training or fine-tuning AI models" | no |
Two details people miss:
ai-inputis the one that covers AI answers. Settingai-input=noasks AI search engines not to use your pages in their answers, which works against the goal of this page. Cloudflare's managed robots.txt line (search=yes, ai-train=no, use=reference) leavesai-inputunset, and per the policy an unset signal "neither grants nor restricts permission".- Signals are requests, not enforcement. Cloudflare's policy post says "content signals express preferences; they are not technical countermeasures against scraping" and "Some companies might simply ignore them."
Know which crawler does what
Some operators run separate crawlers for search and training, so you can block one and keep the other. Others use a single crawler plus a control token.
| Operator | Search or retrieval | Training opt-out |
|---|---|---|
| OpenAI | OAI-SearchBot |
GPTBot |
| Anthropic | Claude-SearchBot |
ClaudeBot |
| Perplexity | PerplexityBot |
none needed: Perplexity says it is "not used to crawl content for AI foundation models" |
Googlebot |
Google-Extended (a token, not a separate crawler) |
|
| Apple | Applebot |
Applebot-Extended |
| Microsoft | Bingbot |
no robots.txt token yet; Cloudflare says one is targeted for early 2027 |
Sources: OpenAI, Anthropic, Perplexity, Google, and Cloudflare's September 15 post for Apple and Microsoft.
OpenAI's documentation is explicit about the cost of getting this wrong: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
One trade-off to know about: Google says Google-Extended controls Gemini training and also grounding in Gemini Apps and Vertex AI, while it "does not impact a site's inclusion in Google Search." Disallowing it may therefore reduce how your pages are used in the Gemini app, even though Google Search is unaffected.
Recommended setup for a B2B site
This is the setup we suggest for a software or online-service company that wants to appear in AI answers but not in training data.
1. Set the three categories
In the Cloudflare dashboard, go to Security Settings, filter by Bot traffic, and open AI Crawler and Agent access policies (Cloudflare docs).
- Search: Allow.
- Training: Disallow AI Training. Never Block or Block on pages with ads.
- Agent: Allow, unless you have a specific reason not to. Agents fetch pages when a person asks an assistant to do something, such as compare vendors. Whether to allow them is a judgment call, not something the evidence settles.
- Turn on Enable Bot Preference Sync and save.
2. Check individual crawlers
In AI Crawl Control, the Crawlers table has an Action column with Allow and Block (Cloudflare docs). Make sure OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and Applebot are allowed. Blocking dedicated training crawlers such as GPTBot and ClaudeBot here is safe; Cloudflare enforces it with a WAF custom rule rather than a polite request.
3. Confirm your robots.txt
Bot Preference Sync prepends its own block to your file. If you write robots.txt yourself instead, the core looks like this:
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
(CCBot is Common Crawl, which is on Cloudflare's managed disallow list.) Then open https://yourdomain.com/robots.txt and read the final file. Make sure no group disallows Googlebot, Bingbot, Applebot, OAI-SearchBot, Claude-SearchBot or PerplexityBot, and that synced and hand-written rules do not contradict each other.
4. Leave Bingbot alone
There is no training-only opt-out for Bingbot in robots.txt yet. Cloudflare lists the NOARCHIVE meta tag and Bing's Block URLs or Content Removal tool as current options. Blocking Bingbot to stop training also removes you from Bing.
What this setup does not do
It keeps the door open; it does not get you cited. Being crawlable is a precondition, and most published evidence on what earns AI citations is correlational. Some crawlers may ignore Content-Signal entirely. And Cloudflare's controls change often (Bot Preference Sync arrived in August 2026, the category changes in September), so check your settings again after each announcement.
Check your site
The free scan at domyaeo.com reads your robots.txt rules for 14 AI and search crawlers, your Cloudflare Content-Signal line, and how your CDN or firewall responds to AI crawler user agents (a hint, not proof). It also flags JavaScript-only rendering, noindex, and whether your key pages open with a direct answer.