Cloudflare's AI-bot defaults change on 15 September 2026. Here is what your site blocks today
By Alex Bouchard · 6 September 2026
On 1 July 2025 Cloudflare said it was "changing the default to block AI crawlers unless they pay creators for their content." Its bot documentation now dates the next step: on 15 September 2026 the legacy "Block AI bots" setting is deprecated and new domains get updated defaults, with Training and Agent bots blocked on pages that display ads and Search left allowed. If your site sits behind Cloudflare, the question is not whether you have an AI-bot policy. You have one. The question is which one, and whether it blocks the crawlers that would cite you.
Three categories, three different consequences
Cloudflare classifies verified bots, and the AI ones fall into categories with very different stakes:
| Cloudflare category | Cloudflare's definition | Example | If you block it |
|---|---|---|---|
| AI Crawler | Crawls websites for content that is used for training AI models. | GPTBot, ClaudeBot | Training opt-out. Citations unaffected. |
| AI Search (now under Search) | Powers AI-driven search experiences. | OAI-SearchBot | Absent from ChatGPT / Perplexity / Claude search answers. |
| AI Assistant | Automated AI bot driven by user action. | Perplexity-User | A person who pastes your URL into an assistant gets nothing. |
Cloudflare notes that as of 1 July 2026 the AI Search category is kept for backward compatibility and new search crawlers are classified under the broader Search behaviour category.
Which setting is actually on
The legacy Block AI bots setting, per Cloudflare, "blocks verified bots that are classified as crawling for the purpose of AI training, as well as a number of unverified bots that behave similarly" and "excludes mixed-purpose bots that are used both for Training and for Search." Its options are block on all pages, block on pages with ads, or allow. AI Crawl Control is the newer per-crawler surface: allow, block, or charge (Pay Per Crawl) for each named crawler, with the category shown next to it. Then there is everything else on the zone that can answer a request before any of this runs: WAF custom rules, managed challenges, Super Bot Fight Mode, rate limiting.
That layering is why "I checked the toggle" is not an answer. The toggle is one of several things that decide what a crawler receives, and the crawler receives one thing.
See what the crawler gets
Fetch your homepage as the search crawler and look at the status and the headers:
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot" \ https://example.com/ | head -20
A 200 with your page is the good case. A 403, a 503 with a cf-mitigated: challenge header, or a tiny HTML body that says "Just a moment" means the edge answered instead of your origin. One honest caveat, which we learned the hard way: a probe from a datacenter IP can be challenged where the real crawler, arriving from its published IP range, is let through. A challenge to a probe is a lead. Your server logs, filtered by user agent and verified against the vendor's IP list, are the proof — and that is exactly what the Verified tier reads.
The setting most companies want
Allow the Search and AI Assistant categories, decide Training on its merits, and make sure no WAF or challenge rule fires on the verified search crawlers. Then re-check after every Cloudflare change, because the defaults are moving and the failure is silent: a blocked crawler leaves no trace in your analytics.
Paste your domain. The scan fetches your homepage as OAI-SearchBot, PerplexityBot, Claude-SearchBot and the training crawlers, shows the status and challenge markers your edge returned to each one, and reads your robots.txt token by token.
Scan free →Sources
Questions people ask
Does Cloudflare block ChatGPT from my site by default?
It depends on when your zone was created and which setting is on. Cloudflare announced in July 2025 that it was changing the default to block AI crawlers. Its legacy 'Block AI bots' setting targets bots classified as crawling for AI training and excludes mixed-purpose bots used for both training and search. From 15 September 2026 new domains get updated defaults: training and agent bots blocked on pages with ads, search allowed. The only way to know what your site does is to observe it.
What is the difference between an AI Crawler, an AI Search bot and an AI Assistant in Cloudflare?
Cloudflare's verified-bot categories define AI Crawler as crawling for content used to train models, AI Search as powering AI-driven search experiences (its example is OAI-SearchBot), and AI Assistant as an automated bot driven by user action (its example is Perplexity-User). Blocking the first is a training opt-out; blocking the second removes you from AI search answers.
I allowed OAI-SearchBot in robots.txt. Why does it still get a 403?
robots.txt is read by the crawler after your edge lets the request through. A WAF rule, a managed challenge or a bot-management block answers before that, so the Allow line never matters. Check the response your edge returns to the crawler's user agent, then confirm it in your logs.
Should I block AI training crawlers?
That is a business decision, and Cloudflare's categories let you make it separately from search. What almost nobody wants is to block the search crawlers by accident, which is what a coarse 'block AI' rule does.
Sixty seconds, no signup: fetch your homepage the way GPTBot, ClaudeBot and PerplexityBot fetch it, count the words that survive, and read what your robots.txt actually tells each AI bot.
Scan free →Corrections: every claim on this page is attributable to vendor documentation or to our own published measurement. If one is wrong, tell us and it gets fixed with a dated note.