Why "They Block GPTBot" Is Almost Always Wrong

The symptom

Our first pass through Y Combinator's Fall 2025 batch flagged 9 companies that looked like they'd made a deliberate choice: reject every AI crawler, let Bing straight through. All eleven AI user-agents we probe — GPTBot, ClaudeBot, PerplexityBot, and eight others — came back 403 on every one of those nine domains. bingbot got 200 on all nine. That is exactly the shape of a headline: "9 YC startups block every AI crawler except Google's search partner."

We checked before writing it. Every one of those nine domains sits behind Cloudflare. Every one of the 403s carried a cloudflare-403 challenge marker. That marker means Cloudflare's edge challenged the request before it ever reached the site's origin — the same mechanism Cloudflare uses to verify a crawler is who it claims to be, by checking the request against that crawler's published IP ranges. Our probe comes from one datacenter IP with a spoofed user-agent. It is not on OpenAI's, Anthropic's, or Perplexity's published IP list, so it fails that check every time — correctly. bingbot passing isn't evidence the site favors Bing over AI models. It's evidence Microsoft's IP ranges are on Cloudflare's verified list and our IP isn't. The real GPTBot, arriving from OpenAI's published ranges, is very likely waved through the same door we were turned away from.

We would have shipped a false story about nine companies that, as far as our probe can actually tell, may not be blocking anyone.

The evidence (verify it yourself)

Here's the check that separates "this looks like a block" from "this is a block," in two curl commands anyone can run:

# Step 1 — same request, browser UA vs. AI-bot UA. A status difference alone proves nothing yet.
curl -s -o /dev/null -w "%{http_code}\n" https://[example].com
#   → 200

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
  https://[example].com
#   → 403

# Step 2 — the step almost everyone skips. Read the headers on that 403.
curl -sI -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
  https://[example].com
#   → server: cloudflare
#   → cf-mitigated: challenge          (or a CF challenge page in the body)

A server: cloudflare header plus a challenge marker on the 403 means you tripped a bot-verification gate that checks IP ranges — not a rule that reads your user-agent string and says no. If the header instead just says server: nginx (or CloudFront, Akamai, Imperva) with no challenge marker and a plain 403 or 404 body, that's a different — and separately ambiguous — case, covered below.

Across the 145 YC-batch domains that returned a clean 200 to our plain baseline fetch, we logged 100 non-200 responses across the 11 AI-bot user-agents (bingbot excluded as the control). 99 of those 100 carried a challenge marker — cloudflare-403, in one case paired with vercel-mitigated. Every one of the 9 domains behind that pattern is on Cloudflare, and every one let bingbot through clean. Exactly 1 of the 100 carried no marker at all — one domain, one bot (CCBot, Common Crawl's crawler), on a Squarespace-hosted site whose own robots.txt explicitly allows GPTBot and ClaudeBot — and GPTBot, ClaudeBot, PerplexityBot, and bingbot all got a clean 200 on that same domain. Even our one unmarked case doesn't look like a considered anti-AI policy; it looks like one crawler tripping something the other ten didn't.

Why this happens (blame the default, not the operator)

Cloudflare's verified-bot feature — the setting behind cloudflare-403 — does exactly what it's designed to do: it checks incoming requests claiming to be GPTBot or ClaudeBot against those companies' published IP ranges, and challenges anything that doesn't match. That's a sane default for a site owner to leave on. It's also indistinguishable, from the outside, from a site owner who sat down and typed "deny GPTBot" into a config file — unless you know to look for the marker. The tool is doing its job. The failure was ours: reading a 403 as an answer instead of as a step that still needs a log to confirm.

The parallel failure runs the other way, and we checked for it too. We pulled the same comparison across our larger dataset of enterprise and major-site domains (455 with a clean 200 baseline). There, 192 AI-bot probe events came back non-200, and only 47 of them carried a challenge marker. The other 145 split as 83 plain 403s, 17 405s, 17 429s, 10 202s, and 9 503s — plus 9 more that were not HTTP responses at all: a connection failure after retry, where our probe never got a server to answer. Those 9 aren't a WAF decision of any kind, and we shouldn't have lumped them in with the rest. That leaves 136 genuine unmarked hard HTTP responses spread across five different status codes — a wider, messier spread than one clean "enterprise WAF signature." 26 distinct domains had at least one such unmarked hard response; checking their baseline servers shows nginx (6), Cloudflare (5), CloudFront (4), seven with no server header set at all, and one each of istio-envoy, AmazonS3, LiteSpeed, and Squarespace — no Akamai, no Imperva, and notably five of the 26 are themselves fronted by Cloudflare, the same product whose challenge marker we're treating as the clean, legible case elsewhere in this piece. That undercuts a tidy "Cloudflare marks its challenges, other WAFs don't" story — apparently some Cloudflare-fronted setups don't leave the marker either, which is itself evidence that active probing can't reliably tell an edge policy from a probe artifact. None of this is evidence those 26 block AI: a plain 403 from a datacenter IP claiming to be GPTBot is exactly what a WAF is built to produce for an unverified caller, whether or not the real GPTBot would ever see one. Of those 26, five domains hard-403'd all eleven of our AI user-agents; a sixth hard-failed all eleven with 503s and connection timeouts, not a single 403 among them. bingbot failed too on every one of those six — 403 on the five hard-403 domains, 503 on the sixth — which reads as our IP being rejected wholesale, not AI being singled out. The other twenty had partial hits; where bingbot did pass clean, the most AI user-agents we saw hard-blocked on any single domain was 8 of 11 — never the full set. Zero domains in that dataset combined "every AI bot hard-403'd" with "bingbot passed clean" — the one pattern that would actually look like a deliberate AI blocklist never showed up, in either direction, in either dataset.

The fix (the method, complete and free)

There's no code fix here — nobody in this piece did anything wrong that we can prove. What we're giving away instead is the check itself, so nobody else has to almost-publish what we almost did:

1. 1

   Run the baseline and bot-UA curl above. A status difference is a lead, not a finding.

2. 2

   Pull the headers on the non-200. server: cloudflare + a challenge marker, or a Vercel vercel-mitigated marker, means an edge bot-verification layer fired — likely an IP-range check your real crawler traffic passes even when your probe doesn't.

3. 3

   No marker, plain WAF response? Don't conclude "blocked" either. Enterprise WAFs allowlist verified crawlers by IP the same way Cloudflare does; a bare 403 to a spoofed UA from a datacenter IP tells you nothing about what the real bot gets.

4. 4

   Cross-check against a non-AI control (bingbot) and the site's own robots.txt. If AI bots 403 and bingbot sails through, that's suggestive. If the control fails too, or robots.txt explicitly allows the bot that 403'd, the story is probably "our probe," not "their policy."

5. 5

   Settle it with logs, not another probe. Pull the site's own server logs, filter by crawler user-agent, and verify the requesting IP against that crawler's published range (openai.com/gptbot.json, bing.com/toolbox/bingbot.json, and the equivalent for each bot). That's what confirms a real crawler was actually turned away — no probe from any IP, ours included, can do that. Our own audit tooling (the log-drain parser in github.com/abouchard11/geo-crawl-audit) does exactly this step, and it's open.

What it costs to get this wrong

AI answer engines can only cite what they successfully read, and the crawl behind that reading is enormous relative to the traffic it sends back — Anthropic's crawler generated roughly 70,900 page requests for every one referral visit it sent in return, per Cloudflare Radar. That figure counts referral visits matched by an HTTP Referer header, and Cloudflare's own writeup notes app-native AI traffic often carries none — so treat it as directional, not a precise visit count. Directional is enough: it's exactly why an unconfirmed "blocked" verdict is expensive in both directions: call a site blocked when the real bot is passing fine, and you've either told them to spend engineering time loosening a rule that was never stopping the crawler that matters, or told everyone reading your research that they're invisible when they're not. Get the direction wrong the other way — waving off a genuine hard block as "probably just our probe" — and a site keeps losing reads it will never know it lost, at that same 70,900-to-1 scale.

How you'd verify this yourself

Run the two-step curl check above against any domain you're curious about, then pull its server logs and grep for the crawler user-agent. If you want a second opinion, our free scan runs the same 12-user-agent probe and reports each response with its challenge marker (or lack of one) shown plainly, rather than collapsed into "blocked" or "not blocked."

We'll help you read your own logs for free. If our probe flags a differential on your domain and you can share (or let us look at) the matching server-log window, we'll run it through the same IP-verification check described above and tell you plainly whether a real crawler was turned away — not whether our spoofed one was.