Reference

AI crawler list (2026): every AI bot user agent, what it does, and whether robots.txt applies

By Alex Bouchard · 6 September 2026

Every table below is built from the same registry our free scan and the ReadableByAI Index use, so the tokens here are the ones actually probed against your site. The grouping matters more than the names: a training crawler decides whether your pages end up in a model, a search crawler decides whether an assistant can find and cite you today, and a user-triggered fetcher is what shows up in your logs when a person pastes your URL into an assistant. Most "block all AI" robots.txt templates block all three at once.

Search / retrieval crawlers

Indexes the web so the assistant can search it and cite you. Disallow it and you are absent from that assistant's search answers.

robots.txt token	Operator	What the vendor says about robots.txt	Docs
`OAI-SearchBot`	OpenAI	Honoured. OpenAI: sites that disallow it will not appear in ChatGPT search answers.	vendor page
`Claude-SearchBot`	Anthropic	Honoured. Anthropic: blocking it can reduce visibility in Claude's search results.	vendor page
`PerplexityBot`	Perplexity	Honoured. Perplexity recommends allowing it to appear in its search results.	vendor page
`bingbot`	Microsoft	Honoured. Feeds Bing and Copilot.	vendor page
`Amazonbot`	Amazon	Honoured per Amazon's documentation.	vendor page
`DuckAssistBot`	—	See operator documentation.	—

User-triggered fetchers

Fetches a page because a person asked the assistant about it. OpenAI and Perplexity say robots.txt rules may not apply to theirs; Anthropic documents Claude-User as controllable through robots.txt.

robots.txt token	Operator	What the vendor says about robots.txt	Docs
`ChatGPT-User`	OpenAI	OpenAI: because the fetch is initiated by a user, robots.txt rules may not apply.	vendor page
`Claude-User`	Anthropic	User-directed fetch. Blocking it prevents retrieval when a person asks Claude about your page.	vendor page
`Perplexity-User`	Perplexity	Perplexity: since a user requested the fetch, it generally ignores robots.txt.	vendor page
`MistralAI-User`	—	See operator documentation.	—

Training crawlers

Collects pages that may be used to train models. Blocking it is a training opt-out; it does not by itself remove you from answers.

robots.txt token	Operator	What the vendor says about robots.txt	Docs
`GPTBot`	OpenAI	Honoured. Disallow = do not use my content for training.	vendor page
`ClaudeBot`	Anthropic	Honoured. Disallow = exclude from training data.	vendor page
`Google-Extended`	Google	Opt-out token only: it never fetches. Controls use of Googlebot's crawl for Gemini training and grounding.	vendor page
`Applebot-Extended`	Apple	Opt-out token only: it never fetches. It tells Apple not to use Applebot's crawl for training.	vendor page
`CCBot`	Common Crawl	Honoured. Its archive is a common training source for many labs.	vendor page
`meta-externalagent`	Meta	Honoured per Meta's documentation.	vendor page
`Bytespider`	ByteDance	No public bot documentation we can cite; treat as a training crawler.	—

Advertisings

Visits pages submitted as ad landing pages. Not a crawler of the open web.

robots.txt token	Operator	What the vendor says about robots.txt	Docs
`OAI-AdsBot`	OpenAI	Only visits pages submitted as ads; not used for training.	vendor page

The robots.txt that keeps you citable and opts out of training

If your policy is "cite me, do not train on me", the tokens split cleanly by vendor. This is the shape most companies actually want and few templates produce:

# Retrieval / search: allow (this is what gets you cited)
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /

# Training: disallow (your call — it does not affect citations)
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /

Two things a robots.txt cannot do: it cannot make a blank page readable (see do AI crawlers run JavaScript?), and it cannot override a WAF or bot-management rule that returns a 403 or a challenge page before the crawler ever reads robots.txt (see Cloudflare's AI-bot defaults).

Check your own site

Paste your domain and the scan lists every token above with its verdict for your site: allowed, explicitly disallowed, or blocked at the edge before robots.txt was ever consulted.

Scan free →

Sources

Questions people ask

Which AI crawler decides whether ChatGPT cites my site?

OAI-SearchBot. OpenAI's documentation says sites that disallow OAI-SearchBot will not appear in ChatGPT search answers. GPTBot is the training crawler; blocking it does not remove you from search answers, and ChatGPT-User is the fetch that happens when a person asks about a specific page.

Does blocking GPTBot hurt my visibility in ChatGPT?

Not by itself, according to OpenAI: GPTBot is the training crawler and OAI-SearchBot is the search crawler. Many robots.txt templates block both under a 'block all AI' rule, which is where the visibility loss comes from.

Do AI crawlers respect robots.txt?

The training and search crawlers from OpenAI, Anthropic and Perplexity document that they honour robots.txt. The user-triggered fetchers are different: OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores robots.txt because a person requested the fetch, and Anthropic documents Claude-User as something you can block through robots.txt.

Which AI crawlers run JavaScript?

Googlebot and Applebot render pages. OpenAI's, Anthropic's and Perplexity's crawlers are not documented as executing JavaScript, and Vercel's network study found none of them do. A page whose text only appears after JavaScript runs is blank to them.

Check your own site

Sixty seconds, no signup: fetch your homepage the way GPTBot, ClaudeBot and PerplexityBot fetch it, count the words that survive, and read what your robots.txt actually tells each AI bot.

Scan free →

Corrections: every claim on this page is attributable to vendor documentation or to our own published measurement. If one is wrong, tell us and it gets fixed with a dated note.