AI crawler list (2026)
Every AI bot token that matters, grouped by job — training, search, user fetch — with what each vendor says about robots.txt.
Learn
AI crawlers, measured
Short answers with sources. Every claim is attributable to vendor documentation or to our own published measurement, and every page ends with the free scan so you can check your own site instead of taking our word for it.
Do AI crawlers run JavaScript?
No. What GPTBot, ClaudeBot and PerplexityBot actually receive, how to check in one command, and the fix.
GPTBot vs OAI-SearchBot vs ChatGPT-User
Three OpenAI bots, three jobs. The one that decides whether ChatGPT cites you is not the one templates block.
Cloudflare's AI-bot defaults change on 15 September 2026
What the categories mean, which setting is on, and how to see what your edge returns to the search crawlers.
Does llms.txt matter?
Adoption up 8.8x; 97% of files got zero requests. Where it sits after the four gates that actually decide readability.
Check your own site
Sixty seconds, no signup: fetch your homepage the way GPTBot, ClaudeBot and PerplexityBot fetch it, count the words that survive, and read what your robots.txt actually tells each AI bot.
Scan free →