GPTBot, ClaudeBot, PerplexityBot: A 2026 Guide to Robots.txt for AI Crawlers
August 19, 2026
robots.txt user-agent groups don't inherit from each other — a named group like User-agent: GPTBot
does not pick up the rules under User-agent: *. Each named bot needs its own complete set of Allow/
Disallow lines, or it can end up with unintended access (or unintended blocking) by omission.
The two categories, concretely
| Purpose | Bots | Effect of blocking |
|---|---|---|
| Training | GPTBot, ClaudeBot, CCBot, Google-Extended | Content excluded from future model training. No effect on being cited in an answer today. |
| Retrieval / live search | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User | Directly prevents citation in live AI answers — this is the one that matters if you want to be recommended. |
A realistic 2026 posture for a business that wants AI visibility: allow the retrieval bots without question, and make a deliberate, not default, decision about the training bots based on whether you're comfortable with your content shaping a future model version — there's no visibility cost either way today.
Worth knowing as a caveat, not a reason to skip this: robots.txt is a voluntary protocol (RFC 9309) with no technical enforcement. Cloudflare documented in mid-2025 that at least one AI crawler was observed using undeclared, rotating IPs that ignored Disallow rules. Configure it correctly anyway — most legitimate bots honor it, and it's the only lever that exists.
See how our own crawler policy is configured on this site, or check whether your own site's current robots.txt is accidentally blocking a retrieval bot at areyoufindable.com.