AI Crawlers and robots.txt: Should You Block or Allow GPTBot, ClaudeBot and PerplexityBot in 2026?

What are AI crawlers, and why do they need their own robots.txt rules?

AI crawlers are automated bots that AI companies use to fetch pages from a site for one of two distinct purposes: training a foundation model on the content, or retrieving a page in real time to answer a specific user's question or search query. Each purpose gets its own bot and its own user-agent token, which is why a single rule can't sensibly cover all of them.

This split matters because the two purposes have opposite implications for a business. A training crawler feeds your content into a model with no direct, attributable return; a citation or search crawler is the mechanism by which your brand actually gets named and linked inside an AI-generated answer. Treating both the same way in robots.txt means either giving up training data for nothing or losing citation visibility you almost certainly want, which is exactly the outcome answer engine optimization work is trying to earn.

GPTBot, ClaudeBot, PerplexityBot and Google-Extended: what each one actually does

The major AI companies each split their crawling into a training bot and one or more citation/search bots, and disallowing one does not disallow the others. Knowing which token controls which behaviour is the whole game — get the user-agent wrong and you block (or allow) the opposite of what you intended.

OpenAI, Anthropic and Perplexity all publish official documentation naming their bots and explaining exactly how to control each one — see OpenAI's crawler overview, Anthropic's crawler support page and Perplexity's robots.txt help article — and it's worth reading these directly rather than relying on a third-hand summary, since the specifics occasionally change.

• GPTBot (OpenAI) — crawls for model training; disallowing it opts your content out of training data. OAI-SearchBot and ChatGPT-User are separate tokens used for search and for live user requests inside ChatGPT.
• ClaudeBot (Anthropic) — crawls for training data collection. Claude-User fetches pages only when a person asks Claude to read one; Claude-SearchBot indexes content for Claude's search features. Each has its own user-agent and its own on/off switch.
• PerplexityBot (Perplexity) — gathers content to answer user queries with citations, and respects robots.txt disallow rules for full-text indexing. Perplexity-User, tied to a live user request, generally does not follow robots.txt the way a background crawler does.
• Google-Extended (Google) — not a separate crawler with its own HTTP requests, but a robots.txt token that lets you opt your already-crawled content out of Gemini and Vertex AI training without affecting Googlebot or your regular Search visibility.

Should you block or allow AI crawlers?

Most businesses should allow the crawlers tied to search and citations, since that is how a brand gets referenced inside AI-generated answers, and should make a deliberate, separate call on training-only crawlers based on whether they are comfortable with content being used to train a model with no direct attribution back. There is no universally correct answer for the training question — it is a business decision, not a technical one.

The stronger the case that a page should compete for AI citations — pricing pages, comparison content, definitional guides, anything written with SEO content writing aimed at answering a specific question — the stronger the case for leaving the relevant search-oriented bots unblocked on that page, even for a business that chooses to disallow training bots elsewhere on the site.

• Default to allowing citation/search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) if AI visibility matters to the business
• Decide on training bots (GPTBot, ClaudeBot, Google-Extended) independently — this is a content-licensing choice, not an SEO one
• Blocking a training bot does not remove you from that company's AI-generated answers, since search and training use separate tokens
• Revisit the decision periodically — bot names, behaviours and documentation have changed multiple times since 2023 and will again

How to write robots.txt rules for AI crawlers

Write AI crawler rules as their own dedicated user-agent blocks in robots.txt, one per bot, rather than trying to fold them into an existing wildcard rule — a wildcard `User-agent: *` block does not reliably control named bots the way an explicit block for that exact token does.

A block to fully disallow a training bot looks like `User-agent: GPTBot` followed by `Disallow: /` on the next line, repeated with the exact user-agent string for each bot you want to block, applied to every subdomain you want covered. Google's own robots.txt guide is a solid reference for syntax even though it predates AI-specific bots, and after any change you should verify the file is reachable and correctly parsed — exactly the kind of crawl-directive issue a routine technical SEO audit is built to catch before it silently costs visibility.

• Use exact, documented user-agent strings — a typo or outdated token simply gets ignored by the bot it was meant to control
• Give each bot its own User-agent/Disallow block instead of relying on a wildcard rule to catch it
• Apply changes to every subdomain that serves content you care about, not just the root domain
• Re-test after any CMS, hosting or platform migration, since robots.txt is a common casualty of site moves
• Remember disallow rules are a request most bots honor, not an access-control mechanism — sensitive content needs real authentication, not robots.txt

Common mistakes that quietly cost you AI visibility

The most common mistake is blocking an entire AI company by name — using one blanket rule that happens to match a company's training bot and its citation bot at once — when the intent was only to opt out of training. Because each purpose has its own token, a careless copy-pasted rule can silently remove a business from AI answers it actually wanted to appear in.

A second, quieter mistake is never checking again after the initial setup. Bot names and behaviours have shifted more than once since AI crawlers first appeared, and a rule that was correct in 2024 can be stale in 2026. Folding a periodic robots.txt review into ongoing on-page SEO and technical maintenance work is a small habit that prevents a large, invisible loss of visibility.

Where this fits into a broader SEO and AEO strategy

robots.txt decisions for AI crawlers are plumbing, not strategy — getting them right simply ensures the content you have already invested in is actually reachable by the systems deciding whether to cite you, which is a prerequisite for AEO working at all rather than a strategy on its own. Blocking the wrong bot can undo months of content and authority work in one file.

Because the right configuration depends on a business's specific goals around AI visibility versus content licensing, and because the bots themselves keep evolving, this is a good candidate to fold into ongoing SEO consulting rather than a one-time setup — checked whenever a new AI crawler appears or an existing one changes its documented behaviour.