ANSWERS

What should go in robots.txt for AI crawlers?

robots.txt controls which crawlers may fetch your pages, one user-agent block per agent. Allow search crawlers so public pages stay discoverable, and decide training permissions separately as a content-policy choice. Training crawlers, search crawlers, and user fetchers are different agents, so one directive doesn't cover the others. ye5's GEO Web audit checks whether your current file helps or hurts.

Which AI crawlers should your robots.txt address?

These user-agent tokens are examples of the agents worth addressing.

Crawler	Operator	Job
GPTBot	OpenAI	Trains OpenAI's models on your public pages
ChatGPT-User	OpenAI	Fetches a specific page live when a user asks ChatGPT about it
OAI-SearchBot	OpenAI	Search crawler for ChatGPT search, not a live user-request fetcher
ClaudeBot	Anthropic	Training crawler for Anthropic's models
Google-Extended	Google	Permission token, not a crawler — controls Gemini and Vertex AI training and specified grounding uses; does not affect Google Search inclusion or ranking, including AI Overviews
PerplexityBot	Perplexity	Crawls and indexes pages for Perplexity's answer engine
CCBot	Common Crawl	Builds the open dataset several AI labs train on

ClaudeBot's own crawl pattern on ye5.ai is covered in detail elsewhere (see: What is ClaudeBot?); GPTBot's is covered on its own page (see: What is GPTBot and what does it do with my content?).

What are the three categories of AI crawler?

AI agents fall into three categories — training crawlers, search crawlers, and user fetchers — and a robots.txt line controls only the agent it names.

—Training crawlers. GPTBot, ClaudeBot, and CCBot collect content that may be used to train future models.
—Search crawlers. OAI-SearchBot, Claude-SearchBot, and PerplexityBot crawl and index pages for their providers' search answers.
—User fetchers. ChatGPT-User, Claude-User, and Perplexity-User fetch a page when a person asks the assistant about it; robots.txt rules may not apply to ChatGPT-User.

Disallowing GPTBot doesn't touch ChatGPT-User. Google-Extended is a permission token, not a crawler — it controls Gemini and Vertex AI training and specified grounding uses, and does not affect Google Search inclusion or ranking, including AI Overviews. Each block is independent.

What should your directives say?

These directives are examples of what most sites address, not a complete set.

—Allow search crawlers such as OAI-SearchBot, Claude-SearchBot, and PerplexityBot, so public pages stay eligible for those providers' search answers.
—Choose training permissions for GPTBot, ClaudeBot, and CCBot separately, as a content-policy decision — training access is not required for search visibility.
—Handle Google-Extended as a permission token rather than a crawler line. It has no separate HTTP user-agent, and it governs Gemini and Vertex AI training and specified grounding uses; Google Search visibility is governed separately.
—Treat user fetchers differently, because they act on a person's request: robots.txt rules may not apply to ChatGPT-User, and Perplexity-User generally ignores robots.txt.
—Decide CCBot as a training permission — it collects content for the open Common Crawl dataset that several AI labs train on.
—Disallow what you'd disallow any crawler — admin paths, checkout flows, duplicate parameter URLs. A narrow disallow list keeps the rest of your site open by default, and staging and private content need authentication, because robots.txt is not access control.

Should you ever block AI crawlers instead?

Blocking is sometimes the right call — for a brand opting out of training data on principle. Staging and private content need authentication instead. The tradeoffs are covered in full (see: Should I block AI crawlers from my site?). For most public marketing pages, the default that keeps you visible is allow.

How do you check whether the directives are working?

Reading your robots.txt tells you what you intended, not what's happening. Two checks confirm the difference.

—Log check. Filter your server or CDN logs for the user-agent strings above and see which ones show up, and how often. User-agent strings can be spoofed, so verify the requesting IPs as well as the user-agent. Google-Extended will not appear, because it has no separate HTTP user-agent.
—Machine-access review. Confirm what a crawler can reach once it's allowed in — JavaScript-only content and login walls block AI systems as effectively as a Disallow line.

ye5's GEO Web audit runs that review, checking your directives alongside everything else blocking machine access to your pages.

FAQ

Does blocking GPTBot stop ChatGPT from citing my site?

No. GPTBot is OpenAI's training crawler. OAI-SearchBot is the separate search crawler behind ChatGPT search, and ChatGPT-User is the live fetcher that pulls a page when someone asks about your URL. Training directives govern collection; the search and user-fetch agents are controlled separately.

Is Google-Extended the same as Googlebot?

No. Googlebot indexes your site for Google Search. Google-Extended is a separate permission controlling Gemini and Vertex AI training and specified grounding uses. It does not affect Google Search inclusion or ranking, including AI Overviews.

Do I need a separate robots.txt file for AI crawlers?

No. One robots.txt file at your root can list directives for any crawler you want to address, AI or otherwise, alongside the search-engine rules you already have.