AI VISIBILITY SUITE
ANSWERS
Blocking GPTBot or ClaudeBot does not directly control the separate search and user-fetch agents used by those providers. Provider controls differ: Google-Extended is a robots.txt permission token that also controls grounding in specified Gemini and Vertex AI products. ye5's GEO Web audit checks your machine-access setup.
Training crawlers copy your pages into a dataset used to build a future model version. Retrieval mechanisms fetch a live page when someone asks a question, or draw on an index or a cached copy, through AI Overviews, ChatGPT and Perplexity browsing, or Copilot. ye5's GEO Web audit treats the two separately: it checks whether automated readers can reach your site today, not whether you allowed last month's training crawl.
The training side runs on its own clock (see: How long does GEO take to show results?). Training directives govern future collection; they do not erase existing datasets or guarantee a change in any model version.
These examples show how three providers separate or combine training and retrieval controls.
| Bot | Runs by | Feeds |
|---|---|---|
| GPTBot | OpenAI | Training data for future ChatGPT versions; a separate choice from OAI-SearchBot |
| ClaudeBot | Anthropic | Training data for future Claude versions; a separate choice from Claude-SearchBot and Claude-User |
| Google-Extended | Google (robots.txt permission token, no separate request user-agent) | Training and grounding in specified Gemini and Vertex AI products, not Google Search inclusion or ranking |
Blocking GPTBot or ClaudeBot keeps future crawls out of that bot's training set. Blocking GPTBot is an independent choice from allowing OAI-SearchBot, and blocking ClaudeBot is independent from allowing Claude-SearchBot and Claude-User, so the training directive does not control those separate search and user-fetch agents. Google-Extended works differently: disallowing it also withholds grounding in the specified Gemini and Vertex AI products (see: What should go in robots.txt for AI crawlers?).
Blocking training bots leaves several things untouched.
The decision depends on what you're protecting. If you want to keep proprietary content out of a future training set, disallowing GPTBot or ClaudeBot is the lever, and the Google-Extended directive is the equivalent control for Google. If your concern is whether AI Overviews, Perplexity, or a shopping agent can read and cite your site, the search and user-fetch agents are separate levers: OAI-SearchBot for OpenAI, Claude-SearchBot and Claude-User for Anthropic, while Google-Extended also governs grounding in specified Gemini and Vertex AI products. The broader trade-off between visibility and control belongs to a wider question (see: Should I block AI crawlers from my site?). ye5's GEO Web audit checks whether your machine-access setup blocks the crawlers your retrieval visibility depends on.
No. Google-Extended is a robots.txt permission token controlling training and grounding in specified Gemini and Vertex AI products, and it does not affect Google Search inclusion or ranking. Googlebot crawls pages; Google Search ranks them.
Unblocking allows future access; it does not guarantee crawling, training, indexing, citations or correction of previous answers.
No. GPTBot and ClaudeBot use separate user-agent strings from the search and user-fetch agents: OAI-SearchBot and ChatGPT-User for OpenAI, Claude-SearchBot and Claude-User for Anthropic. robots.txt can allow one and block the other. User-triggered ChatGPT-User requests may not follow robots.txt. Google-Extended has no separate HTTP request user-agent at all — it is a permission token read in robots.txt.