ANSWERS

Should I block AI crawlers from my site?

For public marketing pages, choose access settings by purpose rather than blocking every AI bot. Training, search and user-requested fetching use different controls. Blocking can reduce access, but does not erase old model knowledge or guarantee removal of every citation. ye5's GEO Web audit checks your site's access and readiness.

Training, search, and user-requested fetching use different controls

AI bots visit your site for three different purposes. Training crawlers such as GPTBot collect content that may be used to train future models. Search crawlers index pages for an assistant's search index. User-requested fetches pull a page in real time, when a person asks their assistant about you. Controls differ by purpose and by provider, so blocking one bot does not necessarily block the others.

OpenAI runs three bots, and each one serves a different purpose:

Bot	Purpose	What blocking it affects
GPTBot	Training	Future model training data
OAI-SearchBot	ChatGPT search index	Whether your pages can be found and cited in ChatGPT search
ChatGPT-User	User-triggered fetching	A page fetched live when a person asks about you; robots.txt rules may not apply to it

Block a training crawler and you opt out of that model's future training data. Restricting a search crawler can reduce eligibility for that provider's search answers, while a user-triggered fetch follows its own rules. Blocking reduces access going forward. It does not erase what a model already holds, and it does not guarantee removal of every existing citation.

When blocking makes sense

—Staging and pre-launch environments. Put them behind authentication, because robots.txt is a request, not access control.
—Gated content. Paywalled research, member pricing, and account-specific pages belong behind a login, not behind a robots.txt rule.
—A principled opt-out. Some brands decide their published work should not train third-party AI models, regardless of the visibility cost. That's a legitimate choice when it's made deliberately, not inherited from a default template.

What blocking changes, and what it doesn't

Restricting search crawlers can reduce how often your pages are discovered and cited. Blocking does not erase what models already hold about your brand, and it does not prevent citations drawn from other sources that describe you. Blocked requests may still appear in your CDN or firewall logs, so a blocked bot stays visible to you.

This is the gap ye5's GEO Web audit is built to close — what automated readers can and cannot reach. That way a block list matches a brand's intent, not a default no one reviewed.

How to block selectively

Blocking doesn't have to be all-or-nothing. robots.txt supports per-bot rules: disallow a training crawler by name while leaving a retrieval agent open, or the reverse (see: What should go in robots.txt for AI crawlers?). The effect of blocking training bots specifically on your AI visibility is a narrower question (see: Does blocking AI training bots hurt my visibility in AI search?).

FAQ

Does blocking GPTBot stop ChatGPT from citing my site?

No — GPTBot is OpenAI's training crawler, and blocking it opts you out of future model training. ChatGPT search runs on OAI-SearchBot, which is blocked independently of GPTBot. Live user-triggered fetching runs on ChatGPT-User, and robots.txt rules may not apply to it. A block also works forward only: it does not remove citations a model already produces from what it holds.

How do I know if I'm already blocking AI crawlers by accident?

Check your robots.txt file and your CDN or firewall's bot-management settings. Some default configurations block unknown bots, which can catch legitimate AI crawlers along with the ones you meant to stop.

Is blocking one AI crawler the same as opting out of AI search overall?

No — each AI system runs its own crawler and its own opt-out signal. Blocking ClaudeBot has no effect on Google's crawler, and blocking Googlebot has no effect on ChatGPT's. Each block is specific to one bot, not a platform-wide signal.