AI VISIBILITY SUITE
ANSWERS
The common blockers are access and rendering problems: robots policy, WAF or CDN blocks, authentication, server errors on crawler requests, and content unavailable to a given renderer. Each one stops a system from reaching or reading your pages, and several can be live at the same time. ye5's GEO Web audit checks your site's access and readiness.
Five blocker types are worth checking: robots policy, WAF or CDN blocks, authentication, server errors on crawler requests, and content unavailable to a given renderer. These overlap with the blockers listed on (see: What's the first thing to fix for AI visibility?); this page explains what to check for each one.
| Blocker | What to check |
|---|---|
| Robots policy | Which agents are disallowed. GPTBot training restrictions do not block the separate search agents. |
| WAF or CDN blocks | Blocked requests appear in CDN or firewall logs, so a blocked bot is still visible to you. |
| Authentication | Staging and private content sit behind authentication. robots.txt is not access control. |
| Server errors on crawler requests | Rate-limiting or bot-detection rules can return 403 or 429 responses to AI crawler user agents. |
| Content unavailable to a given renderer | Facts that never appear in the initial HTML are missed by non-rendering fetchers. |
A missing or broken sitemap is a discovery gap, not an access block. Crawlers that rely on the sitemap to find pages find them more slowly.
Crawler and agent capabilities vary. Some fetch the initial HTML, others render JavaScript, so client-only content may be missed by non-rendering fetchers. The server sends a mostly empty HTML shell; the browser fills in content after the script runs. A person testing the site in a browser sees the finished page (see: Do AI crawlers execute JavaScript?).
This gap survives QA because everyone who tests the site uses a browser. No one on the team sees the page the way a crawler does, so the blocker ships unnoticed and stays live until someone checks.
One fast check, not a diagnosis: load your homepage with JavaScript turned off and see what renders. That screens for one problem only, and the systems you care about should be tested rather than assumed. The full checklist lives on (see: How do I make my website readable to AI crawlers?).
Robots policy, firewall rules, authentication, and server errors take longer to catch by hand, because they surface only on the specific pages or requests where they happen. ye5's GEO Web audit checks your site's access and readiness.
No. A robots policy states which agents you ask to stay out, and robots.txt is not access control. Rendering decides whether the facts appear in the initial HTML once a request goes through. Fixing one leaves the other in place.
Yes. A common pattern is client-only content on product pages paired with a WAF or CDN rule that blocks crawler requests. GPTBot training restrictions do not block the separate search agents, so a training directive is a different decision again.
Not usually. Rate-limiting or bot-detection rules built to stop scraping abuse can catch AI crawler user agents by accident, returning a 403 or 429 response instead of the page.