ANSWERS

How do I check which AI bots visit my site?

AI bots identify themselves in the user-agent string of each request. Search your server or CDN logs for known names: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, and PerplexityBot. Many hosts also expose a verified-bot signal that labels confirmed crawlers. ye5's GEO Web audit checks what these bots can reach on your site.

AI bots name themselves in the user-agent string

Each request to your site carries a user-agent header. Reputable AI crawlers publish their names in it (see: What is GPTBot and what does it do with my content?). A handful of names cover most AI bot traffic:

—GPTBot — OpenAI's crawler for potential model training
—OAI-SearchBot — OpenAI's crawler for ChatGPT search
—ChatGPT-User — OpenAI's user-triggered fetcher
—ClaudeBot — Anthropic's crawler for potential model training (see: What is ClaudeBot?)
—Claude-SearchBot — Anthropic's crawler for Claude search
—Claude-User — Anthropic's user-triggered fetcher
—PerplexityBot — Perplexity's crawler

These agents do different jobs. Search crawlers build an index for answers with citations, training crawlers collect text that may feed a future model, and user-triggered fetchers pull one page because a person asked for it in a conversation. Google-Extended is a robots.txt permission token, not a request agent, so it never appears in your logs as a user-agent. Googlebot requests come from Google's search crawl and do not prove Gemini used your pages. A crawler that hides its identity or spoofs a browser string won't show up this way. A user-agent check tells you about traffic that names itself, not traffic that hides.

Your logs live in your hosting or CDN dashboard

Server and CDN logs are the source of truth for bot traffic. Page-view analytics tools usually miss it, because client-side analytics do not reliably record crawler activity. Common places to look:

—Your host's log viewer or observability panel, if it offers one
—Raw access logs on the server itself — Apache, Nginx, or an equivalent
—CDN-level logs and analytics — Cloudflare, Fastly, or your provider's equivalent
—Cloud logging tools tied to your infrastructure — CloudWatch, Cloud Logging, or similar

Where you look depends on your stack. The constant across each option: you need the raw list of requests, with each request's user-agent string intact.

Filtering by user-agent surfaces the pattern

Once you have access to raw logs, search or filter for the bot names above. Two approaches work:

—Text search — grep or search the raw log file for each bot name directly
—Dashboard filters — many hosting and CDN dashboards let you filter requests by user-agent without touching a raw file

Read each matching log line for four fields: timestamp, requested path, status, and user-agent. Together they show when the agent came, what it asked for, and whether your server answered. Check your CDN or WAF logs for blocked requests as appropriate, since a bot stopped at the edge may never reach the origin log at all.

A user-agent alone can be spoofed. Use the provider's verification method where one is available, or a verified-bot signal from your host or CDN, before you treat a named string as that crawler.

Visiting is not the same as reaching your content

A request does not prove readership, training, indexing, or citation. A robots.txt or sitemap.xml request proves least of all: the agent asked for a file about your site, not for your content. Knowing which bots visit is half the picture. The other half is what they can retrieve once they arrive. Three things commonly block that:

—Pages disallowed in robots.txt (see: What should go in robots.txt for AI crawlers?)
—Content that renders only after JavaScript runs
—Pages buried too deep in the site structure to get crawled

ye5's GEO Web audit reviews what these bots can reach and flags the templates blocking them. It checks access and readiness rather than replaying past visits. One fix scales across the pattern instead of pages getting fixed one at a time.

FAQ

Do bots identify themselves honestly?

No bot is required to. Reputable crawlers publish accurate user-agent strings; bots designed to scrape without permission typically will not. A log check tells you about traffic naming itself as a bot — it doesn't catch traffic hiding that identity.

Does my website analytics tool already show this?

Client-side analytics do not reliably record crawler activity, so page-view reports miss AI crawlers. Server or CDN logs record each request at the network level, capturing traffic invisible to script-based analytics.

What if my host doesn't expose raw logs?

Ask your host or CDN whether it offers a bot-traffic or observability panel; several providers label verified crawlers by name. If neither raw logs nor a labeled panel exists, that access gap is worth flagging before you invest further in AI visibility work.