See which AI bots crawl your website, free

Why your analytics never showed them

Crawlers do not run JavaScript, so a tracking script never sees GPTBot or ClaudeBot fetching your pages. Cloudflare does: it sits in front of every request to a proxied site and knows which bots it has verified. Totallytics reads that log with a read-only token and turns it into a report per bot, page and status code, so you can see whether ChatGPT's search crawler has read your new page, which pages Perplexity fetches, and which bots are getting blocked.

The bots you see by name

• OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
• Anthropic: ClaudeBot, Claude-SearchBot, Claude-User
• Perplexity: PerplexityBot, Perplexity-User
• Google: Googlebot (including Image, Video and News), GoogleOther
• Microsoft: Bingbot
• Apple: Applebot
• Meta: meta-externalagent, meta-externalfetcher
• Others: Amazonbot, Bytespider, CCBot, DuckDuckBot, DuckAssistBot, MistralAI-User, ExaSearchBot, Baiduspider

That is 22 named bots. Any other bot Cloudflare has verified appears under its own user-agent name, and SEO tools, webhooks, link previews and security scanners are left out so the report stays about search and AI.

Verified means verified

A hit counts as verified when Cloudflare has verified the bot in its AI crawler, AI search, AI assistant or search engine categories, or when the request came from the company's published IP ranges (OpenAI, Perplexity, Google, Microsoft, Apple, Common Crawl, DuckDuckGo). The IP check runs during sync and IP addresses are never stored. Requests that merely claim to be a known bot are counted as unverified, so a scraper pretending to be Googlebot does not inflate your numbers.

What the report shows

Four totals against the previous period: AI crawler hits, search engine hits, pages crawled and unverified hits. A Bots panel with verified hits per bot, when each was last seen, how many requests were blocked with 401, 403 or 429, and the unverified count. A Pages panel with the 100 most crawled paths and how many distinct bots fetched each. A Responses panel with status codes, so a page returning 403 to OAI-SearchBot stands out. Hit counts are Cloudflare's sample-adjusted estimates at one-hour granularity.

Setup without touching your site

In the site settings, create a Cloudflare API token with Zone Read and Analytics Read on all zones, paste it, and Totallytics checks it, stores it encrypted and shows only its last 4 characters. One token covers every site in your account. The last 30 days are imported a week at a time, then a sync runs every 15 minutes and re-reads the previous hour for late data; a Sync now button is there when you cannot wait. The site has to be verified in Totallytics and proxied through Cloudflare, on any Cloudflare plan including Free.

Stop tracking on a site deletes that site's crawler history; disconnecting the token stops tracking on every site and deletes all of it. Tracking again imports the last 30 days afresh.

Crawlers are not visitors

This page is about bots fetching your pages. People who click a link inside ChatGPT, Perplexity or Claude are regular visitors, tracked by the script and shown as their own source. That is the AI traffic analytics page.

Limits

FAQ

My site is not on Cloudflare. Can I still see AI crawlers?

No. The data comes from Cloudflare's request log, so the site has to be proxied through Cloudflare. The crawler stats are the only Totallytics feature with that requirement.

Do I need to add code to my site?

No. The tracking script is for visitors; crawler stats come entirely from the Cloudflare token.

Are the hit counts exact?

They are Cloudflare's sample-adjusted estimates at one-hour granularity, which is what Cloudflare's own analytics show.

Can I block a bot from Totallytics?

No. Totallytics reads the log; blocking is done in Cloudflare or robots.txt. The Responses panel shows you whether a block is working.