Measuring human traffic, AI referrals and agent requests separately

Six buckets, plus outcomes

We classify every request into one of six buckets, then attribute outcomes — leads, sales, bookings — back to them. The buckets differ in who is behind the request and in what unit they're counted, which is why they can't be summed.

Buckets 3 and 4 look alike in logs, but they mean different things. A crawler visits on its own schedule. A user-initiated fetch happens because a person asked an assistant something that needed your page — closer to demand, though still not a visit: no human saw your page. The providers also treat them differently: OpenAI says robots.txt rules “may not apply” to ChatGPT-User, Perplexity says its user fetcher “generally ignores robots.txt rules”, and Google says the same of its user-triggered fetchers.

Verify before you count

A user-agent string is a claim, not an identity. Each provider offers a way to check it:

• Published IP ranges. OpenAI publishes a JSON file per bot; Perplexity publishes ranges for both its bots; Anthropic's crawler page now links to a published list; Google publishes separate files for its crawlers, user-triggered fetchers and user-triggered agents.
• Reverse DNS. Google documents a reverse-then-forward DNS check for its crawlers.
• Cryptographic signatures. OpenAI's cloud browser signs requests with HTTP Message Signatures (RFC 9421), sending a Signature-Agent header of "https://chatgpt.com" and publishing its keys at a well-known URL. Google says it is experimenting with the same approach. The protocol around this, Web Bot Auth, is still an IETF Internet-Draft Experimental.
• Your CDN. Cloudflare's verified-bot classification accepts a Web Bot Auth signature, a published IP list with a stable user agent, or reverse DNS — so if you're behind it, use its classification rather than rebuilding it.

Ranges and keys rotate, so verification has to read the published lists on a schedule, not from a copy taken once.

Referrals: what the browser tells you, and what it doesn't

Bucket 2 is the hardest to measure honestly. When a person clicks a link in an assistant, your analytics sees at most two hints:

• The referrer. The Referrer Policy specification makes strict-origin-when-cross-origin the default, which sends only the origin on cross-site navigation. You'll learn the click came from an assistant's domain — never which conversation or prompt. A link copied out of an assistant and pasted into a browser carries no referrer at all and lands in “direct”.
• UTM parameters. Some assistants add a utm_source to outbound links. Segment on the values you actually see in your landing-page data. We haven't found provider documentation that guarantees them on every link, so treat them as a hint, not a census.

Keep a maintained list of assistant hostnames — for example chatgpt.com, perplexity.ai, claude.ai, gemini.google.com, copilot.microsoft.com — and review it against what appears in your data each month. Whatever you report, label it as a floor: real AI-assisted visits are higher than what you can attribute.

What the platforms report — and what they don't

• Google Search Console. AI Overviews and AI Mode are counted in the Performance report under the “Web” search type, mixed in with ordinary results. Google's dedicated Generative AI performance report, rolled out to all sites as of 31 August 2026, lists impressions, pages, countries, devices and dates; clicks aren't among them.
• Bing Webmaster Tools. The AI Performance report shows citations, cited pages and “grounding queries” across Copilot, Bing's AI summaries and some partners. Microsoft states it doesn't measure rankings or importance, and that the data is a sample.
• Everyone else. For ChatGPT, Claude, Perplexity and the rest, we know of no equivalent site-owner console. Referrals, logs and sampled prompt testing are what's left.

Note the units: impressions and citations are things that happened on someone else's screen. They belong next to your sessions and requests, not added to them.

A classification order

Rules overlap, so order matters. We apply these in sequence; the first match wins.

1. Valid HTTP message signature from a known agent operator → bucket 5, recorded by operator.
2. Request to an API with a valid credential → bucket 5, recorded by client.
3. Known bot user agent from a published IP range or verified reverse DNS → bucket 3 or 4, by the purpose the provider documents.
4. Bot-like or claimed-bot traffic that fails any of the above → bucket 6.
5. Everything else that behaves like a browser → bucket 1, then split by referrer and UTM into AI referrals (bucket 2), search, direct and the rest.

Outcomes are attributed last, and only from buckets where a person or their agent could have converted: 1, 2 and 5.

Worked example

Here is the monthly report this produces. The business and every number are invented for illustration.

Reporting rules we follow

• Every number carries its unit and its source.
• No composite “AI visibility score”. Trends are reported per bucket.
• Unverified traffic is shown, then excluded — never quietly added to a bucket it claims to belong to.
• Referral counts are labeled as floors. Sampled data (Bing's report, prompt testing) is labeled as a sample, with dates.
• Provider documentation changes often; classification rules are reviewed against it on a schedule, and the report says which version it used.

For the crawler side of this in more depth, see what AI crawlers actually do. For when agent traffic should lead to building something, see when a website should expose an API for AI agents.

Sources

1. OpenAI — Overview of OpenAI crawlers
2. OpenAI — ChatGPT Work's Cloud browser allowlisting
3. Anthropic — Does Anthropic crawl data from the web, and how can site owners block the crawler?
4. Perplexity — Perplexity crawlers
5. Google — Verifying Google requests
6. Google — Google user-triggered fetchers
7. Google — AI features and your website
8. Google Search Central Blog — Introducing Search Generative AI performance reports in Search Console
9. Microsoft — AI Performance in Bing Webmaster Tools
10. Cloudflare — Verified bots
11. IETF — RFC 9421: HTTP Message Signatures
12. IETF — draft-ietf-webbotauth-httpsig-protocol (Internet-Draft)
13. W3C — Referrer Policy (Editor's Draft)