How detection works

​

The dark traffic problem

When someone clicks a link in ChatGPT or Perplexity, the browser typically does not send a Referer header to your site. The visit looks identical to a user who typed your URL directly. Standard analytics tools (including Google Analytics) record it as “direct” traffic. Studies of SaaS sites with high AI presence suggest that 60-75% of AI-referred visits arrive as dark traffic. Traceten is built to recover this attribution.

​

How a visit is identified

Traceten reads the signals a visit carries, such as the referring address, AI-specific UTM parameters and the user agent, and uses them to decide whether an AI assistant sent it. A missing referrer header does not end the check: a visit with no referrer can still be identified from the other signals it carries. The same check names sources that are definitely not AI. A visit referred by Instagram, Google, or Facebook is labelled by name rather than left unknown.

​

Which assistant gets the credit

AI hostnames are matched exactly, so a dedicated AI hostname always wins: gemini.google.com is attributed to Gemini, copilot.microsoft.com to Copilot, grok.x.ai to Grok. AI features built into a parent product’s main site are the exception. AI Overviews on www.google.com, Grok inside x.com, and the Copilot sidebar on www.bing.com send the same referrer as ordinary search and social traffic, so Traceten currently reports those visits as search or social rather than as AI.

​

Crawlers are a separate question

Everything above is about people: a visitor who read an AI assistant’s answer and clicked through to your site. AI crawlers are the other half of the picture. They fetch your pages directly, they run no JavaScript (so the snippet never sees them), and they identify themselves in the user agent rather than through a referrer. Traceten maintains a registry of those crawlers, and publishes it: the AI crawler directory has a page for each of the 60 crawler tokens we track across 21 providers. Each page gives the user agent, what the crawler is for, the exact robots.txt directive to allow or block it, and what it would take to verify that a request really came from that vendor. That last point is worth reading before you trust any user agent, and it does not divide neatly into “verifiable” and “not”. 34 of the 60 tokens have a machine-readable range file we fetch on a schedule, and for 29 of them that file is the vendor’s own. Those are the ones where an address that matches nothing is evidence the request is not the crawler it claims to be. The other five are Meta’s, and Meta publishes no crawler IP list at all: the only addresses available are BGP prefixes a third party reports Meta’s network announcing, which show where a request came from rather than who sent it. The directory says that on those five pages instead of calling them verified. Three more (You.com, ByteDance and Baidu) document a real check in prose rather than as a file: signed requests under Web Bot Auth, forward-confirmed reverse DNS, or a published address range written as text. That check is one you run at your edge, not one we have run for you, and the directory says so on those pages rather than filing them under “unverifiable”. For 20 crawlers a user agent is a claim and not a proof. One token, Microsoft’s msnbot, is a name Microsoft retired as a user agent in 2010, so anything sending it today is something other than a Bing crawler and no vendor tooling can tell you what. The last two of the 60 are robots.txt control tokens that never appear as a user agent at all. The directory says which is which per crawler instead of implying uniform verification.

​

What gets recorded

Every classification records an attributed AI source (or “Unknown AI” when a source can’t be identified), a confidence score from 0.0 to 1.0, and the detection method that produced it. You can inspect these fields in the Events view for any site.

​

What Traceten does not do

• We do not profile individual users. Detection reads where a visit came from, not who the visitor is.
• We do not share your site’s traffic data with other customers.
• We do not make claims beyond what the confidence score supports. A score of 0.72 means probable AI traffic, not certain.