AI crawler tracking

What this lets you do

See which AI systems fetch your pages, which pages they read, and whether the crawler was really who it claimed to be. AI crawlers do not run JavaScript. They issue one HTTP GET for your HTML and leave, so the Traceten snippet never sees them. The only place a crawl is observable is your server. This guide installs a small middleware there. This is separate from, and additional to, the browser snippet. Keep both: the snippet measures people who arrive from an AI assistant, this measures the crawl that put you in that assistant’s answer.

Before you start

A site already added in Traceten
Node 18 or later, or an edge runtime with fetch
Admin access to your Traceten account, covering the whole account rather than a single site (only admins can create crawl tokens, see Team and roles)

1. Create a crawl token

In the dashboard, open Settings → API keys and select the AI crawler API key tab. Crawl tokens belong to a single site, so check that the site named under Crawler tokens is the one you are installing on. If your account has more than one site, switch with the site picker beside Create token. Then select Create token, give the token a name (your environment name is a good choice) and select Create.

The token is shown once, at creation. Traceten stores a one-way hash of it and cannot show or recover it afterwards. Copy it before you leave the page. If you lose it, revoke it and create a new one.

You also need your site key, shown under the new token and in the Install the middleware section. It is the same public value the snippet uses as data-site. Store both as environment variables:

TRACETEN_SITE_ID="ttid_7Rb4TrC1dTbnD8w3s1TS12"
TRACETEN_CRAWL_TOKEN="tt_bot_9f2c41a7be08d35c6114ea7d20b98f43ac5e6710d92b4f8e3c07a1d6b5f2e934"

The site key is public. The crawl token is a server-side secret: never put it in browser code, and never use the site key in its place. You can hold up to five active tokens per site, which is what makes rotation safe. One token reports for exactly one site: a token minted for site A is rejected outright if it reports for site B.

Rotating a token

There is deliberately no single rotate button. Pressing it would break reporting the instant you did, because your deployed server still holds the old secret. Rotate in this order:

Create the replacement. Both tokens are live at once.
Deploy it. Confirm crawls are still arriving.
Revoke the old one, from the token list.

Revocation is immediate in our database and takes effect at the ingestion edge within 60 seconds, so a revoked token can still be accepted for up to a minute. If you believe a token has leaked, revoke it first and investigate second: the worst case for a leaked crawl token is fabricated crawl records on that one site. It grants no read access, no access to any other site, and no access to your account.

2. Install the package

npm install @traceten/ai-crawl

3. Add the middleware

Pick the one that matches your stack. If your site is already behind Cloudflare, use one of the two Cloudflare options. The tracking runs in front of your origin rather than inside your app, and the crawler’s address is already correct with no proxy configuration to get right. With Cloudflare Workers your application itself never changes: you wrap the Worker that already fronts your origin and deploy that. Cloudflare Pages Functions does need a deploy, because the middleware ships as a file in your project.

import { defineAiCrawlConfig } from "@traceten/ai-crawl";
import { withAICrawlerTracking } from "@traceten/ai-crawl/cloudflare-workers";

const config = defineAiCrawlConfig({
  siteId: "ttid_7Rb4TrC1dTbnD8w3s1TS12",
  authToken: "tt_bot_9f2c41a7be08d35c6114ea7d20b98f43ac5e6710d92b4f8e3c07a1d6b5f2e934",
});

export default {
  fetch: withAICrawlerTracking(async (request, env, ctx) => {
    return fetch(request); // your existing origin logic
  }, config),
};

// functions/_middleware.ts
import { createAICrawlerPagesMiddleware } from "@traceten/ai-crawl/cloudflare-pages";

export const onRequest = createAICrawlerPagesMiddleware({
  siteId: "ttid_7Rb4TrC1dTbnD8w3s1TS12",
  authToken: "tt_bot_9f2c41a7be08d35c6114ea7d20b98f43ac5e6710d92b4f8e3c07a1d6b5f2e934",
});

// proxy.ts (Next 15.5+; middleware.ts on earlier versions)
import { NextResponse, type NextFetchEvent, type NextRequest } from "next/server";
import { defineAiCrawlConfig } from "@traceten/ai-crawl";
import { trackAICrawlerRequest } from "@traceten/ai-crawl/next";

const config = defineAiCrawlConfig({
  siteId: process.env.TRACETEN_SITE_ID!,
  authToken: process.env.TRACETEN_CRAWL_TOKEN!,
});

export function proxy(request: NextRequest, event: NextFetchEvent) {
  trackAICrawlerRequest(request, event, config);
  return NextResponse.next();
}

import express from "express";
import { createTracetenAICrawlerMiddleware } from "@traceten/ai-crawl/express";

const app = express();

app.use(
  createTracetenAICrawlerMiddleware({
    siteId: process.env.TRACETEN_SITE_ID!,
    authToken: process.env.TRACETEN_CRAWL_TOKEN!,
    // Set this. Without it the reported URL is built from the Host header,
    // which a client can set to anything.
    publicOrigin: "https://www.example.com",
    // Uncomment ONLY if a proxy, load balancer or CDN sits in front of this
    // process. On a directly-exposed origin it lets any client choose the
    // address you report. See "Getting the crawler's IP right" below.
    // trustProxy: true,
  }),
);

import { Hono } from "hono";
import { aiCrawlerTracking } from "@traceten/ai-crawl/hono";

const app = new Hono();

app.use(
  "*",
  aiCrawlerTracking({
    siteId: process.env.TRACETEN_SITE_ID!,
    authToken: process.env.TRACETEN_CRAWL_TOKEN!,
  }),
);

Any other framework that exposes fetch-API Request and Response objects can call trackAICrawlerFetch(request, response, config, waitUntil?) from @traceten/ai-crawl/hono. That function requires a config you have already passed through defineAiCrawlConfig; hand it a plain object and it returns without reporting anything. Invalid config fails at startup, not during a request, with one exception. defineAiCrawlConfig throws on invalid input, and the Cloudflare Pages, Cloudflare Workers, Express and Hono factories all validate at construction whether or not you called it yourself, so a missing or malformed token throws while your server is starting. That is the failure you want: it fails your deploy rather than your visitors’ requests. The exception is the Next.js trackAICrawlerRequest, which runs per request and therefore must never throw into one. It validates lazily instead, and invalid config logs [@traceten/ai-crawl] invalid config — crawler tracking is DISABLED once and then reports nothing at all. On Next.js, call defineAiCrawlConfig yourself at module scope, as the example does, so the failure is loud.

Getting the crawler’s IP right

Verification is only as good as the address your server reports. If it reports your own load balancer, nothing matches, every crawl comes back unverified, and no error appears anywhere. This is the single most common way this install goes quietly wrong.

Option	Default	When to set it
`trustProxy`	`false`	Your app is behind a reverse proxy, load balancer, or CDN that sets `x-forwarded-for`. Off by default because that header is trivially forged.
`proxyDepth`	`1`	How many trusted proxies append to `x-forwarded-for`. Two proxies in front of you means `2`.
`trustCfConnectingIp`	`false`	Your origin is behind Cloudflare but you are not using the Cloudflare adapters. Those two set it for you.
`publicOrigin`	none	Your app sees an internal hostname (containers, private networking) and reported URLs would otherwise carry it. The package auto-recovers the common case where the runtime resolves the request against its own bind address (`0.0.0.0`, `[::]`) using the incoming `Host` header — set `publicOrigin` when your `Host` header itself carries an internal or wrong hostname instead.

Resolution order is cf-connecting-ip (only when trusted), then x-forwarded-for (only when trustProxy is set, counting back proxyDepth entries from the right), then the socket address. If no trustworthy value can be derived, the package sends no address at all rather than guessing one.

Do not set trustProxy when nothing is actually in front of your server. Any client can send an x-forwarded-for header, so trusting it on a directly-exposed origin lets a caller choose the address you report to us.

4. Deploy and verify it worked

Deploy, then confirm the middleware is reporting:

curl -A "Mozilla/5.0 (compatible; GPTBot/1.1; +https://openai.com/gptbot)" \
  https://www.example.com/

Open AI Crawlers in the dashboard. Within a minute or two you should see that crawl. It will not come back verified, because you are not OpenAI. Depending on what could be checked at the time, it lands as Spoofed (your address is outside OpenAI’s published ranges) or as Unverified. Either way, seeing your own test arrive is proof that reporting works end to end. Real crawls arrive on their own schedule. If your site is new to AI crawlers it can be days before the first genuine one.

Troubleshooting

Nothing appears at all. Check the token is set in the deployed environment, not only locally. An absent token throws at startup on every adapter except Next.js, so check your boot logs first; on Next.js look for [@traceten/ai-crawl] invalid config. A token that is present but revoked, or issued for a different site, is rejected at our edge instead, and the report is dropped silently by design: the middleware never breaks a request to tell you. Set onError to see those rejections.

Seeing rejected reports

The middleware never throws and never retries, so a rejected report looks the same as a delivered one. A wrong token, or egress your platform blocks, both read as “no AI crawlers visited” for as long as they last. onError reports what happened without changing that behaviour:

export const aiCrawl = defineAiCrawlConfig({
  siteId: "ttid_7Rb4TrC1dTbnD8w3s1TS12",
  authToken: process.env.TRACETEN_BOT_TOKEN,
  onError: (e) => {
    console.warn("[traceten] crawl report failed", e.kind, e.status ?? e.cause);
  },
});

e.kind is "http" when our edge answered, with the status in e.status. A 401 or 403 means the token is wrong, revoked, or issued for another site. e.kind is "network" when the request never completed, with the thrown value in e.cause; repeated network failures usually mean your platform blocks outbound requests. Your callback runs inside a try/catch. Throwing from it cannot break the response, so the guarantee that this middleware never affects your site still holds. Wire it to your logger in staging at minimum. “Site not found” or nothing lands despite a 200. Check siteId is the site key from the dashboard, not the site’s UUID from your browser’s address bar. They are different values and the wrong one is rejected at the edge. Everything shows as Unverified or Spoofed. Your server is probably behind a proxy or CDN and reporting the proxy’s IP rather than the crawler’s. See Getting the crawler’s IP right. Note that a large Unverified share on its own is normal and not a fault: about half of tracked crawlers publish no way to prove their identity. See “Unverified” is normal. URLs look wrong, or use an internal hostname. If the reported URL shows 0.0.0.0 or [::], that’s the server’s own bind address — the package already falls back to your Host header in that case, so check your reverse proxy is forwarding Host at all. If URLs instead carry a different internal hostname (containers, private networking, Host itself set to something internal), set publicOrigin to your real origin. Crawls of /robots.txt or /llms.txt are missing. They should not be. Those paths are always reported even though .txt is otherwise filtered out. If they are absent, the middleware is likely mounted after a route that serves those files directly. Too much traffic reported. Use disableTrainingCrawlers, disableSearchCrawlers, disableAnswerFetch or disableOtherCrawlers to skip whole categories.

What gets sent

For each matched crawl the middleware sends the request URL, method, response status, the crawler’s user agent, a timestamp, and the crawler’s IP address. It never sends cookies, request bodies, or visitor analytics. The IP is required. A crawler’s connection terminates at your server, so its address is observable only there, and it is what lets Traceten check the crawl against the evidence the vendor publishes. Traceten treats it as a claim to be verified, never as proof on its own. Where the published-range check does not match and the crawler’s claimed vendor supports reverse-DNS verification, the address is sent to Cloudflare’s public DNS resolver as part of that lookup. This happens before it is verified, on failed attempts as well as successful ones, unless a recent identical lookup is still cached. Retention depends on the verdict. Traceten keeps the raw address for 90 days in exactly two cases: it fell inside an IP range the vendor publishes for that specific crawler, or its reverse DNS forward-confirmed to the vendor’s own domain. In every other case, including where the only evidence is that the address sits somewhere in the vendor’s wider network, it is hashed (HMAC-SHA-256 under a per-site key) and the address is discarded. No network operator or country is derived from a crawler IP. Individual crawl records are deleted after 90 days. The aggregate counts behind the charts are kept indefinitely and contain no IP data. The full field list, and what happens above your plan’s crawl allowance (which differs between the free plan and the paid plans), are in crawl data and retention. Crawl hits are metered separately from your event allowance and never count against it.

What it costs your server

The trade is worth stating in numbers, because this runs on every request:

Requests that are not crawls, which is nearly all of them: a method check, then a lowercase path compared against the deny-lists, and only if that passes, the user agent lowercased once and searched for the crawler tokens: 58 exact tokens, then 28 provider prefixes for anything unrecognised. All of it is in-process string work: no network call, no I/O, nothing awaited. Most requests never reach the user-agent stage at all, because assets, API routes and browser subresource fetches are rejected by the path and header checks first.
Requests that are crawls: one POST to Traceten, scheduled after your response through waitUntil where the runtime provides one, bounded by a 1500 ms timeout. There is no batching; one crawl is one report, and crawl volume sits orders of magnitude below pageview volume because of the pre-filter.
Failures: silent. The package never throws per-request and never blocks a response. If Traceten is unreachable, your site does not notice.
Bundle size: every adapter bundle is under 8 KB gzipped, enforced in CI.

Getting started

Install guide

AI crawlers

AI visibility

Dashboard

Goals and funnels

Integrations

Privacy

AI crawler tracking

What this lets you do

Before you start

1. Create a crawl token

Rotating a token

2. Install the package

3. Add the middleware

Getting the crawler’s IP right

4. Deploy and verify it worked

Troubleshooting

Seeing rejected reports

What gets sent

What it costs your server

Next

​What this lets you do

​Before you start

​1. Create a crawl token

​Rotating a token

​2. Install the package

​3. Add the middleware

​Getting the crawler’s IP right

​4. Deploy and verify it worked

​Troubleshooting

​Seeing rejected reports

​What gets sent

​What it costs your server

​Next

What this lets you do

Before you start

1. Create a crawl token

Rotating a token

2. Install the package

3. Add the middleware

Getting the crawler’s IP right

4. Deploy and verify it worked

Troubleshooting

Seeing rejected reports

What gets sent

What it costs your server

Next