Skip to main content

​
What this lets you do

Answer, precisely, what leaves your server when you install AI crawler tracking, and what Traceten holds afterwards. Crawl records are records of automated software requests. They are not visits by people, and nothing on this path touches your visitor analytics.

​
What your server sends

One report per matched crawl, containing exactly these fields: That is the whole payload. No cookies, no request bodies, no headers beyond the user agent, no visitor analytics, and nothing about the human traffic on the same server.
Where the query string is removed matters. The middleware sends the URL as your server saw it, query string included; we strip it at our edge before the crawl is written anywhere or placed on any internal queue. This is different from visitor events, where the browser snippet redacts the values of personal-data parameters before anything leaves the browser and our edge applies the same redaction again. If a crawled URL carries a token in its query string, that token reaches our edge in memory. If that matters for your site, exclude the path with extraDenyPathPrefixes.

​
Why the user agent is stored raw

We keep the full claimed user-agent string, unmodified. It is the only record of what the client actually said about itself, and it is what makes a spoofing claim auditable rather than something you have to take our word for. Truncating or normalising it would destroy the evidence. Worth being explicit about the edge case: about half of tracked crawlers cannot be verified at all, so a small share of records in that bucket may not be crawlers. If something with an ordinary browser user agent is misidentified, the string we stored is a browser string. It is still tied to a machine request rather than a session or a person, it carries no visitor identifier, and it expires with the record after 90 days.

​
What happens to the crawler’s IP address

The crawler connects to your server, not to ours, so its address is observable only there. We use it for one purpose: checking whether the crawler was who it claimed to be. We keep the raw address in exactly two cases:
  • It fell inside a range the vendor publishes for that specific crawler. That is infrastructure the vendor publishes as its own.
  • Its reverse DNS forward-confirmed to the vendor’s own domain.
Everything else is hash-only. We discard the address and keep an HMAC-SHA-256 hash of it, computed under a key derived per site. That explicitly includes the case where the address sits somewhere in the provider’s wider network but is not on a published crawler range: that evidence is enough to record a weaker verification result, and not enough to keep the address, so we discard it and keep only the hash. Those hash-only records are exactly that: the hash and nothing else. We do not resolve or store the network operator (ASN) or the country for a crawler address. Reverse-DNS verification sends the address to Cloudflare’s public DNS resolver, before anything about that address is known, on failed attempts as well as successful ones. The full description is in the privacy notes and the sub-processors page.

​
How long we keep it

The 90-day limit is enforced by the database itself rather than by a scheduled job, and it is not a plan feature: there is no tier that keeps individual crawl records for longer. Removal runs as part of the database’s background maintenance, so a record is deleted at 90 days or shortly after, never before. Your long-term trend survives that expiry. The daily rollups are three separate tables of counts: one grouped by crawler, provider, category, verification state and response class, one grouped by page path, and one grouped by page path and crawler. They carry no address, no hash, and no personal identifiers, which is why they can be kept indefinitely. So a year from now you can still see that GPTBot crawled you 40,000 times last quarter; you cannot see the individual 40,000 requests. Be aware of what the two page path rollups keep: the page paths themselves are retained indefinitely, as daily per-path counts (one of them also per crawler), after the individual crawl records containing them have expired. Query strings are already gone by then, and a path with no requester attached to it is not personal data, but it is site data about your URL structure and it does not age out. Closing your account erases all three, for every site on the account. One exception, so the sentence above is not read wider than it is: the metered usage counts behind your invoices (how many crawl hits were recorded in each billing period, and how many of those were billed as overage) are kept as billing records. They are counts, with no URLs, no paths and no crawler identifiers, and we are required to retain them for tax and accounting purposes. See Deleting your account.

​
If you go over your plan’s crawl allowance

What happens depends on whether your plan offers pay-as-you-go crawl overage. Your usage page shows which state you are in, and the API reports it as crawl_allowance_state.

​
On a paid plan

Nothing is rejected and nothing is lost from your totals.
  • Your counts stay exact. Every crawl continues to be counted into the daily rollups, so every chart, total, category split and per-page number remains correct and complete.
  • Verification is reduced. The reverse-DNS check, a live lookup we run per crawl to confirm a claimed crawler really is who it says, is skipped. Affected crawls settle on a lower verification level and are recorded at lower confidence.
  • Per-crawl detail pauses. Individual crawl records stop being written for the rest of the period, so the recent-crawls feed and anything that reads a single request go quiet.
This is a deliberate degrade rather than a limit being hit. The design goal is that the number you report to someone else is never wrong because of billing.

​
On the free Starter plan

Starter has no pay-as-you-go crawl overage, so there is no way to keep serving the volume and bill for it. Measurement pauses at the limit instead.
  • New crawl hits stop being recorded once the cap takes effect, for the rest of the billing period. The daily rollups stop advancing, so your totals stop rising. Enforcement is checked every few minutes, so a small number of hits just past the limit may still be counted, in your favour, and never billed.
  • Everything captured before the limit stays available, and stays accurate. Nothing already measured is deleted or altered.
  • Your allowance resets at the start of your next billing period, and recording resumes automatically.
Upgrading to a paid plan resumes recording within a few minutes and gives you the degrade behaviour above instead.

​
Keeping full detail

Turning on AI crawl overage in Settings → Billing keeps everything above running at full fidelity past your allowance: the reverse-DNS check, the verification level and the per-crawl records. The excess hits are billed per hit. It is a separate switch from event overage; turning one on does not affect the other. See Plans and billing. On a paid plan, leaving it off costs nothing and loses no counts. It only costs detail.

​
Data subject requests

Traceten’s per-visitor access and deletion API does not cover crawl records, and this is deliberate rather than a gap. See Data deletion for the reasoning and for what to do if you receive a request that you believe reaches crawl data.

​
Next