Base URL and authentication
Authorization: Bearer <YOUR_API_KEY> (details) and the stats:read permission.
Crawler traffic is a separate population from visitor traffic. A crawl is a machine fetching a page to index or answer with it. Nothing here appears in your session, visitor or revenue figures, and nothing from those endpoints appears here.
What “verified” means, before the endpoints
Every crawl records how strongly its identity could be checked. The API returns the rung as a string; the dashboard shows the same value as a badge, and Verification levels is the full definition of each one.
Two consequences shape these responses, so they are worth repeating here.
user_agent_only is normally the largest bucket. About half the crawlers we track publish no IP ranges and document no reverse-DNS convention, so there is nothing to check them against and no tool can verify them. A large unverified share is the truthful state of the ecosystem, not a fault in your setup. That is why the summary returns a breakdown by rung and never a single verified percentage.
“No spoofing detected” does not mean everything was checked and passed. Spoofed is reachable two ways: an IP mismatch, which needs the vendor to publish ranges, and a live request carrying a robots.txt permission token, which no crawler ever sends. For a crawler that publishes nothing and sends no permission token, neither route applies and nothing can be confirmed either way. Read a zero in spoofed as “nothing was proven”, not as “nobody impersonated you”.
GET /v1/ai-crawls/summary
Totals and the verification breakdown for a window.
has_crawls: false is the distinguishable empty state. When it is false, verification_breakdown is an empty array rather than a list of zeroes that would imply a 0 percent verified rate.
crawl_allowance_state says what your account’s AI-crawl allowance is doing, so you can tell an empty window apart from a stopped one. under and metered are full fidelity. degraded means these totals are still exact but individual crawl records are not being kept. capped means new crawl hits are no longer recorded at all, so has_crawls: false there does not mean no crawlers came. See the account usage endpoint for the full table.
GET /v1/ai-crawls/by-page
Two views on one endpoint, told apart by mode in the response.
Without path, you get the top crawled paths (mode: "ranking", page_detail: null). With path, you get one page’s detail (mode: "page_detail", pages: null). The unused field is null rather than an empty array, so an empty result and the wrong mode are never confusable.
Parameters
verification=genuine means not spoofed and actually graded. It exists because writing the predicate yourself as “not spoofed” silently includes every ungraded row.
Sending verification or limit together with path is a 422. The drill-down ranks nothing, so there is no ranking for those to apply to, and accepting them would mean returning a number that ignored a filter you sent.
Filters run on the server, and that is load bearing
verification, category and provider filter and re-rank in SQL. If you filter the returned rows yourself instead, you keep the rows we chose under a different ordering: correct numbers attached to the wrong pages.
The same applies to category. Under a category filter, the per-row categories split describes the filtered rows only, so the three unselected buckets read 0 and crawl_count counts the selected category. That is the honest reading, because every number on the row is then about the same rows. To see the full split, ask without the filter.
provider also drops spoofed rows, deliberately
In ranking mode, filtering by provider excludes spoofed crawls. A spoofed row carries the impersonated company’s name, so a plain provider match would print an impersonator’s volume under the real company’s brand. Expect verification.spoofed to read 0 on every row of a provider-filtered response. That is the filter working.
To ask “who is pretending to be this company”, send provider together with verification=spoofed.
Page detail
page_detail carries total_crawls, genuine_crawls, a providers list and a sparse daily series. Providers are bucketed by rung, so a spoofed claim is a separate row from the same company’s genuine traffic. genuine_crawls excludes both spoofed and ungraded rows; it is the figure to print next to a company’s name.
A path nobody crawled returns 200 with total_crawls: 0. That is not an error.
GET /v1/ai-crawls/answer-map
The pages AI companies fetch to answer a live question, ranked.
This endpoint reads one purpose only, answer_fetch: a crawler fetching a page because a person is asking something right now. It is the closest thing in the data to real demand, and it is a much smaller population than total crawling, most of which is training.
Parameters
These counts are lower than by-page, and that is the filter working
This endpoint counts only crawls the network proved: web_bot_auth, ip_range, reverse_dns and asn_match. by-page counts every rung, including user_agent_only, which is usually the largest bucket.
That difference is deliberate. The ungraded bucket is where scanner traffic sits. Requests for /.env, /wp-login.php and /xmlrpc.php reach this data like any other, and a fake GPTBot is only gradeable as spoofed when the vendor it claims publishes IP ranges. For a vendor that publishes nothing, the same request lands in user_agent_only. Counting it here would print a scanner as an assistant reading your page.
asn_match is the weakest of the four. It matches the network operator rather than a published crawler fleet, so it is reported as its own number in verification_tiers rather than folded into a total. None of this verification is cryptographic.
concentration is null below the threshold
concentration names the single most-fetched page and its share of total_answer_fetches. It is null when the window holds fewer than min_crawls_for_concentration verified fetches, which means there is too little data to generalise. It does not mean there is no concentration, and it is not 0%.
The share is computed against every path in the window, not against the pages returned. A smaller limit returns fewer rows and does not change the share.
GET /v1/ai-crawls/reads-vs-visits
The same pages as the answer map, with what the reading actually brought.
Never combine the two columns
answer_fetch_crawls is a count of hits. ai_sessions is a count of visits. They come from different tables, at different grains, and they are returned side by side because both are true, not because they are two halves of one number.
Do not add them. Do not divide one by the other, and do not present the result as a conversion rate. Reads outnumber visits by orders of magnitude: Cloudflare measured Anthropic at 38,066 crawls per referral in July 2025. A ratio built from these two columns measures nothing.
A page read often and visited never is a normal state. An assistant can answer someone’s question from your page without sending them to it. min_reads_for_no_visit_tag is the number of reads we think is enough to be worth pointing out; below it the sample is too small to mean anything.
Revenue is absent, not zero, past the retention floor
ai_visitors and ai_revenue_usd_cents are null when landing_revenue.available is false. That happens when your window reaches further back than session-grain retention, whose boundary is in floor_date. Render a dash. A zero would be a measurement, and this is the absence of one.
When revenue is available, these two are entry-page grain: they count sessions that STARTED on the page. ai_sessions counts the page as viewed at any point in a session. They are different populations, so a page can have sessions and no visitors.
crawls_host_blind is always true
The crawl data stores a path and no hostname. If your site serves several hostnames, every host’s crawls of /pricing are folded into one row, while the traffic side is counted per URL. Multi-hostname sites therefore overstate the crawl column. / is the sharpest case, because every hostname has one.
Two paths that differ only by a trailing slash are different paths on the crawl side, because your middleware reports the path it received.
GET /v1/ai-crawls/fix-list
Pages where a real AI crawler could not read what it asked for.
Failures are ordered by purpose first (answer_fetch, then search_index, then training), then by volume, because a crawler failing to answer someone’s question right now matters more than a missed training fetch.
Parameters
An empty list is not “nothing is wrong”
Readcoverage every time, before you conclude anything from failures.
Read
status before observed_crawls: under no_crawls the count is 0 because there was nothing to count, which says nothing about your install.
observed_crawls: 0 with crawls present is the common one, and it is not a fault in your site. Next.js middleware runs before the response exists, so it never sees a status code, and every Next.js install using the middleware adapter reads zero here. AI crawler tracking lists which adapters can report a status and what to do about it.
rollup_crawls below total_crawls is partial. Per-page status history begins when this feature was deployed, plus a backfill limited by the 90-day raw retention, so a long window can legitimately hold failures we cannot show.
include_unverified widens the list to scanner territory
By default the list carries only crawls the network proved. Turning include_unverified on admits crawls that carried no proof, and those rows come back with unverified: true.
That population is mostly scanners. About half of tracked crawlers publish no IP ranges, so nothing distinguishes a real crawler from something claiming to be one, and requests for paths like /.env sit in exactly this bucket. Never describe an unverified: true row as a real AI crawler being blocked.
Proven impersonators are excluded from both modes. An attacker being refused is not a problem to fix.
blocked versus the other failures
blocked is true for 401 and 403 only: a verified AI crawler was refused. That usually points at a firewall rule, a CDN bot rule or a WAF, not at your application. Every other status is a real failure with a different cause, so blocked stays false for a 404, a redirect loop or a 500.
status_code is the raw code, never a class, so 403 stays distinguishable from 404. It is always 300 or above. A 2xx is not a failure, and a crawl whose status could not be observed is reported through coverage instead of appearing here as a zero.
GET /v1/ai-crawls/timeline
Daily crawl volume, split by by.
by=category (default) returns rows carrying category. by=provider returns rows carrying provider and is_other, covering the top limit companies plus one roll-up row per day.
Each mode takes only the parameters it can use. provider narrows a by=category series to one company, and is a 422 under by=provider, which groups by company already. limit bounds the company list under by=provider, and is a 422 under by=category, which returns every category and has no top N to bound. The alternative was accepting them and dropping them, which returns the wrong scope’s numbers with a 200 beside them.
is_other, never on provider == "other". “Other” is a plausible company name, and unlike the category taxonomy the provider registry is not a closed set.
GET /v1/ai-crawls/series
A ranked daily series. dimension=provider gives a line per company, dimension=path a line per page.
path and provider combine, and they combine on the server. They have to: this response is a top N chosen by an ordering, so narrowing it in your own code keeps rows that were selected under a different predicate.
measure=spoofed ranks by impersonation volume instead of crawl volume. It is a server parameter rather than a client-side re-sort because the two measures name different companies: the most impersonated brand may send very little genuine traffic and would fall outside a crawls-ranked cut entirely.
reach annotates the same key set the series draws. It answers “OpenAI read 41 of your pages” or “41 crawlers read this page”. It is a count, and it can never be expanded into which 41.
GET /v1/ai-crawls/recent
The live feed: individual recent crawl hits, newest first.
limit is 1 to 100, default 50.
It can legitimately be empty while every other endpoint keeps counting. Read crawl_allowance_state on the same response to know why:
degraded: you are past your crawl allowance on a plan that offers overage, so we stop writing individual rows while the rollups keep counting. Your totals are still exact.capped: you are past your allowance on a plan that does not offer crawl overage, so nothing is being recorded any more. The rollups have stopped too.
No IP address is ever returned
We verify some crawls against vendor infrastructure addresses. Those addresses are never serialised by any endpoint on this page, under any parameter. Verification is what they are for, not export. Visitor addresses are hashed on arrival and never stored raw at all. See what we collect.Windows
Every endpoint exceptrecent accepts a window of up to 366 days. recent accepts 7. Send from and to together, or neither, in which case you get the last 7 days.
timezone is not accepted anywhere on this page. Crawl figures are stored as UTC calendar days, so a local day boundary cannot be applied. Sending a zone returns 422 rather than UTC numbers under your zone’s label.
Errors
Next
- Detection for the bot filter, which is about visitor traffic rather than crawls