Base URL and authentication
Authorization: Bearer <YOUR_API_KEY> (details) and the stats:read permission.
Crawler traffic is a separate population from visitor traffic. A crawl is a machine fetching a page to index or answer with it. Nothing here appears in your session, visitor or revenue figures, and nothing from those endpoints appears here.
What “verified” means, before the endpoints
Every crawl records how strongly its identity could be checked. The API returns the rung as a string; the dashboard shows the same value as a badge, and Verification levels is the full definition of each one.
Two consequences shape these responses, so they are worth repeating here.
user_agent_only is normally the largest bucket. About half the crawlers we track publish no IP ranges and document no reverse-DNS convention, so there is nothing to check them against and no tool can verify them. A large unverified share is the truthful state of the ecosystem, not a fault in your setup. That is why the summary returns a breakdown by rung and never a single verified percentage.
“No spoofing detected” does not mean everything was checked and passed. Spoofed is reachable two ways: an IP mismatch, which needs the vendor to publish ranges, and a live request carrying a robots.txt permission token, which no crawler ever sends. For a crawler that publishes nothing and sends no permission token, neither route applies and nothing can be confirmed either way. Read a zero in spoofed as “nothing was proven”, not as “nobody impersonated you”.
GET /v1/ai-crawls/summary
Totals and the verification breakdown for a window.
has_crawls: false is the distinguishable empty state. When it is false, verification_breakdown is an empty array rather than a list of zeroes that would imply a 0 percent verified rate.
crawl_allowance_state says what your account’s AI-crawl allowance is doing, so you can tell an empty window apart from a stopped one. under and metered are full fidelity. degraded means these totals are still exact but individual crawl records are not being kept. capped means new crawl hits are no longer recorded at all, so has_crawls: false there does not mean no crawlers came. See the account usage endpoint for the full table.
GET /v1/ai-crawls/by-page
Two views on one endpoint, told apart by mode in the response.
Without path, you get the top crawled paths (mode: "ranking", page_detail: null). With path, you get one page’s detail (mode: "page_detail", pages: null). The unused field is null rather than an empty array, so an empty result and the wrong mode are never confusable.
Parameters
verification=genuine means not spoofed and actually graded. It exists because writing the predicate yourself as “not spoofed” silently includes every ungraded row.
Sending verification or limit together with path is a 422. The drill-down ranks nothing, so there is no ranking for those to apply to, and accepting them would mean returning a number that ignored a filter you sent.
Filters run on the server, and that is load bearing
verification, category and provider filter and re-rank in SQL. If you filter the returned rows yourself instead, you keep the rows we chose under a different ordering: correct numbers attached to the wrong pages.
The same applies to category. Under a category filter, the per-row categories split describes the filtered rows only, so the three unselected buckets read 0 and crawl_count counts the selected category. That is the honest reading, because every number on the row is then about the same rows. To see the full split, ask without the filter.
provider also drops spoofed rows, deliberately
In ranking mode, filtering by provider excludes spoofed crawls. A spoofed row carries the impersonated company’s name, so a plain provider match would print an impersonator’s volume under the real company’s brand. Expect verification.spoofed to read 0 on every row of a provider-filtered response. That is the filter working.
To ask “who is pretending to be this company”, send provider together with verification=spoofed.
Page detail
page_detail carries total_crawls, genuine_crawls, a providers list and a sparse daily series. Providers are bucketed by rung, so a spoofed claim is a separate row from the same company’s genuine traffic. genuine_crawls excludes both spoofed and ungraded rows; it is the figure to print next to a company’s name.
A path nobody crawled returns 200 with total_crawls: 0. That is not an error.
GET /v1/ai-crawls/timeline
Daily crawl volume, split by by.
by=category (default) returns rows carrying category. by=provider returns rows carrying provider and is_other, covering the top limit companies plus one roll-up row per day.
Each mode takes only the parameters it can use. provider narrows a by=category series to one company, and is a 422 under by=provider, which groups by company already. limit bounds the company list under by=provider, and is a 422 under by=category, which returns every category and has no top N to bound. The alternative was accepting them and dropping them, which returns the wrong scope’s numbers with a 200 beside them.
is_other, never on provider == "other". “Other” is a plausible company name, and unlike the category taxonomy the provider registry is not a closed set.
GET /v1/ai-crawls/series
A ranked daily series. dimension=provider gives a line per company, dimension=path a line per page.
path and provider combine, and they combine on the server. They have to: this response is a top N chosen by an ordering, so narrowing it in your own code keeps rows that were selected under a different predicate.
measure=spoofed ranks by impersonation volume instead of crawl volume. It is a server parameter rather than a client-side re-sort because the two measures name different companies: the most impersonated brand may send very little genuine traffic and would fall outside a crawls-ranked cut entirely.
reach annotates the same key set the series draws. It answers “OpenAI read 41 of your pages” or “41 crawlers read this page”. It is a count, and it can never be expanded into which 41.
GET /v1/ai-crawls/recent
The live feed: individual recent crawl hits, newest first.
limit is 1 to 100, default 50.
It can legitimately be empty while every other endpoint keeps counting. Read crawl_allowance_state on the same response to know why:
degraded: you are past your crawl allowance on a plan that offers overage, so we stop writing individual rows while the rollups keep counting. Your totals are still exact.capped: you are past your allowance on a plan that does not offer crawl overage, so nothing is being recorded any more. The rollups have stopped too.
No IP address is ever returned
We verify some crawls against vendor infrastructure addresses. Those addresses are never serialised by any endpoint on this page, under any parameter. Verification is what they are for, not export. Visitor addresses are hashed on arrival and never stored raw at all. See what we collect.Windows
Every endpoint exceptrecent accepts a window of up to 366 days. recent accepts 7. Send from and to together, or neither, in which case you get the last 7 days.
timezone is not accepted anywhere on this page. Crawl figures are stored as UTC calendar days, so a local day boundary cannot be applied. Sending a zone returns 422 rather than UTC numbers under your zone’s label.
Errors
Next
- Detection for the bot filter, which is about visitor traffic rather than crawls