> ## Documentation Index > Fetch the complete documentation index at: https://docs.traceten.com/llms.txt > Use this file to discover all available pages before exploring further. # AI Crawlers API > Which AI crawlers read your pages, what they read, and how far we could verify them. ## Base URL and authentication ``` https://api.traceten.com/v1/ai-crawls ``` Takes `Authorization: Bearer ` ([details](/api/authentication)) and the `stats:read` permission. Crawler traffic is a separate population from visitor traffic. A crawl is a machine fetching a page to index or answer with it. Nothing here appears in your session, visitor or revenue figures, and nothing from those endpoints appears here. ## What "verified" means, before the endpoints Every crawl records how strongly its identity could be checked. The API returns the rung as a string; the dashboard shows the same value as a badge, and [Verification levels](/dashboard/ai-crawlers#verification-levels) is the full definition of each one. | API value | Dashboard badge | | ----------------- | ------------------- | | `web_bot_auth` | Verified, signature | | `ip_range` | Verified, IP range | | `reverse_dns` | Verified, rDNS | | `asn_match` | Verified, network | | `user_agent_only` | Unverified | | `spoofed` | Spoofed | | `unknown` | Not graded | Two consequences shape these responses, so they are worth repeating here. `user_agent_only` is normally the largest bucket. About half the crawlers we track publish no IP ranges and document no reverse-DNS convention, so there is nothing to check them against and no tool can verify them. A large unverified share is the truthful state of the ecosystem, not a fault in your setup. That is why the summary returns a breakdown by rung and never a single verified percentage. "No spoofing detected" does not mean everything was checked and passed. Spoofed is reachable two ways: an IP mismatch, which needs the vendor to publish ranges, and a live request carrying a `robots.txt` permission token, which no crawler ever sends. For a crawler that publishes nothing and sends no permission token, neither route applies and nothing can be confirmed either way. Read a zero in `spoofed` as "nothing was proven", not as "nobody impersonated you". ## `GET /v1/ai-crawls/summary` Totals and the verification breakdown for a window. ```bash theme={null} curl "https://api.traceten.com/v1/ai-crawls/summary?site_id=ttid_7Rb4TrC1dTbnD8w3s1TS12&from=2026-08-01&to=2026-08-07" \ -H "Authorization: Bearer " ``` ```json theme={null} { "status": "success", "data": { "site_id": "ttid_7Rb4TrC1dTbnD8w3s1TS12", "from": "2026-08-01", "to": "2026-08-07", "has_crawls": true, "total_crawls": 4820, "distinct_agents": 14, "distinct_providers": 8, "verification_breakdown": [ { "verification": "user_agent_only", "crawl_count": 2410 }, { "verification": "ip_range", "crawl_count": 2180 }, { "verification": "spoofed", "crawl_count": 230 } ], "spoofed_crawls": 230, "crawler_facing_hits": 190, "crawl_allowance_state": "under" } } ``` `has_crawls: false` is the distinguishable empty state. When it is false, `verification_breakdown` is an empty array rather than a list of zeroes that would imply a 0 percent verified rate. `crawl_allowance_state` says what your account's AI-crawl allowance is doing, so you can tell an empty window apart from a stopped one. `under` and `metered` are full fidelity. `degraded` means these totals are still exact but individual crawl records are not being kept. `capped` means new crawl hits are no longer recorded at all, so `has_crawls: false` there does not mean no crawlers came. See [the account usage endpoint](/api/account#the-ai-crawl-allowance) for the full table. ## `GET /v1/ai-crawls/by-page` Two views on one endpoint, told apart by `mode` in the response. Without `path`, you get the top crawled paths (`mode: "ranking"`, `page_detail: null`). With `path`, you get one page's detail (`mode: "page_detail"`, `pages: null`). The unused field is `null` rather than an empty array, so an empty result and the wrong mode are never confusable. ```bash theme={null} curl "https://api.traceten.com/v1/ai-crawls/by-page?site_id=ttid_7Rb4TrC1dTbnD8w3s1TS12&from=2026-08-01&to=2026-08-07&category=answer_fetch" \ -H "Authorization: Bearer " ``` ```json theme={null} { "status": "success", "data": { "site_id": "ttid_7Rb4TrC1dTbnD8w3s1TS12", "from": "2026-08-01", "to": "2026-08-07", "mode": "ranking", "pages": [ { "path": "/pricing", "crawl_count": 412, "unique_agents": 6, "genuine_agents": 4, "categories": { "answer_fetch": 412, "search_index": 0, "training": 0, "ai_crawler": 0 }, "verification": { "web_bot_auth": 0, "ip_range": 300, "reverse_dns": 0, "asn_match": 0, "user_agent_only": 112, "spoofed": 0, "unknown": 0 } } ], "page_detail": null } } ``` ### Parameters | Parameter | Type | Required | Notes | | -------------- | ------- | -------- | ------------------------------------------------------------------- | | `site_id` | string | yes | The snippet key. | | `from`, `to` | string | no | `YYYY-MM-DD`, inclusive, sent together. | | `limit` | integer | no | 1 to 500, default 50. Ranking mode only. | | `verification` | string | no | A rung, `genuine`, or `all` (default). | | `category` | string | no | `answer_fetch`, `search_index`, `training`, `ai_crawler`, or `all`. | | `provider` | string | no | Narrow to one crawling company. | | `path` | string | no | Drill into one path, matched exactly. Switches mode. | `verification=genuine` means not spoofed and actually graded. It exists because writing the predicate yourself as "not spoofed" silently includes every ungraded row. Sending `verification` or `limit` together with `path` is a `422`. The drill-down ranks nothing, so there is no ranking for those to apply to, and accepting them would mean returning a number that ignored a filter you sent. ### Filters run on the server, and that is load bearing `verification`, `category` and `provider` filter and re-rank in SQL. If you filter the returned rows yourself instead, you keep the rows we chose under a different ordering: correct numbers attached to the wrong pages. The same applies to `category`. Under a category filter, the per-row `categories` split describes the filtered rows only, so the three unselected buckets read 0 and `crawl_count` counts the selected category. That is the honest reading, because every number on the row is then about the same rows. To see the full split, ask without the filter. ### `provider` also drops spoofed rows, deliberately In ranking mode, filtering by `provider` excludes spoofed crawls. A spoofed row carries the impersonated company's name, so a plain provider match would print an impersonator's volume under the real company's brand. Expect `verification.spoofed` to read 0 on every row of a provider-filtered response. That is the filter working. To ask "who is pretending to be this company", send `provider` together with `verification=spoofed`. ### Page detail ```bash theme={null} curl "https://api.traceten.com/v1/ai-crawls/by-page?site_id=ttid_7Rb4TrC1dTbnD8w3s1TS12&from=2026-08-01&to=2026-08-07&path=/pricing" \ -H "Authorization: Bearer " ``` `page_detail` carries `total_crawls`, `genuine_crawls`, a `providers` list and a sparse `daily` series. Providers are bucketed by rung, so a spoofed claim is a separate row from the same company's genuine traffic. `genuine_crawls` excludes both spoofed and ungraded rows; it is the figure to print next to a company's name. A path nobody crawled returns `200` with `total_crawls: 0`. That is not an error. ## `GET /v1/ai-crawls/timeline` Daily crawl volume, split by `by`. `by=category` (default) returns rows carrying `category`. `by=provider` returns rows carrying `provider` and `is_other`, covering the top `limit` companies plus one roll-up row per day. Each mode takes only the parameters it can use. `provider` narrows a `by=category` series to one company, and is a `422` under `by=provider`, which groups by company already. `limit` bounds the company list under `by=provider`, and is a `422` under `by=category`, which returns every category and has no top N to bound. The alternative was accepting them and dropping them, which returns the wrong scope's numbers with a `200` beside them. ```json theme={null} { "status": "success", "data": { "by": "provider", "timeline": [ { "date": "2026-08-01", "provider": "OpenAI", "crawl_count": 310, "spoofed_count": 12, "is_other": false }, { "date": "2026-08-01", "provider": "other", "crawl_count": 44, "spoofed_count": 0, "is_other": true } ], "limit": 8, "truncated": true } } ``` Match the roll-up on `is_other`, never on `provider == "other"`. "Other" is a plausible company name, and unlike the category taxonomy the provider registry is not a closed set. ## `GET /v1/ai-crawls/series` A ranked daily series. `dimension=provider` gives a line per company, `dimension=path` a line per page. `path` and `provider` combine, and they combine on the server. They have to: this response is a top N chosen by an ordering, so narrowing it in your own code keeps rows that were selected under a different predicate. `measure=spoofed` ranks by impersonation volume instead of crawl volume. It is a server parameter rather than a client-side re-sort because the two measures name different companies: the most impersonated brand may send very little genuine traffic and would fall outside a crawls-ranked cut entirely. `reach` annotates the same key set the series draws. It answers "OpenAI read 41 of your pages" or "41 crawlers read this page". It is a count, and it can never be expanded into which 41. ## `GET /v1/ai-crawls/recent` The live feed: individual recent crawl hits, newest first. ```json theme={null} { "status": "success", "data": { "crawls": [ { "crawl_id": "a2f1c8d4-1f2e-4a3b-9c8d-77aa11bb22cc", "timestamp": "2026-08-07T10:14:02Z", "path": "/pricing", "provider": "OpenAI", "agent": "ChatGPT-User", "category": "answer_fetch", "verification": "ip_range", "confidence": 0.95, "status_code": 200, "is_crawler_facing_path": false, "user_agent": "Mozilla/5.0 (compatible; ChatGPT-User/1.0)" } ], "limit": 50 } } ``` This is the one endpoint here reading individual rows rather than a rollup, so it is capped at 7 days and defaults to the last 7. `limit` is 1 to 100, default 50. It can legitimately be empty while every other endpoint keeps counting. Read `crawl_allowance_state` on the same response to know why: * `degraded`: you are past your crawl allowance on a plan that offers overage, so we stop writing individual rows while the rollups keep counting. Your totals are still exact. * `capped`: you are past your allowance on a plan that does not offer crawl overage, so nothing is being recorded any more. The rollups have stopped too. Do not read an empty feed as "no crawlers". ## No IP address is ever returned We verify some crawls against vendor infrastructure addresses. Those addresses are never serialised by any endpoint on this page, under any parameter. Verification is what they are for, not export. Visitor addresses are hashed on arrival and never stored raw at all. See [what we collect](/privacy/data-collected). ## Windows Every endpoint except `recent` accepts a window of up to 366 days. `recent` accepts 7. Send `from` and `to` together, or neither, in which case you get the last 7 days. `timezone` is not accepted anywhere on this page. Crawl figures are stored as UTC calendar days, so a local day boundary cannot be applied. Sending a zone returns `422` rather than UTC numbers under your zone's label. ## Errors | Status | Cause | | ------ | ---------------------------------------------------------------------------------------------- | | `401` | Missing, malformed, revoked or expired key. | | `403` | The key is valid but lacks `stats:read`. The body names the permission. | | `404` | No such site, or the key does not cover it. | | `422` | Bad `site_id`, a partial window, a window over the cap, a `timezone`, or an unknown parameter. | | `429` | Rate limited. Back off and retry. | | `500` | The query failed. You will never get zeroes standing in for an error. | ## Next * [Detection](/api/detection) for the bot filter, which is about visitor traffic rather than crawls