Skip to main content

​
What this lets you do

Decide which AI crawlers may use your content, state that in the files crawlers look for, and then check whether they actually fetched those files. Traceten reports; it never blocks. We are not inline infrastructure and nothing on your site changes because you installed us. Control lives in your own files, and this page covers the stock directives every crawler documents support for.

​
robots.txt

robots.txt is the only control mechanism every AI vendor documents. It is a request, not an enforcement boundary, but the major vendors state that they honour it. Block training crawlers, allow everything else:
That combination is the common shape: it keeps you out of training corpora while leaving you available to the crawlers that answer live questions and build the search indexes that send you traffic. See the four purposes for why those are different decisions. Two things to get right:
  • The token is not always the crawler name you see in a user agent. Use the vendor’s documented robots.txt token. Each entry in the crawler directory lists the token to use.
  • Some tokens are robots.txt-only. Google-Extended and Applebot-Extended are permission signals, not crawlers. Google and Apple document them as opt-out controls for AI training, and neither is a user agent a crawler sends, so you use them in robots.txt and expect no traffic from them. Because neither is a crawler, neither should produce traffic at all. If one does, something is impersonating a token that never makes requests, and that is what the AI Crawlers page will tell you:
    • A request carrying Google-Extended is not reported at all. Nothing in the middleware matches that token, so it is treated as ordinary traffic and never sent to us. Seeing nothing is the correct outcome.
    • A request carrying Applebot-Extended is reported. Depending on what could be checked at the time, it lands as Spoofed (recorded as its own agent, Applebot-Extended, because a permission token has no crawl to verify, so a live request bearing one cannot be legitimate) or as Unverified with no provider or agent, which means we could not check it against the crawler registry and recorded only what the request claimed. See how verification works.
    On the robots.txt side, the two tokens are separate permissions from the crawlers they relate to. Disallowing Applebot-Extended asks Apple not to use your content for training; it does not ask Applebot to stop crawling, and Apple documents them as independent controls. The same holds for Google-Extended and Google’s crawlers. If you want both, write both.

​
llms.txt

llms.txt is an emerging convention: a Markdown file at your site root that tells AI systems how to use your content and points at the pages worth reading. llms-full.txt is the long-form variant with content inline. It has no formal status and no vendor promises to honour it. It costs almost nothing to publish, and Traceten shows you whether anything actually fetched it, which is the only way to find out.

​
Watching the plumbing files

The AI Crawlers page has a panel for robots.txt, llms.txt, llms-full.txt and sitemaps. Crawls of these files are reported even though other .txt and .xml paths are filtered out, because they are the highest-signal requests a crawler makes: they are what it fetches before deciding how to treat everything else. Read it as a leading indicator:
  • A crawler fetched robots.txt and then nothing else. It read your rules and honoured them. That is the file doing its job.
  • A crawler fetched robots.txt and then kept crawling paths you disallowed. Your rules are being ignored, and now you have dated evidence of it. robots.txt is voluntary, so the remedy is at your CDN or WAF, not in the file.
  • Nothing fetched llms.txt. Nobody is using it yet on your site. Publishing it cost you nothing; now you know.
  • A sudden spike on sitemaps. Something is about to crawl you broadly. Usually a re-index.

​
What we deliberately do not do

  • We do not generate or edit these files for you. They are on your origin and they are yours.
  • We do not offer blocking beyond these stock directives. Traceten never sits between a crawler and your server, so we could not enforce a rule even if we shipped one.
  • We do not tell you what to allow. Whether being in a training corpus is good or bad for your business is a commercial judgement, not a technical one.

​
Next