> ## Documentation Index
> Fetch the complete documentation index at: https://docs.traceten.com/llms.txt
> Use this file to discover all available pages before exploring further.

# robots.txt and llms.txt

> How to tell AI crawlers what they may do with your site, and how to see whether they read it.

## What this lets you do

Decide which AI crawlers may use your content, state that in the files crawlers look for, and then check whether they actually fetched those files.

Traceten reports; it never blocks. We are not inline infrastructure and nothing on your site changes because you installed us. Control lives in your own files, and this page covers the stock directives every crawler documents support for.

## robots.txt

`robots.txt` is the only control mechanism every AI vendor documents. It is a request, not an enforcement boundary, but the major vendors state that they honour it.

Block training crawlers, allow everything else:

```
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
```

That combination is the common shape: it keeps you out of training corpora while leaving you available to the crawlers that answer live questions and build the search indexes that send you traffic. See [the four purposes](/dashboard/ai-crawlers#the-four-purposes) for why those are different decisions.

Two things to get right:

* **The token is not always the crawler name you see in a user agent.** Use the vendor's documented `robots.txt` token. Each entry in the [crawler directory](https://traceten.com/crawlers) lists the token to use.
* **Some tokens are `robots.txt`-only.** `Google-Extended` and `Applebot-Extended` are permission signals, not crawlers. Google and Apple document them as opt-out controls for AI training, and neither is a user agent a crawler sends, so you use them in `robots.txt` and expect no traffic from them.

  Because neither is a crawler, neither should produce traffic at all. If one does, something is impersonating a token that never makes requests, and that is what the AI Crawlers page will tell you:

  * A request carrying **`Google-Extended`** is not reported at all. Nothing in the middleware matches that token, so it is treated as ordinary traffic and never sent to us. Seeing nothing is the correct outcome.
  * A request carrying **`Applebot-Extended`** is reported. Depending on what could be checked at the time, it lands as **Spoofed** (recorded as its own agent, `Applebot-Extended`, because a permission token has no crawl to verify, so a live request bearing one cannot be legitimate) or as **Unverified** with no provider or agent, which means we could not check it against the crawler registry and recorded only what the request claimed. See [how verification works](/dashboard/ai-crawlers#verification-levels).

  **On the `robots.txt` side, the two tokens are separate permissions from the crawlers they relate to.** Disallowing `Applebot-Extended` asks Apple not to use your content for training; it does not ask `Applebot` to stop crawling, and Apple documents them as independent controls. The same holds for `Google-Extended` and Google's crawlers. If you want both, write both.

## llms.txt

`llms.txt` is an emerging convention: a Markdown file at your site root that tells AI systems how to use your content and points at the pages worth reading. `llms-full.txt` is the long-form variant with content inline.

It has no formal status and no vendor promises to honour it. It costs almost nothing to publish, and Traceten shows you whether anything actually fetched it, which is the only way to find out.

```markdown theme={null}
# Example Corp

> Infrastructure monitoring for small engineering teams.

## Docs

- [Quickstart](https://www.example.com/docs/quickstart): install in five minutes
- [Pricing](https://www.example.com/pricing): plans and limits
```

## Watching the plumbing files

The **AI Crawlers** page has a panel for `robots.txt`, `llms.txt`, `llms-full.txt` and sitemaps. Crawls of these files are reported even though other `.txt` and `.xml` paths are filtered out, because they are the highest-signal requests a crawler makes: they are what it fetches before deciding how to treat everything else.

Read it as a leading indicator:

* **A crawler fetched `robots.txt` and then nothing else.** It read your rules and honoured them. That is the file doing its job.
* **A crawler fetched `robots.txt` and then kept crawling paths you disallowed.** Your rules are being ignored, and now you have dated evidence of it. `robots.txt` is voluntary, so the remedy is at your CDN or WAF, not in the file.
* **Nothing fetched `llms.txt`.** Nobody is using it yet on your site. Publishing it cost you nothing; now you know.
* **A sudden spike on sitemaps.** Something is about to crawl you broadly. Usually a re-index.

## What we deliberately do not do

* **We do not generate or edit these files for you.** They are on your origin and they are yours.
* **We do not offer blocking beyond these stock directives.** Traceten never sits between a crawler and your server, so we could not enforce a rule even if we shipped one.
* **We do not tell you what to allow.** Whether being in a training corpus is good or bad for your business is a commercial judgement, not a technical one.

## Next

* [The four crawler purposes](/dashboard/ai-crawlers#the-four-purposes)
* [Verification levels](/dashboard/ai-crawlers#verification-levels)
* [Install AI crawler tracking](/install/ai-crawler-tracking)