Who actually reads my llms.txt?

Checking who touches llms.txt in practice

/llms.txt and /llms-full.txt have been live on richardlemon.com for a while. The obvious next question: who, if anyone, is actually requesting them?

One way to approach that is to look at the web server logs directly. No dashboards, no vibes. Just the Nginx access log on the droplet.

Where the evidence lives: Nginx access logs

On a typical Nginx setup on a small VPS, access logs will usually live somewhere like:

/var/log/nginx/access.log
/var/log/nginx/access.log.1
/var/log/nginx/access.log.2.gz
...

The exact path depends on the distro and how Nginx was installed, but the pattern is similar: a current access.log plus rotated and sometimes gzipped older logs.

For anything related to llms.txt, the relevant fields are the request path and the user agent. That tells you which client asked for which file, and how it chose to introduce itself.

Filtering for llms.txt and llms-full.txt

First step is to pull out only the lines that touch those two files. You can do that with a simple grep across your logs. For example, from the directory where your logs live:

cd /var/log/nginx

grep -hE "llms\.txt|llms-full\.txt" access.log* > llms-requests.log

A few details here:

• -h hides the filename in the output, which keeps the next steps cleaner.
• -E lets you use an extended regular expression so you can match both filenames in one go.
• access.log* will hit the current log and any rotated ones that are not gzipped. If you have .gz files, you can run zgrep similarly.

After this you have a smaller file, llms-requests.log, that only contains requests for your LLM-related files.

Grouping by user agent

The next step is to see who those requests belong to. For that you need the user agent field from each log line.

This depends on the Nginx log format, but on a fairly standard combined log format the user agent is the last quoted string in the line. You can extract and group it with awk and sort:

I wrote about this earlier in When a client asks for SEO and needs GEO.

awk -F '"' '{print $6}' llms-requests.log \
  | sort \
  | uniq -c \
  | sort -nr > llms-user-agents.txt

What this does:

• -F '"' tells awk to split on double quotes. In a combined log, the user agent ends up as field 6.
• sort | uniq -c groups identical user agents and counts them.
• The final sort -nr orders them by frequency, highest first.

At this point you have a list of all user agents that have requested llms.txt or llms-full.txt, plus how often they did it.

Bucketing into GPTBot, ClaudeBot, Perplexity, Googlebot, other

Raw user agent strings are messy. If you want something legible, it helps to bucket them into a small set of known crawlers and an “everything else” bucket.

For LLM-related traffic, the usual suspects are:

• GPTBot (OpenAI)
• ClaudeBot (Anthropic)
• PerplexityBot
• Googlebot (useful for comparison, even though it is not an LLM crawler in the same sense)
• Other (browsers, random scrapers, SEO tools, misconfigured scripts)

You can do a rough classification with another awk pass that looks for substrings in the user agent string and assigns a label.

awk -F '"' '{
  ua = $6
  bucket = "Other"

  if (ua ~ /GPTBot/i) {
    bucket = "GPTBot"
  } else if (ua ~ /ClaudeBot/i) {
    bucket = "ClaudeBot"
  } else if (ua ~ /PerplexityBot/i) {
    bucket = "PerplexityBot"
  } else if (ua ~ /Googlebot/i) {
    bucket = "Googlebot"
  }

  print bucket
}' llms-requests.log \
  | sort \
  | uniq -c \
  | sort -nr

This does a few things:

• Looks at each log line, extracts the user agent, and sets a default bucket of Other.
• Checks for known bot identifiers with case-insensitive regexes.
• Prints the bucket name for each request, then groups and counts.

The result is a small table of how many requests came from each crawler family. You can run the same script periodically to see if the mix changes over time.

What you might expect to see

Given that llms.txt and llms-full.txt have been live for months, it is reasonable to expect at least some of the known LLM crawlers to touch them occasionally if they are crawling the domain at all.

That could look like:

More on this in my article The boring training week I can actually repeat.

• Sporadic hits from GPTBot, ClaudeBot, or PerplexityBot if they respect and look for llms.txt-style files.
• Regular background noise from Googlebot and other search crawlers that might fetch anything that looks like a text file.
• Manual requests from browsers or command-line tools when humans are testing or debugging.

If the logs show zero requests from a specific crawler family over a long period, that suggests either the crawler is not visiting the domain at all or it does not care about these particular files. If you see requests from a bot with a name that sounds like a major provider, that still needs a bit of skepticism.

Why a request is not the same as being in the training set

It is tempting to look at a log entry from something called GPTBot and immediately conclude that your content is now in some training corpus. That leap is not supported by the data.

The logs only prove one thing: at a certain time, from a certain IP, a client that claimed a particular user agent fetched a particular URL and got some HTTP status code back. Everything beyond that is inference.

Some specific limits:

• Identity is self-reported. User agents can be faked. Without reverse DNS and IP range checks, you do not know if a request really came from OpenAI, Anthropic, or anyone else.
• Fetching is not storing. A crawler can request llms.txt, read the policy, and then decide not to crawl or not to retain content. The file might be used purely as a routing or filtering hint.
• Storing is not training. Even if content is cached, that does not prove it ends up in a training dataset. Providers often separate crawling, indexing, and training pipelines.
• Training is not citation. Even if something is in a training set, there is no guarantee it will ever be surfaced as a citation or explicit reference in generated output.

So “I see GPTBot in my logs” is a weaker claim than “GPTBot trained on my content,” and much weaker than “GPTBot will cite my content.” The first is observable. The others are policy and implementation details on the provider side.

What this does and does not say about GEO

There is a lot of talk about geographic opt-out files and LLM-specific robots instructions as a way to control how models use content. Files like llms.txt are one piece of that conversation, but access logs do not tell you much about how GEO policies are actually enforced.

From the logs you can reasonably say:

Read also The client code I will not paste into an AI tool.

• “This crawler family appears to be aware of llms.txt and fetches it.”
• “This crawler seems to visit from these IP ranges and with this frequency.”

From the logs you cannot safely say:

• “This provider is fully complying with my GEO or usage instructions.”
• “My content is excluded from all future training runs in all regions.”
• “If I change llms.txt today, the effect will propagate everywhere by a specific date.”

At best, you can correlate: you publish or change llms.txt, you see a certain bot fetch it, and you assume that whatever policy they have that mentions the file is now in play. The real enforcement logic and the timelines sit behind someone else’s API and release notes.

Why still bother with llms.txt and logs

Even with all those caveats, llms.txt and llms-full.txt are still useful knobs to have:

• They give you a clear, machine-readable place to state your preferences.
• Access logs let you verify at least that some crawlers are paying attention.
• When policies or providers change, you have a concrete file to update instead of guessing which meta tags or headers matter this month.

The flip side is that they are not magic. If you care about how your content is used in LLMs, logs are just the first, low-tech feedback loop. They tell you who knocked on the door. They do not tell you what happened to the data after it left your server.