I checked who reads my llms.txt. Almost nobody, and most of "GPTBot" wasn't GPTBot

Last week I wrote about how to find out which AI crawlers read your llms.txt, using nothing but the Nginx access logs. What that post didn't have was an answer. So I ran the commands on my own server.

The setup: this blog, a single Ghost site on a DigitalOcean droplet, with an llms.txt at the root. Nginx keeps 15 days of access logs, so the window is 7 to 21 October 2026. One small personal site, two weeks. Keep that in mind for everything below.

The short version

In 15 days, /llms.txt and /llms-full.txt were requested 31 times. Two of those came from something that identifies as an AI agent. None came from GPTBot, ClaudeBot, PerplexityBot or Googlebot.

Meanwhile, something calling itself GPTBot requested my site 4,565 times. About 95 percent of that wasn't OpenAI.

Who fetched llms.txt

24 of the 31 got a 200; the other 7 were 301 redirects (http or www to the canonical address).

The one AI hit worth noting is Claude-User. That isn't a crawler. It is Claude fetching a page because someone asked it to, in a conversation. Somebody, somewhere, pointed Claude at my llms.txt once.

The rest is what you would expect: SEO tools checking whether the file exists, and people (or scripts pretending to be browsers) having a look.

What the big crawlers did instead

The crawlers everyone writes about were there, a lot. They just didn't ask for the file.

They ask for robots.txt and thousands of other URLs, and skip llms.txt. On this site, in these two weeks, llms.txt was a file for humans and SEO tools, not for the bots it was written for.

These are counts by user agent, and user agents can be faked. Which brings me to GPTBot.

Most of "GPTBot" wasn't GPTBot

4,565 requests carried GPTBot in the user agent. The most requested paths:

/
/app
/cms/.env
/.env_1
/api
/Dockerfile
/static../.env
/api/orders/..%2f..%2f.env
/.pgpass

GPTBot doesn't go looking for your .env and your Postgres password file. A vulnerability scanner does, and it borrows GPTBot's name to get past filters that wave AI crawlers through.

OpenAI publishes the IP ranges its crawler uses (https://openai.com/gptbot.json, 18 prefixes when I checked). Matching the requests against that list:

• 231 requests came from OpenAI's ranges.
• 4,334 came from somewhere else.

The split shows in the user agent too. The real one ends in (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot). The fake one closes the bracket too early: (KHTML, like Gecko); compatible; GPTBot/1.4. The counts for the two strings match the IP check exactly.

ChatGPT-User, Claude-User and CCBot each show roughly 4,300 to 4,500 requests from a few dozen addresses. I didn't verify those, but the pattern looks like the same scanner rotating through AI names.

The 200 that wasn't

The scanner's requests came back as 404s and redirects, except one 200. That had my attention until I read the path: /?url=.env. It's my homepage with a query string. Ghost ignores the parameter and serves the front page; the response was 97,220 bytes, the size of the homepage, not of a config file. A search for stray .env, .git or .sql files on the server came back empty. Still worth checking on your own server, because on a different stack that 200 could mean something else.

What I take from this

llms.txt costs nothing to publish, so I'm keeping it. But on this site, the crawlers that train and power the big models didn't read it in two weeks. If you added one expecting GPTBot or ClaudeBot to use it as a map of your site, check your own logs before you build on that.

"AI bot traffic" in your logs is partly not AI at all. If you allow or block by user agent, you are trusting a string anyone can type. Verify against published IP ranges where they exist.

And two names you will never find in your logs: Google-Extended and Applebot-Extended. They aren't crawlers, only tokens for robots.txt. Google fetches as Googlebot and Apple as Applebot; the token only says what the content may be used for.

The commands

Everything above came from these, run as root on the server. They read the current and the rotated logs (zcat -f handles both) and print counts, not IP addresses.

Who fetched llms.txt:

zcat -f /var/log/nginx/access.log* | grep -E '"GET /llms(-full)?\.txt' | awk -F'"' '{print $6}' | sort | uniq -c | sort -rn | head -25

Status codes for llms.txt:

zcat -f /var/log/nginx/access.log* | grep -E '"GET /llms(-full)?\.txt' | awk '{print $9}' | sort | uniq -c

Requests per bot, and how many of those were for llms.txt and robots.txt:

for b in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-User PerplexityBot Googlebot bingbot Applebot CCBot Bytespider Amazonbot meta-externalagent; do L=$(zcat -f /var/log/nginx/access.log* | grep -i "$b"); echo "$b total=$(echo "$L" | grep -c .) llms=$(echo "$L" | grep -c 'GET /llms') robots=$(echo "$L" | grep -c 'GET /robots')"; done

The paths a "GPTBot" asked for:

zcat -f /var/log/nginx/access.log* | awk -F'"' 'tolower($6) ~ /gptbot/ {split($2,r," "); print r[2]}' | sort | uniq -c | sort -rn | head -10

Real versus fake GPTBot, against OpenAI's published ranges:

curl -s https://openai.com/gptbot.json -o /root/gptbot.json
zcat -f /var/log/nginx/access.log* | awk -F'"' 'tolower($6) ~ /gptbot/ {split($1,a," "); print a[1]}' | python3 -c "
import sys,json,ipaddress as I
n=[I.ip_network(p.get('ipv4Prefix') or p.get('ipv6Prefix')) for p in json.load(open('/root/gptbot.json'))['prefixes']]
r=f=0
for l in sys.stdin:
  ip=I.ip_address(l.strip()); r+=any(ip in x for x in n); f+=not any(ip in x for x in n)
print('real',r,'fake',f)"

One site, two weeks, one set of logs. Run them on yours and I'd like to know if you see the same.