We Watched the AI Crawlers: A 17-Day Log Study (2026)

A note on funding: CypherpunkGuide carries no surveillance advertising — no ad networks, tracking pixels, or sponsored content. It is funded by transparent streams: reader donations now; subscription and editorially-aligned affiliate later. We answer to our readers, not to advertisers.

Every guide to keeping AI models out of your writing ends at the same three levers: add the crawler’s name to robots.txt (the text file that politely asks bots what not to visit), publish an llms.txt (a newer, AI-specific version of the same idea), or tell your host to “block GPTBot.” All three assume the same thing — that the bot knocking at your door is who its user-agent says it is. A user-agent is just a line of text a program sends to announce itself; it is typed, not proven, and anything can type anything.

So we stopped theorizing and looked. For 17 days — June 19 to July 5, 2026 — we recorded every self-identifying AI crawler that touched this site: 1,635 requests across 14 different commercial AI user-agents, from ClaudeBot and GPTBot down to names most publishers never see. Then we did the step the advice skips: we checked each request’s real source address against the address list the named company itself publishes. The result splits cleanly into three groups, and only one of them is good news. This is the measurement behind a promise we made earlier — that we keep this site’s own crawler logs under watch — and it is the empirical floor under the enclosure of the open web we argued elsewhere.

The uncomfortable finding is not that AI crawlers lie. It is that the ones you most want to stop are exactly the ones your tools cannot touch — because the whole opt-out regime works only when the crawler is both honest enough to obey and transparent enough to be checked. Below is what 17 days of one small site’s logs revealed about which is which.

What We Measured, and How You’d Reproduce It

#

We collected 17 consecutive days of edge logs, grouped every request by its user-agent string, and kept the 14 that name a commercial AI system. That is the raw claim — a bot said “I am GPTBot.” Verification is a separate, harder question, and it has an objective answer: does the request’s source IP fall inside the address range that OpenAI, Anthropic, Google or Amazon publishes for its own crawler? Every major operator posts that list precisely so servers can tell a real bot from an impostor.

The 1,635 hits sorted into a sharp hierarchy. The full per-agent tally ships with this article as a plain-text summary — aggregate counts only, no visitor data, crawler networks named only at the /24 level — a /24 being a block of 256 consecutive IP addresses, the smallest unit these networks are handed out in (verification summary).

Seven more low-volume agents — another 101 requests — round out the 14; the full list is in the summary file.

A verified request is one whose source IP the named company vouches for. The percentages are weighted by request count, and where a bot’s volume is small we say so outright — PerplexityBot’s “21%” is 5 hits out of 24, too thin to indict a company on, and we treat it that way below. Two caveats bind every number here: this is one new, low-traffic site over 17 days, not the whole web; and a published address list is only as current as its date — Perplexity’s is 17 months old, which we return to.

The Declared Crawlers Are Mostly Real

#

Start with the good news, because it is real and it matters: when a big lab’s crawler announces itself, it is almost always telling the truth. GPTBot verified at 97%, ClaudeBot at 99%, and GoogleOther at 100% against each company’s published address ranges. The overwhelming majority of “GPTBot” and “ClaudeBot” traffic genuinely originated from OpenAI and Anthropic infrastructure. One caution for anyone here to opt out of AI training specifically: GoogleOther is a general-purpose Google crawler, a different token from Google-Extended, the one that actually governs Gemini training — and every Google-Extended hit we logged came from the spoof cluster below, not from Google.

This is the part the cynical take gets wrong. If you add GPTBot to your robots.txt or block Anthropic’s ranges at your edge, you will stop the real GPTBot and the real ClaudeBot — they publish their addresses (OpenAI, Anthropic), they respect the file, and independent network-wide data agrees the large declared crawlers are the dominant players — even as their individual shares shift from year to year (Cloudflare, 2025). The honest crawler is a solved problem. If every AI bot behaved like GoogleOther — named, ranged, checkable — this article would be a footnote.

One declared crawler resists this clean sorting, and honesty requires naming it. Only 5 of our 24 PerplexityBot hits matched Perplexity’s published range — but 24 hits is far too thin to judge a company by, and Perplexity’s list is itself 17 months old, so a miss may mean nothing more than a stale file. There is a live dispute in the background: Cloudflare reported in August 2025 that Perplexity used stealth, undeclared crawlers to slip past no-crawl rules (Cloudflare, 2025), and Perplexity publicly rejected that framing (Perplexity, 2025), arguing a user-prompted “agent” fetching a page on demand is not the same thing as a pre-emptive “bot,” and that Cloudflare had misattributed to it the traffic of a third-party cloud-browser service. From an IP address alone we cannot tell an outside impostor from a company’s own undeclared fetcher — so on 24 hits we conclude nothing about Perplexity, and simply flag it as the declared crawler our checks verify least well.

The number that stayed with us is a different, sharper one: the tiny remainder among the crawlers that did verify. ClaudeBot’s 7 unverified hits and GPTBot’s 9 were not random noise. Every one of them traced back to the same place — a place that had no business claiming either name.

One Network Wore Fourteen Faces

#

Here is the finding that named the article. Four of those address blocks — four /24s — sat entirely outside every company’s published range, and in 17 days those four blocks alone produced 99 requests carrying 14 different AI companies’ user-agents. One /24 — 185.213.174.0/24 — impersonated all fourteen by itself: GPTBot, ClaudeBot, PerplexityBot, Amazonbot, Google-Extended, Bytespider, cohere-ai and seven more, all from one small network.

Now put that beside a legitimate range. Two adjacent Anthropic blocks — 216.73.216.0/24 (281 requests) and its neighbour 216.73.217.0/24 (265) — sent 546 requests between them, and every single one said ClaudeBot: one entity, one identity, whichever block you inspect. Look up who owns each network, and the contrast finishes the thought. Ownership comes from RDAP, the public registry that maps an IP to the organization it was assigned to:

The legitimate blocks sit inside Anthropic’s own published range — and the one we resolved in RDAP comes back as Anthropic, PBC. The impersonating subnets are registered to commercial hosting companies — infrastructure anyone can rent by the hour — and two of the four trace to the same rented company, the signature of a single operator running one costume box. A real crawler carries one identity because it is one entity. A spoofer wears all the faces because the face is the cheap part. The user-agent is not an ID card; it is a costume, and robots.txt is a sign that only the costumed can choose to read.

Amazonbot: The Bot You Can’t Tell From a Rented Server

#

Amazonbot is where verification gets genuinely hard, and it is instructive precisely because Amazon is not hiding. The company publishes a crawler address list like the others — yet only 35% of our 344 “Amazonbot” hits (120 of them) matched it. The other 65% — 224 requests from 179 distinct addresses — did not appear on Amazon’s list at all.

The reason is structural. Amazon’s real crawler runs on the same cloud — AWS EC2 — that anyone can rent, so an impostor on EC2 looks, at the network layer, a lot like the real thing. Amazon’s own answer to this is a second check: a reverse-DNS test (FCrDNS — confirm the IP’s name resolves back to an official crawl.amazonbot.amazon host, in both directions). We ran it on a sample of 18 of the unmatched addresses. All 18 resolved to generic ec2-*.compute-1.amazonaws.com hosts — ordinary rented servers — and none to Amazon’s crawler domain (Amazon’s verification method). They were not new, unlisted Amazon IPs; they failed Amazon’s own test for being Amazon.

The lesson generalizes past one bot: a crawler that runs on rented cloud is the easiest to counterfeit, because the counterfeiter rents the identical cloud. Verification there cannot stop at an address list — it needs a cryptographic or DNS-anchored proof the operator controls. Amazon at least offers one. Some don’t offer anything.

The Crawlers You Cannot Check at All

#

The hardest group is not the liars — it is the crawlers you have no way to verify in either direction. meta-externalagent (Meta’s AI crawler) sent 179 requests and Bytespider (ByteDance’s) sent 112, and for neither company could we find a published IP range or an official reverse-DNS method to check a single one against. When we reverse-looked-up a sample of the Meta addresses (9 of them), they returned NXDOMAIN — no registered name at all. There is nothing to match, by design or neglect; you are asked to trust the header and given no way to.

This is the quiet core of the whole problem, and it is a privacy problem, not a webmaster’s inconvenience. The opt-out you were sold — robots.txt, “block the bot,” even the new llms.txt — is only as strong as your ability to tell whether it worked. It divides the world of AI crawlers into three, and your tools reach only the first:

And the newer, AI-specific fix fares no better on our own logs. We publish an llms.txt; across the 17 days, AI crawlers fetched our pages hundreds of times — GPTBot alone 310 times — and fetched the llms.txt file exactly zero times. A 300,000-domain study reached the same verdict at scale: llms.txt files show no measurable correlation with AI crawler behavior (SE Ranking, 2025). The standard the bots are asked to read, they are not reading.

Bottom Line — What Actually Protects You

#

If the goal is to keep your public writing, photos and posts out of AI systems, this 17-day slice says something plain and slightly bleak: the controls work best on the crawlers that would have behaved anyway, and not at all on the ones you most fear. That is not a reason to remove robots.txt — stopping the honest majority still matters, and it is the difference between your words entering the big models or not. It is a reason to stop mistaking the sign on the door for a lock.

Three honest conclusions follow. First, use the levers, but rank them by who obeys: robots.txt and edge-blocking the published ranges genuinely stop GPTBot, ClaudeBot and Google — that is most of the declared volume. Second, verify, don’t trust the label: if you act on your logs, check source IPs against operators’ published lists and reverse DNS, exactly as we did, rather than the user-agent string. Third — the load-bearing one — treat anything you publish in the clear as already readable by systems you cannot audit. The impostor cluster and the no-method crawlers are not going to honor a text file; the only durable privacy control is the choice of what enters the public record in the first place, the same permanence logic we mapped for posts you can’t unpublish and for what machines infer from fragments.

An emerging standard called Web Bot Auth aims to fix the costume problem at its root — cryptographic signatures a crawler cannot fake, so identity is proven instead of typed (Cloudflare, 2025). It is early and not widely adopted, and it will help only with crawlers that want to be identified. The ones wearing fourteen faces were never the target of a trust standard.

Frequently Asked Questions

#

Can you block AI crawlers with robots.txt?

#

Partly, and it depends entirely on the crawler’s honesty. In our 17-day logs, the big declared crawlers — GPTBot, ClaudeBot, GoogleOther — verified as genuine 97–100% of the time and do respect robots.txt, so listing them there stops them. But robots.txt is a voluntary request, not an enforced rule: a crawler that ignores it, or one wearing a fake user-agent, sails straight through. The file stops the polite and informs the honest; it means nothing to an impostor.

How do you verify an AI crawler is really GPTBot or ClaudeBot?

#

Don’t trust the user-agent string — check the source IP. OpenAI, Anthropic, Google and Amazon each publish the address ranges their crawlers use, so a real GPTBot request comes from an IP inside OpenAI’s published list. For crawlers on shared cloud (like Amazonbot on AWS), add a reverse-DNS check: confirm the IP’s hostname resolves back to the operator’s official crawler domain in both directions (FCrDNS). If a “GPTBot” hit comes from an address OpenAI never published, it is not GPTBot.

Does llms.txt keep AI out of my content?

#

On our evidence, no. We publish an llms.txt file, and over 17 days AI crawlers fetched our actual pages hundreds of times while fetching the llms.txt zero times — GPTBot read 310 pages and the file not once. A separate study of 300,000 domains found no measurable correlation between having an llms.txt and any change in AI crawler behavior. It is a proposed standard that the crawlers it addresses are largely not reading yet.

What is a spoofed AI crawler?

#

It is a request that carries a real AI company’s user-agent name but does not come from that company. In our logs, four commercial hosting networks — the kind anyone can rent — sent traffic labeled as 14 different AI companies, including one network that impersonated all fourteen by itself. Because a user-agent is just a text string the sender chooses, impersonation is trivial; only checking the source IP against published ranges tells you whether the name is earned or worn.

If I can’t stop every crawler, what actually protects my privacy?

#

The only control that survives an uncheckable or dishonest crawler is deciding what you make public in the first place. Robots.txt and blocking published ranges will stop the honest majority, which is worth doing. But for the crawlers you cannot verify — and the impostors who ignore every rule — assume that anything posted in the clear is already readable by systems you cannot audit, and treat publication itself as the decision point.

Sources

#

Our test artifact: the verification summary — aggregate per-user-agent request counts, verified counts, and network ownership at the /24 level, from the June 19–July 5, 2026 log window. No visitor data is included; crawler IPs are public infrastructure, disclosed no finer than /24.

A note on what this is and isn’t. This is one new, low-traffic site over 17 days — a case study, not a census. The percentages describe our logs, not the web. What generalizes is not the numbers but the method and its verdict: verify against published identity, and the world of crawlers splits into checkable and not. We will re-run the same measurement at the 90-day mark and publish the delta — whether the impostor cluster grows, whether the no-method crawlers ever become verifiable, and whether the honest ones stay honest.

This is the crawler-side companion to our argument that the open web is being enclosed — read more by machines than people — and the empirical follow-through on the AI-age threat model that opened this series. If you want to see what one of those machines could assemble about you from what is already public, our self-audit tool runs the same verify-don’t-trust discipline on your own footprint.