We Watched the AI Crawlers: A 17-Day Log Study (2026)

Funding disclosure: CypherpunkGuide may earn a commission on qualifying purchases through clearly disclosed affiliate links in relevant articles. We do not carry surveillance advertising, ad-network scripts, tracking pixels, or sponsored articles.

Correction, September 24, 2026: the original article overstated what IP matching could prove about crawler identity and blocking. It also treated llms.txt as an opt-out mechanism and described 14 labels as 14 companies. We have corrected those claims and the accompanying summary, and withdrawn unsupported Amazonbot and DNS conclusions. The observation window remains June 19–July 5, 2026; this update contains no new crawl experiment.

A publisher needs to distinguish three questions: what a visitor calls itself, whether its source address matches a documented crawler range, and whether an access rule actually prevented retrieval. Our logs help with the first two. They do not answer the third.

A user-agent is text a program sends to identify itself. The sender can choose that text. An IP-list check adds evidence about the source address, but the reference list, its scope, and its capture date matter. robots.txt asks compatible crawlers not to retrieve specified paths. A server or CDN block instead enforces an access rule. Neither is interchangeable with llms.txt, a proposal for presenting useful site information to language models.

The AI-age threat model called for checking this site’s crawler traffic. The resulting 17-day record is useful because it exposes the limits of relying on a name in a log, including limits our original interpretation missed.

What the 17-Day Record Contains

#

The saved aggregate records contain 1,635 requests classified into 14 AI-related user-agent families. A family is a category assigned by matching a label in the user-agent text, not a distinct company or a unique complete user-agent string. The classification also includes CCBot and a Google-Extended label; it should not be read as a census of 14 commercial training crawlers.

The corrected verification summary preserves the counts and explains their limitations. It contains aggregate data and selected network prefixes, not full source IPs or request-level logs. Readers can check the arithmetic, but cannot independently replay the source-IP analysis from that file.

The other seven categories account for 101 requests. Percentages are rounded and weighted by requests, not by unique visitors, pages, or source addresses.

Here, a match means that an observed source IP matched the reference set used in the July analysis. It does not authenticate the software, identify the person operating it, or show how retrieved content was used. A non-match can reflect an unrelated sender, an incomplete or outdated reference set, or a processing error. It is not, on its own, proof of impersonation.

IP Matches Do Not Measure Whether Blocking Worked

#

The recorded GPTBot and ClaudeBot match rates concern source addresses. We did not run a controlled before-and-after blocking experiment or observe training use. The original claim that these percentages proved robots.txt worked was not supported.

OpenAI’s crawler documentation distinguishes crawler roles and their controls. Anthropic’s guidance describes robots.txt instructions and warns that IP blocking can interfere with a crawler reading those instructions. These are operator statements about their systems, not results from our experiment.

Google’s verification guidance also depends on the crawler category and its documented verification method. GoogleOther is a general-purpose crawler. Google-Extended is a separate control token, not a separate HTTP user-agent; requests bearing that label in our logs do not measure Google’s use of the control.

For a blocking test, record the rule, its activation time, the crawler category, and whether subsequent requests received content. Attribution checks and access outcomes belong in separate columns. Our published dataset does not contain that experiment, so it cannot establish a blocking success rate or a training exclusion rate.

Four Networks, Fourteen Labels, Unknown Operators

#

The historical network summary records 99 requests across four /24 prefixes carrying 14 labels, with all 14 appearing within one prefix. A /24 is a block of 256 IPv4 addresses; it is a reporting granularity here, not the smallest possible allocation or an identity boundary.

The 14 labels include multiple products associated with the same company and some user-triggered fetchers. This network tally uses a different selection from the 1,635-request table above; it is not a simple subset to add or subtract.

The first row below is a comparison. The remaining four prefixes form the 99-request tally.

The adjacent 216.73.217.0/24 block accounts for another 265 ClaudeBot requests, giving the recorded total of 546 matches. Those counts describe this sample, not a rule that legitimate infrastructure always uses one name.

RDAP, the public registry information used for the organization column, describes address registration. It does not tell us which customer ran a request. Two prefixes registered to the same hosting company do not establish a single actor. The label diversity is a reason to investigate attribution, not evidence that one operator impersonated 14 companies or ignored robots.txt.

Amazonbot: An Unresolved Reference-List Problem

#

The saved analysis reports 120 matches among 344 Amazonbot-labeled requests, but we cannot present 35% as a validated Amazonbot identity rate. The analysis extracted 525 addresses from a page where the research notes recorded an advertised list size of 489. Those notes flagged possible non-crawler addresses from page code and left the discrepancy unresolved.

The 120 matches and 224 non-matches remain in the corrected summary as historical processing results. Revalidating the extracted reference set is necessary before treating that split as an official-list comparison.

The original text also said that 18 unmatched addresses all failed Amazon’s DNS check. The available saved DNS aggregate describes a different 14-address sample, so it cannot substantiate that assertion. We withdraw the 18-address conclusion rather than substitute one sample for another. The unmatched 224 requests have not been established as impostors.

For a fresh check, use the current Amazonbot documentation and preserve the exact reference data and parsing method. A cloud provider’s ownership of an address is not sufficient to identify a specific crawler or its customer.

Missing DNS Names Do Not Establish Identity

#

The July analysis did not establish attribution for the 179 meta-externalagent-labeled requests or the 112 Bytespider-labeled requests. That is a limitation of this analysis, not proof that these crawlers are inherently unverifiable or that no method exists today.

For nine Meta-labeled addresses, the stored DNS result reports no name. The checker mapped all lookup failures to the same NXDOMAIN label, including failures that may have had other causes. We therefore cannot claim nine confirmed nonexistent DNS names. Nor can a failed lookup establish whether the sender obeyed a crawl rule.

llms.txt Is a Content Guide, Not an Opt-Out

#

llms.txt is a proposal for guiding language models to useful site content, not a request to keep them out. The proposal serves a different purpose from robots.txt. Its original 2024 purpose was already informational; this is not a new opt-out feature introduced after our study.

The July research note records zero AI-classified requests to our /llms.txt during the observation window. That path-level count cannot be recalculated from the saved aggregate JSON used for this correction. We retain it as a reported observation with that limitation. The GPTBot total of 310 is a request count, not 310 distinct pages or 310 successful downloads.

Even if the zero count is taken as reported, it says only that this site’s logging and classification recorded no such requests in that window. It does not measure llms.txt adoption across the web, AI citations, or the effectiveness of an opt-out mechanism. Our earlier comparison to an external citation study did not support that conclusion and has been removed.

Bottom Line: Verify Identity and Access Separately

#

The useful result is narrower than our first version claimed: user-agent labels, reference-IP matches, and successful access controls are different kinds of evidence. Keep them separate when deciding what your logs establish.

1. Use each operator’s documented crawl controls for the relevant purpose. Distinguish training crawlers, search crawlers, and user-triggered fetchers. Do not treat an IP match as a compliance test.
2. Preserve the reference and the method. Record its retrieval date, exact address set, matching rules, and unresolved failures. Shared hosting or an unmatched address warrants caution, not automatic attribution.
3. Test access rules by their outcomes. If content must remain private, require authorization before serving it and test that unauthorized requests are refused. A public opt-out file is not an access barrier.
4. Choose what enters the public record. Content already copied elsewhere is a separate problem from future access to your server. The limits are related to posts you cannot fully unpublish and inferences drawn from public fragments.

Frequently Asked Questions

#

Can you block AI crawlers with robots.txt?

#

You can publish instructions for crawlers that honor the relevant rules, but robots.txt does not enforce access control. Our IP-match results do not measure whether a crawler obeyed it. Follow the operator’s documented controls and test actual access outcomes separately. If the content must be private, restrict who can retrieve it.

How do you verify a request labeled GPTBot or ClaudeBot?

#

Compare its source IP with the appropriate operator reference, recording the version and capture date. Follow any verification method documented for that crawler. A match supports source attribution within those limits; a non-match is not by itself proof of impersonation. It does not tell you whether the content entered a model’s training data.

Does llms.txt keep AI out of my content?

#

No. The proposal is a guide to useful site content for language models, not a crawl-denial or training opt-out mechanism. Our July note reports zero AI-classified requests to that file in 17 days, but that count is not a test of exclusion and cannot be recalculated from the retained aggregate JSON.

What is a spoofed AI crawler?

#

A spoofed request uses a crawler’s label without being that crawler. A sender can choose its user-agent text, but a label mismatch or shared hosting alone does not prove spoofing. Our four-prefix observations justify caution; they do not establish who operated the requests or whether the operators ignored robots.txt.

What can I do if I cannot identify every crawler?

#

Use and test access controls for content that should not be public. For public pages, apply documented crawl preferences, monitor outcomes, and keep uncertainty visible in your analysis. No request log can prove that a copy of your content was never obtained elsewhere or used later.

Sources

#

External documentation checked September 24, 2026. Archive dates differ from the observation window and do not recreate the original IP lists.

The corrected aggregate summary preserves the historical counts and documents the withdrawn conclusions. The September 24 update reviews retained aggregates and methods; it does not publish a new 90-day rerun or a controlled blocking experiment.

This case study informs our discussion of the open web’s enclosure, without proving its wider claims from one site’s logs. Our self-audit tool addresses a different question: what public information about you can be found and checked.