================================================================================ AI CRAWLER VERIFICATION — 17-DAY LOG SUMMARY CypherpunkGuide (cypherpunkguide.com) edge logs Window: 2026-06-19 to 2026-07-05 (17 consecutive days) ================================================================================ WHAT THIS FILE IS Aggregate, per-user-agent request counts from our own edge logs, with each user-agent's source IPs checked against the crawler operator's OWN published address list (and, for Amazonbot, forward-confirmed reverse DNS). WHAT THIS FILE IS NOT It contains NO visitor data of any kind. The only IP networks named here are those of automated crawlers, which are public infrastructure that the operators themselves publish. Networks are disclosed no finer than /24. "Verified" = the request's source IP falls inside the range the named company publishes for its crawler, as of the dated source shown. METHOD (reproducible) 1. Parse 17 days of edge logs; group every request by user-agent string. 2. Keep the user-agents that name a commercial AI system. 3. For each, test the source IP against the operator's published crawler address list (CIDR ranges — shorthand for a block of IPs — or individual IPs). 4. For cloud-hosted crawlers (Amazonbot on AWS), add forward-confirmed reverse DNS (FCrDNS): the IP's hostname must resolve to the operator's official crawler domain, and back again. ================================================================================ 1. COMMERCIAL AI CRAWLERS (autonomous) — 14 user-agents, 1,635 requests ================================================================================ user-agent requests verified source (as of) -------------------- -------- ----------------- -------------------------- ClaudeBot 553 546 (99%) Anthropic bot list (2026-05) Amazonbot 344 120 (35%) Amazon crawler IPs (2026-04) + FCrDNS GPTBot 310 301 (97%) OpenAI GPTBot list (2025-10) meta-externalagent 179 — (no method) Meta publishes no IP list / rDNS method Bytespider 112 — (no method) ByteDance publishes no method OAI-SearchBot 65 44 (68%) * OpenAI SearchBot list (2026-01) PerplexityBot 24 5 (21%) Perplexity bot list (2025-02, ~17mo old) CCBot 20 — (0 verified) all from the spoof cluster below GoogleOther 12 12 (100%) Google crawler list (2026-06) Google-Extended 6 0 all from the spoof cluster below cohere-ai 5 — (0) all from the spoof cluster below anthropic-ai 3 0 all from the spoof cluster below Claude-Web 1 0 spoof cluster DuckAssistBot 1 — — -------------------- -------- TOTAL 1,635 * OAI-SearchBot: some of the remaining hits fall inside OpenAI's shared GPTBot IP range (OpenAI shares crawler infrastructure across products); we count these conservatively and do not add them to the verified figure. Consistency check: ClaudeBot verified 546 = 216.73.216.0/24 (281) + 216.73.217.0/24 (265). ClaudeBot's 7 unverified + GPTBot's 9 unverified are accounted for entirely by the spoof cluster. Verified + unverified close. ================================================================================ 2. THE SPOOF CLUSTER — 4 networks outside every published range ================================================================================ In 17 days these four /24s produced 99 requests wearing 14 different AI companies' user-agents. Ownership from RDAP (public IP registry). /24 (network) AI names claimed registered to (RDAP) country ------------------- ---------------------- ------------------------ ------- 185.213.174.0/24 14 companies (all) NextGenWebs (web host) NL 45.45.237.0/24 part of cluster Infraly, LLC US 23.161.169.0/24 part of cluster Infraly, LLC (same org) US 154.58.229.0/24 part of cluster Limestone Networks US 185.213.174.0/24 alone claimed: GPTBot, ClaudeBot, OAI-SearchBot, ChatGPT-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Amazonbot, meta-externalagent, Bytespider, cohere-ai, Google-Extended, CCBot. CONTRAST — legitimate networks carry ONE identity: 216.73.216.0/24 -> ClaudeBot only (281 requests) -> RDAP: Anthropic, PBC 216.73.217.0/24 -> ClaudeBot only (265 requests) -> inside Anthropic published range (546 ClaudeBot requests across the two adjacent blocks; no other name appears) ================================================================================ 3. AMAZONBOT — cloud-native, hardest to verify ================================================================================ 344 "Amazonbot" hits from 288 distinct IPs. - 120 hits (35%) matched Amazon's published crawler IP list. - 224 hits (179 distinct IPs) did NOT match. FCrDNS on a sample of 18 unmatched IPs: - 18 / 18 resolved to generic ec2-*.compute-1.amazonaws.com (rented EC2). - 0 / 18 resolved to Amazon's official *.crawl.amazonbot.amazon domain. => the unmatched majority is not "new unlisted Amazon IPs"; it fails Amazon's own verification test. ================================================================================ 4. NO-METHOD CRAWLERS — unverifiable in principle ================================================================================ meta-externalagent (179 hits): reverse-DNS sample 9/9 = NXDOMAIN (no name). Bytespider (112 hits): observed source RDAP = China Unicom (CN); ByteDance publishes no official attribution method, so "Chinese ISP origin" is the furthest a third party can honestly go. ================================================================================ 5. WITHIN-SITE PAIRED — llms.txt is not being read ================================================================================ Same 17-day window, same site: GPTBot fetched our content pages ............... 310 times AI crawlers fetched our /llms.txt file .......... 0 times ================================================================================ Not shown separately (kept out of the commercial-AI tally): - user-triggered fetchers (human-initiated): ChatGPT-User 17, Claude-User 4, Perplexity-User 3 = 24. - research crawler (non-commercial): ClueWeb (CMU) = 106. - traditional search engines (Googlebot, Bingbot, YandexBot, Applebot, DuckDuckBot) are separate infrastructure and are not counted above. All dates are the "as of" of each operator's published list at time of check. Published lists change; a match is necessary, not permanent, proof. ================================================================================