The AI crawler your host is silently blocking

2026-09-29 · Bálint Szabó

If ClaudeBot is getting HTTP 429 responses from your site, the most likely cause is a user-agent-scoped rate limiter at your hosting provider, not your robots.txt. On the shared-hosting account of a German lab-glass manufacturer we work with, 682 of 1,177 ClaudeBot requests were answered with 429 between 25 August and 9 September 2026, while Amazonbot made 5,616 requests from 432 IP addresses in the same window and received none. The crawler was not blocked outright; it was told to slow down on the one file it must read first, robots.txt, and it obeyed. Nobody had looked at a log file to notice.

What we found in the logs

The site runs on a HostEurope WebPack shared-hosting tier. We exported 16 days of raw access logs (141,499 parsed request lines) and counted every AI-related user agent by status code.

Window: 25 Aug–9 Sep 2026, one shared-hosting account.

Two things stand out. The limiter is not volume-driven at the origin: Amazonbot, Googlebot and Bingbot each sent several times ClaudeBot’s traffic and were never throttled. And it is not a blanket block: single ClaudeBot requests do get through. Bucketing all ClaudeBot hits by requests-in-that-minute shows a clean dose-response, from 107 of 305 rejected at one request per minute to 89 of 100 rejected at five or more. That is a genuine rate limiter, calibrated so low that one request in an otherwise idle minute fails roughly a third of the time.

The 429 responses split cleanly by body size. The 162-byte variant (827 responses, mostly ClaudeBot) comes from the web-server layer. The 564-byte variant (182 responses) comes from a security plugin answering vulnerability probes and is working as intended. Only the first one matters here.

The robots.txt retry loop

The damaging part is where the rejected requests land. Of 1,177 ClaudeBot hits, 645 were for robots.txt or the sitemap index. robots.txt alone was fetched 348 times in 16 days (180 succeeded, 168 returned 429), roughly 22 times a day for a file a crawler is meant to fetch once a day and cache.

Under RFC 9309, most 4xx responses on robots.txt mean “unavailable, proceed”. 429 is the exception: crawlers treat it as an explicit instruction to reduce crawl rate. Google documents 429 as the way to tell its crawlers to slow down. So the site was not accidentally blocking ClaudeBot; it was repeatedly instructing it to back off, on the file it must read before doing anything else. A well-behaved crawler complies, retries later, gets another 429, and spends its whole visit budget in that loop instead of reaching content.

We saw the back-off in the data. In an earlier 14-day window (7–20 August 2026), ClaudeBot sent 1,554 requests and 1,250 came back 429; daily volume then fell from 249 hits on 25 August to a plateau of about 40 a day by 30 August, with the rejection rate holding near 55%. A falling 429 percentage on its own would look like improvement. It is the opposite: the crawler gave up.

What this means for AI visibility

Content a crawler cannot reach cannot be cited. For this client, eighteen months of content work had produced daily retrieval by ChatGPT-User (382 hits in the same 16 days, zero throttled, landing on real posts) and effectively no retrieval by Anthropic’s crawler. The difference had nothing to do with the writing. It was an access problem, and it was invisible from inside any SEO tool, because those tools do not read your server logs.

What to check in your own logs

1. Export at least 14 days of raw access logs. On many shared hosts retention is 14 days with daily rotation; anything older is gone.
2. Filter by user agent: ClaudeBot, GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, Amazonbot, Googlebot.
3. For each, count requests by status code. A 429 share above a few percent on any AI agent, while Googlebot sits at zero, is the signature of a UA-scoped limiter.
4. Look at which URLs the throttled agent is hitting. If most are robots.txt and the sitemap, it is stuck in the loop described above.
5. Check the 429 response body size. Two different sizes usually mean two different layers (host versus plugin).
6. Exclude your own tooling. If you have installed any connector or API endpoint that AI assistants call, its traffic will appear in the logs under an AI user agent and inflate your numbers.

The fix: serve robots.txt from the edge

Raising the limiter needs the host’s cooperation, and on a shared tier the honest expectation is that the support ticket will say no. The fix that does not depend on them is Cloudflare, on the free plan. With Cloudflare’s authoritative DNS in front of the site, robots.txt and the sitemap index are served from edge cache and never reach the origin rate limiter. That alone breaks the retry loop; content requests that follow are spread out enough to pass. It costs nothing per month, but it does require the client to move nameservers, which is a trust conversation rather than a technical one.

For this client, that change ranked above any new content in our remediation queue. It also became a standard step in every audit we run: pull the logs, census the AI crawlers, check status codes per agent, before writing a single word.

If you want to know which AI crawlers can actually read your site, the Access pillar of our audit answers exactly that. Start with the AI Visibility Audit at https://citehelm.com/audit.