The 403 that isn't an outage — and why AI agents get it too

We probe a few hundred services on a schedule. A meaningful share of those probes come back 401, 403, or 429 — and none of those responses mean the service is down.

They mean a bot shield looked at our request and declined to answer. The service behind it is almost always fine. Users are sailing through. We just aren't a user.

This distinction is load-bearing for us: a monitoring tool that reports "Cloudflare said 403" as "the service is down" is a tool that cries wolf, and a tool that cries wolf gets ignored precisely when it matters. So our scoring engine neutralises those three status codes entirely — they're recorded as probe-blocked, never counted toward an outage score.

Why a 403 is not an outage

An HTTP status code answers a specific question, and it's important to be precise about which one.

• 500, 502, 503, 504 — the server tried and failed. This is an availability signal.
• 401, 403 — the server understood you and refused. This is an authorisation signal.
• 429 — the server understood you and is throttling you. This is a rate signal.

Only the first group says anything about whether the service works. The other two say something about your relationship with the edge, and an automated client's relationship with the edge is not the same as a browser's.

Anti-bot systems make this determination from a bundle of signals: the user-agent string, TLS fingerprint, IP reputation and ASN, header ordering, whether you execute JavaScript, and how quickly you request things. A server-side monitor fails several of those tests by construction. It runs from a datacenter IP, it doesn't render JavaScript, and it announces itself honestly in the user-agent.

That last one is worth sitting with. Being honest about who you are is itself a bot signal. A well-behaved crawler that identifies itself is easier to block than one that pretends to be Chrome.

The same rule blocks AI agents

Here's the part that has changed recently.

For years the traffic hitting those bot rules was search crawlers, monitors, scrapers, and attackers. The calculus was simple: allow Googlebot, block most of the rest, don't think about it again.

Now a fourth category shows up in the same bucket — AI crawlers and agents fetching pages on behalf of a user who asked a question. They tend to arrive with identifiable user-agents:

• GPTBot — OpenAI's crawler
• ChatGPT-User — fetches made when a ChatGPT user's request requires browsing
• ClaudeBot — Anthropic's crawler
• PerplexityBot — Perplexity's crawler
• Google-Extended — a robots.txt token governing Google's AI usage rather than a distinct fetcher

Most default bot-management configurations were written before these existed. If your rule is "allow known search engines, challenge everything else," these agents get challenged — which for a non-browser client means blocked.

And the failure mode is completely silent. A blocked monitor retries and eventually pages someone. A blocked agent does neither. The user asked a question, your site returned a challenge page, and the answer got assembled from whoever did respond. You never see an error, because from your edge's perspective nothing went wrong — the rule worked exactly as configured.

Test it yourself in a minute

You don't need a tool to get a first read. Compare a browser-shaped request against an agent-shaped one:

# Browser-shaped
curl -sS -o /dev/null -w '%{http_code}\n' \
  -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0 Safari/537.36' \
  https://example.com/
 
# Agent-shaped
curl -sS -o /dev/null -w '%{http_code}\n' \
  -A 'ChatGPT-User/1.0 (+https://openai.com/bot)' \
  https://example.com/
 
# Crawler-shaped
curl -sS -o /dev/null -w '%{http_code}\n' \
  -A 'GPTBot/1.0 (+https://openai.com/gptbot)' \
  https://example.com/

If the first returns 200 and the others return 403, 429, or a 503 challenge, you have your answer.

Three follow-ups worth running:

1. Check robots.txt before blaming the WAF. A stray User-agent: GPTBot / Disallow: / — sometimes added years ago during an AI-scraping panic, sometimes inherited from a template — is the most common cause and the easiest fix.
2. Fetch without JavaScript. curl a product or article URL and read the raw HTML. If the meaningful content only appears after hydration, an agent that doesn't execute JS sees an empty shell even when it gets a clean 200. That's a different failure than a block, with the same outcome.
3. Test from a datacenter IP, not your laptop. Residential IPs have far better reputation. Running the same curl from a cheap VPS often reproduces a block you can't reproduce at home.

What to actually fix

In rough order of effort-to-payoff:

• Audit robots.txt for AI tokens and decide each one on purpose. Blocking training crawlers while allowing user-initiated fetchers like ChatGPT-User is a coherent position; blocking both by accident is not.
• Allowlist user-initiated agents in bot management if you want to be reachable in AI answers. These fetch a page because a human asked for it — closer to a visitor than a scraper.
• Serve meaningful HTML without JavaScript on pages that matter. Server-render the content, price, and availability. This helps agents, search crawlers, and users on bad connections simultaneously.
• Keep structured data accurate. Agents lean on it heavily precisely because it's unambiguous.
• Return honest status codes. If you're throttling, 429 with a Retry-After is enormously more useful than a 200 containing a challenge page — which is indistinguishable from real content to anything that isn't a browser.

The connection back to status

We keep coming back to the same principle from opposite directions. A monitor that reads 403 as "down" is wrong. A site that returns 403 to a legitimate agent is also producing a wrong answer — it's telling a system "you may not look" and the system reports back "I couldn't find it."

Both are failures of interpretation at the edge, and both are invisible unless you go looking.