Bot Detection Headless Browser: How to Spot a Block

When Your Scraper Doesn't Know It Got Blocked

A headless browser request comes back with HTTP 200 and a full HTML body. Your pipeline marks it as success, extracts whatever fields it can find, and moves on. The problem: that HTML was a CAPTCHA page, or an interstitial telling a human to "verify you are not a robot." Nothing crashed. The job just quietly ingested garbage.

This is the failure mode most scraping pipelines never plan for. Teams build careful retry logic for timeouts and connection errors, but treat "got a response" as synonymous with "got the page." Bot detection scenarios rarely announce themselves with a clean error — they hide inside 200 responses, inside HTML that looks structurally valid but contains none of the content you asked for. Interpreting a response correctly is a distinct skill from evading detection or handling retries. It's the classification layer that decides whether everything downstream — parsing, storage, analytics — is working with real data or noise.

The Three Response Patterns: Rejected, Challenged, and Faked

Bot mitigation systems don't respond in one consistent way, and lumping every anomaly into "blocked" hides useful information. There are three distinct patterns worth separating.

Hard rejection at the edge. The request never reaches the origin in any meaningful sense — it's stopped by a CDN or bot-management layer and answered with a definitive status like 403, 429, or a Cloudflare-specific 1020. No page, no partial content, just a refusal.

Interactive challenge. The response might be a 200 or a 403, but the body is an interstitial — a JavaScript challenge, a CAPTCHA widget, or a "Just a moment" holding page. It's designed for a browser to solve, not for a script to parse. Treating this as content means storing challenge markup as if it were your target data.

Soft block. The trickiest pattern: a genuine 200, real-looking page structure, but the content is thin, decoy, or missing the elements you expect — pagination stripped out, prices replaced with placeholders, or an empty product list where fifty items should be. Nothing here looks like an error. Each pattern breaks a pipeline differently — a hard rejection at least fails loudly, while a soft block fails silently and corrupts your dataset without a single alert firing.

Status Codes and Headers Worth Checking

Status codes are the first, cheapest signal, but they need correct interpretation, not just presence-checking.

• 403 Forbidden usually means a bot-management layer or WAF rule made a fingerprinting decision and rejected the request outright. It's about who is asking.
• 429 Too Many Requests signals rate limiting — the identity might be fine, but the request volume tripped a threshold. It's about how often.
• 503 Service Unavailable is worth treating with more suspicion than it used to be. Historically, Cloudflare served 503s for JavaScript challenges; as of the platform's 2023 changes, challenges are far more likely to arrive as 403s, so a 503 today more often reflects a genuine origin or upstream failure rather than an anti-bot response. The Cloudflare status code breakdown from Stat Proxies covers this shift and what 403/429/503/530 mean specifically behind Cloudflare.

Headers narrow down which system made the decision, which matters because mitigation differs by vendor:

Signal	Likely source
cf-ray, cf-mitigated, server: cloudflare	Cloudflare Bot Management
x-datadome-* headers or cookies	DataDome
akamai-grn	Akamai Bot Manager
PerimeterX/HUMAN cookies (_px*)	PerimeterX/HUMAN Security

The cf-mitigated header in particular is a direct tell — its presence confirms Cloudflare intervened, independent of what status code accompanied it. Vendor header and cookie signatures across Cloudflare, DataDome, Akamai, and Kasada are documented in more depth in Scrapfly's anti-bot protection guide, a useful reference when building a signal-to-vendor mapping table for your own logging.

Body-Level Signals When Headers Don't Help

Plenty of soft blocks arrive with completely clean headers — no vendor cookie, no diagnostic header, just a 200 and HTML. This is where response bodies need their own heuristics.

Compare content-length or DOM node count against a stored baseline for that URL or domain. A product page that's normally 80KB of rendered HTML dropping to 6KB on a given request is a strong signal, even with no other clue. Search the body for known interstitial markers: script tags referencing challenge platforms, CAPTCHA vendor widget markup, or title/meta text like "Just a moment," "Access denied," or "Verify you are human." Also check whether the extracted text length or expected selectors (price, title, listing count) resolve to nothing — a page that renders visually but yields zero matches on your normal selectors is behaving like a decoy, not a genuine empty result. Headless Chromium sessions can be soft-blocked at a meaningful rate even when nothing in the transport layer looks wrong, which is why body-level checks aren't optional — they catch what headers can't, as detailed in Anakin.io's writeup on Cloudflare-blocked automation.

A Practical Classification Pattern

A response classifier doesn't need to be elaborate — it needs to run in a fixed order and stop at the first confident match:

if status in (403, 429, 1020):
    if vendor_headers_present(headers):
        return ESCALATE  # fingerprint/IP-based block
    else:
        return RETRY      # possible rate limit, backoff first

if status == 503 and not vendor_headers_present(headers):
    return RETRY  # likely origin issue, not a challenge

if body_contains_challenge_markers(html):
    return ESCALATE

if content_length(html) < baseline * threshold or selectors_empty(html):
    return ESCALATE  # soft block

return OK

RETRY implies backoff and possibly a proxy swap before trying again — the mechanics of that are covered separately in Headless Browser Timeout Handling: A Resilience Pattern. ESCALATE on IP-reputation-driven 403s often means it's time to reconsider the request path entirely, which is where Rotating Proxies vs Browser Sessions: A Scraper's Framework is the relevant next read.

What Browsevra Returns So You Don't Have to Guess

Every render call through Browsevra returns structured metadata alongside the HTML: the actual status code returned by the origin or edge, the full response header set, and the final rendered document — so vendor headers like cf-ray or akamai-grn aren't buried behind a browser abstraction, they're just fields on the response object. That means the classification pattern above can be wired in directly, checking response.status, response.headers, and response.html without touching raw sockets or manually screenshotting pages to guess what went wrong. Full field-by-field schema detail lives in the Browsevra docs.

Frequently Asked Questions

How do I know if my headless browser request was blocked instead of successful?

Check three things in order: the HTTP status code, known vendor headers (like cf-mitigated or akamai-grn), and body-level signals such as content length versus a stored baseline or the presence of CAPTCHA/interstitial markup. A 200 status alone is not proof of success — many blocks arrive disguised as valid responses.

What's the difference between a 403 and a 429 in terms of what caused the block?

A 403 typically means a bot-management or WAF layer rejected the request based on identity or fingerprint signals — a decision about who's asking. A 429 means the request volume from that identity tripped a rate limit — it's about how often, not who, and usually resolves with backoff rather than a fingerprint or proxy change.

Can a bot-blocking page return HTTP 200 instead of an error code?

Yes, and it's one of the most common ways pipelines get silently corrupted. Interactive challenges and soft blocks frequently return a 200 status with a fully-formed HTML body that contains a CAPTCHA, an interstitial, or decoy content instead of the real page.

What headers indicate Cloudflare, DataDome, or Akamai specifically blocked the request?

Cloudflare exposes cf-ray, cf-mitigated, and server: cloudflare; DataDome uses x-datadome-* headers and cookies; Akamai sets akamai-grn. Their presence confirms which vendor's system intervened, independent of the status code returned alongside them.

How do I detect a 'soft block' where the page loads but the content is fake or empty?

Compare the response's content length or DOM structure against a stored baseline for that page, and check whether expected selectors (price, title, listing items) return empty. A page with clean headers and a 200 status that's dramatically thinner than usual, or missing the fields you always extract, is behaving like a soft block even without any explicit error signal.

Response classification shouldn't require screen-scraping your own screenshots to figure out what happened. Browsevra returns status code, final URL, render time, and full response headers on every call, so the logic in this article drops into a pipeline in a few lines instead of guesswork. Check the response schema in the docs, or head to pricing to test it against a real protected URL — or start at browsevra.