Why Your Scraper Gets Blocked: The 2026 Anti-Bot Stack

There is a specific moment that sends teams looking for a new proxy provider: the IPs are clean, the pool is large, the location is right — and the target still returns 403.

Changing provider almost never fixes it, because the IP was not what failed. Modern bot detection scores five independent layers, and a proxy only touches one of them.

Here is what each layer actually measures, in the order a request encounters them.

Layer 1 — TCP/IP

Before a single byte of TLS is exchanged, your operating system's TCP stack has already said something about itself: initial window size, TTL, maximum segment size, TCP options and their ordering. This is classic p0f-style passive fingerprinting, and it identifies the OS family.

It is weak on its own — millions of real users share a Linux TCP signature. It becomes useful in combination. A request whose TCP stack says Linux server while its User-Agent claims Windows Chrome has contradicted itself before the handshake even finishes.

What your proxy does here: a forward proxy re-originates the TCP connection, so the target sees the proxy's stack, not yours. This layer is genuinely handled for you.

Layer 2 — TLS, and why JA3 became JA4

This is where most scrapers are actually caught.

The first message in a TLS handshake is the ClientHello, and it is sent in the clear. It advertises the TLS version, the cipher suites the client supports, the extensions it offers, the supported groups and EC point formats — and the order of all of them. That combination is highly specific to the library that produced it.

JA3 concatenated those fields in order and hashed them with MD5. It worked well until Chrome 110 shipped extension-order randomisation in early 2023, which made every real Chrome session produce a different JA3 hash — destroying the fingerprint's stability for exactly the client you most want to imitate.

JA4, developed by FoxIO, fixed this by sorting ciphers and extensions before hashing, and splitting the output into a readable three-part structure:

• Part A — metadata in the clear: protocol, TLS version, whether SNI is present, cipher count, extension count, first ALPN protocol
• Part B — truncated SHA256 of the sorted cipher suites
• Part C — truncated SHA256 of the sorted extensions plus signature algorithms

Sorting is the whole trick: shuffling your extension order no longer changes the fingerprint.

How discriminative is this? A 2026 study trained gradient-boosted classifiers on 227,404 records from the JA4DB repository — 148,610 benign and 50,212 labelled malicious bots. A CatBoost model reached 98.63% accuracy, 96.70% precision, 98.00% recall and an AUC of 0.998 using TLS features alone. The single most discriminative feature was ja4_b, the sorted cipher hash.

That is the number to sit with. Before your scraper sends one HTTP header, a well-implemented TLS classifier can separate it from a browser with about 99% accuracy.

What your proxy does here: nothing. A forward proxy passes your TLS handshake through untouched. This is your HTTP client's fingerprint, and it is yours to fix.

Layer 3 — HTTP/2

Once the tunnel is up, HTTP/2 offers its own fingerprint, popularised by Akamai. It is built from:

• The SETTINGS frame the client sends on connection — which parameters, in what order, with what values
• The WINDOW_UPDATE delta
• Stream priority information
• The order of the pseudo-headers — :method, :authority, :scheme, :path

Real browsers are extremely consistent here, and HTTP libraries are consistently different. Pseudo-header order alone separates whole classes of client.

By 2026 the major vendors — Akamai Bot Manager, Cloudflare Bot Management, DataDome, HUMAN — score TLS and HTTP/2 fingerprints together. Matching Chrome's TLS signature while sending a Python library's HTTP/2 settings produces what practitioners call a "wrong-shape Chrome": internally inconsistent, and ranked as a bot precisely because of the inconsistency.

Layer 4 — Browser environment

If the target runs JavaScript challenges, the browser itself is interrogated: canvas and WebGL rendering output, installed fonts, screen dimensions, timezone, language, hardware concurrency, audio stack, and the presence of automation flags.

This is the layer where headless browsers get caught, and it matters because it is the layer TLS fingerprinting cannot reach. The 2026 study was explicit about its own limit: the method cannot detect bots driving a real Chrome binary through Puppeteer, Playwright or Selenium, because those produce a genuine Chrome handshake. Detection moves up the stack instead.

Consistency is scored here too, and it is where most homegrown setups fall apart. A German exit IP, an Accept-Language of en-US and a New York timezone are three answers to the same question that disagree.

Layer 5 — Behaviour

The slowest layer, and the hardest to fake: request timing and its variance, navigation order, whether assets are fetched, mouse movement and scroll patterns, session depth.

Requests at a metronomic 50ms interval, in perfect sequential order, never loading a single image, are not a browsing pattern. This layer is why "slow down and add jitter" remains the highest-yield fix in most real cases.

What this means for how you spend effort

Line the layers up against what a proxy actually controls:

• TCP/IP — fixed by your proxy
• IP reputation and ASN — fixed by your proxy
• TLS (JA3/JA4) — your HTTP client
• HTTP/2 fingerprint — your HTTP client
• Browser environment — your client
• Behaviour — your client

Two of six. A good proxy is necessary and nowhere near sufficient — which is why swapping providers so rarely resolves a 403 problem, and why the honest advice from a proxy vendor is that most of your remaining work is not in the proxy.

A diagnostic order that saves time

Classify before you change anything:

1. 429 — rate limiting. Slow down, add jitter, back off exponentially. Do not rotate harder.
2. 403 or a challenge on a fresh IP — fingerprinting. Changing IPs will not help. Fix the client.
3. 200 containing a challenge page — a soft block. If you are not validating the body, you are recording blocks as successes and your metrics are lying to you.
4. Works in a browser, fails in code, same IP — conclusively a client-side fingerprint problem.
5. Fails everywhere including the browser — now it is the IP or the location.

Only step 5 is a proxy problem. Our five-step proxy test isolates it in a couple of minutes, and why CAPTCHAs persist on good proxies covers the challenge case specifically.

What to actually change, in order of yield

1. Pace and jitter. Cheapest fix, largest effect, and it costs nothing but wall-clock time.
2. Header realism. A current User-Agent with matching Accept, Accept-Language and Accept-Encoding — in the order that browser actually sends them.
3. TLS stack. Use a client that can imitate a real browser's ClientHello. This is the single highest-impact technical change for a non-browser scraper, and standard library HTTP clients cannot do it.
4. Locale coherence. Align timezone, language and exit country. Free, and it eliminates a whole class of contradiction.
5. Session strategy. Stateful flows need a sticky session; stateless collection should rotate. Getting this backwards causes failures people misdiagnose as blocking.
6. Proxy type. Last, not first. If layers 2 to 5 are wrong, no IP will save you.

The uncomfortable conclusion

The industry sold "rotate your IPs" as the answer for a decade because for a decade it was. It is not any more. A TLS classifier can pick your scraper out at 98.6% accuracy before it sends a header, and no amount of IP diversity changes that number.

Get the client right and a modest, well-sourced pool goes a long way. Get the client wrong and the largest pool in the world just gives you more addresses to burn.

Our broader guidance is collected in scraping practices that keep success rates high, and if you want to work out which proxy type your target actually warrants, choosing a proxy type is a shorter path than trial and error. When you are ready to test against a real target, our scraping plans cover the layers we can.

Sources

• When Handshakes Tell the Truth: Detecting Web Bad Bots via TLS Fingerprints — 227,404 JA4DB records, CatBoost AUC 0.998
• JA3/JA4 fingerprint composition reference