Web scraping best practices that keep your success rate high

A good proxy pool solves the network layer. Most blocks that survive it are caused by how the requests themselves are made.

Pace yourself

Sites profile timing. Fire requests at a constant 50ms interval and you have identified yourself before the first page loads. Add jitter, back off after errors, and keep per-IP request rates in a range a person could plausibly produce.

Send believable headers

• A current, real User-Agent — not a five-year-old string and not the default from your HTTP library
• Accept, Accept-Language and Accept-Encoding consistent with that browser
• Header order matching what the browser actually sends, which several anti-bot systems check

Keep sessions coherent

If you are logged in, keep the cookie and the IP together on a sticky session. Do not mix one account's cookies across many IPs at once.

Respect the site

Read robots.txt, avoid endpoints it disallows, stay off login-protected data you have no right to, and never push enough volume to degrade the service for real users. See What sites are not allowed using Limeproxies.

Handle failures properly

• Retry 429 and 5xx with exponential backoff, not immediately
• Treat a 403 as a signal to change something, not to retry harder
• Log the status code and body of every failure so you can tell a block from a bug

Take less

Use the JSON endpoint if there is one, request only the fields you need, and cache aggressively. Every request you avoid is one that cannot be blocked.

Related: How to crawl a website without getting blocked or misled (cloaked).