How to add retry and rotation logic to your scraper

Failures are normal at scale. What separates a reliable scraper from a fragile one is how it reacts to them.

Classify before you retry

Response	Meaning	Action
200 with expected content	Success	Continue
200 with a challenge page	Soft block	New IP, do not count as success
403 / 401	Blocked	New IP, slow down
429	Rate limited	Back off hard, then new IP
5xx	Target error	Retry the same IP after a delay
Timeout / connection reset	Network	Retry, new IP after two failures

Retrying a 403 on the same IP just confirms the block.

Back off exponentially

Wait 1s, then 2s, then 4s, then 8s, with a little randomness added so parallel workers do not retry in lockstep. Cap the total attempts — three to five is usually right. Past that you are burning bandwidth on a page that is not coming back.

Rotate on the right signal

With a rotating endpoint you get a fresh IP automatically on the next request. On a sticky session, drop the session and start a new one when you hit a hard block; keeping it alive keeps you on the address that is already flagged.

Validate the body, not just the status

Anti-bot systems return 200 with a challenge page. Check for a marker you expect on a real page before recording success, or your success metric will be measuring the block page.

Log everything

Status code, elapsed time, exit country and a hash of the body. When a job's success rate falls, that log tells you in minutes whether the cause is the target, the location or your own code.

See also How to test the proxies using Python 3.