For most of the web's history, the default answer to a crawler was yes. Publish something, and anyone could fetch it; robots.txt was a polite request that well-behaved crawlers honoured and nobody enforced.
That default has now flipped, and it flipped fast. If your team collects web data to train, fine-tune or retrieve for a model, the ground moved underneath the pipeline you built two years ago.
What actually changed
The economics stopped being reciprocal
The old bargain was implicit: crawlers took content and sent back visitors. Search worked because both sides got something.
Cloudflare published crawl-to-referral ratios in June 2025 that made the collapse of that bargain measurable — pages crawled per referral sent back:
- Google — 14 : 1
- OpenAI — 1,700 : 1
- Anthropic — 73,000 : 1
Google's ratio is a trade. The others are, from the publisher's side, a one-way transfer. Once you can see that number, blocking stops being an ideological position and becomes an operating decision — which is exactly how site owners started treating it. Cloudflare reported AI bot traffic growing 65% in six months over the same period.
Blocking became one click, then became the default
Cloudflare shipped managed robots.txt: a setting that intercepts requests for /robots.txt and serves directives asking AI operators not to use the content for training, prepended to whatever the origin already had. It named Google-Extended and Applebot-Extended among others, deliberately leaving standard search crawlers alone so SEO was unaffected. More than a million customers enabled it, and it was available on every plan including the free tier.
Then on 1 July — what Cloudflare called Content Independence Day — the default flipped to blocking AI crawlers outright unless they compensate creators.
Refusal became billable
Pay per crawl revived HTTP 402, Payment Required, a status code that sat unused in the spec for decades. A crawler either presents payment intent in its request headers and gets a 200, or it gets a 402 quoting the price. Cloudflare has since generalised the idea into a monetisation gateway covering APIs, data and MCP tool calls.
The important shift is not the money. It is that refusal is now explicit, machine-readable and categorised — and site owners increasingly distinguish between crawlers that train on their content, crawlers that index it for AI search, and agents fetching it live for a user right now. Most will happily allow the last two while refusing the first.
What this means for your pipeline
1. Your source list is now a compliance surface
Two years ago, "we crawl the public web" was a sufficient description of provenance. It is not any more. If a source has published an explicit, machine-readable refusal and you collected it anyway, that is a fact about your dataset that will surface — in a vendor security review, a customer's procurement questionnaire, or a regulator's question.
Keep provenance per source: what you took, when, under what stated terms, and what the robots directives said at the time. Retrofitting that record later is close to impossible.
2. Distinguish training from retrieval, because the web now does
These are different activities with different acceptability, and lumping them together costs you access you could have had:
- Training — bulk, one-off, content leaves and nothing returns. The most restricted, and the hardest to justify against an explicit refusal.
- Retrieval and RAG — fetching a handful of documents to ground an answer. Much smaller volume, and increasingly tolerated because it can carry attribution.
- Agentic browsing — a live fetch on behalf of a user who asked a question. Closest to a human visit, and the category site owners are most willing to permit.
If your system does all three through one undifferentiated crawler, you inherit the permissions of the most restricted category for everything.
3. Volume discipline is now a feature, not an optimisation
When the whole web was open, over-crawling cost you bandwidth. Now it costs you access: aggressive crawling is what gets you categorised as a training crawler and blocked for every purpose, including the ones that would have been allowed.
The practical levers are the same ones that always saved money and now also preserve access — conditional requests with ETag and Last-Modified so unchanged pages return an empty 304, blocking assets you never parse, preferring a JSON endpoint over a rendered page, and caching anything fetched more than once. Our note on reducing bandwidth covers the mechanics.
4. Licensed and cooperative sources deserve a second look
The reflex is to treat licensing as expensive and crawling as free. With per-crawl pricing and blocking as the default, that arithmetic has changed. For a source you depend on continuously, a licence buys stability, a support contact and a provenance record — three things a crawl cannot give you at any volume.
Crawl the long tail. License or partner for the sources you cannot afford to lose.
Where proxies fit, and where they do not
We sell proxies, so let us be direct about the boundary.
A proxy is not a licence. Changing your network path does not change whether a site owner consented. A robots.txt disallow is a refusal whether or not anyone enforces it, and using proxy capacity specifically to circumvent an explicit refusal is outside our acceptable use policy, as it is with any provider worth using.
What proxies legitimately solve in an AI data pipeline:
- Geographic accuracy. If your model needs to know what a page looks like in Germany, you have to fetch it from Germany. Our 100+ country coverage exists for exactly this.
- Rate distribution. Spreading permitted collection so no single address behaves unnaturally — courtesy as much as capability.
- Reliability at volume. Long-running jobs need addresses that stay up, and a support channel when they do not.
- Separating workloads. Keeping a live agentic fetch off the same address range as a bulk crawl, so one does not poison the other's reputation.
What they do not solve: consent, licensing, provenance, or the fact that a 402 means the content has a price.
Choosing capacity for AI workloads
The most common and expensive mistake is buying residential bandwidth for sources that would have served a datacenter IP without complaint. The AI use case downstream does not change what the target checks.
- Public APIs, documentation, open datasets — datacenter, faster and billed per IP
- Defended consumer sites such as retail, travel and social — residential, rotating
- Geo-specific rendering for eval sets — residential in the target country
- Live agentic fetches at low volume — datacenter, unless the target rejects it
- Anything needing a persistent identity — ISP proxies
If you are unsure, measure rather than guess: run a few hundred real requests against your actual target on the cheaper option and look at the success rate. That answers it faster than any specification sheet. Choosing a proxy type walks through the decision.
What a defensible pipeline looks like in 2026
- A source register. Every source, its terms, its robots directives at collection time, and why you believe collection was permitted.
- Separated crawlers for training, retrieval and agentic fetching, each with its own rate profile and identification.
- Honest identification. A user agent that says who you are and a contact address. Disguising a training crawler as a browser is the behaviour that made blanket blocking the default.
- Conditional fetching everywhere — 304s cost nobody anything.
- Right-sized proxy capacity, with residential reserved for targets that genuinely require it.
- A stop rule. When a source starts returning 402 or blocks you outright, that is an answer. Route it to licensing or drop it — do not escalate technically.
The honest summary
The era of collecting the open web by default is over, and it ended for a reason that is legible in Cloudflare's own numbers: at 73,000 crawled pages per referral, the reciprocity that made open crawling tolerable simply was not there.
That does not make web data collection for AI impossible. It makes it a discipline — one where knowing your sources, respecting explicit refusals and sizing your infrastructure honestly matter more than raw crawl capacity. Teams that build that discipline now will still have working pipelines when the next default flips.
If you want help sizing proxy capacity for a collection workload — including being told you do not need residential — ask us. We would rather scope it correctly than sell you bandwidth you will not use.
Sources
- Cloudflare, Control content use for AI training with managed robots.txt — crawl-to-referral ratios, June 2025
- Cloudflare, Introducing pay per crawl — the HTTP 402 mechanism
- Cloudflare, Content Independence Day
Post Quick Links
Jump straight to the section of the post you want to read:
About the author
LimeProxies Editorial Team
We are the engineers and support staff who run the LimeProxies network — the people who provision residential, datacenter, ISP and mobile capacity across 100+ countries and answer the tickets when something breaks. We write about what we see from that seat: why requests get blocked, how detection actually works, and what the numbers say rather than what marketing would prefer. Every claim that comes from research is linked to its primary source so you can check it yourself, and we will tell you when the honest answer is that you do not need what we sell.
View all postsRelated Articles
Are Free Proxies Safe? What 53 Million Exit Nodes Reveal
Free proxies are not free — they are paid for with someone's compromised device, and often with your data. Independent 2026 research measured how much of the free proxy supply runs on malware-infected hardware, and the number is worse than most teams assume.
Why Your Scraper Gets Blocked: The 2026 Anti-Bot Stack
If you are still getting 403s on clean residential IPs, the IP is not the problem. Modern bot detection scores five independent layers, and four of them live in your HTTP client — not your proxy. This is what each layer measures and what you can actually do about it.
Datacenter vs Residential Proxies: Which Should You Choose in 2026?
Datacenter proxies are faster and cheaper for most tasks. Residential proxies handle heavily bot-protected sites. This guide breaks down every difference so you pick the right type — and avoid overpaying.