On July 1, 2025, Cloudflare announced a significant shift in how websites can control automated scraping: new zones will, by default, block known AI crawlers unless site owners explicitly opt in. Alongside that default block, Cloudflare opened a private beta called Pay Per Crawl that lets publishers set fees for AI bots seeking access.
The change bundles several technical and policy tools , managed robots.txt, a Content Signals policy, honeypots, and payment plumbing , into what Cloudflare describes as a permissioned, enforceable approach to scraping. Given Cloudflare's global footprint, the move has immediate consequences across a substantial slice of the web.
What changed on July 1, 2025
Cloudflare made its policy public on July 1, 2025: by default new domains are asked whether they want to allow AI crawlers, and the company began inviting selected publishers into a private beta for Pay Per Crawl. The line feature is a default block for known AI crawlers on newly created zones unless site owners flip the setting.
Pay Per Crawl pairs the block with a pathway for permissioned access: bot operators who want to crawl a site must register, declare identity and intent, and support a payment flow. Cloudflare implemented an HTTP 402 "Payment Required" mechanism so that bots without payment intent receive 402 responses instead of ordinary content.
Cloudflare framed the change as a practical outgrowth of publisher demand and prior opt‑ins: over a million customers had already used anti‑AI crawler controls before the default flip, and managed robots.txt features had seen widespread uptake. The July announcement was also accompanied by documentation and developer guidance for sites that want to adopt the new controls.
Scale and why it matters
Part of what gives Cloudflare's move weight is scale: the company routes a large share of global internet traffic and serves millions of domains. Coverage commonly cites figures such as roughly 16% of global traffic and about 20% of websites or "two million+ customers," meaning a default block for new zones affects a meaningful portion of the public web.
Cloudflare used its telemetry to argue the old bargain , crawl in exchange for referral traffic , has frayed. Its Radar/Noise analyses showed dramatic differences in crawl‑to‑referral ratios between traditional search engines and many AI providers: Google crawled roughly 14, 18 HTML pages per referral in the sampled period, whereas some AI providers showed orders‑of‑magnitude higher ratios.
Those ratios are central to Cloudflare's justification for Pay Per Crawl: if a crawler takes thousands of pages without returning comparable referral traffic, sites argue they're bearing costs without receiving the commercial upside that search used to provide. Cloudflare stresses, however, that telemetry can be imperfect and that native apps, proxies, and missing Referer ers can affect ratios.
How Pay Per Crawl works
At the core of Cloudflare's market approach is an HTTP 402 flow: when a bot requests content but has not declared payment intent, the origin can return a 402 Payment Required. Cloudflare's system expects bot operators to register, provide identity/purpose declarations, and, in the beta, complete payments with Cloudflare acting as merchant‑of‑record.
The platform supports cryptographic signatures and request signing to reduce user‑agent spoofing, and Cloudflare says it will delist or block crawlers that try to evade detection. In the private beta, Cloudflare also handles the payment plumbing and billing, which simplifies adoption for publishers but creates an intermediary role that some companies have questioned.
Cloudflare positions Pay Per Crawl as a marketplace alternative to litigation or licensing negotiations, enabling publishers to set per‑crawl fees or require permissioned access. Early participants invited into the beta included large news and tech publishers, and the beta is intended to iterate as both technical and policy questions arise.
Detection tools: AI Labyrinth, managed robots.txt and Content Signals
Cloudflare has layered detection and deception into its strategy. In March 2025 it introduced AI Labyrinth, an opt‑in honeypot that injects AI‑generated decoy pages and invisible links to slow, confuse and fingerprint crawlers that ignore robots directives. These deceptive pages feed signatures into bot detection systems.
Cloudflare also expanded managed robots.txt capabilities and, on September 24, 2025, published a Content Signals Policy , a machine‑readable extension to robots.txt that lets operators declare preferences for "search", "ai‑input" and "ai‑train". The policy is intended to express fine‑grained opt‑outs or permissions for different downstream uses.
Crucially, Cloudflare emphasizes that Content Signals are preference signals, not guarantees: they work best when paired with bot management, WAF rules, and the company's detection stack, which combines user‑agent and ASN/IP analysis, behavioral fingerprinting, ML models, honeypots and curated bot signature lists.
Publisher adoption and marketplace dynamics
Several major publishers and platforms were named among early program participants and supporters. Cloudflare and press reports listed Condé Nast, The Atlantic, The Associated Press, TIME and Stack Overflow as participants in the early program, while sites like Reddit and Pinterest expressed interest in permissioned crawling.
For publishers, the attraction is straightforward: the ability to limit unfettered scraping or to extract direct revenue from AI firms that consume large amounts of content. The Pay Per Crawl model offers a technical marketplace that can sit alongside, or in some cases substitute for, bilateral licensing deals and litigation strategies pursued by other publishers.
That market logic has real traction but also raises questions about fragmentation: if many publishers set fees or block crawlers by default, the downstream cost and complexity for AI companies could rise, and smaller sites may face tradeoffs between openness and monetization.
Pushback, disputes and the legal context
The rollout has prompted pushback and public disputes. In August 2025 Cloudflare published analysis alleging Perplexity used "stealth, undeclared crawlers" that evaded robots.txt and spoofed user agents; Perplexity denied the claims, arguing Cloudflare conflated third‑party traffic or misinterpreted telemetry. The exchange escalated into a high‑profile back‑and‑forth covered across tech press.
OpenAI reportedly declined to participate in the initial preview, arguing that Cloudflare's intermediary payment and permission model adds a middleman between publishers and AI firms. Commentators and legal observers have warned the approach could create new gatekeepers and fragment the open web, or push disputes into courts if enforcement and definitions remain contested.
Separately, the move sits amid broader litigation and licensing activity: major publishers such as the New York Times and platforms like Reddit have pursued lawsuits or negotiated licensing deals with AI firms, and Cloudflare frames Pay Per Crawl as a complementary marketplace option to enable publishers to be compensated without relying solely on litigation.
Technical limits, caveats and what happens next
Cloudflare is candid about limits: crawl/referral ratios can be affected by native apps that don't emit Referer ers, third‑party proxies and other measurement artifacts. It stresses that managed robots.txt and Content Signals express preferences and must be paired with enforcement tools to handle adversarial actors.
Technically sophisticated crawlers can try evasion techniques , IP rotation, user‑agent spoofing, or routing via third parties , and Cloudflare says it will delist or block crawlers that attempt evasion. The company relies on signatures, ML detection, and honeypot fingerprints to identify misbehavior, but universal enforcement remains challenging.
As of October 27, 2025, Cloudflare's permission‑based approach , default block for new zones, Pay Per Crawl private beta, AI Labyrinth and the Content Signals Policy , has become an active multi‑tool strategy. The approach is shaping how publishers, AI firms and regulators think about access to training data and the economics of scraping.
Cloudflare Pay Per Crawl is now a live experiment at scale: it attempts to convert a technical enforcement problem into a marketplace negotiation. Whether it becomes the dominant model for balancing publisher control, AI development needs and the open web will depend on detection efficacy, legal rulings and how market participants respond.
For site operators, the new tools offer choice: block by default, open permissioned access, or monetize. For AI firms and researchers, they introduce potential costs and friction. The broader debate about gatekeeping, fairness and the future price of training data is likely to continue as the technology, law and business models evolve.
