Announcing Cloudflare pay per crawl beta

Cloudflare is introducing a new technical solution with pay per crawl to experiment with a way to help content creators be compensated for their contributions to the AI economy. Pay per crawl will let creators control access and get paid, ensuring AI companies can use quality content the right way — with permission and compensation. Over time, we expect this to continue to evolve and may incorporate a combination of different tools for publishers and creators.

With pay per crawl, publishers can block AI crawlers, allow specific ones, charge for access, or grant free access. They have full control over how their content is accessed.

AI crawler owners will be able to use pay per crawl to register, see pricing, and choose to pay or walk away — simple, transparent access to quality content.

Cloudflare powers one of the world’s largest networks, helping to manage and protect traffic for 20% of the web. Cloudflare handles trillions of requests daily, and thus has the world’s most advanced bot management solutions, accurately distinguishing between human users and AI crawlers. Cloudflare identifies and distinguishes AI crawlers through its sophisticated bot detection system, which combines machine learning, behavioral analysis, and fingerprinting. These tools accurately classify traffic and separate AI crawlers from search engine bots and other “good” bots. Cloudflare also runs one of the world’s largest Verified Bots programs, which allows it to detect AI crawlers with high precision and for bot operators to self declare the purpose or intent of their crawlers.

AI Audit lets users view and control crawler behavior on their sites. It is available now on all plans. Pay per crawl is a private beta feature of AI Audit that allows publishers to charge for crawler access

As a beta tester, you will get early access to pay per crawl features, allowing you to experience and provide feedback on pay per crawl. You will have the opportunity to shape the development and future of content access and monetization.

Apply for private beta access by filling out the form at the bottom of the page. Make sure to specify whether you're a publisher or a crawler.