Cloudflare Wants Google to Let Sites Block AI without Losing Search Visibility

Cloudflare wants Google to let sites block AI without losing search visibility

Cloudflare recently announced a new "pay-per-crawl" system aimed at pushing back against AI companies that continue to scrape the open web without paying a cent. According to CEO Matthew Prince, the effort should be supported by the very companies doing the crawling.Read Entire Article

Cloudflare's innovative "pay per crawl" lets publishers charge AI crawlers for content

Cloudflare has launched a new "pay per crawl" feature in private beta that allows content creators to monetize their work by charging AI crawlers for access, offering a middle ground between blocking AI entirely or allowing free access.The system leverages HTTP response code 402 (Payment Required) and established web authentication mechanisms. When AI crawlers request content, they either present payment intent through request headers for succes…

Will IT turn the AI bot battle into a money maker? (And is that even a good idea?)

Enterprise IT has been waging a difficult battle against generative AI (genAI) bots and crawlers, and banning some crawlers has simply not worked. That’s led to both data leakage and painful out-of-pocket bandwidth costs. Cloudflare is one of the few companies that has had some success in blocking these unwelcome visitors, although some have complained that by blocking most of the crawlers, Cloudflare is also filtering out some legitimate traffi…

Pay per Crawl vs. Google AI Mode: The battle for search results has already begun

Cloudflare, the web infrastructure company, recently launched “Pay per Crawl“, an experimental platform that allows publishers to charge for each visit made by an artificial intelligence bot to their websites. This initiative is a response from publishers to the Google AI Mode and other search engines such as OpenAI’s ChatGPT Search. ALSO READ. How does Google AI Mode affect SEO? A study reveals its real impact on SEO “Pay per Crawl” proposes a …