Earlier this year, web infrastructure provider Cloudflare launched a marketplace that allows websites to charge AI bots for scraping their content. Now, the company is pushing for increased regulation in the AI sector.
Cloudflare’s chief executive, Matthew Prince, is in London to speak with the U.K.’s Competition and Markets Authority. He is proposing stricter rules on how Google should be able to compete in the AI race, given its dominance in search. The CMA recently designated Google with a special status in the search and advertising markets due to its substantial and entrenched position. This move allows the regulator to impose more stringent regulations that could extend into areas like Google’s AI Overviews, AI Mode, the Discover feed, Top Stories, and News tab.
According to Prince, Cloudflare is in a good position to make recommendations because it is not in the AI business itself but has relationships with many AI companies. He stated that Cloudflare does not have a direct stake in the fight, as it is not an AI company or a media publisher, but is a network that sits between them. He added that 80 percent of the AI companies are Cloudflare’s customers.
Prince believes that Google should have to compete on the same footing as other AI companies, which he says is not happening today. Instead, Google uses its existing web crawler to gather content for its AI products and services in addition to its search engine. Prince claims this gives Google an unfair advantage. He explained that Google asserts a right to all content in the world without payment, based on its past activities over the last 27 years. Furthermore, Google uses the same crawler for both search and AI systems, meaning that if a publisher wants to opt out of one, they must opt out of both.
This situation is not feasible for most businesses, particularly those in media, where losing search traffic could mean losing about 20 percent of revenue. The problem is compounded because blocking Google’s crawler also blocks its ad safety team, which causes advertisements across all platforms to stop working. This makes blocking a non-starter for many publishers.
Because Google bundles its crawler, it gains access to content that other companies, like Anthropic, OpenAI, and Perplexity, would have to pay for. Prince warned that this could effectively hand the game to Google. The solution, he said, is to foster significant competition in the market where thousands of AI companies could compete to buy content from thousands of media businesses and millions of small businesses. He views the CMA’s action of flagging Google as a potential regulatory target as a thoughtful move that shows awareness of Google’s unique advantage.
Cloudflare has provided the CMA with data showing how Google’s crawler works and why it is nearly impossible for other players to replicate Google’s potential success. Prince is not alone in these opinions. Last month, Neil Vogel, the CEO of the largest digital and print publisher in the United States, which operates over 40 media brands, expressed similar concerns. In an interview, he called Google a bad actor, stating that media companies have no choice but to allow Google to crawl their sites for AI content because of the way the crawlers are combined.
Vogel, whose company adopted Cloudflare’s solution to block AI crawlers that do not pay, claimed the system was working. He said deal discussions were underway with several large large language model providers.