AI startup Perplexity is crawling and scraping content from websites that have explicitly indicated they don’t want to be scraped, according to internet infrastructure provider Cloudflare.
On Monday, Cloudflare published research stating it observed the AI startup ignore blocks and hide its crawling and scraping activities. The network infrastructure giant accused Perplexity of obscuring its identity when trying to scrape web pages in an attempt to circumvent the website’s preferences.
AI products like those offered by Perplexity rely on gathering large amounts of data from the internet. AI startups have long scraped text, images, and videos from the web, often without permission, to make their products work. Recently, websites have tried to fight back by using the web standard Robots.txt file, which tells search engines and AI companies which pages can be indexed and which shouldn’t. However, these efforts have seen mixed results so far.
Perplexity appears to be willingly circumventing these blocks by changing its bots’ user agent, a signal that identifies a website visitor by their device and version type. It also alters its autonomous system networks, or ASN, which identifies large networks on the internet, according to Cloudflare.
Cloudflare noted that this activity was observed across tens of thousands of domains and millions of requests per day. The company said it was able to identify the crawler using a combination of machine learning and network signals.
Perplexity spokesperson Jesse Dwyer dismissed Cloudflare’s blog post as a sales pitch, claiming in an email to TechCrunch that the screenshots in the post showed no content was accessed. In a follow-up email, Dwyer stated the bot named in the Cloudflare blog wasn’t even theirs.
Cloudflare said it first noticed the behavior after customers complained that Perplexity was crawling and scraping their sites, even after they added rules to their Robots file to block Perplexity’s known bots. Cloudflare conducted tests and confirmed that Perplexity was bypassing these blocks.
The company observed that Perplexity not only uses its declared user-agent but also a generic browser intended to impersonate Google Chrome on macOS when its declared crawler was blocked.
Cloudflare has since removed Perplexity’s bots from its verified list and added new techniques to block them. The company has recently taken a public stance against AI crawlers. Last month, Cloudflare announced a marketplace allowing website owners to charge AI scrapers for accessing their sites.
Cloudflare’s CEO, Matthew Prince, has warned that AI is disrupting the business model of the internet, particularly for publishers. Last year, the company also launched a free tool to prevent bots from scraping websites for AI training data.
This isn’t the first time Perplexity has faced accusations of unauthorized scraping. Last year, news outlets like Wired alleged that Perplexity was plagiarizing their content. Weeks later, Perplexity’s CEO, Aravind Srinivas, was unable to immediately define plagiarism when asked during an interview at the Disrupt 2024 conference.