Cloudflare has accused artificial intelligence company Perplexity of using hidden tactics to bypass restrictions designed to stop automated bots from collecting website data.
In a statement published Monday, Cloudflare said it had received multiple complaints from its customers claiming that Perplexity was still able to view and collect information from their sites, even though they had taken steps to block its activity. These blocks were implemented through a robots.txt file, a common tool that tells search engine bots which parts of a website they can or cannot access.
According to Cloudflare’s engineers, testing confirmed that Perplexity’s official crawler — the automated system responsible for scanning and indexing web content was being blocked as expected. However, the company claims Perplexity was also using other, less obvious methods to gain access to pages where it was not permitted.
As a result, Cloudflare said it has removed Perplexity from its list of verified bots and updated its own security rules to detect and block what it called “stealth crawling.” The company stressed that trustworthy crawlers should operate transparently, follow site owner instructions, and clearly state their purpose.
This dispute comes shortly after Cloudflare introduced new tools allowing website operators to either block AI crawlers completely or charge them for access. The move is part of a broader debate over how AI firms gather the large amounts of online data needed to train their systems.
When contacted by media outlets, Perplexity did not respond immediately. Later, company spokesperson Jesse Dwyer told TechCrunch that Cloudflare’s claims were exaggerated, describing the blog post as a “sales pitch.” Dwyer also argued that Cloudflare’s screenshots showed no actual data collection, and that one of the bots mentioned “isn’t even ours.”
Perplexity went further in its own blog post, criticizing Cloudflare’s actions as “embarrassing” and “disqualifying.”
The AI company has faced similar accusations before. Earlier this year, the BBC threatened legal action against Perplexity over claims it had copied its content without permission. Perplexity is one of several AI companies caught up in disputes over online data scraping, though some media organizations have instead chosen to sign licensing agreements with AI firms, including Perplexity.
As the tension between AI data gathering and online privacy grows, this case stresses upon the increasing push from technology infrastructure providers like Cloudflare to give site owners more control over how and whether, AI systems can collect their content.