Cloudflare has introduced a suite of new features designed to give website owners greater control over how AI crawlers access and use their content. These tools aim to address concerns from businesses that rely on advertising or subscriptions and want to prevent their material from being used by AI systems without compensation.
Starting September 15, Cloudflare will modify its default settings to permit AI-powered search crawlers while blocking training and automated agent use on pages supported by ads. Website owners will retain full control to adjust these preferences via a new dashboard interface. This approach balances openness for businesses willing to share content with protection for those seeking tighter restrictions.
The company is also rolling out an Attribution Business Insights dashboard, which tracks how much content AI bots consume and provides transparency on traffic generated by different AI firms. Additionally, Cloudflare is testing signals that notify AI crawlers when web pages have changed, aiming to reduce unnecessary data scraping. Another evolution is transforming its Pay Per Crawl model into Pay Per Use, enabling publishers to earn revenue proportional to the value AI systems generate from their content.
Cloudflare emphasizes that most internet traffic now originates from nonhuman sources like bots, making it imperative to establish sustainable practices for web content use. According to the company’s leadership, these enhancements offer website owners clearer visibility, new monetization opportunities, and support AI enterprises operating with transparent intent.
This announcement follows last year’s launch of Cloudflare’s initial AI crawler control system, which allowed site owners to decide on access and set pricing for AI content usage. The company highlighted positive ecosystem impacts stemming from that early initiative.