Cloudflare have recently announced that they’ll be blocking AI crawlers from accessing websites that use its services.
This is important for us – and for you, if you’re a Splitpixel client – because we use Cloudflare for CDNs and security on the vast majority of our websites. By default, your site will be blocked from AI crawls, unless we specifically allow them.
What’s happening?
Cloudflare will now block AI crawlers from accessing websites by default. On top of this, they’re testing a new initiative called Pay Per Crawl, which allows AI companies to pay for access to your website content.
These AI crawlers include bots from platforms such as OpenAI (ChatGPT), Google AI, and other large language model (LLM) tools that automatically crawl web content to train its AI systems and improve the responses they generate.
Why is this important?
According to Cloudflare, “AI Crawlers generate more than 50 billion requests to the Cloudflare network every day.”
With the rise of AI tools, LLM (Large Language Model) traffic is impacting traditional search traffic, and user behaviour is changing rapidly.
According to a recent SEMrush study, one key finding was that AI search visitors are projected to surpass traditional search visitors by 2028.
In addition:
- ChatGPT’s weekly users rose over eight times between October 2023 and April 2025, reaching over 800 million.
- Google has also started rolling out AI Mode, a feature that operates in a similar way to that of ChatGPT, providing a more conversational search experience that will change the traditional search results page.
This move from Cloudfare is designed to give website owners more control over how their content is used, and even offer the chance to monetise access to that content. Recently, many content creators feel their work is being taken with no control and without permission, so this is an attempt to put a stop to this.
However, the flipside is that blocking AI crawlers will limit the ability for businesses to be returned as options in AI searches.
What does “Pay Per Crawl” mean?
In a recent Cloudflare article, they explained:
“Pay per crawl, in private beta, is our first experiment in this area.
“Each time an AI crawler requests content, they either present payment intent via request headers for successful access (HTTP response code 200), or receive a 402 Payment Required response with pricing.
“They can define a flat, per-request price across their entire site. Publishers will then have three distinct options for a crawler:
“Allow: Grant the crawler free access to content.
“Charge: Require payment at the configured, domain-wide price.
“Block: Deny access entirely, with no option to pay.”
To find out more or sign up for the beta, you can do so here.
What should you do?
The choice is entirely yours and depends on whether you want AI bots like ChatGPT to access and use your content, for free (allow), for a fee (charge model – coming soon), or not at all (block).
A few pointers that might be useful to consider when making your decision:
Advantages of Allowing AI Crawlers
- Visibility – Not allowing AI bots to crawl your content could reduce your brand’s/organisation’s visibility across LLMs, whether that’s from link clicks or simply brand awareness from featuring within AI results.
- Growing trend – More users are now using AI tools to conduct their online searches. Blocking AI tools could mean missing out on this growing search behaviour.
Advantages of Disallowing AI Crawlers
- Content protection – You retain full control over your site’s content, reducing the risk of duplication and brand misrepresentation.
- Website performance – Reduces server load to help ensure it’s prioritised for real human visitors.
Summary
Cloudflare’s new AI bot blocking and Pay Per Crawl initiative helps put control of content back into your own hands. Whether you choose to allow, charge, or block these bots, you will now have a way to manage how AI and LLMs interact with your website.
We will be contacting all of our affected clients in due course to ask whether you’d like us to turn it on or off – but if you read this first and would like us to take action, please drop an email to our wonderful account manager El at el@splitpixel.co.uk. They will make sure this gets sorted for you!
