Cloudflare implements a new model for websites to​

How it works

Block AI bots by default

Starting July 1, 2025, all websites using Cloudflare will block AI bots (e.g. OpenAI, Google Gemini...) unless the site owner actively allows it.

Register API bot

The AI bot must register with Cloudflare, declare information, set up authentication (Web Bot Auth – Ed25519 key), and send a confirmation HTTP header when requesting:
If the bot does not authenticate or does not pay a fee → returns HTTP 402 Payment Required with crawler-price header to notify the price.

Pay fees in two ways

Reactive: The bot tries to access, get 402 with crawler-price, then send it back with crawler-exact-price agreeing to pay.
Proactive: Bot immediately sends crawler-max-price; If max ≥ costs the site owner, Cloudflare pays the content and adds crawler-charged to HTTP 200

Payment & Accounting

Cloudflare plays the role of collecting money from bots and returning it to website owners, content is tracked according to crawls using micropayments
CryptoSlate

Why does Cloudflare do this?

Protect content ownership: AI bots are now "sucking out" the web without paying or redirecting users back to the content source page, causing damage to publishers.
Establish a new economic model: Create a marketplace where website owners can control or monetize data, balancing the ecosystem between publishers and AI.
Enhanced Security & Transparency: Cloudflare also deploys AI Labyrinth to deceive bots and tools to identify AI through suspicious behavior.

Who benefits?

Large publishers such as The Atlantic, AP, Condé Nast, The Atlantic, Time, Gannett, Stack Overflow... participated in using this tool or agreed with it.
AI bots that want to access these sites need to be authenticated, licensed, or paid, or will be blocked.

Briefly

Cloudflare has set a new threshold for AI bots: pay or not use data. This is a turning point in protecting online content, while opening up a new market where bots will have to negotiate and pay to collect data.

--------------

Block AI bots in Cloudflare

To enable or disable AI bot blocking in Cloudflare, follow these steps:

1. Access Cloudflare Dashboard
Sign in to your Cloudflare account
Select the website you want to configure

2. Visit the Bot Management section
In the left navigation bar, select Security > Bots (or Bot Fight Mode if you use the free plan)

3. Enable AI bot blocking
If you use Cloudflare Bot Management (in the Pro, Business, or Enterprise plan):
Find the AI bot protection option
Enable Block AI bots so Cloudflare blocks identified AI bots like OpenAI, Anthropic, Google Gemini...

If you use Bot Fight Mode (free plan):
Enable Bot Fight Mode to prevent bad bots in general, including some AI bots

4. Customize using WAF (Firewall Rules)
Go to Security > WAF > Firewall Rules
Create a new rule with the following conditions:
Field: User Agent
Operator: contains
Value: specific bot name (eg. GPTBot, Anthropic, Claude, Google-Extended)
Action: Block or Challenge

5. Check access logs
Go to Security > Events to see which bots are being blocked or accessed

6. Turn off AI bot blocking
Turn off options like Bot Fight Mode
Fix or delete firewall rules that are blocking AI bots