AI crawler rules, WAF controls, and content monetization for modern websites.
Cloudflare helps content sites manage AI crawler traffic with Search, Agent, and Training controls, WAF rules, AI Crawl Control, and Pay Per Crawl / Pay Per Use monetization paths.
Opens official website
Live change feed
Shared with the homepage and tool pages
Agents can pay for HTTP resources and MCP tools through HTTP 402 using x402 or MPP. The client can require human approval before payment, and no pre-created service account or shared API key is required for the exchange.
Verify official source →Content sites need explicit Search, Agent, and Training crawler rules.
Verify official source →On August 6, 2026, Cloudflare open-sourced Cloudflare OS, an agentic platform providing each employee with an AI workspace based on company context and skills. It includes an isolated runtime, security governance framework, and shareable personal apps. Cloudflare also released identity-aware AI Gateway and User Insights, binding every AI request to a user identity and detecting anomalous sessions. An internal version has been used by thousands of Cloudflare employees daily.
Cloudflare is a web performance, security, and edge platform that has become increasingly important for AI-era content sites. In July 2026, Cloudflare launched new AI traffic options that let site owners manage AI crawlers by purpose: Search, Agent, and Training. Starting September 15, 2026, new domains onboarding to Cloudflare receive updated defaults: Training and Agent crawlers are blocked on pages that display ads, while Search remains allowed. Cloudflare also continues to develop AI Crawl Control, WAF integration, and monetization experiments such as Pay Per Crawl and the broader Pay Per Use direction. For publishers, SaaS companies, documentation sites, and SEO-driven content businesses, Cloudflare is now part of the AI visibility and content protection stack, not just a CDN or firewall. On August 6, 2026, Cloudflare open-sourced Cloudflare OS — an agentic platform that gives every employee an AI workspace grounded in company context and skills. It includes an isolated runtime, a security governance framework, and shareable personal apps. An internal version has been used daily by thousands of Cloudflare employees. Cloudflare also launched identity-aware AI Gateway and User Insights, binding every request to a user identity and flagging anomalous sessions. Together, these moves extend Cloudflare from infrastructure into the enterprise AI agent platform layer.
Agentic Payments with x402 and MPP
Cloudflare's Agents SDK can handle HTTP 402 payment challenges for HTTP resources and MCP tools. x402 supports stablecoin payments, while MPP supports methods including cards and stablecoins. Clients can require human approval before paying; production workflows should also enforce spending limits, allowlists, receipts, and separate payment credentials.
Search, Agent, and Training Controls
Separate AI crawlers by purpose so a site can allow search discovery, restrict autonomous agents, and block or charge model-training crawlers.
AI Crawl Control with WAF
Cloudflare documents how AI Crawl Control interacts with WAF custom rules, including cases where WAF blocks can override charging or allow/block choices.
Pay Per Crawl to Pay Per Use
Pay Per Crawl lets publishers choose Allow, Charge, or Block for AI crawlers. Cloudflare's Pay Per Use direction aims to align compensation with content used in AI answers.
Cloudflare OS — Agentic Enterprise Platform
Open-sourced August 6, 2026, Cloudflare OS provides every employee with an AI workspace grounded in company context. It includes an isolated runtime, security governance framework, and shareable personal apps. The identity-aware AI Gateway binds every request to a user identity, while User Insights flags anomalous sessions.
Prices, plan limits, and promotions may change. Verify final pricing on the official site. Last verified: 2026-08-06.
Cloudflare is now a practical AI SEO and content-protection tool for sites that care about how AI systems crawl, cite, train on, or monetize their content. The best default strategy is not to block everything: allow beneficial search crawlers, restrict or charge high-volume training access where appropriate, and test WAF precedence before assuming crawler rules are working.