What are Cloudflare's AI crawler traffic categories?

Cloudflare now lets site owners manage AI traffic by broad use case: Search crawlers, Agent crawlers, and Training crawlers. This helps publishers distinguish discoverability from automation and model-training access.

Should a content site block all AI crawlers?

Usually no. A practical default is to allow crawlers that improve search visibility, restrict or charge crawlers used for training, and review agent traffic separately because it may take actions or gather content for a user task.

How does Pay Per Crawl differ from Pay Per Use?

Pay Per Crawl charges when an AI crawler accesses content. Cloudflare's Pay Per Use direction aims to compensate creators when content is actually used in AI answers, which is closer to the value publishers care about.

Do WAF rules affect AI Crawl Control?

Yes. Cloudflare documents that WAF and Bot Management rules can execute before AI Crawl Control choices, so teams should test whether upstream security rules are overriding allow, block, or charge settings.

AI Crawler Segmentation and Pay Per Use for Content Sites

AI traffic is no longer one thing. A crawler that indexes your article for AI search is different from a crawler training a model, and both are different from an agent gathering content for a user task. Cloudflare's July 2026 AI traffic options make that distinction operational for site owners.

For AILinkBase-style content sites, the goal is not "block every bot." The better goal is controlled visibility: allow useful discovery, restrict high-risk training use, and create a path to compensation when AI systems extract value from your content.

Quick recommendation

Use a three-lane policy: allow Search crawlers that send useful discovery signals, review Agent crawlers separately, and block or charge Training crawlers unless you have a clear licensing reason to allow them.

The Three AI Crawler Lanes

Crawler type	Main risk	Default stance
Search	Visibility without enough referral traffic	Allow and monitor
Agent	Automated extraction or task-specific scraping	Allow selectively
Training	Uncompensated model training	Block or charge

Pay Per Crawl vs Pay Per Use

Pay Per Crawl is the current mechanism: a crawler requests a page, receives either access or a payment-required response, and the publisher can choose Allow, Charge, or Block. It is simple, but crawling is only a proxy for value.

Cloudflare's Pay Per Use direction is more ambitious. The idea is that content owners should be compensated when content is actually used in AI answers, summaries, or agent workflows. That is still an emerging market, but it is strategically important for publishers because value increasingly happens away from the click.

Implementation Checklist

1.Inventory your content: public SEO pages, docs, gated research, affiliate pages, and ad-monetized pages.
2.Define crawler policy by purpose: Search, Agent, and Training.
3.Review WAF and Bot Management rules before assuming AI Crawl Control settings are final.
4.Use analytics to compare crawler volume, AI search visibility, and actual referral traffic.
5.Revisit pricing and allow/block decisions monthly while the Pay Per Use market develops.

AILinkBase Take

Cloudflare has become part of the AI SEO stack. Content teams should treat crawler policy as a product decision, not just a security toggle: what should be discoverable, what should be licensed, and what should never be used for model training?

Sources: Cloudflare AI traffic options · Cloudflare Pay Per Use direction · AI Crawl Control with WAF · Cloudflare tool page