AI crawler policy is becoming an SEO, security, and monetization decision. Cloudflare now gives site owners controls for three AI traffic purposes: Search, Agent, and Training.
Deadline to know
Starting September 15, 2026, new domains onboarding to Cloudflare receive updated defaults: Training and Agent crawlers are blocked on ad-supported pages, while Search remains allowed.
Recommended policy
| Crawler type | Default stance | Why |
|---|---|---|
| Search | Allow and monitor | Supports discoverability and AI search visibility. |
| Agent | Allow selectively | Agent traffic can extract or act on content in task-specific ways. |
| Training | Block or charge | Training access may create value without referral traffic. |
AILinkBase content-site checklist
- Inventory pages with ads, affiliate links, gated research, docs, and original guides.
- Set separate policies for Search, Agent, and Training crawlers.
- Review WAF and Bot Management rules so they do not override AI Crawl Control unexpectedly.
- Monitor AI referral ratio: crawler requests versus actual users sent back to your site.
- Revisit Pay Per Crawl / Pay Per Use as the publisher compensation market matures.
Sources: Cloudflare AI traffic options · Cloudflare changelog · AI Crawl Control with WAF · Cloudflare tool page