Cloudflare Blocks AI Bots: What You Need to Know

AI agent crawlers will be blocked by default on some websites starting September 15th. Cloudflare announced this change on July 1st. Cloudflare has replaced its single block-AI-bots switch with three categories: Search, Agent, and Training. These controls went live on July 1st for all customers. From September 15th, Training and Agent bots will be blocked on pages displaying ads, while Search bots will remain allowed. These new defaults apply to new Cloudflare domains, new sites from existing customers, and all existing free-tier customers. Cloudflare believes an advertisement means a page is for human visitors. A search crawler can send a reader back, but a bot that simply reads the page and provides an answer does not. Cloudflare's blocks operate at the network level, affecting a large portion of web traffic. This means ad-supported pages, which often contain valuable information like news and pricing, will be impacted. Anyone running agents should determine which of their Cloudflare accounts will be classified as Agent-class, as this classification is behavioral. This change matters because it could affect how AI agents gather information from a significant part of the internet.