Did the web just close to AI crawlers?

No. The engines answering your customers’ questions can still read most of the web, including your site. What changed on September 15 is more interesting than a closed door. The web’s front gate now asks every AI crawler what it’s there for. One answer, training, gets turned away more and more. A month in, the data shows which doors closed, which stayed open, and which setting on your own site deserves five minutes.

What actually changed on September 15

Cloudflare, which sits in front of a huge share of the web, began sorting AI crawlers into three buckets: Search, Training, and Agent. Search crawlers index pages and send people back. Training crawlers collect content to teach models. Agents visit on a user’s behalf to do a task. The new rules land on new domains that run ads, and on free accounts nobody adjusted. Those sites now start with training turned off, and with agents blocked on pages that carry ads. A crawler that won’t say which job it’s doing gets blocked on those pages outright. Search stays open by default.

We saw this one coming

Regular readers met this change before it landed. Our September 6 answer post about hosts blocking AI without asking flagged the September 15 expansion two weeks early. The advice from then holds now. Whichever side of this you’re on, it should be your decision, made on purpose. What’s new is a month of numbers.

The numbers behind the gate

Three figures from Cloudflare’s own data tell the story. First, the pressure. In spring 2025, 22 percent of crawler requests were for AI training. By June 2026, it was 52 percent. More than half of all crawling now feeds models. Much of it is pure waste, re-fetching pages that haven’t changed. Second, the response: 17 percent of sites on Cloudflare now block training crawlers. Third, the figure that matters most for you: fewer than 1 percent block search crawlers.

Read the last two together and you have the honest answer. The web is closing to free training. It stays wide open to the crawlers that put businesses in answers.

Why the answer crawlers stayed in

The three-bucket split encodes a simple bargain. Search and answer crawlers send something back: a citation, a name, a visitor. Training crawlers take without returning. Site owners finally got to choose between the two, and they’re keeping the first kind. For your AI visibility, this is good news dressed as scary news. The crawlers that read your page before an engine quotes you are in the protected class almost everywhere.

The toll booth riding along

There’s a second change in the same release. Cloudflare’s new Pay Per Use program pays publishers when their content actually powers an AI answer, not just when a crawler fetches it. The first partners are small, and nothing here touches a typical business site yet. But watch the direction. Content is becoming a licensed input with a meter on it. The open question for next year is whether small sites ever see a check, or whether the meter only spins for big publishers.

The wrinkle worth knowing: the two clocks again

One honest caveat before you block training on your own site. We’ve written about the two clocks of AEO: retrieval, which updates when crawlers re-read you, and the engine’s own memory of your name, which builds slowly, partly from training data. Blocking training protects your content. It may also slow the slow clock. A model that never trains on the web’s mentions of you remembers you from less. Nobody can measure this trade yet, but it is a trade, not a free win. A business that wants to be named should weigh it one way. A publisher protecting an archive, another.

What to do about it

Spend five minutes knowing your own settings. If your site sits behind Cloudflare or any host with bot controls, open the dashboard and look. Is search allowed? Is training blocked? Did you choose this, or did a default? Match the settings to your goals. If being named in AI answers is the goal, keep the search and answer crawlers open, and decide on training deliberately. Then run the free single-page scan to confirm engines can read what you’ve left open. If you’d rather have the whole picture measured, an AI Visibility Audit covers the gate, the pages, and the answers in one pass.

The web didn’t close to AI. It started reading badges at the door. Make sure the crawlers working for your visibility are on the list, on purpose.