Is my web host blocking AI from seeing my site?

Quite possibly, and you would be the last to know. Your site can look perfect in every browser while a security layer you never turned on quietly blocks the AI crawlers reading the web. The engines move on. The answers get written without you. And nothing anywhere looks broken. We know, because it happened to us this week.

How a site blocks AI without anyone deciding to

Between your website and the world sit layers you rarely think about: your host, a CDN, a firewall, security plugins. Many of these ship with bot protection. Some hosts turn it on for you, without asking, as a favor. The protection works with a challenge page. It says something like “Checking your browser” and runs a bit of code to prove the visitor is human.

Here’s the problem. A person’s browser passes the test in a blink. An AI crawler doesn’t run the code. It fails the test and gets the challenge page instead of your content. Your customers see your site. The engines writing answers about your market see a wall.

It happened to us

This week we found that our own host had switched on its CDN for our site. We never asked for it. Its bot protection was challenging non-browser visitors, including the AI crawlers this site exists to serve. Every page looked flawless in a browser. We caught it only because we fetch our own pages the way a crawler does. We’re an AEO consultancy, and it still happened to us. If it can happen here, it can happen to any business. And the owner will never see it.

Blocked-by-default is becoming the norm

This isn’t one careless host. The companies that run the web’s plumbing are moving toward blocking AI crawlers as the default. In July 2025, Cloudflare announced that new domains on its network block AI crawlers unless the owner says yes. On September 15, 2026, just over a week from now, its default blocking grows again. For publishers who sell content, this is protection. For a business that wants AI engines to know it exists, a default you never saw can erase you quietly. Whichever side you’re on, it should be your call, made on purpose.

The five-minute check

• Fetch your site the way a crawler does. If you’re comfortable in a terminal, request your homepage with a crawler’s user agent. A page titled “Just a moment” or “Checking your browser” means the wall is up.
• Run our free scan. The single-page scanner reads your page the way an engine’s crawler does. If the scan can’t read a page that loads fine in your browser, something upstream is filtering non-browser visitors.
• Look for crawler names in your access logs. GPTBot, ClaudeBot, PerplexityBot. If they used to show up and stopped, note the date. Then ask what changed at your host around then.

What to ask your host

Three questions get to the bottom of it. Is a CDN, firewall, or bot protection active on my account? Did I turn it on, or did you? And can I lower the challenge level or let verified AI crawlers through? If the first answer is yes and the second is “we did it for you,” you’ve likely found your wall.

Good bots in, bad bots out

None of this means running unprotected. Modern edge tools can tell real, named crawlers from junk scrapers. The right posture for most businesses is selective. Let verified AI crawlers read you. Keep blocking the junk. This is a setting, not a trade-off, and any decent host can set it up. The point isn’t “never block a bot.” The point is that being invisible to AI engines should never happen to you by default.

Being readable is the first of the three things an engine needs before it can name you. We’ve written about all three, and what happens when a competitor wins them. But readable comes first. When the crawler hits a wall, nothing else you do matters.

What to do about it

Run the free scan on your most important page today. It takes a minute and settles the question. If the wall is up, the fix is usually one setting or one support ticket. And if you want the full picture of how engines see you, an AI Visibility Audit measures walls, content, and entity information together, and hands you the fixes in priority order.

Your site is either readable to the engines answering your customers, or it isn’t. Don’t let a default you never saw make the call.