"Just block the AI bots." It is everywhere now: a setting in your hosting panel, a tick-box in a security plugin, a line of advice in a forum thread. The pitch is that AI companies are reading your website for free, and one switch makes them stop.
The trouble is that it is not one switch. It is at least three, they do different jobs, and the most common version of "block AI" quietly does the opposite of what the owner wanted. You can end up protecting content that was never your moat while making yourself invisible in the place customers are increasingly asking their questions.
So before you flip anything, it is worth knowing what you are actually turning away.
AI crawlers serve different purposes
We covered the crawler list in detail when we wrote about checking whether AI tools cite your site. The short version, because it is the whole decision: AI crawlers split into three kinds of work.
There are training crawlers, which collect pages that may go into building the next model. OpenAI's GPTBot and Anthropic's ClaudeBot are these, as is Google's training token, Google-Extended. Blocking them is mostly about not feeding your content into model training.
There are AI-search crawlers, which index your pages so the tools can quote you when someone asks a question right now. OpenAI's OAI-SearchBot, Anthropic's Claude-SearchBot and PerplexityBot do this job. These are the ones that decide whether you appear in an AI answer.
And there are on-demand fetchers, which grab a single page the moment a user asks about it, like ChatGPT-User. These behave more like a person clicking a link than a bot indexing the web.
The names matter because the switches are independent. OpenAI says plainly that blocking GPTBot has no effect on whether ChatGPT can show you in search, because that is OAI-SearchBot's job. Google says the same about Google-Extended: opt out of training and your normal Search ranking is untouched. You can decline to be training data and still be a source the tools quote. The two decisions only look like one because the advice flattens them.
Blocking search crawlers can make you invisible
Here is the version that costs real money.
A bakery is told to "block AI" to stop companies reading its content for training. Someone applies a blanket rule that turns away anything that looks like an AI bot. A few weeks later, a customer asks ChatGPT where to get a birthday cake in their town this week. The bakery is the obvious answer. It does not appear, because the rule that was meant to stop training also turned away the search crawler that would have quoted it.
Nobody notices, because absence is invisible. There is no error, no warning, no line in the analytics that says "you were the right answer and were left out". The owner thinks they protected something. They took themselves out of the shortlist.
For most small businesses, this is the wrong trade. Your website copy, opening hours and service list are not a secret anybody is going to monetise. They are marketing. The cost of a model reading them is close to nothing. The cost of being missing from the answer a customer reads, instead of visiting Google, is your enquiry going to whoever was quoted instead.
That does not mean nobody should block anything. If your content genuinely is the product, a paid research library, original journalism, a database you sell access to, then training exposure is a real concern and blocking the training crawlers is a reasonable call. The point is to know which camp you are in before you choose, not to apply a publisher's defensive instinct to a six-page services site that wants to be found.
You may already block AI search by accident
The decision has another wrinkle in 2026: you might have made it by accident, or had it made for you.
Since mid-2025, Cloudflare, which sits in front of a large slice of the web, started blocking AI crawlers by default for new sites and gave existing customers a one-click way to do the same. The intent is good, and it has teeth: well over two million sites had opted to disallow AI training by late 2025. There is now even a paid route, where a publisher can charge crawlers for access rather than simply allow or block, with Stack Overflow signing one of the first such deals in early 2026.
This is genuinely useful for big publishers. For a small business, the risk is that a default switch, set up by whoever built the site, is now quietly turning away the search crawlers too, and the owner has no idea it is on. We have found exactly this more than once: a site blocking AI search bots not through any decision, but through a leftover anti-scraper rule from a year earlier, or a security default nobody revisited. The owner wanted to be in AI answers. Their own setup was keeping them out.
This is the part only an agency that both builds the site and watches its AI visibility tends to catch, because the rule lives in the infrastructure and the symptom shows up in the answers, and most people only ever look at one of those.
Blocking limits access but cannot prevent reuse
One limitation the "block AI" advice rarely mentions.
The polite mechanism here, robots.txt, is a request, not a wall. It is a small text file that tells bots what they may and may not read, and the documented crawlers from the big providers honour it. But it is an honour system. A badly behaved scraper, or a tool that does not respect the rules, can ignore it entirely, and reports of fetchers crawling sites that asked them not to are not rare.
So if your goal is to genuinely stop a determined actor, robots.txt will not do it alone. That needs enforcement at the server or network level, the firewall-style blocking Cloudflare offers, which refuses the connection rather than asking nicely. And that is the same layer where it is easiest to lock yourself out of the search crawlers by mistake. The tool that can truly block is also the tool that quietly makes you invisible. Worth respecting in both directions.
Most small businesses should allow search crawlers
If you run a normal small business with a marketing website, the default is not "block AI". It is closer to the opposite, with one deliberate exception.
- Allow the AI-search crawlers unless you have a specific reason not to. Being quoted is the upside of the whole thing. Turning these away is choosing not to compete in AI answers.
- Block the training crawlers only if your content is the product. If people pay for what is on your pages, decline training. If your pages are there to win enquiries, there is little to protect and a real cost to hiding.
- Before you change anything, check what you are already turning away. Read your own robots.txt and your hosting or Cloudflare bot settings. The most common problem we see is not the wrong new decision. It is an old block nobody remembers, doing damage in the dark.
The framing that helps is this: blocking AI crawlers is a content-protection decision, and being visible in AI answers is a marketing decision, and most small businesses have far more to gain from the second than to lose from the first. The mistake is treating them as the same switch.
If you would like someone to check what your site is actually allowing and blocking right now, and whether a leftover rule is keeping you out of AI answers, have a 20-minute chat. We will look at the infrastructure and the answers together, because the problem usually lives in the gap between the two.
