Should You Block AI Crawlers? Training vs Search

"Should we block AI crawlers? I don't want my content taken for free." — it's a question many owners have been asking lately.

The conclusion first: don't rush to block. AI crawlers aren't one thing — there are at least three kinds. One decides whether AI can mention you; one only decides whether your content is used to train models; and one is a temporary visit triggered by a user. Treat them as one and block everything, and the most common outcome is this: you meant to block training, and you blocked your chance of being found by AI as well.

One boundary: not every company needs fine-grained configuration. If your website content is already public and you don't mind it being used for training, leaving things open is usually fine. The decision that actually matters is the training category.

AI crawlers come in three kinds, with completely different purposes

Sort crawlers by what they come to do, and the decision becomes clear.

• Training: fetches content to train or improve AI models. It decides whether your content enters the model's knowledge. Examples: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), Applebot-Extended (Apple), CCBot (Common Crawl), Bytespider (ByteDance).
• Retrieval: fetches content to source live answers to user questions. It decides whether AI can find and cite you when answering. Examples: OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot.
• User-triggered: fetches only when a user explicitly asks AI to go and look at a page. Examples: ChatGPT-User, Claude-User, Perplexity-User.

Scope note: apart from Google, the crawler names and purposes above come from third-party compilations of vendor documentation and have not been verified one by one against the original vendor documents; user-agent strings are added and removed every year, so check each vendor's official documentation before configuring anything.

The key point: what you actually want to block is most likely the first kind; what you actually need is the second.

The most important point: Google-Extended does not affect indexing or ranking

A common worry: "if I block AI, will my search rankings fall too?" For Google, the official answer is clear.

In its developer documentation "Google's common crawlers", Google states two things.

Google-Extended has no separate user-agent string; crawling is still done by Google's existing user agents, and the token in robots.txt acts only as a control switch. What it governs: whether content Google crawls may be used to train future versions of the Gemini models, and for "grounding" in Gemini apps and Vertex AI (supplying content from Google's search index to the model at prompt time).

Then the key official conclusion: "Google-Extended does not affect a site's inclusion in Google Search results, nor is it used as a ranking signal in Google Search."

In other words, for Google, blocking training and being indexed for search are independent. You can opt out of training data while keeping the chance to be discovered by Search and AI features.

Scope note: this official conclusion applies only to Google-Extended. Whether other vendors offer equally fine-grained separation needs checking in their own documentation — don't treat it as an industry-wide rule.

So the question to settle is actually this one

Once the mechanics are clear, "should we block" becomes a much more specific choice.

What exactly bothers you? Is it "my content is being used to train someone else's model", or is it "AI mentions me when answering customers"? The first is training; the second is retrieval.

Does your content contain anything that shouldn't get out? Public product information, industry knowledge and usage instructions are published for exactly that purpose. What genuinely needs protecting is customer lists, pricing detail, process formulas — and those never belonged on public website pages in the first place.

Can you actually block it? This is more practical than many assume — see the next section.

A reality you need to know: robots.txt is a request, not a lock

robots.txt is a voluntary industry protocol — a statement saying "please don't crawl", not a technical barrier.

Vendors that respect it will respect it; but some crawlers have been recorded ignoring it (third-party sources note this of ByteDance's Bytespider). There's also a trickier case: user-triggered visits (such as ChatGPT-User, Perplexity-User) are generally understood in the industry not to be bound by robots.txt, because a person initiated the action rather than a bot crawling on its own.

Scope note: the claims above about which crawlers respect robots.txt come from third-party compilations and are industry observation, not vendor commitments; specific practices may change.

Three practical conclusions follow.

• For public content, don't expect robots.txt to keep secrets. Genuinely sensitive information belongs behind permissions, not behind a declaration.
• For crawlers that ignore the rules, only server-level blocking works (firewall rules, rejecting requests by characteristic). robots.txt can't solve it.
• Review regularly. User-agent strings are added and renamed every year; a ruleset written two years ago is probably out of date.

The two mistakes most often made in implementation

Mistake one: a blanket Disallow: /. That also blocks standard search engine crawlers — effectively removing yourself from search. Never block Googlebot or Bingbot: they're the gateway to the search index, which is a different matter from AI training.

Mistake two: assuming "blocked means deleted". Content already in training data doesn't disappear because you changed the rules now. Rules only affect future crawls.

One more reminder: robots.txt is shared at the domain root — every language version and subdirectory under a domain uses that one file. So think in terms of the whole site, not a single section.

Three common questions

If AI uses my website content for free, am I losing out?

Look at it from the other side: the real loss is when AI can't find you while answering a customer's question. For the vast majority of B2B companies, website content exists to be seen, trusted and cited — it isn't a paid product and doesn't need to be billed by download. So the usual sensible choice is: keep retrieval open (so AI can find you), and decide on training as you see fit. On Google, these two are fully separate.

We're B2B with few customers — does this matter?

Precisely because you have few customers, it matters more — you can't afford "not being in the answer" when a customer asks AI. The cost here is almost zero (editing one file), while the cost of getting it wrong can be an entire channel going dark.

How do we know whether AI is actually crawling?

Check your server or CDN access logs and filter by user agent: you'll see which crawlers hit which pages, and how often. If a retrieval crawler keeps fetching the same article, that usually means it's being used to answer live questions — exactly the kind of visit worth keeping.

In closing

To sum up: "should AI be allowed to crawl" isn't a yes-or-no question — it's a sorting question.

Sort the three kinds of crawler first, then decide what you actually care about, then do only what's necessary: don't block search, decide on training as you wish, don't rely on a declaration to keep public information secret, and remember to review the rules regularly.

One small thing you can do right now: open your site's robots.txt and look for a blanket Disallow: / or a global block under User-agent: *. If there is one, check whether it's accidentally hitting legitimate search engine crawlers — that's the easiest mistake to overlook and the most expensive to make.

Yihang Network, based in Shenzhen, previously worked in cross-border B2B and Alibaba.com-related services, and expanded into GEO (Generative Engine Optimization) from 2024. Our own view: public content should be as readable by AI as possible, and anything that genuinely needs protecting shouldn't be on public pages.

Data and scope note: The mechanics, purpose and scope of Google-Extended described here come from Google's official developer documentation "Google's common crawlers" and its guidance on Google web crawling. Crawler names, purposes and compliance behaviour other than Google's come from third-party public compilations and are industry-aggregate information, not verified line by line against each vendor's official documentation and not endorsed by any vendor. This content describes mechanisms and rules; it does not represent any specific company's results. Following these practices does not guarantee that content will be cited by AI, and no timeline or business result is promised.

1. Google for Developers, "Google's common crawlers" (including the official Google-Extended documentation)
2. Google for Developers, guidance on Google web crawling
3. Google Search Central, "AI features and your website"
4. Third-party compilation: AI crawler user agents and purposes (industry observation, not vendor-official)