Should you let AI bots crawl your website?

"AI bots" aren't all the same thing

You've probably seen the advice to block AI bots, or a one-click setting from your host that promises to do it. Before you flip any switch, it helps to know that the automated visitors people lump together as "AI bots" are actually doing three quite different jobs, and blocking each one has a very different effect on your business.

The three jobs are roughly these. Training crawlers collect content to help build the AI models themselves. Search and answer bots fetch pages so an assistant can find you and cite you when it answers a question. And user-triggered fetchers grab a page in the moment because someone asked an assistant to look at it. Some of these you may be happy to block. One of them is often the very thing that decides whether an AI assistant can mention your business at all.

So the useful question isn't "should I block AI bots?" It's "which bots, and what do I gain or lose by blocking each one?" The rest of this guide walks through the three types, how to actually control them, where the llms.txt idea fits, and a sensible default for a service business that wants enquiries.

Training crawlers: learning from your content

Training crawlers download public pages so their content can be used to train or improve AI models. The best-known ones publish their name so you can identify and control them. OpenAI's is GPTBot, which OpenAI describes as crawling content that may be used to train its foundation models. Anthropic's is ClaudeBot, described as collecting web content for model training. Google uses a control called Google-Extended, which governs whether content Google already crawls can be used to train and ground its Gemini models. Common Crawl's CCBot builds a large public dataset that many AI projects then learn from.

Businesses block these for a few honest reasons: they'd rather their words weren't used to train commercial models, they want more control over their content, or they simply want to cut down on bot traffic. Those are reasonable positions. The key point is what you don't lose: blocking a training-only crawler generally has little to do with whether you turn up in live AI answers or in ordinary search. Google is explicit about this for its own control, stating that Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search."

Google-Extended is worth understanding because it's often misread. It isn't a bot that visits your site, and it doesn't control whether you appear in Google's AI Overviews. It's a robots.txt instruction that tells Google not to use your content for training and grounding Gemini, while your normal search presence, including AI Overviews, keeps working through the usual Googlebot crawl.

Search and answer bots: how you show up in AI results

This is the group most businesses don't mean to block, and the one where a careless setting does real damage. These bots exist so an assistant can discover your pages and cite them when it answers a customer's question. OpenAI runs OAI-SearchBot, which it says is used to surface websites in ChatGPT's search features. Anthropic runs Claude-SearchBot to index content for Claude's search results. Perplexity crawls the web with PerplexityBot for the same purpose.

Google is a special case worth spelling out. Its AI Overviews and AI Mode are part of Google Search, and they run on the same index that ordinary results use. Google's own guidance says that to be eligible for its AI features, "a page must be indexed and eligible to be shown in Google Search with a snippet." In other words, the thing that gets you into Google's AI answers is being crawlable and indexable by the normal Googlebot, not any special AI setting. Block Googlebot and you drop out of both regular results and AI Overviews together.

The lesson is simple: if you want AI assistants to be able to mention your business, these are the bots to keep open. Blocking them to "opt out of AI" is the digital equivalent of taking your number out of the phone book to avoid junk mail. It's your call to make, but make it knowing that it's your visibility you'd be switching off.

Keep readingCheck whether AI search mentions you

User-triggered fetches: when someone asks the assistant to look

The third group fetches a single page because a person asked for it right then. When someone pastes your link into ChatGPT and says "summarise this", or asks an assistant to check your opening hours, the assistant fetches that page on their behalf. OpenAI calls this agent ChatGPT-User; Anthropic calls its equivalent Claude-User.

Because these fetches are user-initiated rather than automatic crawling, the operators note that ordinary robots.txt crawl rules may not apply to them in the same way. That's an honest thing to know: even if you block the automatic crawlers, you can't fully stop a customer from copying your public page into an assistant and asking about it. Robots.txt manages well-behaved bots doing their own crawling, not what a person chooses to do with a page you've published openly.

How to actually allow or block them

The main control is a plain text file at the root of your site called robots.txt, for example yourbusiness.com.au/robots.txt. In it, you name a bot by its user-agent token and tell it what it may or may not access. Reputable crawlers, including the ones named above, check this file and honour it. It's a set of instructions you publish, not a lock, so treat it as a request that well-behaved bots respect rather than a security barrier.

Here's an illustrative example that keeps the search and answer bots welcome while turning away the training-only crawlers. Treat it as a pattern to adapt, not a copy-paste answer, and check each operator's current bot names before you rely on them, because these change over time.

Many hosting platforms and content systems now offer a single "block AI bots" toggle. These are convenient, but check exactly which bots the toggle covers before you switch it on. Some of them block the search and answer bots as well as the training crawlers, which quietly removes you from AI results at the same time as opting you out of training. If you're not sure, editing robots.txt yourself gives you the control to keep the right bots in and the wrong ones out.

• To block a training crawler, add a block like: User-agent: GPTBot then on the next line Disallow: / . Repeat the same pair for ClaudeBot and CCBot, and add User-agent: Google-Extended with Disallow: / to opt out of Gemini training.
• To stay visible in AI answers, do not disallow the search and answer bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and, crucially, do not disallow Googlebot, which is what carries you into both normal results and AI Overviews.
• Apply the rules to every subdomain you want covered, since robots.txt is read per host. A rule on your main site doesn't automatically cover shop.yourbusiness.com.au or a separate blog subdomain.
• Never rely on robots.txt to hide sensitive pages. Anything you truly need private should sit behind a login, not just a Disallow line, because the file is public and only guides cooperative bots.

What about llms.txt?

You may have read that adding a file called llms.txt helps AI tools understand your site. It's a proposed idea: a markdown file that offers a tidy, curated summary of your content for language models to read. It sounds sensible, and some documentation-heavy sites publish one. But for an ordinary business deciding where to spend effort, the honest picture is underwhelming.

Google has been blunt that it doesn't use it. Its guidance states that you "don't need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn't use them," and that creating an llms.txt file "will neither harm nor help your site's visibility or rankings in Google Search, as Google Search ignores them." Google staff have compared it to the old keywords meta tag, and pointed to server logs showing the file simply isn't being fetched by AI systems.

So don't treat llms.txt as a shortcut into AI answers, and definitely don't let it distract from the basics. What actually helps an assistant understand and cite you is the same thing that helps a person: clear, accurate, crawlable pages that explain what you do, who you help and how to get in touch. If you want to publish an llms.txt as well, it won't hurt, but put your time into the real pages first.

A sensible default for most service businesses

For most Australian service businesses whose goal is more enquiries, the safe starting point is to stay open to the bots that help customers find you and be deliberate about the rest. If AI visibility matters to you, the single most important thing is not to accidentally block the bots that provide it.

Blocking or allowing a bot changes who can reach your content, not whether that content is worth reaching. Once you've made a deliberate choice here, the more valuable work is making sure the pages those bots can read actually explain your business well.

• Keep Googlebot and the AI search and answer bots allowed. This is what lets you appear in normal search, in Google's AI Overviews and in assistants' cited answers.
• Decide the training question on your own terms. If you'd rather your content wasn't used to train models, block the training-only crawlers (GPTBot, Google-Extended, ClaudeBot, CCBot). You can do this without losing search or AI-answer visibility.
• Check any one-click "block AI" setting before trusting it. Confirm whether it also blocks the search and answer bots, and reach for a hand-edited robots.txt if you need finer control.
• Don't chase special files. There's no llms.txt shortcut and no secret markup that gets you into AI answers. Spend the effort on clear, crawlable pages instead.
• Recheck the names now and then. The list of bots and controls changes, so glance at each operator's current documentation before you make or trust a blocking rule.

Keep readingHelp AI search understand your business