Reference · AI crawlers
Most robots.txt files we audit get at least one of these wrong. Usually it is the one that quietly removes the business from ChatGPT search.
Updated 2026-08-29
There is no single "AI bot". There are at least nine distinct crawlers from four companies, they do different jobs, and blocking one has nothing like the same consequence as blocking another. The most common mistake we find is a robots.txt that blocks OAI-SearchBot while intending only to opt out of model training. That single rule removes the business from ChatGPT search answers entirely while doing nothing about training.
The single most important fact: with the exception of Googlebot and Applebot, none of these crawlers execute JavaScript. If your site renders its content in the browser, they see an empty shell no matter what your robots.txt says. You can check yours with our free AI visibility tool in about a minute.
The title promises a cost, so here is the size of what is being blocked. Adobe measured a 1,200% rise in traffic reaching US retail sites from generative AI sources in under a year. Semrush found the average AI-referred visitor converts at 4.4 times the rate of one arriving from traditional organic search. BrightLocal puts 45% of consumers asking generative AI for local business recommendations, up from 6% a year earlier. And the channel it displaces is shrinking: Pew Research tracked real browsing and found people click a search result 8% of the time when an AI summary is present, against 15% when it is not.
None of that is a projection. It is the current size of a channel that a single mistaken line in a text file can remove you from entirely.
GPTBot collects data used to train OpenAI's models. It does not control whether you appear in ChatGPT search results; that is a different bot.
Blocking it is a reasonable content-licensing decision and costs you nothing in visibility. Blocking it under the impression that it controls ChatGPT search is the mistake.
The short versionBlocks training. Does not affect ChatGPT search visibility.
This is the one that matters commercially. OAI-SearchBot builds the index ChatGPT search draws on, and OpenAI documents plainly that sites which opt out "will not be shown in ChatGPT search answers".
If you take one action after reading this page, confirm that your robots.txt allows OAI-SearchBot.
The short versionBlocking it removes you from ChatGPT search answers. Almost nobody intends this.
When someone in a ChatGPT session follows a link to your site or asks the assistant to look at a specific page, this is the agent that fetches it in real time.
Blocking it means a user who explicitly asks ChatGPT to read your page gets told it cannot. That is a strange experience to hand a prospect who is actively researching you.
The short versionBlocks the live "go and read this page" request a real user initiated.
Anthropic's training crawler, the direct counterpart to GPTBot. Same reasoning applies: blocking it is a licensing choice, not a visibility one.
The short versionBlocks training for Claude. No search-visibility consequence.
The Anthropic equivalents of OAI-SearchBot and ChatGPT-User: one builds the search index Claude consults, the other fetches a page when a user's conversation calls for it.
Claude's user base skews heavily toward technical and professional buyers, which for a B2B services business is a more valuable audience per visit than its raw traffic share suggests.
The short versionBlocking these removes you from Claude's answers.
Perplexity is answer-first by design: it shows a composed answer with numbered citations rather than a list of links, so a citation there is unusually visible.
It does not execute JavaScript, which means a client-rendered site is invisible to it regardless of how good the content is.
The short versionBlocking it removes you from Perplexity citations.
Googlebot renders JavaScript, and it is the only control that matters for AI Overviews and AI Mode. There is no separate "AI Overviews bot" to allow or block.
This is why classic SEO is the entry requirement for Google's AI surfaces rather than a separate discipline from them.
The short versionBlocking it removes you from Google Search and AI Overviews together.
Google-Extended is frequently misdescribed by agencies as an AI Overviews lever. It is not a crawler at all: it is a token that governs whether your content trains Gemini.
Google states explicitly that it "does not impact a site's inclusion in Google Search", and blocking it does not remove you from AI Overviews.
The short versionBlocks Gemini training only. No effect on Google Search or AI Overviews.
Bingbot renders JavaScript imperfectly and feeds Bing, which in turn feeds part of ChatGPT's search results, so it has an indirect route into an answer engine.
Applebot renders JavaScript and powers Siri and Apple Intelligence surfaces.
The short versionBoth worth allowing; both reach further than their own products.
The common intention is: stay out of model training, stay in the answers. That is expressible, because training and answering are different crawlers. Allowing a bot is the default, so the useful file is the one that names only what you are refusing.
Three lines refuse the training collectors: Disallow: / under User-agent: GPTBot, the same under User-agent: ClaudeBot, and the same under User-agent: Google-Extended. Nothing else needs a rule, because allowing is the default. Add your sitemap line and stop there.
What that leaves free to read you is OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, Claude-User, Googlebot, Bingbot and Applebot, which is the set that decides whether you can be cited at all.
If you want to be in training too, the answer is simpler: publish nothing but the sitemap line. The failure we actually find in audits is never an over-permissive file. It is a bare `User-agent: *` with a `Disallow: /` left behind from a staging environment, which catches every bot on this page at once.
The short versionName only what you refuse. Blocking training does not require blocking the answering crawlers.
The pattern across all nine: the crawlers that decide whether an AI assistant can cite you are not the ones most people think they are, and the ones that matter most cannot run JavaScript. Getting the robots.txt right is a ten-minute job. Getting the rendering right is an engineering one, and it is the work our AEO practice exists to do.
Frequently asked questions
