Yes, one line in your robots.txt can quietly delete your brand from ChatGPT's answers. OpenAI runs separate crawlers for different jobs, and blocking the wrong one has different consequences: blocking OAI-SearchBot removes you from ChatGPT's search answers entirely, while blocking GPTBot only keeps your content out of model training. A blanket Disallow: / rule, or a security plugin's "block all bots" default, hits the retrieval crawlers that AI search engines rely on to cite you. The fix is to audit your robots.txt and explicitly allow the search and user-facing crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, and Googlebot) even if you decide to block the training ones.
The distinction between training crawlers and retrieval crawlers is the whole game, and most "block the AI bots" advice online ignores it. This guide explains the difference, gives you a checklist of the crawlers that matter as of July 2026, and shows how to audit your own file in a few minutes.
Can a robots.txt rule really remove you from ChatGPT?
Yes, and OpenAI says so in its own documentation. In its published crawler docs, OpenAI states that "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." OAI-SearchBot is the crawler that indexes pages for ChatGPT's search feature, so disallowing it in robots.txt tells OpenAI to leave your site out of the answers ChatGPT generates when it searches the web.
This is the trap. Many site owners added User-agent: GPTBot / Disallow: / in 2024 to keep their content out of AI training, then assumed they had "blocked ChatGPT." They had not. GPTBot and OAI-SearchBot are separate systems. Blocking one does nothing to the other. The reverse mistake is worse: a rule that blocks OAI-SearchBot, whether added on purpose or inherited from a plugin, pulls you out of ChatGPT's live citations while you keep waiting to show up.
Training crawlers vs search crawlers: the difference that matters
A training crawler collects content to help build and refine the AI model itself, while a search (retrieval) crawler fetches pages so the AI search engine can cite them in a live answer. Blocking a training crawler affects what the model learns during training. Blocking a retrieval crawler affects whether you appear in answers right now. These are different doors, and closing one does not close the other.
Here is the plain-language version. When ChatGPT, Perplexity, or Claude searches the web to answer a question, they do not consult the model's memory alone. They send out a retrieval crawler, pull current pages, and cite the ones that answer the question. If your robots.txt blocks that retrieval crawler, the engine cannot fetch you, so it cannot cite you, no matter how good your content is. Training crawlers are the ones that gather text to improve the underlying model, a slower and separate process that does not decide whether you show up in today's answer. If you want to understand the retrieval step in more depth, see how AI search engines actually work.
What to do: Decide the two questions separately. "Do I want my content used to train AI models?" is a rights and policy question you can answer either way. "Do I want to appear in AI search answers?" is a visibility question, and for almost every brand the answer is yes. Block training crawlers if you must, but never block the retrieval crawlers by accident.
Why "block all bots" quietly deletes you from AI search
A blanket "block all bots" setting is the most common way brands disappear from AI search without realizing it. Security plugins, WAF rules, Cloudflare bot-management toggles, and copy-pasted robots.txt templates often ship with aggressive defaults that disallow unfamiliar user-agents. AI retrieval crawlers are unfamiliar user-agents. So the same setting that blocks a scraper also blocks OAI-SearchBot, Claude-SearchBot, and PerplexityBot.
The damage is silent because nothing breaks. Your site loads fine, Google still ranks you, and your analytics look normal. The only symptom is an absence: AI search engines stop naming you, and there is no error message telling you why. This is different from a schema or file-based fix you might have tried. Schema markup will not get you into ChatGPT, and you do not need an llms.txt file either, but a bad robots.txt rule can actively lock you out. Robots.txt is one of the few technical files that can subtract visibility rather than add it.
What to do: Treat robots.txt and your CDN bot settings as an allow-list problem, not a block-list problem. Before assuming your AI visibility issue is a content problem, confirm the crawlers can even reach you. A brand that is invisible in AI answers should rule out an accidental block first, because it is the fastest thing to fix.
The AI crawler checklist (July 2026)
The table below lists the AI crawlers that matter as of July 2026, what each one does, and whether most brands should allow it. User-agent tokens and roles are drawn from each operator's own published documentation. The rule of thumb: allow every search/retrieval and user-action crawler, and treat training crawlers as an optional policy choice.
| User-agent | Operator | Role | Recommendation |
|---|---|---|---|
| GPTBot | OpenAI | Training | Optional. Blocking affects training only, not ChatGPT search |
| OAI-SearchBot | OpenAI | Search/retrieval (ChatGPT search) | Allow. Blocking removes you from ChatGPT search answers |
| ChatGPT-User | OpenAI | User action (fetches a page a user asks for) | Allow |
| ClaudeBot | Anthropic | Training | Optional. Blocking excludes future content from training |
| Claude-SearchBot | Anthropic | Search/retrieval | Allow. Blocking reduces visibility in Claude search results |
| Claude-User | Anthropic | User action | Allow |
| PerplexityBot | Perplexity | Search/index (retrieval) | Allow. Blocking makes you undiscoverable in Perplexity |
| Perplexity-User | Perplexity | User action | Allow (see caveat below) |
| Googlebot | Search index (also feeds AI Overviews and AI Mode) | Allow. Never block | |
| Google-Extended | Training/grounding control token (not a crawler) | Optional. Affects Gemini training only, not search | |
| Bingbot | Microsoft | Search index (feeds Copilot retrieval) | Allow. Never block |
| Amazonbot | Amazon | Training plus Alexa/Rufus retrieval | Consider allowing if you sell products |
| Meta-ExternalAgent | Meta | Training and AI indexing | Optional policy choice |
A few caveats worth knowing so you do not misread the table:
- Google-Extended is not a crawler. It is a control token in robots.txt. The actual crawling is still done by Googlebot. Disallowing Google-Extended tells Google not to use your content to train Gemini or ground its AI products, but it does not affect your Google Search ranking. This is why blocking
Googlebotto "stop AI training" is a costly mistake: it removes you from Google Search entirely, while the AI-only opt-out you actually wanted is Google-Extended. - Perplexity-User may not honor robots.txt. Perplexity has publicly argued that Perplexity-User "is an agent, not a bot" acting on a specific user's behalf, and therefore is not bound to follow robots.txt. This position has drawn disputes with publishers and Cloudflare. Treat any robots.txt rule for Perplexity-User as advisory, not guaranteed.
- ChatGPT-User actions are user-initiated. OpenAI notes that because ChatGPT-User is triggered by a person asking ChatGPT to visit a page, robots.txt rules may not apply to it the way they do to automatic crawlers.
- Older Anthropic tokens exist. You may still see
anthropic-aiandClaude-Webin old templates. Anthropic's current documentation describes ClaudeBot, Claude-User, and Claude-SearchBot. Keep the current three allowed and you are covered.
What to do: how to audit your own robots.txt
Auditing your robots.txt takes a few minutes and does not require a developer. Pull up the file, look for anything that blocks the retrieval crawlers, and check the layers above robots.txt that can block bots without touching the file at all.
- Read the file. Open
https://yourdomain.com/robots.txtin a browser, or fetch it from a terminal. Scan for anyDisallow: /underUser-agent: *, and for named blocks against OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, or Bingbot. - Check for a blanket block. A
User-agent: * / Disallow: /blocks everything, including every AI retrieval crawler. If you see it and did not intend a fully private site, that is your problem. - Check the layer above robots.txt. Cloudflare's "Bot Fight Mode" or "Block AI Bots" toggle, a WAF rule, or a security plugin (common on WordPress) can block AI crawlers even when your robots.txt looks clean. These act at the network edge and will not show up in the file. Review those dashboards directly.
- Add explicit allow rules. Rather than relying on the absence of a block, allow the crawlers you want by name so a future template change does not silently lock them out.
A minimal robots.txt that keeps you in AI search while opting out of training might look like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
This allows the retrieval and user-facing crawlers while blocking OpenAI's and Google's training-specific access. Adjust the training lines to your own policy. After you publish a change, expect a lag before it takes effect: OpenAI, for example, notes that its search systems take roughly 24 hours to reflect a robots.txt update.
The training-vs-visibility tradeoff
Blocking training crawlers is a legitimate choice, but blocking retrieval crawlers is almost always a mistake. Some brands do not want their content feeding the next generation of AI models, and disallowing GPTBot, ClaudeBot, or Google-Extended is a reasonable way to express that. The cost of that choice is close to zero for present-day visibility, because those crawlers do not decide whether you appear in today's answers. Blocking retrieval crawlers is different: it removes you from the answers themselves, which is usually the opposite of what a brand wants.
Allowing the crawlers only makes you eligible, though. It opens the door; it does not walk you through it. AI search engines still build recommendations from the sources they trust, so being crawlable is necessary but not sufficient. The higher-impact work is building third-party presence on the sources AI search engines actually pull from, such as Reddit, review sites, YouTube, and editorial coverage. The Loudmink AEO platform tracks which sources each AI search engine pulls from, so you can see whether you are actually getting cited once the crawlers can reach you. Plans from $99/mo.
Frequently Asked Questions
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot is OpenAI's training crawler, and blocking it only keeps your content out of model training. ChatGPT's search answers use a separate crawler, OAI-SearchBot. To stay in ChatGPT's search answers, allow OAI-SearchBot; blocking that one, per OpenAI's own docs, removes you from ChatGPT search results.
Should I allow AI crawlers?
For almost every brand, yes, at least the search and user-facing ones (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Googlebot, and Bingbot). Blocking them removes you from AI search answers. Whether to allow training crawlers (GPTBot, ClaudeBot, Google-Extended) is a separate policy choice with little effect on present-day visibility.
What is OAI-SearchBot?
OAI-SearchBot is OpenAI's crawler for ChatGPT's search feature. It indexes web pages so ChatGPT can cite them in live answers. It is separate from GPTBot, which collects training data. OpenAI states that sites opting out of OAI-SearchBot will not be shown in ChatGPT search answers.
Does blocking ClaudeBot remove me from Claude?
Not from Claude's search answers. ClaudeBot is Anthropic's training crawler, and blocking it excludes your future content from training datasets. Claude's live search uses Claude-SearchBot, and its user-directed fetches use Claude-User. Anthropic says disabling Claude-SearchBot reduces your visibility in Claude's search results, so allow that one to stay citable.
How do I know if my robots.txt is accidentally blocking AI crawlers?
Open https://yourdomain.com/robots.txt and look for Disallow: / under User-agent: * or under any named AI crawler. Also check Cloudflare bot settings, your WAF, and any security plugin, since those can block crawlers without changing the file. If retrieval crawlers are blocked at any layer, AI search engines cannot cite you.