Whether your business appears in ChatGPT, Perplexity, or Google’s AI features depends partly on which crawlers your robots.txt file lets in. The mistake many sites make is treating every “AI bot” as one thing. Operators now separate crawlers that train models, crawlers that build search indexes, and fetchers triggered by a single user’s question.
- OpenAI states that sites opted out of OAI-SearchBot “will not be shown in ChatGPT search answers,” while GPTBot is a separate crawler tied to model training.
- Perplexity says PerplexityBot is for surfacing and linking sites in search results and is “not used to crawl content for AI foundation models.”
- Google says Google-Extended does not affect inclusion in Google Search, and that AI Overviews and AI Mode need only normal indexing and snippet eligibility.
- Operators say robots.txt changes can take up to about 24 hours to be reflected, so verify after a change and do not judge results the same day.
Three kinds of AI crawler
The first step is to sort bots by purpose. Training crawler: a bot that collects content that may be used to build future models. Search crawler: a bot that indexes pages so an AI product can surface and link them in answers. User-triggered fetcher: a bot that visits a page because a person asked a question that needs it.
Blocking all three with one blanket rule is a business decision, not a neutral default. It can remove you from AI answers that buyers read while they compare vendors. Allowing all three may conflict with a content-licensing position. The right setting differs by company, which is why the table below separates them.
| User agent | Operator | Purpose per operator docs | Effect of blocking |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search features | Site not shown in ChatGPT search answers |
| GPTBot | OpenAI | Crawls content that may train generative models | Signals content should not be used for training |
| ChatGPT-User | OpenAI | User-initiated page visits | Not used to decide Search inclusion; robots.txt rules may not apply |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity search | Site may not appear in Perplexity results |
| Perplexity-User | Perplexity | User-triggered visits | Generally ignores robots.txt rules |
| Google-Extended | Controls use for Gemini training and grounding | No effect on inclusion in Google Search |
OpenAI’s three user agents
OpenAI’s bot documentation describes three separate agents. OAI-SearchBot surfaces websites in ChatGPT’s search features, and OpenAI recommends allowing it. GPTBot crawls content that may be used to train generative AI foundation models, and the documentation says disallowing it “indicates a site’s content should not be used in training.” ChatGPT-User visits pages when a user asks ChatGPT or a Custom GPT a question.
The practical consequence is that a company can block GPTBot to keep content out of training while still allowing OAI-SearchBot to remain eligible for ChatGPT search answers. OpenAI also states that ChatGPT-User “is not used to determine whether content may appear in Search.” The implication for a buyer is that a blanket “block AI” instruction from a developer or CMS plugin may be doing more damage than intended.
Perplexity and Google
Perplexity’s documentation splits its agents the same way. PerplexityBot is “designed to surface and link websites in search results on Perplexity” and “is not used to crawl content for AI foundation models.” Perplexity recommends allowing it. Perplexity-User handles requests triggered by a person’s question and “generally ignores robots.txt rules,” since the visit is user-initiated.
Google works differently because it does not run a separate AI-search crawler for these features. Its crawler documentation describes Google-Extended as a robots.txt token controlling use for Gemini training and grounding, and says it “does not impact a site’s inclusion in Google Search.” Its AI features documentation states that a page must be indexed and eligible to be shown with a snippet, with no additional requirements for AI Overviews or AI Mode. Blocking Googlebot would therefore hurt both classic and AI visibility, while blocking Google-Extended would not.
How to decide what to allow
Treat the choice as a short decision process, owned jointly by marketing, legal, and whoever manages the website.
- Decide whether AI answers are a channel you want. If buyers research vendors there, allow the search crawlers: OAI-SearchBot, PerplexityBot, and Googlebot.
- Decide separately whether you accept model training on your content. This governs GPTBot and Google-Extended.
- Check that your robots.txt, CMS security plugin, CDN, and firewall do not contradict each other. A bot allowed in one place and blocked in another is not allowed.
- Record the decision and the date, so later changes in citations can be traced to it.
A simple configuration that allows search crawlers and opts out of training looks like this:
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /Verifying the change
OpenAI’s documentation says it can take about 24 hours after a robots.txt update for its systems to adjust, and Perplexity gives a similar figure of up to 24 hours. Check results after that window, not the same afternoon. Server logs show whether the named user agents are actually reaching your pages, which is stronger evidence than a robots.txt file that looks correct.
Once access is confirmed, measure outcomes over repeated runs, since AI answers vary. Our guide to measuring AI search visibility explains how, and the agentic SEO overview shows where crawler monitoring can be automated safely. Note that a file telling AI systems what to read is a separate, weaker lever, covered in what the evidence shows on llms.txt.
What the evidence doesn’t support
None of the operator documents we read claim that allowing a crawler guarantees citations. They describe eligibility, not ranking. We found no verified study showing that allowing training crawlers improves or harms citation rates, so we make no claim either way. Operator documentation changes, and user-agent names have been added over time, so confirm the current lists on OpenAI’s, Perplexity’s, and Google’s own pages before editing production files.
Frequently Asked Questions
Will blocking GPTBot remove my site from ChatGPT answers?
Not according to OpenAI’s documentation. GPTBot relates to model training, while OAI-SearchBot controls whether a site appears in ChatGPT search answers.
Does blocking Google-Extended hurt my Google rankings?
No. Google states that Google-Extended does not affect a site’s inclusion in Google Search and is not used as a ranking signal.
How long until a robots.txt change takes effect?
OpenAI and Perplexity both indicate up to about 24 hours. Check server logs and citation tracking after that window before drawing conclusions.
Crawler access is a small technical setting with direct commercial consequences, which is why it belongs in a marketing review and not only in a developer ticket. At Grow with Hafsa we help teams audit AI search readiness as part of AI marketing and performance work, starting with what can be verified.
Access is only one input. For what structured data does and does not change, see schema markup for AI search.