Many businesses now ask whether to block AI crawlers in robots.txt, and what that would do to their presence in ChatGPT, Gemini and Google’s AI features. A BuzzStream study of about 4 million citations gives a partial answer. It also shows why a simple allow-or-block rule is a weak way to control AI visibility.
- BuzzStream’s 2026 analysis of about 4 million citations from 3,600 prompts found that most sites blocking AI bots in robots.txt still appeared in AI citations.
- Among the top 50 news sites that blocked ChatGPT-User, 70.6% were still cited. For OAI-SearchBot the figure was 82.4%.
- The sample is news publishers, not typical businesses, and BuzzStream says the causes are educated guesses.
- Treat robots.txt as a policy statement about training and crawling, not as a guarantee about what AI answers show.
What the study found
The claim is simple: blocking an AI bot in robots.txt did not reliably keep a site out of AI answers. BuzzStream, a link-building and digital PR software company, analysed about 4 million citations from 3,600 prompts across 10 industries. It used Citation Labs’ XOFU citation-tracking tool and covered ChatGPT, Gemini, Google AI Overviews and AI Mode.
For ChatGPT’s bots, BuzzStream looked at the top 50 news sites that block them. Of those blocking ChatGPT-User, 70.6% still appeared in citations. Of those blocking OAI-SearchBot, 82.4% did. BuzzStream also reported that about 70% of ChatGPT citations in its data came from sites that block ChatGPT’s retrieval or search bots. The implication for you is that a block does not behave like a removal request.
Which bot does what
The bots serve different purposes, so one blanket decision is rarely right. OpenAI’s own documentation says OAI-SearchBot surfaces sites in ChatGPT’s search features, and that sites opting out will not appear in ChatGPT search answers, though they can still show as navigational links. GPTBot collects content that may be used to train OpenAI’s models. ChatGPT-User visits pages when a person asks ChatGPT to, and OpenAI says robots.txt rules may not apply to those user-initiated actions.
| Bot | Purpose per OpenAI | Blocking sites still cited (BuzzStream) |
|---|---|---|
| ChatGPT-User | Visits a page when a user asks | 70.6% |
| OAI-SearchBot | Surfaces sites in ChatGPT search | 82.4% |
| GPTBot | Crawls content that may train models | 88.2% |
We left out Google-Extended because the BuzzStream article reports its figure two different ways. The study’s sample for the ChatGPT bots is the top 50 news sites, so read the table as a signal, not a benchmark for your site.
Why blocking may not remove you
BuzzStream is candid that it cannot prove why. Its author writes that the team can only take educated guesses. It notes that robots.txt is a directive, not an enforcement mechanism, and that bots can ignore it. It also suggests some AI retrieval may rely on search-result data such as titles, URLs and snippets without fetching the page, and that older crawls, such as Common Crawl, may explain some citations.
These are hypotheses, not findings. The practical reading is narrower. If your robots.txt says no, your content may still be known to an AI system through another route. That matters most if you block because of confidentiality, since robots.txt does not secure anything. Password protection or removal does.
Our earlier guide to which bots to allow in robots.txt covers the access side of this. Our article on JavaScript rendering and AI crawlers covers a different reason a crawler may not see your content.
How to decide for your business
The study samples publishers, whose business model depends on traffic and licensing. A service business that wants to be recommended has a different goal. Treat the steps below as our recommendation, not a rule from BuzzStream or OpenAI.
- Decide your goal first: appear in AI answers, avoid model training, or both.
- Check your current robots.txt and note which AI bots it names.
- If visibility is the goal, allow the search and retrieval bots, such as OAI-SearchBot, and treat training bots like GPTBot as a separate choice.
- Do not rely on robots.txt for anything confidential.
- Test: track whether your pages appear in answers for a fixed prompt set, before and after any change.
Retrieval bot: a crawler that fetches pages to answer a live question. Training bot: a crawler that collects content to build or improve a model. Keeping the two apart stops a privacy decision from quietly costing you visibility.
Because AI answers change between runs, a single before-and-after check proves little. Our guide to measuring AI search visibility when answers change every run explains how to repeat the test.
Frequently Asked Questions
Does blocking GPTBot keep my site out of ChatGPT answers?
Not necessarily. OpenAI describes GPTBot as a training crawler, separate from OAI-SearchBot, which handles ChatGPT search. In BuzzStream’s data, 88.2% of sites blocking GPTBot were still cited.
Should a business block AI crawlers?
It depends on the goal. If you want to be found in AI answers, blocking search and retrieval bots works against you. If you object to model training, block training bots and leave search bots alone.
Does this study apply to small business websites?
Only loosely. The ChatGPT-bot figures come from the top 50 news sites. Run your own prompt tests before assuming the same pattern holds for your pages.
Bot access is one setting among many in an AI search programme, and it is easy to get wrong in either direction. At Grow with Hafsa we help US and UK businesses audit crawler access and measure what changes, as part of our AI marketing and performance marketing services.
Google has since added a separate Search Console control for its generative AI features; our article on Google’s AI search opt-out and the UK CMA requirement explains how it differs from blocking crawlers.