Robots.txt is not a lock: OpenAI's 3 bots and who really decides your ChatGPT citations

OpenAI's documentation now says robots.txt may not apply to the bot ChatGPT uses to fetch pages live, and that the agent deciding whether your site shows up in ChatGPT Search is a different one: OAI-SearchBot, with its own token and its own rules. Blocking all three OpenAI bots to "stop AI traffic" doesn't make you invisible. It removes you from citations, which is exactly the visibility that matters when traffic comes from generated answers. Here's what the docs actually say, why your robots.txt is not a lock, and the access policy we use in Mintec's GEO audits.

What changed on August 14

On August 14, Search Engine Journal covered something many of us assumed until we read the fine print: according to OpenAI's documentation, robots.txt may not apply to ChatGPT's fetch bot, and the agent responsible for whether your site appears in ChatGPT search results is called OAI-SearchBot, not ChatGPT-User.

OpenAI runs three bots with different jobs, each with its own robots.txt token:

The settings are independent: you can allow OAI-SearchBot to appear in results while disallowing GPTBot so your content isn't used for training. The docs also note that after a robots.txt change, the search index can take about 24 hours to adjust.

Why your robots.txt is not a lock

Robots.txt is part of the Robots Exclusion Protocol, formalized as RFC 9309, and the protocol is voluntary: it documents a convention, it doesn't enforce one. Nothing physically stops a bot from ignoring the file.

And AI bots ignore it more and more. Figures that circulated in this week's coverage: 13% of AI bot requests bypassed robots.txt in Q4 2025, up 400% from Q2, and Cloudflare went as far as de-listing Perplexity as a verified bot after catching it masquerading as a Chrome browser. The ones that respect it respect it; the ones that don't don't announce themselves.

The finer point in the SEJ piece is different. Sites that block both OAI-SearchBot and ChatGPT-User to "prevent AI traffic" have traded away the visibility half of the deal while keeping a control that carries a carve-out: ChatGPT-User's live fetch is a user-initiated action, not automated crawling. In practice, the control you thought you had doesn't protect what you wanted to protect.

There's also a layer that never appears in your robots.txt: Cloudflare moved its crawler controls to the network layer. If you switched on the "Block AI Bots" toggle, that overrides your file, including any Allow you wrote for OAI-SearchBot.

The asymmetry between engines matters too. Google sticks to its historical line: Googlebot respects robots.txt and Google-Extended controls whether your content trains Gemini, with no documented exceptions for AI Mode's live fetch. OpenAI documents the exception instead. Perplexity has the shakiest track record of the three, with crawlers caught masquerading as browsers. Your robots.txt doesn't have one behavior, it has one per engine, and the policy you build has to assume some of them will follow it to the letter while others will negotiate.

The access matrix we use in GEO audits

When Mintec audits a site for generative search, the first finding is almost never about content. It's about configuration. Sites that blocked "all AI bots" with a generic robots.txt or a Cloudflare toggle, then wonder why they don't show up in ChatGPT while their competitors do. The decision isn't binary, there are four strategies:

1. Block everything (Disallow GPTBot, OAI-SearchBot, and ChatGPT-User). You disappear from ChatGPT Search and from training. It only makes sense if you want nothing from AI traffic, and even then it's not a guarantee: user-initiated live fetches can still arrive.

2. Citations yes, training no (our recommendation). Allow OAI-SearchBot and ChatGPT-User, block GPTBot. This is the default we set for most business sites:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

3. Training yes, citations no. The rare case: sites that want to sell access to their content for model training. Unless you have a signed licensing deal, don't do this.

4. Allow everything. Maximum reach, zero control. For sites that live on visibility and have no proprietary content to protect, it's a legitimate option.

The rule we carry out of every audit: decide whether you want citations first, then decide about training. They are two decisions, not one.

And a note on who actually makes these blocking calls: almost always an infrastructure or security team that sees AI bots as a scraping threat and applies the broadest block available, without consulting the people who measure results. The outcome is a visibility decision made by the team that doesn't track visibility. If IT owns your robots.txt, this post is worth forwarding to them before the toggle wins by default.

What we're already seeing in our numbers

In our weekly GEO traffic report, we've been measuring LLM referrals for months: small sessions, dozens per week, but converting several times better than organic. The new part is the split between citations and visits: pages ChatGPT cites without the user ever entering, invisible in GA4 but visible in the Search Console AI report.

That split changes the robots.txt math. When traffic was all clicks, blocking a crawler was free. Now every block is a decision about future citations: block OAI-SearchBot and you don't lose visits you already had, you lose the chance to be cited in answers that don't exist yet. It's the same logic as our work with crawlers that don't execute JavaScript: first make sure you can be read, then argue about the terms.

The same principle applies to llms.txt: it's not a blocking file, it's a presentation file. Where robots.txt says what you don't want read, llms.txt says what you do want read and how to cite it. Blocking every AI bot and then publishing an llms.txt is contradictory: you're locking the front door and leaving the window wide open.

Checklist for this week

1. Open your robots.txt and list which AI user-agents are blocked. If you have a generic "AI bots" block, it probably includes OAI-SearchBot.
2. Check Cloudflare or your WAF: if the "Block AI Bots" toggle is on, it overrides your robots.txt.
3. Make the explicit call: do you want to be cited in ChatGPT? If yes, Allow OAI-SearchBot and ChatGPT-User.
4. Verify in your server logs which bots actually arrive. Robots.txt shows what you asked for; logs show what happened.
5. Measure with the Search Console AI report for two weeks before and after the change, and compare against the citation profile of each engine.

Robots.txt is not a lock, it's a letter of preference. OpenAI reads it, honors it in part, and now documents its exceptions. The one thing that stays your call is whether you want to be in the conversation. Our answer is easy: the citation is the new currency, and getting ChatGPT to cite you starts with not blocking yourself.