Which index holds your page decides who can cite it

Answer engines don't read the web. They read an index, and each engine reads a different one. So the first question worth answering about AI search isn't "am I cited" — it's "which index holds this page, and who buys that index". Until you can answer that, every visibility score you buy is a number about somebody else's corpus.

On 2 November the New York Times, the Daily News, the Chicago Tribune and the Orlando Sentinel sued Brave in the Southern District of New York: a 69-page complaint, five counts each, over the browser's search index, its "Ask Brave" answers and its API endpoints. The part worth reading twice is buried in the middle. The complaint names Anthropic, Perplexity, Mistral, xAI and Cohere as customers of the Brave Search API. None of them is a defendant. Ars Technica quotes the filing directly: "AI companies pay Brave to provide them with real-time online content from the Brave search index" — to train models, and to answer specific user queries. Brave's own API page lists Cohere and Mistral among the labs using it.

That's a copyright argument, and it will take years. The operational fact inside it takes an afternoon to act on: your page can sit inside a commercially licensed corpus that you never configured a rule for, and the crawler filling that corpus has no name you can put in robots.txt.

That's not my inference. It's Brave's documentation.

Three gates into an AI answer

"AI crawlers" is treated as one list. It's three different mechanisms with three different owners, and only two of them accept instructions from you.

Start with gate one, because the vendors have finally documented it properly — and the documentation is more specific than the SEO advice built on top of it.

OpenAI's crawler page draws the split itself. OAI-SearchBot is for search: "sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." GPTBot is the training crawler, and disallowing it says nothing about search. ChatGPT-User is the third: a fetch triggered by a person's question, where OpenAI notes that "robots.txt rules may not apply" because the request came from a user.

Perplexity says the same thing in its own words. PerplexityBot "is designed to surface and link websites in search results" and "is not used to crawl content for AI foundation models." Perplexity-User handles live fetches when someone asks a question, and Perplexity publishes a separate IP range file for each bot. Anthropic runs the same three-way split with ClaudeBot for training, Claude-SearchBot for indexing search results and Claude-User for user-directed fetches.

Notice what that means for the standard audit. "We allow all AI crawlers" is not a position, because allowing a training crawler and allowing a search crawler are unrelated decisions with unrelated consequences. And a blanket User-agent: * allowance is the default state of most sites — which means most sites are already in gate one for every engine, and their AI visibility problem is not permissions at all.

Then there's Google, where the confusion is worse. Google-Extended is a switch for Gemini training and grounding. Google documents that it does not affect a page's inclusion in Search, and it does not remove you from AI Overviews or AI Mode either — those draw on the Googlebot index. If you want out of Google's AI surfaces, the only switch that does it is blocking Googlebot, and that takes you out of Google Search too. Teams routinely block Google-Extended, announce they've opted out of AI, and change nothing about what they see in an AI Overview.

Gate two: the index with no token

Brave's crawler help page contains the sentence that makes the whole licensed-corpus problem concrete. The Brave Search crawler "does not advertise a differentiated user agent because we must avoid discrimination from websites that allow only Google to crawl them." It continues: "if a domain or page is not crawlable by Googlebot, then Brave Search's bot will not crawl it either."

Read that again from the site owner's side. You cannot name the Brave crawler, because there is no name. You cannot disallow it specifically. And the rule it obeys is your Googlebot rule — the one most sites never touch, because touching it would cost them their search traffic.

So a site that disallows Googlebot to keep content out of AI training has, without intending it, removed itself from a corpus that at least five named AI companies buy through an API. And a site that writes User-agent: Bravebot / Disallow: / has written a rule that binds nothing, because Brave doesn't publish that token.

That second failure is not hypothetical. An independent audit of 1,025 robots.txt files published in early November found seven files naming a bravebot token, six of which disallowed the site root to it — and all six left Googlebot open at the root. Six sites wrote a rule for a name that doesn't exist and left the switch that does exist untouched. That is the entire failure mode of AI crawler configuration in one data point, and it will repeat for every new index that gets sold before it gets documented.

The test: four steps, one afternoon

This is the procedure I'd run before spending money on any visibility tool, and it's the one I run on client properties.

1. Name the surfaces, not "AI". Pick three to five queries that carry commercial intent for you, and write down which surface each one actually appears on: ChatGPT search, Perplexity, an AI Overview, the Gemini app. "AI search" as a target is unmeasurable; five named surfaces are not.

2. Run the membership test per surface, and date it. Search site:yourdomain.com query-term inside each engine — the search box where it has one, and the assistant where it doesn't. What this proves is narrow and useful: whether that engine's index holds a representation of the page. What it does not prove is that the page will ever be selected, because retrieval and selection are different steps. Run a control query you know is indexed in the same session, so you can tell "page missing" apart from "test broken".

3. Inventory the agents that actually visited. Pull 30 days of server or CDN logs, group by user agent, then verify each hit against the vendor's published IP ranges — OpenAI and Perplexity both publish per-bot JSON files. User agent strings are copyable in one line of code; IP ranges are the part a scraper can't fake. What you're looking for is which documented token touched the site, and whether there are fetches you can't attribute at all. In client work, the unattributable bucket is usually the interesting one.

4. Write the directive table and mark the load-bearing row. One row per token: rule now, surface it changes, surface it can't change, and the check that proves the difference. Then add the row that has no token and mark it as load-bearing, because on a site that allows Googlebot, the licensed corpus is open by default and nobody on the team decided that.

What the test changes

Two decisions usually come out of it, and both are uncomfortable.

If your AI answer is bad, permissions are rarely the cause. When a site is in gate one for every engine and still invisible, the constraint is extraction and selection — what the crawler can read, and whether the passage answers the question. No robots.txt edit reaches that layer, which is why another crawler audit won't move the number.

If your AI answer is good and your risk appetite is low, the options get worse the closer you look. There is no switch that says "index me for search, keep me out of every licensed AI corpus". Blocking Googlebot does both. Blocking Google-Extended does neither. Blocking OAI-SearchBot removes you from one engine's answers and leaves the resold corpus untouched. Anyone selling you a single "AI opt-out" line is selling a control that doesn't exist.

The opinion, since it costs nothing to state plainly: the crawler-config business sells lists of tokens, and what you need is a map of who owns each index and which of your switches each one obeys. Most properties have neither, which is exactly how "we allow every AI crawler" and "we're invisible in AI answers" end up true on the same site at the same time.

And keep the boundary straight: being inside the index is a precondition, not a cause. A page can be in every corpus and still lose the answer to a competitor whose paragraph is clearer — that's a different problem with a different fix, and it's the one most AI visibility audits skip.

The document I'd keep

One page, five columns: surface, index owner, token, current rule, and who else buys that index. Fill it in for your own property and the surprises are all in the last column, because that's where a supplier you've never spoken to turns out to be distributing your pages to five companies you can't address directly.

If the last column is empty because nobody knows who buys the corpus, that's the finding. It's also the point where a GEO engagement earns its fee: not a bigger token list, but a map with names on it and directives that match. The two posts worth reading next are the llms.txt playbook if you're still deciding what to expose, and the AI Overviews versus AI Mode comparison if you're trying to work out which surface is eating your clicks.

The corpus question is upstream of both. Answer it first.