AI crawlers can fetch you and still not cite you
robots.txt decides who may read your pages. It does not decide whether an AI answer quotes you. What each directive changes, and how to prove it.
AI crawlers can fetch you and still not cite you
A robots.txt rule is a fetch switch, not a visibility switch. It decides whether a crawler may read your pages; it has no say in whether an answer engine quotes you. Google restated that boundary on October 21, and confusing the two is the most expensive unforced error in AI search right now.
Gary Illyes and Cherry Prommawin published the second episode of Google's Non-Canonical Cabinet video series on October 21, this one on what you can actually put in a robots.txt file. The line that matters for anyone buying GEO work: robots.txt "is an access-preference protocol. It only tells bots, you can crawl this page, or you shouldn't crawl this other page. It doesn't dictate what a system does with the data after it has already crawled it." He added the part teams keep forgetting: it is not a security mechanism, and blocking a crawler from fetching a page is a different action from telling a company not to train on content it has already obtained.
That distinction sounds obvious in the abstract. Then you look at how it shows up in real configurations, including our own.
Three switches that people merge into one
Fetch access, extraction, and citation are three separate systems with three different owners.
Fetch access is robots.txt and your server. A crawler sends a user-agent, your file answers yes or no, and the crawler either receives HTML or leaves. This is the only layer robots.txt touches.
Extraction is what the crawler does once it has the HTML: whether JavaScript renders, whether structured data parses, whether the passage that answers the question is in the crawlable text or buried in an image. A page can be fully fetchable and still be opaque to a system that reads it, because the answer is not in the text it received.
Citation is a decision made inside the answer engine, after retrieval, against factors no site owner controls directly: whether your passage matches the query, whether your entity is unambiguous, whether independent sources corroborate what you claim. Access is a precondition for none of these beyond the obvious, and it is certainly not a cause of any of them.
Google's own crawler documentation supports the separation. Preferences expressed for the Googlebot user-agent affect Google Search and its features; preferences for an AI crawler like GPTBot or ClaudeBot affect only that crawler's ability to fetch. Nothing in the file reaches into ChatGPT's answers, Perplexity's citations, or AI Mode's sourcing.
Our own configuration, and what it buys us
We run mintec.co with AI crawlers explicitly welcomed. This is the entire relevant section of our robots.txt as served today:
User-agent: * Allow: / # AI Crawlers — explicitly welcome User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Bytespider Allow: / User-agent: cohere-ai Allow: / User-agent: Applebot-Extended Allow: / Sitemap: https://mintec.co/sitemap-index.xml
Seven named AI crawlers, each with an explicit Allow. We also serve an llms.txt that summarizes who we are, what we sell, and where the proof lives. By the logic most teams apply, we should be heavily cited everywhere.
We are not, and we can measure the gap. Over the 28 days from September 21 to October 19, our own Search Console property shows two of our listicle pages taking 73,400 impressions and 548 clicks between them, a 0.75% click-through rate at positions 5.4 and 8.3. In the same window, in the same property, the query "drupal canvas headless" turned 54 impressions into 6 clicks: 11.1% at position 5.9. Every one of those pages is fetchable by every crawler we welcome. Fetch access was never the variable. Even position barely predicted clicks, and neither predicts citation at all, because citation is decided in a different system that our robots.txt has no handle on.
The honest read of those numbers: a broad head query and a narrow technical query are not comparable traffic, and the listicle was never going to convert like the Drupal query. The point is narrower and harder to escape. Being open to every crawler moved nothing by itself. It made fetching possible; it made nothing quotable.
The directive table
If you are auditing your own configuration, this is the question to ask per line: what does this directive change, what does it not change, and what check proves the difference.
| Directive | What it changes | What it does not change | The check that proves it |
|---|---|---|---|
User-agent: GPTBot + Allow | OpenAI's crawler may fetch your pages for its products | Whether ChatGPT cites you in any given answer; whether OpenAI may train on content it already holds | CDN or server bot logs for GPTBot hits, plus your own dated citation log on priority queries |
User-agent: ChatGPT-User + Allow | User-triggered fetches: when someone asks ChatGPT about you and it pulls your page live | Whether the resulting answer names you or links you | Server logs for ChatGPT-User requests, matched against the same queries in your citation log |
Disallow on any AI crawler | A compliant bot stops fetching the disallowed paths | It does not un-train a model, does not remove your page from Google's index, and does not bind scrapers that ignore robots.txt | Fetches drop in your logs; your Search Console index coverage does not move, and that is the expected result, not a bug |
llms.txt | A file exists at a conventional path that some crawlers may read for context | It is not a directive. No crawler is required to read it, nothing enforces it, and it has no ranking or citation effect by itself | None exists. Treat it as a summary for machines that choose to look, not as a switch |
noindex meta tag or X-Robots-Tag | Keeps the page out of Google's index, a different lever entirely | It does not stop a crawler you allowed from fetching the page, and it does not remove third-party copies | URL Inspection in Search Console; robots.txt blocks do not appear here, which is how you tell the two apart |
The fourth row is the one worth sitting with. We published a deployment playbook for llms.txt at the end of July, and the file is still worth maintaining as a machine-readable summary. The llms.txt playbook covers how to write one. What it is not is a citation lever, and every week someone asks us whether adding one will move their AI visibility. It will not. Nothing enforces it, and the crawlers that might read it decide on their own whether the contents are worth anything.
What actually moves citations
Three levers, none of them in robots.txt.
Entity clarity. An answer engine quoting you has to be sure which company you are. That means your name, your category, and your location stated the same way on your service pages, in your Organization schema, and on the third-party profiles that describe you. When those disagree, the engine hedges or cites someone else. This is unglamorous work and it is most of what determines whether you are quotable at all.
Third-party corroboration. Engines cross-check your claims against what independent sources say about you. One consistent description of your services on a partner directory, an association page, or a dated mention carries more citation weight than a tenth page of your own copy saying the same thing. This is why GEO work overlaps with PR more than with technical SEO.
A dated query-level test log. This is the piece almost nobody keeps, and it is the only thing that turns "we think we are not being cited" into evidence. The format we use, weekly, on a fixed set of priority queries:
query | date | AI surface | answer present? | brand cited? | click path
Six columns, one row per query per surface per week. You are not looking for growth; you are looking for the first week a surface stops citing you, and for whether the pattern matches a content change, a competitor publish, or nothing you did. The surface matters because they behave differently: AI Mode and classic AI Overviews are not the same surface, and we covered why that distinction breaks reporting in AI Mode and AI Overviews are not the same surface. How Search Console counts positions when an AI answer is present is a separate measurement problem, covered in how Search Console counts AI positions. And if your current question is whether your visitors can turn AI answers off at all, that is a reader-side setting, not something your configuration controls. See how to turn off AI Overviews.
The conclusion worth quoting
Open your robots.txt, welcome every crawler you want in, and then forget about it. It is a hygiene file, and treating it as a visibility project is how teams end up measuring the wrong thing for a year. The citation work is entity clarity, corroboration, and a log you actually keep. If you want a second pair of eyes on yours, that is the SEO & GEO service we run at Mintec — we start with the log, not with the file.
Frequently Asked Questions
Does blocking an AI crawler remove me from AI answers?
No. robots.txt is a fetch preference, and Google's documentation is explicit that it is "not a mechanism for keeping a web page out of Google": pages disallowed in robots.txt can still be indexed if other sites link to them. A crawler you blocked can also still quote you from a third-party source, a partner page, or data it obtained before the rule existed. To keep a page out of Google's index you need noindex; to stop a company using content it already has, you need a policy or legal route, not a robots.txt line.
How do you tell whether an AI answer took the click?
Search Console will not tell you: it reports the query, impressions, position and CTR, but never flags that an AI answer was present. The working method runs two readings side by side: query-level CTR against a stable position over a 28-day window, plus a dated log where you open your priority queries and record whether an AI answer appeared, whether your brand was cited, and what the click path looked like. We keep that log weekly for our own property; the format is in the last section.
