ChatGPT's index no longer runs through Google: what the Cloudflare pilot changed
The Cloudflare-OpenAI pilot indexed hundreds of millions of unlinked pages into ChatGPT's search index. What it means for GEO, what each OpenAI bot does, and the robots.txt audit to run this week.
ChatGPT's index no longer runs through Google: what the Cloudflare pilot changed
On October 15, Cloudflare and OpenAI published the results of a research pilot that started in July. One-line summary: with network traffic signals from roughly 1% of Cloudflare's network feeding OpenAI's search crawler, the company discovered and indexed "hundreds of millions of new web pages" into the index ChatGPT uses to search. The immediate read for anyone doing GEO work: visibility in ChatGPT just stopped depending on Google's link graph. Useful pages that almost nobody links to can enter the index, and the decision that matters most this week is whether your robots.txt lets OAI-SearchBot through.
If you've been treating GEO as "SEO under a new name," this announcement is worth reading slowly. Not because link building dies today, but because, for the first time, one of the two biggest AI search engines confirms in public that it discovers the web with signals other than links.
What they actually announced (and what they didn't)
The joint Cloudflare and OpenAI post (published October 15, 2026) says, in their words:
- The pilot is research and started in July 2026, with a participating cohort of ~1% of Cloudflare's total network.
- Cloudflare's signals helped OpenAI's search crawler discover useful pages, and OpenAI "independently evaluates and ranks" those pages for relevance and helpfulness.
- The result was the discovery and indexing of "hundreds of millions of new web pages" for ChatGPT search.
- The data used is metadata about the web for search and discovery, "not page content for model training."
- A signal about a page "doesn't grant access permission": your robots.txt and your settings still decide who gets in.
That last line is the one most Twitter threads about the announcement skip. Cloudflare also tested safety signals that flag harmful pages individually (their example was a fake login form planted by an attacker), without penalizing the whole site.
And what the announcement doesn't say: that links stopped mattering, that pay-per-crawl left beta (it's still in private beta), or that your site entered the index without you doing anything. The pilot solves discovery, not access.
The real consequence: two ways into AI, not one
Until now, most brands assumed a chain of dependency: Google ranks your page, ChatGPT reads what Google ranks (via Bing or another inherited index), and your SEO strategy served both. That assumption was fragile, and the pilot confirms it: ChatGPT is building its own discovery muscle, based on which pages real people actually visit (Cloudflare network signals) and evaluated by its own ranking.
Same week, on October 15, Google separately announced "substantial increases" of inline links inside AI Mode and AI Overviews (we covered all five changes here). Two announcements the same day, in opposite directions: Google reinforces its chain of links; ChatGPT builds a door that doesn't go through it.
Does that mean traditional SEO dies? No, and the difference between SEO and GEO is unchanged: SEO puts you in front of people searching on Google; GEO puts you inside the answers models give. What's new is that those answers no longer need Google to have ranked you in order to find you.
OpenAI's three bots: the audit that actually matters
This is the part most marketing teams get wrong. OpenAI runs three agents with different jobs, and the right policy differs for each:
| Bot | What it does | If you block it |
|---|---|---|
| OAI-SearchBot | Discovers and indexes pages for ChatGPT's search answers | Your site won't appear in ChatGPT search answers (it can still appear as a navigational link) |
| GPTBot | Crawls content that may be used to train foundation models | You're only out of training; your search presence is unaffected |
| ChatGPT-User | Fetches pages when a specific user asks ChatGPT for something | User requests can still come through; OpenAI notes robots.txt rules "may not apply" because a person initiated the request |
The setup most teams pick when they want visibility without feeding models looks like this:
User-agent: OAI-SearchBot Allow: / Disallow: /dashboard Disallow: /settings User-agent: GPTBot Disallow: / Sitemap: https://yoursite.com/sitemap.xml
OpenAI indicates robots.txt changes take about 24 hours to be picked up for search. And if your site runs on Cloudflare, the AI Crawl Control panel (available on all plans) shows which AI crawlers actually hit your site and whether they follow your rules, which is a more reliable source than what you think your file says.
What to run this week
- Split the bot policy. If you added a blanket "block all AI bots" rule in 2024 or 2025 and never touched it again, you're probably invisible to ChatGPT search without knowing it. Write the OAI-SearchBot and GPTBot policies separately and document them.
- Find your orphan pages. The ones with zero backlinks that answer a real question from your audience are exactly what signal-based discovery puts in play. Those pages, with a clear title and a direct answer up top, are your new advantage over brands that only publish what their PR campaigns link to.
- Basic indexing hygiene. Made-up URLs should return a real 404, not a soft 200; temporary redirects (302) that should be permanent (301) need fixing; and your sitemap's lastmod field should reflect real changes, not build time. None of this is glamorous, which is why most teams let it rot.
- Measure presence, not just traffic. Ask ChatGPT your key topics directly with search enabled and note whether your brand appears, how it describes your product, and which sources it cites. Search Console's generative AI performance report (worldwide since August 31, 2026) shows your site's impressions inside AI Overviews and AI Mode, page by page.
Our numbers, because GEO talk without data usually means too much confidence
In the last 7 days, mintec.co received 30 sessions from AI search referrals: 16 from ChatGPT, 11 from Gemini, 2 from Claude, and 1 from Perplexity. The page ChatGPT sent people to most isn't a blog post: it's our contact page, with five sessions.
That pattern held through the six months of real AI traffic data we published: when someone arrives from an AI search engine, they arrive decided. The model already described your service, sometimes compared you, and the click is the last step, not the first. That's why clickless visibility is still a business outcome, even if your GA4 dashboard refuses to reward it.
One detail from this week that backs the point: since the GPT-6 rollout (October 7 and 8), links inside ChatGPT's answers started opening without UTM tags, and several panels measured drops of up to 46% in sessions attributed to ChatGPT. People kept showing up; they stopped showing up counted. If your ChatGPT traffic halved this month, read GA4 side by side with Direct before you touch the strategy.
What doesn't change
The announcement doesn't turn GEO into a bag of technical tricks. If anything, why manipulating AI answers won't last still holds: models learn to distrust manufactured signals, and bought-account Reddit spam is already being filtered. The Cloudflare pilot reinforces the same idea from the discovery side: the signals feeding the crawler are real usage patterns, not link campaigns.
In practice, this makes GEO priorities pretty simple. First, your content answers questions people actually ask, with the answer at the very top. Second, the bots that decide your presence can read it. Third, you check presence with your own eyes once a month, not just with whatever the dashboards decide to tell you.
The link graph is still alive; it's just no longer the only door into AI answers.
Frequently Asked Questions
Does the Cloudflare and OpenAI pilot mean links no longer matter?
No. Links still matter for Google and for the reputation models associate with your brand. What changed is that ChatGPT no longer depends on Google's link graph to discover pages: the pilot fed its search crawler network traffic signals from ~1% of Cloudflare's network and indexed hundreds of millions of pages that almost nobody links to. The practical consequence is that a useful page with zero backlinks can now show up in ChatGPT just like a heavily linked one.
How do I know if ChatGPT indexed my site?
Three signals: first, ask ChatGPT directly (with search enabled) about your brand or the topics your key pages cover and see if you appear; second, check the Search Console generative AI performance report, which since August 31, 2026 shows your site's impressions inside AI Overviews and AI Mode; third, if you're on Cloudflare, the AI Crawl Control panel shows which AI crawlers actually hit your site and whether they follow your robots.txt.
Do I have to allow GPTBot to appear in ChatGPT?
No. They are different bots with different jobs: OAI-SearchBot decides whether your site appears in ChatGPT's search answers; GPTBot is the training crawler, and blocking it only keeps you out of training data; ChatGPT-User fetches pages when a specific user asks for something. You can allow OAI-SearchBot and block GPTBot at the same time, which is the setup most teams pick when they want visibility without feeding models.


