Spoke
How AI engines actually get your content
Retrieval is what lets a model quote something you published this morning. Understanding it tells you which levers exist — and which widely repeated tactics are aimed at nothing at all.
Every crawler name, what it does and what blocking it controls comes from that platform's own documentation, retrieved 15 November 2026 and linked below.
A language model gets trained once, on a snapshot, and then it's frozen. On its own it can't tell you anything that happened after that snapshot closed, and it can't cite your page because it doesn't have your page. It has statistical residue from an enormous pile of text.
Retrieval changes that. When an engine goes and fetches live documents at the moment somebody asks a question, and hands them to the model as source material, the model is suddenly reading rather than remembering.
That's the whole mechanism behind citations, and nearly every lever you have sits inside it.
What is retrieval-augmented generation?
It's fetching live documents when somebody asks a question and giving them to a language model as source material for its answer. It's why a model with a year-old cutoff can quote an article from this morning, and it's why your page can be cited at all.
The practical consequence: the old technical requirements didn't go away. They became the price of entry. A page a crawler can't reach, or a renderer can't parse, isn't a weak candidate for citation. It isn't a candidate.
Two ways something can know you
Worth separating clearly, because the most common mistake in this area comes from blurring them.
Through training. Your content, or writing about you, was in the pile a model learned from. This produces knowledge with no link — a model might describe your firm accurately and cite nothing, because there's nothing to cite. You can't update it. It changes when somebody trains a new model.
Through retrieval. The engine went and got a page when the question was asked. This produces a citation with a link, reflects what's on the page right now, and is the only one of the two you can affect this week.
Almost all practical work targets the second. Treating the first as something you can optimize is a category error that burns a lot of effort.
The crawlers are not one crawler
Each company runs several, with different jobs and different controls. This is where most AI-crawler advice falls apart — it treats "block the AI bots" as one decision when it's four or five separate ones.
| Engine | How it gets content | Live web access | Citation display | Documented user agents | What blocking actually controls | Last verified |
|---|---|---|---|---|---|---|
| ChatGPT (OpenAI) | Both. Model-native knowledge from training, plus live retrieval for search answers and user-triggered fetches. | Yes, through OAI-SearchBot for search features and ChatGPT-User for user-triggered visits. | Inline links in search answers. | OAI-SearchBot, GPTBot, OAI-AdsBot, ChatGPT-User | The four are independent. Blocking GPTBot signals only that content should not train foundation models. Only OAI-SearchBot governs whether a site can appear in ChatGPT search answers; opted-out sites may still appear as navigational links. ChatGPT-User is user-triggered and robots.txt rules may not apply to it. | 2026-11-15 — source |
| Google AI Overviews and AI Mode | Retrieval over the Google Search index. | Yes. The same index that serves Search. | Supporting links shown alongside the generated answer. | Googlebot (Search). Google-Extended is a separate token and is not a Search crawler. | Google states there are no additional requirements to appear: a page qualifies as a supporting link if it is indexed and eligible to be shown in Search with a snippet. The preview controls that limit what is shown are nosnippet, data-nosnippet, max-snippet and noindex. Google-Extended is NOT a Search control -- it governs training and grounding in other Google systems, and Google states it does not impact a site's inclusion in Google Search nor is it used as a ranking signal. | 2026-11-15 — source |
| Perplexity | Retrieval-first. | Yes. | Inline numbered citations on essentially every answer. | PerplexityBot, Perplexity-User | PerplexityBot surfaces and links sites in Perplexity results and is documented as not crawling content for AI foundation models. Perplexity-User handles user-initiated requests and generally ignores robots.txt, because a person triggered the fetch. | 2026-11-15 — source |
| Claude (Anthropic) | Both. Model-native knowledge, plus web search and user-triggered fetches. | Yes, through Claude-SearchBot and Claude-User. | Inline links when answering from web results. | ClaudeBot, Claude-SearchBot, Claude-User | ClaudeBot collects content that may contribute to training. Claude-SearchBot crawls to improve search quality, and disabling it may reduce a site's visibility and accuracy in search results. Claude-User accesses sites when a user's question requires it, and disabling it may reduce visibility for user-directed web search. Blocking is per user agent in robots.txt, and must be repeated for each subdomain. Blocking by IP is specifically discouraged, because the bots then cannot read robots.txt at all. | 2026-11-15 — source |
| Microsoft Copilot | Retrieval over Microsoft's own index. | Yes. | Citations shown in AI answers, and reported back to publishers. | Bingbot. PARTIALLY VERIFIED -- see control. | Microsoft reports publisher visibility in its AI experiences through an AI Performance report in Bing Webmaster Tools, covering citations, cited pages and the queries associated with retrieval. Microsoft states that citation activity is not a measure of ranking, authority or page importance. The crawler-control details for Microsoft's AI experiences were still not readable from Microsoft's own documentation when this row was updated, so that cell remains partially verified rather than filled in from secondary coverage. | 2026-11-15 — source |
Read that table before you change a single line of robots.txt. The pattern holds across platforms: the crawler that decides whether you can appear in an AI answer is not the crawler that decides whether your content trains a model.
OpenAI documents four separate crawlers: OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User. They are controlled independently, so blocking one does not imply blocking the others.
Only OAI-SearchBot governs appearance in ChatGPT search answers. GPTBot governs use of content for training foundation models. Conflating the two is the most common error in AI-crawler advice.
The consequence is blunt. Somebody who wants out of model training, and blocks every AI user agent they can find, has in several cases also opted out of being quoted — which is the opposite of what they wanted.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot governs whether your content can be used to train OpenAI's foundation models. Showing up in ChatGPT's search answers is governed by OAI-SearchBot, which is a different user agent with a separate rule.
Google documents the same split from the other side: Google-Extended controls training and grounding in other Google products, and Google states it doesn't affect your inclusion in Google Search and isn't used as a ranking signal. The control for Search — and so for AI Overviews, which run on the Search index — is Googlebot.
What Google says qualifies a page
This is the most useful documented fact in this section, because it contradicts a pile of advice about special AI-era requirements.
Google says there are no extra requirements to appear in AI Overviews or AI Mode. A page qualifies as a supporting link if it's indexed and eligible to show in Google Search with a snippet. That means meeting the ordinary technical requirements and not suppressing your own snippet.
That last part is where sites hurt themselves without noticing. nosnippet, data-nosnippet and a tight max-snippet limit what Search can show from your page — and they take the AI eligibility with them.
What this doesn't tell you
Everything above is documented. The following isn't, and we're not going to pretend otherwise:
- How any engine ranks or chooses between the sources it already retrieved.
- Whether being cited once makes the next citation more likely.
- Whether any particular formatting convention increases how often you get quoted.
Those are the questions everybody wants answered. They're exactly the ones nobody publishes. Advice about them circulates with great confidence and no method behind it. Where we discuss them at all, we label them speculative and say what the reasoning rests on.
More in GEO & entities
What Google documents
Google published a guide to showing up in its AI features, and a big chunk of it is a list of popular tactics it says you don't need. That makes it the most useful source in this field, and the most uncomfortable one.
Training vs retrieval
One of these you can change this afternoon. The other changes when somebody else trains a model. Knowing which is which tells you where effort is worth spending and where it is wasted.
Platform matrix
One row per engine, each with its own last-verified date and a link to the platform's own documentation. Where a row could not be verified, it says so instead of being filled in from memory.
Entity consistency
The highest-value work in getting quoted by AI is also the dullest. Describe your company the same way everywhere, give it one stable identifier, and stop handing machines reasons to hedge about who you are.
Comparative content
Comparison questions are among the most valuable a buyer asks and among the most common an AI engine is given. The format decides whether you can be quoted — a table in HTML can be, the same table as an image cannot.