Spoke
AI crawler access
Letting AI crawlers read your site is a real decision with a real trade-off. Most advice about it mixes up controls that do completely different jobs.
The crawler names and what each is for are factual and linked. How engines will treat these controls over time, and whether anyone reads llms.txt, is unsettled — so the advice here is graded speculative on purpose.
This page is graded speculative, and that grade is the most useful thing on it.
The crawler names are documented. What each one controls is documented. What isn't settled is how these controls get honored over time, whether the proposed conventions catch on, and what blocking costs you in the long run. Anybody giving you a confident recommendation here is extrapolating.
Should I block AI crawlers?
Depends whether your content is the product. A publisher selling access to writing has a coherent reason to block the training crawlers. A business that wants to get found, named and recommended almost never benefits from blocking the retrieval crawlers, because retrieval is the thing that makes a citation possible at all.
The decision that's almost always wrong is the blanket one — hunting down every AI user agent and disallowing the lot. That usually opts you out of being cited while achieving a training opt-out that one specific token would have handled on its own.
Two questions, not one
Do you want your content used to train models? That's a rights and business question. GPTBot, ClaudeBot and Google-Extended are the names that matter.
Do you want to be found and quoted when somebody asks a question? That's a visibility question. OAI-SearchBot, Claude-SearchBot, PerplexityBot and plain Googlebot are the ones that matter.
You're allowed to answer those differently, and the controls let you. Most firms want "no" to the first and a loud "yes" to the second. That's achievable, and it is not what a blanket block gives you.
Google is explicit about the asymmetry on its side. Google-Extended governs training and grounding in other Google products, and Google states it doesn't affect your inclusion in Google Search and isn't used as a ranking signal. The control for Search — and so for AI Overviews, which run on the Search index — is Googlebot.
OpenAI documents four separate crawlers: OAI-SearchBot, OAI-AdsBot, GPTBot and ChatGPT-User. They are controlled independently, so blocking one does not imply blocking the others.
Only OAI-SearchBot governs appearance in ChatGPT search answers. GPTBot governs use of content for training foundation models. Conflating the two is the most common error in AI-crawler advice.
What a thought-through setup looks like
For a professional firm that wants visibility and doesn't sell content, allowing everything is defensible. It's what we do here.
For an organization that wants a training opt-out while staying quotable, the shape is: disallow the training tokens by name, allow the search and retrieval tokens by name, leave Googlebot alone.
Two cautions. Blocking by IP rather than by user agent is specifically discouraged by at least one platform, because the crawler then can't read your robots.txt at all. And the rules have to be repeated on every subdomain.
llms.txt — not just unproven anymore
llms.txt is a proposed convention: a plain-text file summarizing your site for language models. It is not a standard.
This section is a downgrade on what we used to say. We previously said no major platform had documented reading one, which made it unproven. Google has since gone further and documented that it's unnecessary — you don't need new machine-readable files, AI text files, markup or Markdown to appear in Google Search, and such files neither help nor hurt visibility in its AI features.
That moves llms.txt from "unproven" to "documented as ineffective for Google." Everybody else publishes nothing either way.
So the honest summary is narrower than its fans claim and narrower than its critics claim. It does nothing for Google, and nobody else has said anything at all.
Publishing one is nearly free and harmless. Paying somebody for one as an AI-visibility measure is paying for something the largest relevant platform has said doesn't affect visibility in its features.
We publish llms.txt and llms-full.txt, and we say inside the files themselves that they aren't evidence the convention works.
What might replace it
There's a standards effort worth knowing about precisely because it isn't a standard yet.
The IETF has working drafts on AI preferences — a vocabulary for saying how content may be used, and an attachment mechanism defining a Content-Usage HTTP header with a matching robots.txt rule. Neither is published, both are still being revised, and nobody should assume any platform has committed to adopting them.
Why track it: it's the first attempt to solve this through a standards body rather than a convention somebody published and hoped would catch on. If anything replaces today's patchwork of per-platform user agents, it probably looks more like this than like llms.txt.
It's also, right now, a draft. There's nothing in it you can act on yet, and we'll keep saying so until that changes.
What would change this page
The grade moves off speculative when platforms publish how these controls affect citation, or when an adopted standard replaces the current proposals.
Until then: the mechanics are known, and the consequences aren't.
More in Technical SEO
Crawlability & indexing
Crawling and indexing are two different things, and a page can fail at either one. Most diagnosis goes wrong by treating them as a single step.
Architecture & URLs
How your pages are organized decides how they get found, how authority moves between them, and whether anyone can tell what the site is about. URLs are the visible shape of that decision.
Sitemaps & robots
These two files are where confident wrong advice concentrates, probably because they look simple enough that nobody bothers checking the documentation.
Structured data
Markup won't rank your page and won't make a claim true. Its job is to state outright what a machine would otherwise have to work out by guessing.
Core Web Vitals
Three metrics with published thresholds, measured on real visits rather than a test you ran once. No interpretation needed. You either meet them or you don't.
JavaScript & rendering
Google can render JavaScript" is true, and it gets used to justify architectures that cause real problems. The useful questions are when rendering happens, how reliably, and who else is looking.