Spoke
How search works
Three stages, three distinct failure modes, and one diagnostic habit that saves most of the time people waste guessing. Everything else on this site assumes this model.
The crawl, index and serve pipeline and the separation between its stages are described in Google's own documentation on how Search works, linked below.
Three things happen between a page existing and a person seeing it, and they are independent. Confusing them is the single most common reason a diagnosis goes wrong and a month gets spent on the wrong fix.
What are the three stages of how a search engine works?
Crawling, indexing and serving. Crawling is discovery and fetching — a crawler follows links and sitemaps to find URLs and download them. Indexing is analysis and storage — the system parses the content, works out what the page is about, and decides whether to keep it. Serving is selection — for a given query, the system chooses from the index and presents results.
The stages are sequential and none of them is guaranteed. A page that is crawled may not be indexed, and a page that is indexed may never be served for anything.
Crawling
A crawler discovers URLs mostly by following links, supplemented by sitemaps and other submissions. It fetches what it can reach, subject to robots directives, server responses and its own scheduling.
The failure modes here are mechanical and checkable: the URL is disallowed, the server errors, a redirect chain loops, authentication intervenes, or nothing links to the page at all. All of them are visible in a server response, which is why the first diagnostic step is always to fetch the exact URL yourself.
Indexing
The fetched page is parsed, rendered where necessary, analyzed for what it covers, and considered for storage. Duplicate detection runs here, as does canonical selection — which is where a page with a canonical pointing elsewhere drops out.
This is the stage people skip when reasoning, and it is where the confusing outcomes live. "Google can see my page but won't show it" is almost always an indexing decision rather than a crawling problem, and the usual cause is that the page is thin, near-duplicate, or canonicalized to something else.
Serving
For a given query the system selects from what it has stored and arranges a results page. This is the stage that changed most in the last few years, and the stage that changes most often.
What used to be ten links is now frequently a generated answer with supporting links, a map pack, ad units, and various other surfaces. The important structural point is that this is a serving change: the crawling and indexing requirements underneath did not move, which is why the technical pillar still matters exactly as much as it did.
The diagnostic habit
When something is not appearing, resist the temptation to guess at the cause, and walk the stages in order:
- Does anything link to it?
- What does the server return for that exact URL?
- Is the main content present in the rendered output?
- Is there a
noindex, in the markup or in a header? - Does a canonical point somewhere else?
- If all of that is clean, it is a content decision, and no technical fix will change it.
Most wasted effort in this field comes from starting at step six and working backward, or from starting at step one and stopping there.
More in Search fundamentals
Semantic understanding
Search systems model meaning rather than matching strings, which is why naturally phrased content now ranks for queries that contain none of its words — and why keyword density stopped being a lever a long time ago.
Penalties
The tactics search engines act against have barely moved in two decades. AI didn't add a new category. It made the oldest one cheap.
How AI answers are built
It reads the question, fans it out into several searches, retrieves passages, writes an answer, and credits some of it. Knowing the order tells you which stage you failed at, which is the only way to fix anything.