Pillar
Search fundamentals
What happens between somebody asking a question and getting an answer, what changed when answers started replacing lists, and the tricks that have been penalized for twenty years and still get sold to people.
The crawl, index and serve pipeline is documented by Google. Which tactics get penalized comes from its published spam policies.
Nearly every bad decision in this field traces back to a wrong idea of what the machine is doing.
Think a search engine matches keywords, and you'll stuff keywords. Think links are votes, and you'll go buy votes. The model you carry around determines what you do, so it's worth getting right.
Here it is, kept deliberately simple, with the parts that changed marked as changed.
How does a search engine work?
It crawls the web to find and fetch pages. It indexes what it finds, analyzing and storing the content. Then it serves results by picking from that index for a given search. AI answers add a layer on top: for some questions the system pulls from that same index and writes an answer instead of, or alongside, the list of links.
The thing to take from that is that the stages are independent. A page can be crawled but not indexed, or indexed but never served, and the fix is completely different in each case.
What changed and what didn't
Changed: the serving layer. More questions get an answer now instead of ten links, and being the source that gets quoted is a kind of visibility that produces no click at all.
Changed: how queries are understood. Exact-match keywords stopped being necessary once these systems could model meaning. That's why writing the way people actually talk now matches searches it would have missed ten years ago.
Didn't change: crawling and indexing. Same pipeline, same requirements, same ways it breaks. Fetching your page for an AI answer runs straight through it.
Didn't change: what gets penalized. The list below is essentially the list from twenty years ago.
The patterns that get you penalized
These aren't history. Every one is still sold as a service, today, probably to somebody you know.
| Do | Don't |
|---|---|
| Write one strong page per topic | Spin one article into fifty near-duplicates |
| Use the words people actually use | Repeat a keyword until the sentence stops making sense |
| Build real pages for each place you serve | Generate a page per city with the name swapped |
| Earn links by being worth pointing at | Buy links, or trade them in a network |
| Let AI draft, then have someone who knows the subject check it | Publish generated text nobody competent has read |
| Show people and crawlers the same thing | Serve crawlers something different |
| Ask every client for a review | Buy them, write them, or screen out the unhappy ones |
The math on the right-hand column is worse than it looks. A penalty isn't a cost you pay once. It's a liability sitting against everything you've built, and buying services goes through what that actually means when the bill comes.
Why this comes first
Everything else on this site assumes this model.
The answer-engine work only makes sense once you know something is pulling passages out of pages. The entity work only makes sense once you know these systems are trying to figure out who you are, not match your name as text. And the section on hiring vendors only makes sense once you know what gets penalized and why you can't buy your way back out of it.
How search works
Three stages, three distinct failure modes, and one diagnostic habit that saves most of the time people waste guessing. Everything else on this site assumes this model.
Semantic understanding
Search systems model meaning rather than matching strings, which is why naturally phrased content now ranks for queries that contain none of its words — and why keyword density stopped being a lever a long time ago.
Penalties
The tactics search engines act against have barely moved in two decades. AI didn't add a new category. It made the oldest one cheap.
How AI answers are built
It reads the question, fans it out into several searches, retrieves passages, writes an answer, and credits some of it. Knowing the order tells you which stage you failed at, which is the only way to fix anything.