Spoke
Crawlability and indexing
Crawling and indexing are two different things, and a page can fail at either one. Most diagnosis goes wrong by treating them as a single step.
The crawl-render-index pipeline and the directive behavior here are documented by Google. The order we suggest diagnosing in is ours.
Three separate things have to happen before a page can be served. It has to be found. It has to be fetched and rendered. Then it has to be picked for the index.
A page can clear the first two and fail the third. That's why "Google can see my page but won't show it" is such a common and such a baffling situation to be in.
Why is my page crawled but not indexed?
Because they're two separate decisions. A crawler fetching your page isn't a promise to store or serve it. The system still decides whether the page is worth including, and it says no to pages that are thin, near-duplicates of something it already has, or canonicalized to somewhere else.
The usual causes, most common first: the content is too similar to another page, the page is thin next to what already exists for that search, or a canonical or parameter setting is pointing somewhere else.
Diagnose it in this order
Working the stages in sequence saves an enormous amount of wasted effort.
Can anything find it? Does a single page link to it? A page that's only reachable from your sitemap has a structural problem the sitemap is hiding.
Can it be fetched? Check the actual server response for the exact URL. Not staging, not a similar URL. Watch for redirect chains, soft 404s, and anything that needs a session.
Can it be rendered? If the main content shows up via client-side JavaScript, confirm it's actually in the rendered output. Don't assume.
Is it allowed to be indexed? noindex in a meta tag, or in an X-Robots-Tag header. The header is the one people miss, because you can't see it in the page source.
Is it canonicalized somewhere else? A canonical pointing at another URL is you explicitly asking not to have this one indexed. On templated pages, self-referencing canonicals often still point at whatever the template was built from.
Is it worth indexing? If everything above is clean, this is your answer. And it's a content problem, not a technical one.
Crawl budget, and when to ignore it
Crawl budget is real and almost always irrelevant to you.
On a site with a few hundred pages it isn't the constraint, and treating it as one leads to a lot of pointless work.
It starts to matter on large sites, and the fix is cutting waste rather than asking for more crawling: faceted navigation spinning out effectively infinite URLs, session IDs in paths, parameters serving identical content, long redirect chains, and heaps of low-value pages.
The structure that makes this easy
Shallow and densely linked beats deep and sparse, every time. Every page should be a few clicks from the home page, and should link onward rather than sitting there as a dead end.
That's the same argument internal linking and clusters makes from the other direction. Your internal links are simultaneously how things get discovered and your statement about what relates to what.
More in Technical SEO
Architecture & URLs
How your pages are organized decides how they get found, how authority moves between them, and whether anyone can tell what the site is about. URLs are the visible shape of that decision.
Sitemaps & robots
These two files are where confident wrong advice concentrates, probably because they look simple enough that nobody bothers checking the documentation.
Structured data
Markup won't rank your page and won't make a claim true. Its job is to state outright what a machine would otherwise have to work out by guessing.
Core Web Vitals
Three metrics with published thresholds, measured on real visits rather than a test you ran once. No interpretation needed. You either meet them or you don't.
AI crawler access
Letting AI crawlers read your site is a real decision with a real trade-off. Most advice about it mixes up controls that do completely different jobs.
JavaScript & rendering
Google can render JavaScript" is true, and it gets used to justify architectures that cause real problems. The useful questions are when rendering happens, how reliably, and who else is looking.