Spoke
Sitemaps and robots.txt
These two files are where confident wrong advice concentrates, probably because they look simple enough that nobody bothers checking the documentation.
Every limit and behavior here comes from Google's own documentation, recorded in our ledger with retrieval dates.
These two files are where confident wrong advice piles up. Probably because they look simple enough that nobody checks.
Does robots.txt stop a page being indexed?
No. It controls whether a crawler may fetch a URL. A disallowed URL can still show up in an index — discovered through links elsewhere, and listed with no useful description because the crawler was never allowed to read it.
To keep a page out, you have to let it be crawled and serve a noindex on it. Blocking the crawler and serving noindex together defeats itself: the directive is sitting on the page you told the crawler not to read.
The documented numbers
A single sitemap file is limited to 50,000 URLs. Larger sets must be split across multiple files, optionally referenced by a sitemap index.
A single sitemap file is limited to 50MB uncompressed, in every supported format.
Google enforces a robots.txt size limit of 500 kibibytes. Content past that limit is ignored.
Google generally caches robots.txt for up to 24 hours, and may cache it longer when it cannot refresh the file.
This is why a robots.txt change is not an instant control. It also means robots.txt is the wrong instrument for anything urgent.
That cache duration is the most practically important of the four. robots.txt isn't a switch. A change takes effect when the cached copy expires, and longer if the file couldn't be refreshed. Anything you need to happen right now needs a different tool.
What a sitemap is for
Helping a crawler find URLs it might otherwise reach slowly or not at all. That's the whole job.
It doesn't request indexing. It doesn't confer importance. It doesn't make up for a page nothing links to — if a page is only reachable through your sitemap, your real problem is the internal links.
Where sitemaps genuinely earn their place: very large sites, pages with few internal links, recently published content, and telling crawlers when something changed.
Getting lastmod right
Google uses lastmod only when it's consistently and verifiably accurate — actually matching when the page last changed. It should reflect a real update: main content, structured data, links. Updating a copyright year doesn't count.
The usual failure is automated. A build system writing today's date for every page on every deploy produces a sitemap claiming your entire site changed substantively this morning. Every morning.
That isn't a stronger signal. It's a destroyed one.
Our sitemap takes lastmod from each page's modified date — the day a human actually re-read it — and rebuilding doesn't touch it.
The directives worth knowing
Disallow— don't crawl. Not "don't index."Allow— carve an exception out of a wider disallow.Sitemap— where the sitemap is. Independent of any user-agent group.User-agent— scopes a group of rules. Most specific match wins, which is why a wildcard block plus a specific allow behaves differently from what people expect.Crawl-delay— not supported by everything. Some crawlers honor it. Google doesn't use it.
One line
Use robots.txt to manage crawler load and keep crawlers out of pointless corners. Use noindex to manage what shows up in an index. Use the sitemap to help discovery.
Confusing the three is the most expensive technical mistake there is, because every version of it fails silently.
More in Technical SEO
Crawlability & indexing
Crawling and indexing are two different things, and a page can fail at either one. Most diagnosis goes wrong by treating them as a single step.
Architecture & URLs
How your pages are organized decides how they get found, how authority moves between them, and whether anyone can tell what the site is about. URLs are the visible shape of that decision.
Structured data
Markup won't rank your page and won't make a claim true. Its job is to state outright what a machine would otherwise have to work out by guessing.
Core Web Vitals
Three metrics with published thresholds, measured on real visits rather than a test you ran once. No interpretation needed. You either meet them or you don't.
AI crawler access
Letting AI crawlers read your site is a real decision with a real trade-off. Most advice about it mixes up controls that do completely different jobs.
JavaScript & rendering
Google can render JavaScript" is true, and it gets used to justify architectures that cause real problems. The useful questions are when rendering happens, how reliably, and who else is looking.