robots.txt and sitemap.xml solve different problems
| Mechanism | Purpose | Scope | Main effect |
|---|---|---|---|
| robots.txt | Control crawling | Crawler paths | Does not guarantee deindexing |
| sitemap.xml | Discover and signal important URLs | URL list and metadata | Hint, not an indexing guarantee |
| noindex | Exclude from the index | Specific URL or resource | Works after a crawler reads it |
| rel="canonical" | Declare a preferred duplicate version | Specific URL or resource | Strong signal, not an absolute directive |
robots.txt is part of the Robots Exclusion Protocol and tells crawlers which paths they may or may not fetch. It is not authentication or a privacy control. RFC 9309 explicitly states that robots rules are not access authorization. [4]
A sitemap provides search engines with information about pages, videos and other files that you consider important, including relationships and metadata. Google uses sitemaps to discover and crawl content more efficiently. [6]
The mechanisms are not interchangeable. robots.txt controls crawling permission, while sitemap.xml helps with discovery and prioritization of important URLs.
robots.txt location and scope
The file must be named `robots.txt` and placed at the root of the host, such as `https://example.com/robots.txt`. A robots file in a subdirectory does not control the whole site. [2][4]
Its scope is tied to protocol, host and port. Rules from `https://example.com/robots.txt` do not automatically apply to `http://example.com`, `https://m.example.com` or another port. [2]
The file should be UTF-8 plain text. Google enforces a 500 KiB size limit and ignores content beyond that limit. [3]
Basic robots.txt syntax
User-agent: *
Disallow: /admin/
Allow: /admin/public/
Sitemap: https://example.com/sitemap.xmlGoogle supports the fields `user-agent`, `allow`, `disallow` and `sitemap`. Fields such as `crawl-delay` are not supported by Google. [3]
`user-agent` selects the crawler. `disallow` blocks a path from crawling, while `allow` can open a more specific path inside a blocked area. [3]
The `sitemap` field takes a fully qualified sitemap or sitemap-index URL. Multiple `Sitemap` lines are allowed. [2][3]
Rule matching: specificity, case and wildcards
Google applies the most specific matching rule based on path length. If `allow` and `disallow` rules are equally specific, Google uses the less restrictive rule, which is `allow`. [3][4]
Path values in `allow` and `disallow` are case-sensitive. `/Folder/` and `/folder/` can therefore behave differently. [3]
Google supports `*` as a wildcard for zero or more characters and `$` as an end-of-URL marker. For example, `Disallow: /*.pdf$` can match URLs ending in `.pdf`. [3]
robots.txt is not an indexing control
A URL blocked in robots.txt may still appear in Google Search if Google discovers it from links elsewhere. Google may know the URL even though it cannot crawl the page content. [1][3]
To exclude a page from search results, use `noindex` in a robots meta tag or `X-Robots-Tag`. Google must be able to crawl the resource to see that rule. Blocking it in robots.txt at the same time can prevent Google from reading the `noindex`. [5][16]
Putting `noindex` directly in robots.txt is not supported by Google. [5]
robots.txt errors, HTTP status codes and caching
| Status | Google behavior |
|---|---|
| 2xx | Processes the file as robots.txt |
| 3xx | Follows redirects, at least 5 hops |
| 4xx except 429 | Treats it as no robots.txt, so no crawl restrictions |
| 5xx / network | May pause crawling and use the last known good file |
For `2xx`, Google processes the returned robots.txt. For redirects, Google follows at least five hops before it may treat the situation as if the file were missing. Most `4xx` responses except `429` are treated as no valid robots.txt, which means no crawl restrictions. [3]
For `5xx` or network failures, Google can initially stop crawling, then use the last known good robots.txt while retrying. Google documents separate behavior for the first 12 hours and the following 30 days. [3]
Google generally caches robots.txt for up to 24 hours, although the lifetime may change based on errors and `Cache-Control` headers. [3]
What sitemap.xml is for
A sitemap helps search engines discover important pages and files and can carry information about modification dates, localized versions, images, videos and news. [6][7]
Google notes that a small site of about 500 pages or fewer with comprehensive internal linking may not need a sitemap. Sitemaps are especially useful for large sites, new sites with few external links and sites with substantial media content. [6]
A sitemap does not replace internal linking and does not guarantee indexing.
Supported sitemap formats
Google supports XML, RSS, mRSS, Atom 1.0 and plain-text sitemaps as documented. XML is the most versatile because it supports extensions for images, video, news and localized versions. [7]
A text sitemap can list one absolute URL per line but cannot carry the richer metadata available in XML. RSS and Atom can be convenient when a CMS already publishes update feeds. [7]
Google states that it has no preference among the supported formats. The right choice depends on site architecture and the metadata you need to provide. [7]
Sitemap and sitemap-index limits
| File | Maximum entries | Maximum size | Note |
|---|---|---|---|
| Single sitemap | 50,000 | 50 MB | Limit is uncompressed; gzip is allowed |
| Sitemap index | 50,000 sitemaps | 50 MB | Limit is uncompressed; gzip is allowed |
A single sitemap is limited to 50,000 URLs or 50 MB uncompressed. If either limit is exceeded, split the sitemap into multiple files. [7][8]
A sitemap index can list up to 50,000 sitemaps and is also limited to 50 MB. More than one sitemap index can be used. [8]
Sitemaps can be gzip-compressed, but the size limit applies after decompression. [8]
Which URLs belong in a sitemap
Google recommends including the URLs you want to see in search results. Sitemap URLs should be fully qualified and absolute. [7]
If the same content is available at multiple URLs, the sitemap should usually contain the preferred canonical URL. Google treats sitemap inclusion as a weak canonicalization signal, weaker than redirects or `rel="canonical"`. [7][9]
A sitemap should not be a dump of every technically generated URL. Intentionally listing noindex URLs, 404s, redirects or duplicates sends a less coherent signal.
`<lastmod>`, `<priority>` and `<changefreq>`
Google ignores `<priority>` and `<changefreq>` values in sitemaps. There is no SEO benefit in building elaborate logic for those fields solely for Google. [7]
Google can use `<lastmod>` when it is consistently and verifiably accurate. The value should reflect the last significant change to page content, structured data or links. A copyright-year change alone is not considered significant. [7]
Updating `<lastmod>` on every deployment, regardless of page changes, weakens the usefulness of the signal.
How to submit a sitemap to Google
You can submit a sitemap through the Sitemaps report in Search Console, through the Search Console API, or reference it in robots.txt with `Sitemap: https://example.com/sitemap.xml`. [7][11][12]
Search Console is useful because the report exposes access and processing information and sitemap errors. [7][11]
Submitting a sitemap is only a hint. Google explicitly states that submission does not guarantee that the sitemap will be downloaded, used for crawling or that the listed URLs will be indexed. [7]
Multilingual sites and specialized sitemaps
For multilingual sites, Google allows `hreflang` relationships to be supplied in XML sitemaps. Each URL entry should reference all language or regional variants, including itself. [10]
Sitemaps can also be extended for images, videos and news. Google publishes separate requirements for each extension. [13][14][15]
For Google News, news metadata should cover recent articles. Google's current guidance says news sitemap metadata should be kept for articles from the last two days. [15]
Common technical SEO mistakes
One of the most serious mistakes is blocking a page in robots.txt while expecting Google to obey its `noindex`. If crawling is blocked, the indexing directive may never be seen. [5][16]
Another common issue is contradictory canonical signals, such as listing URL A in the sitemap while `rel="canonical"` points to URL B. Google recommends consistency between canonicalization methods. [9]
Sitemaps also become noisy when they contain unnecessary parameters, duplicates, redirects, error pages or faceted URLs that are not intended for search.
- Do not use robots.txt as a replacement for `noindex`.
- Do not block pages whose robots meta directives Google must read.
- Do not put every generated URL into the sitemap.
- Do not change `<lastmod>` without a meaningful page update.
- Do not rely on `<priority>` or `<changefreq>` for Google.
- Do not send conflicting sitemap and canonical signals.
- Remember that subdomains and different protocols may need their own robots.txt.
Deployment checklist for robots.txt and sitemap.xml
Before release, inspect real HTTP responses rather than only repository files. robots.txt should return valid plain text, while the sitemap should return valid XML or another supported format. [2][7]
After publishing, submit the sitemap in Search Console, inspect the Sitemaps report and use URL Inspection for representative pages. During site migrations, Google recommends sitemaps as one mechanism that can help discover the new URLs. [11][17]
For larger sites, automation is valuable: generate sitemaps from the source of truth and validate URL status, canonical consistency, noindex state and meaningful `<lastmod>` values as part of the release process.
- Is robots.txt available exactly at `/robots.txt`?
- Do its rules apply to the intended protocol, host and port?
- Is any important URL accidentally blocked?
- Are noindex pages still crawlable?
- Does the sitemap contain only absolute preferred URLs?
- Is each sitemap below 50,000 URLs and 50 MB?
- Does `<lastmod>` represent real meaningful updates?
- Do canonical tags and sitemaps point to the same preferred URLs?
- Are multilingual sitemap relationships correct?
- Does Search Console report any sitemap fetch or processing errors?
