Analyze a site robots.txt and its sitemap tree. Parse crawler rules, follow sitemap indexes, validate the documents it reads and check the URLs they submit.
Enter a domain or any page on it. robots.txt is always read from the root of that host, so a path in the address changes nothing.
Analyze a specific sitemap as well as the ones found automatically. Useful for a sitemap that robots.txt does not declare.
Rule matching is per crawler. This choice drives the URL tester and the crawlability column.
This analysis runs on POLPROG servers, because reading another site robots.txt and sitemaps cannot be done from your browser. Only publicly available files are requested, nothing is stored after the response is sent, and no report is saved or shared.
A plain-text file at the root of a host, at /robots.txt, that tells crawlers which URL paths they may request. It is a crawl instruction, not an access control: the file is public, and a crawler that ignores it is not breaking anything technical. Only four directives are standardised in RFC 9309 - User-agent, Allow, Disallow and Sitemap.
Disallow stops a page being fetched. It does not stop the URL appearing in search results: a blocked page that other sites link to can still be listed, without a description, because the crawler was never allowed to read the page to find your noindex tag. To keep a page out of the index, leave it crawlable and serve a noindex robots meta tag or an X-Robots-Tag header.
A crawler uses one group: the one whose User-agent token is the longest match for its own name, falling back to *. Within that group the most specific rule wins, measured by the length of the path pattern, not by the order the lines appear in. When an Allow and a Disallow of identical length both match, Allow wins. * matches any run of characters and a trailing $ anchors the end of the URL.
A list of the URLs you want crawled, with optional lastmod, changefreq and priority for each. It helps discovery on large sites, on sites with pages that are not well linked, and on new sites. It does not guarantee indexing, and Google ignores changefreq and priority entirely.
A sitemap of sitemaps. One file can hold at most 50,000 URLs and 50 MB uncompressed, so a large site splits its URLs across several sitemaps and lists those files in a sitemapindex document. That index is the single URL you declare in robots.txt and submit in Search Console.
A sitemap: 50,000 URLs and 50 MB uncompressed, per file. Gzip is allowed and the 50 MB applies to the decompressed size. robots.txt: Google reads the first 500 KiB and discards the rest. A sitemap may only list URLs on the host that serves it, unless cross-submission is configured.
robots.txt is only valid at the root of a host and port: https://example.com/robots.txt governs https://example.com, and nothing else. A file at /blog/robots.txt is ignored, and https://example.com/robots.txt says nothing about http://example.com or about www.example.com. A sitemap can live anywhere on its host, and is found through robots.txt or by submission.
A URL that appears in the sitemap and is disallowed by robots.txt. The sitemap asks for it to be crawled and robots.txt refuses, so it is neither crawled nor reliably indexed, and Search Console reports it as an error. This tool checks every sitemap URL against the rules in the same robots.txt and counts them.
Everything reported here is read from the site own robots.txt and sitemap files during the scan: HTTP status, file size, parsed rules, XML structure and the URLs each sitemap lists. Crawlability is computed by matching those URLs against the same robots.txt, using RFC 9309 rules. The scan reads at most 40 sitemap documents, five levels of index nesting and 50,000 URLs, and it checks HTTP status for a bounded number of URLs rather than all of them; wherever a number covers only part of the site, the report says so beside it. It cannot tell you whether a page is indexed: nothing in robots.txt, a sitemap or an HTTP response carries that, and this tool has no Search Console connection.
No. Disallow stops a crawler fetching the page; it does not remove the URL from the index. A blocked page that other sites link to can still be listed, usually with no description, because the crawler was never allowed to read the page. To keep a page out of the index, leave it crawlable and serve a noindex robots meta tag or an X-Robots-Tag header. A Noindex line inside robots.txt does nothing: Google has never supported it and stopped honouring it in 2019.
At the root of the host and port it governs, and nowhere else. https://example.com/robots.txt applies to https://example.com and to nothing more: not to http://example.com, not to www.example.com, not to a subdomain. A file at /blog/robots.txt is ignored. This tool always reads the root of whatever host you enter, so a path in the address changes nothing.
Up to 50,000 URLs and 50 MB uncompressed per file. Gzip is allowed and the size limit applies to the decompressed file. Past either limit, split the URLs across several sitemaps and list those files from a sitemap index; an index has the same 50,000 and 50 MB limits over the sitemaps it references.
The most specific one, measured by the length of the path pattern, not the order the lines appear in the file. When an Allow and a Disallow of identical length both match, Allow wins. Before that, the crawler picks one group: the one whose User-agent token is the longest match for its own name, falling back to the * group. The URL tester in this report shows which rule decided each answer and on which line it sits.
No. The analysis runs on POLPROG servers, because reading another site robots.txt and sitemaps cannot be done from your browser, and only publicly available files are requested. Nothing is written to a database, no report is saved, and no shareable copy is created. The link in your address bar re-runs the analysis rather than loading a stored one.
Because checking a URL means sending a real request to the analyzed site, and a large sitemap holds tens of thousands of them. The sitemap structure is always parsed in full, so the counts, duplicates, lastmod and crawlability figures cover every URL the scan read. HTTP status is requested for a bounded number: about 40 in Quick analysis and up to 500 in Full URL analysis. The report always states how many were actually requested, and never presents a sample as if it covered the whole site.
Performance is not only a technical topic. Slow pages cost leads, harm SEO, and erode trust, and the things that fix them are usually small, specific, and boringly consistent.
SSR vs CSR vs SSG vs ISR: what to choose for SEO and performance
SSR, CSR, SSG and ISR compared
SSR, CSR, SSG and ISR differ mainly in where and when HTML is generated. For SEO, the key question is whether important content and metadata are available without waiting for client-side JavaScript. For performance, server cost, caching, JavaScript execution, hydration and freshness matter as well. In 2026, the best architecture is usually not one rendering mode for an entire application, but a strategy selected for each route or even each part of a route.
robots.txt and sitemap.xml: a complete technical SEO guide
robots.txt, sitemap.xml, noindex and rel=canonical compared
robots.txt and sitemap.xml solve two different technical SEO problems. The first controls which URLs crawlers may fetch, while the second helps search engines discover important URLs and their metadata. Problems begin when robots.txt is used to block indexing or when a sitemap contains redirects, duplicates and noindex URLs. A solid setup combines crawl control, consistent canonical URLs, accurate sitemaps and Search Console monitoring.
We build custom software and solutions tailored to your needs.