Robots.txt / sitemap checker
Content stays local in your browser. Do not paste sensitive tokens or private data.
How to use this tool
Send direct browser GET requests to robots.txt or sitemap URLs, show HTTP status, and count literal Sitemap: lines or <url> tags.
Audit your robots.txt and sitemap.xml directives to avoid accidental search indexing blocks. Validate syntax rules for User-agent, Disallow, and Sitemap declarations according to Google guidelines.
💡 3 Quick Steps to Get Started
Paste your robots.txt lines or sitemap.xml structure into the validator.
The tool validates crawler User-agent headers, wildcard patterns, and path rules.
Identify unintentional indexing blocks and verify sitemap declarations.
When it helps
A site URL checks same-origin /robots.txt and /sitemap.xml. A robots.txt or .xml URL checks only that exact address.
How it works
The tool uses browser fetch without the GoGo Tools backend. XML counting matches only case-sensitive literal <url> strings, while robots.txt counting matches case-insensitive Sitemap: at the start of a line. It does not parse XML, sitemap indexes, compressed files, or robots directives.
Worked example
Example: entering https://example.com attempts https://example.com/robots.txt and https://example.com/sitemap.xml in sequence.
Limits and data handling
If a cross-origin server does not allow CORS, the browser cannot read the body and the tool shows error text without creating a manual link. There is no custom timeout or cache, and no parsing of sitemap indexes, namespaces, compressed sitemaps, or robots rules.
Your browser sends GET requests directly to the target site, so that site receives a network request from you. The URL is not sent to the GoGo Tools backend, and analytics events do not contain the URL or response body.
Robots.txt Protocols, Sitemap XML Standards & Crawl Budget Guide
1. The Robots Exclusion Protocol (IETF RFC 9309)
The `robots.txt` file serves as the first line of crawler governance. Codified in RFC 9309, directives dictate access permissions (`User-agent`, `Allow`, `Disallow`, `Sitemap`). A critical distinction: Disallow blocks page content extraction but does not prevent indexation if external backlink signals exist; authoritative de-indexing requires `noindex` headers.
2. Sitemaps.org XML Protocol and Hreflang Localization
XML sitemaps provide structured manifests of indexable canonical URLs. Multi-regional architectures embed `xhtml:link` hreflang annotations alongside ISO 8601 `<lastmod>` timestamps, enabling Googlebot to route regional traffic seamlessly.
3. Crawl Budget Management and Enterprise SEO
Optimizing crawl budget involves pruning faceted navigation parameters and staging endpoints from crawler queues, focusing indexing bandwidth entirely on high-value canonical pages.
Feedback
How can we improve this tool?
Send a suggestion or bug report for this tool.
FAQ
Robots.txt / sitemap checker: How are target URLs selected?
A general site URL ignores its path and checks same-origin /robots.txt and /sitemap.xml. If the pathname ends in robots.txt or .xml, only the full entered URL is checked.
Robots.txt / sitemap checker: What does the result show?
Each readable response shows its target URL, HTTP status, and a count of Sitemap: lines or literal <url> tags. This text count does not prove that directives, XML, or listed URLs are valid.
Robots.txt / sitemap checker: What CORS and format limits apply?
If a cross-origin server does not allow CORS, the browser cannot read the body and the tool shows error text without creating a manual link. There is no custom timeout or cache, and no parsing of sitemap indexes, namespaces, compressed sitemaps, or robots rules.
Robots.txt / sitemap checker: Where is the entered URL sent?
Your browser sends GET requests directly to the target site, so that site receives a network request from you. The URL is not sent to the GoGo Tools backend, and analytics events do not contain the URL or response body.
