Sitemap and robots.txt checker
Verifies robots.txt and your XML sitemap exist, parse, and don't block indexing.
What TMOD checks
- Fetches /robots.txt and reports whether it exists and parses as a valid robots file.
- Discovers your XML sitemap, from the Sitemap directive in robots.txt and from the conventional /sitemap.xml location, and follows sitemap index files to their children.
- Counts the URLs the sitemap declares, which the page-count check then uses to judge your real site size rather than only the pages the crawler fetched.
- Flags a robots.txt whose rules would block the crawler from reaching your content at all.
Why it matters
robots.txt is the single most dangerous file on a website, because one wrong line silently removes the entire site from search results and nothing about the site looks broken. A `Disallow: /` left over from a staging environment is a genuinely common and genuinely catastrophic mistake.
The sitemap matters most for pages that are hard to reach by following links, new posts, deep archive pages, anything more than a few clicks from the homepage. It does not guarantee indexing, but it is how a crawler learns those URLs exist without walking your whole link graph.
There is also an important interaction people get wrong: robots.txt controls crawling, not indexing. A URL you disallow can still appear in search results, sourced from external links, because the crawler never fetched it and therefore never saw your noindex tag. To remove a page from the index you must let it be crawled and serve a noindex directive.
How to fix it
01Read your robots.txt as a stranger would
Fetch it and read every line. Anything you do not recognise, do not understand, or cannot explain the purpose of should be removed. A robots.txt with three lines you understand is better than thirty inherited from a tutorial.
02Reference the sitemap from robots.txt
Add a `Sitemap: https://yourdomain.com/sitemap.xml` line. It is the standard discovery mechanism and costs one line. Use the absolute URL, a relative path is not valid in this directive.
03Keep the sitemap honest
Every URL in it should return 200, be canonical, and be a page you actually want indexed. Listing redirects, 404s or noindexed pages wastes crawl budget and reduces trust in the file. If your CMS generates it, verify it after any structural change.
04Never use Disallow to hide a page from search
If you want a page out of the index, allow crawling and serve a noindex directive. Disallowing it prevents the crawler seeing the very instruction you need it to obey. This is the usual trap when people try to hide empty tag and author archives.
Rules that look harmless and are not
The dangerous lines in a robots.txt are rarely dramatic. Blocking an assets directory to keep the file tidy also blocks the CSS and JavaScript that render the page, so what a crawler evaluates is an unstyled document, and on a script-driven site an empty one. That single line can undo every responsive rule you wrote, because a layout cannot be assessed from stylesheets nobody is allowed to fetch.
Wildcard rules on query strings are the next most common. A pattern written to keep session parameters out of the crawl usually catches your filtered, sorted and paginated URLs too, and those are often the only route to a large part of a catalogue. Test any pattern against a handful of real URLs before shipping it rather than reasoning about it in the abstract.
Two more worth knowing. Crawl-delay is not supported by Google, so a site that added it to reduce load is still being crawled at the same rate while believing otherwise. And a Disallow line advertises what it names, which makes robots.txt a poor place to hide an admin path from anyone curious enough to read it.
Sitemap hygiene once a site grows
A sitemap tops out at 50,000 URLs and 50MB uncompressed, after which you split it and point a sitemap index at the parts. Splitting by content type rather than arbitrarily is worth the small effort, because the coverage report then tells you which type has a problem instead of which numbered file does.
The rule for what goes in is narrow: one entry per page, the canonical URL, returning 200, and a page you would be happy to see in results. That excludes redirects, the non-canonical variants of a page, noindexed archives and anything blocked in robots.txt. A sitemap contradicting your own canonical tags is a signal a crawler has to resolve, and it will not always resolve it your way.
Keep lastmod truthful. A file that stamps every URL with today's date, every day, teaches crawlers to ignore the field, and it is the one field that earns faster re-crawls when it is accurate. Regenerate after any structural change, and re-probe the links inside your content at the same time, since the change that moved those URLs moved them everywhere.
Then use the file as a diagnostic rather than a formality. Comparing the number of URLs you declare against the number actually indexed is one of the fastest reads available on whether a site has a crawl problem, a quality problem, or no problem at all.
Questions
Do I need a sitemap if my site is small?
If every page is reachable within two or three clicks of your homepage and your internal linking is sound, a crawler will find everything without one, and a sitemap adds little. It is still worth having: it is nearly free to generate, it gives you a machine-readable inventory of what you think is on your site, and comparing that list against what is actually indexed is a useful diagnostic.
Is a missing robots.txt a problem?
No. A site with no robots.txt is treated as fully crawlable, which is usually what you want. A 404 for that path is completely fine. The dangerous states are a robots.txt that exists and blocks too much, or one that returns a server error, some crawlers treat a 5xx on robots.txt as a signal to back off entirely.
How often should the sitemap be regenerated?
Whenever content is published, unpublished or moved. Most CMS plugins do this automatically. The lastmod dates should be real, a sitemap that reports every page as modified today, every day, trains crawlers to ignore the field entirely, which removes the main benefit of having it.