Free tool
Robots.txt Checker
Run the robots.txt checker
Free, no sign-up. You get the full LaunchSafer report, including the checks below, in 30–90 seconds.
The robots.txt mistakes that cost traffic
The most damaging one is a leftover `Disallow: /` from a staging site, which tells every crawler to stay away. Others are blocking CSS or JavaScript that Google needs to render the page, and forgetting to list the sitemap.
LaunchSafer reports whether robots.txt exists, how many rules apply to all crawlers, and which sitemaps it references. It also respects those rules while crawling, so any important page it couldn't scan is listed as blocked.
robots.txt is not the same as noindex
robots.txt controls crawling, not indexing. To keep a page out of search results it must be crawlable and carry a noindex directive. LaunchSafer's indexability check looks for noindex in meta robots tags and X-Robots-Tag headers on every scanned page.
Frequently asked questions
What happens if I don't have a robots.txt?
Crawlers assume everything is allowed. That's fine for many sites, but a robots.txt is a convenient place to list your sitemap.
Can robots.txt remove a page from Google?
No. Blocking a page in robots.txt stops crawling, but the URL can still appear in results. Use a noindex directive instead, and leave the page crawlable.
Does LaunchSafer obey my robots.txt?
Yes. The LaunchSaferBot crawler follows your robots.txt rules and lists any pages it skipped because of them.