Robots.txt Generator
Build a robots.txt with blocked paths, a sitemap line and optional AI crawler blocks.
Robots.txt and Sitemap Validator
Enter your site address and see what search engines will find. Our server fetches the two files once.
Your files stay with you. Unlike our other tools, where your files never leave your browser, this one sends the web address you type to our server so it can fetch the page for you. Nothing is stored or logged, and only the result is shown to you.
How to use it
- Type your site address, or the address of a sitemap file.
- Press the check button.
- Read the problems found in robots.txt, then in the first sitemap it lists.
Questions people ask
What does it check in robots.txt?
That the file exists and is not a web page in disguise, every line is a known rule, rules sit under a User-agent line, paths start with a slash, Sitemap lines are full URLs, and whether all crawlers are blocked from the whole site.
What does it check in the sitemap?
That it is valid XML with a urlset or sitemapindex, entries have full URLs, there are at most 50,000 of them, no duplicates, dates in the right format, and URLs on the same host as the site.
Does it check every URL in the sitemap?
No. It reads the sitemap file and reports on its contents but does not visit the listed pages, to avoid flooding your site. Use the HTTP status checker for individual pages.
Why is a compressed or very large sitemap only partly checked?
To keep the tool fast and safe it reads only the first 1 MB and plain XML. Compressed .gz sitemaps are valid for search engines but are not read here.
Want this inside your own account?
Marketiism keeps your invoice numbers, GST reports, UPI payment links and WhatsApp reminders in one place. Leave your details and we'll call to set it up for your business, or start the trial yourself.
More SEO tools
XML Sitemap Generator
Turn a list of page URLs into a valid sitemap.xml with optional dates, frequency and priority.
HTTP Status Checker
Check the status code of a URL and follow its redirect chain hop by hop.
Meta Tag Generator
Write the title, description, robots, canonical and viewport tags for a page, ready to paste.
Open Graph Generator
Create Open Graph and X (Twitter) card tags and preview how your link looks when shared.
SERP Snippet Preview
See how your title and description may look in search results, with character and pixel estimates.