Skip to main content
DevToolbox

robots.txt & Sitemap

Nothing leaves your browser

Generate crawl rules and a sitemap, or paste yours to find mistakes. Test whether a crawler may fetch a URL the way Google reads the file. Nothing is fetched from your site.

Nothing is fetched from your site: paste the file or build one here.

Build a robots.txt

Google ignores it; some other crawlers obey it.

One per line, e.g. /private/

One per line, e.g. /private/public/

Full addresses, one per line.

Block these AI crawlers

Tab indents. Press Escape, then Tab, to leave the editor.

Can a crawler fetch this address?

https://example.com/page or /page?x=1

Enter an address to test it against the file above.

Related tools