robots.txt Generator
Build your robots.txt with User-agent, Disallow, Allow, Sitemap and Crawl-delay live in the browser.
💡 What's this for?
With this tool you put together your robots.txt file by choosing which areas of your site search engines may crawl and which they should leave alone. You add your rules and sitemap and get the finished file to upload, so crawlers like Google know exactly where they're allowed to go.
Fill in your values - the robots.txt is generated instantly. Add Disallow and Allow paths one per line.
Result: robots.txt
🐤 Want Canary to keep an eye on this for you?
This tool checks once. Canary checks your whole website every day and alerts you instantly when something breaks technically and costs you Google traffic. Start a free health check →