Free robots.txt Generator

Build robots.txt rules with a visual editor. Add multiple user agents. Copy or download.

Generated robots.txt

User-agent: *
Allow: /

Related Tools

{}

JSON Formatter

🔤

Base64 Encode/Decode

.*

Regex Tester

📝

Markdown Editor

Get notified of new tools

New tools every week. No spam, unsubscribe anytime.

How to Generate robots.txt

Add rules for different user agents (search engine crawlers). Specify which paths to allow or disallow. Add your sitemap URL for better indexing. Copy the output or download it as a robots.txt file to place in your site root.

Why Use robots.txt?

robots.txt tells search engine crawlers which pages to index and which to skip. Proper configuration prevents duplicate content issues, keeps admin pages out of search results, and helps crawlers find your sitemap efficiently.

Frequently Asked Questions

Where does robots.txt go? Place it in your site's root directory: https://yourdomain.com/robots.txt

What does User-agent: * mean? The asterisk (*) means the rules apply to all crawlers. You can specify individual bots like Googlebot, Bingbot, etc.