Robots.txt Generator
Create robots.txt files to control search engine crawlers.
Preview
User Agent Block
Recommended Tools
Hand-picked utilities you might find useful
Creating a robots.txt File to Guide Search Crawlers
Overview
A robots.txt file tells search engine crawlers which parts of your site they may or may not access. It lives at the root of your domain and is the first thing well-behaved bots check before crawling. This generator builds a valid robots.txt from simple choices — which paths to allow or disallow, and where your sitemap lives — so you don't have to memorize the syntax.
How to Use (Step by Step)
- 1
Choose your rules
Pick which user-agents the rules apply to and which paths to Allow or Disallow.
- 2
Add your sitemap
Optionally include your sitemap URL so crawlers can find all your pages.
- 3
Save as robots.txt
Copy the generated text into a file named robots.txt and upload it to your domain's root.
How It Works
You specify user-agents (which crawlers a rule applies to, often * for all), then Allow and Disallow rules for URL paths, and optionally your sitemap URL. The tool assembles these into the exact text format search engines expect. You copy the output into a file named robots.txt and upload it to your site's root. It's a set of directives, not a security barrier — it guides cooperative crawlers.
When to Use This
Keeping crawlers out of admin, staging, or duplicate pages. Pointing search engines to your sitemap for faster discovery. Blocking a specific bot that's hammering your server. Allowing everything while still declaring a sitemap. Preventing thin or private sections from being crawled. Setting up robots rules for a new site launch.
Frequently Asked Questions
No. It only requests that cooperative crawlers skip a path; the page is still publicly accessible and non-compliant bots can ignore it. To truly restrict access, use authentication or server-side controls.
Important Notes
A single wrong Disallow line can hide your whole site from search. Double-check the paths, and remember robots.txt is public — anyone can read it, so don't list secret URLs you're trying to hide.