Robots.txt Generator
Create a robots.txt file to control how search engines crawl your website.
Configuration
Generated robots.txt
Control Bot Access
A robots.txt file gives you control over which parts of your site should be public to search engines.
Recommended Tools
Hand-picked utilities you might find useful
Building a robots.txt File for Your Site
Overview
A robots.txt file tells search-engine crawlers which parts of your site they may or may not access. A correct file keeps private or duplicate areas out of the index and points crawlers to your sitemap. This generator builds the file from simple rules you choose, in your browser.
How to Use (Step by Step)
- 1
Choose what to allow or block
Add Disallow rules for paths you don't want crawled.
- 2
Add your sitemap URL
Include the full sitemap location so crawlers can find it.
- 3
Save as robots.txt
Place the file at the root of your domain (example.com/robots.txt).
How It Works
The file lists rules grouped by user-agent (the crawler), each with Allow and Disallow paths that permit or block crawling of URL patterns. It can also declare your sitemap location. Crawlers read robots.txt before crawling, so blocking a path keeps well-behaved bots out — though it is a guideline for crawling, not a security control or a guaranteed way to hide a page from search.
When to Use This
Blocking crawlers from admin, cart, or staging paths. Pointing search engines to your sitemap. Allowing full crawl access for a new site with a clean default file.
Frequently Asked Questions
Not reliably. It requests that crawlers skip a path, but a blocked URL can still appear in results if linked elsewhere. Use a noindex tag or authentication to truly keep a page out.