Robots.txt Generator

Überblick

A robots.txt file tells search-engine crawlers which parts of your site they may or may not access. A correct file keeps private or duplicate areas out of the index and points crawlers to your sitemap. This generator builds the file from simple rules you choose, in your browser.

So gehts (Schritt für Schritt)

1. 1

   Choose what to allow or block

   Add Disallow rules for paths you don't want crawled.

2. 2

   Add your sitemap URL

   Include the full sitemap location so crawlers can find it.

3. 3

   Save as robots.txt

   Place the file at the root of your domain (example.com/robots.txt).

Funktionsweise

The file lists rules grouped by user-agent (the crawler), each with Allow and Disallow paths that permit or block crawling of URL patterns. It can also declare your sitemap location. Crawlers read robots.txt before crawling, so blocking a path keeps well-behaved bots out — though it is a guideline for crawling, not a security control or a guaranteed way to hide a page from search.

Wann verwenden

Blocking crawlers from admin, cart, or staging paths. Pointing search engines to your sitemap. Allowing full crawl access for a new site with a clean default file.

Häufig Gestellte Fragen