Robots.txt生成器
為搜尋引擎生成robots.txt檔案。
配置
生成的robots.txt
Control Bot Access
robots.txt檔案控制搜尋引擎可以爬取的頁面範圍。
推薦工具
精心挑選的實用工具
Building a robots.txt File for Your Site
Overview
A robots.txt file tells search-engine crawlers which parts of your site they may or may not access. A correct file keeps private or duplicate areas out of the index and points crawlers to your sitemap. This generator builds the file from simple rules you choose, in your browser.
How to Use (Step by Step)
- 1
Choose what to allow or block
Add Disallow rules for paths you don't want crawled.
- 2
Add your sitemap URL
Include the full sitemap location so crawlers can find it.
- 3
Save as robots.txt
Place the file at the root of your domain (example.com/robots.txt).
How It Works
The file lists rules grouped by user-agent (the crawler), each with Allow and Disallow paths that permit or block crawling of URL patterns. It can also declare your sitemap location. Crawlers read robots.txt before crawling, so blocking a path keeps well-behaved bots out — though it is a guideline for crawling, not a security control or a guaranteed way to hide a page from search.
When to Use This
Blocking crawlers from admin, cart, or staging paths. Pointing search engines to your sitemap. Allowing full crawl access for a new site with a clean default file.
Frequently Asked Questions
Not reliably. It requests that crawlers skip a path, but a blocked URL can still appear in results if linked elsewhere. Use a noindex tag or authentication to truly keep a page out.