Robots.txt生成器
为搜索引擎生成robots.txt文件。
配置
生成的robots.txt
Control Bot Access
robots.txt文件控制搜索引擎可以爬取的页面范围。
推荐工具
精心挑选的实用工具
Building a robots.txt File for Your Site
概述
A robots.txt file tells search-engine crawlers which parts of your site they may or may not access. A correct file keeps private or duplicate areas out of the index and points crawlers to your sitemap. This generator builds the file from simple rules you choose, in your browser.
使用步骤
- 1
Choose what to allow or block
Add Disallow rules for paths you don't want crawled.
- 2
Add your sitemap URL
Include the full sitemap location so crawlers can find it.
- 3
Save as robots.txt
Place the file at the root of your domain (example.com/robots.txt).
工作原理
The file lists rules grouped by user-agent (the crawler), each with Allow and Disallow paths that permit or block crawling of URL patterns. It can also declare your sitemap location. Crawlers read robots.txt before crawling, so blocking a path keeps well-behaved bots out — though it is a guideline for crawling, not a security control or a guaranteed way to hide a page from search.
什么时候用
Blocking crawlers from admin, cart, or staging paths. Pointing search engines to your sitemap. Allowing full crawl access for a new site with a clean default file.
常见问题
Not reliably. It requests that crawlers skip a path, but a blocked URL can still appear in results if linked elsewhere. Use a noindex tag or authentication to truly keep a page out.