Create a clean robots.txt file with your sitemap, blocked folders and optional AI crawler rules.
🔒 Generated in your browser.
How to use the robots.txt generator
- Choose what search engines may crawl.
- List folders to block and add your sitemap URL.
- Download robots.txt and upload it to your site's root folder.
What robots.txt is for
It tells crawlers which parts of your site not to fetch – carts, internal search results, admin pages. It is not a security feature and does not remove pages from Google; for that use a noindex tag.
Blocking AI crawlers
GPTBot, ClaudeBot, CCBot and Google-Extended collect content for AI training. Blocking them does not affect your Google rankings.
Frequently asked questions
Where does robots.txt go?
At the root of your domain: yoursite.com/robots.txt.
Can robots.txt hurt my SEO?
Yes – a single 'Disallow: /' hides your whole site. Always test after uploading.
Does WordPress have one already?
WordPress serves a virtual robots.txt; a real file in the root folder replaces it.