How to Create a robots.txt File the Right Way

🤖Free Robots.txt Generator
Create a robots.txt file – including AI crawler rules.Open tool →

What it does

It asks crawlers not to fetch certain URLs. Well-behaved bots – Google, Bing, most AI crawlers – obey. It is not security, and it does not remove pages that are already indexed.

To keep a page out of search results, use a noindex meta tag and let Google crawl the page so it can see it.

Rules worth having

Most small sites need very little: allow everything, block admin and internal search or cart pages, and point to the sitemap. WordPress sites should block /wp-admin/ but allow admin-ajax.php.

Never block your CSS or JavaScript folders – Google needs them to render the page.

Blocking AI crawlers

Add a group for each bot – GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot – with Disallow: /. The Robots.txt Generator adds them with one tick. This does not affect normal Google rankings.

Upload and test

Save the file as robots.txt and upload it to the root folder so it opens at yoursite.com/robots.txt. Then run the Robots.txt & Sitemap Checker to confirm it is reachable and not blocking the whole site.

Frequently asked questions

Do I need a robots.txt at all?

It's optional, but adding your sitemap line is worthwhile.

What does 'Disallow: /' mean?

Block everything – only use it on staging sites.

How fast does Google notice changes?

Usually within a day.