Skip to content
Web4tools

Robots.txt Generator

Control which parts of your site search engines crawl.

Runs in your browserNo accountNo watermark

In short

To create a robots.txt file, choose whether to allow crawling, list paths to block (like /admin/ or /cart/), add your sitemap URL and download the file. Upload it to your site's root so it's available at yourdomain.com/robots.txt. You can also block AI training crawlers.

How to use the Robots.txt Generator

  1. 1Choose the default rule and list paths to block.
  2. 2Add your sitemap URL(s) and any options.
  3. 3Download robots.txt and upload it to your website's root folder.

What robots.txt can and can't do

robots.txt tells well-behaved crawlers which URLs not to fetch. It doesn't hide pages: blocked URLs can still appear in search results if other sites link to them. To keep a page out of Google, allow crawling and add a noindex meta tag instead.

Blocking AI crawlers such as GPTBot stops future training use, but it can also keep your content out of AI search answers. Decide based on your goals.

Frequently asked questions

Where do I put robots.txt?

In the root of your domain, e.g. https://example.com/robots.txt. Each subdomain needs its own file.

Should I block CSS and JS files?

No. Google needs them to render your pages correctly.

Is my data sent anywhere?

No. Everything runs in your browser. What you paste is never uploaded, logged or stored, so it's safe for code, tokens and private content.

Last updated