DA
CompareGet started

Free tool

robots.txt generator: decide who may crawl your website

robots.txt is the file that tells search engines and AI crawlers which parts of your website they may visit. Choose what to allow and block, add your sitemap, and copy a finished robots.txt. No syntax to learn.

Block search engines

Blocks Google Search too.

Block AI crawlers

Gemini training only, not Google Search.

The full address, like https://www.example.dk/sitemap.xml. Leave it empty if you don’t have one.

Bing respects it; Google ignores it.

robots.txt
User-agent: *
Disallow:

Sitemap: https://www.example.dk/sitemap.xml

Save it as robots.txt at the root of your domain, e.g. https://www.example.dk/robots.txt.

What is robots.txt?

robots.txt is a plain text file at the root of your domain: www.example.dk/robots.txt. It tells crawlers which parts of the site they may fetch. Well-behaved crawlers like Googlebot and Bingbot read it before anything else.

It is a request, not a lock. It hides nothing from people, and a crawler that ignores the rules can still fetch the pages. Anything truly private belongs behind a login.

How to make your robots.txt

  1. Choose what everyone else may do: allow all, block all, or block some paths.
  2. Blocking paths? Type one per line, each starting with /. A trailing / blocks the whole folder, like /admin/.
  3. Tick any search engines or AI crawlers you want to keep out.
  4. Add the full address of your sitemap.
  5. Copy the file, save it as robots.txt and upload it to the root of your domain.

Example: a typical robots.txt

A small business that lets everyone in, keeps its admin and basket out of search, and points to its sitemap:

  • User-agent: *
  • Disallow: /admin/
  • Disallow: /cart/
  • Sitemap: https://www.example.dk/sitemap.xml

Blocking crawling is not removing from Google

A page blocked in robots.txt can still show up in search results if other sites link to it, just without a description, because Google wasn’t allowed to read it. To keep a page out of results, let it be crawled and give it a noindex tag instead.

That is also why “Block all” is almost never right for a business website: it asks every search engine to stay away from the whole site.

Block AI crawlers in robots.txt

AI companies crawl the web to train their models. Most announce a user agent you can block: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl, used by many models) and PerplexityBot. Google-Extended is a token, not a separate crawler: blocking it keeps your content out of Gemini training without touching Google Search.

Each blocked bot gets its own group with “Disallow: /”. A bot with its own group ignores the “User-agent: *” rules, so the block is complete.

The sitemap line

Add a “Sitemap:” line with the full address of your sitemap.xml. It helps search engines find every page. It’s the one line almost every site should have.

Common robots.txt mistakes

  • “Disallow: /” left over from a test site, blocking the whole site.
  • Using robots.txt to hide a page from Google instead of noindex.
  • The file in a subfolder instead of the root of the domain.
  • A relative sitemap address instead of a full https:// one.

Frequently asked questions

At the root of the domain, named exactly robots.txt: https://www.example.dk/robots.txt. A file in a subfolder is ignored, and each subdomain needs its own.

No. GPTBot, ClaudeBot, CCBot and Google-Extended have nothing to do with Google Search. Only blocking Googlebot affects how you rank on Google.

Rarely. Google ignores it and sets its own pace. Bing respects it, but a normal small-business site never needs to slow crawlers down.

It allows everything. “Disallow: /” blocks everything. One character apart, which is why a generator is safer than typing it by hand.

No, but it’s a good idea. Without one, crawlers assume everything is allowed. A short file that allows all and names your sitemap costs nothing and helps search engines find your pages.

More free tools