What Is robots.txt? A Beginner's Guide

robots.txt is a small text file at the root of a website that tells crawlers — search engines, AI training bots, and others — which parts of the site they're allowed to visit. It's one of the simplest files on any website, and one of the most misunderstood.

What robots.txt is

When a crawler arrives at a website, the first thing it does is check for a file called robots.txt at the root of the domain. This file contains rules telling the crawler what it may and may not visit.

It's been part of the web since 1994, when a group of search engine engineers agreed on a common format. Google, Bing, and most other major search engines respect it. The file is a simple, well-defined convention that any site owner can use to communicate their preferences to crawlers.

The file must live at exactly this location:

https://example.com/robots.txt

Not in a subfolder, not at a different path. Only one robots.txt per domain.

What robots.txt is not

This is critical: robots.txt is a request, not a security mechanism.

It does not block access. It does not password-protect anything. It does not stop malicious bots. It relies entirely on the goodwill of the crawler visiting it — like a polite sign at a shop saying "employees only." Most well-behaved crawlers honor it. Bad actors ignore it completely.

Anything you truly need to protect — admin panels, staging environments, internal tools, private files — must be protected with server-side authentication, IP allowlists, or a firewall. robots.txt is not that.

The basic syntax

The file is line-based and simple. Each block starts with a User-agent line, followed by one or more Allow or Disallow rules for that agent.

Allow everything

The default state of the web. Every crawler can visit every page.

User-agent: *
Allow: /

Block everything

Useful for staging sites, development environments, or when you want a site temporarily hidden. Note that this blocks legitimate search engines too, which will remove your site from search results.

User-agent: *
Disallow: /

Block specific folders

Common for admin areas, private sections, or directories you don't want indexed.

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /tmp/

Block specific file types

Often used to keep PDFs or images out of search results, though this doesn't prevent someone from opening the file if they know the URL.

User-agent: *
Disallow: /*.pdf$
Disallow: /*.zip$

Point to your sitemap

You can tell crawlers where to find your sitemap. This doesn't affect crawling directly but helps search engines discover all your pages faster.

Sitemap: https://example.com/sitemap.xml

Blocking AI crawlers

Since around 2023, a growing number of site owners have used robots.txt to block AI training crawlers. These bots operate under different names and are run by different companies. Common ones include:

• GPTBot — OpenAI's crawler for training ChatGPT models
• ClaudeBot and anthropic-ai — Anthropic's crawlers
• Google-Extended — Google's crawler for AI training (separate from Googlebot for search)
• CCBot — Common Crawl, whose data feeds many AI training pipelines
• PerplexityBot — Perplexity AI's crawler
• Bytespider — ByteDance's crawler
• Amazonbot — Amazon's crawler
• Applebot-Extended — Apple's AI training crawler

To block them, list each one with a Disallow: / rule. Here's a full example that allows regular search engines but blocks AI training bots:

# Allow regular search engines
User-agent: *
Allow: /

# Block AI training crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Whether these companies honor the request varies. Google and OpenAI have stated publicly that they respect robots.txt. Other bots are less reliable. Treat any robots.txt-based blocking of AI crawlers as a request rather than a guarantee.

What robots.txt cannot do

A few things people often assume robots.txt does, but doesn't:

• Prevent a page from appearing in search results. If a page is linked from elsewhere on the web, Google may index it even if you disallowed crawling. Use a noindex meta tag for that.
• Protect private content. Anything sensitive needs server-side authentication.
• Stop malicious scrapers. Bad actors ignore robots.txt entirely.
• Prevent someone from opening a file if they know the URL. Disallow rules only tell crawlers not to fetch; a user typing the URL into a browser goes right through.
• Improve SEO on its own. It's a crawl-control file, not a ranking factor.

The most common robots.txt mistake

Blocking search engines from pages you actually want indexed.

A common scenario: a developer adds Disallow: / to a staging site to keep it out of Google, then forgets to remove it when the site goes live. The site remains completely invisible to search engines for weeks, and by the time someone notices, the domain has lost all its momentum.

Always check https://yourdomain.com/robots.txt after launching a site, and after any deploy that touches the file. Confirm the rules are what you intend.

The other common mistake is using robots.txt to try to remove a page that's already in Google. If a page is already indexed, robots.txt alone won't remove it. You need either a noindex meta tag or a formal removal request through Google Search Console.

Testing your robots.txt

Once your file is live, you can verify it a few ways:

• Open the URL. Visit https://yourdomain.com/robots.txt in a browser. You should see the file's content. If you see a 404, the file isn't at the root.
• Google Search Console. Under "robots.txt report," Google shows how it parsed the file and flags any syntax errors.
• Ask a specific user-agent to test. Some tools let you simulate a crawl as a particular bot (Googlebot, GPTBot) to see whether it would be blocked.