# robots.txt: what does this file tell search engine crawlers?

> robots.txt tells crawlers which URLs they may visit. How RFC 9309 rules are read, what happens on errors, a small business example and the costly mistakes.

Updated on October 15, 2026 · https://app.pilotiko.com/en/glossary/robots-txt

robots.txt is a plain text file at the root of a website, at the address /robots.txt, that tells crawlers which URLs they are allowed to fetch. Its main job is to keep crawlers from overloading your server. It does not hide pages from Google and does not secure anything.

## In short
- The standard is RFC 9309, the Robots Exclusion Protocol, published in September 2022.
- It always lives at /robots.txt, lowercase, UTF-8 encoded.
- It is not security: crawlers follow it voluntarily.
- To keep a page out of results, use noindex, not robots.txt.

## What is robots.txt for?

Google describes it as the file that tells search engine crawlers which URLs they can access on your site, mainly so your server is not swamped with requests [1].

Since September 2022 the protocol has been formalized as IETF standard RFC 9309. The file must sit at /robots.txt in the top-level path of the site, with an all-lowercase name, UTF-8 encoding and a plain text media type [2].

## How are the rules read?

The file is split into groups. Each group opens with one or more user-agent lines naming the crawlers it targets, followed by allow and disallow rules [2].

For any given URL, the crawler applies the most specific match, meaning the rule whose path matches the most characters. If an allow and a disallow rule are equivalent, allow should win. If no rule matches, the URL is allowed [2].

| Rule | URL requested | Outcome |
|---|---|---|
| Disallow: /portal/ | /portal/invoices | Blocked |
| Disallow: /portal/ plus Allow: /portal/help | /portal/help | Allowed: the allow rule is longer |
| No rule covering /services/ | /services/furnace-repair | Allowed |
| Disallow: / | every URL | Whole site blocked for that crawler |

Longest-match logic as described in RFC 9309 [2].

## Why doesn't robots.txt hide a page?

Google is blunt: robots.txt is not a mechanism for keeping a web page out of Google. A disallowed URL can still be indexed if other sites link to it, so the address can show up in results. To keep a page out, use a noindex rule or password protection [1].

The standard says the same thing in its own words: the rules are not a form of access authorization [2]. A badly behaved bot can ignore them entirely, so never rely on robots.txt to protect quotes, invoices or customer files.

It is not the right tool for choosing between duplicate pages either. Google advises against using robots.txt for canonicalization and points to the [canonical tag](https://app.pilotiko.com/en/glossary/canonical-tag) instead [4].

## What happens when the file is missing or broken?

**Example: four situations for a roofing company's website in Red Deer**

RFC 9309 sets out different behaviour depending on how the server answers a request for /robots.txt [2].

- 404 response (no file) : Crawler may fetch any resource
- 503 response (server down) : Crawler must assume everything is disallowed
- 600 KiB file : Only the first 500 KiB are guaranteed to be parsed
- File edited this morning : A cached copy may be used for up to 24 hours
- **= A long server outage can shut crawlers out of the whole site even without a single disallow rule.**

## Can robots.txt point to the sitemap?

Yes. The sitemaps.org protocol lets you add a Sitemap: line with the full URL of your [XML sitemap](https://app.pilotiko.com/en/glossary/xml-sitemap), anywhere in the file, and you can list several [3]. It is an easy way to tell every search engine where your page list lives.

## Which robots.txt mistakes cost the most?

- Shipping the staging site's Disallow: / rule on launch day.
- Blocking CSS files or images the page needs to render properly.
- Disallowing a page to remove it from Google, which stops the crawler from ever seeing the noindex rule.
- Treating a secret page listed under disallow as protected, when the file is public and advertises it.
- Letting the server return a 5xx error for /robots.txt.

## What does Pilotiko do with robots.txt?

For the business website it builds and hosts, Pilotiko generates robots.txt alongside the sitemap, [structured data](https://app.pilotiko.com/en/glossary/structured-data) and an [llms.txt](https://app.pilotiko.com/en/glossary/llms-txt) file. That covers a [search engine optimization](https://app.pilotiko.com/en/glossary/seo) basic; it is not a substitute for properly protecting private pages.

## Frequently asked questions

### Where does robots.txt have to be?

At the root of the site, at the exact path /robots.txt, in lowercase. A file placed in a subfolder is not where crawlers look for it.

### Can robots.txt remove a page from Google?

No. A blocked page can stay indexed if other sites link to it. Use a noindex rule instead, and leave the page crawlable so the crawler can actually read that rule.

### Does my site need a robots.txt file?

It is not mandatory: under RFC 9309, if the file returns a 404 the crawler may fetch the whole site. A simple file is still handy to point to your sitemap and keep crawlers out of technical areas.

### Do all bots obey robots.txt?

No. Google notes that other crawlers might not, and the standard states the rules are not access authorization. Sensitive data needs real protection.

### How long before a change takes effect?

RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable. A change can therefore take up to a day to be picked up.

### How is robots.txt different from llms.txt?

robots.txt sets access rules for crawlers. llms.txt is a separate proposal that summarizes a site's content for language models; it neither blocks nor allows anything.

## Sources

1. Google Search Central, « Introduction to robots.txt », https://developers.google.com/search/docs/crawling-indexing/robots/intro (accessed on 2026-10-15)
2. IETF (RFC Editor), « RFC 9309 : Robots Exclusion Protocol », https://www.rfc-editor.org/rfc/rfc9309.html (accessed on 2026-10-15)
3. sitemaps.org, « Sitemaps XML format », https://www.sitemaps.org/protocol.html (accessed on 2026-10-15)
4. Google Search Central, « How to specify a canonical URL with rel="canonical" and other methods », https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls (accessed on 2026-10-15)