Available
robots.txt: what does this file tell search engine crawlers?
A tiny text file at the root of your site that rarely gets a second look, yet can make a whole website disappear from search.
robots.txt is a plain text file at the root of a website, at the address /robots.txt, that tells crawlers which URLs they are allowed to fetch. Its main job is to keep crawlers from overloading your server. It does not hide pages from Google and does not secure anything.
In short
- The standard is RFC 9309, the Robots Exclusion Protocol, published in September 2022.
- It always lives at /robots.txt, lowercase, UTF-8 encoded.
- It is not security: crawlers follow it voluntarily.
- To keep a page out of results, use noindex, not robots.txt.
What is robots.txt for?
Google describes it as the file that tells search engine crawlers which URLs they can access on your site, mainly so your server is not swamped with requests [1].
Since September 2022 the protocol has been formalized as IETF standard RFC 9309. The file must sit at /robots.txt in the top-level path of the site, with an all-lowercase name, UTF-8 encoding and a plain text media type [2].
How are the rules read?
The file is split into groups. Each group opens with one or more user-agent lines naming the crawlers it targets, followed by allow and disallow rules [2].
For any given URL, the crawler applies the most specific match, meaning the rule whose path matches the most characters. If an allow and a disallow rule are equivalent, allow should win. If no rule matches, the URL is allowed [2].
| Rule | URL requested | Outcome |
|---|---|---|
| Disallow: /portal/ | /portal/invoices | Blocked |
| Disallow: /portal/ plus Allow: /portal/help | /portal/help | Allowed: the allow rule is longer |
| No rule covering /services/ | /services/furnace-repair | Allowed |
| Disallow: / | every URL | Whole site blocked for that crawler |
Longest-match logic as described in RFC 9309 [2].
Why doesn't robots.txt hide a page?
Google is blunt: robots.txt is not a mechanism for keeping a web page out of Google. A disallowed URL can still be indexed if other sites link to it, so the address can show up in results. To keep a page out, use a noindex rule or password protection [1].
The standard says the same thing in its own words: the rules are not a form of access authorization [2]. A badly behaved bot can ignore them entirely, so never rely on robots.txt to protect quotes, invoices or customer files.
It is not the right tool for choosing between duplicate pages either. Google advises against using robots.txt for canonicalization and points to the canonical tag instead [4].
What happens when the file is missing or broken?
Can robots.txt point to the sitemap?
Yes. The sitemaps.org protocol lets you add a Sitemap: line with the full URL of your XML sitemap, anywhere in the file, and you can list several [3]. It is an easy way to tell every search engine where your page list lives.
Which robots.txt mistakes cost the most?
- Shipping the staging site's Disallow: / rule on launch day.
- Blocking CSS files or images the page needs to render properly.
- Disallowing a page to remove it from Google, which stops the crawler from ever seeing the noindex rule.
- Treating a secret page listed under disallow as protected, when the file is public and advertises it.
- Letting the server return a 5xx error for /robots.txt.
What does Pilotiko do with robots.txt?
For the business website it builds and hosts, Pilotiko generates robots.txt alongside the sitemap, structured data and an llms.txt file. That covers a search engine optimization basic; it is not a substitute for properly protecting private pages.
Frequently asked questions
Where does robots.txt have to be?
At the root of the site, at the exact path /robots.txt, in lowercase. A file placed in a subfolder is not where crawlers look for it.
Can robots.txt remove a page from Google?
No. A blocked page can stay indexed if other sites link to it. Use a noindex rule instead, and leave the page crawlable so the crawler can actually read that rule.
Does my site need a robots.txt file?
It is not mandatory: under RFC 9309, if the file returns a 404 the crawler may fetch the whole site. A simple file is still handy to point to your sitemap and keep crawlers out of technical areas.
Do all bots obey robots.txt?
No. Google notes that other crawlers might not, and the standard states the rules are not access authorization. Sensitive data needs real protection.
How long before a change takes effect?
RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable. A change can therefore take up to a day to be picked up.
How is robots.txt different from llms.txt?
robots.txt sets access rules for crawlers. llms.txt is a separate proposal that summarizes a site's content for language models; it neither blocks nor allows anything.
Sources
- Google Search Central, Introduction to robots.txt (accessed on October 15, 2026).
- IETF (RFC Editor), RFC 9309 : Robots Exclusion Protocol (accessed on October 15, 2026).
- sitemaps.org, Sitemaps XML format (accessed on October 15, 2026).
- Google Search Central, How to specify a canonical URL with rel="canonical" and other methods (accessed on October 15, 2026).
Read next
XML sitemap: what does it do for search engines?
What an XML sitemap is, which tags it uses, the official 50,000-URL limit, whether a small business site needs one and the mistakes that confuse search engines.
Canonical tag: how do you point Google to the right version of a page?
A canonical tag tells Google which URL to prefer among duplicate pages. The methods, how strong each signal is, a worked URL count and the usual mistakes.
llms.txt: what is this file for language models?
The llms.txt file explained: who proposed it, its Markdown structure, a sample for a Quebec service business, and what it does not do for your website.
SEO: what is search engine optimization?
What SEO really covers, how Google finds and ranks pages, a worked example for a Quebec service business and the mistakes that quietly cost visibility.