robots.txt
A file at a site's root telling crawlers which URLs they may request. It controls crawling, not indexing.
robots.txt was standardised as RFC 9309 in 2022. A Disallow line stops compliant crawlers fetching a URL, but a blocked URL can still be indexed from links pointing to it — to keep a page out of results, allow crawling and use noindex.
It is advisory: polite crawlers obey it, scrapers do not. Many AI crawlers now publish their own user-agent names so sites can allow or block them separately.
Crawling & Indexing timeline · Latest crawling & indexing news · What Google said
Latest headlines mentioning it
- “Blocked by robots.txt” vs. “Indexed, though blocked by robots.txt”: Differences and How To Fix Them
- Google Explains 503s, Robots.txt Limits & Bracket URLs – SEO Pulse
- What can you actually put in a robots.txt?
- Google Explains The Limit Of Robots.txt For AI Content Use
- Do LLMs Respect robots.txt? Where robots.txt Can and Can't Block AI
- Analysing Robots.txt at scale with HTTP Archive and BigQuery - transcript
- Analysing Robots.txt at scale with HTTP Archive and BigQuery