robots.txt

robots.txt is a plain text file at the root of a host (example.com/robots.txt, nowhere else) that tells crawlers which paths they may fetch. The grammar is small: 'User-agent:' picks who the rule applies to, 'Disallow:' and 'Allow:' name path prefixes, and a 'Sitemap:' line points at your XML sitemap. It is an honour system standardised as RFC 9309: Googlebot and Bingbot obey it, scrapers and spam bots do not.

Reach for it to keep crawlers out of places that waste their time and your server: internal search results, infinite faceted filter combinations, cart and checkout paths, admin routes, huge generated files. On a small marketing site the honest answer is usually an almost empty file with a sitemap line.

The trap that catches everyone: Disallow does not mean hidden from search. It means 'do not fetch'. A blocked URL that other pages link to can still show up in results as a bare link with no description, because Google knows it exists without having read it. Worse, blocking a page in robots.txt guarantees the crawler never sees the noindex tag on it, so the two directives cancel out. To remove a page, leave it crawlable and let it say noindex. And never ship the staging file ('Disallow: /') to production, which is the single most expensive typo in SEO.