robots.txt
A plain text file at your domain root telling crawlers which paths to skip. Politeness, not a lock: good bots obey it, bad ones ignore it.
See it
What it is
robots.txt is a plain text file at the root of a host (example.com/robots.txt, nowhere else) that tells crawlers which paths they may fetch. The grammar is small: 'User-agent:' picks who the rule applies to, 'Disallow:' and 'Allow:' name path prefixes, and a 'Sitemap:' line points at your XML sitemap. It is an honour system standardised as RFC 9309: Googlebot and Bingbot obey it, scrapers and spam bots do not.
Reach for it to keep crawlers out of places that waste their time and your server: internal search results, infinite faceted filter combinations, cart and checkout paths, admin routes, huge generated files. On a small marketing site the honest answer is usually an almost empty file with a sitemap line.
The trap that catches everyone: Disallow does not mean hidden from search. It means 'do not fetch'. A blocked URL that other pages link to can still show up in results as a bare link with no description, because Google knows it exists without having read it. Worse, pairing a Disallow with a noindex tag hands the engine two controls that fight each other. The crawler cannot fetch the page, so it cannot discover the noindex or refresh one it saw months ago, and what happens next depends entirely on what that engine already observed. To remove a page, leave it crawlable and let it say noindex. And never ship a blanket 'Disallow: /' from staging to production, which is the single most expensive typo in SEO.
Ask AI for it
Create a robots.txt served at the site root. Allow all crawlers by default with 'User-agent: *', then Disallow only the paths that should never be fetched: /api/, /admin/, internal search results, cart and checkout, and any filter or sort query paths that generate near-infinite URL variants. Do not disallow pages that carry a noindex tag, since the crawler must be able to fetch them to see it. Finish with an absolute 'Sitemap: https://example.com/sitemap.xml' line. For staging, put the environment behind authentication if you can; if it has to stay publicly reachable, keep it crawlable and send a noindex header instead of stacking a blanket Disallow on top, so engines can actually read the directive. Either way, make sure the production build can never emit a 'Disallow: /' file.