noindex
A directive telling search engines 'you may read this page, just never list it'. The proper way to keep a live URL out of results.
See it
What it is
noindex is a directive that says 'read this page if you like, just never show it in results'. You deliver it either as a meta tag in the head (<meta name='robots' content='noindex'>) or, for files that have no head like PDFs and images, as an X-Robots-Tag HTTP header. It removes the page from the index on the next crawl and keeps it out.
Typical customers: thank-you and order-confirmation pages, internal search result pages, login and account screens, tag or author archives that are thin by nature, printer-friendly duplicates, and staging environments. If two pages are near-duplicates that both deserve to exist, a canonical URL is the better tool, since it consolidates the ranking signals instead of throwing one page away.
The classic self-defeating combo: blocking the page in robots.txt and tagging it noindex. The crawler never fetches the page, so it never sees the tag, and the URL can linger in results as a bare link forever. Leave noindexed pages crawlable. Also remember noindex is not access control: the page is still public, still shareable, still in anyone's browser history. If it must be private, put it behind auth.
Ask AI for it
Add noindex handling to this site. Render <meta name='robots' content='noindex, follow'> in the head of pages that should stay out of search: thank-you and confirmation pages, internal search results, account and settings screens, and thin tag archives. Drive it from a per-route or per-document flag rather than hardcoding it in a layout. Send an 'X-Robots-Tag: noindex' response header for non-HTML files that need the same treatment. Make sure none of these paths are also disallowed in robots.txt, since a blocked page can never be read to see the tag, and set a blanket noindex header for the entire staging environment.