noindex
A directive telling search engines 'you may read this page, just never list it'. The proper way to keep a live URL out of results.
See it
What it is
noindex is a directive that says 'read this page if you like, just never show it in results'. You deliver it either as a meta tag in the head (<meta name='robots' content='noindex'>) or, for files that have no head like PDFs and images, as an X-Robots-Tag HTTP header. It removes the page from the index on the next crawl and keeps it out.
Typical customers: thank-you and order-confirmation pages, internal search result pages, login and account screens, tag or author archives that are thin by nature, printer-friendly duplicates, and staging environments. If two pages are near-duplicates that both deserve to exist, a canonical URL is the better tool, since it consolidates the ranking signals instead of throwing one page away.
The classic self-defeating combo: blocking the page in robots.txt and tagging it noindex. The two controls conflict. The crawler cannot fetch the page, so it cannot discover the tag or refresh what it saw last time, and the URL can linger in results as a bare link for as long as the engine keeps trusting its old picture. Leave noindexed pages crawlable. Also remember noindex is not access control: the page is still public, still shareable, still in anyone's browser history. If it must be private, put it behind auth.
Ask AI for it
Add noindex handling to this site. Render <meta name='robots' content='noindex, follow'> in the head of pages that should stay out of search: thank-you and confirmation pages, internal search results, account and settings screens, and thin tag archives. Drive it from a per-route or per-document flag rather than hardcoding it in a layout. Send an 'X-Robots-Tag: noindex' response header for non-HTML files that need the same treatment. Make sure none of these paths are also disallowed in robots.txt, since a blocked page can never be read to see the tag. For staging, prefer authentication; when it has to stay publicly reachable, send a blanket 'X-Robots-Tag: noindex' for the whole environment and leave its robots.txt permissive, so crawlers can fetch the pages and actually see the directive.