Robots meta tag
A tag in an HTML page's head that tells search bots whether to index the page or follow its links.
See it
What it is
The robots meta tag is an HTML <meta> element that gives search bots page-level instructions. Put it in the document head, usually as <meta name="robots" content="noindex">. Common directives include noindex, nofollow, nosnippet, and max-image-preview. Without a tag, indexing and link following are allowed by default.
Use it when an HTML page may be visited but should not appear in search, such as an internal search result or a temporary campaign page. A crawler-specific name such as googlebot can narrow an instruction, while name="robots" addresses supporting crawlers generally.
The trap is blocking the URL in robots.txt at the same time. A blocked crawler cannot fetch the HTML, so it cannot see the noindex tag. Let the page be crawled until the noindex is processed, and use an X-Robots-Tag header instead for PDFs and other non-HTML files.
Ask AI for it
Add the correct robots meta directives to these HTML pages. Use <meta name="robots" content="noindex"> for pages that must stay out of search, and use nofollow only when bots should not follow any links on the page. Omit the tag on ordinary indexable pages, and use max-image-preview:large where large image previews are wanted. Put the tag in the server-rendered <head>, remove any conflicting robots.txt Disallow rule, and verify in Google Search Console URL Inspection that Googlebot received the final directive.