X-Robots-Tag
An HTTP response header that tells search bots how to treat a URL, especially PDFs and other files with no HTML head.
See it
What it is
X-Robots-Tag is an HTTP response header carrying the same kinds of indexing directives as a robots meta tag. Because it lives in the response rather than the HTML head, it works for PDFs, images, feeds, and other files that cannot contain HTML metadata. A response can say X-Robots-Tag: noindex without changing the file itself.
Use it at the web server, CDN, or application layer when a whole file type or path needs a rule. It is also useful when templates are outside your control but response headers are not. Inspect the final public response because a proxy or CDN can remove or overwrite the header.
As with a meta noindex, the bot must crawl the URL to read the header. Do not pair it with a robots.txt Disallow and expect removal from search. Also keep the rule off shared static-file routes unless every matching asset really should inherit it.
Ask AI for it
Keep these non-HTML files out of search by returning the HTTP header X-Robots-Tag: noindex on their successful responses. In Nginx, apply add_header X-Robots-Tag "noindex" always; only to the intended PDF or download location, leave the URLs crawlable in robots.txt, and check each public response with curl -I after any CDN. Confirm that 200, redirect, and error responses carry only the intended directive and that ordinary images, CSS, and JavaScript files are unaffected.