X-Robots-Tag HTTP Header
An HTTP response header used to deliver robots exclusion directives (such as noindex, nofollow, or noarchive) for non-HTML resources and dynamic API endpoints.
AI Summary: The X-Robots-Tag is an HTTP response header that provides search indexing instructions for any resource, including non-HTML assets like PDFs, images, and JSON APIs. It offers identical directives to HTML meta robots tags but operates at the HTTP transport layer.
Technical Definition
The X-Robots-Tag is an HTTP response header specified in search engine technical documentation to convey crawler directives at the protocol level.
While HTML <meta name="robots"> tags can only be embedded inside HTML documents, X-Robots-Tag can be returned with any file type, including PDFs, XML feeds, CSV exports, and REST API payloads.
Header Syntax & Directives
HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex, noarchive, nosnippet
Server Configuration Examples
Nginx Configuration
# Apply X-Robots-Tag to sensitive download directory
location /downloads/ {
add_header X-Robots-Tag "noindex, nofollow" always;
}
Apache (.htaccess) Configuration
<FilesMatch "\.(pdf|doc|docx|csv)$">
Header set X-Robots-Tag "noindex, noarchive"
</FilesMatch>
Advantages for AI Governance
Using X-Robots-Tag allows webmasters to selectively serve noindex headers based on dynamic edge conditions—such as user authentication status, subscription level, or specific crawler identity—without altering the underlying asset file.
Verify that your non-HTML assets and API responses deliver correct X-Robots-Tag headers. Audit with Geolify.ai.