Noindex Directive
A robots directive used in HTML meta tags or HTTP headers that explicitly commands search engines and crawlers not to index the designated page.
AI Summary: The noindex directive instructs search engines and AI crawlers to exclude a page from their search indexes. When implemented via HTML meta tag or X-Robots-Tag header, it prevents the page from appearing in both standard search results and AI-synthesized answers.
Technical Definition
noindex is a standardized crawler instruction that explicitly forbids search engines and AI indexing systems from storing, displaying, or referencing a URL in their search repositories.
Implementation Methods
1. HTML Meta Tag
<meta name="robots" content="noindex">
2. HTTP Response Header (X-Robots-Tag)
Essential for non-HTML files such as PDFs, JSON endpoints, and raw data dumps:
HTTP/1.1 200 OK
X-Robots-Tag: noindex, noarchive
Content-Type: application/pdf
Impact on AI Visibility
- When a page is marked
noindex, AI search engines (like Perplexity and ChatGPT Search) will not surface it as a cited source for general queries. - Pages carrying
noindexmay still be read by on-demand assistant crawlers if a user explicitly requests a direct link analysis, unless access is blocked at the HTTP/WAF layer.
Audit your site for accidental noindex tags that harm AI visibility. Run an audit with Geolify.ai.