How many websites pass the Xrobots Noindex check?
Based on 14 audited sites · Updated 2026-07-16
About This Check
Checks whether the X-Robots-Tag response header contains noindex, which prevents indexing at the HTTP header level.
Why This Matters
LLM Discoverability
Header-level noindex directives apply to all content types (HTML, PDF, images) and are respected by AI crawlers at the same level as meta tags, making them the preferred mechanism for server-rendered or dynamically generated pages.
SEO Impact
X-Robots-Tag is the only way to suppress indexing of non-HTML resources such as PDFs; for HTML pages it has identical effect to the meta robots noindex tag.
What To Do Next
Every audited site in the current sample passes xrobots noindex, so this check is currently a baseline expectation rather than a differentiator.
- Prioritize xrobots noindex on templates that create many URLs, because one template fix can improve a large share of bot access & control plane.
- After changing the page, re-run the audit and compare the observed value for this check instead of relying on visual inspection alone.
- Use the 100% pass rate as a benchmark: if your site fails here, passing the check moves it closer to the visible majority of crawlable, AI-readable pages.
Status Breakdown
Compliance Trend
Daily pass, warning, and failure rates across benchmark snapshots (up to 90 days).
Industry Breakdown
How different industries perform on this check.
| Industry | Pass rate | Avg score | Sites |
|---|---|---|---|
| Technology | 100% | 72 | 6limited |
| Media & Entertainment | 100% | 73.8 | 6limited |
Related Checks
Run a free audit to see whether your site passes this check, or browse the leaderboard and the audited-sites directory.