How many websites pass the Robots Present check?
Based on 563 audited sites · Updated 2026-07-16
95% confidence interval: 99.3%–100%
About This Check
Checks whether robots.txt exists at the site root. If it is missing, crawlers may still fetch this page, but you lose a key control surface for discovery guidance.
Why This Matters
LLM Discoverability
Without a robots.txt file, AI crawlers have no explicit guidance and may apply the most conservative behaviour (refusing to crawl) or the most permissive (crawling everything), leading to unpredictable inclusion in AI datasets.
SEO Impact
robots.txt is the primary crawler-management layer; its absence means search engines must infer crawl policy from meta tags and server headers alone, reducing crawl efficiency.
What To Do Next
Every audited site in the current sample passes robots present, so this check is currently a baseline expectation rather than a differentiator.
- Prioritize robots present on templates that create many URLs, because one template fix can improve a large share of bot access & control plane.
- After changing the page, re-run the audit and compare the observed value for this check instead of relying on visual inspection alone.
- Use the 100% pass rate as a benchmark: if your site fails here, passing the check moves it closer to the visible majority of crawlable, AI-readable pages.
Status Breakdown
Compliance Trend
Daily pass, warning, and failure rates across benchmark snapshots (up to 90 days).
Industry Breakdown
How different industries perform on this check.
| Industry | Pass rate | Avg score | Sites |
|---|---|---|---|
| Technology | 100% | 72 | 258 |
| Media & Entertainment | 100% | 73.8 | 84 |
| Travel & Hospitality | 100% | 74.8 | 51 |
| Finance & Banking | 100% | 72.8 | 35 |
| Education | 100% | 71.8 | 29limited |
| E-Commerce | 100% | 69.6 | 29limited |
| Government | 100% | 73.1 | 13limited |
| Food & Beverage | 100% | 71.1 | 11limited |
| Healthcare | 100% | 76.7 | 9limited |
| Other | 100% | 67 | 6limited |
Related Checks
Run a free audit to see whether your site passes this check, or browse the leaderboard and the audited-sites directory.