How many websites pass the Robots Present check?
Based on 488 audited sites · Updated 2026-05-08
95% confidence interval: 95.8%–98.6%
About This Check
Checks whether robots.txt exists at the site root. If it is missing, crawlers may still fetch this page, but you lose a key control surface for discovery guidance.
Why This Matters
LLM Discoverability
Without a robots.txt file, AI crawlers have no explicit guidance and may apply the most conservative behaviour (refusing to crawl) or the most permissive (crawling everything), leading to unpredictable inclusion in AI datasets.
SEO Impact
robots.txt is the primary crawler-management layer; its absence means search engines must infer crawl policy from meta tags and server headers alone, reducing crawl efficiency.
Status Breakdown
Compliance Trend
Daily pass rate across benchmark snapshots (up to 90 days).
Industry Breakdown
How different industries perform on this check.