Robots Present

Based on 421 audited sites · Updated 2026-04-05

95.2%of audited sites pass this check

95% confidence interval: 92.8%–96.9%

About This Check

Checks whether robots.txt exists at the site root. If it is missing, crawlers may still fetch this page, but you lose a key control surface for discovery guidance.

Status Breakdown

95.2%pass(401)

4.5%warn(19)

0.2%fail(1)

Compliance Trend

Daily pass rate for this check across benchmark snapshots (up to 90 days).

Pass Rate by Industry

How different industries perform on this specific check.

Industry Context

Average audit scores by industry across all checks.

Technology74.9n=187

Media & Entertainment73.7n=76

Education74.8n=29

E-Commerce70.9n=28

Finance74.9n=21

Automotive73.6n=11

Fintech72.5n=8

Healthcare75.5n=8