How many websites pass the Ai Not Blocked check?

Based on 596 audited sites · Updated 2026-07-16

81.4%of audited sites pass this check↑2.7% vs last month

95% confidence interval: 78.1%–84.3%

About This Check

Checks whether robots.txt blocks common AI/LLM crawlers. Blocking AI agents can reduce your visibility in LLM-based discovery systems.

Why This Matters

LLM Discoverability

Blocking known AI user-agents (GPTBot, ClaudeBot, etc.) directly prevents those models from citing your content, summarising your pages, or recommending your site in responses.

SEO Impact

Blanket AI agent blocks can extend to Google's AI-driven tools such as Google-Extended; misconfigurations can inadvertently impair standard indexing as well.

What To Do Next

18.6% of audited sites need attention on ai not blocked (111 of 596 sites are warning or failing).

Prioritize ai not blocked on templates that create many URLs, because one template fix can improve a large share of bot access & control plane.
After changing the page, re-run the audit and compare the observed value for this check instead of relying on visual inspection alone.
Use the 81.4% pass rate as a benchmark: if your site fails here, passing the check moves it closer to the visible majority of crawlable, AI-readable pages.

Status Breakdown

81.4%pass(485)

5.5%warn(33)

13.1%fail(78)

Compliance Trend

Daily pass, warning, and failure rates across benchmark snapshots (up to 90 days).

Industry Breakdown

How different industries perform on this check.

100% of audited Travel & Hospitality sites pass this check (n=51), compared with 53.6% of Media & Entertainment sites (n=84).

credible samples first · then pass rateshown-industry average 81.7%

Industry pass rates, average audit scores, and sample sizes. A marker in each pass-rate bar shows the weighted average across the displayed industries.
Industry	Pass rate	Avg score	Sites
Travel & Hospitality	100%	74.8	51
Technology	86.7%	72	278
Finance & Banking	84.6%	72.8	39
Education	73.3%	71.8	30
E-Commerce	73.3%	69.6	30
Media & Entertainment	53.6%	73.8	84
Food & Beverage	100%	71.1	11limited
Automotive	100%	75.6	5limited
Professional Services	100%	80.8	4limited
Retail	100%	78.3	3limited

Related Checks

Run a free audit to see whether your site passes this check, or browse the leaderboard and the audited-sites directory.