B Evidence Grade B Graded accuracy verified by the review judge Safe to Cite Current

87.5%

87.5% -- sites without ai crawler policy for top websites (Tranco sample) (Q1 2026)

Q1 2026  · n=1,460  · AI Visibility

87.5% of the websites in the crawled Tranco sample have no AI crawler policy in their robots.txt file.

Cite this stat Share How we verify
Attribution: AI Visibility View source
Lighthouse Research Team
Last verified:  ·  How we verify data →
Graded Evidence graded

Questions about this data

What is the sites without ai crawler policy for top websites (Tranco sample)?

The sites without ai crawler policy for top websites (Tranco sample) is 87.5% as of Q1 2026 according to AI Visibility. This figure is the primary benchmark in the Lighthouse dataset for this segment. Individual results vary by region, channel mix, and audience quality.

What is a good sites without ai crawler policy benchmark for top websites (Tranco sample)?

A strong sites without ai crawler policy benchmark for top websites (Tranco sample) is 87.5%, using the current serving value as of Q1 2026. This is one serving-eligible observation, not a distribution median; top-quartile comparisons require the underlying distribution. Vertical, audience size, and campaign type all shift what 'good' means.

How does top websites (Tranco sample) sites without ai crawler policy compare to the industry average?

The published sites without ai crawler policy serving value for top websites (Tranco sample) is 87.5% as of Q1 2026. Individual sectors and audiences may differ substantially; this single observation does not establish an aggregate mean. Quartile comparisons require the underlying distribution, which is not implied by this record.

What factors affect sites without ai crawler policy for top websites (Tranco sample)?

Primary drivers of sites without ai crawler policy variation for top websites (Tranco sample) include audience segmentation, timing, and channel selection. Methodology documents campaign factors, but this serving record does not quantify the effect of personalization. The current serving value is 87.5% as of Q1 2026.

What sites without ai crawler policy should I aim for as a digital marketer in 2026?

For a digital marketer, the published sites without ai crawler policy serving value is 87.5%; it is an evidence-grade reference, not a personalized target as of Q1 2026. Any performance goal should account for channel mix and audience. Review the methodology before operational decisions; a gap from this serving value does not by itself indicate a structural issue.

Value
87.5%
Unit
Percentage
Data Vintage
Q1 2026
Entity
top websites (Tranco sample)
Sample Size
1,460
Geography
EMEA
B
Strong, reliable evidence

Approved for serving with evidence grade B. The record passes the documented quality and serving gates; consult the methodology for provenance details.

61/100 confidence

Single source - not yet corroborated

Extraction
95
Source Trust
60
Citation Chain
20
Independence
30
Methodology
45
Freshness
95
AI Visibility Primary Tier A
https://www.ai-visibility.org.uk/blog/how-ai-ready-is-the-web/ ↗
Confidence: 32/100 Vintage: Q1 2026 Channel: Deep Crawl
Source Article
How AI-Ready Is the Web? What Our Research Data Reveals | AI Visibility
The headline finding We crawled nearly 2,000 of the web's most prominent domains to answer a simple question: how prepared is the internet for AI? The answer is worse than we expected. Not because websites aren't trying, but because most of the ones that are trying are getting it wrong. 6.5% of the …
Original Research
AI Visibility · Original study ↗

robots.txt checked for 15 known AI user agents

Methodology documented
IndustryTechnology RegionEMEA RoleAll marketers Channelcontent
Plain Text
top websites (Tranco sample): 87.5% sites without ai crawler policy. Source: AI Visibility (Q1 2026). Via Lighthouse Intelligence -- https://lighthousedata.io/data/no-ai-crawler-policy-q1-2026
HTML Embed
<blockquote cite="https://lighthousedata.io/data/no-ai-crawler-policy-q1-2026" style="border-left:3px solid #2563eb;padding:12px 16px;margin:16px 0;font-family:system-ui,sans-serif;background:#f0f4ff;"><strong>87.5%</strong> -- sites without ai crawler policy for top websites (Tranco sample)<br><small>Source: AI Visibility (Q1 2026) · <a href="https://lighthousedata.io/data/no-ai-crawler-policy-q1-2026" target="_blank" rel="noopener">Lighthouse Intelligence</a></small></blockquote>
Share
Share on X Share on LinkedIn

Use this data in your work

Copy the citation, embed the widget, or explore thousands more verified marketing statistics -- all free.

Copy citation Explore similar stats →