B
Evidence Grade B
Graded
accuracy verified by the review judge
Safe to Cite
Current
78%
78% -- robots txt adoption rate for Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services) (2026)
78% of the top 200,000 domains scanned by Cloudflare have a robots.txt file.
- Vast majority of robots.txt files were written for search engine crawlers, not AI agents
Attribution:
View source
Lighthouse Research Team
Last verified:
·
How we verify data →
Source verified
Evidence graded
Value
78%
Unit
Percentage
Data Vintage
2026
Entity
Sample Size
200,000
Evidence Quality
B
Strong, reliable evidence
Sourced from a well-regarded research firm or industry leader with transparent methodology and a reasonably large sample.
Single source - not yet corroborated
Score breakdown
Extraction
95
Source Trust
60
Citation Chain
20
Independence
30
Methodology
45
Freshness
95
Source
Cloudflare
Primary
Tier A
https://seohandbook.co.uk/ai-search/agent-readiness/ ↗
Source Article
Agent-readiness | The SEO Handbook
Agent-readiness
Last updated
Traditional SEO has two layers: a content layer (what a page says) and a technical layer (whether search engines can access it). The same split is forming for AI agents. Generative Engine Optimisation and agentic search address the content layer: what agents retrieve and…
Original Research
Cloudflare
· Original study ↗
Methodology
Cloudflare scan of 200,000 top domains; article notes the vast majority of robots.txt files were written for search engine crawlers, not AI agents. SEO/agent-accessibility benchmark.
Methodology documented
Segmentation
Industrymarketing
Cite this stat
Plain Text
Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services): 78% robots txt adoption rate. Source: Cloudflare (2026). Via Lighthouse Intelligence -- https://lighthousedata.io/data/cloudflare-top-domains-robots-txt-prevalence
HTML Embed
<blockquote cite="https://lighthousedata.io/data/cloudflare-top-domains-robots-txt-prevalence" style="border-left:3px solid #2563eb;padding:12px 16px;margin:16px 0;font-family:system-ui,sans-serif;background:#f0f4ff;"><strong>78%</strong> -- robots txt adoption rate for Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services)<br><small>Source: Cloudflare (2026) · <a href="https://lighthousedata.io/data/cloudflare-top-domains-robots-txt-prevalence" target="_blank" rel="noopener">Lighthouse Intelligence</a></small></blockquote>
Share
Related Statistics
Use this data in your work
Copy the citation, embed the widget, or explore thousands more verified marketing statistics -- all free.