B Evidence Grade B Graded accuracy verified by the review judge Safe to Cite Current

78%

78% -- robots txt adoption rate for Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services) (2026)

2026  · n=200,000  · Cloudflare

78% of the top 200,000 domains scanned by Cloudflare have a robots.txt file.

  • Vast majority of robots.txt files were written for search engine crawlers, not AI agents
Cite this stat Share How we verify
Attribution: Cloudflare View source
Lighthouse Research Team
Last verified:  ·  How we verify data →
Graded Evidence graded

Questions about this data

What is the robots txt adoption rate for Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services)?

The robots txt adoption rate for Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services) is 78% as of 2026 according to Cloudflare. This figure is the primary benchmark in the Lighthouse dataset for this segment. Individual results vary by region, channel mix, and audience quality.

What is a good robots txt adoption rate benchmark for Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services)?

A strong robots txt adoption rate benchmark for Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services) is 78%, using the current serving value as of 2026. This is one serving-eligible observation, not a distribution median; top-quartile comparisons require the underlying distribution. Vertical, audience size, and campaign type all shift what 'good' means.

How does Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services) robots txt adoption rate compare to the industry average?

The published robots txt adoption rate serving value for Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services) is 78% as of 2026. Individual sectors and audiences may differ substantially; this single observation does not establish an aggregate mean. Quartile comparisons require the underlying distribution, which is not implied by this record.

What factors affect robots txt adoption rate for Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services)?

Primary drivers of robots txt adoption rate variation for Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services) include audience segmentation, timing, and channel selection. Methodology documents campaign factors, but this serving record does not quantify the effect of personalization. The current serving value is 78% as of 2026.

What robots txt adoption rate should I aim for as a digital marketer in 2026?

For a digital marketer, the published robots txt adoption rate serving value is 78%; it is an evidence-grade reference, not a personalized target as of 2026. Any performance goal should account for channel mix and audience. Review the methodology before operational decisions; a gap from this serving value does not by itself indicate a structural issue.

Value
78%
Unit
Percentage
Data Vintage
2026
Entity
Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services)
Sample Size
200,000
B
Strong, reliable evidence

Approved for serving with evidence grade B. The record passes the documented quality and serving gates; consult the methodology for provenance details.

61/100 confidence

Single source - not yet corroborated

Extraction
95
Source Trust
60
Citation Chain
20
Independence
30
Methodology
45
Freshness
95
Cloudflare Primary Tier A
https://seohandbook.co.uk/ai-search/agent-readiness/ ↗
Confidence: 57/100 Vintage: 2026 Channel: Deep Crawl
Source Article
Agent-readiness | The SEO Handbook
Agent-readiness Last updated Traditional SEO has two layers: a content layer (what a page says) and a technical layer (whether search engines can access it). The same split is forming for AI agents. Generative Engine Optimisation and agentic search address the content layer: what agents retrieve and…
Original Research
Cloudflare · Original study ↗

Cloudflare scan of 200,000 top domains; article notes the vast majority of robots.txt files were written for search engine crawlers, not AI agents. SEO/agent-accessibility benchmark.

Methodology documented
Industrymarketing
Plain Text
Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services): 78% robots txt adoption rate. Source: Cloudflare (2026). Via Lighthouse Intelligence -- https://lighthousedata.io/data/cloudflare-top-domains-robots-txt-prevalence
HTML Embed
<blockquote cite="https://lighthousedata.io/data/cloudflare-top-domains-robots-txt-prevalence" style="border-left:3px solid #2563eb;padding:12px 16px;margin:16px 0;font-family:system-ui,sans-serif;background:#f0f4ff;"><strong>78%</strong> -- robots txt adoption rate for Top 200,000 domains scanned by Cloudflare (filtered to exclude redirects, ad servers, tunnelling services)<br><small>Source: Cloudflare (2026) · <a href="https://lighthousedata.io/data/cloudflare-top-domains-robots-txt-prevalence" target="_blank" rel="noopener">Lighthouse Intelligence</a></small></blockquote>
Share
Share on X Share on LinkedIn

Use this data in your work

Copy the citation, embed the widget, or explore thousands more verified marketing statistics -- all free.

Copy citation Explore similar stats →