B
Evidence Grade B
Graded
accuracy verified by the review judge
Safe to Cite
Recent
92%
92% -- robots txt adoption rate for Top million websites (2025)
Robots.txt is present on over 92% of the top million websites according to Ahrefs' 2025 study.
- Based on a crawl study of the top 1 million websites
Attribution:
View source
Lighthouse Research Team
Last verified:
·
How we verify data →
Source verified
Evidence graded
Value
92%
Unit
Percentage
Data Vintage
2025
Entity
Sample Size
1,000,000
Evidence Quality
B
Strong, reliable evidence
Sourced from a well-regarded research firm or industry leader with transparent methodology and a reasonably large sample.
Single source - not yet corroborated
Score breakdown
Extraction
50
Source Trust
60
Citation Chain
20
Independence
30
Methodology
45
Freshness
95
Source
Ahrefs
Primary
Tier C
https://www.gofylo.io/blog/what-llmstxt-actually-does-for-your-ai-search-visibility ↗
Source Article
What llms.txt Actually Does for Your AI Search Visibility
As of 2026, the majority of B2B buying journeys touch an AI engine before they ever reach a traditional search results page. Perplexity, ChatGPT, Claude, and Gemini are now active intermediaries between your content and your prospective customers — and they decide what to surface, summarize, and cit…
Original Research
Ahrefs
· Original study ↗
Methodology
Ahrefs crawl study of top million websites, as cited by gofylo.io
Methodology documented
Segmentation
Industrymarketing
Cite this stat
Plain Text
Top million websites: 92% robots txt adoption rate. Source: Ahrefs (2025). Via Lighthouse Intelligence -- https://lighthousedata.io/data/robots-txt-top-million-sites-2025
HTML Embed
<blockquote cite="https://lighthousedata.io/data/robots-txt-top-million-sites-2025" style="border-left:3px solid #2563eb;padding:12px 16px;margin:16px 0;font-family:system-ui,sans-serif;background:#f0f4ff;"><strong>92%</strong> -- robots txt adoption rate for Top million websites<br><small>Source: Ahrefs (2025) · <a href="https://lighthousedata.io/data/robots-txt-top-million-sites-2025" target="_blank" rel="noopener">Lighthouse Intelligence</a></small></blockquote>
Share
Related Statistics
Use this data in your work
Copy the citation, embed the widget, or explore thousands more verified marketing statistics -- all free.