B
Evidence Grade B
Graded
accuracy verified by the review judge
Safe to Cite
Recent
13.26%
13.26% -- robots txt noncompliance rate for AI crawlers (Q2 2025)
In Q2 2025, 13.26 percent of requests across all AI bots ignored robots.txt directives.
Attribution:
View source
Lighthouse Research Team
Last verified:
·
How we verify data →
Source verified
Evidence graded
Value
13.26%
Unit
Percentage
Data Vintage
Q2 2025
Entity
Evidence Quality
B
Strong, reliable evidence
Sourced from a well-regarded research firm or industry leader with transparent methodology and a reasonably large sample.
Single source - not yet corroborated
Score breakdown
Extraction
75
Source Trust
60
Citation Chain
20
Independence
30
Methodology
45
Freshness
95
Source
Tollbit
Primary
Tier B
https://encypher.com/blog/how_robots_txt_fails_publishers_the_case_for_in_content_rights_signals ↗
Source Article
How Robots.txt Fails Publishers: In-Content Rights Signals
How Robots.txt Fails Publishers: In-Content Rights Signals
In December 2025, a federal judge in the Southern District of New York compared robots.txt to a "keep off the grass" sign. 󠇟󠇠󠇡󠇢󠅝󠅊󠅏󠆰󠅞󠄛󠇠󠇉󠆽󠅈󠆨󠄠󠆵︂󠇚󠇠󠅶󠅡󠄤󠅌󠄯󠆓󠅈󠅅󠄴󠄣󠅁󠅘︍󠆊󠅶󠄍󠇮󠅧󠆵󠅀󠅼󠄃󠄴󠅿The case was Ziff Davis v. OpenAI. 󠇟󠇠󠇡󠇢󠄮󠅯󠄷󠆬󠅕󠆻︍︁󠄙󠆓󠆁󠅓󠅵󠅧󠄅︄󠇢︍󠅛󠇜󠄖󠅍󠆃󠆗︀󠅣󠅧󠆙󠆴󠄧󠄟󠄢󠇘󠅡󠆛…
Original Research
Tollbit
· Original study ↗
Methodology
Data from Tollbit, a crawler monetization company, measuring AI bot requests ignoring robots.txt
Methodology documented
Segmentation
Industrymarketing
Cite this stat
Plain Text
AI crawlers: 13.26% robots txt noncompliance rate. Source: Tollbit (Q2 2025). Via Lighthouse Intelligence -- https://lighthousedata.io/data/ai-bot-robots-txt-noncompliance-q2-2025
HTML Embed
<blockquote cite="https://lighthousedata.io/data/ai-bot-robots-txt-noncompliance-q2-2025" style="border-left:3px solid #2563eb;padding:12px 16px;margin:16px 0;font-family:system-ui,sans-serif;background:#f0f4ff;"><strong>13.26%</strong> -- robots txt noncompliance rate for AI crawlers<br><small>Source: Tollbit (Q2 2025) · <a href="https://lighthousedata.io/data/ai-bot-robots-txt-noncompliance-q2-2025" target="_blank" rel="noopener">Lighthouse Intelligence</a></small></blockquote>
Share
Use this data in your work
Copy the citation, embed the widget, or explore thousands more verified marketing statistics -- all free.