B
Evidence Grade B
Graded
accuracy verified by the review judge
Safe to Cite
Recent
26.1%
26.1% -- false positive rate for A/B testing with regular peeking (2025)
With regular peeking, the actual false positive rate in A/B testing jumps to 26.1% according to calculations by Evan Miller.
Attribution:
View source
Lighthouse Research Team
Last verified:
·
How we verify data →
Source verified
Evidence graded
Value
26.1%
Unit
Percentage
Data Vintage
2025
Entity
Evidence Quality
B
Strong, reliable evidence
Sourced from a well-regarded research firm or industry leader with transparent methodology and a reasonably large sample.
Single source - not yet corroborated
Score breakdown
Extraction
75
Source Trust
60
Citation Chain
20
Independence
30
Methodology
45
Freshness
95
Source
Evan Miller
Primary
Tier B
https://kirro.io/type-1-vs-type-2-errors-in-a-b-testing ↗
Source Article
Type 1 vs type 2 error in A/B testing
A type 1 error means your A/B test says “we have a winner” when there’s no real difference. A type 2 error means your test says “nothing happened” when there actually was a winner hiding in the data.
Both cost you money. One makes you ship changes that don’t work. The other makes you throw away chan…
Original Research
Evan Miller
· Original study ↗
Methodology
Calculation by Evan Miller cited in article
Methodology documented
Segmentation
IndustryAll
RoleAll marketers
Cite this stat
Plain Text
A/B testing with regular peeking: 26.1% false positive rate. Source: Evan Miller (2025). Via Lighthouse Intelligence -- https://lighthousedata.io/data/ab-testing-peeking-false-positive-rate-26-1
HTML Embed
<blockquote cite="https://lighthousedata.io/data/ab-testing-peeking-false-positive-rate-26-1" style="border-left:3px solid #2563eb;padding:12px 16px;margin:16px 0;font-family:system-ui,sans-serif;background:#f0f4ff;"><strong>26.1%</strong> -- false positive rate for A/B testing with regular peeking<br><small>Source: Evan Miller (2025) · <a href="https://lighthousedata.io/data/ab-testing-peeking-false-positive-rate-26-1" target="_blank" rel="noopener">Lighthouse Intelligence</a></small></blockquote>
Share
Related Statistics
Use this data in your work
Copy the citation, embed the widget, or explore thousands more verified marketing statistics -- all free.