B
Evidence Grade B
Graded
accuracy verified by the review judge
Safe to Cite
Current
500,000
500,000 -- count of sites with gptbot robots txt rules for GPTBot (OpenAI) (2026)
Over 500,000 websites grew to include rules for GPTBot in robots.txt in under two years.
- Represents the broader web crawl by HTTP Archive, distinct from the top 1,000 websites sample
Attribution:
View source
Lighthouse Research Team
Last verified:
·
How we verify data →
Source verified
Evidence graded
Value
500,000
Unit
Count
Data Vintage
2026
Entity
Geography
EMEA
Evidence Quality
B
Strong, reliable evidence
Sourced from a well-regarded research firm or industry leader with transparent methodology and a reasonably large sample.
Single source - not yet corroborated
Score breakdown
Extraction
75
Source Trust
60
Citation Chain
20
Independence
30
Methodology
45
Freshness
95
Source
Paul Calvano / HTTP Archive
Primary
Tier B
https://www.ai-visibility.org.uk/blog/is-your-website-blocking-ai/ ↗
Source Article
Is Your Website Blocking AI Crawlers? How to Check | AI Visibility
Why your website might be invisible to AI crawlers
Your website might be invisible to AI, and you'd never know it.
When someone asks ChatGPT, Claude, or Gemini about your industry, your competitors show up. You don't. Not because your content is weak or your site is new, but because something techni…
Original Research
Paul Calvano / HTTP Archive
· Original study ↗
Methodology
HTTP Archive analysis by Paul Calvano, as cited
Methodology documented
Segmentation
IndustryAI & Machine Learning
RegionEMEA
RoleAll marketers
Channelcontent
Cite this stat
Plain Text
GPTBot (OpenAI): 500,000 count of sites with gptbot robots txt rules. Source: Paul Calvano / HTTP Archive (2026). Via Lighthouse Intelligence -- https://lighthousedata.io/data/gptbot-robots-txt-500k-sites
HTML Embed
<blockquote cite="https://lighthousedata.io/data/gptbot-robots-txt-500k-sites" style="border-left:3px solid #2563eb;padding:12px 16px;margin:16px 0;font-family:system-ui,sans-serif;background:#f0f4ff;"><strong>500,000</strong> -- count of sites with gptbot robots txt rules for GPTBot (OpenAI)<br><small>Source: Paul Calvano / HTTP Archive (2026) · <a href="https://lighthousedata.io/data/gptbot-robots-txt-500k-sites" target="_blank" rel="noopener">Lighthouse Intelligence</a></small></blockquote>
Share
Use this data in your work
Copy the citation, embed the widget, or explore thousands more verified marketing statistics -- all free.