Crawling & Bot Access Policy

Robots.txt

Our robots.txt file is available at /robots.txt. We allow search engine crawlers (Googlebot, Bingbot, Slurp, DuckDuckBot) to access public pages including /data, /methodology, and pSEO pages.

Rate Limits

We enforce reasonable rate limits to ensure service availability. Verified search bots are allowed through Cloudflare without challenge. API endpoints require authentication via X-API-Key header.

API Access

Programmatic access to our data is available through the Stats API v2. All API requests require an API key. Free tier: 10 RPS. Pro tier: 100 RPS. See API documentation for details.

Measured Behavior

This page reflects our actual measured crawling behavior, not aspirational claims. Our Cloudflare configuration allows verified bots, blocks malicious crawlers, and rate-limits API traffic per key.