bbc.com
bbc.com blocks OpenAI's training crawler, Anthropic's training crawler, and 3 other AI crawlers. It still allows Google Search.
- GPTBot OpenAI's training crawlerBlocked
- ClaudeBot Anthropic's training crawlerBlocked
- Google-Extended Google's Gemini opt-outBlocked
- PerplexityBot Perplexity's crawlerBlocked
- ChatGPT-User ChatGPT when a person asksBlocked
- Googlebot Google SearchOpen
Other bots this site named
- Applebot-Extended AppleBlocked
- CCBot Common CrawlBlocked
- Google-CloudVertexBot GoogleBlocked
- meta-externalagent MetaBlocked
- OAI-SearchBot OpenAIBlocked
- Perplexity-User PerplexityBlocked
- PetalBot HuaweiBlocked