theguardian.com
theguardian.com blocks Anthropic's training crawler and Perplexity's crawler. It still allows OpenAI's training crawler, Google's Gemini opt-out, and 2 other AI crawlers.
- GPTBot OpenAI's training crawlerOpen
- ClaudeBot Anthropic's training crawlerBlocked
- Google-Extended Google's Gemini opt-outOpen
- PerplexityBot Perplexity's crawlerBlocked
- ChatGPT-User ChatGPT when a person asksOpen
- Googlebot Google SearchOpen
Other bots this site named
- Applebot-Extended AppleBlocked
- CCBot Common CrawlBlocked
- Claude-SearchBot AnthropicBlocked
- Claude-User AnthropicBlocked
- Google-CloudVertexBot GoogleBlocked
- meta-externalagent MetaBlocked
- PetalBot HuaweiBlocked