Crawl Policy Index

Look up

google.com

Any site on the 100,000 we watch. No account.

google.com

Search · United States · Swedish

google.com is open to the main AI crawlers.

  • GPTBot OpenAI's training crawler
    Open
  • ClaudeBot Anthropic's training crawler
    Open
  • Google-Extended Google's Gemini opt-out
    Open
  • PerplexityBot Perplexity's crawler
    Open
  • ChatGPT-User ChatGPT when a person asks
    Open
  • Googlebot Google Search
    Open

Want every site on the list, filters, and a file of today's table?

Subscribe · $49 / month