Crawl Policy Index

Public crawl rules, in English

What does this website tell AI crawlers?

Type a name you already know. We read its robots.txt every day — the public file that says which bots may look around — and we say it so a person can repeat it.

Any site on the 100,000 we watch. No account.

What changed

On 27 Oct 2026, 3,633 sites started blocking a bot. 12 opened the door.

Sites you already know

Sites you already know, in a sentence.

Same reading as the full list of 100,004. These are just the ones you can picture.

Want every site, the filters, and a file? Full table $49 / month. Who flipped, and when — $199 / month.

Subscribe