We read your public rules. We fetch the homepage once.

If you run a website and saw us in your logs, this page is how to recognise us and how to ask us to stop.

Who we are

Crawl Policy Index keeps a record of what websites say about AI crawlers. We are not a search engine. We do not train models. We do not follow links into your articles or products. If robots.txt allows /, we fetch the homepage once so we can record language and a rough site type.

What we request

At most four addresses per site, once per day:

• your crawl-rules file (/robots.txt) — the main one
• an optional AI-instructions file (/llms.txt) — often missing; a “not found” is still useful
• your homepage (/) — only if robots.txt allows it; once, not every page
• your top-level sitemap (/sitemap.xml) — we do not then fetch every page listed there

We keep a copy of those files so we can check our work later. We publish counts and short facts, not a copy of your whole site.

How to recognise us

• At most one successful request per file per site per day
• One request at a time to your server
• Name in the request: CrawlPolicyIndex/1.0 (+https://crawlpolicyindex.org/bot)
• Server address: 178.128.255.20

How to be left out

Email exclude@crawlpolicyindex.org with the website you want excluded (for example, example.com). We stop within 24 hours and keep a record that you asked. We will not keep fetching you. Future lists of sites will leave you out.

We do not want personal data. If a fetched file happens to contain any, we delete it rather than study it.