What these numbers mean — and what they don’t.

This index records what websites say about AI crawlers in a public file called robots.txt. It is not a measure of traffic, and it is not a scrape of page content.

In one minute

• We look at a fixed list of 100,004 websites (this panel, not “the web”).
• Once a day we fetch each site’s public crawl rules.
• We record whether a named bot may crawl the site. Blocked means the homepage is off limits. Open means it may crawl — even if search, login, or other utility URLs are closed.
• If the site never named the bot but said “block all bots,” we still count it — and we label it as that blanket policy, not as a named decision.

What is robots.txt?

It is a short public file most websites already publish. Search engines have used it for years. It is the site speaking in public: “this automated visitor may look around, that one may not.” Anyone can read it. We do.

If robots.txt allows the homepage, we fetch it once for language and a rough site type. We do not follow links into articles, product pages, or comment threads.

Named vs a blanket “no”

Many sites have one rule that says “block everyone.” That is a real policy, and we include it. We do not treat it as the site naming GPTBot or ClaudeBot.

On the overview chart, pink is “named this bot and blocked it.” Purple is “never named this bot, but blocked all bots.” Both mean GPTBot cannot crawl. They are different kinds of decision.

A site with no readable rules file is left out of the count. We do not treat silence as permission.

The file we were served — not Google’s copy

We fetch as ourselves: CrawlPolicyIndex/1.0, from our published address, once a day. We record the robots.txt that server sent us.

Some sites send a different file to crawlers they have verified by IP — Googlebot is the usual example — and a stricter fallback to everyone else. We do not pretend to be Googlebot. We do not guess the private copy. A site can block us, and every unverified bot, while still appearing in Google. That is not a contradiction in these tables. It is a two-channel policy, and we only see one channel.

So “blocked GPTBot” here means: the public rules file we were given named GPTBot and said no — or said “block all bots.” It does not mean we watched GPTBot being refused at the door.

Why you should not write “the web”

These figures come from a frozen list of 100,004 well-known sites (pinned Tranco list, snapshot 2026-10-15). That is a defined panel, not a sample of every website. Do not write a percentage of the internet.

Names in the tables are tokens we checked against each operator’s own documentation. Tokens we could not confirm are stored, but they are not in published figures.

How the table filters work

Country comes from a labeled list when we have one, otherwise from the domain ending (.se → Sweden, .co.uk → United Kingdom). Most .com sites have no country in this table.

Type is labeled news where we maintain a list, otherwise .edu / .gov, otherwise a rough read of the homepage (news article markup, “add to cart”, and so on). Unknown is allowed — it is better than guessing.

Language is the homepage lang tag, fetched once when robots.txt allows /. Swedish language is not the same as Sweden.

Training, search, and a person asking

Companies often run more than one bot. We keep them apart on purpose:

• Training AI — collecting pages to build a model (GPTBot, ClaudeBot).
• AI search — collecting pages to answer questions or cite sources.
• When a person asks — fetching one page because a human requested that link.

A publisher can block training and still allow search. Mixing those two into “AI bots” hides the story.

Switches that are not crawlers

Names such as Google-Extended and Applebot-Extended are opt-out signals. They do not crawl. Never add them into a “how many crawlers were blocked” total.

Googlebot is a sanity check

Almost no serious site wants to disappear from Google. If our tables showed a large share naming Googlebot and blocking it, first ask whether we are looking at the public fallback file — the one served to unverified visitors — not whether Google itself is locked out. A jump in that number is a reason to inspect the files, not to write “the web blocked Google.”

Keeping the record honest

We store the original file so we can re-read it if our software is wrong. A mistake is fixed by reading the same files again — not by quietly changing yesterday’s number. Every figure on the site should be explainable from this page. If it is not, do not cite it.