With growing awareness and concerns about AI, companies are scrambling to address a problem: how do they protect their content and intellectual property without hiding themselves from potential users and customers? The first tool most companies reach for is robots.txt, a plain-text file at the root of a site that says who is allowed in.
This got me thinking. If millions of sites publish one, what do those files actually reveal? Do they show how industries differ in how they are handling AI? Do they accidentally expose AI partnerships or deals?
I pulled the robots.txt files from the top million websites to see if I could glean any interesting patterns from all these policies. Have companies gotten more clever at handling bots?
The answer is mostly no. In a sample of 356 testable sites that declared a GPTBot block, 279 did not block a request with the GPTBot User-Agent at the exact path they blocked. Some of the most repeated AI policies were boilerplate files copied from third parties or CDNs, and a few had contradictory policies.
I’m not the first person to look at these files. FT Strategies analysed 70 major publishers and found a clear pattern: 60 of them restrict at least one AI crawler, and 23 block the training crawler while leaving the search crawler some access. However, FT Strategies stopped at just reading the robots.txt files. I wanted to actually test them and expand analysis across 1 million sites and many industries.
How I did this
I started with Tranco’s list of the top million domains and fetched the robots.txt from 990,825 of them. For each file I recorded which crawlers it excludes, at which paths, and whether it was copied from somewhere else.
Then I took 1,000 sites, looked at what they intended to restrict and crawled as a browser, as a generic crawler, and as GPTBot and comparing what the server did with what the file said. Where a site blocked a specific path rather than the whole site, I tested that exact path. If the page didn’t exist for a normal browser, I threw the case out rather than score it.
The web says less than you would think
Start with the most basic fact: most sites have nothing to say. Of the 589,347 sites that returned a valid robots.txt file, 108,421, about 18%, block at least one AI crawler at the root. Across all 741,498 policy-observable domains, that is 14.6%. Another 46,379 block only below the root, so 154,800 block AI somewhere. The remaining 586,698 show no AI blocking: 434,547 served a valid file that allows AI, 106,871 had no robots.txt, and 45,280 served a malformed file. The other 249,327 domains are outside the observable set.
The spread between industries is the first real signal. News media declares restrictions at 25.7%, roughly three times the rate of software, e-commerce, education, or government. I was expecting a wave of companies protecting themselves from AI, and there is, but it seems concentrated in one industry.
The obvious objection is that media sites are simply the popular sites, and popular sites have lawyers. But the rate does not rise steadily with popularity. It ranges from 12.3% to 18.9% across the ten bands, with a bump in the 400,000–600,000 range and no consistent gradient from the top hundred thousand down to the millionth. Being popular does not make a site more likely to write rules.
The media effect is not a size artifact either. Within every size band, media stays far above everything else. But size has its own independent effect. Smaller companies block more: 11.3% for companies with 1–10 employees, falling to about 6.0% for companies with 10,001 or more, and that downward trend holds within each traffic band. Both what kind of company it is and how large it is matter. Popularity does not.
When they do restrict, they restrict a lot
AI companies split their crawlers in two. There are training crawlers, which collect material to build models (ex. GPTBot, CCBot, ClaudeBot). And there are retrieval crawlers, which fetch a page because a person just asked a question (ex. OAI-SearchBot, ChatGPT-User, PerplexityBot). Restricting the first can help protect your IP. Restricting the second can cost you page views and revenue.
I expected a clean pattern: block the training bots and allow the retrieval bots. What I found was mostly all or nothing. Of the 108,421 sites that block at least one AI crawler at the root, 66,740 (62%) block citation as well, 41,352 (38%) block training only, and 329 (0.3%) do the reverse: they block citation while allowing training. The reversal is rare, but not zero.
The real split is between sites that draw a line and sites that block the whole category. Most blockers choose the whole category.
The ranking tells a consistent story at the margin. The top of the list is training crawlers: GPTBot at 11.9%, ClaudeBot at 11.1%, Amazonbot at 10.9%, Google-Extended at 10.2%. Retrieval crawlers sit lower on average, but the ranges overlap: ChatGPT-User is the highest retrieval crawler at 8.4%, above Bytespider, ImagesiftBot, and Timpibot, while Applebot-Extended is the lowest training crawler at 6.9%. Sites are more willing to block training than retrieval, but the preference is not absolute.
But that preference is not the same as a two-tier policy. The per-site numbers are blunt. 62% of blockers also block citation. The “protect IP, stay discoverable” strategy is real, but it is a minority position, not the industry standard.
The most common repeated AI policy on the web is a Cloudflare template
This is a number I find really interesting (maybe biasedly because I work at Cloudflare lol). The most repeated robots.txt among the top million sites is a block of Cloudflare boilerplate which includes policies like search allowed and AI training denied. It appears on 29,041 domains, in two near-identical versions, and it represents about 3.9% of the 741,498 policy-observable domains.
That is one company’s default, copied onto tens of thousands of sites. The next most common files are unremarkable: “allow everything”, “block everything”, a WordPress default, an old template from a decade ago. How many of these were intentionally set up and how many are boilerplates the company doesn’t even know is there?
Meanwhile, 63% of policy-observable files are unique to a single site. That is not a contradiction. Cloudflare’s file is the most repeated single file at 29,041 copies, but that is still only 3.9% of the 741,498 policy-observable domains. 63% of those files appear exactly once.
All of this matters for what comes next. You cannot enforce a policy you didn’t know you had but with 63% of these files being unique, can it be assumed these are intentional?
The most common AI block mostly isn’t enforced at the User-Agent level
| Server blocks | Server serves | Sample | |
|---|---|---|---|
| Declared block (356) | 77 | 279 | path-aware |
| Declared open (697) | 114 | 583 | root test |
Of 356 testable sites that disallowed GPTBot and were tested at the exact blocked path, 279 (78%) served it anyway. In a separate root-test sample of 697 sites that declared no AI restriction, 114 (16%) blocked it anyway. The GPTBot non-enforcement rate in the path-aware sample held across every industry, 67–91%. The two samples used different methods, so I don't pool them into a single contradiction rate. Sites that blocked the browser itself are excluded.
This is the part I did not expect.
Robots.txt is voluntary. Nothing about the protocol requires a server to honor it. It’s a statement of intent and policy, not an actual security protection.
I ran two checks. The path-aware sample covered 1,000 domains. 525 declared an AI block; 356 of those were testable at a loadable GPTBot-blocked path. A separate root-test sample tested 697 sites that declared no AI restriction. Sites with no file, a missing path, a blocked browser, or an unverifiable result were excluded from the relevant check.
Of those 356 testable sites whose file disallowed GPTBot, tested at the exact path they blocked, with a browser and a generic crawler as controls, 78% served the content to a request identifying as GPTBot anyway. Only 12% blocked GPTBot while allowing the browser and generic crawler. Another 10% blocked every bot that wasn’t a browser, so the GPTBot block was incidental.
It runs the other way too. Of the 697 sites in the root-test sample that declared no AI restriction at all, 16% blocked an AI crawler anyway.
There is a detail that makes this worse and better at the same time. 30% of the 154,800 sites that block AI somewhere (46,379) do it only below the homepage. A site leaves its front page open and fences off their archive, or images or whatever they consider important.
I tested the actual paths. The GPTBot non-enforcement rate got worse, not better: when I requested a path a site wanted locked, it still opened.
Collection disclaimers
None of this is a complete picture, and I want to be specific about where it breaks.
63,147 sites of the 990,825 fetched, or 6.4%, refuse to serve robots.txt at all, returning a block instead of a file. Another 173,539 were unreachable. In the path-aware enforcement sample, 85 of 1,000 domains (8.5%) could not be verified because the browser itself was blocked; the separate root-test sample had 220 of 1,000 (22%). I threw those out rather than count them as enforcement.
And the enforcement test measures User-Agent behaviour, not real crawler identity. GPTBot was the test crawler because it is the most-blocked AI crawler and gives the largest sample. The tradeoff is IP verification: OpenAI publishes its crawler IPs, so a request with the GPTBot User-Agent from a random IP is not the real GPTBot. A cleaner follow-up would test the exact bot each site blocks, including a crawler that does not verify by IP.
Hypotheses I couldn’t prove
A few things looked promising and didn’t survive contact with the data.
Licensing deals. I had hoped robots.txt files would reveal contractual industry patterns. A deal would show up as a single-vendor exception: block everyone except the company you signed with. The full sample has 26,422 files that explicitly allow exactly one AI vendor while blocking at least four training crawlers. But the pattern is not a clean deal signal. 23,684 of them exempt ByteDance’s crawler while blocking the other training bots, spread across 23,672 distinct files. That looks more like a mass-produced SEO pattern than a set of negotiated contracts. The signal is intriguing, but it is swamped by template spam, so I can’t use it to identify deals.
Jurisdiction. I expected a European split, given the TDM opt-out in EU law. Among countries with at least 2,000 policy-observable domains, the rate ranged from 5.7% in Turkey to 13.1% in Poland. The US was 10.6%, Germany 9.8%, France 10.4%, the Netherlands 11.1%, and Spain 12.4%. No EU effect, no US effect.
Enforcement by industry. News media contradicts its own robots.txt most often (35.5%), and software sits near the least (10.6%, roughly tied with professional services at 10.0%), but the intervals overlap too much to claim it. It’s a suggestion, not a finding. A deeper follow-up analysis is needed.
What this means
Robots.txt is a strange kind of document. It isn’t a true security protection. It puts a company’s position on the record, where partners, lawyers, regulators, and AI agents can read it.
With the rise in awareness about AI, I expected clear and intentional restriction patterns, but this turned out not to be the case. I also expected to see a majority of sites with strict enforcement controls, but this was also disproven.
30% of the blocking I found was invisible to a homepage check. In the path-aware sample, roughly three in four declared GPTBot blocks weren’t enforced at the User-Agent level. And the most common robots.txt file on the web belonged to Cloudflare.
If anything, this shows that while some industries and AI power users like me may be hyper-focused on AI, most companies are still playing catch up.
Method: 990,825 domains from the Tranco top-1M list, October 2026. Industry labels from a commercial company database, checked against 250 hand-labelled domains (80% agreement). Enforcement test: two samples. The path-aware sample covered 1,000 domains, including 525 declared blockers, of which 356 were testable at a loadable path and tested at the exact path they block, with browser and generic-crawler controls. The root-test sample covered a separate 1,000 domains, including 697 declared-open sites tested at the root. Sites that blocked the browser itself were excluded (85 of 1,000 path-aware; 220 of 1,000 root-test). The two samples are not pooled. Enforcement is measured by User-Agent only, not verified crawler identity. GPTBot was the only AI crawler tested.