How the FTSE 100 handles AI crawlers

What we checked

For each FTSE 100 company we took its main corporate website, the one that hosts investor relations, and fetched two files: robots.txt and llms.txt. Three domains (Coca-Cola Europacific Partners, JD Sports and Segro) didn't resolve on the day, which leaves 97 sites.

We then worked out whether each of 13 AI user agents may fetch the homepage, with Googlebot and Bingbot as a control. The rules come from the robots.txt standard, RFC 9309: a crawler obeys the group that names it, falls back to the * group if none does, and the most specific matching rule wins. The method section covers the detail and the limits.

Most of the FTSE 100 has no AI crawler policy

91 of the 97 robots.txt files don't name a single AI crawler. Most were written for search engines, with rules for admin folders, internal search pages and CMS paths. A crawler that isn't named follows the * group, so on these sites GPTBot, ClaudeBot, PerplexityBot and the rest get the same access as Google.

That's a reasonable outcome if it's what the company wants. The point is that for most of these businesses nobody appears to have decided. The same file controls whether ChatGPT can cite your annual results and whether your pages can be used to train the next model, and in 94% of cases it says nothing about either.

The FTSE 100 is also more open than the web's busiest sites. An academic study of AI crawler blocking found that 12 to 14% of the 5,000 most popular websites restricted at least one AI crawler, using robots.txt data from 2022 to 2024. Among the FTSE 100 the figure is 4 of 97. The samples differ in date and make-up, since the most visited sites include many news publishers, so treat the comparison as a rough guide.

Six companies name AI crawlers

M&G and IMI are the only two that clearly separate AI search from AI training, which is how the AI companies' own documentation is set up. Two details are worth knowing:

• M&G's file blocks Apple-Extended. Apple's token is Applebot-Extended, so that rule matches nothing and Apple's training opt-out isn't applied.
• IMI's content signals express a preference rather than a block. Cloudflare's policy describes them as a way to state how content may be used after a crawler has accessed it. GPTBot and ClaudeBot aren't named in IMI's file, so robots.txt itself still lets them in.

WPP's pattern, one AI company's training crawler blocked and its competitors' allowed, usually means a rule was added for one crawler and not revisited as others appeared.

Search crawlers and training crawlers do different jobs

The main AI companies run separate crawlers for separate purposes, and each can be allowed or blocked on its own.

OpenAI's documentation says each setting is independent: you can allow OAI-SearchBot and block GPTBot, and sites that block OAI-SearchBot won't appear in ChatGPT search answers. Anthropic says blocking Claude-SearchBot may reduce a site's visibility in Claude's search results. Google says Google-Extended "does not impact a site's inclusion in Google Search" (Google). Perplexity says Perplexity-User generally ignores robots.txt, because a person asked for the page.

That leaves every company with two separate decisions: whether it wants to be found and cited in AI answers, and whether it's happy for its content to be used for training.

Robots.txt is only half the picture

Robots.txt states what a site asks crawlers to do. Whether a crawler actually gets through is up to the hosting, CDN and firewall, and that layer was busier than the files suggest.

When we requested robots.txt with Python's standard library, 52 of the 97 sites refused. With curl's default settings, 11 refused, including AstraZeneca, the company with the most welcoming AI policy in the index. Every one of those files opened normally in a web browser.

That doesn't mean ChatGPT or Claude are blocked from those sites. Many bot management services recognise and allow verified crawlers while refusing generic scripts, and those settings can't be seen from outside. It does mean robots.txt can say yes while the firewall says no. The only way to know is to check your CDN's bot settings and look for the AI user agents in your server logs. Cloudflare, for example, offers AI crawler controls that sit alongside robots.txt.

llms.txt: four companies so far

llms.txt is a proposal, first published by Jeremy Howard in September 2024, for a Markdown file that gives AI tools a short guide to a site's most useful pages. Four FTSE 100 companies publish one:

• Unilever and Weir follow the proposed format, opening with a heading and a one-line summary.
• Croda's is laid out as an entity profile: company name, headquarters, listing and a description.
• Whitbread's was generated by the Rank Math SEO plugin for WordPress and lists the site's pages.

There were two near misses. Reckitt's /llms.txt address serves a copy of its robots.txt, and GSK's returns an empty file. Neither tells an AI system anything.

Five housekeeping problems

• No robots.txt at all. Intertek and Rentokil Initial return a 404. Under RFC 9309, crawlers may then access everything.
• An error page in place of the file. Howdens returns its 404 page with a 200 "OK" status, and Smith & Nephew redirects robots.txt to an error page. Crawlers find no rules in either, so again everything is allowed.
• A redirect loop. Melrose's robots.txt redirects to itself. RFC 9309 lets crawlers treat a file as unavailable after more than five redirects, which also means no restrictions.
• Comments with no rules. Games Workshop's investor site serves only the explanatory text of Cloudflare's content signals policy, with no directives. Cloudflare says that's what sites on its free plan get by default when they haven't set their own robots.txt.
• A mistyped crawler name. M&G's Apple-Extended, covered above. Robots.txt fails silently: a misspelt token matches nothing, and nothing warns you.

None of these affects human visitors, and for most of these companies the practical result is that everything is open. Each one does suggest that nobody is looking after the file.

What to check on your own site

1. Read your robots.txt. Open yourdomain.com/robots.txt and look for AI crawler names. If there are none, every AI bot is allowed by default.
2. Make two decisions, not one. Decide separately on AI search (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and AI training (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot). If you want AI tools to recommend you, allowing the search crawlers is the place to start.
3. Use the exact tokens. Copy names from each company's documentation. A typo blocks nothing.
4. Check the firewall as well. Review your CDN or security plugin's bot settings, and search your server logs for the AI user agents to confirm they're getting successful responses.
5. Make sure the file exists and is plain text. A missing file or an HTML error page means you have no rules at all.
6. Review it every quarter. New AI crawlers keep appearing, and a policy written for one company's bot soon goes out of date.

Crawler access is one part of AI visibility. The AI Search Readiness Scorecard checks the rest, from content structure to brand clarity, in about three minutes.

How we ran the study

• Sample: the 100 FTSE 100 constituents after the quarterly review that took effect on 1 October 2026, as listed on Wikipedia's FTSE 100 Index page. For each company we used its main corporate website, which hosts investor relations, and not its consumer brand sites.
• Date: every file was fetched on 30 October 2026.
• Fetching: we requested each file with Python's standard library, retried with curl where that was refused, and used a normal web browser for the rest. We never presented ourselves as an AI or search crawler.
• Parsing: we applied RFC 9309 matching to decide whether each of 15 user agents (13 AI agents plus Googlebot and Bingbot) may fetch the homepage path /. A site "names an AI crawler" if its file has a user-agent line for at least one known AI agent.
• llms.txt: counted only where /llms.txt returned a non-empty text file, checked by hand.
• Excluded: Coca-Cola Europacific Partners, JD Sports and Segro, whose domains didn't resolve on the day.
• Limits: this is a one-day snapshot of corporate websites only. Robots.txt shows a site's stated rules, not what its firewall does, and we only checked access to the homepage path.

Download the full results: FTSE 100 AI crawler data, October 2026 (CSV). If you spot an error, or a company changes its file, email hello@gmkdigital.com and we'll update the data.

Frequently asked questions