Skip to content
GEOoptimised

Original data · 50 firms · Fetched 2026-07-14

We read the robots.txt of the top 50 UK accountancy firms. Almost none say anything about AI crawlers.

For each firm in a published fee-income ranking we fetched /robots.txt and /llms.txt ourselves, saved every raw response, and parsed each file for the 12 AI user-agent tokens that OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple and Common Crawl publish. This is what the files say, and, just as important, what they do not say.

3/43

readable robots.txt files name no AI crawler at all. Silence is the default posture.

3/50

holds an AI crawler to stricter rules than it gives Googlebot: the only reading of robots.txt that counts as an AI-specific restriction.

12/50

serve a genuine llms.txt. 0 more return a 200 that is really an HTML soft-404.

n = 50 firms · 7 indeterminate: behind bot protection or unreachable, so their stance cannot be read · firm list from a published ranking (source) · every raw file retained

What the files actually show

Silence, not blocking, is the norm

Of 43 firms whose robots.txt we could read, 3 (7%) mention none of the 12 AI tokens. 40 name an AI crawler at all, and in 3 of 50 files an AI crawler is held to stricter rules than Googlebot. The dominant stance is not a decision either way.

Naming an AI crawler is not restricting it

Most firms that name AI tokens treat them exactly like mainstream crawlers. In one file the AI tokens sit in the same directive group as Googlebot and Bingbot, so the policy applies to all named crawlers alike; in others the AI tokens get Allow: / with only the same housekeeping exclusions as the default group, which is explicit permission. Per-firm notes appear under the table.

Bot protection reads more like a gate than robots.txt does

7 of 50 firms answered our request for a plain text file with a WAF challenge (Cloudflare, Vercel, Azure, Incapsula) or did not respond. That gate can turn away a compliant AI fetcher regardless of what the robots.txt would have said, and it is not a robots directive.

The search and on-demand tokens are untouched

Where firms name AI tokens at all, they name the training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot). The tokens that power live answers, OAI-SearchBot, ChatGPT-User, PerplexityBot and Perplexity-User, are almost entirely unaddressed. A rule on a training crawler does not touch retrieval.

llms.txt is barely used, and easy to fake by accident

12 firms serve a genuine llms.txt; 32 return a clean 404. 0 return HTTP 200 for /llms.txt but the body is an ordinary HTML page, a soft-404 that a careless audit would count as present. We do not.

Per-crawler summary

Counted across the 43 readable robots.txt files only. These counts describe the files' syntax shape, token by token; they are not a judgement that a firm restricts AI. "Not mentioned" is not a block; it means the token does not appear, so the site's general rules (or none) apply to it. And a token that is named is usually being permitted, not limited; the stance column in the full table below carries the actual judgement, made against what each file gives Googlebot.

AI crawler directive counts across readable robots.txt files
AI user-agent Allow Block Partial Not mentioned Named at all
GPTBot 9 4 27 3 40
OAI-SearchBot 11 1 28 3 40
ChatGPT-User 13 1 26 3 40
ClaudeBot 8 4 28 3 40
Claude-Web 10 1 29 3 40
anthropic-ai 11 1 28 3 40
PerplexityBot 12 1 27 3 40
Perplexity-User 11 1 28 3 40
Google-Extended 9 4 27 3 40
Bingbot 11 1 28 3 40
CCBot 8 4 28 3 40
Applebot-Extended 7 4 29 3 40

What a block actually prevents

A blocked user-agent token restricts one specific fetch. It does not make a brand "invisible in AI". Models retain what they were trained on before the rule existed, retrieval and training use different tokens, and one assistant often rides another's index. Precise, per-token, this is what each directive in the table above does and does not do.

  • GPTBot is OpenAI's crawler for model training and for GPTBot-driven browsing. Blocking it asks OpenAI not to collect the site for those uses. It does not remove the firm from ChatGPT's existing knowledge, and it does not stop ChatGPT's search feature, which uses OAI-SearchBot.
  • OAI-SearchBot powers results in ChatGPT search. A firm that blocks GPTBot but leaves OAI-SearchBot unmentioned, which is the common pattern here, can still be fetched and cited by ChatGPT search.
  • ChatGPT-User is the live fetch made when a user asks ChatGPT to open a specific link. Blocking it affects on-demand visits, not training.
  • ClaudeBot is Anthropic's active crawler. Claude-Web and anthropic-ai are older tokens Anthropic has published; naming ClaudeBot alone is the current control.
  • PerplexityBot is Perplexity's index crawler; Perplexity-User is its on-demand fetch when a user's question requires opening a page. They are separate decisions.
  • Google-Extended controls use of content for Google's Gemini and Vertex AI, including grounding and training. It does not affect Google Search indexing or eligibility for AI Overviews, both of which follow Googlebot. Blocking Google-Extended does not remove a firm from AI Overviews.
  • Bingbot is Bing's search crawler. Microsoft Copilot draws on the Bing index, so a Bingbot rule reaches Copilot indirectly; there is no separate Copilot opt-out token to set.
  • CCBot is Common Crawl, an open corpus that many models train from at one remove. Blocking it limits that indirect route, not any single assistant.
  • Applebot-Extended opts content out of Apple Intelligence training. Apple's search crawler, Applebot, is a separate token and is not in scope here.

And the rule that sits under all of them: robots.txt is a request, not enforcement. It expresses a preference that a crawler operator may honour or ignore, and honouring it is voluntary and varies by operator. Nothing on this page measures traffic, lost visibility or citation outcomes, and no causal claim of that kind is made. We report the directives as fetched, and only that.

Every firm, every token

All 50 firms in ranked order. Click a column header to sort. The per-token cells describe the file's syntax shape only, not a judgement that the firm restricts AI: allow (named, nothing disallowed), block (named, full disallow), part (named with path rules or an allowlist), n/m (not mentioned), n/a (file unreadable). The judgement is the AI stance column: whether any AI token is held to stricter effective rules than the same file gives Googlebot.

AI crawler directives for all 50 firms
# Firm robots AI stance GPTBotOAI-SearchBotChatGPT-UserClaudeBotClaude-Webanthropic-aiPerplexityBotPerplexity-UserGoogle-ExtendedBingbotCCBotApplebot-Extended llms
1 PwC pwc.co.uk WAF n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a WAF
2 Deloitte deloitte.com read none part part part part part part part part part part part part 404
3 EY ey.com read none allow part allow part part allow allow allow allow part allow part 404
4 KPMG kpmg.com read none part part part part part part part part part part part part yes
5 BDO bdo.co.uk read none part part part part part part part part part part part part 404
6 Grant Thornton UK grantthornton.co.uk read none part part part part part part part part part part part part yes
7 RSM rsmuk.com read none part part part part part part part part part part part part 404
8 Azets Group azets.co.uk read none allow allow allow allow allow allow allow allow allow allow allow allow yes
9 Forvis Mazars forvismazars.com read none part allow allow allow part part allow part allow part part part 404
10 Moore UK moore.co.uk read none part part part part part part part part part part part part 404
11 MHA mha.co.uk read none part part part part part part part part part part part part 404
12 Sumer sumer.co.uk read none part part part part part part part part part part part part 404
13 PKF UKI pkf-uki.com down n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a down
14 Crowe UK crowe.com read none part part part part part part part part part part part part 404
15 Saffery saffery.com read none allow allow allow allow allow allow allow allow allow allow allow allow 404
16 Xeinadin xeinadin.com read none part part part part part part part part part part part part 404
17 Cooper Parry cooperparry.com read none part part part part part part part part part part part part yes
18 TC Group tc-group.com read none allow allow allow allow allow allow allow allow allow allow allow allow yes
19 ETL GLOBAL etl-global.com read none allow allow allow allow allow allow allow allow allow allow allow allow 404
20 Begbies Traynor Group begbies-traynorgroup.com read none part part part part part part part part part part part part 404
21 FRP Advisory frpadvisory.com read none part part part part part part part part part part part part 404
22 Dains dains.com read AI-specific block allow allow block allow allow allow allow block allow block block 404
23 AAB aab.uk down n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a yes
24 Menzies menzies.co.uk read none part part part part part part part part part part part part 404
25 K3 Capital Group k3capitalgroup.co.uk down n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a down
26 Kinbrook Group kinbrookgroup.com read none n/m n/m n/m n/m n/m n/m n/m n/m n/m n/m n/m n/m yes
27 Buzzacott buzzacott.co.uk read none allow allow allow allow allow allow allow allow allow allow allow allow yes
28 UHY Hacker Young uhy-uk.com read none part part part part part part part part part part part part 404
29 HaysMac haysmac.com read none allow allow allow allow allow allow allow allow allow allow allow allow 404
30 TaxAssist Accountants taxassist.co.uk read none n/m n/m n/m n/m n/m n/m n/m n/m n/m n/m n/m n/m 404
31 Shaw Gibbs shawgibbs.com down n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a down
32 DJH djh.co.uk read none block block block block block block block block block block block block 404
33 Gravita gravita.com read AI-specific block allow allow block allow allow allow allow block allow block block 404
34 Affinia affinia.co.uk read none part part part part part part part part part part part part 404
35 Hazlewoods hazlewoods.co.uk read none n/m n/m n/m n/m n/m n/m n/m n/m n/m n/m n/m n/m yes
36 Kreston Reeves krestonreeves.com read none allow allow allow allow allow allow allow allow allow allow allow allow yes
37 BKL bkl.co.uk read none part part part part part part part part part part part part 404
38 Armstrong Watson armstrongwatson.co.uk read none part part part part part part part part part part part part 404
39 Bishop Fleming bishopfleming.co.uk read none part part part part part part part part part part part part 404
40 Streets streetsweb.co.uk read AI-specific block allow allow block allow allow allow allow block allow block block 404
41 Haines Watts hwca.com read none part part part part part part part part part part part part 404
42 Price Bailey pricebailey.co.uk read none part part part part part part part part part part part part 404
43 BHP bhp.co.uk read none part part part part part part part part part part part part 404
44 James Cowper Kreston jamescowperkreston.co.uk read none part part part part part part part part part part part part 404
45 Harris & Trotter harris-and-trotter.co.uk down n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a down
46 Lovewell Blake lovewell-blake.co.uk read none part part part part part part part part part part part part 404
47 Mercer & Hole mercerhole.co.uk WAF n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a n/a WAF
48 Larking Gowen larking-gowen.co.uk read none part part part part part part part part part part part part 404
49 Hillier Hopkins hillierhopkins.co.uk read none allow part allow part part part part part part allow part part yes
50 Wilson Partners wilsonpartners.co.uk read none part part part part part part part part part part part part yes

Notes on every firm that names an AI token

  • Deloitte no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • EY no AI-specific restriction AI tokens named in this file carry rules identical to or more permissive than Googlebot's, or share a group with it.
  • KPMG no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • BDO no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Grant Thornton UK no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • RSM no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Azets Group no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Forvis Mazars no AI-specific restriction AI tokens named in this file carry rules identical to or more permissive than Googlebot's, or share a group with it.
  • Moore UK no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • MHA no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Sumer no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Crowe UK no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Saffery no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Xeinadin no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Cooper Parry no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • TC Group no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • ETL GLOBAL no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Begbies Traynor Group no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • FRP Advisory no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Dains AI-specific restriction GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended are fully disallowed while Googlebot is not equally disallowed in the same file.
  • Menzies no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Buzzacott no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • UHY Hacker Young no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • HaysMac no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • DJH no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Gravita AI-specific restriction GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended are fully disallowed while Googlebot is not equally disallowed in the same file.
  • Affinia no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Kreston Reeves no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • BKL no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Armstrong Watson no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Bishop Fleming no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Streets AI-specific restriction GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended are fully disallowed while Googlebot is not equally disallowed in the same file.
  • Haines Watts no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Price Bailey no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • BHP no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • James Cowper Kreston no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Lovewell Blake no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Larking Gowen no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.
  • Hillier Hopkins no AI-specific restriction AI tokens named in this file carry rules identical to or more permissive than Googlebot's, or share a group with it.
  • Wilson Partners no AI-specific restriction No AI token is named; AI crawlers inherit the same generic rules as every other unnamed crawler.

The 7 firms marked n/a (PwC, PKF UKI, AAB, K3 Capital Group, Shaw Gibbs, Harris & Trotter, Mercer & Hole) answered our request with a bot-protection challenge or did not respond, so their robots.txt stance is unknown, not inferred.

Raw responses for every firm are retained at src/data/ai-crawler-index/raw-accountancy/ in the site repository, and the parsed dataset is src/data/ai-crawler-index-accountancy.json.

Other sector indices

The same methodology applied to three more UK sectors: law firms, fintech companies and private healthcare providers. Want to check one domain instead of fifty? Try the free AI crawler check.

Methodology

The firm list

The population is the top 50 firms from Accountancy Age (in partnership with Intuit QuickBooks), "Top 50+50 Accountancy Firms 2025": the top 50 of the Accountancy Age Top 50+50 2025, ranked by total UK fee income for the last financial year based on voluntary submission of data. We fetched that ranking on 2026-07-14. Rankings are based on firms voluntary submissions to Accountancy Age, not an independent audit. Firm-to-domain mapping was done by inspection; a wrong domain would surface below as unreachable, never as a false directive. The list orders firms by UK fee income; it is not a ranking of AI readiness, and nothing here should be read as one.

The fetch

On 2026-07-14 we requested https://<domain>/robots.txt and https://<domain>/llms.txt for each firm with curl, following redirects, and saved the exact response body for every firm as provenance. Three firms answered only on their www host, so we recorded the effective URL. Each firm's primary domain was mapped by inspection; a wrong domain would appear below as unreachable, never as a false directive.

Reading a file, and refusing to read a fake one

A response counts as a readable robots.txt only when its body is genuinely a robots file, that is, plain text carrying User-agent, Allow or Disallow lines. Several firms returned HTTP 200, 403 or 429 whose body was an HTML bot-protection challenge (Vercel, Cloudflare, Azure WAF, Incapsula). We record those as unreadable and never parse them for directives, because a challenge page is not a policy. The same discipline applies to llms.txt: a 200 whose body is an ordinary HTML page is a soft-404 and is marked as such, not counted as a genuine file.

How each token was classified

robots.txt was parsed with standard user-agent grouping: consecutive User-agent lines share the rules that follow them, until the next group. For each of the 12 AI tokens we then recorded:

  • allow: the token is named and nothing disallows it.
  • block: the token is named with Disallow: / and no Allow override, a full-site block request.
  • partial: the token is named with some paths disallowed, or an allowlist pattern (Disallow: / alongside Allow: rules that re-permit specific sections).
  • not mentioned: the token does not appear. The catch-all User-agent: * group, if any, still applies to it, but no AI-specific rule was set. This is not a block.

Those four labels describe syntax, deliberately. A token can be "partial" because it receives an allowlist, or because it shares the site's ordinary housekeeping exclusions; neither is evidence that the firm singled out AI. The judgement lives in a separate, derived field.

The stance judgement: stricter than Googlebot, or not

For each firm we derived one field, the AI-specific stance, computed from the archived raw files. The benchmark is Googlebot, the mainstream crawler every firm plainly wants. Following RFC 9309 semantics, each token obeys the group or groups naming it exactly; a token that is not named falls back to the User-agent: * group. We then ask one question: is any AI token's effective rule set clearly stricter than the same file's effective rule set for Googlebot? Bingbot sits in the per-token table because Copilot rides its index, but it is a mainstream search crawler, so it is not part of the AI-specific test.

  • Restricts AI specifically: at least one AI token is fully disallowed while Googlebot is not equally disallowed. This is the only pattern we label a restriction.
  • No AI-specific restriction: the AI tokens are unnamed (they inherit the same generic rules as every other unnamed crawler), or named with rules identical to or more permissive than Googlebot's. A restriction shared equally with Googlebot, or inherited from *, is a general crawling policy, not an AI decision.
  • Indeterminate: we could not read the file, or the differences were neither clearly stricter nor clearly equal. Ambiguity is graded conservatively, never as a restriction.

Limitations

  • This is a single snapshot on 2026-07-14. robots.txt and llms.txt change; the retained raw files fix what we saw on that date.
  • The population is a fee-income ranking based on voluntary submissions, not an independent audit, and not a random or complete sample of UK accountancy firms. It says nothing about firms outside the top 50.
  • Firms behind bot protection (7 of 50) have an unreadable stance here. That is a finding about access, not evidence that they block or permit any crawler.
  • robots.txt and llms.txt are voluntary requests. Compliance is at the operator's discretion, so a directive is a stated preference, not an enforced outcome, and we measure no outcome.
  • We checked 12 named AI tokens. Other agents exist (for example Amazonbot, Bytespider, meta-externalagent, which appear in some of these files); they are out of scope and not counted.

The dataset behind this page is public: src/data/ai-crawler-index-accountancy.json plus one raw file per firm. Anyone can refetch a domain and check our reading against the saved response. If a firm's file has changed since 2026-07-14, that is expected, and a fresh fetch is the way to confirm it. See also how we grade and the reproduction study.