We read the robots.txt of every top-50 UK law firm. Almost none say anything about AI crawlers.

Methodology

The firm list

The population is the top 50 firms from Lawyer Mag (lawyermag.co.uk), "Law Firm Rankings: 100 Leading Law Firms in UK in 2025": the top 100 UK law firms ranked primarily by revenue for 2025, of which we take the top 50. We fetched that ranking on 2026-07-14. This is a secondary compilation, not a primary revenue audit. The Lawyer UK 200 and Legal Business LB100 are the primary rankings but are paywalled. Firm-to-domain mapping was done by inspection; a wrong domain would surface below as unreachable, not as a false directive. The two primary rankings in this field, The Lawyer UK 200 and the Legal Business LB100, are the stronger sources but sit behind paywalls, so we used a freely accessible published compilation and state that plainly. The list orders firms by revenue; it is not a ranking of AI readiness, and nothing here should be read as one.

The fetch

On 2026-07-14 we requested https://<domain>/robots.txt and https://<domain>/llms.txt for each firm with curl, following redirects, and saved the exact response body for every firm as provenance. Three firms answered only on their www host, so we recorded the effective URL. Each firm's primary domain was mapped by inspection; a wrong domain would appear below as unreachable, never as a false directive.

Reading a file, and refusing to read a fake one

A response counts as a readable robots.txt only when its body is genuinely a robots file, that is, plain text carrying User-agent, Allow or Disallow lines. Several firms returned HTTP 200, 403 or 429 whose body was an HTML bot-protection challenge (Vercel, Cloudflare, Azure WAF, Incapsula). We record those as unreadable and never parse them for directives, because a challenge page is not a policy. The same discipline applies to llms.txt: a 200 whose body is an ordinary HTML page is a soft-404 and is marked as such, not counted as a genuine file.

How each token was classified

robots.txt was parsed with standard user-agent grouping: consecutive User-agent lines share the rules that follow them, until the next group. For each of the 12 AI tokens we then recorded:

• allow: the token is named and nothing disallows it.
• block: the token is named with Disallow: / and no Allow override, a full-site block request.
• partial: the token is named with some paths disallowed, or an allowlist pattern (Disallow: / alongside Allow: rules that re-permit specific sections).
• not mentioned: the token does not appear. The catch-all User-agent: * group, if any, still applies to it, but no AI-specific rule was set. This is not a block.

Those four labels describe syntax, deliberately. A token can be "partial" because it receives an allowlist, or because it shares the site's ordinary housekeeping exclusions; neither is evidence that the firm singled out AI. The judgement lives in a separate, derived field.

The stance judgement: stricter than Googlebot, or not

For each firm we derived one field, the AI-specific stance, computed from the archived raw files. The benchmark is Googlebot, the mainstream crawler every firm plainly wants. Following RFC 9309 semantics, each token obeys the group or groups naming it exactly; a token that is not named falls back to the User-agent: * group. We then ask one question: is any AI token's effective rule set clearly stricter than the same file's effective rule set for Googlebot? Bingbot sits in the per-token table because Copilot rides its index, but it is a mainstream search crawler, so it is not part of the AI-specific test.

• Restricts AI specifically: at least one AI token is fully disallowed while Googlebot is not equally disallowed. This is the only pattern we label a restriction.
• No AI-specific restriction: the AI tokens are unnamed (they inherit the same generic rules as every other unnamed crawler), or named with rules identical to or more permissive than Googlebot's. A restriction shared equally with Googlebot, or inherited from *, is a general crawling policy, not an AI decision.
• Indeterminate: we could not read the file, or the differences were neither clearly stricter nor clearly equal. Ambiguity is graded conservatively, never as a restriction.

Limitations

• This is a single snapshot on 2026-07-14. robots.txt and llms.txt change; the retained raw files fix what we saw on that date.
• The population is a revenue ranking, not a random or complete sample of UK firms. It says nothing about firms outside the top 50.
• Firms behind bot protection (5 of 50) have an unreadable stance here. That is a finding about access, not evidence that they block or permit any crawler.
• robots.txt and llms.txt are voluntary requests. Compliance is at the operator's discretion, so a directive is a stated preference, not an enforced outcome, and we measure no outcome.
• We checked 12 named AI tokens. Other agents exist (for example Amazonbot, Bytespider, meta-externalagent, which appear in some of these files); they are out of scope and not counted.

The dataset behind this page is public: src/data/ai-crawler-index.json plus one raw file per firm. Anyone can refetch a domain and check our reading against the saved response. If a firm's file has changed since 2026-07-14, that is expected, and a fresh fetch is the way to confirm it. See also how we grade and the reproduction study.