‹Blog

Updated 2026-07-31

Which sites block AI crawlers? A 28-domain access benchmark

AI visibility starts with a simpler question than most teams ask: can the crawler read the page at all? If robots.txt blocks GPTBot, no amount of content optimization will make ChatGPT cite that domain — the page never enters the candidate pool. This benchmark applies the exact parser PilotCite ships in its free checker to 28 public domains, so the results are reproducible with the same tool anyone can run on their own URL.

What the benchmark tests

Date of observation: 2026-07-31. Sample: 28 domains across six categories — AI companies, major SaaS, media publishers, e-commerce, social platforms, and reference sites. The full domain list appears in the results table below and is published so anyone can rerun the test.

For each domain, two files were fetched over HTTPS: /robots.txt and /llms.txt. The robots.txt file was then parsed into per-user-agent groups — not a substring search. A crawler counts as blocked when its own named group or the wildcard (*) group disallows the root path (/) and no matching allow rule overrides it. This is the same logic implemented in PilotCite's site-audit checks: five named agents tested, each evaluated independently against the rules that apply to it.

The five crawlers tested are the ones the checker covers today:

• GPTBot — OpenAI's crawler, feeds ChatGPT search and browsing
• ClaudeBot — Anthropic's crawler, feeds Claude
• PerplexityBot — Perplexity's crawler
• Google-Extended — controls whether Google can use crawled content for AI training and AI Overviews
• Bingbot — Microsoft's crawler, feeds Bing and Copilot

Results: which domains block which crawlers

The table shows every domain in the sample. ✓ means the crawler is allowed; ✗ means robots.txt blocks it from the entire site. The llms.txt column shows whether the domain publishes one.

Which crawlers get blocked most

Bingbot is almost universally allowed — only Reddit blocks it among the 28 domains. The four AI-specific crawlers see meaningfully higher block rates:

The pattern is consistent with what each crawler is used for. Bingbot serves a general-purpose search engine that most sites want to appear in, so blocking it is rare. Google-Extended is slightly less blocked than the three AI-startup crawlers, likely because some sites distinguish between appearing in Google search (where Bingbot-equivalent Googlebot is usually still allowed) and contributing to AI training. The three pure AI crawlers — GPTBot, ClaudeBot, PerplexityBot — carry the highest block rates, reflecting the ongoing debate about whether AI companies should be allowed to train on or surface publisher content without compensation.

One interesting inconsistency: several domains block some AI crawlers but not others. The Guardian blocks ClaudeBot and PerplexityBot but allows GPTBot. Medium blocks GPTBot and ClaudeBot but allows PerplexityBot. These splits suggest that blocking decisions are often made per-crawler based on individual negotiations or policy stances, not as a blanket "no AI" position.

llms.txt adoption remains low

The llms.txt convention is an emerging standard where sites publish a plain-text file pointing AI systems at their most citable pages. In this sample, only 11 of 28 domains serve one — roughly 39%. The adopters skew toward developer-tooling companies (Stripe, Shopify, Vercel, Cloudflare, Notion, GitHub, Linear, Supabase) rather than media publishers, none of whom publish one.

This is not surprising for a convention that is still early in adoption, but it matters: a site that blocks AI crawlers in robots.txt and also skips llms.txt is making a deliberate choice to stay out of AI answers entirely. A site that allows the crawlers but has no llms.txt is leaving discovery to chance.

What this means for AI visibility

The practical takeaway is that crawler access is a binary gate that sits below all other AI visibility work. If your robots.txt blocks GPTBot, ChatGPT will not cite your pages — regardless of how well-structured your content is, how strong your entity consistency is, or how many other sites link to you. Fixing access is the cheapest visibility improvement available, because it costs nothing and unlocks everything downstream.

Three patterns emerged in the data that are worth checking against your own site:

• Unintentional wildcard blocks. A robots.txt that disallows / for User-agent: * blocks every crawler that does not have its own named group. If you added this rule for a specific crawler and forgot it also applies to the wildcard, you may have blocked more agents than intended.
• Inconsistent per-crawler policies. Several domains in this sample allow some AI crawlers but block others. If your team decided to allow GPTBot but the robots.txt was written before ClaudeBot existed, newer crawlers may be falling through to a restrictive wildcard rule.
• Missing llms.txt on an otherwise-open site. Allowing crawlers but not publishing llms.txt means AI systems can find your pages but have no curated pointer to your most important content. Adding one is a few minutes of work.

How to check your own site

The free AI visibility checker runs the same parser used in this benchmark on any URL you enter. It tests all five crawlers, checks for llms.txt, and also reviews extractability (JavaScript walls, heading structure, structured data) and identity (Organization markup). No sign-up required — one URL in, a prioritized fix list out.

For a full picture of whether AI platforms actually mention and cite your brand — not just whether they can crawl your site — PilotCite monitoring runs real prompts across eight AI platforms on a schedule and tracks mentions, citations, competitors, and sources over time.

Limitations

This benchmark tests one binary dimension of AI visibility: can the crawler fetch the page? It does not measure whether crawlers are actually visiting the page, how the page renders for AI systems, or whether the content earns citations. A domain that allows all five crawlers may still be invisible in AI answers because its content is not relevant, not authoritative, or not structured for extraction. The full picture requires monitoring actual AI answers, not just checking crawler permissions.

The sample is 28 domains chosen for category diversity, not statistical representativeness. The robots.txt rules were fetched once on 2026-07-31 and may have changed since. Domains that returned non-200 status codes for robots.txt (midjourney.com returned HTTP 403, stackoverflow.com returned HTTP 418) are treated as "all allowed" because no blocking rules were found — but a missing robots.txt is different from a permissive one.