Why robots.txt Is Only Part of the AI Readability Problem

What robots.txt does and does not do

robots.txt is a public, voluntary instruction file at the root of your domain. Major AI operators publish their user agents and state that they honor it. It is not a security control: it does not authenticate, it does not enforce, and it is readable by anyone. Use it to express intent about crawling, not to protect anything.

The user agents that matter

• GPTBot — OpenAI's indexing and training crawler.
• OAI-SearchBot — OpenAI's search index crawler, separate from training.
• ChatGPT-User — live fetch triggered by a user's request in the moment.
• ClaudeBot and Claude-User — Anthropic's crawler and live fetcher.
• PerplexityBot — Perplexity's index crawler.
• Google-Extended — controls Gemini and AI training use, not classic Google indexing.
• Applebot-Extended, Bytespider, CCBot, meta-externalagent — additional agents worth an explicit decision.

These are distinct controls. Allowing Googlebot says nothing about Google-Extended; allowing GPTBot says nothing about the live fetcher that runs when a buyer asks ChatGPT for options.

The three lines that cost the most

• User-agent: * / Disallow: / — a staging configuration promoted to production. Everything stops.
• User-agent: GPTBot / Disallow: / — added during the 2023 opt-out wave and never revisited, usually by someone who has since left.
• Disallowing the directories where your proof lives — /blog, /case-studies, /learn — which are precisely the pages that earn citations.

A noindex meta tag or header is a related and equally decisive signal. Our AI Machine Readability Checker treats these as blocking issues: a site-wide disallow or a noindex limits the total score no matter how strong the rest of the page is, because nothing further downstream can compensate.

A reasonable default for a business that wants to be recommended

Allow the indexing and live-retrieval agents, declare your sitemap, and disallow only what is genuinely private or duplicative — admin paths, internal search results, parameterized URLs. Publishers who monetize content directly have a legitimate reason to restrict training agents while still allowing search and live retrieval; that nuance is why the agents are separable.

Always include your sitemap declaration. It is a discovery signal that costs one line:Sitemap: https://example.com/sitemap.xml.

Auditing yours

1. Open /robots.txt on your live domain and read every line, including inherited ones.
2. Check for noindex in meta tags and X-Robots-Tag response headers on key pages.
3. Confirm your CDN or WAF is not blocking agents that robots.txt allows — the two disagree more often than teams expect.
4. Verify arrivals in server logs, not just permissions. See How AI Crawlers Access Websites.
5. Re-run the free checker after any change.

What robots.txt cannot tell you

Even with every AI agent allowed, several things can still stand between a request and usable content:

• A CDN, WAF, or bot-management rule that returns a challenge page rather than the page.
• A noindex meta tag or X-Robots-Tag header that contradicts the permission.
• Content that only renders client-side, leaving thin server-rendered HTML.
• Copy locked inside images, embeds, or scripts, where no text can be extracted.
• Canonical tags pointing somewhere other than the page itself, muddying which URL the content belongs to.

Those belong to the content and signal layers, not the permissions layer. The practical checklist for all five is in What Makes a Website Machine-Readable for AI Search.

Permission is the beginning

Unblocking crawlers does not produce recommendations. It makes them possible. Once access is clean, the work moves to interpretation and evidence — see Can AI Actually Read Your Website? for the full sequence.