How AI Crawlers Access Websites

Two mechanisms, two failure modes

Indexing and training crawlers — GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and others — visit on their own schedule, at scale, and build the corpus a model draws on. Being excluded here means long-term absence from a system's background knowledge of your market.

Live retrieval agents — the fetchers behind ChatGPT search, Perplexity, and browsing-enabled assistants — request your page in real time while composing an answer for a specific user. Being excluded here means absence from the exact moment a buyer is asking for a shortlist.

Most sites fail one and not the other, which is why a single "are we indexed?" check is not a diagnosis.

What a crawler actually does

1. Resolves DNS and requests the URL with its own user agent string.
2. Passes — or does not pass — your CDN, WAF, rate limiter, and bot-management rules.
3. Reads robots.txt and applies any directive addressed to it.
4. Receives HTML. Some execute JavaScript; many do not, or do so with strict time budgets.
5. Extracts text, headings, links, and structured data from whatever it received.
6. Follows internal links and sitemap entries to discover further pages.

Every step is a place to lose. The most common losses happen at steps two and four, and neither shows up in human analytics.

The quiet blockers

• Bot management defaults. Managed challenge rules on a CDN can serve a JavaScript interstitial to any unfamiliar agent. The crawler records an empty page, not a block.
• Aggressive rate limiting. A crawler that gets 429s on its first few requests may not return for weeks.
• Client-side rendering. If content only exists after hydration, a non-rendering fetcher sees an empty shell. Prerendered or server-rendered HTML solves this permanently.
• Geo or ASN blocking. Datacenter IP ranges are routinely restricted; crawlers live in them.
• Login and cookie walls. Anything gated is functionally invisible.
• Redirect chains and orphaned pages. Your best proof page is unreachable if nothing links to it and the sitemap omits it.

How to verify access rather than assume it

• Request your key pages with an AI crawler user agent and compare the response to what a browser sees.
• Check whether substantive text is present in the raw HTML with JavaScript disabled.
• Review server and CDN logs for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended hits — presence is the proof, robots.txt is only the permission.
• Confirm your sitemap is declared in robots.txt and contains every page you want cited.
• Run the free AI Machine Readability Checker, which flags blocking issues before anything else.

Ongoing observation of which crawlers actually arrive is what BotIQ™ — AI crawler accessibility and activity monitoring — was built for. Directives declare intent; logs record reality.

Access is necessary and insufficient

Once crawlers reliably reach and render your pages, the next constraint is interpretation, not access. That is covered in Being Crawlable Is Not the Same as Being Understood. For the permissions layer specifically, see Why robots.txt Is Only Part of the AI Readability Problem, and for the full sequence, the pillar: Can AI Actually Read Your Website?