How AI Crawlers Work: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Applebot-Extended

AI crawlers are bots that fetch web pages for three different jobs: collecting training data, building the search indexes that AI answers cite, and fetching a page when a user asks. Google-Extended and Applebot-Extended are control tokens, not crawlers. robots.txt controls most of these bots by name, most read raw HTML without running JavaScript, and your server logs show what they actually did.

This is part five of our series on how search and AI visibility work. The earlier parts explained how engines retrieve and cite pages. None of that happens if the right bot cannot fetch the page. This explainer is the reference: what each bot does according to its operator, how robots.txt is evaluated, where firewalls and rendering get in the way, and how to confirm it all in logs. For the policy decision, read should you block AI crawlers. For the silent failures, read is your WAF blocking AI crawlers and Cloudflare AI crawler settings.

The short version

1. Identify The bot announces itself with a user agent token that tells you its operator and job.
2. Check rules It reads robots.txt and applies the group that names it, or the wildcard group.
3. Pass the edge Your CDN or firewall lets it through, challenges it or blocks it.
4. Fetch It downloads the HTML your server returns, along with a status code.
5. Parse It reads the raw HTML; most AI crawlers do not run your JavaScript.
6. Use The content goes into a training dataset, a search index or a single live answer.

Three jobs, three kinds of bot

Training crawlers collect public pages that may be used to train future models. Blocking them keeps your content out of future training runs. It does not remove you from answers produced by search. Search crawlers build and refresh the index an engine retrieves from when it answers with citations. Blocking them removes your route into those answers. User-triggered agents fetch a specific page because a person asked a question that needs it, or pasted a link. They behave more like a browser acting for a user than like a crawler, and some operators say robots.txt may not apply to them.

The distinction matters because many sites still carry a single "block AI" rule that treats all three the same. For most B2B marketing sites, being retrievable by search crawlers and user agents is the point of the website.

AI crawler reference: what each bot does

Every row below comes from the operator's own documentation: OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl.

Two more belong in any AI crawler review even though they predate the category. Googlebot feeds Google Search, including AI Overviews and AI Mode. Bingbot feeds Bing, which powers Microsoft Copilot. OpenAI also runs OAI-AdsBot, which only visits pages submitted as ads on ChatGPT.

What happens when an AI crawler visits your site, step by step

Step 1: The bot gets a URL

Search and training crawlers discover URLs the way Googlebot does, from links and sitemaps. User-triggered agents get URLs from the conversation: a search result the assistant chose, or a link the user pasted. That is why a page can be fetched by ChatGPT-User the day it launches, before any crawler has found it.

Step 2: The bot reads robots.txt

The rules are standardized in RFC 9309, the Robots Exclusion Protocol. A compliant crawler looks for groups whose user-agent line matches its product token, case-insensitively. If one or more groups match, it combines and obeys them. Only if none match does it fall back to the * group. Within the rules it obeys, the most specific match wins, measured by the length of the path, and when an allow and a disallow match equally, allow is used.

The RFC also covers failure cases that catch real sites. A crawler should not rely on a cached robots.txt for more than 24 hours unless the file is unreachable. If robots.txt returns a 4xx error, the crawler may access anything. If it returns a 5xx server error, the crawler must assume everything is disallowed. And the RFC states that these rules are not a form of access authorization. Google makes the same point: robots.txt manages crawling and is not a mechanism for keeping a page private.

Step 3: The request passes your edge

Before your server sees the request, a CDN or web application firewall may inspect it. Bot protection often answers unfamiliar bots with a challenge page, which a crawler cannot solve. Anthropic says its bots will not try to bypass CAPTCHAs. The result is a site whose robots.txt says "welcome" while the firewall says "go away".

This became common quickly. Cloudflare announced that from July 1, 2025 new domains would be asked at sign-up whether to allow AI crawlers, with blocking as the default, and reported that more than one million customers had already turned on its AI crawler blocking. Anthropic also warns that blocking its IP addresses is unreliable and can stop its bots from reading your robots.txt at all.

Step 4: The bot fetches the page

The server returns HTML and a status code. Size and error handling matter. Googlebot documents that it processes the first 2MB of an HTML file; AI crawler operators do not publish equivalent limits, so keep critical content early in the document. Broken URLs waste fetches: Vercel's December 2024 analysis found ChatGPT's crawler spent 34.82 percent of its fetches on 404 pages and Claude's 34.16 percent, against 8.22 percent for Googlebot. Clean redirects and current sitemaps reduce that waste.

Step 5: The bot parses, usually without JavaScript

Googlebot renders pages in headless Chromium. Most AI crawlers do not. The same Vercel analysis found that none of the major AI crawlers it studied executed JavaScript; ChatGPT's and Claude's crawlers downloaded script files (11.50 and 23.84 percent of fetches) without running them. AppleBot renders through a browser-based crawler, and Gemini benefits from Googlebot's rendering. Google's own JavaScript documentation notes that not all bots can run JavaScript and recommends server-side rendering or prerendering. Test with View Source, not the browser inspector. Our guide on AI crawlers and JavaScript rendering covers framework-specific fixes.

Step 6: The content is used

What happens next depends on the job. Training crawlers add content to datasets for future model versions. Search crawlers add it to an index that answers retrieve from, as described in how AEO works. User-triggered agents pass the text straight into one conversation.

robots.txt patterns for a B2B site

A common B2B default: allow search and user-triggered bots on all public pages, keep application and account areas closed to everyone, and make the training decision deliberately.

# Search crawlers that feed cited AI answers
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /app/
Disallow: /account/

# Training crawlers: allowed here; change to Disallow: / to opt out
User-agent: GPTBot
User-agent: ClaudeBot
Allow: /
Disallow: /app/
Disallow: /account/

User-agent: *
Disallow: /app/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

Note the repeated Disallow lines. Under RFC 9309, a bot that matches a named group ignores the * group entirely, so rules you want applied to named bots must be repeated inside their groups. To opt out of training while staying in search, set Disallow: / for GPTBot, ClaudeBot, CCBot, Google-Extended and Applebot-Extended, and leave the search groups unchanged. Anthropic notes rules apply per subdomain, so each subdomain needs its own file.

Reading the logs: what actually happened

robots.txt states intent. Logs show behavior. Pull 30 days of CDN or server logs and check five things for each bot:

1. Identity. Filter by user agent token, then match the source IP against the operator's published ranges. Google notes user agent strings are frequently spoofed.
2. Status codes. A wall of 403 or 429 responses for a verified search crawler means your edge is blocking it.
3. robots.txt fetches. Confirm the file returns 200. Intermittent 5xx responses tell compliant crawlers to stay away.
4. Paths. See which sections get fetched. Search bots that only ever reach the homepage point to discovery or linking problems.
5. Trend. Compare month over month after any CDN, CMS or migration change.

203.0.113.24 - - [08/Sep/2026:14:02:11 +0000] "GET /platform/integrations HTTP/1.1" 403 1520
"-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"

That sample line uses a documentation IP address. In a real log, a verified GPTBot receiving 403 means a rule somewhere is blocking a bot your robots.txt allows. Our AI crawler log file analysis guide has queries for common log formats.

Four misconceptions about AI crawlers

• "Blocking AI in robots.txt blocks ChatGPT, Claude and Perplexity." Only the named tokens are affected. A GPTBot rule does nothing to OAI-SearchBot, and user-triggered agents may not consult robots.txt at all.
• "Allowing bots in robots.txt means they can reach us." robots.txt is guidance for compliant crawlers. Your CDN, firewall, rate limits and the availability of robots.txt itself decide what actually happens, and only logs show it.
• "If Google can see our JavaScript content, so can AI engines." Googlebot renders pages; most AI crawlers do not. A page that looks complete in Google's URL Inspection tool can still be nearly empty to OAI-SearchBot.
• "An llms.txt file controls AI crawlers." llms.txt is a proposed format for listing key pages in Markdown, not an access control standard, and Google's generative AI guide says AI text files neither help nor harm visibility in its Search. Access rules still live in robots.txt. Our review of whether llms.txt works covers the evidence.

A ten-minute self-test catches most problems. Open robots.txt on every subdomain and confirm it loads. Search it for each token in the reference table. Open a key product page, choose View Source, and search for a sentence from the body copy. Ask your security team whether a bot management or AI crawler setting is enabled at the CDN. Then pull one week of logs filtered to OAI-SearchBot, Claude-SearchBot and PerplexityBot and count the status codes.

What you control: the AI crawler checklist

• robots.txt returns HTTP 200 on every subdomain, reliably.
• Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot) are allowed on public pages.
• Training crawler rules reflect a recorded decision, not a default.
• Named groups repeat any disallow rules that must apply to them.
• CDN and firewall rules allow verified AI search and user agents, checked against published IP ranges.
• Main copy, links and tables are present in the server-rendered HTML.
• Broken internal links and redirect chains are cleaned up, and the sitemap lists live, canonical URLs.
• Private areas use authentication, not robots.txt.
• Logs are reviewed monthly and after every infrastructure change.

Illustrative example: a data platform that vanished from Perplexity

Illustrative example. The company, logs and findings below are invented to show the diagnostic sequence. They are not a client result.

A B2B data integration company notices that Perplexity stopped citing its connector documentation, which used to appear for "how to sync Salesforce to Snowflake" prompts. Its robots.txt allows everything. The team works through the chain:

1. Logs by token. PerplexityBot requests over the last 30 days: all returned 403 since a date six weeks earlier. Claude-SearchBot shows the same pattern.
2. Verification. The source IPs match Perplexity's and Anthropic's published ranges, so these are real crawlers.
3. Edge rules. Six weeks earlier, the security team switched on a managed bot protection setting during a scraping incident. It challenges any non-browser traffic not on the vendor's verified list.
4. robots.txt health. The docs subdomain serves robots.txt from a different origin that returns 503 during nightly deploys. Under RFC 9309, compliant crawlers treat that as full disallow.
5. Rendering. The connector pages are server-rendered, but the new pricing calculator is client-side only. It is noted for a later fix.
6. Fix and confirm. The team allowlists verified AI search and user agents at the edge, moves the docs robots.txt to static hosting, and checks logs a week later for 200 responses before rerunning its prompt set.

Glossary

• Control token: a robots.txt name, such as Google-Extended, that governs how content is used without being a separate crawler.
• Crawl-delay: a non-standard robots.txt line asking a bot to wait between requests; ClaudeBot supports it.
• Product token: the bot name in a user agent string that robots.txt groups match against.
• RFC 9309: the IETF standard that defines the Robots Exclusion Protocol.
• Search crawler: a bot that builds the index an engine retrieves from when it cites sources.
• Training crawler: a bot that collects content that may be used to train future models.
• User-triggered agent: a fetcher that loads a page because a user's request needs it.
• WAF: web application firewall, which can block or challenge bots before your server responds.

Next in the series

Once crawlers can read your pages, engines still have to understand who you are. How entities and schema work covers knowledge graphs, JSON-LD and sameAs. To have your robots.txt, edge rules and rendering checked against every bot above, request a free audit, or use the crawler chapter of the B2B AEO and GEO playbook.