Cloudflare AI Crawler Settings: What Enterprises Should Allow After the Default Block

What Do Cloudflare AI Crawler Settings Actually Control?

Cloudflare AI crawler settings decide which AI clients may fetch pages from your domain, sorted by purpose: model training, search and answer retrieval, or user-triggered agent requests. Cloudflare launched a one-click toggle to block AI bots in July 2024, and on September 15, 2026 it will apply new defaults to new domains, new sites and existing free-plan sites that have not changed their settings. Existing sites on paid plans keep the settings they already have, so on many enterprise zones the current configuration is whatever someone chose, or never revisited, since that toggle appeared. Either way, access is a setting someone owns, and it is now a marketing concern as much as a security one.

That default matters more than it sounds. A control that used to sit quietly in a security console now determines whether a company appears inside ChatGPT, Perplexity, Google AI Mode and the growing set of assistant surfaces where buyers begin research. Marketing teams are discovering the problem backwards: citations decline, a competitor starts appearing in answers where the brand used to, and the investigation ends at an infrastructure toggle nobody in the marketing organization knew existed.

The reason this catches enterprises specifically is ownership. The setting lives with security or platform engineering, the consequence lands on demand generation, and the two functions rarely review the same dashboards. In most enterprise environments we see, the setting was switched on during onboarding or a security review and then left unexamined for months before anyone connected it to declining AI visibility.

The correct response is not to switch everything back on. It is to treat crawl access as a deliberate policy with four distinct decisions behind it, each with a different owner, a different risk profile and a different commercial consequence.

What changes on September 15, and how it can block Googlebot

On July 1, 2026, Cloudflare replaced its single AI bot switch with three behaviors that every customer, including those on the Free plan, can manage separately, according to its changelog and announcement:

• Search: crawlers that collect or index your content so they can answer questions about it later.
• Agent: automated activity acting in real time for a person, such as ChatGPT-User or a browser agent driving Chrome.
• Training: crawlers that take your content to train or fine-tune a model.

For each behavior, a site can block on all pages, block only on pages that display ads, or not block. The three line up with the gates later in this guide: Training is gate one, Search feeds gate two and Agent is gate three.

Which sites get the new defaults

From September 15, the default is Search allowed, with Training and Agent blocked on pages that display ads. Cloudflare's reasoning is that an ad signals a page meant for human attention. Whether a site receives that default depends on how it got there:

The free-plan change comes from Cloudflare's July 1 press release, which says the changes will also be made for existing free customers that have not changed their settings by September 15. Search Engine Journal and TechCrunch reported the same list of affected sites. Cloudflare's own changelog and blog announcement describe the scope differently, applying the new defaults to new domains onboarding to Cloudflare, so the press release and the product documentation do not match. Check the Search, Agent and Training settings in your dashboard rather than relying on either description. Remote Work Europe, which audited its own Cloudflare setup on September 11, advises checking the configuration directly rather than assuming a plan exempts you. That is sound advice on any plan, because the last row of the table has nothing to do with plan type.

The multi-purpose crawler rule

The second change matters more for B2B sites. From September 15, Cloudflare treats a crawler that does both Search and Training according to all of its behaviors and enforces the most restrictive rule that applies. Its announcement names Googlebot, Applebot and Bingbot as examples of these multi-purpose crawlers, and says they will be blocked for customers who block Training, whether through the new options or the older "Block AI bots" setting. The press release adds that under the new defaults, mixed crawlers that do not let site owners choose between search, agent use and training will be blocked on all pages with ads.

What that means for the crawlers most B2B sites depend on:

• Googlebot builds Google's search index, and Google Search is where AI Overviews and AI Mode appear, as TechCrunch notes. Google's own training control is the Google-Extended robots.txt token, which Google says does not affect inclusion or ranking in Google Search. A Cloudflare Training block is different: it acts on the crawler itself at the edge.
• Applebot powers search features in Spotlight, Siri and Safari, according to Apple. Apple's training opt-out is Applebot-Extended, which does not remove content from search results.
• Bingbot crawls for Bing search, so blocking it takes the site out of the index Bing's features draw on.

Search Engine Journal makes the key point for anyone who owns the console: robots.txt is an advisory instruction, while a Cloudflare block operates at the network level. A Googlebot that cannot reach the site cannot crawl it effectively, and search visibility can suffer over time.

The old Block AI bots toggle now counts as a Training block

Cloudflare folds its older "Block AI bots" setting, which began as a one-click toggle in July 2024, into this rule: a site that switched it on is treated as blocking Training. A zone where someone enabled that toggle at any point since July 2024 and never looked again is exactly the zone at risk on September 15, on any plan. Cloudflare has also stopped treating Verified bots as allowed by default. A verified bot is now allowed only if its category is allowed, so Googlebot's verified status does not exempt it. Cloudflare says customers can mark in their Security settings, before September 15, that they want no changes for Training crawlers that also crawl for Search, and the press release says settings can be changed in the dashboard at any time.

Pay Per Crawl becomes Pay Per Use

Cloudflare is also reshaping Pay Per Crawl into Pay Per Use, so publishers are paid when their content is used, for example when it appears in Ceramic.ai's AI search results or when a You.com agent buys a specific piece of premium content, rather than each time it is fetched, according to the press release. Both partnerships are early experiments, and for most B2B sites Pay Per Use is a publisher tool for the reasons in the Pay Per Crawl section below.

A September 15 checklist for B2B marketing sites

Most B2B marketing sites do not run ads, and their goal is to be cited, not to be paid per crawl. Work through these steps with whoever holds the Cloudflare login, before September 15 if you can, and repeat the checks afterward either way.

1. Confirm plan and current settings Note the plan for each zone and record the current Search, Agent and Training choices, including whether the older Block AI bots setting was ever switched on. Take screenshots before changing anything. On a free plan, unchanged settings move to the new defaults on September 15.
2. Allow Search and Agent crawlers Set both to not blocked on public marketing pages. Search crawlers put your pages into search results and AI answers, and Agent traffic is the fetch that happens when a buyer asks an assistant to read your page.
3. Decide on Training deliberately Write down the decision and the reason. On Cloudflare, blocking Training also blocks multi-purpose crawlers such as Googlebot, Applebot and Bingbot wherever the block applies. If the goal is to limit model training without touching search, robots.txt tokens such as Google-Extended and Applebot-Extended are the documented controls, as covered in how AI crawlers work.
4. Never apply a blanket block to ad-free marketing pages A block on all pages for Training or Agent reaches the product, solution, integration, documentation and blog pages a SaaS company or services firm built to be found. If one section genuinely needs protection, such as proprietary research, scope a narrower rule to that path or put the asset behind a form.
5. Test with real user agents and curl Request a few key pages with each crawler's user agent and read the status code, using the loop below. Remote Work Europe's audit recommends this direct test over trusting the dashboard figure.
6. Check WAF and bot-fight rules Custom WAF rules, rate limits, managed challenges and Bot Fight Mode can block crawlers regardless of the AI settings. Security teams own these, so bring log evidence and a narrow request, as our guide to WAF rules blocking AI crawlers explains. If you market a cybersecurity company, expect your own security team to ask for exactly that evidence.
7. Verify in Search Console and server logs Run a live URL Inspection test on key pages, then open the Crawl Stats report, which groups Googlebot's responses by type, including 403 and 429, and shows host status. In server or CDN logs, match crawler requests to each operator's published IP ranges and count status codes. Our AI crawler log file analysis guide has the queries.
8. Re-check after September 15 On September 16 and again a week later, reread the settings, rerun the curl tests and review Crawl Stats and logs. Record the final configuration in the written crawl access policy described below.

# Status code each user agent receives on a key page
for ua in "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "bingbot" "Applebot" "OAI-SearchBot" "ChatGPT-User" "Claude-SearchBot" "PerplexityBot" "GPTBot"; do
  printf "%s: " "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "$ua" https://www.example.com/platform/
done

Copy the full user agent strings from each operator's documentation for a closer match, and read the results with care. User agents are easy to fake, so Cloudflare does not decide whether a request comes from a verified bot on the user agent alone. A request from your own machine shows how your origin and any user agent rules respond, not what Cloudflare does to the real crawler. A 403 or a challenge page deserves a look. A 200 is a good sign, not proof, which is why the Search Console and log checks in step seven matter.

Which AI Crawler Categories Exist, and Why the Distinction Matters

There are three functional categories of AI crawler and they behave nothing alike: training crawlers that collect text to build future models, search and retrieval crawlers that fetch pages so an assistant can cite them in a live answer, and user-triggered agent fetches where a person asks an assistant to read a specific URL. Blocking all three with one rule is the most common and most expensive configuration error.

Training crawlers take content and give nothing back in the near term. Whatever they ingest surfaces later, unattributed, inside a model that may never link to the source. This is the category most legal and content teams want restricted, and restricting it costs very little traffic today, as long as the restriction does not also catch multi-purpose crawlers such as Googlebot.

Retrieval crawlers are the opposite. They fetch a page because a user asked a question right now, and the result is a citation with the brand name attached and, often, a link. Blocking retrieval is functionally identical to blocking your own listing from a search results page. The site does not appear as a source, the assistant answers from whichever competitor did allow access, and there is no error message anywhere in the marketing stack to explain it.

The third category, user-triggered fetch, is where the pressure is now. If a prospect pastes a pricing page into an assistant and asks it to summarize the terms, a blanket block means the assistant reports that it cannot access the page. Cloudflare has said it hopes its September 15 defaults will encourage operators of mixed-use crawlers to separate search from agent use and training, so site owners can make this distinction rather than choosing all or nothing.

The Four-Gate Crawl Access Model

The Four-Gate Crawl Access Model gives enterprise teams a defensible way to set policy: four independent gates, each opened or closed on its own merits, rather than a single switch owned by whoever touched the console last. It replaces the argument about whether AI crawlers are good or bad with four narrower questions that different functions can actually answer.

Gate one is training. The question is whether your published content may be used to build future models with no attribution and no traffic. Most enterprises close this gate for proprietary research, original data and gated-equivalent material, and leave it closed until a licensing conversation exists. Closing it carries almost no visibility cost when it is done with robots.txt tokens or a narrowly scoped rule. On Cloudflare after September 15, a site-wide Training block is not free, because it also stops crawlers that combine Search and Training.

Gate two is retrieval and citation. The question is whether an assistant may fetch a page in order to cite it in a live answer. This gate should be open for nearly every commercial site. It is the mechanism by which a brand appears in AI answers at all, and closing it removes the company from consideration in the surface where a growing share of B2B research now begins.

Gate three is user-triggered agent fetch, where a named human has already asked for that specific page. Blocking here creates visible friction for prospects and customers mid-evaluation, so the default position is open, with rate limits rather than refusals. Gate four is commercial licensing: whether crawl access should be sold rather than simply granted or denied. Work through the gates in order, because gate four only makes sense once one to three are settled.

How Do You Verify What Your Site Is Actually Blocking?

Verification means reading live request logs by user agent, not reading the intended configuration, because the two diverge more often than teams expect. Start with the bot analytics view in Cloudflare, filter to AI clients over the longest date range available, and compare the categories that are being served against the categories that are being challenged or blocked.

Three artifacts need to agree. The Cloudflare AI bot rules are the first. The robots.txt file is the second, and it frequently contradicts the edge configuration because it was written by a different team in a different year. Any WAF rules, rate limits or managed challenges that catch AI clients as collateral damage are the third, and these are the ones that produce the confusing outcome where the configuration says allow and the logs say otherwise.

Then test from the outside. Run the same commercial prompts across ChatGPT, Perplexity and Google AI Mode, several times each, and record whether the domain appears as a cited source. Single runs prove nothing: the same prompt repeated returns different brands and different sources, which is why a check needs repeated runs of each prompt before the pattern is real. A domain that never appears across dozens of runs, while competitors appear consistently, is showing an access problem rather than a content problem.

Reading the Cloudflare dashboard: unsuccessful does not mean blocked

Cloudflare's AI Crawl Control reports a count of unsuccessful AI crawler requests, and that number is easy to misread. When Remote Work Europe audited its own Cloudflare configuration on September 11, the figure read about 2,000 in 24 hours, up 74 percent, which looked like crawlers being turned away in volume. They were not. The count combines three different responses in one number: 429 (rate limited), 403 (blocked) and 404 (page not found). In that audit, most were 404s caused by old external links pointing at URLs that no longer exist.

Each code points to a different fix:

• 403: a rule refused the request. Find the rule, whether an AI crawler setting, a custom WAF rule or bot protection, before changing anything.
• 429: a rate limit throttled the crawler. Review limits that were tuned for human traffic.
• 404: the crawler reached your site and asked for a page that does not exist. That is a redirect and link hygiene task, not an access problem.

Before anyone concludes a crawler is blocked, split the unsuccessful count by status code and by crawler, then test the specific pages with the steps in the checklist above. A spike that turns out to be mostly 404s calls for redirects, not a change to the AI settings.

Is the Pay Per Crawl Trade-Off Worth Taking?

Pay Per Crawl lets a site charge AI companies for crawl access rather than granting or denying it for free, and it makes clear sense for publishers whose product is the content itself. Cloudflare is now reshaping it into Pay Per Use, which pays when content is used rather than fetched, and the reasoning below applies to both. For most enterprise B2B sites, whose product is software or services and whose content exists to generate pipeline, charging for access trades away the visibility the content was built to create.

The economics separate cleanly. A publisher loses revenue when an assistant answers from its article and the reader never arrives, so pricing that access recovers something real. A B2B company loses nothing when an assistant summarizes its solution page and cites it, because the citation is the point. Charging for that access is closer to charging search engines for the right to index you.

There is a defensible middle position. Price access to genuinely proprietary assets such as original benchmark data, research libraries and technical documentation that took real investment to produce, while leaving product, solution and comparison pages open to retrieval. That keeps gate four available for the material that has independent commercial value without closing the visibility path for everything else.

How Should Marketing, Security and Legal Coordinate the Decision?

Crawl access needs a single documented owner with input from three functions, because no one function holds all the relevant information. Marketing knows what visibility is worth, security owns the infrastructure and the abuse surface, and legal holds the position on content rights and licensing. Left uncoordinated, the default wins by inertia.

The practical mechanism is a short written policy that states which gates are open, which are closed, what triggers a review and who signs off on a change. Two pages is enough. What matters is that the document exists somewhere both the security team and the demand generation team can find it, so the next platform update does not silently reset a commercial decision.

Expect the objections to be legitimate rather than obstructive. Security will point out that AI crawler traffic can be heavy and that user agents can be spoofed, which argues for verified bot signatures and rate limits rather than blanket blocks. Legal will want training access restricted, which gate one already handles. Once the categories are separated, most of the disagreement dissolves, and the remaining conversation is usually short.

Set a review cadence of roughly every six months, and add an unscheduled review whenever Cloudflare changes its defaults, as it does on September 15, 2026. Bot categories, user agent names and the separation between training and retrieval crawlers are all still moving, and a policy written against last year's category names will quietly stop matching reality.

How Do You Confirm Access Has Actually Been Restored?

Confirmation comes in two stages: server-side evidence that retrieval crawlers are being served successfully, then answer-side evidence that the domain reappears as a cited source. The first shows up within days. The second takes considerably longer and teams that expect otherwise conclude the fix failed.

Server-side, look for successful responses to retrieval and user-triggered agent user agents in the logs, with the error rate for those clients falling toward zero. Answer-side, rerun the same prompt set used during diagnosis, at the same cadence, and track the share of runs in which the domain appears. Citation share takes longer to recover than server-side access, because assistants need to recrawl, reindex and then start selecting the pages again.

Measure this as a rate, not an event. Because the same prompt returns different sources on different runs, the meaningful metric is the percentage of runs across a fixed prompt set where the brand is cited, tracked monthly against the competitors that appear in the same answers. That is the number worth putting in a board deck, and it is the number that shows whether an infrastructure decision helped or hurt.

This is the diagnostic work Lemniscate Growth runs for enterprise clients before touching content strategy, because a visibility problem caused by an edge configuration cannot be fixed by writing more pages. The free AEO Checkers and AI Citation Checkers in The GrowthGPT toolset cover the answer-side half of that check, and pairing them with the log review above tells a team within a week whether it has an access problem or a content problem. To have your Cloudflare settings, WAF rules and logs reviewed together, request a free audit, or see how this fits into our AEO, GEO and SEO service.