What AI crawlers actually send back

For two years the advice has been to make your site easy for AI systems to read, on the assumption that being read leads to being cited, and being cited leads to visitors. The first two steps happen. The third one is now measurable, and the numbers are not what the advice implied. Here is what the exchange actually looks like, what it costs you to host, and what you can do about it — including the option, new this year, of charging for it.

Pages fetched per human visitor referred

Cloudflare Radar, Q1 2026. The ratio counts pages fetched against human clicks referred back to the same site. It is a network-wide average across sites on Cloudflare, so your own figure will differ — but the order of magnitude is the finding, not the decimal. Googlebot is included as a reference point for what a search crawler that sends traffic looks like; it is not from the same measurement.

The exchange nobody agreed to

A search crawler and a website have always had a bargain. The crawler takes your pages, and in return it sends people who searched for what you wrote. The rate was never one-to-one, but it was close enough that nobody thought of it as a trade at all.

An AI crawler takes the same pages and, in the measured average, returns almost nothing. Content is read at industrial scale, absorbed into an answer, and the person who needed it never arrives. If your site earns its keep through pageviews, that is not a nuisance — it is the business model being quietly removed while the traffic graph still looks busy.

What it costs you before anyone visits

The first cost is bandwidth and CPU. A crawler pulling tens of thousands of pages is a load pattern, and on shared or entry-level hosting it competes with real visitors for the same resources. Sites on constrained plans see it as slow afternoons with no matching rise in sessions, which is a genuinely hard thing to diagnose if you are not looking at your logs by user agent.

The second cost is analytical. Bot traffic that is not filtered inflates the numbers you use to make decisions — page popularity, bandwidth planning, even which articles look worth updating. If you have never split your logs by user agent, the split is usually the most surprising report you will run this year.

Three positions, and none of them is obviously right

You can allow everything, on the argument that presence in AI answers is brand-building even when it never resolves into a click. That is a real effect and it is the position most small businesses should probably take, because the alternative is being absent from the place a growing number of people now ask their questions.

You can block, via robots.txt or a CDN rule, which protects your content and your server and removes you from AI answers entirely. And you can now license: Cloudflare’s pay-per-crawl uses an HTTP 402 response with a signed payment header, so a crawler is quoted a price per request instead of being allowed or refused. That third option is what changes the conversation from a permission question into a commercial one.

The date to put in the calendar

From 15 September 2026, Cloudflare blocks mixed-use crawlers by default on any page that carries advertising. Mixed-use means a bot that both indexes for search and collects for model training — which describes most of them. If your site is behind Cloudflare and monetises through ads, this changes your defaults without you doing anything, and it is worth knowing before your traffic composition shifts rather than after.

The practical step is smaller than it sounds: look at what your CDN is currently doing, decide deliberately rather than by default, and write the decision down somewhere your future self will find it. A setting you did not choose is the one that surprises you.

What this does not mean

It does not mean the work of being readable was wasted. Clean structure, unambiguous facts and correct schema are what earn a citation, and a citation is worth having even at a poor click rate — it puts your name in the answer when someone asks for a recommendation. The mechanics are in our GEO guide, and how long it takes to get cited at all is a separate and slower question.

What it does mean is that traffic is the wrong thing to promise yourself from this channel. Measure it the way you would measure a brand campaign — branded search, direct enquiries, being named when someone asks — and not as a referral line that was never going to arrive in volume. If your reporting still leads with sessions, the metrics worth swapping in are here.

Sources

Figures checked September 2026. The ratios are network-wide averages and are quoted as orders of magnitude, not as guarantees about any single site.

• TechCrunch — Cloudflare’s new policy pushes AI companies to pay for publishers’ content ↗

  The pay-per-crawl launch: default blocking of AI crawlers for new domains, and a marketplace in which publishers set a price for access.

• Digital Applied — AI crawler access control: the 2026 decision matrix ↗

  Source of the crawl-to-refer figures quoted in the table (Cloudflare Radar Q1 2026), and of the 15 September 2026 change to mixed-use crawler defaults on ad-bearing pages.

• Learn & Work Ecosystem Library — Pay-per-crawl ↗

  The mechanism itself: HTTP 402 Payment Required with Ed25519-signed payment headers, charged per request.