A crawler pays per page

The situation

An AI company runs a crawler that reads articles on publisher sites behind Cloudflare. The publishers want to be paid for each page. No human is present: the crawler decides, pays and fetches by itself.

Cloudflare offers two different mechanisms, and they must not be merged. The Monetization Gateway closed beta of 2026-09-30 settles USDC on Base through Coinbase’s x402 facilitator.^{Source [1]} The credit-backed cloudflare:402 network with RFC 9421 signatures is a separate batch-settlement variant, not the gateway’s settlement path.^{Source [1, 2]} This page walks through each in turn, steps 1 to 6 for variant (a) and steps 7 to 13 for variant (b).

There is a fit question too. The gateway targets per-use resources such as websites, APIs, MCP tools and datasets.^{Source [1]} Its announcement places pages that are crawled once and used many times under Cloudflare’s Pay Per Use instead.^{Source [1]} The cloudflare:402 binding, by contrast, sends crawlers to Cloudflare’s pay-per-crawl documentation to register.^{Source [2]}

Actors

• Crawler, an autonomous agent with a wallet (variant a) or a Cloudflare billing account (variant b).
• Cloudflare edge, which prices requests and checks payment.
• Publisher origin, the seller.
• Coinbase x402 facilitator and Base, in variant (a) only.^{Source [1]}
• Crawler key directory and Cloudflare billing, in variant (b) only; Cloudflare acts as merchant of record there.^{Source [2]}

Step by step

Variant (a): Monetization Gateway, closed beta.

1. Request (x402). The crawler asks for a priced resource. Sellers set which requests pay, the cost and the destination, with rules on URL, headers or query parameters.^{Source [1]} Only eligible US-based sellers and buyers can join.^{Source [1]}
2. 402 (x402). The server answers with 402 and a PAYMENT-REQUIRED header holding base64 PaymentRequired JSON.^{Source [3, 4]} The announcement names no scheme identifier.^{Source [1]}
3. Pay (x402). The crawler retries with PAYMENT-SIGNATURE holding a base64 PaymentPayload.^{Source [3]}
4. Verify and settle (x402). The server calls the facilitator’s POST /verify and then POST /settle.^{Source [4]} Cloudflare routes this through Coinbase’s facilitator.^{Source [1]}
5. On chain (Base). The facilitator broadcasts the USDC transfer and usually pays the gas.^{Source [5, 6]}
6. Serve (x402). The server returns 200 with a PAYMENT-RESPONSE holding the SettleResponse.^{Source [3]}

Variant (b): batch-settlement on cloudflare:402, credit-backed.

7. Register once (x402 extension). The crawler publishes keys at /.well-known/http-message-signatures-directory and registers its signature agent; Cloudflare ties it to a billing identity.^{Source [2]}
8. Request (x402). The crawler asks for the article.
9. 402 (x402). The requirement names batch-settlement, cloudflare:402, payTo set to the constant merchant, and asset USD.^{Source [2]} Schema and optional fields are dropped because large headers may be lost.^{Source [2]}
10. Sign (x402). The crawler sends Signature-Agent, Signature-Input and Signature, plus PAYMENT-SIGNATURE with amount and asset.^{Source [2]} The signature covers @authority, signature-agent and payment-signature.^{Source [2]}
11. Verify at the edge (x402). The edge checks the signature, the known agent, amount and asset, freshness and the accepted offer.^{Source [2]}
12. Serve (x402). The PAYMENT-RESPONSE has an empty transaction field.^{Source [2]}
13. Bill and pay out. Cloudflare aggregates commitments, bills the crawler’s identity and pays publishers periodically over off-chain rails.^{Source [2]}

Which protocol does what

The batch-settlement scheme, its cloudflare:402 binding and the signatures extension were merged on 2026-04-15.^{Source [7]}

Where controls apply

• Before authorization. Budgets are out of scope for x402 and live in SDKs and wallets.^{Source [4, 8]} The TypeScript client caps each payment at $1 by default.^{Source [9, 10]} In variant (b) the credit terms set at registration act as the limit.^{Source [2]}
• At signing. In variant (a) the client’s signature fixes recipient and amount.^{Source [11]} In variant (b) the signature binds the payment header to the request.^{Source [2]}
• At the edge. Cloudflare’s gateway is the enforcement point in both; in variant (b) RFC 9421 identity replaces a wallet.^{Source [2, 12, 13]}
• At settlement. Variant (a) has the facilitator, which usually pays gas.^{Source [5, 6]} Variant (b) has Cloudflare’s billing.
• After the fact. The exact scheme defines no refund, and x402 has no dispute process.^{Source [14, 15]} In variant (b) Cloudflare is merchant of record with its own billing and dispute terms.^{Source [2, 4]}

What is observable from outside

Variant (a) leaves a USDC transfer on Base. Header contents such as resource URLs and amounts stay between client, server and facilitator, so the chain shows the transfer but not the page.^{Source [4, 16]} Variant (b) leaves nothing on any chain: commitments never reach one, and Cloudflare alone holds the link from signature identity to billing account.^{Source [2]} The crawler’s key directory is public, since the edge must fetch it.^{Source [2, 17]} The gateway post gives no fee or payout schedule.^{Source [1]}

What can go wrong

• Unknown or unregistered crawler (b). The binding returns signature_agent_unknown, invalid_signature, payment_failed or price_not_acceptable.^{Source [2]}
• No identity check (a). The gateway post mentions no crawler signature requirement and says identity primitives are planned.^{Source [1]} Anyone with a funded wallet can pay.
• Settle timeout (a). A timeout is indeterminate; settlement_pending is retried and reconciled.^{Source [4, 6]}
• Concentration. Critics note that Cloudflare’s gateway, like most x402 traffic, routes through Coinbase’s facilitator (reported).^{Source [1, 18, 19]}
• Who bears the loss. In (a) a paid but failed fetch is the publisher’s arrangement to fix, because exact has no refund.^{Source [14]} In (b) pages are served before the bill, and what happens when a batched payment fails is set by the registration terms, not by x402.^{Source [2]}

Variants

• Pre-authorized crawler (b). A crawler with a pre-authorized agreement can sign and pay in its first request and skip the 402.^{Source [2]}
• Paid inference instead of pages (a). Cloudflare AI Gateway accepts PAYMENT-METHOD: x402 to pay for inference.^{Source [20]}