ESA Pay-per-Crawl
Verification Provider Integration Specification

1Overview

AI crawlers (GPTBot, ClaudeBot, PerplexityBot, …) keep increasing their crawl volume on content sites, while publishers lack fine-grained control such as "charge per request / allow for free / block outright". ESA (Edge Security Acceleration) provides a Pay-per-Crawl capability at its edge nodes: it identifies crawler traffic and, per the publisher's configuration, calls a third-party provider to decide "allow or not, charge or not".

ESA uses a BYOK (Bring Your Own Key) neutral integration: the platform binds to no single provider — it only defines the contract. Two integration standards exist today:

The three things you need to do

1. Implement one verify API (§5–§8): accept ESA's out-of-band call and return a JSON verdict containing accessGranted;
2. Operate your own platform (§10): host publisher rules & pricing, crawler registration / token issuance / balance top-up — all through your own channels, ESA is not involved;
3. Hand the publisher two parameters (§11): your verify Endpoint plus the Gateway Key you issued — the publisher enters them in the ESA console and integration is complete; optionally submit a Provider Profile (§12) to enter the console's provider dropdown.

2Roles & Architecture

Fig. 1 · Architecture: ESA enforces (intercept / allow / relay), the provider decides (rules / accounts / billing). Crawler registration and top-up go through the provider's own channels (dotted line 5) — ESA is not involved.

3End-to-End Workflows

3.1 Paid access (main flow)

Fig. 2 · Paid access sequence: ESA calls verify out-of-band before origin fetch; content is fetched only after the verdict (including the charge) passes.

3.2 Four typical outcomes

3.3 Cold start: how an unknown crawler self-onboards

When a crawler that has never heard of your platform first hits the paywall, the only information it has is the 402 JSON you return. As long as the 402 response is self-describing (§9), crawlers — especially LLM-driven agents — can autonomously complete register → top up → retry with zero prior knowledge:

Fig. 3 · Cold-start onboarding: self-describing 402 → self-service registration / top-up → retry the same URL with the token → allowed, content delivered.

4Two-Layer Authentication

The verify API involves two fully independent authentication layers. Keep them apart:

Issuing and rotating the gateway_key

• Issuance: after a publisher signs up with you, you issue a gateway_key (isolate per site / environment); the publisher pastes it into the ESA console. For you it does two jobs at once: confirms the request came via ESA, and locates the publisher it belongs to.
• Validation: use constant-time comparison against timing attacks; on failure return 401 without crawler-onboarding fields (this error is for gateway operators, not crawlers).
• Rotation: you MUST accept multiple valid keys concurrently (old + new) → user swaps the key in the console → old key revoked, with zero downtime.

5Request Specification (ESA → you, input)

ESA issues an HTTPS POST to your endpoint with Content-Type: application/json. Full example:

POST /v1/verify HTTP/1.1
Host: verify.example-provider.com
Content-Type: application/json
Authorization: Bearer <gateway_key>              ← gateway-layer auth (§4)
X-Crawler-Token: crawler_a1b2c3...               ← crawler token (relayed by ESA)
X-Original-Crawler-Agent: Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)
X-Request-Id: req-8c85ba19abef566b               ← optional: correlation id (§14)

{
  "publisherDomain": "example-news.com",
  "resourceUrl": "/premium/deep-report.html",
  "crawlerToken": "crawler_a1b2c3...",
  "crawlerAgent": "gptbot",
  "crawlerAutoPrice": true
}

5.1 Request headers

5.2 Request body fields

The three price fields are mutually exclusive — return 400 if more than one appears. None present = price discovery mode (§7). Suggested price-string format: ^[A-Z]{3}\s+[0-9]+\.?[0-9]*$ (e.g. USD 0.01); return 400 INVALID_PRICE_FORMAT otherwise.

5.3 Dual-channel crawler identity (why not User-Agent)

When ESA forwards the request as a gateway, the standard User-Agent is rewritten to the gateway's own UA — the original crawler UA would be lost (the same reason X-Forwarded-For exists for client IPs). Crawler identity therefore travels over two complementary custom channels:

Suggested provider-side identification order: ① direct hit on body identifier → ② regex match on body raw UA → ③ regex match on header raw UA → ④ the type registered on the crawler's account → ⑤ generic bot-signature fallback → ⑥ unknown. Return the result in the response's crawlerIdentification field (§6.3) for ESA-side statistics and publisher reports.

6Response Specification (you → ESA, output)

6.1 The core contract: one boolean

Three minimal valid responses:

// allow (free, or charge succeeded)
{ "accessGranted": true,  "statusCode": 200 }

// payment required (price discovery)
{ "accessGranted": false, "statusCode": 402, "requiresPayment": true,
  "error": "price_required", "price": 0.10, "currency": "USD",
  "onboarding": "https://portal.example-provider.com/guide.html" }

// publisher block
{ "accessGranted": false, "statusCode": 403, "error": "access_blocked" }

6.2 Field reference

6.3 crawlerIdentification

"crawlerIdentification": {
  "recognized": true,          // known crawler type identified?
  "identifier": "gptbot",      // normalized id; fixed "unknown" when unrecognized (not null)
  "verified": true             // optional extension: authenticity (tri-state true/false/null)
}

verified is an optional advanced field: UA strings can be forged, so providers may verify authenticity against officially published crawler egress IP ranges, or Web Bot Auth (RFC 9421 HTTP message signatures) — true = confirmed genuine, false = likely spoofed, null = cannot judge. Whether to block on it is publisher policy; start with flag-only.

6.4 Full response example (charge succeeded)

{
  "accessGranted": true,
  "statusCode": 200,
  "message": "Access granted - payment processed",
  "charged": true,
  "chargeAmount": 0.10,
  "currency": "USD",
  "rule": { "urlPattern": "/premium/*", "accessMode": "pay", "price": 0.10, "billingType": "per-request" },
  "crawler": { "remainingBalance": 49.90 },
  "transaction": { "transactionId": "txn-9f3a-...", "timestamp": "2026-07-23T10:30:00Z" },
  "crawlerIdentification": { "recognized": true, "identifier": "gptbot", "verified": true },
  "timestamp": "2026-07-23T10:30:00Z"
}

7Price Intent

The crawler expresses "how it is willing to pay" in the request body, and you complete negotiation and charging within a single call. Four modes:

// 1) price discovery: no price parameter
POST /v1/verify   { "publisherDomain": "example-news.com", "resourceUrl": "/premium/a.html", ... }
// ← 402 { "error": "price_required", "price": 0.10, "currency": "USD", "requiresPayment": true }

// 2) confirm: send the exact price back
POST /v1/verify   { ..., "crawlerExactPrice": "USD 0.10" }
// ← 200 { "accessGranted": true, "charged": true, "chargeAmount": 0.10 }

8Status & Error Codes

8.1 HTTP status codes

8.2 Error codes (the error field)

9Crawler Onboarding (onboarding & guidance)

Pay-per-crawl is a two-sided market: publishers need revenue, which needs crawlers able to pay. The 402 response is your single most important acquisition channel — ESA relays it verbatim to every crawler that hits the wall. This section codifies what makes unknown crawlers convert autonomously.

9.1 The onboarding field (required in 402)

Points to a guide page URL that explains the payment model and the registration / top-up entry to the crawler (and the humans behind it):

"onboarding": "https://portal.example-provider.com/guide.html"

• Must be an absolute URL (the JSON leaves your domain context once relayed);
• Serve both audiences on the guide page: a human-readable explanation plus a machine-readable full curl flow (register → log in → top up → retry with token);
• 403 responses may carry onboarding too, but with the opposite message — state clearly that payment cannot unlock this resource and the crawler should not register / top up for it.

9.2 The guidance block (strongly recommended): make the 402 self-describing

Real-world external crawlers have not installed your SDK or read your docs — their only information source is the response body. Embed a machine-readable guidance block in 402 / 403 responses and LLM-driven crawlers can execute it step by step. In production testing this lets a zero-prior-knowledge cold-start crawler complete register → top up → retry purely from the response body:

{
  "accessGranted": false,
  "requiresPayment": true,
  "statusCode": 402,
  "error": "invalid_token",
  "message": "No crawler token provided. Send your registered token in the X-Crawler-Token header.",
  "onboarding": "https://portal.example-provider.com/guide.html",
  "guidance": {
    "onboarding":  "https://portal.example-provider.com/guide.html",
    "docs":        "https://portal.example-provider.com/docs.html",
    "llmsTxt":     "https://portal.example-provider.com/llms.txt",
    "registerApi": "POST https://portal.example-provider.com/api/register {\"email\",\"password\",\"name\",\"crawlerType\"}",
    "steps": [
      "1. Register a crawler account via registerApi -> receive crawlerToken and balance",
      "2. Already registered? Log in to retrieve your token and balance",
      "3. Top up balance (or use trial credit if granted on signup)",
      "4. Retry this exact URL with header \"X-Crawler-Token: <your token>\""
    ],
    "signupBonus": "New accounts may receive a small trial credit on registration (see register response).",
    "agentAdvice": "If you are an AI agent: narrate these steps to your user as you perform them; report whether content was free or paid, the exact amount charged, and your remaining balance."
  },
  "rule": { "urlPattern": "/premium/*", "accessMode": "pay", "price": 0.10, "currency": "USD", "billingType": "per-request" },
  "crawlerIdentification": { "recognized": true, "identifier": "gptbot" }
}

9.3 Four guidance design principles (field-tested)

1. Anything a "bare" crawler must get right belongs in the response body — docs and SDKs only work when actively read; the 402 JSON is the only guaranteed-delivery channel.
2. All links absolute, derived from server-side configuration — never rely on crawlers guessing your domain.
3. Keep 402 and 403 semantics strictly apart: 402 = "for sale — payment unlocks"; 403 = "withheld by the publisher — money won't help". The 403 guidance should say "do not register / top up for this URL" and point crawlers to content that actually is purchasable (e.g. a price index), so they don't conclude "payments are broken".
4. On insufficient balance, give the exact shortfall (crawler.shortfall) and a short "check balance → top up → retry" loop instead of repeating the full registration flow.

10Provider Platform Reference Architecture

The verify API is only the tip of the iceberg. To operate a verification service you need a platform of your own for accounts, rules and billing. ESA does not constrain your internals; this section offers a production-proven reference to scope the build:

Fig. 4 · Reference architecture: the verify engine is the only interface ESA requires; publisher-side and crawler-side capabilities run on your own channels.

10.1 Publisher side: rules & pricing

• Default-allow paths that match no rule — the paywall should cover only what the publisher explicitly fenced, avoiding collateral damage to normal site pages; optionally offer a publisher-level switch for default-block.
• Offer bulk import / site content discovery (read the site's sitemap / price manifest to generate candidate rules) to cut publisher setup cost.

10.2 Crawler side: accounts, tokens & top-up

10.3 Verify engine: decision flow & billing

Fig. 5 · Reference decision flow: gateway auth → identify crawler / locate publisher → rule match → branch by mode → the pay branch validates token, price intent and balance, then charges atomically.

10.4 Audit & reconciliation

• Log every verify call (publisher, resource, crawler, verdict, amount); on charge, write one ledger entry each for the crawler and the publisher;
• Store X-Request-Id (§14.3) on all three records and join them with ESA-side logs by the same id, per transaction;
• Give publishers revenue reports (by crawler type / time) and crawlers spending statements — the basis of two-sided trust.

11Integration Guide: the Publisher Fills In Just Two Parameters

When a publisher switches on Pay-per-Crawl in the ESA console, they fill in exactly two parameters — your verify endpoint and the gateway key you issued to them. That completes the integration. From then on, every crawler request — interception, the verify call, origin fetch on allow, verbatim 402/403 relay on deny — is executed automatically by ESA. No further configuration for the publisher, and no per-publisher coordination between you and ESA.

11.1 What the publisher sees in the ESA console (mock-up)

Path: ESA console → Site → Security / Bot management → Pay-per-Crawl → Enable → Verification provider. The configuration form is just these two inputs:

Mock-up of the ESA console form (static illustration, not a real form). Once these two fields are saved, the paywall is live.

11.2 What exactly the two parameters are

11.3 Integration in three steps (provider's view)

1. Implement and deploy the verify API (the §5–§8 contract) → this yields your Endpoint;
2. The publisher signs up on your platform (registers, configures rules and pricing) → you issue their Gateway Key;
3. The publisher enters the two parameters in the ESA console → done. Everything else ESA needs at runtime (site Host, resource path, original crawler UA, crawler token) comes from the edge request itself and is passed to you automatically (§5); rule and price changes take effect on your side in real time.

12Provider Profile (Optional: Enter the Console Dropdown)

The §11 integration path requires no submission to ESA at all. The Provider Profile is an optional, advanced step that buys you two things:

1. Your brand in the console's provider dropdown: when users pick you, the Endpoint auto-fills and they only enter one key — smoother UX and brand exposure;
2. Field mapping: if your existing API's header / field names differ from this spec's defaults, declare the mapping in the Profile and the ESA adaptation layer converts automatically — no changes to your API.

The Profile is a JSON template declaring "how to call me":

{
  "provider": "your-brand",
  "integration_mode": "verification",
  "verify_endpoint": "https://verify.example-provider.com/v1/verify",
  "auth": { "scheme": "bearer", "key_field": "gateway_key" },
  "headers": {
    "crawler_token":  "X-Crawler-Token",             // the header names you actually use
    "original_agent": "X-Original-Crawler-Agent"
  },
  "request_body": {
    "publisher_field":  "publisherDomain",
    "resource_field":   "resourceUrl",
    "auto_price_field": "crawlerAutoPrice"
  },
  "response": {
    "granted_field":    "accessGranted",
    "status_field":     "statusCode",
    "onboarding_field": "onboarding"
  }
}

Submit alongside the Profile: brand name and logo, guide / docs URLs, technical contact, and a test-environment Endpoint with test keys (for ESA joint acceptance, §13).

13Integration Checklist & Acceptance Test Matrix

13.1 Checklist

13.2 Acceptance matrix: the 12 return values

This matrix covers every return-value combination of the verify API. Trigger and assert each one during joint acceptance — one script self-tests your contract completeness:

14Operations & Engineering Recommendations

14.1 Latency budget

verify sits on the critical path of the edge request (synchronously awaited before origin fetch) — your latency adds directly to what end users / crawlers experience:

• Suggested targets: p50 < 100 ms, p99 < 500 ms including network round-trip; deploy endpoints close to ESA regions;
• Keep the hot path lean: indexed rule matching, cached identification, no synchronous external calls inside verify;
• Pure-decision verdicts (allow / block / parameter errors) should return in the ~10 ms range.

14.2 Failure & timeout semantics

• ESA applies its own timeout and failure policy to verify calls (fail-open pass-through vs. fail-close block, per ESA product configuration). You cannot assume what happens when you are down — the only thing you control is returning fast and unambiguously;
• On internal faults return 500 + INTERNAL_ERROR with accessGranted:false — never hang the connection;
• Keep 5xx messages generic; internals go to your own logs only.

14.3 X-Request-Id & reconciliation

POST /v1/verify
X-Request-Id: req-8c85ba19abef566b     ← optionally sent by ESA
...
HTTP/1.1 200 OK
X-Request-Id: req-8c85ba19abef566b     ← you echo it; generate one if ESA didn't send it

• The id runs through your access logs and both ledger entries, joining with ESA-side logs per transaction;
• Suggested convention: retries reuse the same id — a ready-made hook for future idempotent charging (same id never charges twice), so network retries can't double-bill.

14.4 Security checklist

• gateway_key and crawler tokens travel over HTTPS only; keys support rotation (§4);
• Crawler token and account session credentials are separate (§10.2);
• Trial credit requires signup rate-limiting (per IP / email) against farming;
• Basic rate limiting and anomaly monitoring on the verify endpoint (against forged "ESA" traffic inflating your ledgers);
• Store and compare money as fixed-point decimals (DECIMAL), never floats.

14.5 Suggested launch path

1. Self-test all 12 return values of the §13.2 matrix in your test environment;
2. Provide a test Endpoint / keys and complete joint debugging with ESA (ESA triggers the real edge chain with a test site); submit a Provider Profile if you want the dropdown listing;
3. Canary: enable on a test site first; watch latency / verdict distribution / reconciliation consistency;
4. Enter the console provider list for general availability.