feat(core): multi-batch external merkle signing + spill-backed prepared uploads

[Nic-dorman]

Linear issue

V2-946

Implements ADR-0003. Builds on #167 (its commit is included until it merges; the diff collapses after).

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

• Wire: none — nodes see identical per-tree payment records and PUT bundles; the payment contract is untouched (the wallet path already pays N trees in N transactions).
• Storage: none.
• API: breaking (0.6.0 window, coordinated consumers per ADR-0003):
  • ExternalPaymentInfo::Merkle now carries prepared_batches: Vec<PreparedMerkleBatch> and an opaque on-disk ExternalChunkStore (replacing prepared_batch + resident chunk_contents: Vec<Bytes>).
  • New Client::finalize_upload_merkle_multi(_with_progress) takes one Option<[u8; 32]> winner hash per batch; finalize_upload_merkle remains as the single-batch special case and now refuses multi-batch uploads with a pointer to the multi API.
  • New Client::file_prepare_upload_with_mode (additive); ClientConfig gains merkle_external_batch_cap (additive, Default covers it).
  • Desktop (V2-948) and FFI (V2-947) updates are tracked and blocked on this PR; both keep working unchanged for ≤256-chunk uploads via the single-batch wrapper after recompiling against the new variant shape.

Semver impact

• breaking
• feature
• fix

Test evidence

• Unit (435/435 pass): partition caps incl. the cap-3 floor proof over totals 2..200 (a cap of 2 cannot partition odd totals — caught during test design and fixed by clamping to 3..=MAX_LEAVES); per-batch receipt fold (proof union, cost sum, oldest-timestamp expiry semantics); winner-hash validation (count mismatch, all-unpaid, k-of-N mix).
• E2E (new, in the Merkle E2E CI job): test_external_merkle_multi_batch_round_trip — 500 KB public upload, forced merkle, cap 3 → genuine [2, 2] partition → two Wallet::pay_for_merkle_tree calls as the simulated signer → finalize_upload_merkle_multi → retrieval via the public DataMap address → byte equality. test_external_merkle_partial_payment_is_partial_upload — pay 1 of 2 batches → PartialUpload { stored_count: 2, failed_count: 2 }, proving k-of-N forward progress. Both pass locally against real 35-node testnets + Anvil: 2 passed; 0 failed in 182 s total.
• Real-size devnet run at the DEFAULT batch cap (no test seam; examples/external-merkle-large.rs, devnet-gated, in this PR): a 1228 MiB incompressible file against a 25-node LocalDevnet of real ant-node 0.14.3 processes + Anvil — prepare produced 312 chunks as 2 sub-batches in 12.5 s with 441 MiB RSS (pre-ADR-0003 this held ≥ 1.2 GiB resident through the whole signing window); two on-chain payments (depth 8 → 3 ANT, depth 6 → 0.75 ANT — the 256:64 padded-leaf ratio visible in the settlement); 312/312 chunks stored, 0 failed, in 43 s (peak 1056 MiB during the store fan-out); download via the public DataMap address verified byte-identical; 89 s end-to-end. (Overall process peak 2.7 GiB was in the download/verify phase, which this PR does not touch — noted as a pre-existing observation.)
• Bonus fix found by the harness: LocalDevnet::create_funded_client connected with allow_loopback: false, so the loopback-devnet convenience client could never reach its own devnet (empty routing table → InsufficientPeers on the first witnessed lookup). Fixed in this PR.
• Cross-machine LAN run (harness MANIFEST mode; nodes on a separate Linux box via ant-devnet --nodes 25 --host <lan-ip> --serve-port 8088 --enable-evm, v0.17.0 tag): a 2200 MiB file from a macOS client → 555 chunks as 3 sub-batches ([256, 256, 43→64], depths 8/8/6 — the depth-8 single-tree limit crossed twice), 3 payments (3 + 3 + 0.75 ANT), 555/555 stored, 0 failed in 8.4 min over real QUIC (~9 GiB egress at 4 client-sent copies/chunk), prepare RSS 346 MiB for the 2.2 GiB file. Then a second Linux machine, knowing only the public address and size (harness MODE=download), fetched the DataMap and byte-verified the full 2.2 GiB in 40 s — cross-device retrievability of a multi-batch external upload.
• Observations parked from the LAN run (pre-existing, not this PR): store-phase client RSS spiked to ~3.6 GiB against a slow sink (transport send queues; loopback peaked ~1 GiB — the spill reads themselves stay bounded), and ant-devnet --enable-evm --host publishes a manifest whose Anvil rpc_url is loopback-only (worked around with an SSH tunnel; external signers on other devices can't pay without it).
• cargo clippy --all-targets --all-features -- -D warnings and cargo fmt --all -- --check clean.
• Merkle E2E job budget raised 40 → 60 min: the job now runs 6 tests (was 4), and the 40-min cap was already grazed at 4 tests (the timeout flake on this repo's PR fix(core): surface partial external-signer merkle uploads as PartialUpload #167 first run).

New dependency

none

ADR

ADR-0003: Multi-Batch External Merkle Signing with Spill-Backed Prepared Uploads (in this PR; Proposed — needs human acceptance)

Mitigation / rollback

Single-commit revert restores the single-batch external API (the #166 fix in #167 is a separate commit and survives a revert of this one). No stored data or on-chain state depends on the new shape — a prepared upload is ephemeral session state, so rollback only affects not-yet-finalized sessions.

Notes

• Peak client RAM for external merkle uploads drops from ≈ file size (held across the entire signing window — the documented "NOT memory-bounded" path) to the wallet path's ~256 MB bound; the external store also gains the spill engine's deferred retry rounds.
• The spill directory now lives as long as the PreparedUpload (disk ≈ 1.05× file until finalize/cancel); leaked sessions fall to the existing stale-spill reaper.
• Wave-batch external (< 64 chunks) keeps resident bodies this pass — noted as follow-up in the ADR.
• Consumer UX guidance (single ERC-20 allowance + "payment i of N") lives in V2-948; contract-level one-tx batching is deferred to V2-949.

🤖 Generated with Claude Code

[dirvine]

Panel review (6 seats: correctness, storage/lifecycle/concurrency, API/compatibility, tests/operations, security, and independent GLM-5.2), against head 4595737c2acd6ef0bd1ab1a4d96e1575a4109e57.

Verdict: no material code blocker found. The panel found the spill ownership/cleanup, bounded read/store concurrency, multi-batch payment/proof merge, and partial-failure accounting coherent. #168 also contains the #167 fix.

One actionable documentation correction:

• ant-core/src/data/client/merkle.rs:718 says cap is clamped to 2..=MAX_LEAVES, but merkle_batch_sizes_with_cap and Client::merkle_external_batch_cap both correctly clamp to 3..=MAX_LEAVES. Please change the public method documentation to 3..=MAX_LEAVES.

Non-blocking test gap:

• #168 removes #167's unit test for an all-paid external upload that later falls short of storage quorum. The new E2E coverage exercises a missing/unpaid batch, which is a different path. A focused assertion that quorum shortfall after deferred retries returns PartialUpload and that stored + failed == total would preserve that regression coverage.

Verification performed:

• git diff --check, formatting and cargo check -p ant-core: passed
• focused external-Merkle/batch-size tests: 7 passed
• independent seat additionally built ant-core --tests and the new example, ran Merkle/file unit suites and both new external-Merkle E2Es: passed
• current GitHub build, unit, Clippy, format, docs, security and ADR validation: passed; Merkle/general E2E jobs remain pending

Dissent checked: the GLM seat initially reported that the breaking external-signer field changes lacked a changelog entry. That finding is not valid: CHANGELOG.md:10-12 explicitly records prepared_batches and ExternalChunkStore replacing chunk_contents.

Governance note: ADR-0003 is still Proposed; CI confirms its format/immutability rules, but it is not accepted guidance until maintainers record human acceptance.

[Nic-dorman]

Both review items addressed in db82913:

• Doc correction: prepare_merkle_batches_external's cap doc now says 3..=MAX_LEAVES (with a pointer to merkle_batch_sizes_with_cap for the floor rationale), matching the actual clamps.
• Test gap: restored the fix(core): surface partial external-signer merkle uploads as PartialUpload #167-level regression at the seam the spill path composes — one single-attempt store pass followed by the deferred rounds — asserting an all-paid quorum shortfall survives every retry with stored + failed == total, fatal: None, and failed_addresses exactly the shortfall set (the fold upload_merkle_from_spill turns into Error::PartialUpload). Chose the engine seam over reinstating the deleted fold test because the fold itself no longer exists — the spill store engine is now the single accounting authority for both paid-but-short (this test) and unpaid-batch (the E2E) partials, without needing a network in either case.

The branch also carries the post-#165 lockfile via the main merge, so the Security Audit red is cleared natively on this run.

On the governance note: agreed — ADR-0003 stays Proposed pending human acceptance; nothing in this PR treats it as accepted.