#345: narrow HEAD+Range shim so DuckDB-WASM uses range requests (74 MB -> 3 MB cold load) - #348
Open
rdhyee wants to merge 3 commits into
Open
#345: narrow HEAD+Range shim so DuckDB-WASM uses range requests (74 MB -> 3 MB cold load)#348rdhyee wants to merge 3 commits into
rdhyee wants to merge 3 commits into
Conversation
…uests again
THE PROBLEM. A cold load of the Explorer transfers ~74 MB; it should transfer
~3 MB. DuckDB-WASM 1.24.0 (the version Quarto's OJS runtime pins) decides
whether a server supports partial reads by sending exactly
`HEAD` + `Range: bytes=0-` and requiring a 206. We answer 200, so it logs
"falling back to full HTTP read" and downloads WHOLE FILES — including
samples_map_lite_v3.parquet (62.9 MB) when it needs ~1.5 MB of it.
Measured on production: facet panel at 168s on 4G, 423s on 3G, and NEVER within
10 minutes on slow 3G.
THIS IS A WORKAROUND, NOT A BUG FIX — and the code says so. RFC 9110 §14.2 is
explicit that Range is defined only for GET and servers MUST IGNORE it on other
methods, so our 200 was CORRECT and DuckDB's probe is the nonconforming party.
Codex talked me out of framing this as a fix, and out of the wider version I
first wrote.
SCOPE is as tight as it can be:
- ONLY the exact probe shape `bytes=0-` gets 206
- every other ranged HEAD (bytes=0-99, bytes=100-199, bytes=-100) stays
standards-correct at 200, so the divergence cannot leak to other clients or
harden into an accidental contract
- a REMOVAL PATH is named in the comment: delete this when the Explorer stops
using Quarto's pinned duckdb-wasm and does its own init on a conformant
version
EVIDENCE. Controlled A/B with a transparent proxy that changed only this one
behaviour: 74,202,598 B -> 3,341,812 B cold, 8 full-read fallbacks -> 0, and on
3G the facet panel 440.6s -> 94.2s (4.7x).
Warm cache measured too, since it could have reversed the recommendation: the
current whole-file path caches perfectly (0 bytes on a repeat visit) while the
ranged path re-fetches ~1.47 MB. So this trades ~71 MB on first visit for
~1.47 MB per revisit, with no warm time regression (4.2s either way).
Adds test/range_contract.sh, a 23-assertion HTTP contract test run against
`wrangler dev --local` with a seeded local R2. Against the UNFIXED Worker it
fails 3/20, all of them HEAD+Range. With this change: 23/23, including four new
assertions that specifically prove the shim does NOT widen.
NOT YET VERIFIED ON THE REAL EDGE. Everything above is a local wrangler Worker
plus a Python proxy on localhost. Deploy to a NON-PRODUCTION workers.dev URL and
canary it before touching the data.isamples.org route — see the PR body.
Refs isamplesorg#345, isamplesorg#313
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa
…samplesorg#345) The 74 MB regression was invisible to every existing test, and was actively MIS-CLEARED in June (ISSUE_313_FINDINGS_2026-06-26.md) by a curl probe that used GET where DuckDB uses HEAD. This is the gate that catches that class. Deliberately cheap: no browser, no DuckDB, a handful of bytes on the wire, 2.5s total. Designed to run BEFORE the Playwright smoke gate so the failure is caught before anything downloads 74 MB to discover it. Asserts: - ranged GET returns 206 with a Content-Range matching the release manifest (passes today, always has — which is exactly why it was not sufficient alone) - CORS exposes Content-Range/Accept-Ranges/Content-Length, without which cross-origin JS cannot read them - DuckDB-WASM 1.24.0's HEAD+Range probe returns 206 — marked xfail until the isamplesorg#345 Worker shim deploys; remove the marker then - the isamplesorg#345 shim does NOT widen: other ranged HEADs must stay standards-correct at 200 with no Content-Range, so a knowing divergence cannot leak into an unintended contract - manifest sizes match what the origin actually serves (the data/doc drift class) The HEAD test carries a long comment explaining it encodes a DELIBERATE standards divergence (RFC 9110 section 14.2 says servers MUST ignore Range on non-GET), and names its own deletion condition. A future reader should not mistake it for correct HTTP. A note on my own first draft: it reported a 404 from a partially-seeded test origin as 'the shim has widened' — a confidently wrong diagnosis, which is the precise failure mode this file exists to prevent. It now skips instead. Design developed with Codex, which argued for bytes-and-protocol-shape as the blocking signal and wall-clock timings as telemetry only. Refs isamplesorg#345, isamplesorg#313 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa
Deploying this shim is the risky part: the production wrangler.toml binds the Worker to the data.isamples.org/* ROUTE, and there is no staging data host — so a plain `wrangler deploy` goes live for every consumer of that hostname immediately, including collaborators. The earlier instruction was 'comment out the routes block first', which is a bad instruction: easy to forget, easy to half-revert, and a mistake is a production incident. Replaced with a SEPARATE wrangler.canary.toml (different name, no routes key at all) so the production config is never read or modified. deploy-canary.sh does the whole loop and refuses to hand over a URL it has not verified: - checks wrangler auth and prints the exact login command if missing - deploys with the canary config only - recovers the workers.dev URL from the deploy output - asserts HEAD 'bytes=0-' -> 206 with the correct Content-Range - asserts the shim did NOT widen: bytes=0-99 / 100-199 / -100 stay 200 - asserts ranged and plain GET are unchanged - exits non-zero with 'do not promote' if any check fails - prints the ready-to-paste staging Explorer URL and measurement command - --verify re-checks an existing canary; --teardown deletes it Validated: `wrangler deploy -c wrangler.canary.toml --dry-run` parses, lists the R2 binding, and shows no routes; the script correctly detects the unauthenticated state rather than failing obscurely. Refs isamplesorg#345 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the 74 MB cold load in #345 — but read the framing, because this is a deliberate standards divergence, not a bug fix.
What changes
DuckDB-WASM 1.24.0 probes for range support with exactly
HEAD+Range: bytes=0-and requires a206. We answer200, so it logs "falling back to full HTTP read" and downloads whole files — 62.9 MB ofsamples_map_lite_v3.parquetwhen it needs ~1.5 MB.This makes only that exact probe return
206. Every other rangedHEADstays200.Why it's framed as a shim
RFC 9110 §14.2:
Rangeis defined only forGET; servers MUST ignore it on other methods. Our200was correct. DuckDB's probe is the nonconforming party. So this is a compatibility measure with a named removal path — delete it when the Explorer stops using Quarto's pinned duckdb-wasm and does its own init on a conformant version.Codex talked me out of calling this a fix, and out of the wider version I first wrote (which would have answered 206 to any ranged HEAD).
Evidence
Controlled A/B — a transparent proxy changing only this one behaviour:
So: ~71 MB saved on first visit, ~1.47 MB cost per repeat visit, no warm time regression. The warm number is included because it could have reversed the recommendation — and the original issue omitted it.
Test coverage
test/range_contract.sh— 23 assertions againstwrangler dev --localwith a seeded local R2:HEAD+RangeFour assertions exist specifically to prove the shim doesn't widen (
bytes=0-99,bytes=100-199,bytes=-100must all stay200). Caching, CORS, ETag-stability and rangedGETare all asserted unchanged — this Worker exists for caching, so that contract is guarded.There is no staging data host. This Worker serves
data.isamples.orgon a route, so a plainwrangler deploygoes straight to production for every consumer.Safe path — deploy to a workers.dev URL instead, then point the staging Explorer at it:
Only promote to the
data.isamples.orgroute once the canary shows 206s and reduced bytes on the real edge.What is NOT verified
Everything measured here is a local
wrangler devWorker plus a Python proxy overhttp://localhost— not the real Worker on the real Cloudflare edge. That canary is the gap.Also relevant: Workers Caching appears disabled today (production returns no
CF-Cache-Status, and the Worker demonstrably still seesRange). If it is ever enabled, Cloudflare stripsRange, caches a full200and slices its own206s — which would replace this path entirely and require re-testing.Refs #345, #313
🤖 Generated with Claude Code
https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa