Explorer downloads ~74 MB cold: DuckDB-WASM 1.24.0's HEAD+Range probe expects 206; our (correct) 200 defeats range requests

[rdhyee]

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: overnight work on making isamples.org more robust at scale. What I did: measured the Explorer across connection speeds and browsers, root-caused a large cold-load cost, and proved the cause with a controlled A/B. Filing because the fix lives in the Cloudflare Worker, not this repo.

Summary

A cold load of the Interactive Explorer transfers ~74 MB before the facet panel appears. Our own docs claim "typically less than 1 MB for initial exploration" (index.qmd) and "only the bytes you need are transferred" (explorer.qmd). Both are wrong by roughly two orders of magnitude.

The whole thing comes down to one status code.

DuckDB-WASM decides whether a server supports partial downloads by sending a HEAD request with a Range header. data.isamples.org answers 200 instead of 206, so DuckDB concludes the server can't do partial reads and downloads every file whole — including samples_map_lite_v3.parquet (62.9 MB) when it actually needs about 1.5 MB of it.

Proposed fix

In the Cloudflare Worker fronting data.isamples.org: when a HEAD carries a Range header and the object supports ranges, respond 206 with Content-Range (no body, per HEAD semantics) rather than 200.

Content-Range is already in Access-Control-Expose-Headers, so nothing else needs to change.

Evidence — controlled A/B, not inference

A transparent reverse proxy forwarded every request to the real host unchanged, altering exactly one thing: HEAD+Range → 206. Same build, same cold cache, one variable.

Unthrottled

	Control (200)	Treatment (206)
Bytes from data host	74,202,598	3,341,812
full HTTP read fallbacks	8	0
samples_map_lite_v3.parquet	62,924,115 B (whole file)	1,467,731 B (8 ranged reads)
sample_facet_masks.parquet	10,138,648 B (whole file)	767,000 B (48 ranged reads)

3G fast (1.6 Mbps) — the user-visible result

	Control	Treatment
Globe drawn	54.0 s	54.1 s
Facet panel	440.6 s	94.2 s
Bytes	74.2 MB	3.0 MB

4.7× faster to a usable filter panel, 24× less data. Globe and table are unchanged — they read the small H3 summary, not the big files.

Why this went unnoticed

I cleared the server three times before finding it, because I kept testing with GET:

Probe	Result
GET + Range	206 + Content-Range ✅
HEAD + Range	200, no Content-Range ❌ ← what DuckDB actually sends
OPTIONS preflight	204, allows Range ✅
CORS exposure of Accept-Ranges	correct ✅

Everything was right except the one verb that mattered. Please verify any fix with HEAD, not GET.

Worth noting this is not an RFC violation — raw.githubusercontent.com answers 200 to HEAD+Range too. It's about interoperating with DuckDB-WASM's capability probe, which is by far this bucket's most important client.

Also answers #313

The same harness answers @akthom's #313, open since June and never done.

Production, cold cache, desktop:

Connection	Globe	Facet panel
Unthrottled	2.2 s	10.8 s
4G	15.4 s	168.7 s
3G fast	38.7 s	423.1 s
3G slow	156.6 s	never (>600 s)

On a slow connection the facet panel never appears at all.

Cross-browser (good news): works in Chromium, Firefox and WebKit/Safari, desktop and mobile viewports, zero uncaught page errors, comparable timings. The 74 MB is identical in all three, so this is not a browser quirk.

Reproducing

Both instruments are committed in perf/313-bandwidth-findings on the rdhyee fork:

• tests/playwright/bandwidth_matrix.py — the measurement harness
• tests/playwright/range_fix_proxy.py — the A/B proxy that isolates the cause

Full write-up with method notes and limitations: PERF_BANDWIDTH_FINDINGS_2026-08-06.md on that branch.

After the fix

index.qmd's "less than 1 MB" and explorer.qmd's "only the bytes you need" should be updated to the measured figure (~3.3 MB cold) — still a good story, and an honest one.

---

Caveat on the numbers: single runs on one machine and uplink, DevTools throttling rather than a real cellular link, cold-cache first visits only. Order-of-magnitude, not benchmarks. The A/B comparison is the reliable part, since both arms ran under identical conditions.

[rdhyee]

🤖 rbotyee (Claude, operated by @rdhyee) — Raymond's intent: he asked "is this a slam dunk?", so I stress-tested my own finding with Codex instead of defending it. What I did: three corrections to the issue above, one of them to a claim that was simply wrong, plus the warm-cache measurement that was missing.

Short version: the mechanism holds and the effect is large, but the framing above is wrong in two important ways, and I owe a correction to a colleague's prior work.

---

Correction 1 — I was wrong that #313 had never been investigated

ISSUE_313_FINDINGS_2026-06-26.md is a real prior investigation. It raised this exact hypothesis (its "H2 — DuckDB-WASM full-HTTP-read fallback") two months ago, and dismissed it with:

"The server does correctly respond 206 for range requests (curl confirmed)"

That is the same GET-versus-HEAD mistake I made, and it sent that investigation off toward Firefox HTTP/2 connection-pooling theories instead. So this finding isn't new ground — it's the resolution of a hypothesis that was correctly identified in June and mis-cleared by a probe that used the wrong verb. That prior work deserves the credit for finding the right suspect.

Correction 2 — the proposed fix is NOT standards-correct, and I should not have called it a fix

RFC 9110 §14.2 is unambiguous: Range is defined only for GET, and a server MUST ignore Range on other methods, including HEAD.

So the Worker's current HEAD+Range → 200 is correct. DuckDB-WASM 1.24.0's capability probe is the nonconforming party. What I proposed is an interoperability workaround, and the issue title/body above frame it as a bug fix. That's wrong and I'd have misled anyone who acted on it.

Consequences for the proposal, all adopted:

• Scope the shim as narrowly as possible. Answer 206 only to the exact probe observed (Range: bytes=0-). Every other ranged HEAD stays standards-correct at 200, so the divergence cannot leak to other clients.
• Ship it with a removal path, not as a permanent contract.
• Say "DuckDB-WASM 1.24.0's probe", not DuckDB-WASM generally.

Correction 3 — the warm-cache trade-off, which the issue omitted entirely

The 74 MB figure is a cold load. Today those files carry Cache-Control: max-age=1yr, immutable, so repeat visitors already pay very little. Measured properly (fresh profile per arm, full browser close + relaunch between visits, passive instrumentation — Playwright's route() disables the HTTP cache and would have destroyed the thing under test):

	Control (200 / full reads)	Treatment (206 / ranged)
Cold bytes	74,202,598	3,341,812
Warm bytes (relaunch)	0	1,467,731
Warm time-to-facets	4.2 s	4.2 s

So there is a real warm-cache cost: ~1.47 MB per repeat visit, against ~71 MB saved on the first. The illustrative byte crossover is around 48 revisits. Treat that as a crossover under the measured visit-2 behaviour, not a user break-even — visits 3+ weren't measured, and the byte counts come from Playwright rather than the proxy's own network accounting.

No warm time regression (4.2 s either way).

A hypothesis we tested and killed

Codex predicted from Chromium source that a repeat HEAD+Range might be served from the sparse cache entry and normalised 206→200 by FixHeadersForHead(), re-triggering the full-read fallback and making the treatment much worse warm. Falsified: on the warm relaunch every HEAD+Range was observed as 206, with zero fallback logs. Worth recording because it was the most plausible way this could have backfired.

What else changed

• Reproduced against the real Worker, not just my proxy: wrangler dev --local with a seeded local R2 and a contract test (workers/data-isamples-org/test/range_contract.sh). Unfixed Worker: 17 pass / 3 fail, all three being HEAD+Range. Caching, CORS, ETag stability and ranged GET all pass.
• The standards-correct client-side alternative does not work on our pinned version. DuckDBClient.of(sources, config) does pass through to db.open(config), so it was a one-line change. On Quarto's pinned duckdb-wasm 1.24.0: reliableHeadRequests: false had no effect (still 73.6 MB, 8 fallbacks); adding allowFullHTTPReads: false dropped bytes but broke the app — facets never render, because DuckDB can neither range nor full-read.
• Workers Caching appears disabled. Production returns no CF-Cache-Status, and the Worker demonstrably still sees Range (it computes content-range itself). If it is ever enabled, Cloudflare strips Range, caches a full 200 and slices 206s itself — which would replace/bypass the Worker's manual range path, and the nonstandard HEAD probe would need re-testing under that configuration.

Where this leaves it

Option	Works?	Standards-correct?
Narrow Worker shim (Range: bytes=0- → 206)	✅ proven	❌ knowing divergence
Client reliableHeadRequests on 1.24.0	❌ no effect / breaks app	✅
Upgrade duckdb-wasm (Explorer does its own init)	untested	✅ — the removal path

Recommendation: ship the narrow shim as a documented, reversible compatibility measure, with the duckdb-wasm upgrade as the removal path. ~71 MB saved on first visit and a usable filter panel 4.7× sooner on 3G is worth 1.47 MB per revisit and no warm time cost — but it should be labelled honestly as a shim for one client's nonconforming probe, not as fixing a server bug.

Before deploying, Codex recommends a real-Cloudflare-edge canary: everything above was measured on headless Chromium over http://localhost through a Python proxy, and the local wrangler dev Worker test — not the real Worker on the real edge.

---

Excluded from the headline: one control run reported 143.6 MB cold rather than 74.2 MB, because sample_facets_v4 (69 MB) crossed the measurement boundary asymmetrically between arms. That weakens that single run, not the mechanism — but I'm not using it affirmatively either.