From ce5fc360a2f0a52479be0cfe0a7da8d7e2a8efba Mon Sep 17 00:00:00 2001 From: Raymond Yee Date: Wed, 5 Aug 2026 23:55:22 -0700 Subject: [PATCH] perf: measure the Explorer across bandwidth/browsers; root-cause a 74 MB cold load MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Answers #313 (open since 2026-06-26, never done: what browser/OS/bandwidth combinations does the Explorer work in?) and root-causes the 'falling back to full HTTP read' finding from the 2026-08-05 coherence audit. HEADLINE: a cold load of the default world view transfers ~74 MB. The site's own docs claim 'typically less than 1 MB for initial exploration'. Wrong by ~2 orders of magnitude. ROOT CAUSE: DuckDB-WASM probes range support with a HEAD carrying a Range header. data.isamples.org answers 200 instead of 206, so DuckDB concludes the server cannot do partial reads and downloads every file whole — including samples_map_lite_v3.parquet (62.9 MB) when it needs ~1.5 MB of it. PROVEN, not inferred. A transparent reverse proxy forwarded everything unchanged except HEAD+Range -> 206. Same build, same cold cache, one variable: unthrottled 74,202,598 B -> 3,341,812 B; 8 full-read fallbacks -> 0 3g-fast facet panel 440.6s -> 94.2s (4.7x), 74.2 MB -> 3.0 MB Fix is server-side in the Cloudflare Worker; no application code changes. WHY IT WAS MISSED: the server was cleared three times by curl tests that used GET. GET+Range correctly returns 206; HEAD+Range returns 200. Only the verb DuckDB actually sends was wrong. Documented so the next person checks HEAD. #313 RESULTS (production, cold cache, desktop): unthrottled globe 2.2s facets 10.8s 4g globe 15.4s facets 168.7s 3g-fast globe 38.7s facets 423.1s 3g-slow globe 156.6s facets NEVER (>600s budget) Cross-browser: works in Chromium, Firefox AND WebKit, desktop and mobile, zero uncaught page errors, comparable timings — that had been an open unknown. The 74 MB is identical in all three, so it is not a browser quirk. Adds two committed, reproducible instruments: tests/playwright/bandwidth_matrix.py — the measurement harness tests/playwright/range_fix_proxy.py — the A/B proxy that isolates the cause INSTRUMENTATION TRAPS documented in the report, both of which produced confidently wrong results before being caught: - page-level CDP does NOT see Web Worker traffic. DuckDB runs in a worker, so the page session reported 568 KB / 3 requests — a textbook 'only the bytes you need' result, off by 130x. Real accounting uses context-level events. - the instrument changed the measurement twice: waitForDebuggerOnStart paused workers that were never resumed (globe never rendered), and route interception inflated time-to-globe from 2.5s to 7.3s. Final harness is passive. No application code changed. No fix applied yet — the Worker change is not in this repo and wants RY's call. Refs #313 Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01QCCDurpcLzMe7L72y2HDAa --- PERF_BANDWIDTH_FINDINGS_2026-08-06.md | 173 +++++++++++++++ tests/playwright/bandwidth_matrix.py | 294 ++++++++++++++++++++++++++ tests/playwright/range_fix_proxy.py | 134 ++++++++++++ 3 files changed, 601 insertions(+) create mode 100644 PERF_BANDWIDTH_FINDINGS_2026-08-06.md create mode 100644 tests/playwright/bandwidth_matrix.py create mode 100644 tests/playwright/range_fix_proxy.py diff --git a/PERF_BANDWIDTH_FINDINGS_2026-08-06.md b/PERF_BANDWIDTH_FINDINGS_2026-08-06.md new file mode 100644 index 0000000..79a3256 --- /dev/null +++ b/PERF_BANDWIDTH_FINDINGS_2026-08-06.md @@ -0,0 +1,173 @@ +# The Explorer downloads 74 MB to show you a globe — and why + +**Date:** 2026-08-06 · **Status:** root-caused and proven by controlled experiment; fix not yet applied +**Harness:** `tests/playwright/bandwidth_matrix.py` (reproducible, committed alongside this doc) + +--- + +## Plain English + +Opening the Interactive Explorer with an empty cache transfers **about 74 MB** before the +facet panel appears. The site's own documentation says *"only the data you need is +downloaded — typically less than 1 MB for initial exploration."* That claim is wrong by +roughly two orders of magnitude. + +The cause is **one HTTP status code**. DuckDB-WASM checks whether a server supports partial +downloads by sending a `HEAD` request with a `Range` header. Our data host answers `200` +instead of `206`, so DuckDB concludes "this server can't do partial reads" and downloads +**every file whole** — including a 63 MB one it only needs about 1.5 MB of. + +Changing that single response to `206` cuts the cold load from **74 MB to 3.3 MB** and makes +the facet panel appear **4.7× sooner on a 3G connection**. The fix is server-side, in the +Cloudflare Worker in front of the bucket. No application code changes. + +--- + +## The measurements + +### Time to a usable Explorer, by connection (production, cold cache, desktop) + +| Connection | Globe drawn | **Facet panel** | Table rows | +|---|---:|---:|---:| +| Unthrottled | 2.2 s | 10.8 s | 1.2 s | +| 4G (4 Mbps) | 15.4 s | **168.7 s** | 7.3 s | +| 3G fast (1.6 Mbps) | 38.7 s | **423.1 s** | 17.6 s | +| 3G slow (400 kbps) | 156.6 s | **never — not reached in 600 s** | 67.7 s | + +On a slow connection the facet panel — the Explorer's main filtering affordance — **never +appears at all** within ten minutes. + +### Cross-browser and viewport (unthrottled) + +| Browser | Viewport | Globe | Facets | Bytes | Page errors | +|---|---|---:|---:|---:|---:| +| Chromium | desktop | 2.4 s | 7.6 s | 73.7 MB | 0 | +| Chromium | mobile | 2.3 s | 7.5 s | 73.7 MB | 0 | +| Firefox | desktop | 2.2 s | 6.3 s | 73.6 MB | 0 | +| Firefox | mobile | 2.2 s | 5.3 s | 73.6 MB | 0 | +| WebKit (Safari) | desktop | 2.4 s | 7.5 s | 73.7 MB | 0 | +| WebKit (Safari) | mobile | 2.3 s | 7.4 s | 73.6 MB | 0 | + +**Good news for [#313](https://github.com/isamplesorg/isamplesorg.github.io/issues/313):** the +Explorer works in all three engines, desktop and mobile, with no uncaught errors and +comparable timings. That had been an open unknown since 2026-06-26. + +**Bad news:** the 74 MB is identical in all three, so this is not a browser quirk. + +--- + +## Root cause, and the experiment that proved it + +DuckDB-WASM's per-file handshake, observed on production: + +``` +HEAD range=None -> 200 +HEAD range=bytes=0- -> 200 <-- a range-capable server answers 206 +GET range=None -> 200 <-- full 62,924,115 bytes +``` + +followed by `falling back to full HTTP read for: …` in the console, for eight files. + +**A caution about how this was diagnosed.** The server was cleared *three times* on the basis +of `curl` tests before the real problem was found, because those tests used `GET`: + +| Probe | Result | +|---|---| +| `GET` + `Range` | **206** + `Content-Range` ✅ | +| `HEAD` + `Range` | **200**, no `Content-Range` ❌ ← what DuckDB actually sends | +| `OPTIONS` preflight | 204, `Access-Control-Allow-Headers: Range` ✅ | +| `HEAD` CORS exposure | `Accept-Ranges` correctly exposed ✅ | + +Everything was right except the one verb that mattered. + +### Controlled A/B + +A transparent reverse proxy forwarded every request to `data.isamples.org` unchanged, altering +**exactly one thing**: a `HEAD` carrying `Range` returned `206` + `Content-Range`. The Explorer +was served locally and pointed at each proxy via `?data_base=`. Same build, same cold cache, +same everything else. + +**Unthrottled** + +| | Control (`HEAD`+Range → 200) | Treatment (→ 206) | +|---|---:|---:| +| Bytes from data host | **74,202,598** | **3,341,812** | +| `full HTTP read` fallbacks | 8 | **0** | +| `samples_map_lite_v3.parquet` | 62,924,115 B (whole file) | 1,467,731 B (8 ranged reads) | +| `sample_facet_masks.parquet` | 10,138,648 B (whole file) | 767,000 B (48 ranged reads) | + +**3G fast — the user-visible result** + +| | Control | Treatment | +|---|---:|---:| +| Globe drawn | 54.0 s | 54.1 s | +| **Facet panel** | **440.6 s** | **94.2 s** | +| Table rows | 30.8 s | 30.7 s | +| Bytes | 74.2 MB | 3.0 MB | + +**4.7× faster to a usable filter panel, 24× less data**, from one status code. + +Globe and table are unchanged because they read the small H3 summary, not the big files. + +--- + +## Recommended fix + +In the Cloudflare Worker fronting `data.isamples.org`: when a `HEAD` request carries a `Range` +header and the object supports ranges, respond **`206`** with `Content-Range` (and no body, as +`HEAD` requires) instead of `200`. + +Notes for whoever implements it: + +- Returning `200` to `HEAD`+`Range` is **not** an RFC violation — `raw.githubusercontent.com` + does the same. This is about interoperating with DuckDB-WASM's capability probe, which is the + single most important client this bucket has. +- `Content-Range` must stay in `Access-Control-Expose-Headers` (it already is) or the browser + cannot read it cross-origin. +- **Verify with `HEAD`, not `GET`.** That mistake cost three false "server is fine" conclusions. +- The A/B proxy (`range_fix_proxy.py`, referenced in the issue) reproduces both arms in minutes. + +### Also worth correcting once fixed + +`index.qmd` claims *"typically less than 1 MB for initial exploration"* and `explorer.qmd` +claims *"only the bytes you need are transferred."* Both are currently false. After the fix the +measured figure is ~3.3 MB cold — still a good story, and an honest one. + +--- + +## Method notes (read before trusting or extending the harness) + +Two instrumentation traps were hit and corrected; both would have produced confidently wrong +conclusions: + +1. **Page-level CDP does not see Web Worker traffic.** DuckDB-WASM runs in a worker. Measuring + only the page session reported **568 KB and 3 requests** — which looks exactly like a healthy + "only the bytes you need" result and is off by a factor of 130. Real accounting uses + context-level `requestfinished` events, which do observe worker requests. +2. **The instrument changed the measurement, twice.** `Target.setAutoAttach` with + `waitForDebuggerOnStart: true` paused workers that were never resumed, so the globe never + rendered and the run looked broken. And `ctx.route(...) + continue_()` interception inflated + time-to-globe from 2.5 s to 7.3 s. The final harness is passive. + +Also: `sizes()` returns `-1` for indeterminate bodies; the harness counts those separately +rather than letting negatives offset real totals (an early version reported a file as −885 bytes). + +**Limits.** Timings are single runs on one machine and one uplink, not medians over repeats — +treat them as order-of-magnitude, not benchmarks. Throttling is Chrome DevTools emulation, not a +real cellular link. Firefox/WebKit runs are unthrottled (CDP is Chromium-only). Only cold-cache +first visits were measured; a returning visitor with a warm cache is a different, much better +story that was not characterised here. + +--- + +## Reproduce + +```bash +# the matrix (production, desktop, four connection profiles) +python tests/playwright/bandwidth_matrix.py https://isamples.org \ + --profiles unthrottled,4g,3g-fast,3g-slow --budget 600 --out /tmp/matrix.json + +# cross-browser +python tests/playwright/bandwidth_matrix.py https://isamples.org \ + --browsers chromium,firefox,webkit --viewports desktop,mobile --profiles unthrottled +``` diff --git a/tests/playwright/bandwidth_matrix.py b/tests/playwright/bandwidth_matrix.py new file mode 100644 index 0000000..1f349f8 --- /dev/null +++ b/tests/playwright/bandwidth_matrix.py @@ -0,0 +1,294 @@ +#!/usr/bin/env python3 +""" +#313-lite — measure the Interactive Explorer across network profiles and viewports. + +Answers two open questions at once: + + #313 (Andrea, open since 2026-06-26, never done): what browser / OS / bandwidth + combinations does the Explorer actually work in? + + The "falling back to full HTTP read" finding (2026-08-05): DuckDB-WASM logs that + it is NOT using HTTP range requests for eight files, including the 63 MB + map-lite parquet — which contradicts the architecture's central claim that + only the bytes you need are transferred. The server was already ruled out + (HEAD 200 + accept-ranges, ranged GET 206, CORS preflight allows Range), so + this measures what the CLIENT actually does. + +Design notes: + * Byte accounting uses CDP Network.loadingFinished `encodedDataLength`, which is + real bytes on the wire (post-compression), not decoded size. + * Every run gets a fresh browser context => cold HTTP cache. This is the honest + first-visit case and the one a reviewer or a new user experiences. + * Runs are budgeted. A run that does not reach a milestone inside the budget is + recorded as NOT REACHED rather than being retried or waited out — "the globe + never appeared within 3 minutes on slow 3G" is a finding, not a failure. + +Usage: + python bandwidth_matrix.py [BASE_URL] [--budget SECONDS] [--profiles a,b] +""" +import argparse, json, statistics, sys, time +from playwright.sync_api import sync_playwright + +# Chrome DevTools-style presets. Values are bytes/sec and milliseconds. +PROFILES = { + "unthrottled": dict(download=-1, upload=-1, latency=0), + "4g": dict(download=4_000_000 / 8, upload=3_000_000 / 8, latency=20), + "3g-fast": dict(download=1_600_000 / 8, upload=750_000 / 8, latency=300), + "3g-slow": dict(download=400_000 / 8, upload=400_000 / 8, latency=2000), +} + +VIEWPORTS = { + "desktop": dict(width=1440, height=900), + "mobile": dict(width=390, height=844), # iPhone 14-ish +} + +MILESTONES = ("cesium_canvas", "globe_drawn", "facet_trees", "table_rows") + + +def probe(page): + """Return which milestones have been reached. Cheap; polled.""" + return page.evaluate("""() => { + const c = document.querySelector('.cesium-viewer .cesium-widget canvas'); + const box = c ? c.getBoundingClientRect() : null; + return { + cesium_canvas: !!c, + globe_drawn: !!(box && box.width > 0 && box.height > 0), + facet_trees: document.querySelectorAll('.facet-treenode').length > 0, + table_rows: document.querySelectorAll('#samplesTable tbody tr, table tbody tr').length > 0, + }; + }""") + + +def run_one(pw, base_url, profile_name, viewport_name, budget_s, query="", browser_name="chromium"): + prof = PROFILES[profile_name] + vp = VIEWPORTS[viewport_name] + + engine = {'chromium': pw.chromium, 'firefox': pw.firefox, 'webkit': pw.webkit}[browser_name] + browser = engine.launch() + ctx = browser.new_context(viewport=vp) # fresh context => cold cache + page = ctx.new_page() + + # CDP (and therefore network throttling) is chromium-only. Firefox/WebKit runs + # are unthrottled by construction; the harness records that rather than + # pretending the profile was applied. + cdp = None + throttled = False + if browser_name == "chromium": + cdp = ctx.new_cdp_session(page) + cdp.send("Network.enable") + cdp.send("Network.emulateNetworkConditions", { + "offline": False, + "downloadThroughput": prof["download"], + "uploadThroughput": prof["upload"], + "latency": prof["latency"], + }) + throttled = profile_name != "unthrottled" + + # --- wire-level accounting ------------------------------------------------- + # CRITICAL: DuckDB-WASM runs in a Web Worker, and a page-level CDP session does + # NOT see worker network traffic. Measuring only the page session reports ~568 KB + # and 3 requests (just the main-frame hits) — which looks like + # a textbook "only the bytes you need" result and is completely wrong. We must + # auto-attach to workers and enable Network on each of their sessions too. + req_url, status_by_id, bytes_by_id = {}, {}, {} + console_errors, page_errors, fallback_logs = [], [], [] + worker_sessions = [] + + def on_req(p): + req_url[p["requestId"]] = p["request"]["url"] + + def on_resp(p): + req_url[p["requestId"]] = p["response"]["url"] + status_by_id[p["requestId"]] = p["response"]["status"] + + def on_done(p): + bytes_by_id[p["requestId"]] = p.get("encodedDataLength", 0) + + def wire(sess): + sess.on("Network.requestWillBeSent", on_req) + sess.on("Network.responseReceived", on_resp) + sess.on("Network.loadingFinished", on_done) + + if cdp: wire(cdp) + + def on_attached(params): + """Attach to each worker target and mirror the Network domain onto it.""" + try: + sid = params["sessionId"] + ws = cdp.session(sid) if hasattr(cdp, "session") else None + if ws is None: + return + worker_sessions.append(ws) + wire(ws) + ws.send("Network.enable") + # Throttling is per-session; workers need their own emulation or they + # would download at full speed while the page is throttled. + ws.send("Network.emulateNetworkConditions", { + "offline": False, + "downloadThroughput": prof["download"], + "uploadThroughput": prof["upload"], + "latency": prof["latency"], + }) + ws.send("Runtime.runIfWaitingForDebugger") + except Exception as e: + fallback_logs.append(f"[worker-attach-failed] {e}") + + if cdp: cdp.on("Target.attachedToTarget", on_attached) + try: + if not cdp: raise RuntimeError("no cdp (non-chromium)") + # waitForDebuggerOnStart MUST be False. With True, workers are paused at + # start and — unless every one is explicitly resumed — DuckDB-WASM never + # boots, so the globe and facets never render. That is the instrument + # changing the measurement: an earlier run reported globe=None purely + # because of this flag. + cdp.send("Target.setAutoAttach", { + "autoAttach": True, "waitForDebuggerOnStart": False, "flatten": True, + }) + except Exception as e: + fallback_logs.append(f"[autoattach-failed] {e}") + + # Real accounting happens here, via PASSIVE context-level request events, which + # (unlike a page-scoped CDP Network domain) do observe dedicated-worker traffic. + # + # Deliberately passive: an earlier version used ctx.route(...)+continue_() and + # the interception round-trip inflated time-to-globe from 2.5s to 7.3s. The + # instrument must not distort the timings it is reporting. + route_hits = {} + + def on_finished(request): + u = request.url + # Match the real data host OR a local proxy standing in for it (used by the + # range-request A/B experiment), so both arms are measured identically. + if not (("data.isamples.org" in u) or ("localhost" in u and ".parquet" in u) + or ("127.0.0.1" in u and ".parquet" in u)): + return + name = u.rsplit("/", 1)[-1].split("?")[0] + e = route_hits.setdefault( + name, {"requests": 0, "with_range_header": 0, "response_bytes": 0, "statuses": {}}) + e["requests"] += 1 + if "range" in {k.lower() for k in request.headers}: + e["with_range_header"] += 1 + # sizes() returns -1 when Playwright cannot determine a body size (small / + # cached / redirected responses). Clamp to 0 and count the occurrences + # rather than letting negatives silently offset real totals — an earlier + # version reported -885 bytes for a file, which is obviously not a size. + try: + n = request.sizes().get("responseBodySize", 0) + if n is None or n < 0: + e["unknown_size_responses"] = e.get("unknown_size_responses", 0) + 1 + else: + e["response_bytes"] += n + except Exception: + e["unknown_size_responses"] = e.get("unknown_size_responses", 0) + 1 + try: + r = request.response() + if r: + e["statuses"][str(r.status)] = e["statuses"].get(str(r.status), 0) + 1 + except Exception: + pass + + ctx.on("requestfinished", on_finished) + + page.on("console", lambda m: ( + console_errors.append(m.text[:200]) if m.type == "error" else + fallback_logs.append(m.text[:200]) if "full HTTP read" in (m.text or "") else None)) + page.on("pageerror", lambda e: page_errors.append(str(e)[:200])) + + # --- run ------------------------------------------------------------------- + reached, t0 = {}, time.time() + try: + page.goto(f"{base_url}/explorer.html{query}", wait_until="commit", timeout=budget_s * 1000) + except Exception as e: + reached["_goto_error"] = str(e)[:160] + + while time.time() - t0 < budget_s: + try: + st = probe(page) + except Exception: + time.sleep(1); continue + for k in MILESTONES: + if st.get(k) and k not in reached: + reached[k] = round(time.time() - t0, 1) + if all(k in reached for k in MILESTONES): + break + time.sleep(1) + + elapsed = round(time.time() - t0, 1) + + # --- aggregate ------------------------------------------------------------- + per_file, total = {}, 0 + ranged = partial = full = 0 + for rid, url in req_url.items(): + n = bytes_by_id.get(rid, 0) + total += n + st = status_by_id.get(rid) + if "data.isamples.org" in url: + name = url.rsplit("/", 1)[-1].split("?")[0] + e = per_file.setdefault(name, {"requests": 0, "bytes": 0, "statuses": {}}) + e["requests"] += 1 + e["bytes"] += n + e["statuses"][str(st)] = e["statuses"].get(str(st), 0) + 1 + if st == 206: ranged += 1 + elif st == 200: full += 1 + else: partial += 1 + + ctx.close(); browser.close() + + return { + "browser": browser_name, + "throttling_applied": throttled, + "profile": profile_name, + "viewport": viewport_name, + "budget_s": budget_s, + "elapsed_s": elapsed, + "milestones": {k: reached.get(k, None) for k in MILESTONES}, + "all_milestones_reached": all(k in reached for k in MILESTONES), + "goto_error": reached.get("_goto_error"), + "total_bytes_all_hosts": total, + "data_host_bytes": sum(v["bytes"] for v in per_file.values()), + "data_host_requests": {"status_206_ranged": ranged, "status_200_full": full, "other": partial}, + "per_file": dict(sorted(per_file.items(), key=lambda kv: -kv[1]["bytes"])), + "worker_sessions_attached": len(worker_sessions), + "worker_aware_total_bytes": sum(v["response_bytes"] for v in route_hits.values()), + "worker_aware_per_file": dict( + sorted(route_hits.items(), key=lambda kv: -kv[1]["response_bytes"])), + "full_read_log_lines": len(fallback_logs), + "diagnostics": fallback_logs[:10], + "console_errors": console_errors[:10], + "page_errors": page_errors[:10], + } + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("base_url", nargs="?", default="https://isamples.org") + ap.add_argument("--budget", type=int, default=180) + ap.add_argument("--profiles", default="unthrottled,4g,3g-fast,3g-slow") + ap.add_argument("--viewports", default="desktop") + ap.add_argument("--browsers", default="chromium") + ap.add_argument("--out", default="/tmp/bandwidth_matrix.json") + ap.add_argument("--query", default="", help="extra query string, e.g. ?data_base=http://localhost:8099") + a = ap.parse_args() + + results = [] + with sync_playwright() as pw: + for bname in a.browsers.split(","): + for vname in a.viewports.split(","): + for pname in a.profiles.split(","): + print(f"--- {bname} / {pname} / {vname} (budget {a.budget}s) ...", flush=True) + r = run_one(pw, a.base_url.rstrip("/"), pname, vname, a.budget, a.query, bname) + results.append(r) + m = r["milestones"] + print(f" globe={m['globe_drawn']}s facets={m['facet_trees']}s " + f"table={m['table_rows']}s bytes={r['data_host_bytes']:,} " + f"206/200={r['data_host_requests']['status_206_ranged']}/" + f"{r['data_host_requests']['status_200_full']}", flush=True) + + payload = {"base_url": a.base_url, "query": a.query, "results": results} + with open(a.out, "w") as f: + json.dump(payload, f, indent=2) + print(f"\nwrote {a.out}") + + +if __name__ == "__main__": + main() diff --git a/tests/playwright/range_fix_proxy.py b/tests/playwright/range_fix_proxy.py new file mode 100644 index 0000000..564840d --- /dev/null +++ b/tests/playwright/range_fix_proxy.py @@ -0,0 +1,134 @@ +#!/usr/bin/env python3 +""" +Controlled experiment for the DuckDB-WASM full-read finding. + +Transparent reverse proxy to data.isamples.org that changes EXACTLY ONE THING: +a HEAD request carrying a Range header gets 206 + Content-Range instead of 200. + +Everything else (GET, ranged GET, bodies, caching headers) is forwarded verbatim. +So if pointing the Explorer at this proxy makes the "falling back to full HTTP +read" behaviour disappear, the HEAD+Range response is the cause. If it does not, +the hypothesis is wrong and something else is triggering the fallback. + +Run: python range_fix_proxy.py [PORT] [--passthrough] +Use: https://isamples.org/explorer.html?data_base=http://localhost:PORT + (--passthrough disables the fix, giving the A-side control run) +""" +import sys, threading, urllib.request, urllib.error +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +UPSTREAM = "https://data.isamples.org" +PASSTHROUGH = "--passthrough" in sys.argv +PORT = next((int(a) for a in sys.argv[1:] if a.isdigit()), 8099) + +CORS = { + "Access-Control-Allow-Origin": "*", + "Access-Control-Allow-Methods": "GET, HEAD, OPTIONS", + "Access-Control-Allow-Headers": "Range", + "Access-Control-Expose-Headers": "Content-Length, Content-Range, Accept-Ranges, ETag", +} + +stats = {"HEAD": 0, "HEAD_ranged": 0, "HEAD_upgraded": 0, "GET": 0, "GET_ranged": 0} +lock = threading.Lock() + + +class H(BaseHTTPRequestHandler): + protocol_version = "HTTP/1.1" + + def log_message(self, *a): + pass + + def _upstream(self, method): + url = UPSTREAM + self.path + req = urllib.request.Request(url, method=method) + # The data.isamples.org Worker 403s the default urllib User-Agent + # (documented gotcha, 2026-07). Identify honestly as a proxy rather than + # spoofing a browser. + req.add_header("User-Agent", + "isamples-range-experiment/1.0 (local proxy; contact @rdhyee)") + req.add_header("Accept", "*/*") + rng = self.headers.get("Range") + if rng: + req.add_header("Range", rng) + try: + return urllib.request.urlopen(req, timeout=60), rng + except urllib.error.HTTPError as e: + return e, rng + + def do_OPTIONS(self): + self.send_response(204) + for k, v in CORS.items(): + self.send_header(k, v) + self.send_header("Content-Length", "0") + self.end_headers() + + def do_HEAD(self): + resp, rng = self._upstream("HEAD") + total = resp.headers.get("Content-Length") + with lock: + stats["HEAD"] += 1 + if rng: + stats["HEAD_ranged"] += 1 + + status = resp.status + extra = {} + # THE ONE CHANGE UNDER TEST. + if rng and not PASSTHROUGH and status == 200 and total: + try: + spec = rng.split("=", 1)[1] + start_s, _, end_s = spec.partition("-") + start = int(start_s or 0) + end = int(end_s) if end_s else int(total) - 1 + status = 206 + extra["Content-Range"] = f"bytes {start}-{end}/{total}" + extra["Content-Length"] = str(end - start + 1) + with lock: + stats["HEAD_upgraded"] += 1 + except Exception: + status = resp.status + + self.send_response(status) + for k, v in resp.headers.items(): + if k.lower() in ("content-length", "content-range", "transfer-encoding", + "connection", "access-control-allow-origin", + "access-control-expose-headers"): + continue + self.send_header(k, v) + for k, v in extra.items(): + self.send_header(k, v) + if "Content-Length" not in extra and total: + self.send_header("Content-Length", total) + for k, v in CORS.items(): + self.send_header(k, v) + self.end_headers() + + def do_GET(self): + resp, rng = self._upstream("GET") + with lock: + stats["GET"] += 1 + if rng: + stats["GET_ranged"] += 1 + body = resp.read() + self.send_response(resp.status) + for k, v in resp.headers.items(): + if k.lower() in ("content-length", "transfer-encoding", "connection", + "access-control-allow-origin", "access-control-expose-headers"): + continue + self.send_header(k, v) + self.send_header("Content-Length", str(len(body))) + for k, v in CORS.items(): + self.send_header(k, v) + self.end_headers() + self.wfile.write(body) + + +if __name__ == "__main__": + mode = "PASSTHROUGH (control)" if PASSTHROUGH else "HEAD+Range -> 206 (treatment)" + print(f"proxy on :{PORT} -> {UPSTREAM} mode={mode}", flush=True) + srv = ThreadingHTTPServer(("127.0.0.1", PORT), H) + try: + srv.serve_forever() + except KeyboardInterrupt: + pass + finally: + print(f"stats: {stats}", flush=True)