zspace: Fisher-Z entry tax without materializing z (ZGamma codes) + binomial Hamming null; batch Welford recovered from #323

[AdaWorldAPI]

Why

Operator rules:

• Anything cosine-shaped pays the Fisher-Z entry tax before it is thresholded, averaged, banded or ranked by margin. There are no exceptions, including lab and calibration code.
• Anything bitpacked goes through HDR popcount stacking on the Hamming distance itself.
• Fisher-Z is never materialized. A value is normalized once and stays in that space.
• Bit-exact on every target.

Before this PR, ndarray had no Fisher transform at all.

This PR also carries two commits that were meant for #323 but missed it: #323 was merged at edbd97f, one push before them. They are cherry-picked here unchanged: moments_u32 + Cascade::observe_batch, and their re-export.

What

hpc::zspace (7bb45fb → bb14de6 → fd45c89 → f2a461a)

ZGamma { z_min, z_range } is the z-space envelope: 8 little-endian bytes.

• fit(cosines) scans only the cosine min and max. Fisher-Z is monotone, so no z value is ever stored.
• encode / encode_batch go straight from cosine to an i8 code, and only the code is written. Both call one shared quantize().
• The code: ((z−z_min)/z_range)·254 − 127, rounded ties-to-even and saturated to the symmetric range −127..=127. −128 is reserved as ZGamma::NAN_CODE.
• threshold_code(c) converts a threshold into code space once. There is no decode back to cosine.

Bit-exact across targets (f2a461a). simd_ln_f32 is a per-lane libm .ln(), so the first version inherited the platform libm. Measured over an 8,539-value grid:

	f32 z	f64 z
values differing, x86-64 glibc vs wasm32	624	566
size of the difference	1–2 ulp	1–2 ulp

The codes happened to agree on that grid, but a one-ulp difference at a rounding boundary flips a code.

ln_det replaces libm in the code path. It is musl/FreeBSD logf, built only from IEEE add, sub, mul, div and bit operations in a fixed order. Rust never contracts these into FMA, so every conforming target gets the same bits. Its error is under 1 ulp over its domain [2⁻²⁴, 2). Its x86 z digest now equals the digest wasm's (musl-derived) libm produced before.

Where this departs from bgz-tensor. It shares FamilyGamma's byte layout, but not its code. FamilyGamma predates this analysis and was not calibrated for exactness:

FamilyGamma behaviour	measured effect
truncation toward zero	the code-0 bucket is twice as wide as the others, and every code is pulled half a step inward (median bias: +0.49 / −0.49). Rounding: +0.006 / −0.002.
−128 used as a data level	asymmetric range
NaN mapped to 0	collides with a real code
platform libm ln	not bit-exact across targets

Unchanged:

• fisher_z / hyperbolic_depth stay, for single f64 scalars only (a report statistic, a threshold). They keep libm and are documented as not bit-exact.
• hamming_null_z stays. It is the binomial-null z for bitpacked data.

Removed:

• fisher_z_f32_batch wrote a buffer of z values.
• fisher_z_inv was the way back to cosine.

FidelityReport pays the tax (b8babbd)

• Adds pearson_z / spearman_z / cosine_z (scalar statistics).

Batch Welford, recovered (90f1128, 414aa8e)

• moments_u32 computes exact integer (n, Σx, Σx²) through U64x8.
• MomentsU32::merge is exact and order-independent.
• Cascade::observe_batch uses the Chan–Golub–LeVeque merge.

wasm enforcement (aa3dfd7)

CI never runs cargo test on wasm. So crates/wasm-simd-parity's node selfcheck now rebuilds the grid and checks the pinned codes digest (rc 0x501) and batch == scalar (rc 0x502).

Tests

zspace (14 tests):

• the 12 earlier tests: bit identity, symmetric saturation, spacing in z, degenerate inputs, threshold, round-trip through 8 LE bytes, no median bias, NaN sentinel, the Fisher-Z and binomial pins;
• ln_det is within 1 ulp across its domain, sampling every 257th bit pattern;
• pinned digests (GOLDEN_CODES, GOLDEN_Z32) that must hold on every target.

Disable checks — each deliberate bug makes the listed test fail:

deliberate bug	what fails
truncate instead of round	median-bias test
saturate to −128	3 tests
NaN → 0	sentinel test
clamp removed from the batch lanes	bit-identity test
Fisher transform removed	3 tests
degenerate-range guard removed	degenerate-population test
libm .ln() back in the scalar path	pinned z digest
altered wasm golden constant	selfcheck returns rc 0x501

Test runs by build:

build	result
config-v3 (AVX2)	14/14
config-v4 (AVX-512)	14/14
aarch64 NEON under qemu	14/14
wasm32 simd128 under node	selfcheck OK (codes digest)

The doc tests pass. clippy -D warnings and fmt are clean.

Open questions (not decided in this PR)

• What does code 0 mean? Here code 0 is the midpoint of the fitted z range, not zero correlation. The alternative is a scale-only envelope, code = round(127·z/z_absmax): the sign of a code becomes the sign of the evidence. It costs half the resolution when a population is one-sided.
• HDR popcount stacking vs Fisher-Z. Bitpacked data stays in Hamming, measured against the binomial null, and is never converted Hamming → cosine → Fisher-Z. The two scores are comparable only in σ units: fisher_z·√(n−3) and hamming_null_z are both ~N(0,1) under their nulls.
• simd_ln_f32 itself is still per-lane libm, used by callers outside zspace. Whether the facade's ln should become ln_det (bit-exact, and vectorizable in the same op order) is a separate decision.

Not in this PR

• lance-graph sites that skip the Fisher-Z tax:
  • the raw SQL cosine UDFs;
  • the contract's similarity_z / fisher_z_inverse.
• Rim-clamp disagreements between Fisher-Z copies.
• helix's hardcoded γ.
• bgz-tensor FamilyGamma: truncation, −128, NaN and libm are all present there too. Fixing it changes stored codes, so it needs its own decision.

🤖 Generated with Claude Code

https://claude.ai/code/session_019HnekoM1EidTwQLS3oFVFm

Summary by CodeRabbit

• New Features
  • Added Fisher-Z score views for Pearson, Spearman, and cosine similarity results, making these values available for consistent comparison and thresholding.
  • Added statistical scoring for bitpacked Hamming distances and compact encoding of cosine similarity values, including batch encoding and threshold lookup.
  • Added batch observation support that combines distance measurements and can flag significant shifts in the observed mean.
  • Added utilities for calculating and combining exact moments—count, mean, and variance—from unsigned integer measurements.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Essentials

Run ID: bcb9bbea-6691-4844-9cb0-d4ff773a4197

📥 Commits

Reviewing files that changed from the base of the PR and between 2c91538 and aa3dfd7.

📒 Files selected for processing (7)

• crates/wasm-simd-parity/src/lib.rs
• src/hpc/cascade.rs
• src/hpc/mod.rs
• src/hpc/reliability.rs
• src/hpc/statistics.rs
• src/hpc/zspace.rs
• src/simd.rs

 ______________________________________________________
< Weeks of programming can save you hours of planning. >
 ------------------------------------------------------
  \
   \   (\__/)
       (•ㅅ•)
       / 　 づ

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_15974d73-f3de-4803-a11d-f88b772c16b3)

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_00acef45-6d8f-44ca-a39b-f70902f9b659)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aa3dfd7365

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve alerts when a batch crosses the warm-up boundary

When a batch starts with 1–10 prior observations and brings the total above 10, checking old_n suppresses the shift alert even though observe checks the incremented observation count. For example, a cascade calibrated with 10 non-constant distances followed by one extreme distance can alert through observe but not observe_batch(&[distance]), making alert behavior depend on batching. Apply the warm-up check to the merged observation count instead.

Useful? React with 👍 / 👎.