Skip to content

Keep embedding bootstrap samples large enough for their statistics - #1994

Open
Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:fix/valid-embedding-bootstrap-sizes-20261112
Open

Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:fix/valid-embedding-bootstrap-sizes-20261112

Conversation

@Afloat16

Copy link
Copy Markdown

Problem

Distance and MMD embedding drift choose bootstrap sample sizes by truncating population-proportional counts. For small or imbalanced inputs, this can produce zero observations for a mean, or one observation for the unbiased MMD statistic whose denominator contains n*(n-1).

The empty mean makes Euclidean distance raise on NaN input; cityblock distance instead produces a NaN threshold and reports no drift. MMD raises ZeroDivisionError. This also affects substantial imbalanced datasets: native Reports fail for reference/current sizes 20/500 with distance and 20/200 with MMD.

Change

Apply the minimum sample count required by each statistic to both bootstrap subsamples: one for distance between means, and two for unbiased MMD. The bootstrap already samples with replacement. Preserve the proportional counts whenever they are large enough, the seeds, the statistic formulas, bandwidths, and quantile settings.

Validation

  • 24 new tests on the upstream base: 10 failed, 14 passed.
  • Fixed tests plus the existing embedding-drift Report test: 25 passed on CPU.
  • Independent mean-distance and pairwise RBF U-statistic references verify scores and seeded bootstrap decisions. Cases cover small/imbalanced inputs, sufficient-size and non-bootstrap controls, the cityblock NaN decision, and native Reports for both methods.
  • Tests restore the caller's global NumPy random state.
  • Ruff 0.3.7 lint/format and git diff --check pass.

This fixes the bootstrap subsample domain. It does not relax the original input requirements (MMD still needs at least two observations in each population), solve degenerate bandwidths, or establish reliable inference from arbitrarily small samples. The full repository suite was not run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant