Skip to content

Handle zero median bandwidth in embedding MMD drift - #1989

Open
Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:fix/mmd-zero-bandwidth-20261112
Open

Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:fix/mmd-zero-bandwidth-20261112

Conversation

@Afloat16

Copy link
Copy Markdown

Problem

Constant or heavily repeated embeddings can have a zero median pairwise distance. The embedding MMD method then passes gamma=1/0 to its RBF kernel, producing NaN entries and a NaN drift score. The same failure occurs in the bootstrap reference kernel when the reference embeddings are constant, even when the current embeddings differ.

Change

Use the zero-bandwidth limit of the RBF kernel when the estimated squared bandwidth is exactly zero: equal vectors have kernel value 1 and distinct vectors have value 0. Group equal vectors with np.unique, then compare their group labels, avoiding a samples-by-samples-by-features intermediate. Positive bandwidths keep the existing pairwise_kernels calculation.

Both the observed embedding kernel and its bootstrap reference kernel use this helper. This change is confined to legacy/metrics/data_drift/embedding_drift_methods.py; it does not change the separate univariate mmd_stattest.py distance calculation addressed in #1966.

Validation

  • Identical constant embeddings return a finite zero score with and without bootstrap, including nonzero constant values.
  • A changed constant embedding is still detected; its score matches 2 * (1 - exp(-1)).
  • A repeated-vector example with zero median bandwidth matches a hand-counted unbiased MMD statistic, preserving distinct groups.
  • A positive-bandwidth control matches an independently constructed RBF statistic.
  • The final eight new tests fail 6/pass 2 before the change; all eight pass after it. The existing native embedding-drift report test also passes (9 total).
  • Ruff 0.3.7 checks and formatting, plus git diff --check, pass.

The full repository suite and mypy were not run. This resolves undefined kernel values for degenerate bandwidths; it does not establish a general detection-quality improvement or change bootstrap sample-size policy.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant