Skip to content

Preserve the caller's NumPy random state during embedding drift bootstrap - #1991

Open
Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:fix/embedding-bootstrap-random-state-20261112
Open

Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:fix/embedding-bootstrap-random-state-20261112

Conversation

@Afloat16

Copy link
Copy Markdown

Problem

Embedding distance and MMD bootstrap repeatedly call np.random.seed(i) and draw from the process-global generator. The model-based drift bootstrap does the same through calc_roc_auc_random. Running a drift check therefore changes the random sequence used by later training, sampling, or evaluation code in the same process.

Change

Use a local np.random.RandomState(i) for each bootstrap repetition and for the random-AUC helper. Keep the same seeds, legacy generator algorithm, draw order, bootstrap samples, and threshold calculations. The change isolates the generator rather than changing the statistical procedure.

Validation

  • Twelve new native regression tests: base code has 6 failures and 6 passing controls.
  • Fixed code: 13 tests pass including the existing embedding-drift report smoke test.
  • All three drift methods are checked with bootstrap enabled and disabled. Tests compare the caller's subsequent random draws against an untouched generator.
  • Random AUC values match an independently seeded reference for seeds 0, 17 and 99.
  • Repeated bootstrap results remain independent of the external NumPy seed.
  • Each test restores the global state it found, so the checks do not change later tests' random state.
  • Pinned Ruff 0.3.7 lint/format and git diff --check pass.

This concerns the embedding-drift bootstrap paths. It is separate from #1985, which changes the permutation implementation under legacy/calculations/stattests/utils.py. The full repository suite and end-to-end training benchmarks were not run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant