Skip to content

Stratify the train/test split for model-based embedding drift - #1992

Open
Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:fix/stratified-embedding-drift-split-20261112
Open

Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:fix/stratified-embedding-drift-split-20261112

Conversation

@Afloat16

Copy link
Copy Markdown

Problem

ModelDriftMethod creates a binary source label for reference/current embeddings, then uses an unstratified 50/50 split. Even when both populations contain at least two rows, the fixed random split can leave the training set with one class. For example, reference/current sizes 2/8 and 3/4 raise SGDClassifier's single-class error. Training the classifier and evaluating its ROC AUC both require the source classes to remain represented.

Change

Stratify train_test_split by the existing source labels, preserving the 50/50 ratio, shuffle setting, random seed, classifier, and thresholds. This change is independent of the separate fix for using an embedding dimension named target.

Validation

  • 10 new tests on the upstream base: 5 failed, 5 passed.
  • Fixed tests plus the existing embedding-drift Report test: 11 passed on CPU.
  • Cases cover small imbalanced samples in both directions, a balanced control, bootstrap and fixed-threshold decisions, an explicit stratified SGD/ROC reference, and a real Report integration control.
  • The tests restore the caller's NumPy random state so the existing bootstrap behavior cannot contaminate other tests.
  • Ruff 0.3.7 lint/format and git diff --check pass.

Stratification can change numeric scores on existing datasets because it changes which rows enter each split. This does not provide a statistically reliable drift estimate from arbitrarily small datasets: each population must have enough rows for both splits. The Report plotting layer's independent t-SNE/KDE sample requirements are unchanged; the Report control uses a sufficient sample size.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant