Skip to content

Keep embedding data separate from model drift labels - #1988

Open
Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:fix/embedding-model-drift-inputs-20261112
Open

Afloat16 wants to merge 1 commit into
evidentlyai:mainfrom
Afloat16:fix/embedding-model-drift-inputs-20261112

Conversation

@Afloat16

Copy link
Copy Markdown

Problem

ModelDriftMethod writes a target column directly into each embedding DataFrame and then removes it from the classifier features. This mutates the caller's data and silently discards an existing embedding dimension named target. If the same DataFrame is supplied for both samples, the second assignment overwrites both sets of labels, so fitting fails with a single-class error.

Change

Construct the domain-classification target as a separate array in the same reference/current order as the concatenated embeddings. Preserve all input dimensions and leave the original DataFrames unchanged. PCA, train/test splitting, classifier configuration, and drift threshold logic stay the same.

Validation

Seven new tests cover input preservation with and without bootstrap/PCA, invariance under renaming an embedding dimension, and comparing a DataFrame with itself. On a fixture whose only separating dimension is named target, the old implementation reports ROC AUC 0.4125 and no drift; renaming that dimension yields AUC 1.0. The fix makes the result independent of that column name. This is a correctness example, not a general accuracy or performance claim.

  • Baseline: 4 failures and 3 passing controls with the final regression tests.
  • Fixed: all 7 new tests plus the existing native embedding-drift report test pass (8 total).
  • Ruff 0.3.7 checks and formatting, plus git diff --check, pass.
  • Tested on Python 3.12 with CPU dependencies. The complete repository suite and mypy were not run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant