feat: add class overlap explorer with F1, N1 and N2 complexity measures - #864
Open
CristobalSantana wants to merge 1 commit into
Open
Conversation
Adds a Data Complexity explorer category and a Class Overlap explorer that measures how separable the classes of a dataset are, without training any model. It reports three geometrical complexity measures surveyed in Lorena et al. (2019): F1 (maximum Fisher discriminant ratio), N1 (fraction of borderline points, from a minimum spanning tree) and N2 (intra over extra class nearest neighbour ratio). All three are normalised to [0, 1] where lower means easier to separate. The measures live in a standalone module with no DashAI imports, so they can be reused unchanged if the project later introduces a dedicated data-metric component type. This follows the discussion in DashAISoftware#836, where the architectural decision is still open: implementing it as an explorer reuses the existing notebook UI, job queue and persistence without pre-empting that call. N1 and N2 both need a pairwise distance matrix, so max_samples caps the rows used for them with a class-proportional subsample. F1 is linear in the sample and always uses every row. No new dependencies: numpy, scikit-learn and its scipy are already required.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Class Overlap explorer under a new Data Complexity category. It measures how separable the classes of a dataset are from the geometry of the data alone, without training any model.
This is the first of the four families discussed in #836. The maintainers suggested starting with the single dataset measures, since they are the least likely to be affected by whatever is decided for the comparison surface.
Three measures from Lorena et al. (2019), all normalised to [0, 1] where lower means easier to separate:
Type of Change
Changes (by file)
DashAI/back/exploration/complexity_measures.py: the three measures as plain functions over(x, y), with no DashAI imports.DashAI/back/exploration/data_complexity_explorer.py: the new Data Complexity category.DashAI/back/exploration/explorers/class_overlap.py: the explorer. The target column is a schema parameter and is appended to the selection inprepare_dataset, following the pattern already used byScatterPlotExplorerfor its grouping columns.tests/back/exploration/test_complexity_measures.py: 29 tests.DashAI/back/initial_components.py: registration.Testing
29 unit tests cover the value range, the direction of the convention, degenerate cases (coincident points, single member classes, missing values), subsampling, reproducibility and scale invariance.
Sanity values against known datasets: iris gives F1 0.0004, N1 0.10, N2 0.18, which matches the known overlap between versicolor and virginica. Random noise gives 0.40, 0.73 and 0.50.
No new dependencies. F1 needs only numpy, and N1 and N2 use scipy, which comes with scikit-learn.
Notes
Why an explorer and not a new component type. #836 is still open on that question, so this reuses the existing exploration subsystem instead of pre-empting the decision. The measures live in a standalone module with no DashAI imports, so if a
DataMetrictype is introduced later, only the wrapper needs to change.Limitation. N1 and N2 need a pairwise distance matrix, so
max_samplescaps the rows used for them at 2000, with a class proportional subsample. F1 is linear in the sample and always uses every row. This is reported in the result table.Found while implementing. The target column has to be typed by hand, as a name or an index, because the schema system has no column picker field. A dropdown of the dataset columns would be better, but it needs a new field type in
schema_fieldsand would affect every component, so it felt out of scope here.