Skip to content

feat: add class overlap explorer with F1, N1 and N2 complexity measures - #864

Open
CristobalSantana wants to merge 1 commit into
DashAISoftware:developfrom
CristobalSantana:feat/data-complexity-explorer
Open

feat: add class overlap explorer with F1, N1 and N2 complexity measures#864
CristobalSantana wants to merge 1 commit into
DashAISoftware:developfrom
CristobalSantana:feat/data-complexity-explorer

Conversation

@CristobalSantana

Copy link
Copy Markdown
Contributor

Summary

Adds a Class Overlap explorer under a new Data Complexity category. It measures how separable the classes of a dataset are from the geometry of the data alone, without training any model.

This is the first of the four families discussed in #836. The maintainers suggested starting with the single dataset measures, since they are the least likely to be affected by whatever is decided for the comparison surface.

Three measures from Lorena et al. (2019), all normalised to [0, 1] where lower means easier to separate:

  • F1: maximum Fisher discriminant ratio. Near 0 when one feature separates the classes on its own.
  • N1: fraction of points on a class boundary, from a minimum spanning tree.
  • N2: ratio between the nearest same class distance and the nearest different class distance.

Type of Change

  • Backend change
  • Frontend change
  • CI / Workflow change
  • Build / Packaging change
  • Bug fix
  • Documentation

Changes (by file)

  • DashAI/back/exploration/complexity_measures.py: the three measures as plain functions over (x, y), with no DashAI imports.
  • DashAI/back/exploration/data_complexity_explorer.py: the new Data Complexity category.
  • DashAI/back/exploration/explorers/class_overlap.py: the explorer. The target column is a schema parameter and is appended to the selection in prepare_dataset, following the pattern already used by ScatterPlotExplorer for its grouping columns.
  • tests/back/exploration/test_complexity_measures.py: 29 tests.
  • DashAI/back/initial_components.py: registration.

Testing

29 unit tests cover the value range, the direction of the convention, degenerate cases (coincident points, single member classes, missing values), subsampling, reproducibility and scale invariance.

Sanity values against known datasets: iris gives F1 0.0004, N1 0.10, N2 0.18, which matches the known overlap between versicolor and virginica. Random noise gives 0.40, 0.73 and 0.50.

No new dependencies. F1 needs only numpy, and N1 and N2 use scipy, which comes with scikit-learn.


Notes

Why an explorer and not a new component type. #836 is still open on that question, so this reuses the existing exploration subsystem instead of pre-empting the decision. The measures live in a standalone module with no DashAI imports, so if a DataMetric type is introduced later, only the wrapper needs to change.

Limitation. N1 and N2 need a pairwise distance matrix, so max_samples caps the rows used for them at 2000, with a class proportional subsample. F1 is linear in the sample and always uses every row. This is reported in the result table.

Found while implementing. The target column has to be typed by hand, as a name or an index, because the schema system has no column picker field. A dropdown of the dataset columns would be better, but it needs a new field type in schema_fields and would affect every component, so it felt out of scope here.

Adds a Data Complexity explorer category and a Class Overlap explorer that
measures how separable the classes of a dataset are, without training any
model. It reports three geometrical complexity measures surveyed in Lorena
et al. (2019): F1 (maximum Fisher discriminant ratio), N1 (fraction of
borderline points, from a minimum spanning tree) and N2 (intra over extra
class nearest neighbour ratio). All three are normalised to [0, 1] where
lower means easier to separate.

The measures live in a standalone module with no DashAI imports, so they can
be reused unchanged if the project later introduces a dedicated data-metric
component type. This follows the discussion in DashAISoftware#836, where the architectural
decision is still open: implementing it as an explorer reuses the existing
notebook UI, job queue and persistence without pre-empting that call.

N1 and N2 both need a pairwise distance matrix, so max_samples caps the rows
used for them with a class-proportional subsample. F1 is linear in the sample
and always uses every row.

No new dependencies: numpy, scikit-learn and its scipy are already required.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant