You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
An unused category in a pandas Categorical column is counted as a zero-row group and lowers the score #994
Where:faircode/profiler.py_dimension() (line 335) and _analyze_groups(), tests/test_profiler.py.
Current gap:value_counts() on a Categorical column lists every declared category, including ones with no rows (a Parquet file written from a categorical dtype does this). The profile then reports a group with count: 0, imbalance_ratio: None, the flag "a subgroup is effectively absent (0 rows)" and a lower score, although no row is missing. CSV, JSON and the web engine can never produce such a group, so the same data scores differently by file format. #975 added tests for how this situation is rendered; this is about whether it should happen at all.
Repro:
>>> import pandas as pd; from faircode import profile
>>> col = pd.Series(pd.Categorical(["M"] * 30 + ["F"] * 30, categories=["M", "F", "X"]))
>>> d = profile(pd.DataFrame({"sex": col}))["dimensions"][0]
>>> d["n_groups"], d["imbalance_ratio"], d["dimension_score"]
(3, None, 63) # with a plain object column: (2, 1.0, 100)
Suggested approach: count only observed values (observed=True, or .astype(object) before value_counts()), keep the zero-row path for reference baselines only if it is wanted there, and add a Parquet round-trip test.
Activity
fix: unused pandas categories are no longer counted as zero-row groups (
yakew7 commentedon Oct 16, 2026
Fixed in f9052c7 (on main). Full suite green (941 passed).