Skip to content

Prune redundant distributions arrays in finalize_serialization - #101

Merged
cmutel merged 1 commit into
mainfrom
feat/prune-trivial-distributions
Jun 4, 2026
Merged

cmutel merged 1 commit into
mainfrom
feat/prune-trivial-distributions

Conversation

@cmutel

@cmutel cmutel commented Jun 4, 2026

Copy link
Copy Markdown
Member

Summary

  • Adds _prune_trivial_distributions(), called from finalize_serialization(), which removes a distributions resource when it carries no stochastic information beyond what the data array already encodes
  • A distributions resource is considered trivial when all rows satisfy both conditions:
    1. uncertainty_type is a no-information ID (NoUncertainty or UndefinedUncertainty from stats_arrays; falls back to 0 and 1 when stats_arrays is not installed, following the same pattern already used in matrix_entry.py)
    2. loc equals the corresponding data-array amount (compared at float32 precision, the dtype of the loc field)
  • Both the in-memory resource entry and the on-disk file are deleted before datapackage.json is written, so reloaded packages are clean

Notes

  • stats_arrays is not a declared dependency of this package; the import is wrapped in try/except ImportError identical to the existing pattern in matrix_entry.py
  • The add_persistent_vector method already declines to store distributions arrays where all types are trivial (line ~573). _prune_trivial_distributions is a complementary defence for arrays that arrive through other code paths — e.g. loaded from an older datapackage file and then re-serialized

Test plan

  • test_finalize_removes_trivial_distributions_type0 — type 0 + loc==amount → pruned
  • test_finalize_removes_trivial_distributions_type1 — type 1 + loc==amount → pruned
  • test_finalize_removes_trivial_distributions_file_deleted — .npy file is physically removed from disk
  • test_finalize_keeps_distributions_real_uncertainty — any type ≥ 2 → retained
  • test_finalize_keeps_distributions_loc_mismatch — all trivial types but loc≠amount → retained
  • test_finalize_mixed_groups_prunes_only_trivial — two groups: only the trivial one is pruned
  • Full test suite: 50/50 passing, no regressions

Add _prune_trivial_distributions(), called from finalize_serialization(),
which removes a distributions resource when every row carries no useful
stochastic information: all uncertainty_type IDs must be in
_NO_UNCERTAINTY_IDS (NoUncertainty / UndefinedUncertainty from stats_arrays,
falling back to 0 and 1 when stats_arrays is not installed), AND the loc
values must equal the corresponding data-array amounts at float32 precision.

Such a distributions array is redundant with the data array, so both the
in-memory resource entry and the on-disk file are deleted before the
datapackage metadata is written.
@cmutel
cmutel merged commit e259bbc into main Jun 4, 2026
6 checks passed
@cmutel
cmutel deleted the feat/prune-trivial-distributions branch June 4, 2026 06:55
@cmutel cmutel mentioned this pull request Jun 4, 2026
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant