Repository navigation
Prune redundant distributions arrays in finalize_serialization - #101
Merged
Merged
Conversation
Add _prune_trivial_distributions(), called from finalize_serialization(), which removes a distributions resource when every row carries no useful stochastic information: all uncertainty_type IDs must be in _NO_UNCERTAINTY_IDS (NoUncertainty / UndefinedUncertainty from stats_arrays, falling back to 0 and 1 when stats_arrays is not installed), AND the loc values must equal the corresponding data-array amounts at float32 precision. Such a distributions array is redundant with the data array, so both the in-memory resource entry and the on-disk file are deleted before the datapackage metadata is written.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_prune_trivial_distributions(), called fromfinalize_serialization(), which removes a distributions resource when it carries no stochastic information beyond what the data array already encodesuncertainty_typeis a no-information ID (NoUncertaintyorUndefinedUncertaintyfromstats_arrays; falls back to0and1whenstats_arraysis not installed, following the same pattern already used inmatrix_entry.py)locequals the corresponding data-array amount (compared atfloat32precision, the dtype of thelocfield)datapackage.jsonis written, so reloaded packages are cleanNotes
stats_arraysis not a declared dependency of this package; the import is wrapped intry/except ImportErroridentical to the existing pattern inmatrix_entry.pyadd_persistent_vectormethod already declines to store distributions arrays where all types are trivial (line ~573)._prune_trivial_distributionsis a complementary defence for arrays that arrive through other code paths — e.g. loaded from an older datapackage file and then re-serializedTest plan
test_finalize_removes_trivial_distributions_type0— type 0 + loc==amount → prunedtest_finalize_removes_trivial_distributions_type1— type 1 + loc==amount → prunedtest_finalize_removes_trivial_distributions_file_deleted—.npyfile is physically removed from disktest_finalize_keeps_distributions_real_uncertainty— any type ≥ 2 → retainedtest_finalize_keeps_distributions_loc_mismatch— all trivial types but loc≠amount → retainedtest_finalize_mixed_groups_prunes_only_trivial— two groups: only the trivial one is pruned