Repository navigation
perf: build resource groups in one pass - #113
Open
simonvanlierde wants to merge 3 commits into
Open
simonvanlierde wants to merge 3 commits into
simonvanlierde wants to merge 3 commits into
Conversation
groups filtered the resources once per group label, and each filter checked membership against a list, so mapping a datapackage with many small groups took quadratic time. filter_by_attribute and exclude now share a helper that indexes the lists directly.
simonvanlierde
marked this pull request as draft
October 2, 2026 14:49
- __contains__ looks up group labels in a set instead of building a filtered package per group - del_resource_group checks indices against a set
simonvanlierde
marked this pull request as ready for review
October 2, 2026 15:09
Author
|
Hi Chris, Could you approve the CI run on this one? GitHub holds it until a maintainer does, so it hasn't run yet. Best, |
This was referenced Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi Chris,
This makes
Datapackage.groupslinear in the number of resources. It calledfilter_by_attributeonce per group label, and each call walks every resource and then checksi in to_includeagainst a list. So a datapackage with many small groups took quadratic time to map into a matrix. I ran into it in bw_timex, which used to add one vector per matrix entry (brightway-lca/bw_timex#244).Now
groupscollects the indices per label in one pass.filter_by_attributeandexcludeshare a small_filtered(indices)helper that indexes the lists directly. Same group order (first seen), same resource objects, no copies. I added a test that interleaves resources from different groups and checks that each group keeps its resources in their original order.Two small ones in the same spirit:
__contains__now looks up group labels in a set instead of building a filtered package per group, anddel_resource_groupchecks indices against a set.One behaviour change: if
dataandresourcesdiffer in length (only possible by editing them by hand), filtering now raisesIndexErrorinstead of quietly pairing data with the wrong resource.Timings on Linux, one vector per resource group (1 entry each):
dp.groupsmainMappedMatrixmainMappedMatrixneeds the matrix_utils fix too (brightway-lca/matrix_utils#48):has_relevant_datarescans the whole package for every group. With only this PR,MappedMatrixat 16,000 groups is 7.3 s. What's left after both is fixed per-group work inResourceGroup, which grows linearly.Each PR alone stays mostly quadratic. At 300 groups of 3 resources,
filter_groups_for_packagestakes 27.4 ms on main, 2.1 ms with only bw_processing#113, 25.2 ms with only matrix_utils#48, and 0.47 ms with both.On bw_timex main with both fixes,
lci()on my synthetic case (5,000 background activities x 4 years) goes from 1.51 to 0.93 s (median of 5), with identical scores.Full test suite passes. Let me know what you think!