Skip to content

perf: build resource groups in one pass - #113

Open
simonvanlierde wants to merge 3 commits into
brightway-lca:mainfrom
simonvanlierde:perf/single-pass-groups
Open

simonvanlierde wants to merge 3 commits into
brightway-lca:mainfrom
simonvanlierde:perf/single-pass-groups

Conversation

@simonvanlierde

@simonvanlierde simonvanlierde commented Oct 2, 2026 •

Copy link
Copy Markdown

Hi Chris,

This makes Datapackage.groups linear in the number of resources. It called filter_by_attribute once per group label, and each call walks every resource and then checks i in to_include against a list. So a datapackage with many small groups took quadratic time to map into a matrix. I ran into it in bw_timex, which used to add one vector per matrix entry (brightway-lca/bw_timex#244).

Now groups collects the indices per label in one pass. filter_by_attribute and exclude share a small _filtered(indices) helper that indexes the lists directly. Same group order (first seen), same resource objects, no copies. I added a test that interleaves resources from different groups and checks that each group keeps its resources in their original order.

Two small ones in the same spirit: __contains__ now looks up group labels in a set instead of building a filtered package per group, and del_resource_group checks indices against a set.

One behaviour change: if data and resources differ in length (only possible by editing them by hand), filtering now raises IndexError instead of quietly pairing data with the wrong resource.

Timings on Linux, one vector per resource group (1 entry each):

groups dp.groups main this PR MappedMatrix main this PR + matrix_utils fix
1,000 0.12 s 0.001 s 0.31 s 0.15 s
4,000 1.98 s 0.005 s 3.20 s 0.64 s
16,000 36.4 s 0.021 s 45.8 s 3.0 s

MappedMatrix needs the matrix_utils fix too (brightway-lca/matrix_utils#48): has_relevant_data rescans the whole package for every group. With only this PR, MappedMatrix at 16,000 groups is 7.3 s. What's left after both is fixed per-group work in ResourceGroup, which grows linearly.

Each PR alone stays mostly quadratic. At 300 groups of 3 resources, filter_groups_for_packages takes 27.4 ms on main, 2.1 ms with only bw_processing#113, 25.2 ms with only matrix_utils#48, and 0.47 ms with both.

On bw_timex main with both fixes, lci() on my synthetic case (5,000 background activities x 4 years) goes from 1.51 to 0.93 s (median of 5), with identical scores.

Full test suite passes. Let me know what you think!

groups filtered the resources once per group label, and each filter checked membership against a list, so mapping a datapackage with many small groups took quadratic time. filter_by_attribute and exclude now share a helper that indexes the lists directly.
- __contains__ looks up group labels in a set instead of building a filtered package per group
- del_resource_group checks indices against a set
@simonvanlierde
simonvanlierde marked this pull request as ready for review October 2, 2026 15:09
@simonvanlierde

Copy link
Copy Markdown
Author

Hi Chris,

Could you approve the CI run on this one? GitHub holds it until a maintainer does, so it hasn't run yet.

Best,
Simon

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant