Skip to content

PoC for consistent collection filtering - #970

Draft
paulgessinger wants to merge 14 commits into
AIDASoft:masterfrom
paulgessinger:feat/filtering
Draft

PoC for consistent collection filtering#970
paulgessinger wants to merge 14 commits into
AIDASoft:masterfrom
paulgessinger:feat/filtering

Conversation

@paulgessinger

@paulgessinger paulgessinger commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

This is PoC by Claude for filtering collections in a frame based on user-supplied predicates including mark-and-sweep unreachable object (orphan) removal and optional cascading deletes through relationships.

I'm not necessarily proposing to merge this "as is", we might want to change the approach entirely, but I thought it's still useful as the basis for discussion.

The user facing API looks something like this:

podio::Frame out = podio::FrameFilter{inFrame}
    .keep("MCParticles", [](const ExampleMC& p) { return p.energy() > 1.0; })
    .keepReferenced("hits")
    .cascade("ExampleCluster.Hits")
    .run();
  • Think about integration with Gaudi, can only happen at the end
  • Potentially interaction with DataSource, but unclear if relevent
  • Do some basic performance measurements
  • Drop cascade for now

BEGINRELEASENOTES

  • Thank you for writing the text to appear in the release notes. It will show up
    exactly as it appears between the two bold lines
  • ...

ENDRELEASENOTES

@paulgessinger paulgessinger changed the title PoC for PODIO filtering PoC for consistent collection filtering Jun 10, 2026
@paulgessinger

Copy link
Copy Markdown
Contributor Author

@tmadlener tmadlener left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't looked at the details (or the code) yet too much. This is definitely useful though. Conceptually, I think this is a pretty nice approach. Is there any indication already if / how this would carry over to python? One reason to ask is because this might become useful in combination with the proposal in #939 to potentially do some collection level filtering first and then some whole event skimming afterwards. Or potentially that functionality can be entirely absorbed here(?).

I wasn't present when this was discussed, so maybe someone who was can clarify / extend a bit more on the open points? Most importantly about the Gaudi and DataSource interactions.

I suppose the generic functionality of the FrameFilter would remain largely in the hands of experts and only some of the functionality would in the end also be exposed via e.g. Gaudi?

Finally, I think I understand what cascade is supposed to do, but I am not sure I would have intuitively guessed it from the name.

Comment thread doc/filtering.md
Comment on lines +33 to +34
type. (To keep an empty collection instead, use an always-false `keep`
predicate.)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This confused me for a bit. There are two ways we can end up with empty collections after the filter has run. Either we have filtered out all elements of a collection, or we started the filtering with an empty collection. Not sure if this needs to be clarified in the end, I just wanted to not I stumbled over this while reading.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants