Skip to content

perf(arrow): build partition filter mask with BooleanBufferBuilder - #3204

Merged
laskoviymishka merged 1 commit into
apache:mainfrom
anoopj:perf-partition-filter-buffer
Sep 14, 2026
Merged

laskoviymishka merged 1 commit into
apache:mainfrom
anoopj:perf-partition-filter-buffer

Conversation

@anoopj

@anoopj anoopj commented Sep 11, 2026

Copy link
Copy Markdown
Member

What changes are included in this PR?

The partition splitter built each group's selection mask as a throwaway vec![false; num_rows] and bit-packed it into a BooleanArray via .into(), allocating an n-byte Vec<bool> plus a full packing pass per group per batch on every partitioned write. Build the packed buffer directly with BooleanBufferBuilder, setting only the selected bits.

Are these changes tested?

Covered by existing tests

The partition splitter built each group's selection mask as a throwaway
`vec![false; num_rows]` and bit-packed it into a `BooleanArray` via
`.into()`, allocating an n-byte `Vec<bool>` plus a full packing pass per
group per batch on every partitioned write. Build the packed buffer
directly with `BooleanBufferBuilder`, setting only the selected bits.
@anoopj
anoopj force-pushed the perf-partition-filter-buffer branch from 651780d to 0766714 Compare September 11, 2026 23:54

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Asked my LLM friend for a benchmark, and it is more than I expected:

Results

Micro-benchmark isolating exactly the changed code (building each group's selection mask), arrow 59.3, criterion, macOS. Each iteration builds masks for all groups in one batch (i.e. one full split):

┌──────────────────────────────┬───────────────┬──────────────────────────┬─────────┐
│ config (batch rows × groups) │ old Vec<bool> │ new BooleanBufferBuilder │ speedup │
├──────────────────────────────┼───────────────┼──────────────────────────┼─────────┤
│ 1024 × 1                     │ 1.47 µs       │ 1.07 µs                  │ 1.37×   │
├──────────────────────────────┼───────────────┼──────────────────────────┼─────────┤
│ 1024 × 10                    │ 3.65 µs       │ 0.69 µs                  │ 5.30×   │
├──────────────────────────────┼───────────────┼──────────────────────────┼─────────┤
│ 8192 × 1                     │ 11.28 µs      │ 8.34 µs                  │ 1.35×   │
├──────────────────────────────┼───────────────┼──────────────────────────┼─────────┤
│ 8192 × 10                    │ 24.12 µs      │ 3.37 µs                  │ 7.15×   │
├──────────────────────────────┼───────────────┼──────────────────────────┼─────────┤
│ 8192 × 100                   │ 266.8 µs      │ 7.67 µs                  │ 34.8×   │
├──────────────────────────────┼───────────────┼──────────────────────────┼─────────┤
│ 65536 × 1                    │ 89.3 µs       │ 69.2 µs                  │ 1.29×   │
├──────────────────────────────┼───────────────┼──────────────────────────┼─────────┤
│ 65536 × 10                   │ 192.8 µs      │ 24.7 µs                  │ 7.79×   │
├──────────────────────────────┼───────────────┼──────────────────────────┼─────────┤
│ 65536 × 100                  │ 2128.9 µs     │ 35.8 µs                  │ 59.5×   │
└──────────────────────────────┴───────────────┴──────────────────────────┴─────────┘

@laskoviymishka laskoviymishka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🌬️ ⛵

@laskoviymishka
laskoviymishka added this pull request to the merge queue Sep 14, 2026
Merged via the queue into apache:main with commit f11a20d Sep 14, 2026
23 checks passed
@anoopj
anoopj deleted the perf-partition-filter-buffer branch September 14, 2026 21:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants