Skip to content

feat: setup datasketches and update hll impl - #2868

Open
coderfender wants to merge 3 commits into
apache:mainfrom
coderfender:column-stats-datasketches
Open

coderfender wants to merge 3 commits into
apache:mainfrom
coderfender:column-stats-datasketches

Conversation

@coderfender

@coderfender coderfender commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Part of #1827. See #2783.

Rationale for this change

Use Apache DataSketches for the HyperLogLog sketch instead of our hand-written one.

What changes are included in this PR?

  • Add datasketches (hll feature) and back HyperLogLog with its HllSketch.
  • Merge task and stage sketches with a HyperLogLogUnion.
  • Remove the unused ColumnStats proto message.

Are there any user-facing changes?

HyperLogLog::as_bytes is now to_bytes and merge is replaced by HyperLogLogUnion. HLL sketch bytes use the DataSketches format.

@coderfender coderfender changed the title setup datasketches and update hll impl feat: setup datasketches and update hll impl Nov 15, 2026

@andygrove andygrove left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this! It's a nice cleanup, and it's great to lean on DataSketches rather than maintain our own estimator. The union logic in MergedSketch::add and ColumnStatsStream::drop looks right to me. Merging mixed precisions at the lowest one is a nice improvement over returning Absent. Memory looks the same or better too, since the bytes stored per task roughly halve.

The main thing I'd like to see before merging is benchmark numbers, because the per-row sketch cost went up on a default-on path. Details are inline, along with a rebase request and an upgrade guide note.

Comment thread ballista/core/src/hll.rs
create_hashes([array], &HLL_RANDOM_STATE, hashes)?;
match array.logical_nulls() {
None => hashes.iter().for_each(|&hash| self.add_hashed(hash)),
None => hashes.iter().for_each(|&hash| self.0.update(hash)),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I noticed is that HllSketch::update hashes its input again with Murmur3, so each key value now gets hashed twice on the shuffle write path (once by create_hashes, once inside the sketch). A quick microbenchmark of HyperLogLog::update over 8M distinct Int64 values at precision 12 went from about 15 ms to about 40 ms on my machine.

Since ballista.shuffle.collect_distinct_counts is on by default, could you share before and after TPC-H numbers against main? Something like:

cargo run --release --bin tpch -- benchmark ballista \
  --host localhost --port 50050 \
  --iterations 3 --path <tpch parquet dir> --format parquet

at SF10 or higher would be ideal. If the overhead shows up, one option is to skip create_hashes and feed the typed values to the sketch directly so each value is only hashed once. Coupon can't be built from an existing hash since pack is private, so that might be worth raising upstream too.

Comment thread Cargo.lock Outdated
dependencies = [
"heck 0.5.0",
"itertools 0.13.0",
"itertools 0.14.0",

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you rebase on main? Cargo.lock conflicts right now, and it also picks up some unrelated bumps here (itertools 0.14 for prost-build and getrandom 0.4 for tempfile). datasketches has no dependencies, so taking main's lockfile and letting cargo add just that entry should keep the diff minimal.

datafusion_common.ScalarValue max_value = 2;
uint32 null_count = 3;
uint32 distinct_count = 4;
reserved 4;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reserving field 4 is the right call on the wire side. Since ColumnStats and PartitionStats::column_stats were public in 55.0.0, downstream code that builds a PartitionStats with a struct literal will stop compiling. Could you add a short note to docs/source/upgrading/56.0.0.md? We should also add the api change label.

Comment thread ballista/core/src/hll.rs
precision,
registers: bytes.into(),
})
let sketch = HllSketch::deserialize(bytes).ok()?;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small question. The old from_bytes rejected any register above max_rank, and now we rely on HllSketch::deserialize. That checks lengths and counts, but Array4::update still has an .expect(...) for an AUX_TOKEN register with no aux entry. Executor to scheduler traffic is trusted, so this isn't a big worry. Do you know if upstream guarantees that a sketch that deserializes OK can't panic later in HllUnion::update?

Comment thread ballista/core/src/hll.rs Outdated
//! task's sketch for a column into a stage-level distinct count with a
//! [`HyperLogLogUnion`].
//!
//! Values are hashed with DataFusion's `HLL_RANDOM_STATE`, the hash state

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: since the sketch hashes the values again, HLL_RANDOM_STATE doesn't really decide anything about the result anymore. Might be worth saying so here, so nobody assumes the sketch is compatible with DataFusion's approx_distinct.

@coderfender

Copy link
Copy Markdown
Contributor Author

TPC-H SF10, local cluster (1 scheduler, 2 executors, 18 cores), 3 iterations, minimum per query in seconds:

Query collect_distinct_counts=false main (74c0924) this PR PR vs main
q1 0.292 0.276 0.277 +0.2%
q2 0.097 0.091 0.086 -5.4%
q3 0.200 0.182 0.185 +1.5%
q4 0.194 0.170 0.175 +2.7%
q5 0.242 0.257 0.253 -1.5%
q6 0.101 0.098 0.099 +0.4%
q7 0.569 0.577 0.596 +3.2%
q8 0.284 0.289 0.285 -1.6%
q9 0.584 0.582 0.619 +6.4%
q10 0.375 0.360 0.394 +9.5%
q11 0.055 0.066 0.053 -20.0%
q12 0.233 0.234 0.237 +1.2%
q13 0.300 0.297 0.308 +3.5%
q14 0.133 0.136 0.137 +0.5%
q15 0.228 0.229 0.231 +0.7%
q16 0.094 0.094 0.099 +4.6%
q17 0.953 0.856 0.769 -10.2%
q18 1.326 1.423 1.214 -14.7%
q19 0.257 0.248 0.253 +1.9%
q20 0.359 0.392 0.380 -3.1%
q21 0.955 0.899 0.874 -2.7%
q22 0.132 0.130 0.129 -0.9%
Total 7.96 7.89 7.65 -3.0%

Per-query differences are within about ±10% run-to-run noise in both directions, and turning the sketches off isn't faster than either implementation. The extra hash per value is real but doesn't show end to end at this scale. I'll raise a Coupon::from_hash constructor upstream so the second hash can be dropped.

@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants