Repository navigation
feat: setup datasketches and update hll impl - #2868
coderfender wants to merge 3 commits into
Conversation
andygrove
left a comment
There was a problem hiding this comment.
Thanks for this! It's a nice cleanup, and it's great to lean on DataSketches rather than maintain our own estimator. The union logic in MergedSketch::add and ColumnStatsStream::drop looks right to me. Merging mixed precisions at the lowest one is a nice improvement over returning Absent. Memory looks the same or better too, since the bytes stored per task roughly halve.
The main thing I'd like to see before merging is benchmark numbers, because the per-row sketch cost went up on a default-on path. Details are inline, along with a rebase request and an upgrade guide note.
| create_hashes([array], &HLL_RANDOM_STATE, hashes)?; | ||
| match array.logical_nulls() { | ||
| None => hashes.iter().for_each(|&hash| self.add_hashed(hash)), | ||
| None => hashes.iter().for_each(|&hash| self.0.update(hash)), |
There was a problem hiding this comment.
One thing I noticed is that HllSketch::update hashes its input again with Murmur3, so each key value now gets hashed twice on the shuffle write path (once by create_hashes, once inside the sketch). A quick microbenchmark of HyperLogLog::update over 8M distinct Int64 values at precision 12 went from about 15 ms to about 40 ms on my machine.
Since ballista.shuffle.collect_distinct_counts is on by default, could you share before and after TPC-H numbers against main? Something like:
cargo run --release --bin tpch -- benchmark ballista \
--host localhost --port 50050 \
--iterations 3 --path <tpch parquet dir> --format parquetat SF10 or higher would be ideal. If the overhead shows up, one option is to skip create_hashes and feed the typed values to the sketch directly so each value is only hashed once. Coupon can't be built from an existing hash since pack is private, so that might be worth raising upstream too.
| dependencies = [ | ||
| "heck 0.5.0", | ||
| "itertools 0.13.0", | ||
| "itertools 0.14.0", |
There was a problem hiding this comment.
Could you rebase on main? Cargo.lock conflicts right now, and it also picks up some unrelated bumps here (itertools 0.14 for prost-build and getrandom 0.4 for tempfile). datasketches has no dependencies, so taking main's lockfile and letting cargo add just that entry should keep the diff minimal.
| datafusion_common.ScalarValue max_value = 2; | ||
| uint32 null_count = 3; | ||
| uint32 distinct_count = 4; | ||
| reserved 4; |
There was a problem hiding this comment.
Reserving field 4 is the right call on the wire side. Since ColumnStats and PartitionStats::column_stats were public in 55.0.0, downstream code that builds a PartitionStats with a struct literal will stop compiling. Could you add a short note to docs/source/upgrading/56.0.0.md? We should also add the api change label.
| precision, | ||
| registers: bytes.into(), | ||
| }) | ||
| let sketch = HllSketch::deserialize(bytes).ok()?; |
There was a problem hiding this comment.
Small question. The old from_bytes rejected any register above max_rank, and now we rely on HllSketch::deserialize. That checks lengths and counts, but Array4::update still has an .expect(...) for an AUX_TOKEN register with no aux entry. Executor to scheduler traffic is trusted, so this isn't a big worry. Do you know if upstream guarantees that a sketch that deserializes OK can't panic later in HllUnion::update?
| //! task's sketch for a column into a stage-level distinct count with a | ||
| //! [`HyperLogLogUnion`]. | ||
| //! | ||
| //! Values are hashed with DataFusion's `HLL_RANDOM_STATE`, the hash state |
There was a problem hiding this comment.
Nit: since the sketch hashes the values again, HLL_RANDOM_STATE doesn't really decide anything about the result anymore. Might be worth saying so here, so nobody assumes the sketch is compatible with DataFusion's approx_distinct.
|
TPC-H SF10, local cluster (1 scheduler, 2 executors, 18 cores), 3 iterations, minimum per query in seconds:
Per-query differences are within about ±10% run-to-run noise in both directions, and turning the sketches off isn't faster than either implementation. The extra hash per value is real but doesn't show end to end at this scale. I'll raise a |
Which issue does this PR close?
Part of #1827. See #2783.
Rationale for this change
Use Apache DataSketches for the HyperLogLog sketch instead of our hand-written one.
What changes are included in this PR?
datasketches(hllfeature) and backHyperLogLogwith itsHllSketch.HyperLogLogUnion.ColumnStatsproto message.Are there any user-facing changes?
HyperLogLog::as_bytesis nowto_bytesandmergeis replaced byHyperLogLogUnion. HLL sketch bytes use the DataSketches format.