[feature](bucket) Support custom distribution_hash_type for Hash Bucketing - #66477
Open
zghong wants to merge 17 commits into
Open
[feature](bucket) Support custom distribution_hash_type for Hash Bucketing#66477zghong wants to merge 17 commits into
zghong wants to merge 17 commits into
Conversation
zghong
requested review from
924060929,
CalvinKirs,
dataroaring,
englefly,
gavinchou,
morningman,
morrySnow,
starocean999,
xiaokang and
yiguolei
as code owners
August 5, 2026 07:28
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29153 ms |
Contributor
TPC-DS: Total hot run time: 166066 ms |
Contributor
ClickBench: Total hot run time: 23.98 s |
…ion_hash_type identity
…ash_type # Conflicts: # be/src/exec/partitioner/partitioner.h # fe/fe-core/src/main/java/org/apache/doris/planner/HashDistributionPruner.java # fe/fe-core/src/test/java/org/apache/doris/planner/HashDistributionPrunerTest.java
Carry the effective storage hash type through FE-planned and BE-native local bucket exchanges. Add fragment protocol metadata and execution-version gating so older backends cannot silently execute IDENTITY layouts as CRC32.
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 17044 ms |
Contributor
TPC-DS: Total hot run time: 82231 ms |
Contributor
ClickBench: Total hot run time: 14.59 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Author
|
run buildall |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
TPC-H: Total hot run time: 16772 ms |
Contributor
TPC-DS: Total hot run time: 82337 ms |
Contributor
ClickBench: Total hot run time: 14.83 s |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE Regression Coverage ReportIncrement line coverage |
…egacy values, and local-exchange hash
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
TPC-H: Total hot run time: 16901 ms |
Contributor
TPC-DS: Total hot run time: 81949 ms |
Contributor
ClickBench: Total hot run time: 14.7 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #66231
Problem Summary:
Hash Bucketing previously supported only the CRC32 hash function to map a row to its bucket. There was no way for users to choose a different bucketing scheme, which prevents scenarios (e.g. bucket pruning / colocation with external systems) that rely on a plain identity mapping of the bucket key, as shown in #66231.
This PR introduces a table-level
distribution_hash_typeproperty for Hash Bucketing tables, supporting two values:crc32(default, keeps existing behavior), meanscrc32(key) % bucket_num.identity(map rows to buckets by the raw key value), meanskey % bucket_num.Main changes:
distribution_hash_typeproperty inPropertyAnalyzer; carry the hash type throughHashDistributionInfo,HashDistributionDesc,ColocateGroupSchema,DataPartition,DistributionSpecHashand related create-table / colocate paths; a newFeMetaVersionis added for metadata persistence.ADD PARTITIONnow inherits the table hash type.HashDistributionPruner,PruneOlapScanTablet) still works correctly for identity buckets.TDistributionHashTypeis added to thrift and threaded through the exchange / local-exchange sink operators and the tablet-sink partitioner so that the backend uses the same bucketing function as the FE plan, keeping FE/BE bucketing consistent. An identity partitioner is implemented.TDistributionHashTypeand propagate the field inDescriptors,Partitions,PlanNodesandTypes.Before this PR: only CRC32 bucketing was available.
After this PR: users can create Hash Bucketing tables with
"distribution_hash_type" = "identity", and FE plan optimization plus BE data sink produce consistent bucketing results. And more hash functions can be extended in the future.Release note
Support a new table property
distribution_hash_typefor Hash Bucketing tables. It acceptscrc32(default) andidentitynow, allowing users to choose the hash function used to map rows to buckets.Check List (For Author)
Test
Behavior changed:
distribution_hash_typeproperty is added; but default behavior (crc32) is unchanged.Does this need documentation?
Check List (For Reviewer who merge this PR)