Skip to content

[improvement](hash) Xxhash 128 - #67318

Open
philipandwaffle wants to merge 3 commits into
apache:masterfrom
philipandwaffle:xxhash_128
Open

[improvement](hash) Xxhash 128#67318
philipandwaffle wants to merge 3 commits into
apache:masterfrom
philipandwaffle:xxhash_128

Conversation

@philipandwaffle

@philipandwaffle philipandwaffle commented Aug 30, 2026

Copy link
Copy Markdown

Using XXHash_64 results in a hash collision chance of ~93% with 10 billion items (based on this calculator https://kevingal.com/apps/collision.html).

Implementing the XXHash_128 function allows for more items to be hashed before a collision can realistically occur.

Issue Number: close #66704

Problem Summary:

Available hash functions with more entropy such as MD5 or SHA have issues that only xxhash_128 can solve. MD5 is slower and would effect performance when a large number of hashes are being performed. SHA results in a hash that is too large needing to use string representation, preventing its use as a table key.

Release note

XXhash_128 implemented

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 35.29% (6/17) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.00% (34460/45341)
Line Coverage 61.01% (388312/636435)
Region Coverage 57.17% (326052/570318)
Branch Coverage 57.97% (148670/256472)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16983 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 47f5e8e8f7ad32b7f2973906ce93306f6ca19568, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17570	3010	3020	3010
q2	2130	260	227	227
q3	10207	900	519	519
q4	4672	257	206	206
q5	7666	551	385	385
q6	136	116	92	92
q7	529	507	382	382
q8	9232	880	905	880
q9	3514	2415	2409	2409
q10	6535	864	730	730
q11	389	204	182	182
q12	607	263	200	200
q13	18114	1532	1166	1166
q14	160	158	139	139
q15	q16	434	394	377	377
q17	1396	982	781	781
q18	3190	2323	2286	2286
q19	1117	879	789	789
q20	359	288	199	199
q21	5149	1790	1872	1790
q22	329	263	234	234
Total cold run time: 93435 ms
Total hot run time: 16983 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3357	3303	3297	3297
q2	515	396	369	369
q3	2321	2421	2301	2301
q4	1216	1196	901	901
q5	2237	2169	2205	2169
q6	174	117	89	89
q7	1009	956	856	856
q8	1625	1427	1421	1421
q9	3259	3222	3210	3210
q10	1900	1916	1664	1664
q11	367	276	258	258
q12	451	432	351	351
q13	1495	1547	1160	1160
q14	175	178	172	172
q15	q16	392	398	358	358
q17	3789	3445	3388	3388
q18	4974	4568	5106	4568
q19	947	847	840	840
q20	1027	985	841	841
q21	3845	3158	3219	3158
q22	420	343	312	312
Total cold run time: 35495 ms
Total hot run time: 31683 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 83083 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 47f5e8e8f7ad32b7f2973906ce93306f6ca19568, data reload: false

query5	4274	405	334	334
query6	384	137	124	124
query7	4967	435	241	241
query8	288	137	119	119
query9	8677	2901	2896	2896
query10	390	214	173	173
query11	5394	1063	925	925
query12	126	70	72	70
query13	1188	457	315	315
query14	6112	2264	2155	2155
query14_1	2049	2047	2033	2033
query15	170	121	116	116
query16	931	389	365	365
query17	1065	462	364	364
query18	2376	339	253	253
query19	170	147	112	112
query20	87	70	74	70
query21	200	102	87	87
query22	5463	5465	5364	5364
query23	7127	6603	6382	6382
query23_1	6444	6167	6193	6167
query24	7253	1132	807	807
query24_1	823	778	792	778
query25	433	305	274	274
query26	1227	217	130	130
query27	2806	434	263	263
query28	4655	1497	1512	1497
query29	913	417	341	341
query30	249	156	130	130
query31	822	408	332	332
query32	123	72	69	69
query33	452	217	164	164
query34	986	846	470	470
query35	412	432	355	355
query36	584	589	523	523
query37	116	89	69	69
query38	1015	864	893	864
query39	509	505	461	461
query39_1	468	466	475	466
query40	203	88	75	75
query41	55	55	51	51
query42	77	70	72	70
query43	238	249	215	215
query44	1038	544	560	544
query45	112	106	99	99
query46	764	833	518	518
query47	753	759	717	717
query48	306	315	228	228
query49	532	259	187	187
query50	746	271	210	210
query51	7914	8123	8018	8018
query52	81	86	78	78
query53	199	200	157	157
query54	226	188	146	146
query55	77	60	58	58
query56	210	163	174	163
query57	723	660	675	660
query58	205	156	156	156
query59	1262	1288	1126	1126
query60	233	195	172	172
query61	117	119	119	119
query62	343	202	186	186
query63	171	148	144	144
query64	2834	666	554	554
query65	1672	1684	1650	1650
query66	1760	259	201	201
query67	9886	9710	10059	9710
query68	2758	1258	780	780
query69	337	219	192	192
query70	661	639	577	577
query71	255	184	157	157
query72	2303	1721	1567	1567
query73	639	542	320	320
query74	1569	1258	1159	1159
query75	1195	1127	968	968
query76	2277	742	538	538
query77	263	279	218	218
query78	4104	3799	3302	3302
query79	2631	849	572	572
query80	1664	331	285	285
query81	502	155	139	139
query82	610	123	100	100
query83	315	211	189	189
query84	247	113	90	90
query85	816	367	300	300
query86	395	172	170	170
query87	1032	987	923	923
query88	2805	2097	2109	2097
query89	275	197	176	176
query90	1966	130	124	124
query91	131	124	97	97
query92	79	72	70	70
query93	1761	1111	674	674
query94	653	254	229	229
query95	531	257	232	232
query96	808	568	266	266
query97	1113	1090	1046	1046
query98	172	141	134	134
query99	420	350	315	315
Total cold run time: 179186 ms
Total hot run time: 83083 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.76 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 47f5e8e8f7ad32b7f2973906ce93306f6ca19568, data reload: false

query1	0.00	0.00	0.00
query2	0.08	0.04	0.04
query3	0.25	0.11	0.11
query4	1.60	0.08	0.10
query5	0.17	0.16	0.15
query6	1.26	0.69	0.72
query7	0.04	0.01	0.00
query8	0.04	0.02	0.02
query9	0.29	0.22	0.21
query10	0.36	0.38	0.34
query11	0.17	0.12	0.12
query12	0.14	0.12	0.12
query13	0.31	0.31	0.29
query14	0.48	0.46	0.45
query15	0.36	0.35	0.35
query16	0.22	0.23	0.23
query17	0.67	0.71	0.73
query18	0.18	0.17	0.16
query19	1.20	1.07	1.18
query20	0.01	0.02	0.02
query21	15.46	0.16	0.11
query22	5.09	0.04	0.04
query23	16.18	0.26	0.10
query24	3.00	0.34	0.25
query25	0.11	0.04	0.04
query26	0.76	0.16	0.13
query27	0.05	0.03	0.03
query28	3.62	0.53	0.27
query29	12.47	3.18	2.57
query30	0.26	0.11	0.11
query31	2.77	0.38	0.18
query32	3.53	0.32	0.24
query33	1.42	1.48	1.44
query34	15.38	2.17	1.79
query35	1.75	1.77	1.79
query36	0.45	0.29	0.28
query37	0.07	0.04	0.04
query38	0.04	0.03	0.03
query39	0.04	0.02	0.03
query40	0.11	0.08	0.08
query41	0.08	0.03	0.03
query42	0.03	0.03	0.03
query43	0.03	0.02	0.03
Total cold run time: 90.53 s
Total hot run time: 14.76 s

@philipandwaffle philipandwaffle changed the title Xxhash 128 [improvement](hash) Xxhash 128 Sep 2, 2026
@philipandwaffle

Copy link
Copy Markdown
Author

/review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Enhancement] XXHASH_128 Implememtation to Prevent Hash Collision

2 participants