[feat](array) support trim_array function by vajaw · Pull Request #67397 · apache/doris

vajaw · 2026-09-01T12:52:18Z

What problem does this PR solve?

Issue Number: Related to #48203

Problem Summary:

Add the trim_array(array, n) scalar function. The BE implementation
trims trailing elements directly from array storage and avoids virtual
function calls in the hot loop.

Register the function and its signature in the Nereids planner. Add FE
constant folding so constant invocations can be evaluated during
planning.

Cover normal and constant columns, NULL values, nullable elements,
nested arrays, multiple element types, zero, negative and out-of-range
sizes, and the maximum BIGINT value.

Release note

Add the TRIM_ARRAY scalar function.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason

Test results:

BE ASAN build passed.
FunctionArrayTrimTest.all_argument_combinations passed.
FE unit tests passed: 2 tests, 0 failures and 0 errors.
The trim_array regression suite passed with no mismatches.
run-be-ut.sh and run-regression-test.sh completed
successfully for the relevant tests.
Behavior changed:
- No.
- Yes. Adds the new trim_array scalar function.
Does this need documentation?
- No.
- Yes.
  [doc] Add trim_array function documentation doris-website#4106

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Implement trim_array in BE and register its Nereids signature. Add constant folding, BE tests for const-column combinations, and regression coverage for nulls, nested arrays, boundary sizes, and types. Related to apache#48203

hello-stephen · 2026-09-01T12:52:25Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

vajaw · 2026-09-02T03:25:07Z

run buildall

hello-stephen · 2026-09-03T11:45:36Z

FE UT Coverage Report

Increment line coverage 40.91% (9/22) 🎉
Increment coverage report
Complete coverage report

vajaw · 2026-09-03T12:13:17Z

run vault_p0

vajaw · 2026-09-03T12:16:47Z

/review

hello-stephen · 2026-09-03T13:40:34Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	76.28% (34651/45425)
Line Coverage	61.37% (391172/637400)
Region Coverage	57.45% (328322/571522)
Branch Coverage	58.40% (149994/256828)

hello-stephen · 2026-09-03T14:06:10Z

FE Regression Coverage Report

Increment line coverage 25.61% (21/82) 🎉
Increment coverage report
Complete coverage report

linrrzqqq · 2026-09-03T14:36:49Z

+            if (UNLIKELY(size < 0)) {
+                return Status::InvalidArgument("size must not be negative: {}", size);
+            }
+            if (UNLIKELY(static_cast<size_t>(size) > cardinality)) {


这里估计得重写 use_default_implementation_for_nulls

外层默认的 null 处理，对于 array 列来说会替换为[], which cardinality == 0, 所以如果使用默认处理框架，L156应该会报错，得手动处理 null

linrrzqqq · 2026-09-03T14:40:31Z

+    Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
+                        uint32_t result, size_t input_rows_count) const override {
+        auto array_column =
+                block.get_by_position(arguments[0]).column->convert_to_full_column_if_const();


use unpack_if_const and index_check_const better， you can refer to: #64175

linrrzqqq · 2026-09-03T14:52:19Z

+        auto offset_column = ColumnInt64::create(input_rows_count, 1);
+        slice_array(dst, src, *offset_column, length_column.get());


slice_array 里面看起来 array 的每个元素都调用了一次虚函数insert_from/insert_default, 这里就别用了，每行调用一次insert_range_from应该就行, 参考：

Suggested change

auto offset_column = ColumnInt64::create(input_rows_count, 1);

slice_array(dst, src, *offset_column, length_column.get());

size_t keep = cardinality - size;

if (keep > 0) {

result->insert_range_from(src.array_col->get_data(), offset, keep);

}

res_offset += keep;

res_offset_data.push_back(res_offset);

linrrzqqq · 2026-09-03T15:08:19Z

作为一个支持新函数的 pr，我认为这个 pr 里面非必要尽量别改动原有的test_util结构

linrrzqqq · 2026-09-03T15:10:48Z

需要加强一波测试，这里面 null 只和 size = 0 配对，还有需要支持(const, col), (col, const) 这种行为的测试等等

linrrzqqq · 2026-09-03T15:11:33Z

+// under the License.
+
+suite("test_trim_array") {
+    qt_trim_two "select trim_array([1, 2, 3, 4], 2)"


also need testFoldConst to double check

linrrzqqq · 2026-09-06T07:46:56Z

/review

github-actions

Review status: complete (converged after two rounds)

I found one material BE allocation issue and left it inline.

Checkpoint conclusions:

Goal and proof: the new trim_array(array, n) behavior is implemented across BE execution, Nereids binding/constant folding, visitor/registration paths, and regression/unit coverage; valid, NULL, nested, boundary, and mixed const/vector cases are represented.
Scope: all 10 changed files and the relevant framework, column, registration, and evaluator paths were reviewed. The change stays focused on the new function and its tests.
Concurrency, lifecycle, and configuration: this stateless scalar function adds no shared mutable state, threads, lifecycle hook, or configuration surface.
Compatibility and parallel paths: FE signature/nullability/folding and BE factory/runtime routes are aligned. No persisted format, RPC, mixed-version field, storage layout, or alternate execution registration is affected.
Conditions and error handling: NULL precedence and negative/out-of-range validation are consistent between FE and BE; the const/nullable physical-shape and offset paths are valid on this head.
Tests and results: the BE, FE, and regression cases cover core valid/error/null/type combinations and their expected results are consistent with the implementation. I did not run builds or tests because the review instructions prohibit builds; the PR reports the targeted BE ASAN, BE/FE unit, and regression runs passed.
Observability, persistence, transactions, and data writes: not applicable; this function has no distributed operation, durable state, catalog mutation, transaction visibility, or data-write path.
Performance and memory: the prior per-element virtual insertion concern is fixed by bulk range copying, but the output reserve still uses the physical source size and creates the material peak-allocation/reallocation issue described inline.
Other: all six existing live review threads were duplicate-fenced, and no second distinct issue survived the final sweep.

User-provided focus: none beyond the full PR review.
Reviewed head: 89106bc9371f9ac6d0d6942885a6d61d20b9dfd0.

github-actions · 2026-09-06T08:17:59Z

+                                                ColumnArray::ColumnOffsets::create());
+        auto& result_data = result_array->get_data();
+        auto& result_offsets = result_array->get_offsets();
+        result_data.reserve(src.array_col->get_data().size());


result_data.reserve uses the physical source element count before any trim size is examined. For a block that trims large numeric/nullable arrays completely, this eagerly allocates almost another full input buffer (and nullable null map) while the input is still live, even though the result data is empty; near the memory limit that can fail an otherwise tiny-result query. A const array has the opposite problem because its physical source contains one array while the logical result may repeat it for every row, so this estimate under-reserves and grows repeatedly. This is separate from the earlier per-element insertion thread. Please validate sizes/build offsets in a first pass, compute the overflow-checked total retained element count, reserve that exact total, and then bulk-copy the retained ranges.

linrrzqqq · 2026-09-06T08:59:27Z

+    qt_trim_zero "select trim_array([1, 2, 3, 4], 0)"
+    qt_trim_one "select trim_array([1, 2, 3, 4], 1)"
+    qt_trim_all "select trim_array([1, 2, 3, 4], 4)"
+    qt_trim_string "select trim_array(['a', 'b', 'c', 'd'], 1)"
+    qt_trim_null_element "select trim_array(['a', 'b', null, 'd'], 1)"
+    qt_trim_nested "select trim_array([[1, 2, 3], [4, 5, 6]], 1)"
+    qt_trim_empty "select trim_array(cast([] as array<int>), 0)"
+    qt_trim_boolean "select trim_array(cast([true, false, true] as array<boolean>), 1)"
+    qt_trim_tinyint "select trim_array(cast([-128, 0, 127] as array<tinyint>), 1)"
+    qt_trim_bigint "select trim_array(cast([-9223372036854775808, 0, 9223372036854775807] as array<bigint>), 1)"
+    qt_trim_double "select trim_array(cast([-1.7976931348623157E308, 0.0, 1.7976931348623157E308] as array<double>), 1)"
+    qt_trim_decimal "select trim_array(cast([-99999999.99, 0.00, 99999999.99] as array<decimal(10, 2)>), 1)"
+    qt_trim_date "select trim_array(cast(['0000-01-01', '2024-02-29', '9999-12-31'] as array<date>), 1)"
+    qt_trim_null_array "select trim_array(cast(null as array<int>), 0)"
+    qt_trim_null_array_invalid_size "select trim_array(cast(null as array<int>), 9223372036854775807)"
+    qt_trim_null_size "select trim_array([1, 2, 3], cast(null as bigint))"


这些常量的，最好都再测一遍testFoldConst 保证 be 和 fe 常量折叠行为一致

感谢您的审查！两个评论已在11f8aa5c中得到处理，能否请您再看一遍？谢谢。
1、BE实现目前会在第一遍遍历中验证修剪大小并计算输出偏移量，同时进行溢出检查。随后在第二遍遍历中预留总保留元素数量，并批量复制保留的范围。
2、目前，所有17个成功的常量输入案例均采用testFoldConst来验证前端常量折叠与后端执行的一致性，涵盖NULL值、可空元素、嵌套数组、空数组及不同元素类型。

…tests

vajaw · 2026-09-07T12:28:59Z

/review

Support trim_array function

96db74f

Implement trim_array in BE and register its Nereids signature. Add constant folding, BE tests for const-column combinations, and regression coverage for nulls, nested arrays, boundary sizes, and types. Related to apache#48203

vajaw requested review from 924060929, englefly, morrySnow and starocean999 as code owners September 1, 2026 12:52

linrrzqqq self-assigned this Sep 3, 2026

vajaw changed the title ~~Support trim_array function~~ [feat](array) support trim_array function Sep 3, 2026

linrrzqqq reviewed Sep 3, 2026

View reviewed changes

[fix](array) address trim_array review comments

89106bc

github-actions Bot requested changes Sep 6, 2026

View reviewed changes

linrrzqqq reviewed Sep 6, 2026

View reviewed changes

[fix](array) reserve retained trim_array elements and extend folding …

11f8aa5

…tests

		auto offset_column = ColumnInt64::create(input_rows_count, 1);
		slice_array(dst, src, *offset_column, length_column.get());

-        auto offset_column = ColumnInt64::create(input_rows_count, 1);
-        slice_array(dst, src, *offset_column, length_column.get());
+        size_t keep = cardinality - size;
+        if (keep > 0) {
+            result->insert_range_from(src.array_col->get_data(), offset, keep);
+        }
+        res_offset += keep;
+        res_offset_data.push_back(res_offset);

Uh oh!

Conversation

vajaw commented Sep 1, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Sep 1, 2026

Uh oh!

vajaw commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hello-stephen commented Sep 3, 2026

FE UT Coverage Report

Uh oh!

vajaw commented Sep 3, 2026

Uh oh!

vajaw commented Sep 3, 2026

Uh oh!

hello-stephen commented Sep 3, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Sep 3, 2026

FE Regression Coverage Report

Uh oh!

linrrzqqq Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

linrrzqqq Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

linrrzqqq Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

linrrzqqq Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

linrrzqqq Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

linrrzqqq Sep 3, 2026

Choose a reason for hiding this comment

Uh oh!

linrrzqqq commented Sep 6, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

linrrzqqq Sep 6, 2026

Choose a reason for hiding this comment

Uh oh!

vajaw Sep 7, 2026

Choose a reason for hiding this comment

Uh oh!

vajaw commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

vajaw commented Sep 2, 2026 •

edited

Loading

linrrzqqq Sep 3, 2026 •

edited

Loading