Describe the bug
With Comet enabled, regr_r2 returns NULL where Spark returns 1.0 when the x argument is constant (a literal, or a column with a single value). Spark alone returns the result shown first. The executed plan runs it through CometColumnarToRow, CometExchange, CometHashAggregate, CometNativeScan.
Steps to reproduce
CREATE TABLE test_regr(y double, x double, grp string) USING parquet;
INSERT INTO test_regr VALUES
(1.0, 2.0, 'a'),
(3.0, 4.0, 'a'),
(5.0, 6.0, 'a'),
(7.0, 8.0, 'a'),
(9.0, 10.0, 'a'),
(2.0, 1.0, 'b'),
(4.0, 3.0, 'b'),
(6.0, 5.0, 'b'),
(NULL, 3.0, 'b'),
(4.0, NULL, 'b'),
(NULL, NULL, 'b');
CREATE TABLE test_regr_const_x(y double, x double) USING parquet;
INSERT INTO test_regr_const_x VALUES (1.0, 5.0), (2.0, 5.0), (3.0, 5.0), (4.0, 5.0);
CREATE TABLE test_regr_const_y(y double, x double) USING parquet;
INSERT INTO test_regr_const_y VALUES (7.0, 1.0), (7.0, 2.0), (7.0, 3.0), (7.0, 4.0);
SELECT regr_slope(y, 3.0), regr_sxx(y, 3.0), regr_r2(y, 3.0) FROM test_regr;;
Spark:
| regr_slope(y, 3.0) |
regr_sxx(y, 3.0) |
regr_r2(y, 3.0) |
| NULL |
0.0 |
1.0 |
Comet:
| regr_slope(y, 3.0) |
regr_sxx(y, 3.0) |
regr_r2(y, 3.0) |
| NULL |
0.0 |
NULL |
Expected behavior
The same rows, schema and nullability as Spark.
Additional context
- Comet built from
apache/main at commit 481aefea9c60592612650de73bd2e1c7aa173979 (2026-09-13T19:58:05+00:00)
- Spark 3.5.3, JDK 17, Linux x86_64,
local[4]
- Configuration:
spark.comet.enabled=true
spark.comet.exec.enabled=true
spark.comet.exec.shuffle.enabled=true
spark.memory.offHeap.enabled=true
spark.memory.offHeap.size=8g
spark.plugins=org.apache.spark.CometPlugin
spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager
Further failing variants of the same expression (same fixture, different arguments):
SELECT regr_slope(y, x), regr_intercept(y, x), regr_r2(y, x) FROM test_regr_const_x
SELECT regr_slope(y, x), regr_intercept(y, x), regr_r2(y, x) FROM test_regr_const_y
Describe the bug
With Comet enabled,
regr_r2returns NULL where Spark returns 1.0 when the x argument is constant (a literal, or a column with a single value). Spark alone returns the result shown first. The executed plan runs it through CometColumnarToRow, CometExchange, CometHashAggregate, CometNativeScan.Steps to reproduce
Spark:
Comet:
Expected behavior
The same rows, schema and nullability as Spark.
Additional context
apache/mainat commit481aefea9c60592612650de73bd2e1c7aa173979(2026-09-13T19:58:05+00:00)local[4]spark.comet.enabled=truespark.comet.exec.enabled=truespark.comet.exec.shuffle.enabled=truespark.memory.offHeap.enabled=truespark.memory.offHeap.size=8gspark.plugins=org.apache.spark.CometPluginspark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManagerFurther failing variants of the same expression (same fixture, different arguments):
SELECT regr_slope(y, x), regr_intercept(y, x), regr_r2(y, x) FROM test_regr_const_xSELECT regr_slope(y, x), regr_intercept(y, x), regr_r2(y, x) FROM test_regr_const_y