Skip to content

regr_r2 returns NULL instead of 1.0 when x is constant #5931

Description

@andygrove

Describe the bug

With Comet enabled, regr_r2 returns NULL where Spark returns 1.0 when the x argument is constant (a literal, or a column with a single value). Spark alone returns the result shown first. The executed plan runs it through CometColumnarToRow, CometExchange, CometHashAggregate, CometNativeScan.

Steps to reproduce

CREATE TABLE test_regr(y double, x double, grp string) USING parquet;
INSERT INTO test_regr VALUES
  (1.0,  2.0, 'a'),
  (3.0,  4.0, 'a'),
  (5.0,  6.0, 'a'),
  (7.0,  8.0, 'a'),
  (9.0, 10.0, 'a'),
  (2.0,  1.0, 'b'),
  (4.0,  3.0, 'b'),
  (6.0,  5.0, 'b'),
  (NULL,  3.0, 'b'),
  (4.0,  NULL, 'b'),
  (NULL, NULL, 'b');
CREATE TABLE test_regr_const_x(y double, x double) USING parquet;
INSERT INTO test_regr_const_x VALUES (1.0, 5.0), (2.0, 5.0), (3.0, 5.0), (4.0, 5.0);
CREATE TABLE test_regr_const_y(y double, x double) USING parquet;
INSERT INTO test_regr_const_y VALUES (7.0, 1.0), (7.0, 2.0), (7.0, 3.0), (7.0, 4.0);
SELECT regr_slope(y, 3.0), regr_sxx(y, 3.0), regr_r2(y, 3.0) FROM test_regr;;

Spark:

regr_slope(y, 3.0) regr_sxx(y, 3.0) regr_r2(y, 3.0)
NULL 0.0 1.0

Comet:

regr_slope(y, 3.0) regr_sxx(y, 3.0) regr_r2(y, 3.0)
NULL 0.0 NULL

Expected behavior

The same rows, schema and nullability as Spark.

Additional context

  • Comet built from apache/main at commit 481aefea9c60592612650de73bd2e1c7aa173979 (2026-09-13T19:58:05+00:00)
  • Spark 3.5.3, JDK 17, Linux x86_64, local[4]
  • Configuration:
  • spark.comet.enabled=true
  • spark.comet.exec.enabled=true
  • spark.comet.exec.shuffle.enabled=true
  • spark.memory.offHeap.enabled=true
  • spark.memory.offHeap.size=8g
  • spark.plugins=org.apache.spark.CometPlugin
  • spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager

Further failing variants of the same expression (same fixture, different arguments):

  • SELECT regr_slope(y, x), regr_intercept(y, x), regr_r2(y, x) FROM test_regr_const_x
  • SELECT regr_slope(y, x), regr_intercept(y, x), regr_r2(y, x) FROM test_regr_const_y

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions