Skip to content

PandasCursor infers float64 for integer and JSON columns with NULL on managed query result storage #1027

Description

@laughingman7743

Problem

With managed query result storage, PandasCursor builds its DataFrame from the GetQueryResults rows (AthenaPandasResultSet._as_pandas_from_api(), pyathena/pandas/result_set.py:844-849) and lets pandas infer the dtypes.
A BIGINT column with a NULL becomes float64, so integers above 2**53 lose precision and NULL becomes NaN.
With an S3 result file, the same column is Int64 and keeps the value.

A json column whose values are numbers and NULL becomes float64 on both paths, so large JSON integers also lose precision there.

Expected: the managed path returns the same dtypes and values as the S3 path (Int64 with <NA>), and JSON numbers keep their decoded Python values.

Reproduction

Measured on Athena on master 16aef64:

from pyathena import connect
from pyathena.pandas.cursor import PandasCursor

sql = """
SELECT id, b, j FROM (VALUES
  (1, BIGINT '9007199254740993', json_parse('9007199254740993')),
  (2, NULL, NULL)
) AS t(id, b, j) ORDER BY id
"""
cursor = connect(work_group="<managed work group>", s3_staging_dir="", cursor_class=PandasCursor).cursor()
cursor.execute(sql)
cursor.fetchall()
# [(1, 9007199254740992.0, 9007199254740992.0), (2, nan, nan)]
cursor.as_pandas().dtypes  # id int64, b float64, j float64

With an S3 result file: [(1, 9007199254740993, 9007199254740992.0), (2, <NA>, nan)], dtypes Int64, Int64, float64.

Environment

  • PyAthena master 16aef64, Python 3.13.1, pandas 3.0.6, PandasCursor (the async and aio pandas cursors share the result set), a work group with managed query result storage.

Found during the review of #1010 (#934).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions