Problem
With managed query result storage, PandasCursor builds its DataFrame from the GetQueryResults rows (AthenaPandasResultSet._as_pandas_from_api(), pyathena/pandas/result_set.py:844-849) and lets pandas infer the dtypes.
A BIGINT column with a NULL becomes float64, so integers above 2**53 lose precision and NULL becomes NaN.
With an S3 result file, the same column is Int64 and keeps the value.
A json column whose values are numbers and NULL becomes float64 on both paths, so large JSON integers also lose precision there.
Expected: the managed path returns the same dtypes and values as the S3 path (Int64 with <NA>), and JSON numbers keep their decoded Python values.
Reproduction
Measured on Athena on master 16aef64:
from pyathena import connect
from pyathena.pandas.cursor import PandasCursor
sql = """
SELECT id, b, j FROM (VALUES
(1, BIGINT '9007199254740993', json_parse('9007199254740993')),
(2, NULL, NULL)
) AS t(id, b, j) ORDER BY id
"""
cursor = connect(work_group="<managed work group>", s3_staging_dir="", cursor_class=PandasCursor).cursor()
cursor.execute(sql)
cursor.fetchall()
# [(1, 9007199254740992.0, 9007199254740992.0), (2, nan, nan)]
cursor.as_pandas().dtypes # id int64, b float64, j float64
With an S3 result file: [(1, 9007199254740993, 9007199254740992.0), (2, <NA>, nan)], dtypes Int64, Int64, float64.
Environment
- PyAthena master 16aef64, Python 3.13.1, pandas 3.0.6,
PandasCursor (the async and aio pandas cursors share the result set), a work group with managed query result storage.
Found during the review of #1010 (#934).
Problem
With managed query result storage,
PandasCursorbuilds its DataFrame from theGetQueryResultsrows (AthenaPandasResultSet._as_pandas_from_api(),pyathena/pandas/result_set.py:844-849) and lets pandas infer the dtypes.A
BIGINTcolumn with a NULL becomesfloat64, so integers above 2**53 lose precision and NULL becomesNaN.With an S3 result file, the same column is
Int64and keeps the value.A
jsoncolumn whose values are numbers and NULL becomesfloat64on both paths, so large JSON integers also lose precision there.Expected: the managed path returns the same dtypes and values as the S3 path (
Int64with<NA>), and JSON numbers keep their decoded Python values.Reproduction
Measured on Athena on master 16aef64:
With an S3 result file:
[(1, 9007199254740993, 9007199254740992.0), (2, <NA>, nan)], dtypesInt64,Int64,float64.Environment
PandasCursor(the async and aio pandas cursors share the result set), a work group with managed query result storage.Found during the review of #1010 (#934).