Skip to content

PandasCursor with engine="pyarrow" changes numeric-looking names in DDL results such as SHOW TABLES and DESCRIBE #1073

Description

@laughingman7743

Problem

With engine="pyarrow", PandasCursor changes the values of DDL statement results (the tab-separated .txt result files) when every value in a column looks like a number.
pyarrow infers a number type for such a column, and the str dtype then turns the numbers back into text, so leading zeros, exponent notation, and the padding Athena adds are lost.
The C engine (the default, engine="auto") returns the values unchanged.

Athena accepts names that consist of digits when they are quoted with backticks, such as a table or column named 001, so this happens for SHOW TABLES, SHOW COLUMNS, and DESCRIBE of such objects.
The PyArrow engine is used only for result files of at least AthenaPandasResultSet.PYARROW_MIN_FILE_SIZE_BYTES (100) bytes.

#1065 (#1062) fixed the same problem for query results, whose CSV files have a header, by reading string-dtype columns as text.
It kept the inferred types for header-less files, because pandas assigns their names to the last fields only after reading, and counting the fields of the first record before reading disagreed with pyarrow on some input (quoted delimiters, long fields, bare CR line ends, a byte order mark).
This is noted in docs/pandas.md.
The behavior predates #1057, which made _read_csv_with_pyarrow() reproduce pandas' PyArrow engine.

Reproduction

Measured on Athena with master 6d268b1, pandas 3.0.6, pyarrow 25.0.1, in a database with 30 tables named 000 to 029 and a table wide with 30 int columns named 000 to 029:

import pandas as pd
from pyathena import connect
from pyathena.pandas.cursor import PandasCursor

cursor = connect(cursor_class=PandasCursor).cursor()
cursor.execute("SHOW TABLES IN db", engine="pyarrow")
cursor.as_pandas().iloc[:3, 0].tolist()  # ['0', '1', '2']
cursor.execute("SHOW TABLES IN db")
cursor.as_pandas().iloc[:3, 0].tolist()  # ['000', '001', '002']
statement result file C engine PyArrow engine
SHOW TABLES IN db 119 bytes '000', '001', … '0', '1', …
SHOW COLUMNS IN db.wide 629 bytes '000 ', … '0', …
DESCRIBE db.wide (first column) 1889 bytes '000 ', … '0', …

The results are the same with future.infer_string on and off.
A column that mixes text and numbers is read as text, so for example SHOW TBLPROPERTIES was not affected in the same measurement: its value column always includes EXTERNAL/TRUE or table_type/ICEBERG.

Expected

DDL results have the same values with engine="pyarrow" as with the C engine.

Notes

Environment

  • PyAthena master 6d268b1, Python 3.13.1, pandas 3.0.6, pyarrow 25.0.1, PandasCursor with engine="pyarrow".

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions