Problem
With engine="pyarrow", PandasCursor changes the values of DDL statement results (the tab-separated .txt result files) when every value in a column looks like a number.
pyarrow infers a number type for such a column, and the str dtype then turns the numbers back into text, so leading zeros, exponent notation, and the padding Athena adds are lost.
The C engine (the default, engine="auto") returns the values unchanged.
Athena accepts names that consist of digits when they are quoted with backticks, such as a table or column named 001, so this happens for SHOW TABLES, SHOW COLUMNS, and DESCRIBE of such objects.
The PyArrow engine is used only for result files of at least AthenaPandasResultSet.PYARROW_MIN_FILE_SIZE_BYTES (100) bytes.
#1065 (#1062) fixed the same problem for query results, whose CSV files have a header, by reading string-dtype columns as text.
It kept the inferred types for header-less files, because pandas assigns their names to the last fields only after reading, and counting the fields of the first record before reading disagreed with pyarrow on some input (quoted delimiters, long fields, bare CR line ends, a byte order mark).
This is noted in docs/pandas.md.
The behavior predates #1057, which made _read_csv_with_pyarrow() reproduce pandas' PyArrow engine.
Reproduction
Measured on Athena with master 6d268b1, pandas 3.0.6, pyarrow 25.0.1, in a database with 30 tables named 000 to 029 and a table wide with 30 int columns named 000 to 029:
import pandas as pd
from pyathena import connect
from pyathena.pandas.cursor import PandasCursor
cursor = connect(cursor_class=PandasCursor).cursor()
cursor.execute("SHOW TABLES IN db", engine="pyarrow")
cursor.as_pandas().iloc[:3, 0].tolist() # ['0', '1', '2']
cursor.execute("SHOW TABLES IN db")
cursor.as_pandas().iloc[:3, 0].tolist() # ['000', '001', '002']
| statement |
result file |
C engine |
PyArrow engine |
SHOW TABLES IN db |
119 bytes |
'000', '001', … |
'0', '1', … |
SHOW COLUMNS IN db.wide |
629 bytes |
'000 ', … |
'0', … |
DESCRIBE db.wide (first column) |
1889 bytes |
'000 ', … |
'0', … |
The results are the same with future.infer_string on and off.
A column that mixes text and numbers is read as text, so for example SHOW TBLPROPERTIES was not affected in the same measurement: its value column always includes EXTERNAL/TRUE or table_type/ICEBERG.
Expected
DDL results have the same values with engine="pyarrow" as with the C engine.
Notes
Environment
- PyAthena master 6d268b1, Python 3.13.1, pandas 3.0.6, pyarrow 25.0.1,
PandasCursor with engine="pyarrow".
Problem
With
engine="pyarrow",PandasCursorchanges the values of DDL statement results (the tab-separated.txtresult files) when every value in a column looks like a number.pyarrow infers a number type for such a column, and the
strdtype then turns the numbers back into text, so leading zeros, exponent notation, and the padding Athena adds are lost.The C engine (the default,
engine="auto") returns the values unchanged.Athena accepts names that consist of digits when they are quoted with backticks, such as a table or column named
001, so this happens forSHOW TABLES,SHOW COLUMNS, andDESCRIBEof such objects.The PyArrow engine is used only for result files of at least
AthenaPandasResultSet.PYARROW_MIN_FILE_SIZE_BYTES(100) bytes.#1065 (#1062) fixed the same problem for query results, whose CSV files have a header, by reading string-dtype columns as text.
It kept the inferred types for header-less files, because pandas assigns their names to the last fields only after reading, and counting the fields of the first record before reading disagreed with pyarrow on some input (quoted delimiters, long fields, bare CR line ends, a byte order mark).
This is noted in
docs/pandas.md.The behavior predates #1057, which made
_read_csv_with_pyarrow()reproduce pandas' PyArrow engine.Reproduction
Measured on Athena with master 6d268b1, pandas 3.0.6, pyarrow 25.0.1, in a database with 30 tables named
000to029and a tablewidewith 30 int columns named000to029:SHOW TABLES IN db'000','001', …'0','1', …SHOW COLUMNS IN db.wide'000 ', …'0', …DESCRIBE db.wide(first column)'000 ', …'0', …The results are the same with
future.infer_stringon and off.A column that mixes text and numbers is read as text, so for example
SHOW TBLPROPERTIESwas not affected in the same measurement: its value column always includesEXTERNAL/TRUEortable_type/ICEBERG.Expected
DDL results have the same values with
engine="pyarrow"as with the C engine.Notes
.txtresults return small files, so reading them with the C engine, for example by having_get_csv_engine()choose it for.txtoutput, would avoid the inference without the field counting that Read string columns as text for the pandas PyArrow engine #1065 dropped.docs/pandas.mdwould then drop the DDL exception.Environment
PandasCursorwithengine="pyarrow".