Skip to content

Result set defects: duplicate column names, header-like rows on fallback pages, and Arrow fetch after close #1032

Description

@laughingman7743

Problem

Three defects in the pandas, Arrow, and Polars result sets, found by the independent reviews of #1023 and #1029.
All results below were measured on master 9a373fd (2026-10-03). "managed" means managed query result storage, where these cursors read every row through the GetQueryResults fallback (AthenaResultSet._fetch_all_rows()).

1. Duplicate column names lose or mix values

SELECT 1 AS x, 2 AS x; the expected row is (1, 2):

Cursor S3 result file managed
PandasCursor [(1, 1)] [(1, 1), (2, 2)]
ArrowCursor [(2,)] [(1,), (2,)]
PolarsCursor [(1, 1)] [(1, 1), (2, 2)]
  • The fallback builds its table with _rows_to_columnar() (pyathena/result_set.py), which keys the columns by name, so both values are appended to one column and every row becomes two rows.
  • On the S3 path, the fetch methods look up values by column name: pandas and Polars rows are dicts keyed by name (row[1][d[0]], row_dict.get(col)), and Arrow uses RecordBatch.to_pydict(), which keeps one column per name.

2. The fallback drops a data row that looks like the header at the start of a later page

_fetch_all_rows() runs _is_first_row_column_labels() on every GetQueryResults page, while _pre_fetch() checks only the first page.
SELECT CASE WHEN n = 1000 THEN 'a' ELSE CAST(n AS varchar) END AS a FROM UNNEST(sequence(1, 1500)) AS t(n) ORDER BY n returns 1500 rows with the S3 result file but 1499 rows (the 'a' row is missing) on the managed path, for all three cursors.

3. AthenaArrowResultSet fetch after close() raises an incidental TypeError

close() assigns self._batches = [], so a later fetch calls next([]) and raises TypeError: 'list' object is not an iterator instead of returning no rows (pyathena/arrow/result_set.py).

Expected

  1. Columns with the same name keep their own values in both modes, as Cursor does.
  2. Only the first GetQueryResults page can carry the header row.
  3. Fetching from a closed Arrow result set behaves like the other result sets (no incidental TypeError).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions