Return Athena JSON values from SQLAlchemy JSON columns and the pandas, Arrow, and S3FS cursors

[laughingman7743]

WHAT

SQLAlchemy JSON columns, and the cursors that feed them, return the JSON values Athena produces.

SQLAlchemy dialect:

• Map SQLAlchemy JSON to a new AthenaJSON colspec in the dialect.
  Its result processor returns values of the Athena json type unchanged, because PyAthena's default converters have already decoded them, and decodes JSON text of other Athena types (for example a varchar column declared as JSON) with json.loads() or the dialect's json_deserializer, as before.
  The decision uses the coltype SQLAlchemy passes from cursor.description, so a TypeDecorator with impl = JSON and reflected json columns (ischema_names maps json to JSON) take the same path.
• Update the JSON section of docs/sqlalchemy.md with json_parse() examples and the CAST('...' AS JSON) string-scalar result.

Cursors (found while reviewing the above):

• The pandas and Arrow converters, which read CSV result files, map json to a new _csv_to_json that returns None for an empty field (how CSV results write SQL NULL), so PandasCursor and ArrowCursor no longer raise JSONDecodeError on a NULL json value. _to_json is unchanged, so an empty JSON string inside array(json) / map(..., json) stays "".
• With managed query result storage, ArrowCursor and PolarsCursor build their table/DataFrame with json values kept as text, as the CSV result file does, and decode only the json columns in the fetch methods. pyathena.converter._json_text_converter() builds the converter, and AthenaResultSet._json_converters() picks the json columns. Before, pyarrow and Polars inferred struct or list columns from the decoded values, and execute() failed when rows held different JSON shapes (ArrowInvalid: cannot mix list and non-list, non-null values, Polars TypeError). For Arrow this builds on Do not convert ArrowCursor fallback values twice #1023, which stopped converting the fallback values twice; the fetch methods still resolve the converters when they fetch.
• S3FSCursor converts GetQueryResults rows (managed query result storage) with its own converter, as it does for the CSV result file, instead of DefaultTypeConverter, so a custom converter applies on both paths.

Release notes:

• Behavior change: type_coerce(literal_column("CAST('<text>' AS JSON)"), JSON) now returns the str that Athena returns for the cast, instead of a second decode of that string into a dict or list. Use json_parse('<text>') to get the parsed value.
• Behavior change: when the cursor returns the raw text of a json result (for example through a custom converter whose json mapping does not decode it), JSON columns now return that text, and the dialect's json_deserializer no longer applies to json results.
• Fix: SQLAlchemy JSON columns no longer raise TypeError for JSON objects and arrays (json_parse(...), CAST(ROW/MAP ... AS JSON)).
• Fix: PandasCursor and ArrowCursor return None for SQL NULL json values instead of raising.
• Fix: with managed query result storage, ArrowCursor and PolarsCursor no longer fail on json columns whose rows hold different JSON shapes.
• Behavior change: with managed query result storage, json columns in as_arrow() / as_polars() are string columns, as with S3 results, instead of inferred struct or list columns.
• Behavior change: with managed query result storage, S3FSCursor uses the configured converter instead of DefaultTypeConverter.

WHY

Closes #934.

The converters decode the Athena json type, and SQLAlchemy's JSON.result_processor decoded the value again, so every real JSON object or array read through JSON raised TypeError: the JSON object must be str, bytes or bytearray, not dict.
CAST('<text>' AS JSON) only appeared to work because Athena returns it as a JSON string scalar that the second decode parsed.

Measured on Athena through SQLAlchemy with all five drivers (rest, pandas, arrow, polars, s3fs): cursor.description reports json for JSON expressions and varchar for text.
UNLOAD (unload=true) rejects the json type (NOT_SUPPORTED: Unsupported Hive type: json).

The cursor defects were found by the measurement and the independent review of this PR; the maintainer asked to fold them in.
Measured before the fix:

• SQL NULL json (S3 result file): PandasCursor raised OperationalError, ArrowCursor raised JSONDecodeError; the other cursors returned None.
• One json column with rows {"a": 1}, {"b": 2}, [1, "x"], "s": with managed storage, Arrow and Polars execute() failed (ArrowInvalid / TypeError); with S3 results, and with the other cursors, the values came back.
• Custom S3FS converter with managed storage: ignored (DefaultTypeConverter values).

TEST

Tested commit: df476e4 (rebased on master fe21250, which includes #1023, #1012, and #1029; df476e4 only renames the S3FS test helper)

• just lint: passed. just docs lint: passed.
• Offline: tests/pyathena/test_converter.py, tests/pyathena/arrow/test_converter.py, and tests/pyathena/sqlalchemy/test_types.py: 148 passed. test_csv_to_json covers the empty field, and test_nested_json_empty_string checks that array(json) / map(varchar,json) keep an empty JSON string; with the dialect change reverted, 13 of the 24 AthenaJSON tests fail.
• Live: test_fetch_all_rows (default S3 and managed work group) for Arrow, pandas, and Polars now selects two rows with TIME, VARBINARY, JSON values of different shapes, JSON string scalars, and NULL json; Arrow and Polars also check that the json column of as_arrow() / as_polars() is a string column. The new TestS3FSCursor::test_fetch_all_rows_custom_converter (default and managed) checks that a custom json converter applies on both paths. With the cursor changes of this PR reverted on master fe21250, Arrow default and managed, pandas default, Polars managed, and S3FS custom-converter managed fail; with them, all 10 pass.
• Live: uv run --env-file .env pytest -n 4 tests/pyathena/arrow tests/pyathena/aio/arrow tests/pyathena/polars tests/pyathena/aio/polars tests/pyathena/s3fs tests/pyathena/aio/s3fs: 315 passed, 1 skipped.
• Live: pytest -n 4 tests/pyathena/pandas tests/pyathena/aio/pandas -k "json or fetch or complex or managed or null": 74 passed.
• Live: pytest -n 1 tests/pyathena/sqlalchemy/test_base.py -k test_json_type: passed (one query instead of four); with the pyathena/sqlalchemy changes reverted, it fails with the TypeError from SQLAlchemy JSON columns fail on JSON objects and arrays returned by Athena #934.
• Related pre-existing defects found during review and filed separately: ArrowCursor drops NULL rows of single-column CSV results #1026 (ArrowCursor drops NULL rows of single-column CSV results), PandasCursor infers float64 for integer and JSON columns with NULL on managed query result storage #1027 (PandasCursor float64 inference on managed storage), Custom converters do not apply to pandas, Arrow, and Polars results with managed query result storage #1028 (custom converters with managed storage). The chunked Polars UNLOAD metadata finding was PolarsCursor with unload=True and chunksize returns the UNLOAD statement's metadata and None rows #1007, fixed by Describe chunked Polars UNLOAD results with the Parquet schema #1012.
• Not run locally: the full just test pyathena, just test sqla, and just test sqla-async; they run in CI once the PR is Ready. The SQLAlchemy compliance suites do not enable json_type.

🤖 Generated with Claude Code

[laughingman7743]

Self-review round 1 (implementation behavior) — FINDINGS (1, repaired)

Scope: git diff 4e7b55ff700edfc92d51baa219e92add9783fcb4..fc1ddb4e4cc2f0ae82a47b6c0d16ddb171744de6 (all 5 files).

Covered:

• Behavior: the coltype SQLAlchemy passes comes from cursor.description. Measured with all five drivers, it is json for JSON expressions and varchar for text, so json results pass through and text is decoded as before. None stays None on both paths.
• Callers: AthenaAioDialect inherits colspecs (AthenaAioDialect.colspecs[types.JSON] is AthenaJSON). A TypeDecorator with impl = JSON resolves its impl through colspecs and gets the same processor (checked through dialect_impl(...).result_processor). STRUCT/MAP processors do not touch JSON, and the ARRAY path decodes JSON elements in _ArrayValueProcessor._decode without a result processor, so both are unchanged.
• Bind side: bind_processor is inherited unchanged, so a bound dict is sent as JSON text (varchar) and decoded on fetch.
• Framework contract: like every colspec, adapt_type now replaces a user subclass of types.JSON with AthenaJSON, which is the standard SQLAlchemy behavior (for example psycopg2 maps JSON to its own type); custom result handling belongs in a TypeDecorator, which keeps working.
• Tests: the offline tests fail 13/24 and the live test_json_type fails with the SQLAlchemy JSON columns fail on JSON objects and arrays returned by Athena #934 TypeError when the pyathena/ changes are reverted.

Finding: tests/pyathena/sqlalchemy/test_types.py:23 called SQLAlchemy's private _cached_result_processor; the public dialect_impl(...).result_processor(...) exercises the same path for both JSON and TypeDecorator.
Repair: 13d2c0e switches to the public method; the tests still pass with the fix and fail 13/24 without it.

Out of scope (pre-existing, not introduced here): PandasCursor and ArrowCursor fail on SQL NULL of the json type (CAST(NULL AS JSON) → JSONDecodeError from _to_json(''); one Arrow run returned no row). The live test therefore runs the SQL NULL case on the default rest engine only.

[laughingman7743]

Self-review round 2 (claims, compatibility, operations) — FINDINGS (PR text only, corrected)

Scope: git diff 4e7b55ff700edfc92d51baa219e92add9783fcb4..13d2c0e0e25eda6c433736c37e85bf1f3a79d287; claims in the PR body, commit messages, the AthenaJSON docstrings, and docs/sqlalchemy.md.

Claims checked:

• "converters decode the Athena json type": the default, pandas, Arrow, and Polars mappings use _to_json, and S3FS copies _DEFAULT_CONVERTERS. Measured through SQLAlchemy with all five drivers: description = json/varchar, values dict/list/str/None. pandas and Arrow were re-measured without the SQL NULL column because of the pre-existing NULL defect recorded in round 1. UNLOAD rejects json (NOT_SUPPORTED), so it never reaches this path.
• Docs: json_parse() vs CAST('...' AS JSON) and varchar decoding: the examples were run live on rest. The first example's output comment uses Athena's key order (items before name).
• Other docs: the CAST(ROW/MAP/ARRAY ... AS JSON) examples (docs/sqlalchemy.md:878, :1026, :1300; docs/usage.md:708) use text() or cursors without the JSON type and cast structured values, so they are unaffected.
• Existing callers: SQLAlchemy >= 2.0 keeps the result_processor(dialect, coltype) signature. A bound value still serializes through the inherited bind_processor. The CAST(text AS JSON) → str change is release-noted, and the PR now has the 4.0.0 milestone and the bug label.

Corrections to the PR body:

• "reflected json view columns" and "JSON view columns" were not measured. Narrowed to the reflected json → JSON mapping and the measured expressions.
• Added the compatibility consequence for a custom converter that returns raw text for json (it now gets that text back).
• Tested commit updated to 13d2c0e; the live runs used fc1ddb4 (it differs only in the offline helper).

Operations: the live JSON test now issues 1 Athena query instead of 4. No retry or quota behavior changed.

Limits: async dialect not tested live (it inherits colspecs); the full pyathena/sqla/sqla-async suites are left to CI after Ready. just docs build passed locally.

[laughingman7743]

Independent review (relayed): Codex CLI 0.160.0, model gpt-6-astra, reasoning effort max (from the user config), codex exec -s read-only --ephemeral, session 01a10177-84e0-7ec0-9890-dc85396aa8fc.
The reviewer worked from a detached snapshot at 13d2c0e0e25eda6c433736c37e85bf1f3a79d287 (merge-base 4e7b55ff700edfc92d51baa219e92add9783fcb4) and received only the literal diff and a neutral prompt; it did not see the PR number, description, commits, or prior findings. Static review only: no edits, builds, tests, or network access. Afterwards the snapshot and the PR worktree were confirmed unchanged.

Covered (reviewer): all five sync and five asyncio dialects; shared cursor conversion, CSV and managed-result paths; SQLAlchemy 2.0.46 colspecs, TypeDecorator, with_variant(), nested ARRAY/MAP/STRUCT processing and reflection; changed tests, docs, and docstrings.

Verdict: FINDINGS

1. P2, introduced: raw custom converters bypass JSON deserialization (pyathena/sqlalchemy/types.py:73). With DefaultTypeConverter().remove("json") passed through connect_args, json_parse('{"a":1}') read through JSON used to return {"a": 1}; it now returns the raw string, and a configured json_deserializer is skipped. The reviewer also said the blanket converter claim needs qualification.
2. P2, pre-existing: with managed result storage and no output_location, Arrow and Polars decode JSON a second time (pyathena/arrow/result_set.py:256, pyathena/polars/result_set.py:138). _fetch_all_rows() decodes json with DefaultTypeConverter, and the fetch then applies _to_json to the dict, raising TypeError before SQLAlchemy receives the value.
3. P2, pre-existing: pandas CSV results raise on SQL NULL json (pyathena/pandas/converter.py:27, pyathena/pandas/result_set.py:605), because _to_json("") reaches json.loads("").

The reviewer also judged the tests to assert observable values and to fail for the original defect, found no other defect in adaptation, reflection, or the typed ARRAY path, and confirmed the json_parse() vs CAST and DDL statements as accurate.

Author verification and disposition

• (1) Confirmed. The behavior is intentional and already listed as a release-note item in the PR body. For an Athena json value that is a str, the value alone cannot tell a decoded JSON string scalar from undecoded text, so returning string scalars as str (the expected behavior in SQLAlchemy JSON columns fail on JSON objects and arrays returned by Athena #934) requires relying on the converter. Repair 9906bd3: the docstring and docs/sqlalchemy.md now say "default converters" and state that a custom converter that does not decode json gets its text back. The implementation is unchanged.
• (2) Confirmed statically: ArrowResultSet._fetch and AthenaPolarsResultSet.iterrows apply the cursor's json converter to values built by _fetch_all_rows() with DefaultTypeConverter (pyathena/result_set.py:682-731). This is a cursor-level defect independent of this diff, and the SQLAlchemy layer never receives the value. Not measured (no managed-results work group used here). Deferred to a separate issue.
• (3) Confirmed and measured live (also recorded in self-review round 1): CAST(NULL AS JSON) raises OperationalError from JSONDecodeError with PandasCursor (and ArrowCursor CSV). Pre-existing; deferred to a separate issue.

[laughingman7743]

Self-review of repair 9906bd3 (both perspectives) — CLEAN

Scope: git range-diff 4e7b55ff700edfc92d51baa219e92add9783fcb4..13d2c0e0e25eda6c433736c37e85bf1f3a79d287 4e7b55ff700edfc92d51baa219e92add9783fcb4..9906bd32b01fdde2bbc360d735814ea32df7b464, which adds one commit and changes only the AthenaJSON docstring and docs/sqlalchemy.md prose.

• Behavior: no code change. just lint, just docs lint, and the offline test_types.py (24 passed) were rerun on 9906bd3.
• Claims: "PyAthena's default converters decode json" holds because the default, pandas, Arrow, and Polars mappings use _to_json and S3FS copies _DEFAULT_CONVERTERS. The pre-existing NULL and managed-result failures in (2) and (3) are raised in the cursors before decoding completes, so they do not contradict it. "A custom converter that does not decode json gets its text back" follows from coltype == "json" returning no processor. The PR body now also states that json_deserializer no longer applies to json results, and the tested commit was updated.

[laughingman7743]

Independent follow-up (relayed): Codex CLI 0.160.0, gpt-6-astra, codex exec -s read-only --ephemeral, session 01a101c4-5925-7ce0-a409-4d1b66be085b. The snapshot was at 9906bd32b01fdde2bbc360d735814ea32df7b464, and the reviewer received the range-diff from 13d2c0e plus the incremental diff. Static review only. Covered: all ten SQLAlchemy drivers (sync + aio), converter selection, row fetching, managed-result fallback, UNLOAD variants, AthenaJSON processing.

Verdict: FINDINGS

• P2, docs/sqlalchemy.md:1370 and pyathena/sqlalchemy/types.py:56: "custom converter → text" overstates the guarantee. With awsathena+s3fs and managed result storage, pyathena/s3fs/result_set.py:124 calls _fetch_all_rows() without the cursor's converter, so DefaultTypeConverter (pyathena/result_set.py:713) decodes json to a dict even when a custom converter is configured. The reviewer suggested describing the actual boundary instead: when the cursor returns raw JSON text, JSON passes it through.

Author verification and repair f2cda29: confirmed at pyathena/s3fs/result_set.py:124. The docstring and docs now say: if the cursor returns the text of a json result, for example through a custom converter that does not decode it, JSON returns that text unchanged. The PR body uses the same boundary. No code change; just lint and just docs lint pass.
Self-review of the repair (both perspectives): the new wording claims only what coltype == "json" → no processor guarantees (the value is passed through), so it holds on every fetch path, including the managed-result fallbacks. The S3FS fallback ignoring a custom converter is pre-existing cursor behavior outside this diff; it is added to the deferred cursor-level issues with (2) and (3).

[laughingman7743]

Independent follow-up 2 (relayed): Codex CLI 0.160.0, gpt-6-astra, codex exec -s read-only --ephemeral, session 01a101c9-e825-7820-bca0-f943a5910c3f. The snapshot was at f2cda2981ee7d5b2becfae2199eea945ee367b3c, and the reviewer received the incremental diff from 9906bd3. Static review only. Covered: all ten SQLAlchemy drivers, converter selection, cursor metadata, CSV/UNLOAD and managed-result paths, AthenaJSON, and "Querying JSON data".

Verdict: FINDINGS. The reviewer considers the previous custom-converter finding resolved.

• P2, docs/sqlalchemy.md:1369 and pyathena/sqlalchemy/types.py:55: with managed result storage, Arrow and Polars (sync and aio) return CAST('[1, 2, 3]' AS JSON) as a list, and raise TypeError for objects and arrays, because _fetch_all_rows() decodes through DefaultTypeConverter (pyathena/result_set.py:713) and the fetch applies _to_json again (pyathena/arrow/result_set.py:256, pyathena/polars/result_set.py:138). The reviewer suggests qualifying the result-shape claims for these fallback paths.

Author disposition: not applied to this PR (pre-existing, deferred). This is pre-existing defect (2) from the first review: the cursor decodes json twice before SQLAlchemy sees the value, and AthenaJSON preserves whatever the cursor returns, as the reviewer notes. The same cursors return the wrong shape for these values without SQLAlchemy too. Following the project's practice, the docs describe the intended behavior rather than a known cursor defect, and the fix belongs in the Arrow/Polars managed-result fetch path. It is deferred to a cursor-level issue together with the pandas/Arrow SQL NULL json failure and the S3FS managed fallback ignoring a custom converter.

[laughingman7743]

Self-review round 1, expanded scope (implementation behavior): CLEAN for this diff; one pre-existing defect recorded below.

The contract expanded to the cursors (5792a2e, c0ebcaa) after the branch was rebased on master 16aef64d10dce92cbf8b8d957c178bef9bd494b6, so this is a full pass over git diff 16aef64d10dce92cbf8b8d957c178bef9bd494b6..c0ebcaaebc83eeaa7b635b753ef063e6fd8e441f (15 files).

Covered:

• _to_json: on the REST API path a NULL is None and a JSON "" scalar arrives as the text "", so an empty string only reaches it as a CSV NULL. pandas converters and the Arrow CSV string column hand it ""; Polars and S3FS already pass None.
• Arrow: _table_converters is read after _as_arrow(), as self.converters was before at fetch time, so UNLOAD metadata updates are kept. On the CSV/UNLOAD path the converters are the same as before. On the managed path only the json columns are converted.
• Arrow/Polars managed path: _json_text_converter() keeps json as text, and _json_converters() selects the cursor's own json converter (_to_json) for those columns. Polars iter_chunks() and the chunked managed case use _df_converters, and the chunked S3 path keeps self.converters (pyathena/polars/result_set.py:367). The other managed values are unchanged (already converted by DefaultTypeConverter). An all-NULL json column becomes a null-typed column and fetches None.
• S3FS: DefaultS3FSTypeConverter.convert returns None for None and delegates type hints to a DefaultTypeConverter, so API strings go through the same conversions as CSV values.
• Callers: the aio/async Arrow, Polars, and S3FS cursors share these result sets (tests/pyathena/aio/{arrow,polars,s3fs} passed). Pandas managed still decodes json eagerly into object columns, which matches its S3 path.
• Tests: each change has a case that fails without it: Arrow default/managed, pandas default, and S3FS custom-converter managed without 5792a2e; Arrow/Polars managed without c0ebcaa.

Pre-existing, outside JSON (not changed here): ArrowCursor drops rows of a single-column CSV result whose value is NULL (SELECT x FROM (VALUES 1, NULL, 2) → [(1,), (2,)]; pandas, Polars, and S3FS return 3 rows; any type, e.g. CAST(NULL AS VARCHAR) → 0 rows). An empty CSV line is skipped by the pyarrow CSV reader. Left for a separate issue.

[laughingman7743]

Self-review round 2, expanded scope (claims, compatibility, operations): CLEAN

Scope: git diff 16aef64d10dce92cbf8b8d957c178bef9bd494b6..c0ebcaaebc83eeaa7b635b753ef063e6fd8e441f, plus the updated PR title/body and the commit messages of 5792a2e and c0ebcaa.

Claims checked:

• "An empty string, which CSV results use for SQL NULL" (here): docs/s3fs.md:216-218 and the measurement agree (unquoted empty field = NULL). JSON text is never empty; a JSON empty-string scalar is "".
• "PolarsCursor got the same fix in Keep the whole pandas and Polars results reusable #1005": 5a60dd1 sets _df_converters = {} for the GetQueryResults path.
• PR table and the heterogeneous-shape failure: measured on master 16aef64 before the fix (ArrowInvalid: cannot mix list and non-list, non-null values; Polars TypeError: unexpected value while building Series of type Int64); after the fix, Arrow and Polars return the same rows on both paths as Cursor.
• "Defaults to DefaultTypeConverter with json values kept as text, as in the CSV result file" (_as_arrow_from_api, _as_polars_from_api): the CSV readers type json as pa.string() / pl.String (docs/arrow.md:159, docs/polars.md:178), and the tests now assert a string column on both paths.
• _fetch_all_rows docstring (subclass converters may not handle API strings) stays true; only S3FS, whose converter reads text like DefaultTypeConverter, now passes its own.

Compatibility (all release-noted in the PR body, 4.0.0 milestone):

• With managed storage, the json columns of as_arrow() / as_polars() / Polars iter_chunks() change from inferred struct/list columns to string columns; code that used struct operations on them must decode instead. The fetch methods return the same values as before for shapes that worked.
• With managed storage, an S3FS custom converter now applies; the default converter gives the same values as DefaultTypeConverter there.
• _to_json("") returns None instead of raising; no previously working input changes.

Docs: docs/s3fs.md, docs/arrow.md, docs/polars.md, docs/pandas.md, docs/usage.md (managed storage), and docs/aio.md:223 contain no statement that this change makes false.

Operations: the cursor tests add columns and a UNION ALL row to existing queries. The new S3FS test adds 2 queries, and test_json_type saves 3.

Limits: the full pyathena / sqla / sqla-async suites run in CI after Ready; locally: the Arrow/Polars/S3FS sync+aio suites (432 passed) and the pandas subset (106 passed, before c0ebcaa, which does not touch pandas).

[laughingman7743]

Independent review of the expanded scope (relayed): Codex CLI 0.160.0, gpt-6-astra, codex exec -s read-only --ephemeral, session 01a101f7-f1dd-7353-ab60-dba323d2576c. The snapshot was at c0ebcaaebc83eeaa7b635b753ef063e6fd8e441f (merge-base 16aef64d10dce92cbf8b8d957c178bef9bd494b6), and the reviewer got the full diff with a neutral prompt (no PR text, commits, or prior findings). Static review only. Afterwards the snapshot and the PR worktree were confirmed unchanged.

Covered (reviewer): SQLAlchemy JSON adaptation, TypeDecorator/Variant, nested ARRAY/MAP/STRUCT, reflection, sync/aio dialects, shared converters, and pandas/Arrow/Polars/S3FS CSV, UNLOAD, managed-storage, chunk, and DataFrame paths including async callers; tests and changed docs.

Verdict: FINDINGS

1. P2, introduced: pyathena/converter.py:131 (_to_json returning None for "") turns nested empty JSON strings into NULL: DefaultTypeConverter().convert("array", '[""]', type_hint="array(json)") → [None] instead of [""].
2. P2, introduced: pyathena/arrow/result_set.py:155: with managed storage, a custom Arrow converter for a non-json type (e.g. varchar → str.upper) no longer applies ("abc" instead of "ABC").
3. P2, pre-existing: pyathena/arrow/result_set.py:325: single-column SQL NULL rows are dropped from CSV results (ignore_empty_lines).
4. P2, pre-existing: pyathena/pandas/result_set.py:849: with managed storage, DataFrame inference turns a JSON column of numbers + NULL into float64 (9007199254740993 → 9007199254740992.0).
5. P2, pre-existing: pyathena/pandas/result_set.py:844: with managed storage, a configured pandas json converter is ignored.
6. P2, pre-existing: pyathena/polars/result_set.py:367: chunked Polars UNLOAD uses the initial metadata names, so row values are missing.
   The reviewer judged the added tests to assert observable behavior and to expose the double decoding and the mixed-shape failures.

Author verification and disposition

• (1) Confirmed offline ([None] / {'k': None} at c0ebcaa vs [''] / {'k': ''} at 16aef64). Repaired in c5d78c7: _to_json restored; the pandas and Arrow converters (CSV readers) map json to the new _csv_to_json, which treats only the empty field as NULL. Polars and S3FS already pass None for NULL and keep _to_json. New offline tests test_csv_to_json and test_nested_json_empty_string.
• (2) Confirmed live (managed: [(1, 'abc')], S3: [(1, 'ABC')]). Maintainer decision: keep it, matching Polars since Keep the whole pandas and Polars results reusable #1005 and pandas, which never ran cursor converters on managed values; release-noted in the PR body and tracked in Custom converters do not apply to pandas, Arrow, and Polars results with managed query result storage #1028.
• (3) Confirmed live, pre-existing, any column type → ArrowCursor drops NULL rows of single-column CSV results #1026.
• (4) Confirmed live: managed BIGINT with NULL is float64 (S3: Int64); a JSON number + NULL column is float64 on both paths → PandasCursor infers float64 for integer and JSON columns with NULL on managed query result storage #1027.
• (5) Confirmed live for pandas, Arrow, and Polars (custom varchar converter) → Custom converters do not apply to pandas, Arrow, and Polars results with managed query result storage #1028.
• (6) Pre-existing, already filed as PolarsCursor with unload=True and chunksize returns the UNLOAD statement's metadata and None rows #1007.

Self-review of repair c5d78c7 (both perspectives): CLEAN. _to_json is byte-identical to master again, so the REST cursor, the typed parser, and Polars are unchanged from master. The pandas pyarrow-engine check uses self.converters, which tests mapping membership, not function identity (pyathena/pandas/result_set.py:405, :509), so the engine choice is unchanged. On the Arrow managed path, _csv_to_json receives None for NULL and the text "" for a JSON empty-string scalar, so values are unchanged. No docs reference _to_json (only build artifacts). test_fetch_all_rows for all four cursors: 10 passed on c5d78c7; just lint passed.

[laughingman7743]

Independent follow-up on c5d78c7 (relayed): Codex CLI 0.160.0, gpt-6-astra, codex exec -s read-only --ephemeral, session 01a10209-9aa8-7873-8919-6782c2b4725f. The snapshot was at c5d78c7556c295640924cd4f8ad3bc3bb08d35f1, and the reviewer got the incremental diff from c0ebcaa. Static review only; the snapshot and the PR worktree were confirmed unchanged.

Covered: nested array(json) / map(varchar,json) conversion; the pandas and Arrow CSV paths across sync, async, and aio cursors, including pandas chunks and Arrow batches; the Arrow managed results; the added regression tests and directly affected callers.

Verdict: CLEAN. The nested empty-string regression is fixed, empty CSV fields stay SQL NULL, a JSON "" stays an empty string, and Arrow managed JSON text is decoded once with None kept as None. No new actionable defects.

[laughingman7743]

Self-review of refactor 4eb5b0b (both perspectives): CLEAN

Maintainer request: the result sets should only choose converters. Scope: git diff c5d78c7556c295640924cd4f8ad3bc3bb08d35f1..4eb5b0b202d1df3000042a9c4a648ac7b8f47c5c (4 files).

• The json-text converter (a DefaultTypeConverter with json → _to_default) moves from the AthenaResultSet._json_text_converter() staticmethod to pyathena.converter._json_text_converter(). Arrow and Polars call it from _as_arrow_from_api() / _as_polars_from_api(), and each call still builds a fresh converter. AthenaResultSet._json_converters() stays in the result set because it picks columns from description.
• Arrow's new _identity is removed; _table_converters.get(k, _to_default) uses _to_default, which returns its argument unchanged (pyathena/converter.py:477), so behavior is the same. The pre-existing Polars _identity is untouched.
• There is no import cycle (pyathena.converter does not import result sets). Both helpers were added in this unreleased PR, so no caller depends on the old location.
• Validation: just lint passed; offline test_converter.py, arrow/test_converter.py, and sqlalchemy/test_types.py (148 passed); live tests/pyathena/arrow tests/pyathena/aio/arrow tests/pyathena/polars tests/pyathena/aio/polars (212 passed). The PR body now names the new location.

[laughingman7743]

Independent follow-up on 4eb5b0b (relayed): Codex CLI 0.160.0, gpt-6-astra, codex exec -s read-only --ephemeral, session 01a1022a-9e31-7610-a5a5-40bc2467440b. The snapshot was at 4eb5b0b202d1df3000042a9c4a648ac7b8f47c5c, and the reviewer got the incremental diff from c5d78c7. Static review only; the snapshot and the PR worktree were confirmed unchanged.

Covered: sync, async/Future, and aio Arrow and Polars cursors; CSV, UNLOAD/Parquet, the managed-result fallback, and Polars chunked iteration; JSON/NULL handling, explicit converter precedence, converter isolation, and Arrow identity conversion; helper references, import dependencies, and callable/return typing.

Verdict: CLEAN. The relocated helper keeps its implementation, both callers are updated, _to_default returns values unchanged and fits the converter mapping, and there is no new import cycle or typing problem.

[laughingman7743]

Self-review after the rebase onto master fe21250 (both perspectives): CLEAN

The branch conflicted with #1023 (#1021), which fixed the ArrowCursor fallback double conversion independently. I rebased onto fe21250a3e228e3d38e61898334c5baacc1285c2. Scope: git range-diff 16aef64d10dce92cbf8b8d957c178bef9bd494b6..4eb5b0b202d1df3000042a9c4a648ac7b8f47c5c fe21250a3e228e3d38e61898334c5baacc1285c2..d5eac4c78ef6fb7c2f009137d9dd537087598487, plus the upstream changes to the touched files (#1023 Arrow, #1012 and #1029 Polars).

Resolution:

• The Arrow part of the old 5792a2e (convert only S3 tables) duplicated Do not convert ArrowCursor fallback values twice #1023, so master's version is kept and the commit (now a96a81e) is renamed to the pandas and S3FS fixes.
• The json-as-text commit (now 90537de) is rebuilt on Do not convert ArrowCursor fallback values twice #1023's structure. _convert_rows stays. _fetch() resolves self.converters when it fetches, as Do not convert ArrowCursor fallback values twice #1023 requires (a converter.set() after execute() still applies), and narrows it with _json_converters() on the fallback path. With no json columns there are no converters, and the fallback keeps Do not convert ArrowCursor fallback values twice #1023's plain zip path. The old _table_converters captured at __init__ and the Arrow _identity are gone, and Do not convert ArrowCursor fallback values twice #1023's comment now mentions the json exception.
• test_fetch_all_rows for Arrow keeps Do not convert ArrowCursor fallback values twice #1023's query layout and adds the NULL json column, the second row with other JSON shapes, and the string-column check.
• 4eb5b0b (now d5eac4c) only moves _json_text_converter(); its message no longer mentions the identity function.

Upstream checks: #1012 changes the chunked Polars UNLOAD metadata and #1029 changes PolarsDataFrameIterator.close(); neither touches the GetQueryResults path or _df_converters. Codex's earlier chunked-UNLOAD finding (#1007) is fixed by #1012.

Validation on d5eac4c: just lint passed; offline 148 passed; live Arrow/Polars/S3FS sync+aio suites 315 passed, 1 skipped; pandas subset 74 passed; test_json_type passed. With this PR's cursor changes reverted on fe21250, five test_fetch_all_rows cases fail (Arrow default and managed, pandas default, Polars managed, S3FS custom-converter managed); with them, all 10 pass. The PR body and #1028 now credit #1023 for the Arrow fallback change.

[laughingman7743]

Independent follow-up after the rebase (relayed): Codex CLI 0.160.0, gpt-6-astra, codex exec -s read-only --ephemeral, session 01a10239-6234-7140-af28-4bdf226e9084. The snapshot was at d5eac4c78ef6fb7c2f009137d9dd537087598487, and the reviewer got the range-diff from 4eb5b0b and the final Arrow/Polars/result-set/converter diff against fe21250a3e228e3d38e61898334c5baacc1285c2. Static review only; the snapshot and the PR worktree were confirmed unchanged.

Covered: sync, async, and aio Arrow/Polars cursors across CSV, UNLOAD, and managed fallback paths, including Polars chunking; Arrow's fetch-time converter lookup (a converter.set() after execute()); managed JSON kept as text and decoded once while other fallback values pass through; heterogeneous JSON, string scalars, and NULL json; the rewritten Arrow test.

Verdict: CLEAN. The conflict resolution keeps both the #1023 intent and this change's intent.

[laughingman7743]

Self-review of df476e4 (both perspectives): CLEAN

Maintainer question: the S3FS test helper had the same name as the production pyathena.converter._json_text_converter(), but builds a different thing. Scope: git diff d5eac4c78ef6fb7c2f009137d9dd537087598487..df476e484c3c0c8208288c54d27761cecf6f2662 (test file only).

• tests/pyathena/s3fs/test_cursor.py: _json_text_converter() → _s3fs_converter_with_json_text(), which builds a DefaultS3FSTypeConverter (the S3FS cursor's converter type) with json → _to_default. The production helper returns a DefaultTypeConverter for the GetQueryResults fallback; the test does not reuse it, so it does not depend on a private helper meant for another path.
• DefaultS3FSTypeConverter.__init__ deep-copies _DEFAULT_CONVERTERS, so set() changes only that instance; there is no shared-dict mutation.
• No source change. just lint passed; pytest -n 2 tests/pyathena/s3fs/test_cursor.py -k test_fetch_all_rows: 4 passed (default/managed, default/custom converter). PR body tested commit updated.

[laughingman7743]

Independent follow-up on df476e4 (relayed): Codex CLI 0.160.0, gpt-6-astra, codex exec -s read-only --ephemeral, session 01a10269-23f8-78a2-8741-ad8297b5dee3. The snapshot was at df476e484c3c0c8208288c54d27761cecf6f2662, and the reviewer got the incremental diff from d5eac4c. Static review only; the snapshot and the PR worktree were confirmed unchanged.

Covered: the incremental diff, helper references, fixture forwarding, and the S3 CSV and managed API conversion paths.

Verdict: CLEAN. The rename is complete (definition and both call sites). Both cases still pass a custom DefaultS3FSTypeConverter that keeps JSON text, and the unchanged assertion checks it. The managed case keeps its skip when AWS_ATHENA_MANAGED_WORKGROUP is unset.