Read multi-line CSV values that cross a block in ArrowCursor

[laughingman7743]

WHAT

ArrowCursor now reads CSV results whose quoted multi-line values cross a block boundary of the pyarrow CSV reader.
AthenaArrowResultSet._read_csv() sets newlines_in_values=True in the csv.ParseOptions of the .csv branch, so pyarrow does not split blocks inside a quoted value.
AsyncArrowCursor and AioArrowCursor share the result set and get the same fix.
The .txt branch (tab-separated, unquoted DDL output) is unchanged.

WHY

Closes #1040.
Without newlines_in_values, pyarrow splits blocks at any newline, and reading fails with CSV parser got out of sync with chunker when a value containing a newline crosses a block boundary.
This happens with CSV results larger than the 128 MiB default block_size, or with a smaller block_size passed to execute().
The parse options date from the original ArrowCursor (e743003, v2.7.0).

A single value larger than block_size still fails with newlines_in_values=True (straddling object straddles two block boundaries), as noted in the issue; that is a pyarrow reader limit, and the default block size is 128 MiB.

Cost of the option, measured locally with pyarrow 25.0.1 on a 250 MiB in-memory CSV (2,000,000 rows, best of 3):

block_size	values	newlines_in_values=False	True
128 MiB	single-line	0.099 s	0.132 s
8 MiB	single-line	0.050 s	0.092 s
128 MiB	every 7th value two-line	0.093 s	0.136 s
8 MiB	every 7th value two-line	ArrowInvalid	0.100 s

The added parse time is 0.03–0.04 s per 250 MiB, small next to downloading that result from S3.

TEST

Tested commit: aef05e4.

• just lint: passed.
• New TestArrowCursor.test_multiline_values_across_blocks reads 50 two-line 301-byte values with block_size=1024 on Athena. It passed, and with the result_set.py change reverted it fails with OperationalError: CSV parser got out of sync with chunker.
• uv run --env-file .env pytest -n 2 tests/pyathena/arrow/test_cursor.py: 56 passed (live Athena, 2026-10-04).
• Offline, with these parse options and block_size=64, 200 empty lines (the single-column NULL rows of Keep NULL rows of single-column CSV results in ArrowCursor #1031) interleaved with 200 two-line values read as 400 rows.
• Not run locally: the async and aio Arrow cursor tests (same result set). AWS CI runs them when the PR is Ready.

🤖 Generated with Claude Code

[laughingman7743]

Self-review round one (implementation behavior): base 28ec68d9f70b5bc2fb447f6a4fb745ee0a467683, head aef05e4a36ec49af94cf685c1e51b638a2b6a597.

Covered: the .csv branch of AthenaArrowResultSet._read_csv() and every Arrow reader of it (ArrowCursor, AsyncArrowCursor, AioArrowCursor); the .txt branch (tab-separated, quote_char=False, unchanged); UNLOAD/Parquet and managed (GetQueryResults) paths, which do not use these options; the interaction with ignore_empty_lines=False from #1031 (empty lines are single-column NULL rows); block_size plumbing from execute(); other CSV readers in the package (only this module uses pyarrow.csv.ParseOptions, while pandas, Polars, and S3FS use their own quote-aware readers); newlines_in_values availability (present well before the pyarrow>=22 floor); the new test.

Result: CLEAN.

• Offline, with these options and block_size=64, 200 empty lines interleaved with 200 two-line values read as 400 rows, so Keep NULL rows of single-column CSV results in ArrowCursor #1031's NULL rows survive the quote-aware chunker.
• The test reads 50 two-line 301-byte values at block_size=1024 (about 15 blocks) and asserts every row. With the fix reverted, it fails with CSV parser got out of sync with chunker.
• Live: tests/pyathena/arrow/test_cursor.py 56 passed.

[laughingman7743]

Self-review round two (claims, compatibility, operations): base 28ec68d9f70b5bc2fb447f6a4fb745ee0a467683, head aef05e4a36ec49af94cf685c1e51b638a2b6a597.

Claims checked:

• 128 MiB default: DEFAULT_BLOCK_SIZE = 1024 * 1024 * 128 (result_set.py:66).
• "A single value larger than block_size still fails": reproduced offline with pyarrow 25.0.1, where 2001-byte values at block_size=1024 raise straddling object straddles two block boundaries with the option on, and the out-of-sync error with it off.
• Parse cost: measured locally on a 250 MiB CSV, adding 0.033–0.043 s (best of 3). The PR text originally said "about 0.04 s" and now says 0.03–0.04 s. This is local CPU time only; the S3 transfer is not measured.
• Origin: the options date from e743003 (v2.7.0), so 3.x has the same defect; the PR targets master.

Existing callers: no signature or default changes; results that did not cross a block parse identically (the existing Arrow suite passes); a result that used to raise OperationalError now returns rows. No extra AWS requests.

Result: CLEAN after the wording correction. Async and aio Arrow tests were not run locally; AWS CI runs them when the PR is Ready.

[laughingman7743]

Independent review (relayed): Codex CLI 0.160.0, model gpt-6-astra, codex exec -s read-only --ephemeral, session 01a1029d-81d3-7f82-ac4e-5526ad97f04a. Base 28ec68d9f70b5bc2fb447f6a4fb745ee0a467683, head aef05e4a36ec49af94cf685c1e51b638a2b6a597, detached snapshot (unchanged afterwards). Prompt had the literal diff and repository conventions only, with no PR text or prior findings. This was a static review: no builds, tests, or network access.

Reviewer coverage: Arrow result-set init, _as_arrow(), _read_csv(), conversion and fetching; sync, threaded-async, asyncio, and SQLAlchemy callers; the new test and fixture; the CSV/TSV/Parquet/API branches; the pandas/Polars/S3FS sibling readers; dependency sources matching uv.lock.

Verdict: FINDINGS, none of them in this diff. No regression was found in the Arrow change, and the new test reaches the parser and crosses block boundaries.

1. P2, pre-existing sibling defect (pyathena/pandas/result_set.py:594, pyathena/polars/result_set.py:498): PandasCursor with engine="pyarrow" and PolarsCursor with use_pyarrow=True let pandas or Polars build pyarrow ParseOptions without newlines_in_values, so multi-line values crossing their 1 MiB blocks raise the same out-of-sync error.

Author verification: confirmed offline. On a 9.6 MB CSV of 8,000 two-line values, pd.read_csv(engine="pyarrow") and pl.read_csv(use_pyarrow=True) raise the error, while engine="c" and the native Polars reader read 8,000 rows. Deferred to #1048: neither wrapper exposes newlines_in_values, so a fix needs a PyAthena-controlled read path, which goes beyond this PR. The default engines are unaffected. The pandas file is also being edited for #1027.

No change to this PR from the independent review.