Correct docstrings that contradict the implementation - #939
Conversation
| rowcount: Number of rows affected by the last query (-1 for SELECT queries). | ||
| arraysize: Default number of rows to fetch with fetchmany(). | ||
| max_workers: Maximum number of worker threads for concurrent execution. | ||
| arraysize: Default number of rows that fetchmany() returns on the result |
There was a problem hiding this comment.
Self-review round one (implementation behavior): CLEAN
Base 9b74767cf65e0ef513338fb2bb2da678cdfbcd2a, head 763004580effd74f444ca6a0045f25eb987f9fa7 (full diff, 31 files, docstrings only).
I traced each changed claim into the code it describes:
- AsyncCursor:
execute()returns(query_id, future)(async_cursor.py:219).arraysizereaches the result sets through_collect_result_set.description(query_id)returns a Future. - Fetch wording:
fetchmany()falls back toarraysizefor non-positive sizes in the base result set (result_set.py:516), in the aio result set (aio/result_set.py:202), and throughWithFetch/WithAsyncFetchdelegation.- Dict rows come from
AthenaDictResultSet/AthenaAioDictResultSet.
- Pandas
retry_config: it is used for GetQueryResults and for_get_content_length/_read_data_manifest. CSV and Parquet reads go throughS3FileSystem(connection=...), which takesconnection.retry_config(filesystem/s3.py:182). - Polars converter: dtypes feed
schema_overridesin bothread_csvandscan_csv(polars/result_set.py:440,625). The conversion mapping covers date/time/varbinary/json only. S3StorageClassBUCKET/DIRECTORYare assigned infilesystem/s3.py:284,319,378,581.ExecuteOptionsis a frozen dataclass, sodataclasses.replacecan clear a field.S3AioExecutorusesasyncio.to_thread. The sync filesystem creates one executor per opened file (_open→_create_executor).- Type compiler: FLOAT/REAL render
FLOAT(compiler.py:104-110), CAST usesREAL(:852-856), and string types render CHAR(n)/VARCHAR(n) or STRING (:180-198). - Attributes sweep: a script compared each class's
Attributes:with its public attributes. Only theTypeNodedataclass fields remain flagged, and they exist.
No code or test changes. Checks run: just lint and sphinx-build (no new warnings).
| SELECT * FROM sales WHERE year = 2023 | ||
| ) | ||
| TO 's3://my-bucket/results/unload/20231215/uuid//' | ||
| TO 's3://my-bucket/results/unload/20231215/<uuid>/' |
There was a problem hiding this comment.
Self-review round two (claims, callers, documentation reader): CLEAN
Base 9b74767cf65e0ef513338fb2bb2da678cdfbcd2a, head 763004580effd74f444ca6a0045f25eb987f9fa7.
Claims checked:
- PR body:
- "19 places": the replacement script changed 19
result_set_type_hintsentries;options.pywas reworded separately. - The Sphinx warning delta was rerun on 7630045. The only change is that one duplicate
AsyncCursor.descriptionwarning is gone. - The
Attributes:sweep result was rerun on the head. - The tested commit in the PR body was updated to 7630045.
- "19 places": the replacement script changed 19
- Examples:
wrap_unloadwas run offline. The real output ends in<uuid>/, and an empty query raisesProgrammingError.DefaultParameterFormatterformatsdate/datetimeand raisesTypeErrorfordatetime.time.AioConnection.createisasync def, soasync with awaitmatchesaio/connection.py:26and the README.
- Caller impact: none. Only docstrings changed, and
just lint(ruff D, mypy) passes on the head. - Related prose outside the diff:
_last_query_execution,uuid//, andAioConnection.create(...)withoutawaitno longer appear anywhere inpyathena/.fetchall() # Returnsnow appears only for S3FS, where it is correct (tuples).aio/common.pyWithAsyncFetch.fetchmanywas missed in the first pass and fixed in 7630045.
- Deferred to Correct user guide claims that contradict the implementation #940: the same claims in
docs/*.md. Apply dict_type only to the dict cursor it was given to #930 edits thedict_typeparagraph of the dict cursor__init__docstrings, which this PR does not touch; a rebase may be needed if it merges first.
| >>> df = cursor.fetchall() | ||
| >>> df = cursor.as_pandas() | ||
| >>> | ||
| >>> # Or iterate through chunks for memory efficiency |
There was a problem hiding this comment.
Independent review (relayed): FINDINGS
Reviewer: Codex CLI 0.160.0, model gpt-6-astra, read-only sandbox. This was a static review: no builds, tests, edits, or GitHub access.
Session: 01a0ffef-2c3f-7be3-bc60-150e8300a47d. Base 9b74767cf65e0ef513338fb2bb2da678cdfbcd2a, head 763004580effd74f444ca6a0045f25eb987f9fa7.
Input: a detached snapshot plus the literal diff, without the PR number, description, commit messages, or earlier findings.
Limitation: Codex loaded its own local memory summaries at startup; none of them concern this change.
The snapshot stayed clean (git status empty after the run).
Reviewer output follows verbatim.
Covered all 33 changed files: cursor/result-set contracts and examples across sync, Future, aio, pandas, Arrow, Polars, and S3FS; type hints; attribute lists; converters/formatters; UNLOAD; execution options; S3 filesystem/executors; Spark/models; SQLAlchemy rendering; and Sphinx/MyST syntax. Followed base classes and mixins, including locally installed dependencies matching the lockfile.
FINDINGS
-
pyathena/pandas/result_set.py:216 — The revised example still does not enable chunking. It promises memory-efficient chunk iteration but constructs a cursor with chunking disabled. PandasCursor defaults and CSV loading show that the entire CSV is loaded. A reader following this example for oversized results can exhaust memory during
execute(). Running the complete snippet also consumes the sole DataFrame throughas_pandas(), leaving the subsequent loop empty. -
pyathena/polars/async_cursor.py:213 — The single-Future return claim remains. The prose says
execute()returns a future, although the implementation at line 258 returns(query_id, future). Following that claim withfuture = cursor.execute(...); future.result()raisesAttributeError. This leaves the same incorrect contract corrected forAsyncCursorelsewhere in this diff. -
pyathena/filesystem/s3_async.py:591 — The claim that execution avoids threads remains.
AioS3Filestill says its executor uses the event loop “instead of threads.”S3AioExecutor.submit()explicitly dispatches throughasyncio.to_thread; parallel range reads use that executor. A reader choosing this backend to avoid worker-thread consumption will still consume the loop’s worker pool during parallel S3 reads. -
pyathena/filesystem/s3_async.py:63 — The new sync-wrapper restriction is too broad. It requires a caller outside any running event loop. In the matching fsspec 2026.9.0 implementation, construction selects a separate loop, and
sync()rejects only calls from that same loop. A notebook or asyncio coroutine can callAioS3FileSystem().ls(...), blocking its caller while fsspec’s separate loop executes. The new text incorrectly excludes this supported case.
No Markdown/Sphinx syntax defects found by static inspection. No builds, tests, edits, or GitHub access occurred; the snapshot remains clean at the supplied head.
There was a problem hiding this comment.
Repairs in a301f5f (all 4 findings verified against the code):
pandas/result_set.py: the chunk example now creates a cursor withchunksize=50_000and runs a newexecute(). That enables chunking and no longer reuses the iteratoras_pandas()consumed. The consumption itself is the code defect filed as as_pandas() and as_polars() consume the result iterator, so repeated calls and later fetches return nothing #935.polars/async_cursor.py:213: now says it returns the query ID with a future. No other*async_cursor.pydescribes the return value as a single future (checked with grep).filesystem/s3_async.pyAioS3File: now says work goes through the event loop withasyncio.to_thread, not "instead of threads".filesystem/s3_async.py:63: dropped the "outside a running event loop" restriction. Withasynchronous=False, fsspec runs its own IO loop (fsspec/asyn.py:445), andsync()rejects only calls made from that loop (:81). The comment now says the wrappers block the caller. The same text indocs/aio.mdis fixed in Correct user guide claims that contradict the implementation #940.
Validation: just lint passes on a301f5f. Both self-review perspectives on the repair: only docstrings changed, the claims are traced to the cited code, and grep found no other copies of these claims in pyathena/.
There was a problem hiding this comment.
Independent follow-up (relayed): CLEAN
Reviewer: Codex CLI 0.160.0, gpt-6-astra, read-only, static. Session 01a0ffff-2eb1-7ed1-b8b6-62de9fb8d82a. Patch 763004580effd74f444ca6a0045f25eb987f9fa7..a301f5fc5d7c7c1b9958f0a7a94fc635f3eb8691.
Result: all 4 findings RESOLVED. No new false claims, failing examples, or Markdown/MyST problems in the repair patch. The reviewer confirmed the fsspec 2026.9.0 wrapper behavior against the installed source. The snapshot stayed clean.
Validation on a301f5f: just lint passes, and sphinx-build has no new warnings compared with master (one duplicate warning removed).
There was a problem hiding this comment.
Rebase onto master 7be04bd (#928, #930 merged)
Force-pushed with --force-with-lease=docs/927-docstrings:a301f5fc. New head a27fd581fab01d74ce3918186043995e2fb27283.
git range-diff 9b74767c..a301f5fc 7be04bdf..a27fd581: commits 2 and 3 are identical. Commit 1 differs only where it conflicted in aio/result_set.py, which now keeps both the new result_set_type_hints wording and #930's new **kwargs (dict_type) entry.
Upstream impact: #930 adds the dict_type constructor arguments, and #928 changes S3 listing code. Neither touches the claims this PR changes. Grep found no new copies of the corrected wording.
Re-run on a27fd58: just lint passes, the Attributes sweep reports only the TypeNode dataclass fields, and sphinx-build has no new warnings compared with master 7be04bd (one duplicate removed). The patch content is unchanged, so no further independent review was requested.
Fix the docstring claims verified in #927: examples that use a nonexistent attribute, treat AsyncCursor.execute() as returning a future, omit await on AioConnection.create(), call fetchall() for DataFrames or Spark output, iterate a cursor for chunks, or pass a tuple to the pyformat style. Also remove Attributes entries that are not public attributes, describe result_set_type_hints keys and the fetchmany() fallback as implemented, and correct the wrap_unload output, the supported formatter types, the pandas arraysize/retry_config/kwargs descriptions, the Polars converter scope, the S3 storage class list, and the type compiler mapping. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a301f5f to
a27fd58
Compare
WHAT
This PR changes docstrings only, under
pyathena/. It corrects claims that do not match the code.AthenaQueryExecutionusedcursor._last_query_execution, which does not exist.AsyncCursor/AsyncDictCursorexamples treated theexecute()return value as a future.AioCursor/AioDictCursorcalledAioConnection.create(...)withoutawait.fetchall()to get a DataFrame or Table, and iterated the cursor to get chunks.SparkCursorcalledfetchall(), which Spark cursors do not have.Cursorpassed a tuple with%s, but the pyformat style takes a dict.AioS3FileSystemsync-wrapper example used anasynchronous=Trueinstance.description/rowcount/max_workersonAsyncCursor,chunksizeonPandasCursor,engine/chunksizeonAsyncPandasCursor,engine_configuration/max_workerson the Spark cursors,defaultonConverter/Formatter, andsession/client/config/retry_configonS3FileSystem.result_set_type_hintsnow says the keys can be names (case-insensitive) or zero-based indexes, in all 19 places that said "column names".fetchone()/fetchmany()docs now cover dict rows and the fallback for non-positive sizes.wrap_unloadshows its real output and lists itsRaises:.DefaultParameterFormatterno longer liststime.arraysize,retry_config, and the constructor**kwargsare described as implemented, and the chunk-size optimization is marked opt-in.S3StorageClasslistsBUCKET/DIRECTORY.ExecuteOptionsno longer saysmergecan reset a field.S3AioExecutordescription is corrected.AthenaCalculationExecutionAPI link now points toGetCalculationExecution.WHY
Part of #927. The audit results are in #927 (comment).
Readers copy these examples and rely on the documented attributes, so each wrong claim turns into a failing call.
Code defects found by the same audit are filed separately (#934-#938) and are not documented here.
The
docs/*.mduser guides follow in a separate PR.TEST
Tested commit a27fd58 (rebased onto master 7be04bd after #928 and #930):
just lintpasses (license headers, ruff check including pydocstyle, ruff format, mypy, cfn-lint).sphinx-build -b html docs <out>has no new warnings compared with master 9b74767. One duplicate-object warning forAsyncCursor.descriptionis gone.wrap_unloadoutput ends in<uuid>/.DefaultParameterFormatterformatsdate/datetimeand raisesTypeErrorfordatetime.time.Attributes:section with its public attributes now reports only theTypeNodedataclass fields, which do exist.🤖 Generated with Claude Code