Skip to content

Correct docstrings that contradict the implementation - #939

Merged
laughingman7743 merged 3 commits into
masterfrom
docs/927-docstrings
Oct 3, 2026
Merged

laughingman7743 merged 3 commits into
masterfrom
docs/927-docstrings

Conversation

@laughingman7743

@laughingman7743 laughingman7743 commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

WHAT

This PR changes docstrings only, under pyathena/. It corrects claims that do not match the code.

  • Examples that fail as written:
    • AthenaQueryExecution used cursor._last_query_execution, which does not exist.
    • The AsyncCursor / AsyncDictCursor examples treated the execute() return value as a future.
    • AioCursor / AioDictCursor called AioConnection.create(...) without await.
    • The pandas and Arrow examples called fetchall() to get a DataFrame or Table, and iterated the cursor to get chunks.
    • SparkCursor called fetchall(), which Spark cursors do not have.
    • Cursor passed a tuple with %s, but the pyformat style takes a dict.
    • The AioS3FileSystem sync-wrapper example used an asynchronous=True instance.
  • Attributes that are not public attributes: removed description/rowcount/max_workers on AsyncCursor, chunksize on PandasCursor, engine/chunksize on AsyncPandasCursor, engine_configuration/max_workers on the Spark cursors, default on Converter/Formatter, and session/client/config/retry_config on S3FileSystem.
  • Behavior descriptions:
    • result_set_type_hints now says the keys can be names (case-insensitive) or zero-based indexes, in all 19 places that said "column names".
    • The fetchone()/fetchmany() docs now cover dict rows and the fallback for non-positive sizes.
    • wrap_unload shows its real output and lists its Raises:.
    • DefaultParameterFormatter no longer lists time.
    • Pandas: arraysize, retry_config, and the constructor **kwargs are described as implemented, and the chunk-size optimization is marked opt-in.
    • The Polars converter scope is corrected.
    • S3StorageClass lists BUCKET/DIRECTORY.
    • The type compiler's FLOAT/string mapping is corrected.
    • ExecuteOptions no longer says merge can reset a field.
    • The S3AioExecutor description is corrected.
    • The AthenaCalculationExecution API link now points to GetCalculationExecution.

WHY

Part of #927. The audit results are in #927 (comment).
Readers copy these examples and rely on the documented attributes, so each wrong claim turns into a failing call.
Code defects found by the same audit are filed separately (#934-#938) and are not documented here.
The docs/*.md user guides follow in a separate PR.

TEST

Tested commit a27fd58 (rebased onto master 7be04bd after #928 and #930):

  • just lint passes (license headers, ruff check including pydocstyle, ruff format, mypy, cfn-lint).
  • sphinx-build -b html docs <out> has no new warnings compared with master 9b74767. One duplicate-object warning for AsyncCursor.description is gone.
  • Offline checks:
    • The wrap_unload output ends in <uuid>/.
    • DefaultParameterFormatter formats date/datetime and raises TypeError for datetime.time.
    • A script that compares each class's Attributes: section with its public attributes now reports only the TypeNode dataclass fields, which do exist.
  • The changes are docstrings only, so no AWS tests are needed.

🤖 Generated with Claude Code

Comment thread pyathena/async_cursor.py
rowcount: Number of rows affected by the last query (-1 for SELECT queries).
arraysize: Default number of rows to fetch with fetchmany().
max_workers: Maximum number of worker threads for concurrent execution.
arraysize: Default number of rows that fetchmany() returns on the result

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round one (implementation behavior): CLEAN

Base 9b74767cf65e0ef513338fb2bb2da678cdfbcd2a, head 763004580effd74f444ca6a0045f25eb987f9fa7 (full diff, 31 files, docstrings only).

I traced each changed claim into the code it describes:

  • AsyncCursor: execute() returns (query_id, future) (async_cursor.py:219). arraysize reaches the result sets through _collect_result_set. description(query_id) returns a Future.
  • Fetch wording:
    • fetchmany() falls back to arraysize for non-positive sizes in the base result set (result_set.py:516), in the aio result set (aio/result_set.py:202), and through WithFetch/WithAsyncFetch delegation.
    • Dict rows come from AthenaDictResultSet / AthenaAioDictResultSet.
  • Pandas retry_config: it is used for GetQueryResults and for _get_content_length/_read_data_manifest. CSV and Parquet reads go through S3FileSystem(connection=...), which takes connection.retry_config (filesystem/s3.py:182).
  • Polars converter: dtypes feed schema_overrides in both read_csv and scan_csv (polars/result_set.py:440,625). The conversion mapping covers date/time/varbinary/json only.
  • S3StorageClass BUCKET/DIRECTORY are assigned in filesystem/s3.py:284,319,378,581.
  • ExecuteOptions is a frozen dataclass, so dataclasses.replace can clear a field.
  • S3AioExecutor uses asyncio.to_thread. The sync filesystem creates one executor per opened file (_open → _create_executor).
  • Type compiler: FLOAT/REAL render FLOAT (compiler.py:104-110), CAST uses REAL (:852-856), and string types render CHAR(n)/VARCHAR(n) or STRING (:180-198).
  • Attributes sweep: a script compared each class's Attributes: with its public attributes. Only the TypeNode dataclass fields remain flagged, and they exist.

No code or test changes. Checks run: just lint and sphinx-build (no new warnings).

Comment thread pyathena/formatter.py
SELECT * FROM sales WHERE year = 2023
)
TO 's3://my-bucket/results/unload/20231215/uuid//'
TO 's3://my-bucket/results/unload/20231215/<uuid>/'

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review round two (claims, callers, documentation reader): CLEAN

Base 9b74767cf65e0ef513338fb2bb2da678cdfbcd2a, head 763004580effd74f444ca6a0045f25eb987f9fa7.

Claims checked:

  • PR body:
    • "19 places": the replacement script changed 19 result_set_type_hints entries; options.py was reworded separately.
    • The Sphinx warning delta was rerun on 7630045. The only change is that one duplicate AsyncCursor.description warning is gone.
    • The Attributes: sweep result was rerun on the head.
    • The tested commit in the PR body was updated to 7630045.
  • Examples:
    • wrap_unload was run offline. The real output ends in <uuid>/, and an empty query raises ProgrammingError.
    • DefaultParameterFormatter formats date/datetime and raises TypeError for datetime.time.
    • AioConnection.create is async def, so async with await matches aio/connection.py:26 and the README.
  • Caller impact: none. Only docstrings changed, and just lint (ruff D, mypy) passes on the head.
  • Related prose outside the diff:
    • _last_query_execution, uuid//, and AioConnection.create(...) without await no longer appear anywhere in pyathena/.
    • fetchall() # Returns now appears only for S3FS, where it is correct (tuples).
    • aio/common.py WithAsyncFetch.fetchmany was missed in the first pass and fixed in 7630045.
  • Deferred to Correct user guide claims that contradict the implementation #940: the same claims in docs/*.md. Apply dict_type only to the dict cursor it was given to #930 edits the dict_type paragraph of the dict cursor __init__ docstrings, which this PR does not touch; a rebase may be needed if it merges first.

>>> df = cursor.fetchall()
>>> df = cursor.as_pandas()
>>>
>>> # Or iterate through chunks for memory efficiency

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed): FINDINGS

Reviewer: Codex CLI 0.160.0, model gpt-6-astra, read-only sandbox. This was a static review: no builds, tests, edits, or GitHub access.
Session: 01a0ffef-2c3f-7be3-bc60-150e8300a47d. Base 9b74767cf65e0ef513338fb2bb2da678cdfbcd2a, head 763004580effd74f444ca6a0045f25eb987f9fa7.
Input: a detached snapshot plus the literal diff, without the PR number, description, commit messages, or earlier findings.
Limitation: Codex loaded its own local memory summaries at startup; none of them concern this change.
The snapshot stayed clean (git status empty after the run).

Reviewer output follows verbatim.


Covered all 33 changed files: cursor/result-set contracts and examples across sync, Future, aio, pandas, Arrow, Polars, and S3FS; type hints; attribute lists; converters/formatters; UNLOAD; execution options; S3 filesystem/executors; Spark/models; SQLAlchemy rendering; and Sphinx/MyST syntax. Followed base classes and mixins, including locally installed dependencies matching the lockfile.

FINDINGS

  1. pyathena/pandas/result_set.py:216 — The revised example still does not enable chunking. It promises memory-efficient chunk iteration but constructs a cursor with chunking disabled. PandasCursor defaults and CSV loading show that the entire CSV is loaded. A reader following this example for oversized results can exhaust memory during execute(). Running the complete snippet also consumes the sole DataFrame through as_pandas(), leaving the subsequent loop empty.

  2. pyathena/polars/async_cursor.py:213 — The single-Future return claim remains. The prose says execute() returns a future, although the implementation at line 258 returns (query_id, future). Following that claim with future = cursor.execute(...); future.result() raises AttributeError. This leaves the same incorrect contract corrected for AsyncCursor elsewhere in this diff.

  3. pyathena/filesystem/s3_async.py:591 — The claim that execution avoids threads remains. AioS3File still says its executor uses the event loop “instead of threads.” S3AioExecutor.submit() explicitly dispatches through asyncio.to_thread; parallel range reads use that executor. A reader choosing this backend to avoid worker-thread consumption will still consume the loop’s worker pool during parallel S3 reads.

  4. pyathena/filesystem/s3_async.py:63 — The new sync-wrapper restriction is too broad. It requires a caller outside any running event loop. In the matching fsspec 2026.9.0 implementation, construction selects a separate loop, and sync() rejects only calls from that same loop. A notebook or asyncio coroutine can call AioS3FileSystem().ls(...), blocking its caller while fsspec’s separate loop executes. The new text incorrectly excludes this supported case.

No Markdown/Sphinx syntax defects found by static inspection. No builds, tests, edits, or GitHub access occurred; the snapshot remains clean at the supplied head.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repairs in a301f5f (all 4 findings verified against the code):

  1. pandas/result_set.py: the chunk example now creates a cursor with chunksize=50_000 and runs a new execute(). That enables chunking and no longer reuses the iterator as_pandas() consumed. The consumption itself is the code defect filed as as_pandas() and as_polars() consume the result iterator, so repeated calls and later fetches return nothing #935.
  2. polars/async_cursor.py:213: now says it returns the query ID with a future. No other *async_cursor.py describes the return value as a single future (checked with grep).
  3. filesystem/s3_async.py AioS3File: now says work goes through the event loop with asyncio.to_thread, not "instead of threads".
  4. filesystem/s3_async.py:63: dropped the "outside a running event loop" restriction. With asynchronous=False, fsspec runs its own IO loop (fsspec/asyn.py:445), and sync() rejects only calls made from that loop (:81). The comment now says the wrappers block the caller. The same text in docs/aio.md is fixed in Correct user guide claims that contradict the implementation #940.

Validation: just lint passes on a301f5f. Both self-review perspectives on the repair: only docstrings changed, the claims are traced to the cited code, and grep found no other copies of these claims in pyathena/.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up (relayed): CLEAN

Reviewer: Codex CLI 0.160.0, gpt-6-astra, read-only, static. Session 01a0ffff-2eb1-7ed1-b8b6-62de9fb8d82a. Patch 763004580effd74f444ca6a0045f25eb987f9fa7..a301f5fc5d7c7c1b9958f0a7a94fc635f3eb8691.

Result: all 4 findings RESOLVED. No new false claims, failing examples, or Markdown/MyST problems in the repair patch. The reviewer confirmed the fsspec 2026.9.0 wrapper behavior against the installed source. The snapshot stayed clean.

Validation on a301f5f: just lint passes, and sphinx-build has no new warnings compared with master (one duplicate warning removed).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase onto master 7be04bd (#928, #930 merged)

Force-pushed with --force-with-lease=docs/927-docstrings:a301f5fc. New head a27fd581fab01d74ce3918186043995e2fb27283.
git range-diff 9b74767c..a301f5fc 7be04bdf..a27fd581: commits 2 and 3 are identical. Commit 1 differs only where it conflicted in aio/result_set.py, which now keeps both the new result_set_type_hints wording and #930's new **kwargs (dict_type) entry.
Upstream impact: #930 adds the dict_type constructor arguments, and #928 changes S3 listing code. Neither touches the claims this PR changes. Grep found no new copies of the corrected wording.
Re-run on a27fd58: just lint passes, the Attributes sweep reports only the TypeNode dataclass fields, and sphinx-build has no new warnings compared with master 7be04bd (one duplicate removed). The patch content is unchanged, so no further independent review was requested.

laughingman7743 and others added 3 commits October 3, 2026 13:45
Fix the docstring claims verified in #927: examples that use a nonexistent
attribute, treat AsyncCursor.execute() as returning a future, omit await on
AioConnection.create(), call fetchall() for DataFrames or Spark output,
iterate a cursor for chunks, or pass a tuple to the pyformat style.

Also remove Attributes entries that are not public attributes, describe
result_set_type_hints keys and the fetchmany() fallback as implemented, and
correct the wrap_unload output, the supported formatter types, the pandas
arraysize/retry_config/kwargs descriptions, the Polars converter scope, the
S3 storage class list, and the type compiler mapping.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@laughingman7743
laughingman7743 marked this pull request as ready for review October 3, 2026 04:49
@laughingman7743
laughingman7743 merged commit 91835a2 into master Oct 3, 2026
12 checks passed
@laughingman7743
laughingman7743 deleted the docs/927-docstrings branch October 3, 2026 05:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant