Tighten the Spark and SQL cancellation notes in the docs - #920
Conversation
The cancellation list stated that a running Spark job always stops within seconds and ends in the CANCELED state. A stop request sent right after the calculation starts can occasionally have no effect (measured for #841: 1 of 6 attempts), so qualify the statement and list that case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
|
||
| - A running Spark job, such as a DataFrame action, stops within seconds. | ||
| The calculation ends in the `CANCELED` state, and the session remains usable for later calculations. | ||
| - A running Spark job, such as a DataFrame action, usually stops within seconds. |
There was a problem hiding this comment.
Self-review rounds 1 and 2 (docs-only): range 2fcbcdbe7ae01b31a4a3bb613e6bccdfcd382014..b304b3c5a12546f7b8a0288b60e2a05cad10e326. Result: CLEAN
Round 1, content against the code:
cancel()and the interrupt paths sendStopCalculationExecutiononce, throughSparkBaseCursor._cancel/AioSparkCursor._cancel, and wait for whatever terminal state Athena reports. The cursor guarantees no particular outcome, which matches "usually".- The new bullet does not conflict with the existing start-phase sentence further down the section ("can occasionally have no effect, so the calculation can still end in the
COMPLETEDstate").
Round 2, claims against the evidence:
- "Usually stops within seconds": Spark: verify best-effort calculation cancellation without mandatory session termination #797 measured a stop of about 2.6 s on a running Spark job.
- "Occasionally have no effect" right after start: the Spark: an interrupt during StartCalculationExecution leaves the calculation running #841 measurement, quoted in Cancel a Spark calculation interrupted while it is being started #861's description, cancelled within 0.4 s in 5 of 6 attempts and had no effect in 1 of 6 (
COMPLETEDafter about 95 s). - "Runs as if it had not been canceled" matches that ignored case.
- The PR description's claim that no AWS jobs run is checked against
paths-ignore: ['docs/**', '**.md']in.github/workflows/test.yaml. - Only one place in
docs/orpyathena/contains the "within seconds" wording (git grep).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| - A running Spark job, such as a DataFrame action, usually stops within seconds. | ||
| The calculation then ends in the `CANCELED` state, and the session remains usable for later calculations. | ||
| - A request sent right after the calculation starts can occasionally have no effect. | ||
| The calculation then runs as if it had not been canceled. |
There was a problem hiding this comment.
Independent review (relayed): Codex CLI, reported model gpt-6-astra, reasoning effort max (as reported), session 01a0ff8d-b6b0-70c2-b1ad-802d54551ff1. Result: FINDINGS (unchanged text in the same section)
Scope: 2fcbcdbe7ae01b31a4a3bb613e6bccdfcd382014..b304b3c5a12546f7b8a0288b60e2a05cad10e326 plus the whole Cancellation section. The reviewer ran in a read-only detached snapshot with no PR framing, against the measured facts supplied in the prompt. Static review. The snapshot and the PR worktree were unchanged afterwards.
Covered surfaces: the exact hunk, the entire Cancellation section, cancellation/startup/polling/state/token paths in all four named source files, and the supplied Athena observations. Static review only.
FINDINGS
- P2 — docs/spark.md:290: Startup interruption does not always wait and expose a calculation ID. If
KeyboardInterruptarrives before the helper begins the request,future.cancel()succeeds. The interrupt propagates immediately, the helper sends no request, andcalculation_idremainsNone, having been cleared byexecute(). Qualify the wait/cancel/ID claims to requests the helper has already begun. This finding concerns unchanged text in the requested section.
The changed list matches the supplied observations and introduces no contradiction.
Disposition: fixed in the next commit. Verified: BaseCursor._start_execution() abandons a request the helper has not begun (future.cancel()), and the interrupt then propagates with calculation_id still None, because SparkCursor.execute() clears it. docs/spark.md now adds after the start-phase sentences: "If execute() has not begun the request when the interrupt is handled, the request is never sent and calculation_id is None." The wording matches docs/usage.md.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
| [StartCalculationExecution](https://docs.aws.amazon.com/athena/latest/APIReference/API_StartCalculationExecution.html) | ||
| request to finish, and then cancels the calculation it started in the same way. | ||
| The `calculation_id` property returns that calculation's ID. | ||
| If `execute()` has not begun the request when the interrupt is handled, the request is never sent and `calculation_id` is `None`. |
There was a problem hiding this comment.
Independent follow-up review (relayed): Codex CLI, reported model gpt-6-astra, reasoning effort max (as reported), session 01a0ff92-5cc9-71d0-a765-4aad595cadc6. Scope: b304b3c5..0e6657343e78c47f3245df4f4d337bac0537e524 plus the whole Cancellation section, read-only snapshot, static review. The snapshot and the PR worktree were unchanged afterwards.
Covered: the new diff, the entire Cancellation section, all four named source files, and the supplied Athena observations. Static review only.
Previous finding: resolved by docs/spark.md:294.
FINDINGS
- P2 — docs/spark.md:287: Terminal-state guarantee omits polling failures. During a long
time.sleep(), Ctrl-C can successfully send the stop request, followed by status polling exhausting its retries. The calculation continues, butKeyboardInterruptpropagates andcursor.stateremainsNone: the state assignment never completes, and the handler chains the polling error. Lines 286–288 should qualify the wait/state guarantee for polling failures as well as cancellation-request failures.
Disposition: fixed in 0c64829 (pre-existing text from #833, kept in scope because this PR tightens this section).
Verified: BaseCursor._poll() chains any Exception from _cancel_and_wait(), including a failed status request, as __cause__. On that path, SparkBaseCursor._cancel_and_wait() never stores the execution, so state returns None after the reset in SparkCursor.execute().
docs/spark.md now says: "If the cancellation request or that wait fails, the KeyboardInterrupt propagates with the error as its cause, and the state property returns None." The SQL notes in docs/usage.md and docs/aio.md (from #853) had the same omission, and now say "the cancellation request or that wait".
| requests cancellation, waits until the calculation reaches a terminal state, and then propagates. | ||
| The `state` property returns that terminal state. | ||
| If the cancellation request fails, the `KeyboardInterrupt` propagates with the error as its cause. | ||
| If the cancellation request or that wait fails, the `KeyboardInterrupt` propagates with the error as its cause, |
There was a problem hiding this comment.
Self-review of the repairs (both perspectives) and independent follow-up (relayed). Result: CLEAN
Self-review of b304b3c5..0c64829d34e36dd87a896196f44f88e267117734, checked against the code:
- The abandon sentence matches
BaseCursor._start_execution()(future.cancel()) and the reset inSparkCursor.execute(). - The failed-wait wording matches
BaseCursor._poll()/AioBaseCursor._poll(), which chain anyExceptionfrom_cancel_and_wait(). - Spark
stateisNoneon that path because the execution is never stored. just docs lint: 0 errors.
Codex CLI, reported model gpt-6-astra, reasoning effort max (as reported), session 01a0ff96-3239-7133-96c1-f5c4d1bbae53. Scope: 2fcbcdbe..0c64829d34e36dd87a896196f44f88e267117734. In one pass, the reviewer checked every sentence of the Spark Cancellation section, including its asyncio counterpart, the usage "Query cancellation on interrupt" section, and the aio "Task cancellation" subsection, against nine source files and the measured facts. Read-only snapshot, static review. The snapshot and the PR worktree were unchanged afterwards.
Covered: every sentence in Spark’s Cancellation section and asyncio counterpart, Query cancellation on interrupt, and Task cancellation, against all nine named source files and the supplied Athena observations.
Previous finding: resolved. The notes cover failed cancellation requests and failed subsequent waits, including Spark’s state=None.
CLEAN — no remaining actionable false statements found. Static review only; no files changed or tests/network operations performed.
WHAT
In
docs/spark.md, the cancellation list stated that a running Spark job "stops within seconds" and "ends in theCANCELEDstate" without exception.The first bullet now says the job usually stops within seconds and then ends in
CANCELED.A new bullet in the same list says that a stop request sent right after the calculation starts can occasionally have no effect, and the calculation then runs as if it had not been canceled.
The start-phase interrupt paragraph now also says that a request
execute()has not begun when the interrupt is handled is never sent, and thatcalculation_idis thenNone. This is the abandon path from Cancel a Spark calculation interrupted while it is being started #861;SparkCursor.execute()clearscalculation_idbefore starting (Re-raise and stop interrupted queries in the SQL cursors, sharing the Spark handling #853).The cancellation failure notes in
docs/spark.md,docs/usage.md, anddocs/aio.mdnow cover a failed wait as well as a failed cancellation request. Both become the interrupt's__cause__(BaseCursor._poll()/AioBaseCursor._poll()chain anyExceptionfrom_cancel_and_wait()). For Spark,statethen returnsNone, becauseSparkCursor.execute()cleared the previous execution and the wait never stored one.Docs only; no code change.
WHY
Follow-up to the independent review of #853, which flagged this sentence as contradicting the start-phase paragraph further down the same section ("A cancellation request sent right after a calculation starts can occasionally have no effect").
The text came from #833 and was out of scope for #853.
Measured for #841 on the CI account:
StopCalculationExecutionsent right afterStartCalculationExecution(stateCREATED) cancelled the calculation within 0.4 s in 5 of 6 attempts. In 1 of 6, it had no effect, and the calculation endedCOMPLETEDafter about 95 s.A stop sent to a running Spark job cancelled it in about 2.6 s (#797).
TEST
Tested commit: 0c64829.
just docs lint: 0 errors.docs/**and**.md(paths-ignore), so no AWS jobs run.🤖 Generated with Claude Code