Skip to content

Tighten the Spark and SQL cancellation notes in the docs - #920

Merged
laughingman7743 merged 3 commits into
masterfrom
docs/spark-cancel-best-effort
Oct 3, 2026
Merged

laughingman7743 merged 3 commits into
masterfrom
docs/spark-cancel-best-effort

Conversation

@laughingman7743

@laughingman7743 laughingman7743 commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

WHAT

In docs/spark.md, the cancellation list stated that a running Spark job "stops within seconds" and "ends in the CANCELED state" without exception.

  • The first bullet now says the job usually stops within seconds and then ends in CANCELED.

  • A new bullet in the same list says that a stop request sent right after the calculation starts can occasionally have no effect, and the calculation then runs as if it had not been canceled.

  • The start-phase interrupt paragraph now also says that a request execute() has not begun when the interrupt is handled is never sent, and that calculation_id is then None. This is the abandon path from Cancel a Spark calculation interrupted while it is being started #861; SparkCursor.execute() clears calculation_id before starting (Re-raise and stop interrupted queries in the SQL cursors, sharing the Spark handling #853).

  • The cancellation failure notes in docs/spark.md, docs/usage.md, and docs/aio.md now cover a failed wait as well as a failed cancellation request. Both become the interrupt's __cause__ (BaseCursor._poll() / AioBaseCursor._poll() chain any Exception from _cancel_and_wait()). For Spark, state then returns None, because SparkCursor.execute() cleared the previous execution and the wait never stored one.

Docs only; no code change.

WHY

Follow-up to the independent review of #853, which flagged this sentence as contradicting the start-phase paragraph further down the same section ("A cancellation request sent right after a calculation starts can occasionally have no effect").
The text came from #833 and was out of scope for #853.

Measured for #841 on the CI account: StopCalculationExecution sent right after StartCalculationExecution (state CREATED) cancelled the calculation within 0.4 s in 5 of 6 attempts. In 1 of 6, it had no effect, and the calculation ended COMPLETED after about 95 s.
A stop sent to a running Spark job cancelled it in about 2.6 s (#797).

TEST

Tested commit: 0c64829.

  • just docs lint: 0 errors.
  • Docs-only change: no tests run; the Test workflow ignores docs/** and **.md (paths-ignore), so no AWS jobs run.

🤖 Generated with Claude Code

The cancellation list stated that a running Spark job always stops
within seconds and ends in the CANCELED state. A stop request sent right
after the calculation starts can occasionally have no effect (measured
for #841: 1 of 6 attempts), so qualify the statement and list that case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread docs/spark.md

- A running Spark job, such as a DataFrame action, stops within seconds.
The calculation ends in the `CANCELED` state, and the session remains usable for later calculations.
- A running Spark job, such as a DataFrame action, usually stops within seconds.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review rounds 1 and 2 (docs-only): range 2fcbcdbe7ae01b31a4a3bb613e6bccdfcd382014..b304b3c5a12546f7b8a0288b60e2a05cad10e326. Result: CLEAN

Round 1, content against the code:

  • cancel() and the interrupt paths send StopCalculationExecution once, through SparkBaseCursor._cancel / AioSparkCursor._cancel, and wait for whatever terminal state Athena reports. The cursor guarantees no particular outcome, which matches "usually".
  • The new bullet does not conflict with the existing start-phase sentence further down the section ("can occasionally have no effect, so the calculation can still end in the COMPLETED state").

Round 2, claims against the evidence:

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread docs/spark.md
- A running Spark job, such as a DataFrame action, usually stops within seconds.
The calculation then ends in the `CANCELED` state, and the session remains usable for later calculations.
- A request sent right after the calculation starts can occasionally have no effect.
The calculation then runs as if it had not been canceled.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent review (relayed): Codex CLI, reported model gpt-6-astra, reasoning effort max (as reported), session 01a0ff8d-b6b0-70c2-b1ad-802d54551ff1. Result: FINDINGS (unchanged text in the same section)

Scope: 2fcbcdbe7ae01b31a4a3bb613e6bccdfcd382014..b304b3c5a12546f7b8a0288b60e2a05cad10e326 plus the whole Cancellation section. The reviewer ran in a read-only detached snapshot with no PR framing, against the measured facts supplied in the prompt. Static review. The snapshot and the PR worktree were unchanged afterwards.

Covered surfaces: the exact hunk, the entire Cancellation section, cancellation/startup/polling/state/token paths in all four named source files, and the supplied Athena observations. Static review only.

FINDINGS

  • P2 — docs/spark.md:290: Startup interruption does not always wait and expose a calculation ID. If KeyboardInterrupt arrives before the helper begins the request, future.cancel() succeeds. The interrupt propagates immediately, the helper sends no request, and calculation_id remains None, having been cleared by execute(). Qualify the wait/cancel/ID claims to requests the helper has already begun. This finding concerns unchanged text in the requested section.

The changed list matches the supplied observations and introduces no contradiction.


Disposition: fixed in the next commit. Verified: BaseCursor._start_execution() abandons a request the helper has not begun (future.cancel()), and the interrupt then propagates with calculation_id still None, because SparkCursor.execute() clears it. docs/spark.md now adds after the start-phase sentences: "If execute() has not begun the request when the interrupt is handled, the request is never sent and calculation_id is None." The wording matches docs/usage.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@laughingman7743 laughingman7743 changed the title Describe Spark cancellation as usually taking effect Tighten the Spark and SQL cancellation notes in the docs Oct 3, 2026
Comment thread docs/spark.md
[StartCalculationExecution](https://docs.aws.amazon.com/athena/latest/APIReference/API_StartCalculationExecution.html)
request to finish, and then cancels the calculation it started in the same way.
The `calculation_id` property returns that calculation's ID.
If `execute()` has not begun the request when the interrupt is handled, the request is never sent and `calculation_id` is `None`.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independent follow-up review (relayed): Codex CLI, reported model gpt-6-astra, reasoning effort max (as reported), session 01a0ff92-5cc9-71d0-a765-4aad595cadc6. Scope: b304b3c5..0e6657343e78c47f3245df4f4d337bac0537e524 plus the whole Cancellation section, read-only snapshot, static review. The snapshot and the PR worktree were unchanged afterwards.

Covered: the new diff, the entire Cancellation section, all four named source files, and the supplied Athena observations. Static review only.

Previous finding: resolved by docs/spark.md:294.

FINDINGS

  • P2 — docs/spark.md:287: Terminal-state guarantee omits polling failures. During a long time.sleep(), Ctrl-C can successfully send the stop request, followed by status polling exhausting its retries. The calculation continues, but KeyboardInterrupt propagates and cursor.state remains None: the state assignment never completes, and the handler chains the polling error. Lines 286–288 should qualify the wait/state guarantee for polling failures as well as cancellation-request failures.

Disposition: fixed in 0c64829 (pre-existing text from #833, kept in scope because this PR tightens this section).

Verified: BaseCursor._poll() chains any Exception from _cancel_and_wait(), including a failed status request, as __cause__. On that path, SparkBaseCursor._cancel_and_wait() never stores the execution, so state returns None after the reset in SparkCursor.execute().

docs/spark.md now says: "If the cancellation request or that wait fails, the KeyboardInterrupt propagates with the error as its cause, and the state property returns None." The SQL notes in docs/usage.md and docs/aio.md (from #853) had the same omission, and now say "the cancellation request or that wait".

Comment thread docs/spark.md
requests cancellation, waits until the calculation reaches a terminal state, and then propagates.
The `state` property returns that terminal state.
If the cancellation request fails, the `KeyboardInterrupt` propagates with the error as its cause.
If the cancellation request or that wait fails, the `KeyboardInterrupt` propagates with the error as its cause,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review of the repairs (both perspectives) and independent follow-up (relayed). Result: CLEAN

Self-review of b304b3c5..0c64829d34e36dd87a896196f44f88e267117734, checked against the code:

  • The abandon sentence matches BaseCursor._start_execution() (future.cancel()) and the reset in SparkCursor.execute().
  • The failed-wait wording matches BaseCursor._poll() / AioBaseCursor._poll(), which chain any Exception from _cancel_and_wait().
  • Spark state is None on that path because the execution is never stored.
  • just docs lint: 0 errors.

Codex CLI, reported model gpt-6-astra, reasoning effort max (as reported), session 01a0ff96-3239-7133-96c1-f5c4d1bbae53. Scope: 2fcbcdbe..0c64829d34e36dd87a896196f44f88e267117734. In one pass, the reviewer checked every sentence of the Spark Cancellation section, including its asyncio counterpart, the usage "Query cancellation on interrupt" section, and the aio "Task cancellation" subsection, against nine source files and the measured facts. Read-only snapshot, static review. The snapshot and the PR worktree were unchanged afterwards.

Covered: every sentence in Spark’s Cancellation section and asyncio counterpart, Query cancellation on interrupt, and Task cancellation, against all nine named source files and the supplied Athena observations.

Previous finding: resolved. The notes cover failed cancellation requests and failed subsequent waits, including Spark’s state=None.

CLEAN — no remaining actionable false statements found. Static review only; no files changed or tests/network operations performed.

@laughingman7743
laughingman7743 marked this pull request as ready for review October 3, 2026 02:32
@laughingman7743
laughingman7743 merged commit 775874c into master Oct 3, 2026
3 checks passed
@laughingman7743
laughingman7743 deleted the docs/spark-cancel-best-effort branch October 3, 2026 02:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant