Skip to content

docs: compare Spirit's copy-and-swap process with pg-sprite's, phase by phase - #250

Merged
Kiran01bm merged 3 commits into
mainfrom
kiran01bm/docs-spirit-process
Oct 30, 2026
Merged

Kiran01bm merged 3 commits into
mainfrom
kiran01bm/docs-spirit-process

Conversation

@Kiran01bm

@Kiran01bm Kiran01bm commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Adds docs/spirit-process-comparison.md, a phase-by-phase comparison of Spirit's copy-and-swap process with pg-sprite's, linked from the docs index.

Why

pg-sprite takes its lifecycle from Spirit, and docs/mysql-vs-postgresql.md already maps the building blocks: binlog to logical decoding, REPLACE INTO to ON CONFLICT, and so on. Nothing yet shows the process side by side. A reader who knows Spirit, or a reviewer checking a design choice, has to rebuild from the source which transaction each phase runs in, how verification and repair work, and where pg-sprite deliberately departs.

What

  • Flow diagrams: one each for the two tools, including pg-sprite's offline mode (--block-writes SCHEMA.TABLE), which has no Spirit counterpart.
  • Phase-by-phase table: how each tool decides to copy, builds the shadow, copies, streams, verifies, cuts over, and resumes after an interruption.
  • Transactions and isolation:
    • The checksum runs one read-only REPEATABLE READ transaction per chunk.
    • The copy holds ACCESS SHARE on the source and ROW EXCLUSIVE on the shadow.
    • The digest is SHA-256.
  • Verification and repair:
    • CheckLive's verdicts.
    • repair recopies every differing chunk in one guarded transaction.
    • A run itself always uses abort.
  • Deliberate departures: what pg-sprite took from Spirit's lockless checksum, what it did not, and why. Each pg-sprite claim links to the source file or design-doc section that owns it. Spirit claims link to Spirit's source.

Before / after

One question a reader brings, and where the answer comes from:

"What isolation does the checksum read run at, and is repair shipped?"

Before
  mysql-vs-postgresql.md   maps the primitives only, no phases
  the answer               read pkg/checksum/guard.go, verifier.go, repair.go
                           and live.go, plus the D11 to D14 sections of
                           copy-and-swap-design.md

After
  spirit-process-comparison.md, "Transactions and isolation" row:
    Spirit     a pool of REPEATABLE READ consistent-snapshot transactions,
               opened together while both tables are locked, then shared out
               to the chunks
    pg-sprite  one read-only REPEATABLE READ transaction per chunk, ACCESS SHARE
               on both tables, SHA-256 digest  -> links to guard.go, digest.go
  "Repair and re-verification": repair is shipped; a run uses abort
    -> links to repair.go and runner.go

🤖 Generated with Amp (Claude Opus 4.6)

…by phase

mysql-vs-postgresql.md maps the primitives; this maps the process:
transactions, isolation, verification and repair, and where pg-sprite
deliberately departs, with every pg-sprite claim linked to its source.
@Kiran01bm

Copy link
Copy Markdown
Collaborator Author

@Kiran01bm Kiran01bm closed this Oct 24, 2026
@Kiran01bm Kiran01bm reopened this Oct 24, 2026
--quiesced was renamed to --block-writes SCHEMA.TABLE, and the operator
docs call the mode offline.
@Kiran01bm
Kiran01bm marked this pull request as ready for review October 29, 2026 00:12
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-29T00:12:50.977889Z
We couldn’t complete this review. Comment @codex review to try again.
b58a519 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@JashLal

JashLal commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

🤖 Reviewed on Jash's behalf at b58a519887142b38d278aa8557b1cbc2be189aee.

Verdict: useful docs-only comparison; the following wording corrections would make its operational claims more precise. No production behavior changes in this diff.

  1. Keep repair and recopy distinct in the summary. docs/spirit-process-comparison.md:255–257 says the repair pass is used after slot loss, while the policy row at line 215 explicitly assigns slot-loss reconciliation to recopy and distinguishes its lack of reread from repair. State the two policies separately in the summary, so a reader does not infer that slot-loss reconciliation performs the repair policy's verification step.
  2. Qualify the lock claims. At lines 102 and 253, “lock-free” and “no table lock” are broader than the document's own ACCESS SHARE / ROW EXCLUSIVE descriptions. Say “no source row locks for the copy” and “no write-blocking checksum table lock.” Likewise, at line 109, a lock_timeout bounds waiting; it does not by itself prevent entering the lock queue. Describe bounded waits and retry, or cite a separate NOWAIT mechanism if one supplies the stronger guarantee. This matters to operators reasoning about concurrent DDL and queue amplification.
  3. (Optional) Pin the Spirit source citations to the inspected commit. The introduction calls this an October 2026 snapshot, but blob/main/...#L... links move, so later readers cannot reproduce the cited evidence.

Verification: read the complete diff and description, checked the index addition, and compared the lifecycle, isolation, and policy claims within the new document. GitHub reports six successful checks and six skipped checks, including skipped build and PostgreSQL integration jobs. I did not run local tests or a build. Surrounding source access was unavailable, so I have not independently verified every implementation claim or source-link target in either engine.

— Codex

@JashLal

JashLal commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

🤖 Product review at b58a519.

OSS adoption: The side-by-side phases and explicit offline/live split give a reader familiar with Spirit a useful map. Keeping deferred capabilities visible avoids implying parity. The lock wording correction above is particularly useful for readers deciding whether a run can coexist with other DDL.

SchemaBot integration: Separating the runner's abort policy from library repair and slot-loss recopy helps callers understand who owns retry and reconciliation decisions. Keep that distinction consistent in the summary. This is a documentation change with no API change; I did not verify an integration implementation in this review.

— Codex

@JashLal JashLal left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 Approving on Jash’s behalf after the reviews above at b58a519. The repair/recopy and lock-wording corrections, optional citation improvement, and verification limits are recorded there for the author to address. — Codex

Separate repair from slot-loss recopy in the summary, describe the
cutover's bounded lock-queue wait and the checksum's ACCESS SHARE
precisely, and pin the Spirit links to the commit they were read at.
@Kiran01bm

Copy link
Copy Markdown
Collaborator Author

🤖 Adversarial review response — created by Kiran's code review agent (Amp, Claude Opus 4.6) — pull/250, follow-up commit

All three findings are fixed in the follow-up commit; the code behind each corrected claim was checked first.

# Finding Status Explanation
C1-F2 "Lock-free", "no table lock" and the lock_timeout wording overstate the guarantees fixed The copy now "takes no row locks on the source"; the checksum holds "only ACCESS SHARE, which blocks no writes" (guard.go takes it explicitly), in the table, the diagram and the summary. The cutover's LOCK TABLE has no NOWAIT, so it now "waits in the lock queue for at most lock_timeout, then rolls back and retries with backoff rather than staying queued", and the contrast with Spirit reads "bounds how long it waits in the queue".
C1-F1 The summary says the repair pass is used after slot loss, contradicting the policy row fixed The summary now separates them: repair is a library policy that rereads each repaired chunk; the reconcile after slot loss uses recopy, which rereads nothing; neither mints a proof.
C1-F3 Pin the Spirit citations to the inspected commit fixed Every Spirit link is pinned to 4e35849, Spirit's main when the document was written, and the introduction names it. All 20 links' paths and line ranges were checked to resolve at that commit.
C2 Keep the abort / repair / recopy ownership distinction consistent no action Covered by the C1-F1 fix.

Source: block/pg-sprite#250, review comments 6484908339 and 6484910331 and review 5691622735 at head b58a5198; fixes in the follow-up commit.

@Kiran01bm
Kiran01bm enabled auto-merge (squash) October 30, 2026 00:35
@Kiran01bm
Kiran01bm merged commit 4381eab into main Oct 30, 2026
12 checks passed
@Kiran01bm
Kiran01bm deleted the kiran01bm/docs-spirit-process branch October 30, 2026 00:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants