[Kiran01bm]

A chunk whose insert collides with a shadow row now commits every row the shadow admits, keeps only the held-off keys pending, and retries those keys alone, so the passes a chunk makes stay bounded whatever the rate of key moves.

Why

The copier's collision retry reran the whole chunk — the strict INSERT … ON CONFLICT (pk) DO NOTHING over the full key range — after each backoff. The chunk collided as a whole on the first row whose unique value another shadow row still held, so none of its rows committed, and the next pass read the whole range again. Under sustained key moves into the chunk's range, each pass could collide on a different landed row the flush had not rewritten yet; a chunk could be sent back to the start as often as moves arrived, and whether it landed within CollisionAttempts depended on the move rate, not on the bound. A move rate above one per backoff into one chunk's range was enough to exhaust the attempts on a chunk whose every row but one the shadow would have admitted.

What

• pkg/internal/chunksql: two statements beside Insert, selecting rows exactly as it does. InsertSkipping runs over the chunk's range with ON CONFLICT DO NOTHING on every unique index and exclusion constraint, and returns the rows inserted plus the ascending keys of the rows it read that are in the shadow neither now nor from before — the rows held off. InsertKeys is the same statement over a bigint[] of keys. The source rows are read once into a MATERIALIZED CTE so the insert and the held-off query see the same rows.
• pkg/copier: the first pass is the strict insert as before, so the chunk that collides on nothing — the common case — still costs one statement, and chunksql.Insert stays the one text the copier and the verifier share. On a collision the second pass runs InsertSkipping with no wait and settles the chunk: it stays in flight, but its Position entry gains Pending{Chunk, Keys}, and Classify reads every key of that chunk outside Keys as landed. Each later pass waits one backoff longer than the one before and runs InsertKeys over the pending keys alone — never the range again, so a value that moves into the range after the second pass is the applier's to write. Pending keys only shrink (the flush rewrites the row holding a key's value and the next pass lands it, or the source deletes the row and the next pass no longer reads it); the chunk lands when they are gone. A pass refused as a whole (a deferred unique constraint at commit) is rerun as it was. CollisionAttempts now bounds passes, and an exhausted chunk stays in flight with its keys pending, so a checkpoint records none of it and the resume clear removes the rows committed around them.
• ledger tracks a settled chunk's rows and pending keys; RowsInserted counts the rows of every committed pass. chunkTx is the guarded transaction copyChunk, copySkipping, and copyKeys share.
• Tests: chunksql freezes the new statements; ledger and Position unit tests cover settling and the pending-key classification; the copier integration test holds slot 15 under shadow key 2000, asserts the first chunk commits around it with key 15 alone pending (key 16 classifies as landed, key 15 in flight, the shadow holds every row of the chunk but one), then removes the row and asserts the chunk lands with every row; the exhaustion test asserts the chunk stops with key 15 still pending and absent. A new catch-up proof runs primary-key moves faster than one per backoff from before the first chunk is read, so every move lands in the last chunk while it is in flight, asserts the copier counted collisions (the moves demonstrably reached an in-flight chunk), and converges within the attempts.
• Docs: CO-4's Enforced text in invariants.md, the pkg/copier row of the design doc, and the Options and Position doc comments.

Before / after

The seats table has id as the primary key and a unique slot. The first chunk covers keys 1–100 and is read while the shadow already holds a row under key 2000 whose slot is 15 (a move the applier wrote, whose other half — the delete of the row that used to hold slot 15 — the flush has not rewritten yet). Before, every pass reran all 100 rows and the whole chunk failed on key 15 each time; after, the second pass commits 99 rows, key 15 alone waits, and the pass after the row under key 2000 is rewritten inserts key 15 by itself. The applier can write keys 1–14 and 16–100 from the second pass onward instead of deferring every change to the chunk.

Before                                                After
──────                                                ─────
pass 1  INSERT keys 1..100 ON CONFLICT (id)           pass 1  INSERT keys 1..100 ON CONFLICT (id)
        key 15: slot 15 held by key 2000 → 23505              key 15: slot 15 held by key 2000 → 23505
        whole chunk rolls back, 0 rows committed              whole chunk rolls back, 0 rows committed
        │                                                     │
        ▼ wait 1 × backoff                                    ▼ no wait
pass 2  INSERT keys 1..100 again                      pass 2  INSERT keys 1..100 ON CONFLICT DO NOTHING
        a new move landed slot 40 under key 2001              commits 99 rows; held off: [15]
        → 23505 on key 40, 0 rows committed                   chunk settled: Pending = {[1,100], keys [15]}
        │                                                     Classify(16) = landed, Classify(15) = in flight
        ▼ wait 2 × backoff                                    │
pass 3  INSERT keys 1..100 again                              ▼ wait 1 × backoff
        another move → 23505 on key 73 …                pass 3  INSERT keys [15] ON CONFLICT DO NOTHING
        …                                                     key 2000 rewritten by the flush → 1 row
passes keep colliding on whichever row moved                  held off: [] → chunk lands, 100 rows
last; the chunk lands only when moves pause,
or the copy stops at CollisionAttempts              a move into keys 1..100 after pass 2 is the
                                                    applier's to write; pass 3 never reads the range

An exhausted chunk reads the same on both sides for the applier — in flight — but after this change the rows committed around the pending key stay in the shadow until the resume clear removes them, and Position.Pending names the key that never landed.

🤖 Generated with Amp (Claude Opus 4.6)