[RLC] Case 3 — exclude relocation (dataChange=false) files from the append-conflict check

[sezruby]

Part of the row-level-concurrency umbrella (#3) — Case 3. Full design: #6. Opened upstream as delta-io#7331 (this fork PR is superseded by that one).

Problem

A concurrent transaction that only rearranges existing rows — most importantly OPTIMIZE (compaction / Z-ORDER) — commits its outputs as AddFiles with dataChange = false and removes its compacted inputs as RemoveFiles, also with dataChange = false. Because OPTIMIZE is not a blind append (isBlindAppend = false), the WinningCommitSummary classifies those adds as changedDataAddedFiles.

As a result an insert-only / append-only writer running concurrently fails spuriously:

1. ConflictChecker.checkForAddedFilesThatShouldHaveBeenReadByCurrentTxn treats the OPTIMIZE output as newly-arrived data → ConcurrentAppendException if the writer's read predicate overlaps.
2. If the writer registered actual readFiles (e.g. a whole-table read), checkForDeletedFilesAgainstCurrentTxnReadFiles sees OPTIMIZE removed a file it read → ConcurrentDeleteReadException.

Both fire even though OPTIMIZE introduced no new logical rows and deleted none — it only relocated existing rows across file boundaries. This is the exact symptom in delta-io#626 ("Prevent insert-only transactions from failing due to concurrent data preserving transactions"), the stale delta-io#1305, and the closed issue delta-io#326. Both prior PRs stalled on the reviewer's request (delta-io#1305) for a general safety argument rather than a targeted patch. This PR supplies that argument — now materially stronger than it could have been in 2022 — and closes both failure modes for an append-only writer.

The safety argument (now backed by a validated invariant)

A file committed with dataChange = false carries, by contract, only rows that already existed in the table at that version. It contributes zero new logical rows and deletes zero logical rows — it rearranges/compacts existing rows into new file boundaries.

Added-files check. Its purpose is to catch rows a loser should have read but didn't because a concurrent writer added them. A dataChange = false add can never hold such a row: every logical row in it was already present (in some other file) in the loser's read snapshot. So it is safe to drop from that check.

Removed-files-vs-read check (new in this PR). Its purpose is to catch a loser that read a file whose rows were concurrently deleted. A dataChange = false remove deleted no rows — the rows it dropped from that file still exist under a new file boundary. So an append-only writer that read the relocated file read nothing that was logically removed, and can safely reconcile. This exclusion is gated on the current transaction being append-only (it adds no RemoveFile and no AddFile carrying a deletion vector): a DML loser (DELETE/UPDATE/MERGE) reads a file precisely to rewrite it, so if that file was concurrently relocated it must still conflict — otherwise its own RemoveFile/DV would target a file that no longer exists and rows could be lost or resurrected.

The 2022 concern was essentially "could a commit mix data-changing and data-preserving files such that filtering by dataChange skips something that matters?" That mixed case can no longer exist: delta-io#6937 / delta-io#6969 ("Validate consistent dataChange across commit FileActions", merged upstream) enforce as a validated invariant that all FileActions in a single commit share one dataChange value (ConflictChecker.trackConsistentDataChange, fatal mode throws on violation). Therefore both .filter(_.dataChange) sites are provably whole-commit gates: either the entire winning commit was data-preserving (safe) or it changed data (checked exactly as before).

The change cannot mask a genuine conflict: a winner that adds real rows commits them dataChange = true (still fully checked), and a winner that genuinely deletes rows commits its RemoveFile dataChange = true (never excluded) — both covered by safety-floor tests.

(Databricks' row-level-concurrency contract documents that non-Z-Order OPTIMIZE "cannot conflict" with concurrent INSERT/UPDATE/DELETE/MERGE. This PR was implemented independently against that observable contract and the OSS ConflictChecker semantics.)

Scope (validated by tests)

Behind a single internal flag (default off), for an append-only current transaction:

• Added-files check — dataChange = false adds are dropped → no spurious ConcurrentAppendException.
• Removed-files-vs-read check — dataChange = false removes are dropped → no spurious ConcurrentDeleteReadException, for both a predicate-only reader and a whole-table (readFiles-populated) reader.

Deliberately not touched:

• A DML loser (writes a RemoveFile/DV) overlapping an OPTIMIZE remove → still ConcurrentDeleteReadException. Fully reconciling that is the harder row-level-concurrency problem (Case 4, [RLC] Case 4 — OPTIMIZE ⟂ DML reconciliation (compaction offset remap) #7).
• A genuine delete by the winner (dataChange = true remove) → still conflicts with a reader of that file.

Change

Internal flag spark.databricks.delta.conflictDetection.excludeNoDataChangeFiles.enabled (default off). Added-files site filters addedFilesToCheckForConflicts.filter(_.dataChange); removed-files-vs-read site filters winningCommitSummary.removedFiles.filter(_.dataChange), gated on the current txn being append-only (forall over its actions: no RemoveFile, no AddFile with a deletion vector).

Tests

OptimisticTransactionSuite:

• dataChange = false append (OPTIMIZE-style) vs concurrent partition read — flag on → reconcile, off → legacy ConcurrentAppendException.
• dataChange = true append still conflicts … (safety floor) — a genuine new-data append still conflicts with the flag on.
• full OPTIMIZE (removes + dataChange=false adds) vs insert-only partition reader reconciles — predicate-only reader, Prevent insert-only transactions from failing due to concurrent data preserving transactions delta-io/delta#626 shape.
• full OPTIMIZE (removes + dataChange=false adds) vs whole-table APPEND-ONLY reader reconciles — whole-table readFiles reader now reconciles too (the removed-files half of the fix).
• full OPTIMIZE (dataChange=false removes) vs concurrent DELETE loser still raises ConcurrentDeleteReadException — boundary: the append-only guard holds; a DML loser is not silenced.
• genuine delete (dataChange=true remove) vs append-only reader still raises ConcurrentDeleteReadException — safety floor for removes: real deletions are never excluded.

Prior art

• Prevent insert-only transactions from failing due to concurrent data preserving transactions delta-io/delta#626 — open PR, same intent (insert-only vs data-preserving).
• Fix ConflictChecker for READ-APPEND after OPTIMIZE delta-io/delta#1305 — earlier attempt; the general-safety concern raised there is answered above.
• No need to consider ConcurrentAppendException/ConcurrentDeleteReadException when check commit conflict with Compaction's commit Actions delta-io/delta#326 — original issue.
• [SPARK] Validate consistent dataChange across commit FileActions delta-io/delta#6937 / Validate consistent dataChange across commit FileActions (follow-up) delta-io/delta#6969 — the consistent-dataChange validation this argument builds on.

[sezruby]

Superseded — opened publicly as delta-io#7331 (branch unchanged). Closing this fork-internal review pass.