Skip to content

fix(storage): stop reader lease release deadlocks from failing MySQL runs - #268

Merged
LouisHaftmann merged 1 commit into
devfrom
fix/reader-lease-release-deadlock
Sep 13, 2026
Merged

LouisHaftmann merged 1 commit into
devfrom
fix/reader-lease-release-deadlock

Conversation

@LouisHaftmann

Copy link
Copy Markdown
Collaborator

📝 Summary

The gcs, mysql unit tests failed (run) with every test passing. The cause was an unhandled ER_LOCK_DEADLOCK rejection. When a download stream ends, we release its Storage Reader Lease without awaiting it. If the storage location gets deleted at the same time, its cascade delete locks the same lease rows in the opposite order. MySQL then kills the release, and nothing catches the error.

The release now retries on lock conflicts with the existing retryOnLockConflict helper. If it still fails, we log a warning instead of leaving an unhandled rejection. That's safe because an unreleased lease expires anyway.

📦 Changes

  • releaseReaderLease retries on deadlocks and lock wait timeouts
  • The release at the end of a download stream logs failures instead of dropping them

✅ Verification

  • Race repro (throwaway test, not committed): 300 rounds, each creating a location with 4 reader leases and then releasing them all while deleting the location in parallel. On dev with MySQL: 5-6 ER_LOCK_DEADLOCKs per run, always on the release. With this fix: 0.
  • VITEST_DB_DRIVER=mysql VITEST_STORAGE_DRIVER=gcs vitest run tests/cleanup-lifecycle.test.ts tests/storage-lifecycle.test.ts: 12 passed, 1 skipped
  • tsc --noEmit, eslint, prettier --check on changed files: clean

⚠️ Needs attention

  • I didn't commit the repro test because it's probabilistic and takes ~25s on MySQL. retryOnLockConflict already has unit tests.

🤖 Generated with Claude Code

Releasing a Storage Reader Lease raced the cascade delete of its storage
location, which locks the same rows in the opposite order. MySQL picked the
release as the deadlock victim, and since it was fire-and-forget the error
surfaced as an unhandled rejection that failed CI.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@LouisHaftmann
LouisHaftmann merged commit b0e6147 into dev Sep 13, 2026
24 checks passed
@LouisHaftmann
LouisHaftmann deleted the fix/reader-lease-release-deadlock branch September 13, 2026 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant