Skip to content

fix(checkpoint): allow failed checkpoint cleanup - #3199

Open
spichen wants to merge 1 commit into
kagent-dev:mainfrom
spichen:fix/failed-checkpoint-deletion
Open

spichen wants to merge 1 commit into
kagent-dev:mainfrom
spichen:fix/failed-checkpoint-deletion

Conversation

@spichen

@spichen spichen commented Oct 14, 2026

Copy link
Copy Markdown

A checkpoint whose snapshot creation fails cannot be deleted: the store only admits READY and DELETING records, so the owner receives NotFound and the failed checkpoint keeps its runtime revision pinned.

Allow FAILED checkpoints to enter the existing deletion workflow. When Substrate confirms the tag is absent, remove the checkpoint record without another delete-by-name request. If a tag exists, require a retained UID and matching identity before deleting it. Backend failures preserve the record for retry, and ownership and fork-reference checks remain in place.

Refs #2746. This addresses failed-checkpoint deletion only; automatic recovery of abandoned CREATING/DELETING operations remains follow-up work.

Validation

  • New database regression fails on unchanged main with begin delete Session checkpoint: record not found and passes with the fix.
  • New service regressions fail on unchanged main and pass with the fix.
  • go test -race ./core/internal/service/checkpoint -count=1 passes.
  • go test -race ./core/internal/database -skip '^TestSessionExpiration' -count=1 passes, including inline SQL preparation and runtime-revision retention coverage.
  • go test -race ./core/pkg/migrations -count=1 passes.
  • Focused golangci-lint v2.13.0 reports zero issues for the database and checkpoint service packages; formatting and git diff --check pass.

The unfiltered database suite has failures in five session-expiration tests. The same failures were reproduced with the production code restored to the base main revision. Full repository lint was blocked during tool setup by local Go checksum-cache permissions; live Substrate E2E validation has not been run.

Tests were run against base revision f2d7cc4. The change also documents failed-checkpoint cleanup semantics.

AI assistance: Codex helped implement and validate this contribution.

Signed-off-by: Salah <nh.salah@gmail.com>
@github-actions github-actions Bot added the bug Something isn't working label Oct 14, 2026
@spichen
spichen marked this pull request as ready for review October 14, 2026 08:55
@spichen
spichen requested a review from a team as a code owner October 14, 2026 08:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant