fix: clear metadata finalizers in namespace purge - #538
Open
JoseSzycho wants to merge 1 commit into
Open
JoseSzycho wants to merge 1 commit into
JoseSzycho wants to merge 1 commit into
Conversation
…e purge Project deletion was stuck because Phase D of the purger was trying to clear nso.Spec.Finalizers instead of nso.ObjectMeta.Finalizers. In Kubernetes, the finalizers that prevent resource deletion are stored in metadata.finalizers, not spec. This fix allows namespaces to be properly force-finalized during project deletion, preventing Phase E from timing out. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Contributor
Author
This was referenced Aug 6, 2026
scotwells
added a commit
that referenced
this pull request
Aug 6, 2026
Closes #750. ## Impact Deleting a project could strand resources that operators created for it. The project finished deleting while objects inside it still held finalizers, so those finalizers never ran. For network-services-operator, gateway configuration outlived the project. It kept propagating to the edge and serving traffic, and NSO held a finalizer it could never release, failing reconcile forever. A project now stays in `Terminating` until its control plane is empty, and reports what blocks it. Consumers finish their work against a control plane that still answers. A project with nothing stuck still deletes in seconds. ## What changed **Namespaces are no longer force-finalized.** Kubernetes guarantees a namespace outlives its contents, and consumers resolve state through it. Namespaces are deleted and left to the namespace controller each project control plane already runs. **The completion check covers `default`.** It previously accepted "no namespaces left". The API server refuses to delete `default`, so resources held there stayed invisible and the project completed on top of them. Completion now requires `default` to be empty. **The `ResourceCleanup` condition names the blocker:** ``` Waiting for project resources to be removed: configmaps "default/held-resource" (finalizers: test.miloapis.com/consumer-cleanup) ``` Two per-project metrics back it and clear on completion: `milo_resourcemanager_project_deletion_pending_seconds` and `milo_resourcemanager_project_deletion_blocking_resources`. **The project control plane is deleted after cleanup rather than before**, so it no longer disappears while a consumer is still finalizing against it. Deletion stays asynchronous, as #553 made it. A stuck project polls on its own and holds up nobody else. There is no escape hatch. A stuck project means a consumer bug, which should be visible and fixed rather than overridden. If operators need one later, make it an explicit gated admin action. ## Testing New e2e `project-deletion-blocked-resources` covers both places project resources live: held in `default`, and held in a project namespace. Neither project may be removed while its resource is held, and both must delete once the finalizers are released. Against unmodified Milo it fails as expected. The `default` case loses its project 45 seconds after deletion while the held ConfigMap remains. Verified end to end against network-services-operator on a two-cluster environment. The gateway finalizer runs while the namespace exists, every downstream object is removed, and the project deletes cleanly. While getting the new test green, project-creation turned out to share the cluster-scoped User `user-admin` with project-deletion. The two run concurrently, so one could delete it while the other waited for it to be ready — project-creation now uses its own user. Unit tests cover the completion check. ## Note on #538 #538 fixes the field the force-finalize path clears. This PR removes that path. Force-finalizing is inert today because the guard returns early on a successful get, so repairing it would arm the behaviour #750 describes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix project deletion stuck state caused by incorrect finalizer clearing in the project purger.
Project deletion was timing out in Phase E because Phase D was trying to clear
nso.Spec.Finalizersinstead ofnso.ObjectMeta.Finalizers. In Kubernetes, the finalizers that prevent resource deletion are stored inmetadata.finalizers, not in spec. This caused namespaces to remain in Terminating state indefinitely.Root Cause
The purger's force-finalize logic was modifying the wrong field, so namespaces never got their deletion-blocking finalizers cleared.
Fix
Changed line 198 from
nso.Spec.Finalizers = niltonso.ObjectMeta.Finalizers = nilso that the actual blocking finalizers are cleared during namespace force-finalization.Testing
🤖 Generated with Claude Code