Repository navigation
Pattern to replace soft-delete #111
Description
Activity
collecting a list of places to remove references:
- Clean up AIP-132: List #108 (comment)
- 135#soft-delete
The background
As of 2025-09-13, the aeps have a pattern in which one can "soft delete" a
resource - where it not actually removed from a collection, but is actually
flagged as deleted. This allows the resource to be recoverable later.The soft delete pattern, I believe, exists to serve the purpose of situations
where recovery is desired, such as most situations where persistent data is
stored:- A database, where recreating the resource does not restore any valuble data
stored. - A filesystem, where a file may need to be recovered after the fact.
The implementation of soft delete today
The current implementation of AEPs' soft delete boils down to the following:
- a new custom method,
undelete, on the resource. - an added field
show_deletedto theListstandard method, where resources
are hidden unless specified. - an added field
show_deletedto theGetstandard method, where the resource will
return 410 without it.
There are a few other details (such as the usage of expiry_date) that are
additional, but do not force any additional fields on the standard methods.The problems: soft delete is not compatible with declarative clients
Declarative clients expect consistent resource-oriented operations on each
resource - for simple CRUD on a resource, declarative clients are able to easily
map each operation to the resource lifecycle.Loadinggraph LR exists not_exists exists -- "update" --> exists not_exists -- "create" --> exists exists -- "delete" --> not_exists
For soft delete, where would that fit in? Effectively it creates another state - soft deleted.
Loadinggraph LR exists soft_deleted not_exists not_exists -- "create" --> exists exists -- "update" --> exists soft_delete -- "undelete" --> exists soft_delete -- "expiry hit" --> not_exists
But this may also result in weird cases like:
A resource "create" throwing an error because the resource is soft-deleted, and
therefore exists. But a get would return a 404.Proposed alternative
Since the primary use case is about backing up and enabling restoration of data,
the proposal would be to replace these augmentations on the primary resource, to
adding a second resource for "backups" or "snapshots".The pattern would look something like:
/database/foo /database-snapshots/foo/snapshots/latestThe database-snapshot could be support a custom method,
restore, that would be
similar to undelete and bring an old resource back.This would be similar to the resource revision pattern -
where a design pattern is introduced by adding a new resource, not by augmenting
standard methods.- A database, where recreating the resource does not restore any valuble data
We discussed on the call 3 options:
- Some sort of
statefield in the resource itself to mark it as soft deleted. - A shadow collection of some nature (such as snapshots as mentioned above)
- Using the pattern of resource revisions.
These all have their pros and cons.
After the call, I ran some of this past ChatGPT and it came to similar conclusions that we had, but it argued a bit harder for the first option as the most straightforward, and citing how Kubernetes resources have
metadata.deletionTimestamp, that mark something as in the process of being deleted.Adapting this idea to AEP patterns, AEP-148 already has
delete_timeandpurge_timefields defined.This still leaves scenarios that need addressing:
- How to un-delete something - probably just
applyit with empty delete_time? - What should the behavior be when re-creating something that has been soft deleted? Should it be allowed? I think not as it does not respect the soft delete. It should be forcibly deleted first. You could use
applyhowever to undelete it and update it. - How to force deletion? Possibly query param for
force_delete=true? - How to list with or without soft deleted items? Which is the default?
- Whether or not soft-delete is the default action for a collection, or it's opt-in at delete time? I think I favor it being set only at the collection level (opt-out of soft delete, or take the default behavior for the collection)
- How to document whether a collection implements soft delete by default?
The other options may still be worth considering, I was just thinking through this option a bit further.
- Some sort of
Discussion:
-
we can adjust the existing pattern where create would overwrite the existing resource.
-
@kindermoumoute feels that the soft deleted resource pattern is better.
-
@rambleraptor says:
- if you have multiple backups, use resource revisions.
use cases:
- list soft-deleted resources.
- restore a soft-deleted resource
- restore from a backup to a new resource.
-
Answering my own questions from our discussion:
- We could use the :undelete custom method, but I selfishly like the
applyoption above because AWS API Gateway is still challenging for custom methods - Re-creating should overwrite a soft-deleted resource, and that soft-deleted resource should be deleted/overwritten and no longer available for restore.
- We did not speak to forced deletion, this should probably be addressed.
- Listing without soft-deleted should be the default. A query parameter could be added for show_deleted or something to that effect.
5 and 6. We did not address how to document collection delete default behavior, or opting-in or out, but this should be addressed as well.
Workflow considerations
Create
- If soft delete exists, it would be overwritten.
- The backend system may need to implement a delete & create behind the scenes, depending on how the resource is implemented.
Update/Apply
- These should 404 by default if the resource is soft deleted.
- Should you be able to pass in a query parameter, or use a custom method to update a soft deleted thing?
- If we use a query poaram to allow manipulation of soft deleted resource, is that the path to undelete via removing the delete_time and purge_time fields?
Get
- Should 404 by default for resources that are soft deleted.
- Should 200 and return resource when query param is added (something like
?show_deleted=true - One thing I don't like about this is that you are adding a query parameter, which is typically used for filtering results of a collection, but now it exists on a single resource. Maybe it should be a custom method instead.
List
- By default, soft deleted resources are not listed
- Query parameter
?show_deleted=truemay show them, or possibly this gets included in the CEL filter? - Would it be desirable to have a list of ONLY deleted items? A mixed list? Both options?
Delete
- Do we still return a 204 no-content result here, or do we return the resource with additional fields for
delete_timeandpurge_time? Changing 204 to 200 might be a breaking change for clients.
- We could use the :undelete custom method, but I selfishly like the
restore from a backup to a new resource.
In our approach a
DatabaseBackupis a resource of its own with its own lifecycle. Thus the restore action is considered as any custom method:rpc RestoreDatabaseBackup(RestoreDatabaseBackupRequest) returns (DatabaseBackup)
This backup can be restored on any non-deleted database.
My preference for infrastructure operation is to use a custom method, weither isCancelDeletionduring the deletion grace period, orRestoreBackup. Can it be generalized to any asynchronously deleted resource? Maybe 🤔restore a soft-deleted resource
I feel I would only implement this kind of method for a resource that is synchronously deleted, such as an IAM policy. And in that case, it would keep the same ID and get/list the deleted resources with a
show_deleted=trueparameter. Also should thedeletednature of the resource be displayed in the resource full path somehow?If we use a query poaram to allow manipulation of soft deleted resource, is that the path to undelete via removing the delete_time and purge_time fields?
Do we still return a 204 no-content result here, or do we return the resource with additional fields for delete_time and purge_time?Update and Delete should just return 404 IMO.
Update and Delete should just return 404 IMO.
Sorry, for this I was talking about the create and then [soft] delete pattern. If the delete operation produces a soft-deleted resource, do we return that information in the delete response? This is different from a delete in a collection that does not offer soft-delete. I think we could return the normal no content response, which is maybe not as user-intuitive, but is more machine consistent with normal operations.
Reacted by Olivier CanoWe discussed this offline today in the sync. the general consensus was that we make the change to soft-delete as a sibling collection. Rationale:
- removing it now allows us to remove a bunch of if/else casing on soft-deleted resources (all of the edge cases you raised @thegagne).
- we can always re-introduce this pattern later.
Thoughts? @thegagne
I'll come back to this next week when I have more time to consider it.
I agree that the if-then logic is problematic. I do think that there is need to support some pattern of "disabled but not deleted", which is probably different than soft delete.
A simple
disabled: truesort of field could work here. They would show up in lists, but could be filtered out (but not by default).I would want to better understand the mechanics of the sibling collection for soft delete.
How does it get backed up?
How does it get restored?Are those both custom methods?
- added a commit that references this issue
on Oct 18, 2025 - added 5 commits that reference this issue
on Oct 18, 2025 In #365, there was additional discussion or the viability of the pattern. @odsod brings up valid points that the cost to implement my proposed pattern is quite a bit different (requires two different collections), and perhaps implements a sort of "snapshot" resource collection than a soft-deleted resource, whose main intention is primarily around having a resource that exists until some expiry date rather than a snapshot that should be recovered.
In addition, most who have joined AEP live meetings have raised valid examples of soft deleted resources in their APIs (e.g. cryptographic keys in Azure), and therefore the omissions of this pattern might invalidate those resources.
As such, combined with the primary motivation here to eliminate the poor UX with declarative clients which may need to overwrite the resource, I'm proposing #371 as a more surgical fix to ensure that a create behaves like any other resource (specifically allowing the recreation of one), even when using the soft delete pattern.
Overwriting a resource accidentally isn't an ideal behavior. Terraform (especially) will create resources without user intervention, which can cause unintended data loss.
I'd like to propose that a POST call on a soft-deleted resource should fail unless a "force" flag is included in the POST request. The flag is a no-op if no soft-deleted resource exists.
Clients can then choose / not to pass along the force flag.
- added a commit that references this issue
on Nov 1, 2025 - added a commit that references this issue
on Nov 9, 2025 - added a commit that references this issue
on Jul 25, 2026
Metadata
Metadata
Assignees
Type
Projects
- Status
@toumorokoshi mentioned having an idea for a design pattern that could serve as an alternative to soft-delete, letting us further standardize
Delete. He kindly offered to draft an AEP describing that pattern.