Repository navigation
Description
Merged into #9489 when #9483 was reorganized into five parts.
Task Summary
Creating, renaming and deleting a resource, and making a version, are written once per kind, each with its own conventions. A workflow is created with POST /workflow/create and an environment with POST /pve/db; a dataset is deleted with DELETE /dataset/{did} and a workflow with POST /workflow/delete. A workflow version is made as a side effect of POST /workflow/persist, while datasets and models have POST .../version/create. And ModelResource.scala repeats 31 of the 32 endpoints of DatasetResource.scala.
This task moves each kind's writes and versions onto the shared code, and its rows into resource and resource_version, one kind at a time. For each kind:
- New endpoints.
POST /{kind},PATCH /{kind}/{rid},DELETE /{kind}/{rid}and, for a versioned kind,POST /{kind}/{rid}/versions, served from the generated interfaces. - A kind handler. One small class says what is specific to the kind: how its fields are validated, where its version content is stored, and which resources it references. The shared code never branches on the kind. Operations that only one kind has, such as uploading a file to a dataset or executing a workflow, are declared in the contract and implemented by that kind.
- Legacy routes as adapters. An old route such as
POST /workflow/persistcalls the same shared code as the new one (Xiao-zhen-Liu in Proposal: Unify Backend Resource Management On Texera #9395), so each write has one implementation and the frontend keeps working unchanged. In the same step, the agent service's calls for the kind move to the generated client. - The rows move. The kind's rows move into
resourceandresource_version, and its old table names remain as views until their last reader moves. The first kind to move turnsresourceandresource_versioninto tables, in the way Give every resource one id and one common record #9487 settles.
Before
flowchart LR
fe["Frontend"]
dr["DatasetResource"]
mr["ModelResource<br/>repeats 31 of 32 endpoints"]
wr["WorkflowResource"]
dt[("dataset, dataset_version,<br/>dataset_user_access")]
mt[("model, model_version,<br/>model_user_access")]
wt[("workflow, workflow_version,<br/>workflow_of_user")]
fe -- "POST /dataset/create" --> dr
fe -- "POST /model/create" --> mr
fe -- "POST /workflow/persist" --> wr
dr --> dt
mr --> mt
wr --> wt
After
flowchart LR
fe["Frontend today"]
nc["Generated client"]
ad["Legacy route as an adapter<br/>POST /dataset/create"]
ifc["Generated interfaces<br/>POST /datasets<br/>POST /models"]
sh["Shared code"]
h["Kind handlers<br/>validate fields, store content,<br/>name references"]
t[("resource, resource_version,<br/>access")]
fe --> ad
nc --> ifc
ad --> sh
ifc --> sh
sh --> h
sh --> t
The kinds move from the ones with the fewest ties to the rest of Texera to the ones with the most, and each move becomes its own sub-issue when it starts:
flowchart LR
a["Warehouses and environments<br/>no sharing, no versions"]
b["Datasets and models<br/>already share their access code"]
c["Computing units<br/>sharing, executions"]
d["Workflows<br/>versions on save, executions,<br/>shared editing"]
a --> b --> c --> d
Scope
Every write of the six kinds and of their versions, the switch of resource and resource_version from views to tables, and the executions that point to workflow versions, which move to (rid, vno) together with the workflows. Reads are #9489, lineage is #9491, and removing the adapters and views is #9492.
Discussion Points
- When a workflow version is made. Today it is made on every save that changes the workflow, and stored as a reverse JSON patch. I propose to keep that rule behind the new endpoint, and to discuss separately whether a version should be made only when what executes changes, as my Texera 2.0 prototype does.
- Dataset and model versions. Each has a name and a lakeFS commit. The name stays, because file paths in workflows use it, next to the
vnofrom Give every resource one id and one common record #9487. - Executions. Design of sharing and access control of resources (computing units, datasets, and workflows) #3749 decided to make an execution a weak entity of its workflow version, computing unit and user. The diagram in the parent issue draws an execution as an entity of its own that points to a workflow version, a computing unit and a warehouse. Either fits the shared tables, and this task should settle which.
- Behavior that differs between kinds. Moving a kind may reveal behavior that differs from the others. Each difference is settled by a test written before the move, so that the move itself changes no behavior.
Done when every kind is created, updated and deleted, and every versioned kind versioned, through the shared code, and today's tests pass without change.
Task Type
- Refactor / Cleanup
- DevOps / Deployment / CI
- Testing / QA
- Documentation
- Performance
- Other
Activity