Skip to content

Move each kind's writes and versions onto the shared resource code #9490

Closed as not planned

Description

@bobbai00

Merged into #9489 when #9483 was reorganized into five parts.

Task Summary

Creating, renaming and deleting a resource, and making a version, are written once per kind, each with its own conventions. A workflow is created with POST /workflow/create and an environment with POST /pve/db; a dataset is deleted with DELETE /dataset/{did} and a workflow with POST /workflow/delete. A workflow version is made as a side effect of POST /workflow/persist, while datasets and models have POST .../version/create. And ModelResource.scala repeats 31 of the 32 endpoints of DatasetResource.scala.

This task moves each kind's writes and versions onto the shared code, and its rows into resource and resource_version, one kind at a time. For each kind:

  1. New endpoints. POST /{kind}, PATCH /{kind}/{rid}, DELETE /{kind}/{rid} and, for a versioned kind, POST /{kind}/{rid}/versions, served from the generated interfaces.
  2. A kind handler. One small class says what is specific to the kind: how its fields are validated, where its version content is stored, and which resources it references. The shared code never branches on the kind. Operations that only one kind has, such as uploading a file to a dataset or executing a workflow, are declared in the contract and implemented by that kind.
  3. Legacy routes as adapters. An old route such as POST /workflow/persist calls the same shared code as the new one (Xiao-zhen-Liu in Proposal: Unify Backend Resource Management On Texera #9395), so each write has one implementation and the frontend keeps working unchanged. In the same step, the agent service's calls for the kind move to the generated client.
  4. The rows move. The kind's rows move into resource and resource_version, and its old table names remain as views until their last reader moves. The first kind to move turns resource and resource_version into tables, in the way Give every resource one id and one common record #9487 settles.

Before

flowchart LR
  fe["Frontend"]
  dr["DatasetResource"]
  mr["ModelResource<br/>repeats 31 of 32 endpoints"]
  wr["WorkflowResource"]
  dt[("dataset, dataset_version,<br/>dataset_user_access")]
  mt[("model, model_version,<br/>model_user_access")]
  wt[("workflow, workflow_version,<br/>workflow_of_user")]
  fe -- "POST /dataset/create" --> dr
  fe -- "POST /model/create" --> mr
  fe -- "POST /workflow/persist" --> wr
  dr --> dt
  mr --> mt
  wr --> wt
Loading

After

flowchart LR
  fe["Frontend today"]
  nc["Generated client"]
  ad["Legacy route as an adapter<br/>POST /dataset/create"]
  ifc["Generated interfaces<br/>POST /datasets<br/>POST /models"]
  sh["Shared code"]
  h["Kind handlers<br/>validate fields, store content,<br/>name references"]
  t[("resource, resource_version,<br/>access")]
  fe --> ad
  nc --> ifc
  ad --> sh
  ifc --> sh
  sh --> h
  sh --> t
Loading

The kinds move from the ones with the fewest ties to the rest of Texera to the ones with the most, and each move becomes its own sub-issue when it starts:

flowchart LR
  a["Warehouses and environments<br/>no sharing, no versions"]
  b["Datasets and models<br/>already share their access code"]
  c["Computing units<br/>sharing, executions"]
  d["Workflows<br/>versions on save, executions,<br/>shared editing"]
  a --> b --> c --> d
Loading

Scope

Every write of the six kinds and of their versions, the switch of resource and resource_version from views to tables, and the executions that point to workflow versions, which move to (rid, vno) together with the workflows. Reads are #9489, lineage is #9491, and removing the adapters and views is #9492.

Discussion Points

  1. When a workflow version is made. Today it is made on every save that changes the workflow, and stored as a reverse JSON patch. I propose to keep that rule behind the new endpoint, and to discuss separately whether a version should be made only when what executes changes, as my Texera 2.0 prototype does.
  2. Dataset and model versions. Each has a name and a lakeFS commit. The name stays, because file paths in workflows use it, next to the vno from Give every resource one id and one common record #9487.
  3. Executions. Design of sharing and access control of resources (computing units, datasets, and workflows) #3749 decided to make an execution a weak entity of its workflow version, computing unit and user. The diagram in the parent issue draws an execution as an entity of its own that points to a workflow version, a computing unit and a warehouse. Either fits the shared tables, and this task should settle which.
  4. Behavior that differs between kinds. Moving a kind may reveal behavior that differs from the others. Each difference is settled by a test written before the move, so that the move itself changes no behavior.

Done when every kind is created, updated and deleted, and every versioned kind versioned, through the shared code, and today's tests pass without change.

Task Type

  • Refactor / Cleanup
  • DevOps / Deployment / CI
  • Testing / QA
  • Documentation
  • Performance
  • Other

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions