Repository navigation
Description
Merged into #9484 when #9483 was reorganized into five parts.
Task Summary
Each kind keeps its common parts in its own way. Every kind has its own id (wid, did, mid, cuid, veid, whid) and records its owner in its own place: the workflow_of_user table, an owner_uid column, or a uid column. Each versioned kind has its own version table: dataset_version and model_version have the same columns, while workflow_version stores reverse JSON patches. As #9395 counts, 17 of the 42 tables are per-kind copies, so no code can name "a resource" or "a version" without knowing its kind.
This task gives every resource a global id, rid, and every version a number within its resource, vno, and adds the two records that every kind shares: resource and resource_version. To change nothing that works today, both start as views over today's tables. They become tables when the first kind moves its rows into them, in #9490.
Before
Three of the six kinds show the pattern. Models follow datasets, and environments and warehouses follow computing units.
erDiagram
user ||--o{ workflow_of_user : "owns through"
workflow ||--o{ workflow_of_user : "is owned in"
workflow ||--o{ workflow_version : "has"
user ||--o{ dataset : "owns by owner_uid"
dataset ||--o{ dataset_version : "has"
user ||--o{ workflow_computing_unit : "owns by uid"
workflow {
int wid PK
string name
text content
}
workflow_version {
int vid PK
int wid FK
text content "reverse JSON patch"
}
dataset {
int did PK
int owner_uid FK
string name
}
dataset_version {
int dvid PK
int did FK
string name
string version_hash
}
workflow_computing_unit {
int cuid PK
int uid FK
string name
}
After
erDiagram
user ||--o{ resource : "creates"
resource ||--o{ resource_version : "has"
resource ||--o| workflow : "kind fields"
resource ||--o| computing_unit : "kind fields"
resource ||--o| warehouse : "kind fields"
resource_version ||--o| workflow_version : "content"
resource {
uuid rid PK
string kind
string name
string description
bool is_public
int created_by FK
json spec "fields of datasets, models, environments"
}
resource_version {
uuid rid PK
int vno PK
string name
int created_by FK
string content_ref "lakeFS commit of a dataset or model"
}
workflow {
uuid rid PK
text content
}
workflow_version {
uuid rid PK
int vno PK
text content
}
computing_unit {
uuid rid PK
string uri
}
warehouse {
uuid rid PK
string lakekeeper_warehouse_name
}
At first, resource and resource_version are views over today's tables, which gain the new keys:
flowchart TB
r[("resource<br/>a view")]
rv[("resource_version<br/>a view")]
subgraph kinds["Today's six tables, each with a new rid"]
w["workflow"]
d["dataset"]
m["model"]
c["workflow_computing_unit"]
e["virtual_environments"]
h["user_warehouse"]
w ~~~ c
d ~~~ e
m ~~~ h
end
subgraph versions["Today's three version tables, each with rid and vno"]
wv["workflow_version"]
dv["dataset_version"]
mv["model_version"]
end
r --> kinds
rv --> versions
Scope
- The keys. Every existing resource gets a
rid, and every existing version avno. Each kind keeps its old id until the legacy routes are gone (Remove the legacy resource routes, tables and services #9492), so links, and the result URIs stored with executions, keep working. - The two views.
resourceandresource_versionpresent every kind in the shape above. - The shared read code.
common/resourcereads any resource and any version of any kind through the views.
This task adds no endpoint (#9489) and no access rule (#9488), and it moves no rows (#9490).
Discussion Points
- What
ridis. The diagrams show a UUID, which follows Design of sharing and access control of resources (computing units, datasets, and workflows) #3749's request that keys not reveal how many objects a deployment holds. An integer from one sequence is the alternative, and is shorter in a URL. - Views first, or tables. Views change nothing that works today, but have to become tables when the first kind moves. Tables from the start, kept in step with today's tables until each kind moves, avoid that switch, but need triggers or double writes.
- Which kinds get a table. A kind's own fields go into
resource.specas JSON, checked against the contract's schema, unless SQL needs them for an index, a constraint or a foreign key. That gives tables to workflows, computing units and warehouses, and JSON to datasets, models and environments. - What a version stores.
resource_versionrecords who made a version and when, andcontent_refpoints to where the kind keeps the content: the workflow itself in PostgreSQL, or a lakeFS commit for a dataset or a model (Xiao-zhen-Liu in Proposal: Unify Backend Resource Management On Texera #9395). Replacing lakeFS later then changes no table. Identifying a version by its resource and a local number also keeps the decision recorded in Design of sharing and access control of resources (computing units, datasets, and workflows) #3749. - Names. A workflow names a dataset file by the owner's email and the dataset's name (
FileResolver), so(kind, created_by, name)stays unique for datasets and models.
Done when every resource and every version has a rid and a vno, and common/resource reads any kind through the two views.
Task Type
- Refactor / Cleanup
- DevOps / Deployment / CI
- Testing / QA
- Documentation
- Performance
- Other
Activity