Skip to content

Give every resource one id and one common record #9487

Closed as not planned
Closed as not planned

Description

@bobbai00

Merged into #9484 when #9483 was reorganized into five parts.

Task Summary

Each kind keeps its common parts in its own way. Every kind has its own id (wid, did, mid, cuid, veid, whid) and records its owner in its own place: the workflow_of_user table, an owner_uid column, or a uid column. Each versioned kind has its own version table: dataset_version and model_version have the same columns, while workflow_version stores reverse JSON patches. As #9395 counts, 17 of the 42 tables are per-kind copies, so no code can name "a resource" or "a version" without knowing its kind.

This task gives every resource a global id, rid, and every version a number within its resource, vno, and adds the two records that every kind shares: resource and resource_version. To change nothing that works today, both start as views over today's tables. They become tables when the first kind moves its rows into them, in #9490.

Before

Three of the six kinds show the pattern. Models follow datasets, and environments and warehouses follow computing units.

erDiagram
  user ||--o{ workflow_of_user : "owns through"
  workflow ||--o{ workflow_of_user : "is owned in"
  workflow ||--o{ workflow_version : "has"
  user ||--o{ dataset : "owns by owner_uid"
  dataset ||--o{ dataset_version : "has"
  user ||--o{ workflow_computing_unit : "owns by uid"
  workflow {
    int wid PK
    string name
    text content
  }
  workflow_version {
    int vid PK
    int wid FK
    text content "reverse JSON patch"
  }
  dataset {
    int did PK
    int owner_uid FK
    string name
  }
  dataset_version {
    int dvid PK
    int did FK
    string name
    string version_hash
  }
  workflow_computing_unit {
    int cuid PK
    int uid FK
    string name
  }
Loading

After

erDiagram
  user ||--o{ resource : "creates"
  resource ||--o{ resource_version : "has"
  resource ||--o| workflow : "kind fields"
  resource ||--o| computing_unit : "kind fields"
  resource ||--o| warehouse : "kind fields"
  resource_version ||--o| workflow_version : "content"
  resource {
    uuid rid PK
    string kind
    string name
    string description
    bool is_public
    int created_by FK
    json spec "fields of datasets, models, environments"
  }
  resource_version {
    uuid rid PK
    int vno PK
    string name
    int created_by FK
    string content_ref "lakeFS commit of a dataset or model"
  }
  workflow {
    uuid rid PK
    text content
  }
  workflow_version {
    uuid rid PK
    int vno PK
    text content
  }
  computing_unit {
    uuid rid PK
    string uri
  }
  warehouse {
    uuid rid PK
    string lakekeeper_warehouse_name
  }
Loading

At first, resource and resource_version are views over today's tables, which gain the new keys:

flowchart TB
  r[("resource<br/>a view")]
  rv[("resource_version<br/>a view")]
  subgraph kinds["Today's six tables, each with a new rid"]
    w["workflow"]
    d["dataset"]
    m["model"]
    c["workflow_computing_unit"]
    e["virtual_environments"]
    h["user_warehouse"]
    w ~~~ c
    d ~~~ e
    m ~~~ h
  end
  subgraph versions["Today's three version tables, each with rid and vno"]
    wv["workflow_version"]
    dv["dataset_version"]
    mv["model_version"]
  end
  r --> kinds
  rv --> versions
Loading

Scope

  1. The keys. Every existing resource gets a rid, and every existing version a vno. Each kind keeps its old id until the legacy routes are gone (Remove the legacy resource routes, tables and services #9492), so links, and the result URIs stored with executions, keep working.
  2. The two views. resource and resource_version present every kind in the shape above.
  3. The shared read code. common/resource reads any resource and any version of any kind through the views.

This task adds no endpoint (#9489) and no access rule (#9488), and it moves no rows (#9490).

Discussion Points

  1. What rid is. The diagrams show a UUID, which follows Design of sharing and access control of resources (computing units, datasets, and workflows) #3749's request that keys not reveal how many objects a deployment holds. An integer from one sequence is the alternative, and is shorter in a URL.
  2. Views first, or tables. Views change nothing that works today, but have to become tables when the first kind moves. Tables from the start, kept in step with today's tables until each kind moves, avoid that switch, but need triggers or double writes.
  3. Which kinds get a table. A kind's own fields go into resource.spec as JSON, checked against the contract's schema, unless SQL needs them for an index, a constraint or a foreign key. That gives tables to workflows, computing units and warehouses, and JSON to datasets, models and environments.
  4. What a version stores. resource_version records who made a version and when, and content_ref points to where the kind keeps the content: the workflow itself in PostgreSQL, or a lakeFS commit for a dataset or a model (Xiao-zhen-Liu in Proposal: Unify Backend Resource Management On Texera #9395). Replacing lakeFS later then changes no table. Identifying a version by its resource and a local number also keeps the decision recorded in Design of sharing and access control of resources (computing units, datasets, and workflows) #3749.
  5. Names. A workflow names a dataset file by the owner's email and the dataset's name (FileResolver), so (kind, created_by, name) stays unique for datasets and models.

Done when every resource and every version has a rid and a vno, and common/resource reads any kind through the two views.

Task Type

  • Refactor / Cleanup
  • DevOps / Deployment / CI
  • Testing / QA
  • Documentation
  • Performance
  • Other

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions