Skip to content

Record lineage between resource versions #9491

Closed as not planned
Closed as not planned

Description

@bobbai00

Merged into #9484 (the lineage table) and #9489 (recording lineage) when #9483 was reorganized into five parts.

Task Summary

Texera records no lineage between resources. A workflow names the dataset files it reads as path strings inside its content, resolved only when it executes. An execution points to the workflow version it executed, the computing unit it ran on and the warehouse it stored in, but not to the dataset and model versions it read. So "which workflows read this dataset version?" and "where did this result come from?" have no answer, and a dataset or a model that workflows still use can be deleted without warning.

This task adds one lineage table between versions, with two relations:

  • uses: a version needs another version, such as a workflow version that reads a dataset version. It is written when the version is saved.
  • derived_from: a version was made from another, such as a dataset version that holds an execution's exported results, or a cloned workflow. It is written when the new version is made.

Clients never write lineage. The shared code of #9490 writes it, and every version answers GET /{kind}/{rid}/versions/{vno}/lineage in both directions.

Before

flowchart LR
  ex["workflow_executions"]
  wv["workflow_version"]
  cu["workflow_computing_unit"]
  wh["user_warehouse"]
  dv["dataset_version"]
  ex -- "vid" --> wv
  ex -- "cuid" --> cu
  ex -- "whid" --> wh
  wv -. "file paths as strings<br/>inside the workflow, no key" .-> dv
Loading

After

erDiagram
  resource_version ||--o{ lineage : "is the version"
  resource_version ||--o{ lineage : "is the source"
  resource_version {
    uuid rid PK
    int vno PK
  }
  lineage {
    uuid rid PK
    int vno PK
    uuid source_rid PK
    int source_vno PK
    string relation PK "uses or derived_from"
  }
Loading

For example, a forecast workflow that reads a dataset and a model, and whose execution's results are exported as a new dataset, is stored as two uses rows and one derived_from row. The arrows show how the data flows:

flowchart LR
  d["Dataset sales<br/>version 2"]
  m["Model churn<br/>version 1"]
  w["Workflow forecast<br/>version 3"]
  o["Dataset forecast<br/>version 1"]
  d -- "used by" --> w
  m -- "used by" --> w
  w -- "an execution exported" --> o
Loading

Scope

The lineage table, its writes from the shared code when versions are saved, cloned or made from an execution, and the lineage endpoint. Lineage inside a workflow, between operators and ports, is not part of this task, and reading an execution's results is #8995.

Discussion Points

  1. References and lineage in one table. zyratlo asked in Proposal: Unify Backend Resource Management On Texera #9395 where "this workflow uses this dataset version" would live. Keeping uses beside derived_from answers both questions with one query; the alternative is that each kind keeps its own references.
  2. Deleting a resource whose versions others use. It can be refused, warned about, or allowed with its lineage rows kept.
  3. Executions as nodes. An execution could be a node of its own, between the versions it read and the versions it wrote, instead of being folded into derived_from.
  4. Existing versions. uses can be back-filled from the file paths in existing workflow versions, or recorded only from now on.

Done when saving a workflow version records the dataset and model versions it uses, a cloned workflow and a dataset version made from exported results record where they came from, and the lineage endpoint answers both directions for any version.

Task Type

  • Refactor / Cleanup
  • DevOps / Deployment / CI
  • Testing / QA
  • Documentation
  • Performance
  • Other

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions