Skip to content

Unify backend resource management around one schema, one API contract and one access check #9483

Open
0 of 5 issues completed

Description

@bobbai00

Task Summary

after-er-diagram

This implements #9395: every kind of resource (workflows, datasets, models, computing units, environments and warehouses) follows one definition, so creating, versioning and sharing a resource are written once instead of once per kind. The work has five parts:

  1. Schema (Adjust the database schema for unified resource management #9484). Add tables for resources, versions, access and lineage next to today's tables, and keep them in step with today's tables until each kind moves. Every deployment applies schema updates by itself.
  2. API definition (Define the resource service's API in one contract and generate code from it #9485). Describe the resource service's API in one contract, and generate the server interfaces, the TypeScript client and, later, the MCP tools from it.
  3. Authentication and access control (Put authentication and access control behind replaceable interfaces #9486). Put "who is calling" and "what may they do" behind two interfaces, with our own implementations first, so that Keycloak or OpenFGA can replace them later.
  4. Resource service (Introduce the unified resource service and move each kind onto it #9489). A new service that implements the API on shared code: reads for every kind first, then each kind's writes, one kind at a time, with the old routes calling it.
  5. Cleanup (Remove the legacy resource routes, tables and services #9492). Once the frontend and the agent service use the new API, remove the old routes, tables and per-kind code, and fold config-service and access-control-service into the resource service.

The order: #9484 and #9486 first, in parallel, then #9485 once the schema is agreed. #9489 needs all three, and #9492 comes last. Today's code keeps working at every step, because each kind has one source of truth at a time and the other side follows it: by triggers before the kind moves, and by read-only views after.

For background, see #9395 and #3749. Related: #5423 moves the Dashboard Service to Dropwizard 4.

Task Type

  • Refactor / Cleanup
  • DevOps / Deployment / CI
  • Testing / QA
  • Documentation
  • Performance
  • Other

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions