Skip to content

Storage: Standardize S3 paths #1170

Description

@kartpop

Is your feature request related to a problem?
Different modules are inconsistent in how they store objects in S3, which complicates data management and raises security concerns.

Describe the solution you'd like

  • Standardize all storage paths to use project.storage_path for documents, voice notes, eval datasets, batch data, and LLM call/chain artifacts.
  • Remove the per-module cleanup logic to facilitate project-reference-based deletion of data from S3.
Original issue

Context

Different modules store objects in S3 differently — documents use project.storage_path (a UUID-based path, good), while audio/TTS-STT paths embed org/project IDs directly.

Decision

Everything (documents, voice notes, eval datasets, batch data, LLM call/chain artifacts) must use project.storage_path.

Rationale

  • Security: UUID paths mean S3 access alone can't be correlated to an org/project without DB access.
  • Cleanup: enables deleting all of a project's data from S3 by project reference (DPDP-relevant), and removes per-module if/else cleanup logic.

Acceptance criteria

  • Documents, LLM call, LLM chain, and eval artifacts all resolve their S3 paths via project.storage_path.
  • Per-module cleanup if/else logic is removed in favor of project-reference-based deletion.

@vprashrex @Prajna1999 @AkhileshNegi

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions