Repository navigation
fix(observability): replace Langfuse's bundled MinIO with aws-compatible-storage - #74
Conversation
…ble-storage The langfuse chart bundles Bitnami MinIO (bitnamilegacy/minio, Docker Hub) as Langfuse's S3 backend. Switch to the shared ai-architecture-charts aws-compatible-storage subchart; S3 behaviour is otherwise unchanged. - Chart.yaml: add aws-compatible-storage 0.2.0. The condition falls back to langfuse.enabled, so the store exists only with Langfuse. Commit Chart.lock and the packaged chart (subcharts are vendored in this repo). - values: langfuse.s3.deploy=false and point the client at pca-langfuse-s3:7480 with path-style addressing. The MinIO-only keys (fullnameOverride/auth/persistence) and the dead persistence.minioSize are replaced by the subchart block; no UI or S3 Route, so the API stays cluster-internal. - Credentials: dedicated generated Secret pca-langfuse-s3-credentials (AWS_ACCESS_KEY_ID plus a lookup-stable randAlphaNum secret) so the store pod only sees the keys it needs. Langfuse and the bucket Job reference it; the now-dead s3-root-user/s3-root-password are dropped from pca-langfuse-credentials. - Bucket: the subchart's sampleFileUpload Job creates `langfuse` (urls: []). - README: Persistence no longer mentions MinIO / persistence.minioSize.
|
@pzhao0 I don't think -> aws-compatible-storage-0.2.0.tgz should be added to the repo? |
| secretKeyRef: | ||
| name: pca-langfuse-s3-credentials | ||
| key: AWS_SECRET_ACCESS_KEY | ||
| # Object storage for Langfuse, replacing the MinIO bundled with the langfuse chart. |
There was a problem hiding this comment.
Is there potentiallyu an update to the langfuse chart that would replace minio instead of this work around ?
There was a problem hiding this comment.
The Langfuse chart is from https://github.com/langfuse/langfuse-k8s and is not maintained by us. It looks like they switched over to the legacy version of MinIO hosted by Bitnami Legacy:
- chore: switch to bitnamilegacy registry for support after 28.08.2025 langfuse/langfuse-k8s#221
- fix: run the v1-to-v2 object-storage mirror with the v1 MinIO image langfuse/langfuse-k8s#460
Since MinIO is a minor component of this quickstart (just used by Langfuse for observability), we could continue with their solution for now to not add in complexity and I can close this PR. What do you think?
Looks like the repo follows this pattern, langfuse-1.5.38.tgz is in the same |
Summary
Replace the Bitnami MinIO that the upstream
langfusechart bundles with the sharedaws-compatible-storagechart, leaving Langfuse's S3 behaviour otherwise unchanged.Langfuse keeps its S3 client; only the endpoint, credential source and bucket bootstrap move.
Verification
Before — the deployed namespace runs the MinIO subchart from Docker Hub:
After — fresh namespace,
helm installof the chart with Langfuse enabled:Langfuse resolves the new client config at runtime:
End-to-end write (ingestion API → worker → S3):
Also checked: no S3 errors in the web/worker logs;
helm upgradeleaves the generated Secret byte-identical and the store pod un-restarted; wrong credentials →SignatureDoesNotMatch(S3 auth is still enforced withauth.enabled: false, which only governs the UI).Merge Danger
Blast Radius: Langfuse only.
pca-langfuse-s3Deployment selector changes, so an in-place upgrade from a MinIO install fails on that immutable field. Fresh namespaces are unaffected — notepca-ai-serving's Argo app runs withprune: false, so leftover MinIO objects are orphaned rather than deleted.pca-ai-servingwith Langfuse disabled is byte-identical; with Langfuse on, only the store objects and Langfuse'sLANGFUSE_S3_*env change.sampleFileUploadJob (plain Job,ttlSecondsAfterFinished: 86400), idempotent, re-runs when TTL prunes it under ArgoselfHeal.