Skip to content

fix(observability): replace Langfuse's bundled MinIO with aws-compatible-storage - #74

Merged
itay1551 merged 1 commit into
rh-ai-quickstart:mainfrom
pzhao0:feat/aws-compatible-storage
Nov 3, 2026
Merged

itay1551 merged 1 commit into
rh-ai-quickstart:mainfrom
pzhao0:feat/aws-compatible-storage

Conversation

@pzhao0

@pzhao0 pzhao0 commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Replace the Bitnami MinIO that the upstream langfuse chart bundles with the shared aws-compatible-storage chart, leaving Langfuse's S3 behaviour otherwise unchanged.

charts/pca-ai-serving/charts/pca-observability/
├── Chart.yaml                                # + aws-compatible-storage 0.2.0; condition follows langfuse.enabled
├── Chart.lock                                # dependency lock
├── charts/aws-compatible-storage-0.2.0.tgz   # vendored subchart (repo convention)
├── values.yaml                               # langfuse.s3 → external store; MinIO-only keys replaced
├── templates/
│   ├── object-storage-secret.yaml            # new: dedicated, generated AWS_* credentials
│   └── langfuse-secret.yaml                  # drop now-dead s3-root-user / s3-root-password
└── README.md                                 # Persistence: MinIO ~10Gi → object storage

Langfuse keeps its S3 client; only the endpoint, credential source and bucket bootstrap move.

 langfuse.s3
-  deploy: true                         # bundled bitnami/minio (bitnamilegacy/minio, Docker Hub)
-  fullnameOverride: pca-langfuse-s3
-  auth: {rootUser: minio, rootPasswordSecretKey: s3-root-password, ...}
+  deploy: false                        # MinIO subchart off
+  bucket: langfuse
+  endpoint: http://pca-langfuse-s3:7480
+  forcePathStyle: true
+  accessKeyId/secretAccessKey → secretKeyRef: pca-langfuse-s3-credentials

+aws-compatible-storage:
+  fullnameOverride: pca-langfuse-s3    # Service/PVC names carried over from the MinIO era
+  s3: {existingSecret: pca-langfuse-s3-credentials}
+  route/auth: disabled                 # S3 API and UI stay cluster-internal
+  sampleFileUpload: {bucket: langfuse, urls: []}   # creates the bucket, uploads nothing

Verification

Before — the deployed namespace runs the MinIO subchart from Docker Hub:

pca-langfuse-s3-594d88bf7f-rhxl8   1/1   Running   bitnamilegacy/minio:2024.12.18-debian-12-r1

After — fresh namespace, helm install of the chart with Langfuse enabled:

deployment/pca-langfuse-s3          1/1     quay.io/rh-aiservices-bu/s4:0.3.2
svc/pca-langfuse-s3                 7480/TCP,5000/TCP      (no Route)
pvc/pca-langfuse-s3-data            Bound 10Gi gp3-csi
secret/pca-langfuse-s3-credentials  AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY   (only these)
job/pca-langfuse-s3-upload-sample-docs   Complete   → "Created bucket langfuse"
oc get all | grep -i minio → none

Langfuse resolves the new client config at runtime:

LANGFUSE_S3_EVENT_UPLOAD_ENDPOINT=http://pca-langfuse-s3:7480
LANGFUSE_S3_EVENT_UPLOAD_BUCKET=langfuse
LANGFUSE_S3_EVENT_UPLOAD_REGION=us-east-1
LANGFUSE_S3_EVENT_UPLOAD_FORCE_PATH_STYLE=true

End-to-end write (ingestion API → worker → S3):

POST /api/public/ingestion → 207
bucket contents: pca-llm/trace/<trace-id>/<event-id>.json   13,297 B
GET /api/public/traces      → returns that trace

Also checked: no S3 errors in the web/worker logs; helm upgrade leaves the generated Secret byte-identical and the store pod un-restarted; wrong credentials → SignatureDoesNotMatch (S3 auth is still enforced with auth.enabled: false, which only governs the UI).

Merge Danger

Blast Radius: Langfuse only.

  • New deployments only: the pca-langfuse-s3 Deployment selector changes, so an in-place upgrade from a MinIO install fails on that immutable field. Fresh namespaces are unaffected — note pca-ai-serving's Argo app runs with prune: false, so leftover MinIO objects are orphaned rather than deleted.
  • Rendering pca-ai-serving with Langfuse disabled is byte-identical; with Langfuse on, only the store objects and Langfuse's LANGFUSE_S3_* env change.
  • The bucket comes from the chart's own sampleFileUpload Job (plain Job, ttlSecondsAfterFinished: 86400), idempotent, re-runs when TTL prunes it under Argo selfHeal.
  • Full-stack smoke was not run (cluster DSC not Ready); the observability chart was exercised directly on OCP 4.21.

…ble-storage

The langfuse chart bundles Bitnami MinIO (bitnamilegacy/minio, Docker Hub) as
Langfuse's S3 backend. Switch to the shared ai-architecture-charts
aws-compatible-storage subchart; S3 behaviour is otherwise unchanged.

- Chart.yaml: add aws-compatible-storage 0.2.0. The condition falls back to
  langfuse.enabled, so the store exists only with Langfuse. Commit Chart.lock
  and the packaged chart (subcharts are vendored in this repo).
- values: langfuse.s3.deploy=false and point the client at
  pca-langfuse-s3:7480 with path-style addressing. The MinIO-only keys
  (fullnameOverride/auth/persistence) and the dead persistence.minioSize are
  replaced by the subchart block; no UI or S3 Route, so the API stays
  cluster-internal.
- Credentials: dedicated generated Secret pca-langfuse-s3-credentials
  (AWS_ACCESS_KEY_ID plus a lookup-stable randAlphaNum secret) so the store pod
  only sees the keys it needs. Langfuse and the bucket Job reference it; the
  now-dead s3-root-user/s3-root-password are dropped from
  pca-langfuse-credentials.
- Bucket: the subchart's sampleFileUpload Job creates `langfuse` (urls: []).
- README: Persistence no longer mentions MinIO / persistence.minioSize.
@mhdawson

mhdawson commented Nov 3, 2026

Copy link
Copy Markdown
Member

@pzhao0 I don't think -> aws-compatible-storage-0.2.0.tgz should be added to the repo?

secretKeyRef:
name: pca-langfuse-s3-credentials
key: AWS_SECRET_ACCESS_KEY
# Object storage for Langfuse, replacing the MinIO bundled with the langfuse chart.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there potentiallyu an update to the langfuse chart that would replace minio instead of this work around ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Langfuse chart is from https://github.com/langfuse/langfuse-k8s and is not maintained by us. It looks like they switched over to the legacy version of MinIO hosted by Bitnami Legacy:

Since MinIO is a minor component of this quickstart (just used by Langfuse for observability), we could continue with their solution for now to not add in complexity and I can close this PR. What do you think?

@pzhao0

pzhao0 commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

@pzhao0 I don't think -> aws-compatible-storage-0.2.0.tgz should be added to the repo?

Looks like the repo follows this pattern, langfuse-1.5.38.tgz is in the same charts/ directory even though it does run a helm dependency update in the install command.

@itay1551
itay1551 merged commit a08ccbd into rh-ai-quickstart:main Nov 3, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants