Skip to content

Enable container CPU metrics for API pods #111

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Alex R (alexr17) wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Enable container CPU metrics for API pods #111

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions braintrust/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -469,3 +469,18 @@ Example values files for different cloud providers and configurations are locate
- `examples/google-autopilot-cel/values.yaml`: GKE Autopilot deployment with CEL-friendly security settings.
- `examples/google-api-isolation-autoscaling/values.yaml`: Minimal example for API workload isolation and per-pool autoscaling on GKE (combine with an Autopilot or Standard values file).
- `examples/google-standard/values.yaml`: GKE Standard deployment.

## API container CPU metrics

Compatible API images collect container CPU time on Linux when `ENABLE_CONTAINER_CPU_METRICS=true`, which this chart sets for every API pool. The `api.container.cpu.time` counter measures CPU seconds across all processes in the API container using cgroup v1 or v2 accounting. Metrics use the existing OTEL exporter and require metrics telemetry to be enabled. Older API images ignore the setting.

To disable collection, set:

```yaml
api:
extraEnvVars:
- name: ENABLE_CONTAINER_CPU_METRICS
value: "false"
```

Remove this override to enable collection again. Existing `extraEnvVars` entries should be retained when adding the override.
1 change: 1 addition & 0 deletions braintrust/templates/api-configmap.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ data:
{{- end }}
BRAINSTORE_BACKFILL_DISABLE_HISTORICAL: {{ .Values.api.backfillDisableHistorical | quote }}
BRAINSTORE_BACKFILL_DISABLE_NONHISTORICAL: {{ .Values.api.backfillDisableNonhistorical | quote }}
ENABLE_CONTAINER_CPU_METRICS: "true"
CONTROL_PLANE_TELEMETRY: {{ .Values.global.controlPlaneTelemetry | quote }}
BRAINSTORE_INSERT_ROW_REFS: "true"
# Logs v2 table. Requires pg_partman extension (default enabled in our TF modules)
Expand Down
115 changes: 115 additions & 0 deletions braintrust/tests/api-container-cpu-metrics_test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
suite: API container CPU metrics
templates:
- api-configmap.yaml
- api-deployment.yaml
tests:
- it: enables CPU collection on aws
template: api-configmap.yaml
values:
- __fixtures__/base-values.yaml
set:
cloud: aws
asserts:
- equal:
path: data.ENABLE_CONTAINER_CPU_METRICS
value: "true"
- it: enables CPU collection on google
template: api-configmap.yaml
values:
- __fixtures__/base-values.yaml
set:
cloud: google
asserts:
- equal:
path: data.ENABLE_CONTAINER_CPU_METRICS
value: "true"
- it: enables CPU collection on azure
template: api-configmap.yaml
values:
- __fixtures__/base-values.yaml
set:
cloud: azure
asserts:
- equal:
path: data.ENABLE_CONTAINER_CPU_METRICS
value: "true"
- it: shares CPU configuration with the default pool with isolation false
template: api-deployment.yaml
values:
- __fixtures__/base-values.yaml
set:
api.workloadIsolation.enabled: false
documentIndex: 0
asserts:
- contains:
path: spec.template.spec.containers[0].envFrom
content:
configMapRef:
name: braintrust-api
- it: shares CPU configuration with the default pool with isolation true
template: api-deployment.yaml
values:
- __fixtures__/base-values.yaml
set:
api.workloadIsolation.enabled: true
documentIndex: 0
asserts:
- contains:
path: spec.template.spec.containers[0].envFrom
content:
configMapRef:
name: braintrust-api
- it: shares CPU configuration with the ingest pool with isolation true
template: api-deployment.yaml
values:
- __fixtures__/base-values.yaml
set:
api.workloadIsolation.enabled: true
documentIndex: 1
asserts:
- contains:
path: spec.template.spec.containers[0].envFrom
content:
configMapRef:
name: braintrust-api
- it: shares CPU configuration with the background pool with isolation true
template: api-deployment.yaml
values:
- __fixtures__/base-values.yaml
set:
api.workloadIsolation.enabled: true
documentIndex: 2
asserts:
- contains:
path: spec.template.spec.containers[0].envFrom
content:
configMapRef:
name: braintrust-api
- it: allows explicitly setting CPU collection to false
template: api-deployment.yaml
values:
- __fixtures__/base-values.yaml
set:
api.extraEnvVars:
- name: ENABLE_CONTAINER_CPU_METRICS
value: "false"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: ENABLE_CONTAINER_CPU_METRICS
value: "false"
- it: allows explicitly setting CPU collection to true
template: api-deployment.yaml
values:
- __fixtures__/base-values.yaml
set:
api.extraEnvVars:
- name: ENABLE_CONTAINER_CPU_METRICS
value: "true"
asserts:
- contains:
path: spec.template.spec.containers[0].env
content:
name: ENABLE_CONTAINER_CPU_METRICS
value: "true"
Loading