feat: OTEL traces (OTLP push) + mesh-aware request/trace ids

Makefile

@@ -49,6 +49,12 @@ update:
 	go get github.com/swaggo/gin-swagger@latest
 	go get github.com/swaggo/swag@latest
 	go get go.uber.org/zap@latest
+	go get go.opentelemetry.io/otel@latest
+	go get go.opentelemetry.io/otel/sdk@latest
+	go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp@latest
+	go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc@latest
+	go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin@latest
+	go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp@latest
 	go get golang.org/x/net@latest
 	go get golang.org/x/crypto@latest
 	go get golang.org/x/sys@latest

README.md

@@ -378,7 +378,22 @@ curl -sS localhost:8080/version | jq
 # {"version":"...","gitHash":"...","hostname":"..."}
 ```
 
-### 12. From inside the cluster (with cluster-utils shell)
+### 12. Traces + Istio-style request ids
+
+```bash
+# simulate gateway/mesh headers
+curl -sS -D- \
+  -H 'X-Request-Id: istio-style-id-001' \
+  -H 'traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01' \
+  localhost:8080/debug -o /dev/null | grep -iE 'x-trace-id|x-request-id'
+
+# with OTEL enabled, X-Trace-Id is the Tempo id; X-Request-Id echoes the mesh id
+# logs: {"msg":"Request","trace_id":"...","request_id":"istio-style-id-001",...}
+```
+
+See **Observability** for push vs scrape and full header table.
+
+### 13. From inside the cluster (with cluster-utils shell)
 
 ```bash
 # port-forward
@@ -472,7 +487,8 @@ curl -sS -X PUT -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/
 | `GET /ping` | `PONG` (not a kube probe) |
 | `GET /headers` | request headers |
 | `GET /debug` | hostname / ip / headers / uri |
-| `GET /metrics` | prometheus (OpenMetrics): request count/latency/in-flight + Go/process |
+| `GET /metrics` | prometheus **scrape** (OpenMetrics): request count/latency/in-flight + Go/process |
+| OTEL traces | **push** OTLP to Alloy/collector (not scrape) — see Observability |
 | `GET /status/:code` | respond with that http status (100-599) |
 | `GET /delay/:seconds` | sleep then 200 (cap `MAX_DELAY_SECONDS`, default 120) |
 | `ANY /echo` | bounce method / query / headers / body |
@@ -490,6 +506,134 @@ curl -sS -X PUT -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/
 
 ---
 
+## Observability (metrics vs traces)
+
+Two different pipelines — don't mix them up:
+
+| Signal | How it leaves the app | Endpoint / protocol | Typical sink |
+|--------|----------------------|---------------------|--------------|
+| **Metrics** | **Scrape** (pull) | `GET /metrics` Prometheus/OpenMetrics | Alloy `prometheus.scrape` → Mimir/Prometheus |
+| **Traces** | **Push** | **OTLP** http/protobuf (default) or grpc | Alloy OTLP receiver → **Tempo** |
+| **Logs** | stdout JSON (zap) | not OTLP yet | Alloy/loki.source.kubernetes → Loki |
+
+Traces are **not** scraped. The app **exports** spans to a collector. Grafana Alloy is the usual middle hop: receive OTLP → forward to Tempo.
+
+### How end-to-end tracing works (Istio / meshes)
+
+Normal path in 2024–26 stacks:
+
+1. **Edge / sidecar (Envoy, Istio, Linkerd)** accepts the request and either  
+   - continues an existing **W3C `traceparent`**, or  
+   - creates/propagates **B3** (`x-b3-traceid`, …) if the mesh is still on Zipkin-style config  
+2. **App SDKs** extract that context, create child spans, inject the same headers on outbound calls  
+3. App **pushes** spans via OTLP → Alloy → Tempo  
+4. **`x-request-id`** (Envoy/Istio) is a **separate correlation id** used in access logs — it is *not* the OTEL trace id. Join them by putting `x-request-id` on the span (we do) and echoing both on the response.
+
+| Header | What it is | We do |
+|--------|------------|--------|
+| `traceparent` / `tracestate` | W3C trace context (modern default) | extract + inject |
+| `x-b3-*` / `b3` | Zipkin B3 (common with Istio) | extract + inject |
+| `uber-trace-id` | Jaeger | extract + inject |
+| `x-request-id` | Envoy request id (logs) | span attr `http.request_id` + response echo |
+| `x-correlation-id` | app/gateway variant | span attr + echo if present |
+
+So: we **match** mesh traffic by speaking **W3C + B3 + Jaeger**, and we **use** Istio’s request id as an attribute / response header so you can jump from Envoy logs to Tempo (`X-Trace-Id`).
+
+### Defaults (when env is unset)
+
+| Variable | Default here |
+|----------|----------------|
+| export / SDK | **disabled (no-op)** until `OTEL_EXPORTER_OTLP_ENDPOINT` (or traces endpoint) is set |
+| `OTEL_SERVICE_NAME` | `cluster-utils-api` |
+| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` (port **4318** on Alloy) |
+| `OTEL_TRACE_SAMPLE_RATIO` | `1.0` (all traces when enabled) |
+| `OTEL_TRACE_PROBES` | off (no spans for `/livez` `/readyz` `/startupz` `/metrics` `/ping`) |
+| propagators | always **tracecontext + baggage + b3 + jaeger** |
+| `OTEL_EXPORTER_OTLP_INSECURE` | unset (exporter default); set `"true"` in-cluster without TLS |
+
+On **every startup** we log a single line `otel config (effective)` with enabled flag, endpoints, protocol, sample ratio, probe tracing, and propagators — grep pod logs for `otel config`.
+
+### Enable traces (OTLP push)
+
+```yaml
+env:
+  - name: OTEL_SERVICE_NAME
+    value: cluster-utils-api
+  - name: OTEL_EXPORTER_OTLP_ENDPOINT
+    value: "alloy.observability.svc.cluster.local:4318"   # http/protobuf default
+  - name: OTEL_EXPORTER_OTLP_PROTOCOL
+    value: http/protobuf   # or grpc (often :4317)
+  - name: OTEL_EXPORTER_OTLP_INSECURE
+    value: "true"          # TLS off inside the mesh
+  # optional:
+  # - name: OTEL_TRACE_SAMPLE_RATIO
+  #   value: "1.0"
+  # - name: OTEL_SDK_DISABLED
+  #   value: "true"
+  # - name: OTEL_TRACE_PROBES
+  #   value: "true"   # also span kube probes (noisy)
+```
+
+What gets instrumented:
+
+- **Inbound HTTP** — Gin (`otelgin`) + mesh header attributes  
+- **Outbound `/a/proxy`** — `otelhttp` client span + header inject east-west  
+- **Response** `X-Trace-Id` (OTEL) and `X-Request-Id` (if the mesh/client sent one)
+
+```bash
+curl -sS -D- -H 'X-Request-Id: demo-from-gateway' localhost:8080/debug -o /dev/null | grep -iE 'x-trace-id|x-request-id'
+```
+
+**Important:** `X-Request-Id` (Istio/Envoy) ≠ `X-Trace-Id` (OpenTelemetry/Tempo).  
+They are both useful; we keep both. In Tempo, search by trace id, or by span attribute `http.request_id` when the mesh sent a request id. Pod logs include `trace_id` + `request_id` on each request line when present.
+
+### Join Envoy / Istio access logs ↔ Tempo
+
+```bash
+# 1) call through the mesh (or simulate Envoy's header)
+curl -sS -D /tmp/hdrs -H 'X-Request-Id: 0a1b2c3d-demo' localhost:8080/debug -o /dev/null
+grep -iE 'x-trace-id|x-request-id' /tmp/hdrs
+
+# 2) app logs (same ids)
+# kubectl logs deploy/cluster-utils-api | grep 0a1b2c3d-demo
+
+# 3) Tempo: search TraceID = value of X-Trace-Id
+#    or attribute http.request_id = 0a1b2c3d-demo
+```
+
+On boot, always check:
+
+```bash
+kubectl logs deploy/cluster-utils-api | grep 'otel config'
+# → enabled, endpoint, protocol, sample_ratio, propagators, mesh_headers, …
+```
+
+### Alloy sketch
+
+```hcl
+// metrics: scrape this app
+prometheus.scrape "cu_api" {
+  targets      = [{ __address__ = "cluster-utils-api-svc:8080" }]
+  metrics_path = "/metrics"
+  forward_to   = [prometheus.remote_write.mimir.receiver]
+}
+
+// traces: receive OTLP *push* from the app
+otelcol.receiver.otlp "default" {
+  http { endpoint = "0.0.0.0:4318" }
+  grpc { endpoint = "0.0.0.0:4317" }
+  output { traces = [otelcol.exporter.otlp.tempo.input] }
+}
+
+otelcol.exporter.otlp "tempo" {
+  client { endpoint = "tempo:4317" tls { insecure = true } }
+}
+```
+
+Istio tip: prefer mesh config that emits **W3C** (or dual W3C+B3). If you only have B3 today, our B3 propagator still joins the chain.
+
+---
+
 ## Security
 
 This image is a **cluster debug tool**, not a public SaaS. Treat it like you treat `kubectl` access.

RELEASE-v2.5.0.md

@@ -0,0 +1,64 @@
+## cluster-utils-api v2.5.0 — OpenTelemetry traces
+
+Adds OTLP **trace export** (push). Metrics are unchanged and still scraped.
+
+### Push vs scrape (read this twice)
+
+| Signal | How | Path |
+|--------|-----|------|
+| **Metrics** | **Scrape** (pull) | `GET /metrics` → Alloy/Prometheus → Mimir |
+| **Traces** | **Push** | OTLP → Alloy → **Tempo** |
+
+Tempo does **not** scrape `/metrics` for spans. The app exports OTLP when configured.
+
+### Mesh / Istio — request id vs trace id
+
+This is the detail that usually bites people:
+
+- **`x-request-id`** (Envoy/Istio) is a **request correlation id** for access logs.
+- **`traceparent` / OTEL trace id** (`X-Trace-Id` on our responses) is the **distributed trace** id in Tempo.
+- They are **not the same value**. Both matter.
+
+**What we do:**
+
+- Propagate **W3C + B3 + Jaeger + baggage** so modern *and* Istio-era meshes join the same trace.
+- Attach mesh ids as span attributes (`http.request_id`, correlation ids, …).
+- Echo **`X-Trace-Id`** and **`X-Request-Id`** (when present) on responses.
+- Put `trace_id` + `request_id` on JSON request logs so you can join Envoy logs ↔ Tempo.
+
+**How to join:** Tempo search by `X-Trace-Id`, or by attribute `http.request_id` = Envoy’s id.
+
+Other meshes (Linkerd, etc.) follow the same pattern: W3C is the normal default; B3 still shows up; product-specific headers are extra correlation.
+
+### Enable (defaults)
+
+Tracing is a **no-op** until an OTLP endpoint is set.
+
+| Variable | Default |
+|----------|---------|
+| export | off until `OTEL_EXPORTER_OTLP_ENDPOINT` (or traces endpoint) |
+| protocol | `http/protobuf` (Alloy **:4318**) |
+| service name | `cluster-utils-api` |
+| sample ratio | `1.0` |
+| probe spans | off (`OTEL_TRACE_PROBES=true` to include `/livez` etc.) |
+| propagators | W3C + baggage + B3 + Jaeger |
+
+```yaml
+OTEL_SERVICE_NAME=cluster-utils-api
+OTEL_EXPORTER_OTLP_ENDPOINT=alloy.observability.svc:4318
+OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
+OTEL_EXPORTER_OTLP_INSECURE=true
+```
+
+On **every startup** logs include `otel config (effective)` — grep pod logs for `otel config` to see exactly what it started with.
+
+### What is instrumented
+
+- Inbound HTTP (Gin), with mesh header attributes
+- Outbound `/a/proxy` (linked east-west spans + header inject)
+- Probe/metrics/ping paths skipped unless `OTEL_TRACE_PROBES=true`
+
+### Images
+
+- `donkeyx/cluster-utils-api:2.5.0` / `:latest`
+- `ghcr.io/donkeyx/cluster-utils-api:2.5.0` / `:latest`

go.mod

@@ -9,6 +9,16 @@ require (
 	github.com/swaggo/files v1.0.1
 	github.com/swaggo/gin-swagger v1.6.1
 	github.com/swaggo/swag v1.16.6
+	go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin v0.70.0
+	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0
+	go.opentelemetry.io/contrib/propagators/b3 v1.45.0
+	go.opentelemetry.io/contrib/propagators/jaeger v1.45.0
+	go.opentelemetry.io/otel v1.45.0
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0
+	go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.45.0
+	go.opentelemetry.io/otel/sdk v1.45.0
+	go.opentelemetry.io/otel/trace v1.45.0
 	go.uber.org/zap v1.28.0
 )
 
@@ -17,51 +27,63 @@ require (
 	github.com/PuerkitoBio/purell v1.1.1 // indirect
 	github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
-	github.com/bytedance/gopkg v0.1.3 // indirect
-	github.com/bytedance/sonic v1.15.0 // indirect
-	github.com/bytedance/sonic/loader v0.5.0 // indirect
+	github.com/bytedance/gopkg v0.1.4 // indirect
+	github.com/bytedance/sonic v1.15.2 // indirect
+	github.com/bytedance/sonic/loader v0.5.2 // indirect
+	github.com/cenkalti/backoff/v5 v5.0.3 // indirect
 	github.com/cespare/xxhash/v2 v2.3.0 // indirect
-	github.com/cloudwego/base64x v0.1.6 // indirect
+	github.com/cloudwego/base64x v0.1.7 // indirect
 	github.com/davecgh/go-spew v1.1.1 // indirect
-	github.com/gabriel-vasile/mimetype v1.4.12 // indirect
-	github.com/gin-contrib/sse v1.1.0 // indirect
+	github.com/felixge/httpsnoop v1.1.0 // indirect
+	github.com/gabriel-vasile/mimetype v1.4.15 // indirect
+	github.com/gin-contrib/sse v1.1.1 // indirect
+	github.com/go-logr/logr v1.4.4 // indirect
+	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/go-openapi/jsonpointer v0.19.5 // indirect
 	github.com/go-openapi/jsonreference v0.19.6 // indirect
 	github.com/go-openapi/spec v0.20.4 // indirect
 	github.com/go-openapi/swag v0.19.15 // indirect
 	github.com/go-playground/locales v0.14.1 // indirect
 	github.com/go-playground/universal-translator v0.18.1 // indirect
-	github.com/go-playground/validator/v10 v10.30.1 // indirect
-	github.com/goccy/go-json v0.10.5 // indirect
+	github.com/go-playground/validator/v10 v10.30.3 // indirect
+	github.com/goccy/go-json v0.10.6 // indirect
 	github.com/goccy/go-yaml v1.19.2 // indirect
+	github.com/google/uuid v1.6.0 // indirect
+	github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
-	github.com/klauspost/cpuid/v2 v2.3.0 // indirect
-	github.com/leodido/go-urn v1.4.0 // indirect
+	github.com/klauspost/cpuid/v2 v2.4.0 // indirect
+	github.com/leodido/go-urn v1.5.0 // indirect
 	github.com/mailru/easyjson v0.7.6 // indirect
-	github.com/mattn/go-isatty v0.0.20 // indirect
+	github.com/mattn/go-isatty v0.0.24 // indirect
 	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
 	github.com/modern-go/reflect2 v1.0.2 // indirect
 	github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
-	github.com/pelletier/go-toml/v2 v2.2.4 // indirect
+	github.com/pelletier/go-toml/v2 v2.4.3 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/prometheus/client_model v0.6.2 // indirect
 	github.com/prometheus/common v0.70.1 // indirect
 	github.com/prometheus/procfs v0.21.1 // indirect
 	github.com/quic-go/qpack v0.6.0 // indirect
-	github.com/quic-go/quic-go v0.59.0 // indirect
+	github.com/quic-go/quic-go v0.61.0 // indirect
 	github.com/twitchyliquid64/golang-asm v0.15.1 // indirect
 	github.com/ugorji/go/codec v1.3.1 // indirect
-	go.mongodb.org/mongo-driver/v2 v2.5.0 // indirect
+	go.mongodb.org/mongo-driver/v2 v2.8.0 // indirect
+	go.opentelemetry.io/auto/sdk v1.2.1 // indirect
+	go.opentelemetry.io/otel/metric v1.45.0 // indirect
+	go.opentelemetry.io/proto/otlp v1.11.0 // indirect
 	go.uber.org/multierr v1.10.0 // indirect
-	golang.org/x/arch v0.22.0 // indirect
+	golang.org/x/arch v0.29.0 // indirect
 	golang.org/x/crypto v0.54.0 // indirect
 	golang.org/x/mod v0.37.0 // indirect
 	golang.org/x/net v0.57.0 // indirect
 	golang.org/x/sync v0.22.0 // indirect
 	golang.org/x/sys v0.47.0 // indirect
 	golang.org/x/text v0.40.0 // indirect
 	golang.org/x/tools v0.47.0 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d // indirect
+	google.golang.org/grpc v1.83.0 // indirect
 	google.golang.org/protobuf v1.36.11 // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect