Skip to content

feat(telemetry): add OpenTelemetry traces and metrics - #792

Merged
tobiasehlert merged 5 commits into
mainfrom
feat-add-opentelemetry-traces-and-metric
Oct 21, 2026
Merged

tobiasehlert merged 5 commits into
mainfrom
feat-add-opentelemetry-traces-and-metric

Conversation

@tobiasehlert

@tobiasehlert tobiasehlert commented Oct 21, 2026 •

Copy link
Copy Markdown
Member

Closes #749

Summary

Adds optional OpenTelemetry instrumentation. It is off by default and turned on with TIBIADATA_OTEL_ENABLED=true. All other settings use the standard OTEL_* environment variables.

  • Traces: each incoming Gin request gets a server span, via otelgin. Incoming W3C traceparent headers are honoured. Upstream calls to tibia.com and the fansite API get child client spans and propagate traceparent, via an otelhttp transport on the shared resty client.
  • Metrics: HTTP server and client semconv metrics, such as http.server.request.duration, which gives request count, latency, method, route and status code.
  • Export: uses autoexport, so OTEL_TRACES_EXPORTER / OTEL_METRICS_EXPORTER can be set to otlp (default), prometheus (metrics only, served on its own port, default localhost:9464), console or none.
  • Probes: /ping, /health, /healthz and /readyz are counted in metrics but not traced. This is done with a small tracer-provider wrapper, because otelgin's WithFilter would also drop their metrics.
  • Log correlation: traceLogAttrs now adds trace_id / span_id to the access log and the panic-recovery log (carried over from feat: add structured logging #750).
  • Shutdown: spans and metrics are flushed when the server shuts down gracefully.
  • README: new TIBIADATA_OTEL_ENABLED row and an OpenTelemetry section.

Notes

  • Collectors accept the incoming request's context as their first parameter, including the houses overview's injected collector. Context is not stored in TibiaDataRequestStruct. Upstream requests are therefore also cancelled when the client disconnects.
  • When telemetry is disabled, the resty transport is left unchanged and the global providers stay no-op.
  • A signal whose exporter is none keeps the global no-op provider. Setting OTEL_TRACES_EXPORTER=none (for example, metrics only in prod) therefore creates no spans at all.
  • When telemetry is disabled, or both exporters are none, the otelgin middleware and the otelhttp transport are not installed, so requests pay no instrumentation cost. In metrics-only mode otelgin still builds some span attributes per request (included in the numbers below).
  • Client spans record the full upstream URL in url.full, including the query string (for example, character or guild names). These values are public, but they end up in the trace backend when tracing is enabled.
  • The server now also handles SIGTERM (sent by Kubernetes and docker stop), and shuts down gracefully: readiness turns false, http.Server.Shutdown drains in-flight requests for up to 20s (below Kubernetes' 30s default grace period), then telemetry is flushed. Before this change SIGINT called server.Close(), which dropped in-flight requests.

Binary size

Stripped linux/amd64 builds (-ldflags="-w -s", CGO_ENABLED=0):

Build Raw gzip -9
main (no OpenTelemetry) 14.9 MB 5.4 MB
Tracing + metrics SDK, no exporters 16.4 MB 5.8 MB
OTLP over HTTP only 26.3 MB —
This PR (autoexport) 27.9 MB 9.6 MB

Most of the increase comes from the OTLP exporters: the protobuf message types plus gRPC, which the OTLP proto package links even for HTTP export. Restricting to OTLP over HTTP saves only 1.6 MB. A build tag could keep the default image near 16 MB, but then the published image couldn't export telemetry and a second image variant would be needed. I kept autoexport with a single image; that adds about 4.2 MB compressed to each image pull.

Runtime overhead

Full middleware stack benchmark (Go benchmark through telemetry, access log, recovery and gzip middleware, on route /v4/worlds/:world with a static JSON handler and no upstream call):

Mode Time/request Allocated bytes/request (B/op) Allocs/request
Disabled ~4.9 µs 4.5 KB 42
Metrics only (OTEL_TRACES_EXPORTER=none) ~5.7 µs 4.5 KB 42
Traces + metrics ~10.6 µs 9.8 KB 57

B/op is short-lived allocation per request, not retained memory. Process idle RSS after 20 requests is about 29–30 MB in all three modes. Real endpoints spend hundreds of milliseconds fetching and parsing tibia.com, so the overhead is negligible.

Testing

  • New unit tests: traceLogAttrs; telemetry init (disabled/enabled); resource defaults and OTEL_SERVICE_NAME override; server span, parent extraction and log correlation; probes producing metrics but no spans; outbound traceparent propagation with a client span; explicit context propagation through houses collectors and cancellation of HTML and JSON requests.
  • go vet ./... and go test ./... -race pass.
  • Manual smoke test with OTEL_TRACES_EXPORTER=console OTEL_METRICS_EXPORTER=prometheus: /versions produced a span whose IDs matched the access log, /healthz produced no span, both routes showed up in Prometheus metrics, and spans were flushed on SIGINT and SIGTERM.
  • Drain check: with a 2s upstream, SIGTERM during a request still returned 200, new connections were refused, and the process exited only after the request finished.

Instrument Gin requests and upstream calls with OpenTelemetry, gated by
TIBIADATA_OTEL_ENABLED. Exporters are selected via the standard OTEL_*
env vars (OTLP by default, Prometheus and console supported). Probe
endpoints are kept in metrics but not traced. Request logs now include
trace_id and span_id for correlation.

Closes #749

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@tobiasehlert tobiasehlert added the enhancement New feature or request label Oct 21, 2026
@tobiasehlert tobiasehlert self-assigned this Oct 21, 2026
@tobiasehlert tobiasehlert added the go Pull requests that update Go code label Oct 21, 2026
@codecov

codecov Bot commented Oct 21, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.66667% with 28 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.68%. Comparing base (3cd8524) to head (6acdba0).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #792      +/-   ##
==========================================
- Coverage   88.87%   88.68%   -0.19%     
==========================================
  Files          24       25       +1     
  Lines        3064     3173     +109     
==========================================
+ Hits         2723     2814      +91     
- Misses        341      359      +18     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

tobiasehlert and others added 4 commits October 21, 2026 14:45
Remove context from request data and propagate it as the first parameter.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Kubernetes stops pods with SIGTERM; without handling it the server exited without draining requests or flushing telemetry.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use http.Server.Shutdown with a 20s bound on SIGTERM/interrupt, mark
the server not ready, and wait for draining to finish before flushing
telemetry. Only install otelgin and otelhttp instrumentation when at
least one OpenTelemetry signal is exported, and skip the probe-filter
context when tracing is off.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@sonarqubecloud

Copy link
Copy Markdown

@tobiasehlert
tobiasehlert merged commit f9009fa into main Oct 21, 2026
8 checks passed
@tobiasehlert
tobiasehlert deleted the feat-add-opentelemetry-traces-and-metric branch October 21, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request go Pull requests that update Go code

Development

Successfully merging this pull request may close these issues.

feat: add OpenTelemetry traces and metrics

1 participant