Repository navigation
feat(telemetry): add OpenTelemetry traces and metrics - #792
Merged
Merged
Conversation
Instrument Gin requests and upstream calls with OpenTelemetry, gated by TIBIADATA_OTEL_ENABLED. Exporters are selected via the standard OTEL_* env vars (OTLP by default, Prometheus and console supported). Probe endpoints are kept in metrics but not traced. Request logs now include trace_id and span_id for correlation. Closes #749 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #792 +/- ##
==========================================
- Coverage 88.87% 88.68% -0.19%
==========================================
Files 24 25 +1
Lines 3064 3173 +109
==========================================
+ Hits 2723 2814 +91
- Misses 341 359 +18 🚀 New features to boost your workflow:
|
Remove context from request data and propagate it as the first parameter. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Kubernetes stops pods with SIGTERM; without handling it the server exited without draining requests or flushing telemetry. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Use http.Server.Shutdown with a 20s bound on SIGTERM/interrupt, mark the server not ready, and wait for draining to finish before flushing telemetry. Only install otelgin and otelhttp instrumentation when at least one OpenTelemetry signal is exported, and skip the probe-filter context when tracing is off. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Closes #749
Summary
Adds optional OpenTelemetry instrumentation. It is off by default and turned on with
TIBIADATA_OTEL_ENABLED=true. All other settings use the standardOTEL_*environment variables.otelgin. Incoming W3Ctraceparentheaders are honoured. Upstream calls to tibia.com and the fansite API get child client spans and propagatetraceparent, via anotelhttptransport on the shared resty client.http.server.request.duration, which gives request count, latency, method, route and status code.autoexport, soOTEL_TRACES_EXPORTER/OTEL_METRICS_EXPORTERcan be set tootlp(default),prometheus(metrics only, served on its own port, defaultlocalhost:9464),consoleornone./ping,/health,/healthzand/readyzare counted in metrics but not traced. This is done with a small tracer-provider wrapper, because otelgin'sWithFilterwould also drop their metrics.traceLogAttrsnow addstrace_id/span_idto the access log and the panic-recovery log (carried over from feat: add structured logging #750).TIBIADATA_OTEL_ENABLEDrow and an OpenTelemetry section.Notes
TibiaDataRequestStruct. Upstream requests are therefore also cancelled when the client disconnects.nonekeeps the global no-op provider. SettingOTEL_TRACES_EXPORTER=none(for example, metrics only in prod) therefore creates no spans at all.none, the otelgin middleware and the otelhttp transport are not installed, so requests pay no instrumentation cost. In metrics-only mode otelgin still builds some span attributes per request (included in the numbers below).url.full, including the query string (for example, character or guild names). These values are public, but they end up in the trace backend when tracing is enabled.docker stop), and shuts down gracefully: readiness turns false,http.Server.Shutdowndrains in-flight requests for up to 20s (below Kubernetes' 30s default grace period), then telemetry is flushed. Before this change SIGINT calledserver.Close(), which dropped in-flight requests.Binary size
Stripped linux/amd64 builds (
-ldflags="-w -s",CGO_ENABLED=0):main(no OpenTelemetry)autoexport)Most of the increase comes from the OTLP exporters: the protobuf message types plus gRPC, which the OTLP proto package links even for HTTP export. Restricting to OTLP over HTTP saves only 1.6 MB. A build tag could keep the default image near 16 MB, but then the published image couldn't export telemetry and a second image variant would be needed. I kept
autoexportwith a single image; that adds about 4.2 MB compressed to each image pull.Runtime overhead
Full middleware stack benchmark (Go benchmark through telemetry, access log, recovery and gzip middleware, on route
/v4/worlds/:worldwith a static JSON handler and no upstream call):B/op)OTEL_TRACES_EXPORTER=none)B/opis short-lived allocation per request, not retained memory. Process idle RSS after 20 requests is about 29–30 MB in all three modes. Real endpoints spend hundreds of milliseconds fetching and parsing tibia.com, so the overhead is negligible.Testing
traceLogAttrs; telemetry init (disabled/enabled); resource defaults andOTEL_SERVICE_NAMEoverride; server span, parent extraction and log correlation; probes producing metrics but no spans; outboundtraceparentpropagation with a client span; explicit context propagation through houses collectors and cancellation of HTML and JSON requests.go vet ./...andgo test ./... -racepass.OTEL_TRACES_EXPORTER=console OTEL_METRICS_EXPORTER=prometheus:/versionsproduced a span whose IDs matched the access log,/healthzproduced no span, both routes showed up in Prometheus metrics, and spans were flushed on SIGINT and SIGTERM.