[APMSVLS-469] fix(traces): filter sampled-out trace chunks when tracer or extension computes stats - #1358
Conversation
…xtension computes stats Trace intake now drops sampled-out chunks whenever stats are computed before the backend, by either the tracer (Datadog-Client-Computed-Stats) or the extension, instead of only when extension-side stats are enabled. The backend still receives them when it owns stats computation, since it needs them to compute stats. The error sampler now rescues eligible errored drops on both non-backend paths.
|
🔗 Commit SHA: b57d663 | Docs | View more details | Give us feedback! |
lucaspimentel
left a comment
There was a problem hiding this comment.
Found one high-severity trace-loss issue.
Review by Claude Code
| config.ext.lambda_extension_compute_stats, | ||
| header_tags.generic.client_computed_stats, | ||
| ); | ||
| if stats_computed_by != StatsComputedBy::Backend |
There was a problem hiding this comment.
bottlecap/src/traces/trace_processor.rs:583: This newly runs drop_sampled_out_chunks for tracer-owned stats, but that helper drops a chunk solely from its sampling priority. Tracers can send a priority-0 partial trace containing spans retained by single-span sampling and marked with _dd.span_sampling.mechanism (Java does this); the standard Trace Agent preserves those spans and promotes the chunk to user-keep. Here the whole chunk is discarded, so configured single-span sampling silently loses the selected spans whenever Datadog-Client-Computed-Stats: true. Before removing a dropped chunk, retain its span-sampled spans and promote the resulting chunk as the Trace Agent does, with a tracer-owned regression test.
Overview
Follow-up to #1320. Filters sampled-out trace chunks from trace intake whenever trace stats are computed before the backend, by either the tracer or the extension. Previously the filtering gate only checked
lambda_extension_compute_stats, so tracers that compute stats themselves and reportDatadog-Client-Computed-Stats: true(for example,dd-trace-javav1.63.0 withDD_TRACE_STATS_COMPUTATION_ENABLEDon by default) still had their rejected chunks forwarded to intake.lambda_extension_compute_stats=false,client_computed_stats=false)lambda_extension_compute_stats=true)client_computed_stats=true)Behavior notes:
client_computed_stats=truemeans the tracer owns stats even when extension-side stats are also configured.AutoDropchunks; explicit/user drops (negative priority) remain dropped, and_dd.errors_sris stamped on rescued root spans.Testing
trace_processor.rs: table-driven over all four ownership combinations, asserting filtering vs. no-filtering, the pre-filter stats payload contents, and error rescue on both non-backend paths. The tracer-owned case was confirmed failing before the fix.apm_integration_test.rs:e2e_sampled_out_chunks_filtered_by_stats_owner(backend keeps all chunks; extension and tracer paths filter to the kept trace only) ande2e_error_sampler_rescues_only_errored_p0_tracesextended to both non-backend owners.cargo nextest run --workspace: 678 passed, 0 failed.cargo fmt --all -- --check,cargo check --workspace, and both clippy passes (default and fips features) pass.Found while investigating excess trace volume in the
serverless-e2e-testsjava17-samplingscenario: 20 traces observed at a 0.2 sample rate over 50 invocations where roughly 10 were expected.