fix(traces): resource identity hygiene in the extension, plus aws-sdk tarball build - #90
fix(traces): resource identity hygiene in the extension, plus aws-sdk tarball build#90GuyMoses wants to merge 4 commits into
Conversation
…ation The extension built its own resource for the spans it creates (the synthetic root, init and overhead spans) while the in-process auto-instrumentation sent its own. The two could never match: the SDK's detectors contribute `process.*` and `telemetry.sdk.*`, which a separate process cannot reproduce, and the extension added `cloud.platform`, `cloud.resource_id` and `cloud.account.id`, which the runtime never sees. A resource differing by one attribute is a different resource downstream, so a single Lambda rendered as two nodes in the trace graph -- one for the extension's SERVER root span, one for the handler span beneath it. Capture the resource from the auto-instrumentation's payload as it passes through, enrich it in place with the attributes only the extension knows, and reuse that exact resource for the extension's own spans. Both sets now go out under one resource. Falls back to building a resource when no auto-instrumentation payload arrives, which is the case when auto-instrumentation is switched off or an error-path synthetic trace is built for an invocation that never reported.
opt/shared.sh wrote the resource attribute with an underscore, so every span the auto-instrumentation produced carried a non-semconv `cloud_platform` key and no `cloud.platform` -- despite the README documenting `cloud.platform`. The extension's own spans set the correct key, so the two disagreed on exactly the attribute that identifies the resource as a Lambda. shared.sh is sourced by the Node, Python and Java wrappers, so this affected all three runtimes. Nothing read the misspelled key.
…rball The fork's monorepo root install compiles better-sqlite3 for a workspace we do not build, which needs a C toolchain and fails outright on a machine without Xcode Command Line Tools. instrumentation-aws-sdk needs no native module.
Validated on a deployed LambdaThe review note above said this had only been checked against the Runtime Every gap the emulator left is closed:
So Two functions were deployed to the same dataset for comparison: one on the Resource equality itself was verified locally, where a collector can be put in |
A Lambda's downstream identity is cloud.platform + cloud.account.id + cloud.region + faas.name. cloud.account.id is therefore not an ordinary attribute: writing "unknown" when the value is unavailable does not record a gap, it asserts an identity, and one that disagrees with anything else reporting the real account for the same function -- which is exactly the input that makes one function resolve to two resources. Only enrich the auto-instrumentation's resource with the attributes whose values the extension actually has.
Four small fixes around the resource the extension puts on telemetry, plus one build-script fix. None of them fixes a reported bug — see "What this does not fix". They are correctness/robustness changes in an area I had to map out in detail for SIG-364, and I would rather the mapping land as code and comments than stay in a ticket.
Background: what decides that a Lambda is one node
Two OTLP streams leave a Lambda: the in-process auto-instrumentation's spans (forwarded through the extension) and the spans and logs the extension creates itself. They carry different resources — the distro's is
defaultResource()+envDetector+processDetector(opt/node/distro/src/bootstrap.ts), the extension's isget_resources_attributes().Downstream, a resource's identity is not its full attribute set. For
cloud.platform=aws_lambdait is exactly four attributes:So
cloud.resource_id,service.name,telemetry.sdk.*andprocess.*may differ freely between the two streams — and they do — without any effect. Only those four matter, andcloud.platformandcloud.account.idare precisely the two the in-process SDK cannot detect. That is the whole reason the extension needs to contribute to the forwarded resource at all.The changes
1. Share one resource between the extension and the auto-instrumentation (
resources.rs,span_mutations.rs,invocation_entry.rs)process_trace_requestnow enriches the incoming resource with the extension-known attributes and captures it per invocation; spans the extension creates for that invocation reuse it verbatim instead of building a second one. Both streams then resolve through the same identity, deterministically, rather than depending on which attributes the in-process instrumentation happens to put where.2. Never assert a placeholder
cloud.account.id(resources.rs)Enrichment used
unwrap_or("unknown"). Sincecloud.account.idis an identity attribute, writing"unknown"does not record a missing value — it claims an identity, and one that disagrees with anything reporting the real account for the same function. Now only attributes with known values are added. (get_resources_attributes, used for the extension's own resource when no payload arrived, keeps its existing behaviour.)3.
cloud.platform, notcloud_platform(opt/shared.sh)The wrapper wrote
cloud_platform=aws_lambdaintoOTEL_RESOURCE_ATTRIBUTES. It reaches the backend ascloud.platformanyway, because ingest also registers the Prometheus-normalized variant of every resource attribute key and rewrites it. Butcloud.platformis an identity attribute: relying on that normalization is a bet, and any other OTLP consumer sees a key that means nothing. One occurrence, no consumers in-tree;README.mdalready documented it ascloud.platform.4.
--ignore-scriptswhen building the aws-sdk tarball (opt/node/scripts/build-aws-sdk-tarball.sh)npm installat the fork's monorepo root compiles native modules for workspaces we never build (better-sqlite3), so the script fails on any machine without a C++ toolchain.instrumentation-aws-sdkneeds none of them.What this does not fix
I originally opened this claiming it fixed a Lambda rendering as two nodes in the trace graph. That was wrong, and the PR title and description have been corrected.
The two-node symptom reproduces only under the Runtime Interface Emulator, where the extension's two sources for the account id — the Extensions API registration response and the invoke event's
invokedFunctionArn— disagree, and the two resources end up with differentcloud.account.idvalues. On a deployed Lambda both are the real account id, the four identity attributes match, and the function renders as one node both before and after these changes. Verified on two container-image Lambdas, one with the published layer and one with this branch.It also does not touch SIG-364 itself: the AWS SDK span being dropped from the graph view in favour of its child HTTP span. That is a graph-view change and lives in another repo. For the record, the shape the Node distro emits is parent/child, not siblings as the ticket says:
which is standard OTel JS output — Java suppresses the nested HTTP span via
SpanKey.HTTP_CLIENT, Node does not, and suppressing it in Node would lose retry and transport detail.Verification
cargo test: 293 pass.cargo clippy --all-targets -- -D warningsreports the same 22 pre-existing errors asmaster.