Skip to content

fix(traces): resource identity hygiene in the extension, plus aws-sdk tarball build - #90

Open
GuyMoses wants to merge 4 commits into
masterfrom
fix/share-resource-between-extension-and-distro
Open

fix(traces): resource identity hygiene in the extension, plus aws-sdk tarball build#90
GuyMoses wants to merge 4 commits into
masterfrom
fix/share-resource-between-extension-and-distro

Conversation

@GuyMoses

@GuyMoses GuyMoses commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Four small fixes around the resource the extension puts on telemetry, plus one build-script fix. None of them fixes a reported bug — see "What this does not fix". They are correctness/robustness changes in an area I had to map out in detail for SIG-364, and I would rather the mapping land as code and comments than stay in a ticket.

Background: what decides that a Lambda is one node

Two OTLP streams leave a Lambda: the in-process auto-instrumentation's spans (forwarded through the extension) and the spans and logs the extension creates itself. They carry different resources — the distro's is defaultResource() + envDetector + processDetector (opt/node/distro/src/bootstrap.ts), the extension's is get_resources_attributes().

Downstream, a resource's identity is not its full attribute set. For cloud.platform=aws_lambda it is exactly four attributes:

cloud.platform : cloud.account.id : cloud.region : faas.name

So cloud.resource_id, service.name, telemetry.sdk.* and process.* may differ freely between the two streams — and they do — without any effect. Only those four matter, and cloud.platform and cloud.account.id are precisely the two the in-process SDK cannot detect. That is the whole reason the extension needs to contribute to the forwarded resource at all.

The changes

1. Share one resource between the extension and the auto-instrumentation (resources.rs, span_mutations.rs, invocation_entry.rs)

process_trace_request now enriches the incoming resource with the extension-known attributes and captures it per invocation; spans the extension creates for that invocation reuse it verbatim instead of building a second one. Both streams then resolve through the same identity, deterministically, rather than depending on which attributes the in-process instrumentation happens to put where.

2. Never assert a placeholder cloud.account.id (resources.rs)

Enrichment used unwrap_or("unknown"). Since cloud.account.id is an identity attribute, writing "unknown" does not record a missing value — it claims an identity, and one that disagrees with anything reporting the real account for the same function. Now only attributes with known values are added. (get_resources_attributes, used for the extension's own resource when no payload arrived, keeps its existing behaviour.)

3. cloud.platform, not cloud_platform (opt/shared.sh)

The wrapper wrote cloud_platform=aws_lambda into OTEL_RESOURCE_ATTRIBUTES. It reaches the backend as cloud.platform anyway, because ingest also registers the Prometheus-normalized variant of every resource attribute key and rewrites it. But cloud.platform is an identity attribute: relying on that normalization is a bet, and any other OTLP consumer sees a key that means nothing. One occurrence, no consumers in-tree; README.md already documented it as cloud.platform.

4. --ignore-scripts when building the aws-sdk tarball (opt/node/scripts/build-aws-sdk-tarball.sh)

npm install at the fork's monorepo root compiles native modules for workspaces we never build (better-sqlite3), so the script fails on any machine without a C++ toolchain. instrumentation-aws-sdk needs none of them.

What this does not fix

I originally opened this claiming it fixed a Lambda rendering as two nodes in the trace graph. That was wrong, and the PR title and description have been corrected.

The two-node symptom reproduces only under the Runtime Interface Emulator, where the extension's two sources for the account id — the Extensions API registration response and the invoke event's invokedFunctionArn — disagree, and the two resources end up with different cloud.account.id values. On a deployed Lambda both are the real account id, the four identity attributes match, and the function renders as one node both before and after these changes. Verified on two container-image Lambdas, one with the published layer and one with this branch.

It also does not touch SIG-364 itself: the AWS SDK span being dropped from the graph view in favour of its child HTTP span. That is a graph-view change and lives in another repo. For the record, the shape the Node distro emits is parent/child, not siblings as the ticket says:

<function> [SERVER] -> handler [INTERNAL] -> DynamoDB.GetItem [CLIENT] -> POST [CLIENT]

which is standard OTel JS output — Java suppresses the nested HTTP span via SpanKey.HTTP_CLIENT, Node does not, and suppressing it in Node would lose retry and transport detail.

Verification

  • cargo test: 293 pass. cargo clippy --all-targets -- -D warnings reports the same 22 pre-existing errors as master.
  • Deployed two container-image Lambdas (published layer, and this branch) making a real DynamoDB call: Telemetry API subscription accepted, wrapper honoured, exports accepted, one graph node in both.

…ation

The extension built its own resource for the spans it creates (the synthetic
root, init and overhead spans) while the in-process auto-instrumentation sent
its own. The two could never match: the SDK's detectors contribute `process.*`
and `telemetry.sdk.*`, which a separate process cannot reproduce, and the
extension added `cloud.platform`, `cloud.resource_id` and `cloud.account.id`,
which the runtime never sees.

A resource differing by one attribute is a different resource downstream, so a
single Lambda rendered as two nodes in the trace graph -- one for the extension's
SERVER root span, one for the handler span beneath it.

Capture the resource from the auto-instrumentation's payload as it passes
through, enrich it in place with the attributes only the extension knows, and
reuse that exact resource for the extension's own spans. Both sets now go out
under one resource. Falls back to building a resource when no
auto-instrumentation payload arrives, which is the case when auto-instrumentation
is switched off or an error-path synthetic trace is built for an invocation that
never reported.
opt/shared.sh wrote the resource attribute with an underscore, so every span the
auto-instrumentation produced carried a non-semconv `cloud_platform` key and no
`cloud.platform` -- despite the README documenting `cloud.platform`. The
extension's own spans set the correct key, so the two disagreed on exactly the
attribute that identifies the resource as a Lambda.

shared.sh is sourced by the Node, Python and Java wrappers, so this affected all
three runtimes. Nothing read the misspelled key.
…rball

The fork's monorepo root install compiles better-sqlite3 for a workspace we do
not build, which needs a C toolchain and fails outright on a machine without
Xcode Command Line Tools. instrumentation-aws-sdk needs no native module.
@GuyMoses

GuyMoses commented Sep 6, 2026

Copy link
Copy Markdown
Contributor Author

Validated on a deployed Lambda

The review note above said this had only been checked against the Runtime
Interface Emulator. It has now run on a real container-image Lambda (arm64),
against real DynamoDB, with the locally built binary and patched shared.sh
swapped into the published extension layer.

Every gap the emulator left is closed:

emulator real Lambda
account id Registration response did not contain accountId resolved from the registration response
Telemetry API Telemetry API is not supported status=200 OK
platform.runtimeDone had to be synthesised delivered by the platform
parent span id empty populated
export 200 OK 200 OK, zero Error sending

So cloud.account.id and cloud.resource_id resolve to real values rather than
the unknown / placeholder-account the emulator produces. Those are the two
attributes this change copies onto the shared resource, so it matters that they
were checked outside the emulator.

Two functions were deployed to the same dataset for comparison: one on the
published extension (pre-fix) and one on this branch.

Resource equality itself was verified locally, where a collector can be put in
front of the extension: the two resourceSpans batches went from differing by
six attributes to byte-identical, order included. The deployed run exercises the
same code path with real values in those attributes.

@GuyMoses GuyMoses changed the title fix(traces): share one resource between the extension and the auto-instrumentation fix(traces): cloud.platform typo, shared resource, aws-sdk tarball build Sep 6, 2026
A Lambda's downstream identity is cloud.platform + cloud.account.id +
cloud.region + faas.name. cloud.account.id is therefore not an ordinary
attribute: writing "unknown" when the value is unavailable does not record
a gap, it asserts an identity, and one that disagrees with anything else
reporting the real account for the same function -- which is exactly the
input that makes one function resolve to two resources.

Only enrich the auto-instrumentation's resource with the attributes whose
values the extension actually has.
@GuyMoses GuyMoses changed the title fix(traces): cloud.platform typo, shared resource, aws-sdk tarball build fix(traces): resource identity hygiene in the extension, plus aws-sdk tarball build Sep 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant