Skip to content

perf: stop eager-building HTTPS_CLIENT before extension registration - #95

Open
RaphaelManke wants to merge 1 commit into
masterfrom
perf/lazy-https-client
Open

perf: stop eager-building HTTPS_CLIENT before extension registration#95
RaphaelManke wants to merge 1 commit into
masterfrom
perf/lazy-https-client

Conversation

@RaphaelManke

@RaphaelManke RaphaelManke commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • route::init() force-built HTTPS_CLIENT (via Lazy::force) before the extension registers with the Lambda Extensions API. Building it invokes hyper-rustls's .with_native_roots(), which synchronously parses the entire OS trust store.
  • That client is only actually needed later, at first outbound OTLP export -- not at registration time. Since AWS Lambda won't let the customer runtime start initializing until this extension registers, that parse sat directly in front of the cold-start critical path for no benefit.
  • Fix: drop the eager Lazy::force call and let the client build lazily on first real use.

Evidence

Caught via live profiling (native arm64 build against a local mock of the Lambda Extensions/Runtime API): debug logs showed with_native_roots processed 162 valid and 0 invalid certs logged before the Registered with accountId line. After the fix, that log line no longer appears before registration.

Measured with a small local perf harness (added separately, see the harness PR): median spawn-to-registered time dropped ~1ms (7.93ms -> 6.89ms) on this dev machine. The real win should be larger on the actual Lambda runtime, where the OS trust-store bundle is likely bigger than this Mac's 162-cert one.

Real-AWS cold-start verification

Confirmed against actual Lambda cold starts, not just the local harness:

  • Deployed this PR's stack (pr-95-IntegrationTestsStack, account 175948133317, eu-central-1) via the existing CI integration-test pipeline.
  • Baseline: the current public release layer (v23, published from master the same day this branch was cut -- see release v23, layer ARN arn:aws:lambda:<region>:115813213817:layer:dash0-extension-node:23), temporarily attached to one of this PR's integration-test functions (pr-99-nodejs24-x-success-true-invocation-end-false-arm64) in place of this PR's own layer, then reverted afterward.
  • For each sample: force a fresh execution environment (update-function-configuration with a unique description, wait for the update, 100ms bake), then invoke with LogType=Tail and parse Init Duration from the Lambda REPORT log line. n=15 cold starts per variant.
  • Result: 210.1ms mean (stdev 22.4) vs 243.2ms baseline (stdev 27.8) -- 33.1ms faster, Welch's t = -3.59. This clears the noise floor comfortably at this sample size.
  • Correction/caveat: at the time of this specific measurement, an unrelated debugging mistake (an update-function-configuration --environment call that overwrites rather than merges the variable map) had inadvertently cleared DASH0_TOKEN_SECRET_ARN on both functions used here, so neither arm of this comparison actually made a Secrets Manager call. That doesn't invalidate the result for this fix specifically -- HTTPS_CLIENT's eager build has nothing to do with the token path, and both arms ran the identical (fast, no-op) "no token configured" branch of resolve_token(), so the 33.1ms delta is still cleanly attributable to the fix under test here. It does mean this number says nothing about Secrets Manager behavior; see PR perf: resolve the Dash0 token lazily, off the extension's cold-start critical path #99 for that measurement, done after the environment was fixed.

Test plan

  • cargo build --release
  • cargo test --release (325/325 passing)
  • Confirmed via debug-level logs that with_native_roots no longer runs before Registered with accountId
  • Confirmed against real AWS Lambda cold starts (n=15) that Init Duration drops ~33ms vs the current public release baseline

route::init() force-built HTTPS_CLIENT via Lazy::force ahead of extension
registration. Building it invokes hyper-rustls's with_native_roots(),
which synchronously parses the entire OS trust store -- confirmed via
debug logs during local profiling (162 certs on the profiling host).

HTTPS_CLIENT is only used later, for outbound OTLP export. Registration
gates when AWS lets the customer runtime start initializing, so this was
pure CPU-bound work sitting in front of that gate for no benefit. Letting
it stay lazy (first real use constructs it) removes it from the
cold-start critical path entirely.

Measured locally with perf-harness (native arm64, mock Extensions API):
median spawn-to-registered time dropped ~1ms (7.93ms -> 6.89ms) on top of
whatever the real trust-store parse costs on the actual Lambda runtime,
which is likely larger than this dev machine's 162-cert bundle.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant