fix(otel): align dependencies with ADOT layer

[zhongkechen]

Summary

• Resolve and validate the latest regional ADOT Python Lambda layer before cloud deployment.
• Pass the layer ARN into the generated SAM template instead of pinning a regional layer version in the examples catalog.
• Cap opentelemetry-api, opentelemetry-sdk, and opentelemetry-exporter-otlp at 1.42.1, matching the stable OTel family bundled by ADOT 0.18.0 / layer :33.
• Document the temporary compatibility ceiling.

Root cause
OpenTelemetry 1.44.0 removed the deprecated Events API. ADOT still imports opentelemetry._events; when the function bundle supplies OTel 1.44.0 ahead of the Lambda layer on sys.path, ADOT initialization fails while importing _OTEL_PYTHON_EVENT_LOGGER_PROVIDER. Lambda continues with a proxy tracer provider, so spans are non-recording and never reach X-Ray.

Updating only the layer to :33 still reproduced both X-Ray failures because that ADOT release predates OTel 1.44.0. The dependency ceiling keeps the application and layer on the compatible 1.42.1 / 0.63b1 family.

Testing

• hatch run dev-otel:test (53 passed)
• hatch run dev-examples:test (79 passed, 2 skipped cloud-only X-Ray tests)
• hatch run examples:build resolves OTel API, SDK, exporters, and proto to 1.42.1, with semantic conventions 0.63b1
• hatch fmt --check
• Cloud example-test matrix passes on Python 3.11, 3.12, 3.13, and 3.14, including both X-Ray span assertions

[ayushiahjolia]

Are we going to revert the capped version change after issue is fixed?
What happens if there is a new version of ADOT layer based off newer version of Otel and we try to run it against older version of Otel (1.42.1)?

[zhongkechen]

Not sure. Let's see how it behaves when new version of ADOT is released