You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Test gaps and decisions that the test audit found #83
The test audit on the test/test-audit-campaign branch found these gaps. They existed before the audit, and the audit did not remove their proof.
Decisions:
The second parameter of telemetryDev in @telemetry-dev/ai-sdk and @telemetry-dev/tanstack-ai (overrides), and MediaWrapperOverrides, have no caller outside the tests. Two options: document them as public options, or change the tests to decode the OTLP requests from a fetch spy and remove the parameter in a major release.
postOtlp and maybeGzip are public exports of @telemetry-dev/otel, but only the package itself uses them. A removal is a breaking change.
Coverage:
No test runs the ai@6 integration through the real ai-v6 dispatcher. Every ai@6 test sends hand-built events.
These AI SDK paths have no test: the gateway relabel in the ai@7 and media paths, the ai@7 step without onStepStart, executeTool, wrapGenerateVideo, wrapGetBatchStatus, wrapCancelBatch, the transcription duration, the video counts, and the wrapper passthrough without a key.
The onConfig phase guard in packages/tanstack-ai/src/middleware.ts has no test. Supported engines do not call onConfig after onStart.
No release check makes sure that each exports and publishConfig.exports target is in the packed tarball.
The Python core has no wire test for on_error with a server that returns an error.
Python compatibility code:
The Bedrock, Google GenAI, LiteLLM, and OpenRouter integrations keep fallbacks for span handle methods (capture_masked, record_output_chunk) that each supported core version has. The Anthropic and OpenAI integrations no longer have them.
CI tests each Python package against the core in this repository, also in the lowest-bounds job. If the telemetry-dev floor of an integration goes below the version that has these handle methods, no test fails.
In the Python core, CaptureBudget.from_client and the public metrics_recorder parameter of TelemetrySpanProcessor have no caller.
I'd like to take the coverage part of this. I checked it against main today. The tarball check is covered now by #155 and #160, and the Python on_error wire test by #87. The other three are still open. For example, removing the onConfig phase guard in tanstack-ai still passes all its tests.
My plan:
run the ai@6 integration through the real ai-v6 dispatcher
test the AI SDK paths listed above, from the ai@7 and media gateway relabel to the batch wrappers and the passthrough without a key
test the onConfig phase guard
I'll leave the decisions and the Python compatibility items to you. I'd send the tanstack-ai test first as a small PR, then the AI SDK tests in a second one.
Quick question before I send the AI SDK tests. They capture spans through the overrides argument, like the rest of the ai-sdk and tanstack-ai tests (#162 too). Which way are you leaning on that decision?
I'd lean toward keeping it. The SDKs send OTLP as protobuf, so a fetch spy would mean decoding spans and metrics from the request bodies in every test. Each test file also captures through one helper, so a later switch would mostly touch those helpers. If you'd rather remove it, I can take that rewrite too.
@lofoneh sorry for the slow answer. Let's keep overrides. Decoding protobuf in every test would cost more than the argument does, and it lets the tests check the real spans and metric points. #169 can keep capturing through it as it does now.
I'll document it as a supported way to capture spans and metrics in tests, in the ai-sdk and tanstack-ai READMEs and on the type in @telemetry-dev/otel, so it stops looking like a leftover.
Activity
lofoneh commentedon Oct 31, 2026
I'd like to take the coverage part of this. I checked it against main today. The tarball check is covered now by #155 and #160, and the Python
on_errorwire test by #87. The other three are still open. For example, removing theonConfigphase guard in tanstack-ai still passes all its tests.My plan:
ai-v6dispatcheronConfigphase guardI'll leave the decisions and the Python compatibility items to you. I'd send the tanstack-ai test first as a small PR, then the AI SDK tests in a second one.
lofoneh commentedon Oct 31, 2026
Quick question before I send the AI SDK tests. They capture spans through the
overridesargument, like the rest of the ai-sdk and tanstack-ai tests (#162 too). Which way are you leaning on that decision?I'd lean toward keeping it. The SDKs send OTLP as protobuf, so a
fetchspy would mean decoding spans and metrics from the request bodies in every test. Each test file also captures through one helper, so a later switch would mostly touch those helpers. If you'd rather remove it, I can take that rewrite too.test(sdk-tanstack-ai): cover the onConfig phase guard (#162)
aikins01 commentedon Nov 4, 2026
@lofoneh sorry for the slow answer. Let's keep
overrides. Decoding protobuf in every test would cost more than the argument does, and it lets the tests check the real spans and metric points. #169 can keep capturing through it as it does now.I'll document it as a supported way to capture spans and metrics in tests, in the ai-sdk and tanstack-ai READMEs and on the type in
@telemetry-dev/otel, so it stops looking like a leftover.test(ai): cover the AI SDK paths the test audit found