[watilde]

Follow-up to OHDSI/Atlas3#390, where reviewing the OpenTelemetry setup turned up this problem.

Problem

Batch jobs run on batchTaskExecutor. When the OpenTelemetry Java agent is attached, it carries the submitting HTTP request's context into that pool. So cohort generation SQL, and every other job's work, joins the trace of the request that launched it, and that trace can then run for minutes or hours. The agent's own Spring Batch instrumentation does not support Spring Batch 5 (opentelemetry-java-instrumentation#7306).

Change

All jobs are launched through JobTemplate → asyncJobLauncher, so the fix is in one place. asyncJobLauncher is now a TracingJobLauncher, a TaskExecutorJobLauncher that:

• wraps each launched Job so that execute() starts a new root span BatchJob <name> with setNoParent(), linked to the launching span. The job's SQL and other work become its children.
• records spring_batch.job.execution.id on the launching (request) span, so the job's trace can be found from the request.
• sets job.system, spring_batch.job.name, spring_batch.job.execution.id and spring_batch.job.status on the job span. Unsuccessful jobs are marked as errors without the exit description, which can carry values from failing SQL. STOPPED (cancel) is not an error.

The wrapper delegates the rest of the Job contract (name, restartability, validator, incrementer, toString() used in launcher logs).

Short request traces and per-job traces also make tail sampling (keep errors and slow traces) practical in a collector.

Dependencies

Only io.opentelemetry:opentelemetry-api is added, with its version from the Spring Boot BOM (1.49.0), plus opentelemetry-sdk-testing in test scope. The API is a no-op unless an agent or SDK is present, so deployments without telemetry are unaffected. With the agent attached, the agent bridges the application's API calls.

Verification

• TracingJobLauncherTest (8 tests) covers: the job runs in a new trace linked to the request; work inside the job is parented to the job span; the execution id appears on the request span; FAILED becomes an error with no description; STOPPED is not an error; an exception from the job is recorded and rethrown; a job launched outside a request is a root with no links; and the wrapper delegates the Job contract.
• Rebased onto master. The full mvn test passes there (171 unit + 35 integration tests, 0 failures), including VocabularyServiceIT, which Point VocabularyServiceIT at the source-keyed vocabulary endpoints #2578 fixed.
• End to end in the Atlas3 docker stack with agent 2.32.0 and Grafana LGTM: generating a cohort on EUNOMIA produced a 98 ms request trace (GET /WebAPI/cohortdefinition/{id}/generate/{sourceKey}, spring_batch.job.execution.id=1, no generation SQL), and a separate BatchJob generateCohort trace. That trace has no parent, a link to the request's trace and span, spring_batch.job.status=COMPLETED, and the generation's batch, source and SQL spans as children.

[chrisknoll]

This is feeling a little high risk since I had to make significant changes to batch execution because it was not handling cancel tasklets correctly. And I'm not sure what the implication to these 'new root spans' etc based on other fixes.

Considering we're far overdue for releasing 3.0, and we are training on the software in 5 days, let's leave this out for 3.0 and we can incorporate it in 3.1.

[watilde]

Makes sense, thanks. With the cancel/tasklet work just landed and training in 5 days, 3.0 isn't the time for this. I'll convert this to a draft and retarget it once there's a 3.1 branch, rebased on your batch execution changes, so we can look at how it interacts with them then. Until then, Atlas3's observability README documents the request/job trace merging as a known limitation.