test(neuron): integration coverage + fixture for multi-runtime per-co…

[spanaik]

Description of the issue

Related: aws/amazon-cloudwatch-agent#2263

The Neuron per-core data-loss defect fixed in
aws/amazon-cloudwatch-agent# only appears on a node running two or more Neuron
runtimes, and no fixture in this repo produced that shape — so nothing here could have
caught it.

neuron-burn-core is replicas: 1 holding one of inf2.xlarge's two NeuronCores, so its
node has exactly one runtime. It also carries a self-referential
requiredDuringScheduling podAntiAffinity on neuron-test=true, which actively
prevents a second Neuron pod from joining it. Raising replicas to 2 does not help: the
anti-affinity pushes the second replica onto the idle node, giving one runtime per node
again and destroying the idle-node fixture.

Worth noting the two existing tests that look like they should catch the defect and do
not:

• TestNeuronRuntimeTagInResourceScope passes on the broken agent — it asserts the tag
  exists, is non-empty, and is absent from the datapoint, and all three hold. The collapsed
  resource still carries one (wrong) tag.
• TestNeuronNoDuplicateSeries passes because the collision happens in-agent: duplicate
  identities are merged before export, so the surface shows too few series rather than
  duplicated ones.

Cardinality is the signal, not duplication.

Description of changes

Fixture (terraform/eks/daemon/otel-neuron/main.tf) — adds a neuron-burn-peer
Deployment taking the second core, co-located with neuron-burn-core via podAffinity, so
one node runs two runtimes burning different cores.

Three constraints, each of which breaks an existing test if changed:

Constraint	Why
Not named neuron-burn-core-*	TestNeuronBurnWorkloadLabels and TestNeuronBurnCorePodColor match HasPrefix(pod, "neuron-burn-core") and assert workload.name == "neuron-burn-core" / pod-color == orange
No neuron-test: "true" label	neuron-burn-core's podAntiAffinity targets it, which would make the two mutually exclusive on a host and push neuron-burn-core onto the idle node
podAffinity, not just nodeSelector	both node groups use var.instance_type, so a nodeSelector alone could land it on the idle node, breaking the uncorrelated-results fixture

wait_neuron_monitor now waits for both burn deployments to be Available and fails if
they did not co-locate — a silent split would make every multi-runtime test vacuously
skip.

Tests (test/otel/neuron/multi_runtime_test.go) — 8 tests. The value-level one requires
≥2 busy cores reported by ≥2 runtime tags; the structural one requires cores × tags
series, which catches a collapse even on an idle node where values cannot distinguish it.
Also included: a tripwire asserting no series carries runtime_tag="DEFAULT", which is what
the in-agent-aggregation alternative stamps — changing that trade-off should mean changing
the test, not a silent behaviour swap.

Deliberately no assertion on distinct pod count: two runtimes need not be two pods, and
one pod can host both, so requiring it would encode this fixture's shape rather than the
invariant. (That over-specification was caught by running against a real cluster.)

License

By submitting this pull request, I confirm that you can use, modify, copy, and
redistribute this contribution, under the terms of your choice.

Tests

Validated against a live 2-runtime inf2.xlarge node in both directions:

• With the agent fix: 7 of 8 pass. The 8th is FixtureIsBothBurnWorkloads, which
  asserts the neuron-burn-core / neuron-burn-peer pod names and therefore needs this
  terraform cluster; the validation cluster used a differently-named workload.
• With the fix reverted on the live agent: all 8 fail. Confirmed the defect
  reproduced first — 2 series instead of 4, one surviving runtime tag, core 0 reporting 0
  while pinned at ~75% — then restored and re-ran to green.

That negative control also exposed a diagnostic problem, now fixed: all 8 failed via the
shared fixture guard rather than their own assertions, because the collapse destroys the
runtime tag itself, so >= 2 distinct tags never holds. The guard's message previously
blamed the fixture only. It now names both causes — the defect and a missing fixture —
with the kubectl command to tell them apart, because a collapsed node is structurally
indistinguishable from a genuine single-runtime node on the metrics surface.

gofmt clean, go vet -tags integration ./test/otel/neuron/ clean, terraform fmt -check
and terraform validate clean.

Not yet run: terraform apply. The fixture is validated by terraform validate and by
the assertions passing against an existing 2-runtime node, but the specific
neuron-burn-peer placement — that the device plugin grants one core to each pod and
podAffinity co-locates them — has not been exercised on a fresh cluster. The
wait_neuron_monitor co-location check is there to fail loudly if it doesn't.