Add production observability baseline with routed alarms

[colinmxs]

The stack had 13 CloudWatch alarms and none of them notified anybody. Three constructs carried a comment saying so. Two of those alarms were worse than silent: they watched metric names that exist in no CloudWatch namespace, so they sat in INSUFFICIENT_DATA from the day they were created, which an operator reads as healthy.

Every alarm now publishes to one SNS topic. 77 alarms, zero unrouted.

Routing is structural, not conventional. AlarmFactory attaches AlarmActions and OKActions as a consequence of being used at all, so an unrouted alarm requires deliberately bypassing it, and a source-level test fails the build if anyone calls new cloudwatch.Alarm() directly. The previous gap was not carelessness — the broken form was the shorter one.

Verified against the live account rather than documentation:

• The AgentCore alarms used namespace bedrock-agentcore with InvocationCount / InvocationErrors / InvocationLatency. That namespace is real but holds only the OpenTelemetry/Strands application metrics; those three names exist nowhere. Corrected to AWS/Bedrock-AgentCore with the verified Resource + Operation + Name dimensions, split so SystemErrors (AWS's fault) is separate from UserErrors (ours), plus a new throttle alarm.

• The latency threshold of 30s sat BELOW the observed maximum. Measured over 14 days, turns average 3.0-4.5s with daily peaks to 24.4s, because the chat path is SSE and the runtime does not finish a request until the stream closes. Floors now default to 120s. AgentCore Latency is in milliseconds while ALB TargetResponseTime is in seconds — a 1000x trap in either direction, so both units were confirmed with get-metric-statistics.

• DynamoDB has never throttled: ReadThrottleEvents, WriteThrottleEvents and SystemErrors all had zero metric streams, since every table is on-demand. Account-level UserErrors had live data with nothing watching it. That traded 78 alarms on signals that have never fired for 27 that include one firing today.

• No Cognito failure alarm and no Browser alarm. AWS/Cognito on the ESSENTIALS feature plan publishes only success metrics, and Browser has zero streams. Alarming on either would recreate the permanently-green alarm this change exists to remove. Both omissions are asserted by tests.

New coverage: ALB (6), ECS service (3), DynamoDB (27), Lambda (21), AI path (9), AgentCore Runtime (4). Plus a {prefix}-platform-health dashboard ordered by triage rather than service inventory, which links to the two existing dashboards instead of restating them — keeping the stack at exactly 3, the CloudWatch free ceiling.

Configuration is 18 single scalars with cost-conscious defaults. No config.production branching: this repo is forked by many institutions, so a fork with one environment should not reason about a production boolean and a fork with three should not be limited to two. Per-environment values live in the forker's deployment config. Enforced by test. The clearest case is X-Ray sampling, which was fixedRate 1.0 for any fork that never set production — a recorded trace for every agent invocation at $5/million. Now 1% by default.

Log retention becomes one configured value across all 15 groups, replacing 14 hardcoded literals (13 ONE_WEEK plus one ONE_MONTH that differed silently). Two gaps needed more than per-site edits: the AgentCore Runtime's group is created by the service rather than CloudFormation, so an AwsCustomResource calls PutRetentionPolicy on it; and CDK creates groups for its own machinery that default to 731 days and are declared nowhere here, so a LogRetentionAspect rewrites every group in the tree. The second was found by diffing a real cdk synth — unit tests missed it because a bare cdk.App lacks the cdk.json feature flags that materialise those groups.

Resource budget: 382 of CloudFormation's 500-resource limit, up from 308. This is a deliberate single-stack architecture with nowhere to spill, so the DynamoDB allocation was decided by measurement rather than by covering every documented metric. A guard test fails above 460.

Subscriptions are deliberately not infrastructure-as-code. Several teams need to hear about failures and their membership changes far more often than the infrastructure does; requiring a PR and a deploy to add one address is how a notification list goes stale and stops being trusted. A test asserts zero subscription resources exist so the decision cannot be quietly reversed. The required post-deploy step is documented in step-05-verify.

Verification: real cdk synth produces 382 resources, 77 alarms with 0 empty AlarmActions, 15 log groups all at the configured retention. Full suite 781 tests across 40 suites, sharded 4 ways. Guards proven non-vacuous by planting a file violating all three rules and confirming three tests fail.