diff --git a/CHANGELOG.md b/CHANGELOG.md index 9f9f86a6..cab8e0de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,39 +4,77 @@ All notable changes to this project are documented in this file. Format follows For narrative release notes written for operators and product owners, see [RELEASE_NOTES.md](RELEASE_NOTES.md). -## [Unreleased] +## [1.17.0] - 2026-09-02 -Production observability baseline. Every CloudWatch alarm in the stack now publishes to a single SNS topic β€” before this, the stack had 13 alarms and **none of them notified anybody**, and two of those watched metric names that exist in no CloudWatch namespace, so they had sat in `INSUFFICIENT_DATA` since creation and read as healthy. **Requires a CDK deploy**, and one manual step after it: subscribe your team to the new alarm topic (see [step-05-verify](.github/docs/deploy/step-05-verify.md#6-subscribe-to-platform-alarms-required--not-automated)). Subscriptions are deliberately not infrastructure-as-code. +Reliability, security and observability. Every CloudWatch alarm in the stack now notifies somebody β€” before this the stack had 13 alarms and **none of them were routed**, two of which watched metric names that exist in no namespace and had read as healthy since the day they were created. A production outage post-mortem (session `5f34d2b0`) drives four chat-path changes: Bedrock's transient faults are retried, a retry and a long silence are both visible to the user, and attachments a failed turn never delivered are re-sent. Four security findings are closed, including a High-severity OIDC login CSRF in the BFF auth flow and a privilege-escalating stored XSS in skill resources. The Bedrock Managed Knowledge Base migration β€” still off by default β€” gets eleven defects fixed from its first real runs in dev. **Requires a CDK deploy**, and one manual step after it: subscribe your team to the new alarm topic (see [step-05-verify](.github/docs/deploy/step-05-verify.md#6-subscribe-to-platform-alarms-required--not-automated)). ### πŸš€ Added -- **Single SNS alarm topic** (`{prefix}-alarms`) that every alarm routes to, encrypted with a customer-managed KMS key. The CMK is required, not a preference: CloudWatch cannot publish to a topic encrypted with the AWS-managed `alias/aws/sns` key, and that failure is silent β€” the alarm fires, the console shows it, the notification is dropped. Topic ARN published to SSM at `/{prefix}/observability/alarm-topic-arn` and as a CfnOutput -- **`AlarmFactory`** β€” the only sanctioned way to create an alarm. Attaches `AlarmActions` *and* `OKActions` as a consequence of being used, so an unrouted alarm now requires deliberately bypassing it. A source-level test fails the build if any file under `lib/` calls `new cloudwatch.Alarm()` directly -- **ALB alarms** (6): ELB 5xx, target 5xx, unhealthy hosts, target connection errors, rejected connections, and a streaming-aware p99 latency floor -- **ECS service alarms** (3): CPU, memory, and running-task-count below desired -- **DynamoDB alarms** (27): a combined read+write throttle alarm per table naming that table, plus one account-level `UserErrors` alarm -- **Lambda alarms** (21): errors and throttles across every runtime function, including `artifact-render`, `rag-ingestion` and the four kb-migration functions which previously had none, plus dead-letter-queue depth on the kb-ingestion DLQ -- **AI-path alarms** (9): Bedrock invocation throttles, server errors and `EstimatedTPMQuotaUsage` (the only *leading* indicator in the set β€” visible before throttling starts), AgentCore Memory hot-path errors and throttles, Gateway MCP errors and throttles, Code Interpreter session errors and concurrent-session count -- **`{prefix}-platform-health` dashboard** β€” one pane answering "is the platform healthy right now": traffic and errors, then saturation, then every alarm's current state. Links to the two existing dashboards rather than restating them, which keeps the stack at exactly 3 (CloudWatch's free ceiling) -- **`observability` configuration section** β€” 18 single-scalar tunables with `CDK_OBSERVABILITY_*` overrides, cost-conscious defaults, and validation that rejects retention values CloudWatch does not accept and X-Ray sampling rates given as percentages -- `.kiro/steering/observability.md` β€” the gotchas that silently break alarms, plus a first-response runbook for every alarm +- **Single SNS alarm topic** (`{prefix}-alarms`) that every alarm routes to, encrypted with a customer-managed KMS key. The CMK is required, not a preference: CloudWatch cannot publish to a topic encrypted with the AWS-managed `alias/aws/sns` key, and that failure is silent β€” the alarm fires, the console shows it, the notification is dropped. Topic ARN published to SSM at `/{prefix}/observability/alarm-topic-arn` and as a CfnOutput (#910) +- **`AlarmFactory`** β€” the only sanctioned way to create an alarm. Attaches `AlarmActions` *and* `OKActions` as a consequence of being used, so an unrouted alarm now requires deliberately bypassing it. A source-level test fails the build if any file under `lib/` calls `new cloudwatch.Alarm()` directly (#910) +- **ALB alarms** (6): ELB 5xx, target 5xx, unhealthy hosts, target connection errors, rejected connections, and a streaming-aware p99 latency floor (#910) +- **ECS service alarms** (3): CPU, memory, and running-task-count below desired (#910) +- **DynamoDB alarms** (27): a combined read+write throttle alarm per table naming that table, plus one account-level `UserErrors` alarm (#910) +- **Lambda alarms** (21): errors and throttles across every runtime function, including `artifact-render`, `rag-ingestion` and the four kb-migration functions which previously had none, plus dead-letter-queue depth on the kb-ingestion DLQ (#910) +- **AI-path alarms** (9): Bedrock invocation throttles, server errors and `EstimatedTPMQuotaUsage` (the only *leading* indicator in the set β€” visible before throttling starts), AgentCore Memory hot-path errors and throttles, Gateway MCP errors and throttles, Code Interpreter session errors and concurrent-session count (#910) +- **`{prefix}-platform-health` dashboard** β€” one pane answering "is the platform healthy right now": traffic and errors, then saturation, then every alarm's current state. Links to the two existing dashboards rather than restating them, which keeps the stack at exactly 3 (CloudWatch's free ceiling) (#910) +- **`observability` configuration section** β€” 18 single-scalar tunables with `CDK_OBSERVABILITY_*` overrides, cost-conscious defaults, and validation that rejects retention values CloudWatch does not accept and X-Ray sampling rates given as percentages (#910) +- **`model_retry` SSE event** β€” the backend now tells the SPA when it is retrying a failed model call instead of surfacing it. The loading indicator swaps its cycling phrases for a fixed amber notice, cleared on `message_start`/`done` (#905) +- **Stall indicator** β€” the loading indicator says "Still working…" after 30s of silence and "Still working β€” this is taking longer than usual." after 90s, driven by a client-side liveness stamp on every stream event. A known retry outranks the stall notice (#907) +- **Attachment recovery** β€” a turn that dies before the model reads its inline documents now re-sends them on the next turn, via a write-ahead marker on the session row that survives every way a turn can die, including a dropped stream. Bounded to the single following turn, discarded after an hour, and the user's own attachments always win (#905) +- **`BedrockTransientRetryStrategy`** β€” `ServiceUnavailableException`, `InternalServerException`, `ModelNotReadyException` and `ModelTimeoutException` are now retried under the existing four-attempt backoff. Only when they fire *before* the response stream opens, so a retry can never replay visible output. `RETRY_TRANSIENT_SERVICE_ERRORS=false` restores stock behavior without a deploy (#905) +- `.kiro/steering/observability.md` β€” the gotchas that silently break alarms, plus a first-response runbook for every alarm (#910) +- `scripts/local-dev/run-kb-migration.py`, `kb-doc-timings.py` and `kb-compare-engines.py` β€” in-process drivers and read-only diagnostics for the managed-KB state machine (#898, #899) -### πŸ› Fixed +### ✨ Improved -- **The two AgentCore Runtime alarms were watching metrics that do not exist.** They used namespace `bedrock-agentcore` with `InvocationCount` / `InvocationErrors` / `InvocationLatency`. Verified against the live account: that namespace is real but holds only the OpenTelemetry/Strands *application* metrics, and those three names exist in no namespace at all. Corrected to `AWS/Bedrock-AgentCore` with the verified `Resource` + `Operation` + `Name` dimension set, and split into four alarms β€” `SystemErrors` (AWS's fault) separated from `UserErrors` (ours), plus a new throttle alarm -- **The AgentCore latency alarm would have fired on healthy traffic.** Its 30-second threshold sat *below* the observed maximum: measured over 14 days, average turns run 3.0–4.5s with daily maxima reaching 24.4s, because the chat path is SSE and the runtime does not finish a request until the stream closes. Floors now default to 120s -- **The `agentcore-observability` dashboard's token-usage widget was always empty** β€” `InputTokens`/`OutputTokens` do not exist, and the token metrics that do exist in that namespace are Memory-strategy counters dimensioned by `StrategyId`, not model tokens. Removed; the header now points at the prompt-cache dashboard for real token accounting -- **X-Ray recorded a trace for every single agent invocation** in any deployment that never set `production` β€” `fixedRate` was `1.0` with a 50/sec reservoir on that branch, at $5 per million traces recorded. Now a single configured value defaulting to 1% with a 1/sec reservoir -- **The AgentCore Runtime's log group had no retention policy and grew forever.** It is created by the AgentCore service rather than CloudFormation, so a CDK `LogGroup` cannot set it. An `AwsCustomResource` calls `logs:PutRetentionPolicy` instead β€” idempotent, and it creates the group if the runtime has not yet been invoked +- A Bedrock 503 no longer reads as "I ran into a problem with the AI model". Two classifiers both keyed on `"throttl"` and neither recognized service-unavailable; a shared `is_service_unavailable_error` predicate now says the fault is on the provider's side and that we already retried (#905) ### ⚠️ Changed -- **Log retention is one configured value** (`observability.logRetentionDays`, default 30) applied to all 14 log groups through `logRetentionFor(config)`. Previously every construct hardcoded `ONE_WEEK`, except AgentCore Memory which used `ONE_MONTH` β€” differing silently rather than deliberately. A source guard fails the build on any hardcoded `RetentionDays` -- **No `config.production` branching in observability code.** This repo is forked by many institutions: a fork with one environment should not have to reason about a `production` boolean, and a fork with three should not be limited to two. Per-environment differences now live in the forker's deployment config as single values. Enforced by test +- **Log retention is one configured value** (`observability.logRetentionDays`, default 30) applied to all 15 log groups through `logRetentionFor(config)`. Previously every construct hardcoded `ONE_WEEK`, except AgentCore Memory which used `ONE_MONTH` β€” differing silently rather than deliberately. A source guard fails the build on any hardcoded `RetentionDays` (#910) +- **No `config.production` branching in observability code.** This repo is forked by many institutions: a fork with one environment should not have to reason about a `production` boolean, and a fork with three should not be limited to two. Per-environment differences now live in the forker's deployment config as single values. Enforced by test (#910) +- **Skill resource files are served as `attachment`, never `inline`**, with their media type re-derived from the filename at serve time rather than reflected from the stored row β€” which is what neutralizes rows written before the upload allowlist existed, so no data migration is needed. Both SPA callers fetch these over XHR for an in-app viewer; nothing navigates to the URL (#904) +- **Managed KB no longer pins the embedding model.** `embeddingModelType: CUSTOM` and `rerankingModelType: MANAGED` are mutually exclusive β€” AWS rejects the combination β€” and measurement settled it: the pin scored 1.00/0.982/0.952 (flat, unusable for a small context cap) against managed reranking's 0.413/0.199. On managed retrieval Bedrock embeds both sides, so query/index consistency is its invariant, not ours. `embeddingModelId`/`embeddingDimensions` are no longer recorded, since nothing here knows what Bedrock chose (#898) + +### πŸ› Fixed + +- **A completed answer could tell the model it was cut short.** The client's Stop writes `lastTurnInterrupted` immediately, but the server only observes the armed cancel on a lease heartbeat that sleeps 10s *before* its first check β€” so a turn finishing inside that window completed normally and left the marker behind. The next turn then prepended a note saying the user stopped the previous response and not to resume it, every clause of which was false, and it demonstrably steered the answer. A turn that reaches the end of the success path now clears the marker, whatever the client signalled (#909) +- **A session id could be forked across two users.** `_get_session_by_gsi` returned `None` both for "no such session" and for "exists, but owned by someone else", so opening someone else's `/s/{sessionId}` link created a *second* metadata row under the requester β€” the `attribute_not_exists(PK)` guard cannot catch it, because the new row has a different PK. Not a confidentiality bug (conversation content lives in AgentCore Memory keyed by actor id, so the second user saw an empty thread), but it duplicated the row, mis-attached spend, and made the original owner's session resolve non-deterministically afterwards. Both GSI lookups now scan for the caller's own row rather than reading `items[0]`, since forked rows already exist (#906) +- **Bedrock's transient service faults were never retried.** Strands' stock retry strategy covers `ModelThrottledException` only, and `BedrockModel` maps exactly one error code to it β€” every other fault re-raised as a raw botocore `ClientError`, so the configured four-attempt backoff never ran. A prod `ConverseStream` carrying two PDFs failed with `ServiceUnavailableException` after 95.6s, billed 56,440 uncached input tokens, and returned nothing (#905) +- **The two AgentCore Runtime alarms were watching metrics that do not exist.** They used namespace `bedrock-agentcore` with `InvocationCount` / `InvocationErrors` / `InvocationLatency`. Verified against the live account: that namespace is real but holds only the OpenTelemetry/Strands *application* metrics, and those three names exist in no namespace at all. Corrected to `AWS/Bedrock-AgentCore` with the verified `Resource` + `Operation` + `Name` dimension set, and split into four alarms β€” `SystemErrors` (AWS's fault) separated from `UserErrors` (ours), plus a new throttle alarm (#910) +- **The AgentCore latency alarm would have fired on healthy traffic.** Its 30-second threshold sat *below* the observed maximum: measured over 14 days, average turns run 3.0–4.5s with daily maxima reaching 24.4s, because the chat path is SSE and the runtime does not finish a request until the stream closes. Floors now default to 120s (#910) +- **The `agentcore-observability` dashboard's token-usage widget was always empty** β€” `InputTokens`/`OutputTokens` do not exist, and the token metrics that do exist in that namespace are Memory-strategy counters dimensioned by `StrategyId`, not model tokens. Removed; the header now points at the prompt-cache dashboard for real token accounting (#910) +- **X-Ray recorded a trace for every single agent invocation** in any deployment that never set `production` β€” `fixedRate` was `1.0` with a 50/sec reservoir on that branch, at $5 per million traces recorded. Now a single configured value defaulting to 1% with a 1/sec reservoir (#910) +- **The AgentCore Runtime's log group had no retention policy and grew forever.** It is created by the AgentCore service rather than CloudFormation, so a CDK `LogGroup` cannot set it. An `AwsCustomResource` calls `logs:PutRetentionPolicy` instead β€” idempotent, and it creates the group if the runtime has not yet been invoked (#910) +- **Managed KB: the legacy pipeline never stood down for a promoted knowledge base.** Routing exclusivity was enforced on one side only, so every document added to a promoted KB was indexed twice β€” and worse, two writers owned one `status` field and the last one won by luck. Observed both ways in dev: a PDF marked `complete` 65s before the managed KB could answer for it, and an image-only PDF marked `failed` by Docling while Bedrock's image extraction had indexed and served it correctly. `handler.py` now resolves the engine before writing any status and returns early for `managed` (#900) +- **Managed KB: deleting a document never removed it from the managed knowledge base.** `cleanup_service` removed the legacy S3 Vectors copy and the `DOC#` row and left the managed copy indexed forever β€” paid for at $5.00/GB-month against S3 Vectors' ~$0.15, silently consuming `top_k` slots that the status filter then dropped, with the one fail-open branch of that filter left load-bearing. A third engine-gated phase is conjoined into `all_succeeded`, and unlike the ingestion gate an unreadable record **fails** rather than assuming legacy, because reporting success on a failed managed delete would remove the row *and* leave the content (#900) +- **Managed KB: provisioning could strand a knowledge base unrecoverably.** `CreateKnowledgeBase` returns while still `CREATING` (measured 47–124s to `ACTIVE`), and `CreateDataSource` was called immediately; `awsKbId` was only written after *both* creates succeeded, so a failure between them left a record with no identifier and every later attempt was refused permanently on the unique name. Now: an explicit bounded wait for `ACTIVE`, `awsKbId` persisted the moment the create returns (guarded on `attribute_not_exists`), and adopt-by-name recovery on a name-collision `ConflictException` β€” skipping terminal statuses, so adoption cannot take a knowledge base that is being deleted (#889, #898) +- **Managed KB: ingestion gave up before indexing finished, and re-ingested while it ran.** The consumer polled a *retrieval* for 30s β€” smaller than the documented lower bound for PDF ingestion β€” and, because `IngestKnowledgeBaseDocuments` is fire-and-forget, every EventBridge redelivery re-submitted the document and restarted the work it was waiting for. A 1.5MB PDF sat at `uploading` indefinitely with a fully retrievable copy in the KB. `handle_object` now probes `GetKnowledgeBaseDocuments` first and branches on the real `DocumentStatus` enum, with a 600s budget that a cross-language test asserts fits inside the Lambda timeout (#901) +- **Managed KB: `indexedAt` was fabricated** β€” set from the local clock the moment the ingest call returned, presented as when indexing finished, minutes apart for a large document. Now Bedrock's own `updatedAt` (#901) +- **Managed KB: `verify` failed good migrations for being asked too early.** The canary queried a freshly-ingested document before it was retrievable (measured ~45s on a fresh KB, against a warm 0.75–1.03s figure) and treated the empty result as terminal. Now defers via `records.defer_verify`, bounded by `MAX_VERIFY_ATTEMPTS` (#898) +- **Admin "Discover from server" returned 403 for every IAM-authenticated MCP server.** `POST /admin/tools/discover` signs with the app-api task role, which held `AddPermission`/`RemovePermission`/`GetFunctionUrlConfig` but never `lambda:InvokeFunctionUrl` β€” so admins had to type each tool name by hand in every environment (#911) + +### πŸ”’ Security + +- **OIDC login CSRF / session fixation in the BFF auth flow (High).** `GET /auth/login` minted a `state`, stored it server-side, and issued no browser-side material at all β€” no state cookie, no PKCE, no nonce β€” and `GET /auth/callback` treated "this state exists in the store" as proof the request continued a login *this* browser started. An attacker could mint a state anonymously, authenticate at the IdP themselves, and lure a victim to `/auth/callback?code=&state=`, silently issuing the victim a live session for the attacker's account (reported against a `system_admin` identity). Login now returns a 32-byte secret in a `__Host-bff_oauth_state` cookie and commits only its SHA-256 digest, checked with `secrets.compare_digest` **before** the state-store lookup so a probe cannot burn an in-flight state. PKCE (S256) and OIDC nonce verification are added end-to-end as defense in depth. A state row carrying no digest fails closed (#903) +- **Privilege-escalating stored XSS in skill resources.** A zero-privilege user could upload bytes labelled `text/html` β€” the routes persisted the client-supplied multipart Content-Type verbatim and permitted an `.html` filename β€” and the read routes reflected that type with `Content-Disposition: inline` while the CloudFront `/api/*` behavior carried no response-headers policy. Because app-api shares an origin with the SPA, the file parsed as a top-level HTML document and its inline `