From bd159083f3ab88ce135bfdcc5a77612475f3a3a3 Mon Sep 17 00:00:00 2001 From: PatternCatalyst Date: Fri, 23 Oct 2026 07:18:39 -0400 Subject: [PATCH] docs(site): Geewax voice pass on all 34 chapters Apply the updated lgtm-professional-voice rules to every chapter: - Rule 12 (em dash, zero tolerance): recast every em dash in prose, figure captions, bullet/glossary term definitions, and the verification-status footer lines to colon/semicolon/comma/parentheses or a split sentence per the clause relationship. Em dashes inside fenced code are left byte-identical. - Rule 15 (no rule-of-three): trimmed rhetorical triads; kept genuine factual enumerations. - Rules 1/2/7/8/9: dropped generated-text tells and padding, converted ALL-CAPS emphasis, cut hedges. No content lost (net -1 line across 34 files; recasts, not deletions). validate_site.py clean; 0 ban-tier in _docs; Jekyll build green. --- _docs/00-setting-up.md | 32 +-- _docs/01-cloud-native-principles.md | 54 ++-- _docs/02-communications.md | 34 +-- _docs/03-composition.md | 50 ++-- _docs/04-data.md | 62 ++-- _docs/05-event-driven.md | 266 +++++++++--------- _docs/06-stream-processing.md | 52 ++-- _docs/07-workflows-and-jobs.md | 48 ++-- _docs/08-api-management.md | 42 +-- _docs/09-api-registry.md | 32 +-- _docs/10-api-metadata.md | 32 +-- _docs/11-observability.md | 64 ++--- _docs/12-security.md | 82 +++--- _docs/13-anti-patterns.md | 40 +-- _docs/14-appendix-a-protocols.md | 56 ++-- _docs/15-appendix-b-versioning.md | 62 ++-- _docs/16-appendix-c-websockets.md | 74 ++--- _docs/17-appendix-d-sagas.md | 35 ++- _docs/18-appendix-e-errors.md | 86 +++--- _docs/19-appendix-f-ddd-hexagonal.md | 74 ++--- _docs/20-appendix-g-coupling.md | 102 +++---- _docs/21-appendix-h-shutdown.md | 54 ++-- _docs/22-appendix-i-l7-routing.md | 82 +++--- _docs/23-appendix-j-dotnet-toolkit.md | 36 +-- ...24-appendix-k-monolith-to-microservices.md | 94 +++---- _docs/25-appendix-l-caching.md | 72 ++--- _docs/26-appendix-m-failure-modes.md | 112 ++++---- _docs/27-appendix-n-feature-flags.md | 42 +-- _docs/28-appendix-o-newman.md | 44 +-- _docs/29-appendix-p-dotnet-on-linux.md | 98 +++---- _docs/29-glossary.md | 24 +- _docs/30-appendix-q-blazor-signalr.md | 30 +- _docs/31-appendix-r-minikube.md | 6 +- _docs/32-appendix-s-openshift-local.md | 6 +- 34 files changed, 1039 insertions(+), 1040 deletions(-) diff --git a/_docs/00-setting-up.md b/_docs/00-setting-up.md index c6b5677d..e661a229 100644 --- a/_docs/00-setting-up.md +++ b/_docs/00-setting-up.md @@ -10,7 +10,7 @@ AI coding tools turn intent into code at a rate that wasn't possible before. Ask one, in a sentence, for a microservices decomposition, a saga, a cache layer, a strangler-fig wrapper, or a CQRS write side, and you get back something plausible enough to compile, deploy, and run. The bottleneck has moved. It is no longer -typing; it is *evaluation* — telling a good answer from a confident, plausible, +typing; it is *evaluation*: telling a good answer from a confident, plausible, wrong one. That is what this book is about. The patterns here are not nostalgic @@ -24,17 +24,17 @@ amplifier; the patterns decide which direction it amplifies. Every concrete choice in this book applies four ideas. Keep them in view: -- **The least-worst decision.** Every architecture choice is a set of trades — +- **The least-worst decision.** Every architecture choice is a set of trades: latency for throughput, consistency for availability, simplicity for flexibility. The job is not to find the option with no downside; it is to make the trades visible and explicit. - **Risk management.** You manage the *distribution* of failures, not their existence. Failure doesn't go away; you choose where it lands and how it's contained. -- **No free lunch.** Every pattern — a cache, a saga, eventing, decomposition — - has a cost: a stale-read window, a compensation path, a coordination tax, - operational complexity. The question is whether you are buying the right thing - for what you are paying. +- **No free lunch.** Every pattern, whether a cache, a saga, eventing, or + decomposition, has a cost: a stale-read window, a compensation path, a + coordination tax, operational complexity. The question is whether you are + buying the right thing for what you are paying. - **No problem cannot be made worse.** A confidently-wrong intervention amplifies the original pain. AI tooling makes confident interventions cheap, which is exactly why this maxim is needed. @@ -42,17 +42,17 @@ them in view: ## Three layers of patterns literacy "Pattern" spans three layers, and they build on each other. **Design patterns** -are code-level vocabulary — the Gang of Four: factory, strategy, observer, -repository. **Architecture patterns** describe how an application fits together — +are code-level vocabulary, the Gang of Four: factory, strategy, observer, +repository. **Architecture patterns** describe how an application fits together: hexagonal, microservices, BFF, federation, event-driven. **System design -patterns** — the focus of this book — span services, data, and runtime: sagas, +patterns**, the focus of this book, span services, data, and runtime: sagas, the outbox, CQRS, caching, the strangler fig, back-pressure, the bulkhead, L7 routing. {% include excalidraw.html file="00-patterns-literacy" - alt="Three stacked layers: design patterns (Gang of Four) at the bottom, architecture patterns in the middle, and system design patterns — the focus of this book — on top, with arrows showing each lower layer informing the next" - caption="Figure 0.1 — Each lower layer informs the one above it" %} + alt="Three stacked layers: design patterns (Gang of Four) at the bottom, architecture patterns in the middle, and system design patterns, the focus of this book, on top, with arrows showing each lower layer informing the next" + caption="Figure 0.1: Each lower layer informs the one above it" %} The layers are not independent. You can't reason about a saga (system layer) without knowing what a state machine and a compensation are (architecture and @@ -62,19 +62,19 @@ what lets you ask an AI for the right thing and judge what it returns. ## The running example -Every concern in this book is designed against one small system — the same +Every concern in this book is designed against one small system: the same example, re-examined from each angle. Anchor on it now; we return to this picture many times. {% include excalidraw.html file="00-running-example" alt="A client calls order-service over REST; order-service calls inventory, payment, and shipping over gRPC and emits an order.placed event to Kafka; a notification service consumes that event; a graphql-gateway composes reads across the internal services" - caption="Figure 0.2 — Five services and a gateway, each exposing only the protocols that fit its role" %} + caption="Figure 0.2: Five services and a gateway, each exposing only the protocols that fit its role" %} The point of the picture is that **each service exposes only the protocols that fit its role**, not a uniform surface: -- **order-service** owns the order lifecycle and exposes REST at the edge — the +- **order-service** owns the order lifecycle and exposes REST at the edge: the one public, client-facing surface. - **inventory**, **payment**, and **shipping** are internal. They are reached by gRPC for low-latency service-to-service calls, and they contribute fields to @@ -95,10 +95,10 @@ choice; then the synchronous surface (communications, composition); then data ownership; then the asynchronous backbone (events, streams); then the operational concerns (workflows and the platform planes); and finally anti-patterns as a reflective close. The deep-dive appendices at the end go deeper on the questions -that always come up — protocols, versioning, WebSockets, sagas, and more — and +that always come up (protocols, versioning, WebSockets, sagas, and more) and can be read on demand. --- -*Verification status: conceptual chapter — no runnable code. The patterns named +*Verification status: conceptual chapter; no runnable code. The patterns named here are defined and applied in their own chapters, where the code ships with a runnable example per language.* diff --git a/_docs/01-cloud-native-principles.md b/_docs/01-cloud-native-principles.md index 9523e3c9..9274b899 100644 --- a/_docs/01-cloud-native-principles.md +++ b/_docs/01-cloud-native-principles.md @@ -2,33 +2,33 @@ title: "Cloud-Native Principles" order: 1 part: "Foundations & the system" -description: "Cloud-native as a set of runtime properties, not a stack of tools — the six pillars, leaning on the platform, design-first contracts, and the twelve-factor app reframed by who owns each factor." +description: "Cloud-native as a set of runtime properties, not a stack of tools: the six pillars, leaning on the platform, design-first contracts, and the twelve-factor app reframed by who owns each factor." duration: 17 minutes --- Before any pattern, the principles that justify them. "Cloud-native" is not a -list of products you adopt; it is a set of **runtime properties** a system has — +list of products you adopt; it is a set of **runtime properties** a system has, and every later choice in this book either earns or spends one of them. ## Six pillars -A cloud-native service exhibits six runtime properties — and the platform exists to +A cloud-native service exhibits six runtime properties, and the platform exists to make each of them the default: {% include excalidraw.html file="01-six-pillars" alt="A central hub labelled Cloud-native API on Kubernetes radiates spokes to six surrounding pillars: Disposable (fast start, clean stop, no local state), Resilient (timeouts, retries, circuit breakers, graceful degradation), Automatable (declarative config, GitOps and CRDs, self-healing), Observable (traces, metrics, logs, health, RED/USE signals), Elastic (scale to load, scale to zero with KEDA), and Loosely coupled (contracts not calls, async where it fits, independent deploy). A note reads: cloud-native is a set of runtime properties an API must exhibit to thrive on Kubernetes, not a checklist of tools." - caption="Figure 1.1 — Cloud-native in six pillars: the runtime properties an API must exhibit on Kubernetes" %} + caption="Figure 1.1: Cloud-native in six pillars: the runtime properties an API must exhibit on Kubernetes" %} Read each as a behaviour you design toward, not a product you install: -- **Disposable** — twelve-factor: fast start, clean stop, no local state, so the +- **Disposable**: twelve-factor: fast start, clean stop, no local state, so the platform can move or restart it at will. -- **Observable** — it emits traces, metrics, and logs, and exposes health probes. -- **Resilient** — timeouts, retries, circuit breakers, graceful degradation. -- **Elastic** — it scales to load, and to zero. -- **Automatable** — declarative config, GitOps, self-healing. -- **Loosely coupled** — contracts not calls, async where it fits. +- **Observable**: it emits traces, metrics, and logs, and exposes health probes. +- **Resilient**: timeouts, retries, circuit breakers, graceful degradation. +- **Elastic**: it scales to load, and to zero. +- **Automatable**: declarative config, GitOps, self-healing. +- **Loosely coupled**: contracts not calls, async where it fits. These six are the rubric for every pattern in this book. Hold each design decision up to them. @@ -46,11 +46,11 @@ default; your job is to stay out of its way and use it. ## Design-first: the contract precedes the code Design-first is the cultural prerequisite for everything that follows. The -contract — OpenAPI, `.proto`, AsyncAPI — is agreed and registered *before* code +contract (OpenAPI, `.proto`, AsyncAPI) is agreed and registered *before* code is written, owned by the producing domain, and gated in CI. Consumers code against the contract and its generated stubs, never against your database. That is what delivers independent deployability: the contract *is* the -service boundary. Skip it and you get distributed coupling — the worst of both +service boundary. Skip it and you get distributed coupling: the worst of both worlds. ## The twelve-factor app, revisited @@ -62,47 +62,47 @@ Here are all twelve, one line each. {% include excalidraw.html file="01-twelve-factors" alt="A grid of the twelve factors, each with a number, name, and one-line summary: 01 codebase (one repo, many deploys), 02 dependencies (declare and isolate), 03 config (read from the environment), 04 backing services (treat as attached resources), 05 build/release/run (keep the stages separate), 06 processes (stateless, share-nothing), 07 port binding (export a service via a port), 08 concurrency (scale out with processes), 09 disposability (fast start, graceful stop), 10 dev/prod parity (keep environments alike), 11 logs (treat as event streams), 12 admin processes (run as one-off tasks)." - caption="Figure 1.2 — The twelve factors at a glance" %} + caption="Figure 1.2: The twelve factors at a glance" %} The useful way to read it today is not to recite all twelve, but to split them by -**who owns each one** — the few you still write, versus the many the platform now +**who owns each one**: the few you still write, versus the many the platform now provides built in. {% include excalidraw.html file="01-twelve-factor-ownership" alt="Two columns: the factors you still write per service (config from the environment, stateless processes, fast clean start and stop, logs as event streams, health and readiness probes) versus the factors the platform provides (build/release/run, concurrency and scaling, disposability, port binding, backing services, retries and timeouts)" - caption="Figure 1.3 — The twelve factors, split by who owns them" %} + caption="Figure 1.3: The twelve factors, split by who owns them" %} The left column is your job; the right column is the platform's. The whole point of a cloud-native service is to do the left column well and trust the right -column — which is exactly what this code does. +column, which is exactly what this code does. ## Twelve factors, mapped to our system Grouped by what each factor *asks of you*, the twelve land on concrete choices in -the running example — and most resolve to a platform default rather than code you +the running example, and most resolve to a platform default rather than code you write. {% include excalidraw.html file="01-factors-mapped" alt="Five rows mapping groups of factors to the system: build and ship becomes built once, run anywhere (one repo, pinned requirements, one image per release); configure becomes config from the environment (pydantic-settings, per-profile, secrets injected); run on the platform becomes disposable, stateless, scaled (KEDA scale-out and to-zero, fast start/stop, behind Istio); attach resources becomes backing services as swappable URLs (Postgres, Kafka via Strimzi, Apicurio, never in-process); operate becomes signals and admin off the hot path (OpenTelemetry to LGTM, admin as Jobs and CronJobs). Each row carries a section reference." - caption="Figure 1.4 — The twelve factors grouped by ask, mapped to where each lives in the system" %} + caption="Figure 1.4: The twelve factors grouped by ask, mapped to where each lives in the system" %} Five groups cover all twelve. *Build & ship* (codebase, dependencies, build/release/run, parity) becomes one repo and one image per release. *Configure* (config) is read from the environment. *Run on the platform* (stateless processes, concurrency, disposability, -port binding) is the runtime's job — KEDA scales it, Istio fronts it, the platform +port binding) is the runtime's job: KEDA scales it, Istio fronts it, the platform disposes of it. *Attach resources* (backing services) makes Postgres, Kafka, and the schema registry swappable URLs, never libraries compiled in. And *operate* (logs, admin processes) pushes telemetry and one-off tasks off the request hot path, into OpenTelemetry and Kubernetes Jobs. The section tags on each row point to where that -group is treated in depth — most of the twelve are now adopted, not authored. +group is treated in depth: most of the twelve are now adopted, not authored. ## A twelve-factor service, ready for the platform Two factors dominate the code you write: **config from the environment** (factor III) and **health and readiness probes** the platform calls. Here it is -in each stack — pick your tab. +in each stack: pick your tab. {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -273,11 +273,11 @@ The same two ideas appear in every tab: variables, .NET reads `ConnectionStrings__Default` through its configuration (env vars override `appsettings.json`), Python binds a `BaseSettings` class to env vars, and C++ and Go read `std::getenv` and `os.Getenv`. One - image, many environments — factor III, and the thing that lets the same + image, many environments: factor III, and the thing that lets the same artifact run in dev, staging, and production unchanged. -2. **Two probes, not one.** *Liveness* answers "is the process up?" — if it fails, +2. **Two probes, not one.** *Liveness* answers "is the process up?" If it fails, the platform restarts the pod. *Readiness* answers "are my dependencies - reachable?" — if it fails, the platform stops routing traffic to this instance + reachable?" If it fails, the platform stops routing traffic to this instance but leaves it running. Conflating the two is a classic mistake: a readiness blip should drain traffic, not trigger a restart loop. The probe reads the **status code**, not the body: Kubernetes counts any answer from 200 to 399 as @@ -288,13 +288,13 @@ The same two ideas appear in every tab: framework routes to the same pair, and Kubernetes calls them the same way. Notice what *isn't* here: no retry logic, no scaling code, no port-binding -plumbing. Those are the right column of Figure 1.3 — the platform's job. +plumbing. Those are the right column of Figure 1.3: the platform's job. ### Cross-check it yourself With the service running, the probes are just HTTP. Hit liveness and readiness with `curl` and confirm liveness returns healthy while the database is down but -readiness does *not* — that difference is the whole point. `curl -i +readiness does *not*. That difference is the whole point. `curl -i localhost:8080/readyz` should flip from `200` ready to `503` down as you stop the database and back as you start it, without the process ever restarting. @@ -302,5 +302,5 @@ The code is in [`examples/01-cloud-native-principles/`](https://github.com/patte and runs it; its `README.md` covers what it does and how to drive it. --- -*Verification status: verified — [`examples/01-cloud-native-principles/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/01-cloud-native-principles/) passes 13/13 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified; [`examples/01-cloud-native-principles/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/01-cloud-native-principles/) passes 13/13 checks in each of the six languages on Docker Engine (2026-10-13) (health probes, config from environment, CRUD, readiness flip on dependency failure).* diff --git a/_docs/02-communications.md b/_docs/02-communications.md index c44d7e58..a6365741 100644 --- a/_docs/02-communications.md +++ b/_docs/02-communications.md @@ -2,7 +2,7 @@ title: "Communications" order: 2 part: "Foundations & the system" -description: "Four interaction styles and four defaults — REST at the edge, gRPC for internal calls, GraphQL for composed reads, async events for facts — and pushing resilience onto the mesh instead of into your code." +description: "Four interaction styles and four defaults (REST at the edge, gRPC for internal calls, GraphQL for composed reads, async events for facts) and pushing resilience onto the mesh instead of into your code." duration: 20 minutes --- @@ -11,28 +11,28 @@ for everything. There are four interaction styles, and each has a default that fits it. Real systems use all four; the skill is matching each call to the right one. -- **REST** for the public, edge surface — reach and caching matter most. -- **gRPC** for internal service-to-service calls — latency and streaming matter. +- **REST** for the public, edge surface: reach and caching matter most. +- **gRPC** for internal service-to-service calls: latency and streaming matter. - **GraphQL** for client-driven reads that compose many sources in one round trip. - **Async events** for decoupled facts that fan out to whoever cares. {% include excalidraw.html file="02-pick-protocol" alt="Four columns of interaction styles, each tagged sync or async: REST/HTTP (public and edge surface, wide reach, cacheable, resource-shaped, OpenAPI-described), sync; gRPC (internal service-to-service, low latency, streaming, schema-first proto, grpc + protobuf), sync; GraphQL (client-driven reads, compose many sources, one round-trip, schema (SDL) + resolvers), sync; and Async/events (fire-and-forget facts, decoupled in time, fan-out to many, Kafka/Pulsar), async." - caption="Figure 2.1 — Four interaction styles, four defaults — three synchronous, one asynchronous" %} + caption="Figure 2.1: Four interaction styles, four defaults: three synchronous, one asynchronous" %} Read it by the sync/async tags along the bottom: REST, gRPC, and GraphQL are all -request/response — the caller waits — while events are fire-and-forget. Three of the +request/response (the caller waits) while events are fire-and-forget. Three of the four couple the caller to the callee's availability for the duration of the call; only the fourth breaks that coupling. Choosing a style is therefore choosing how much -availability you are willing to inherit — the thread that runs through this chapter. +availability you are willing to inherit: the thread that runs through this chapter. REST wins where reach and caching matter; gRPC where an internal hop must be fast and strongly typed; GraphQL where one client read would otherwise be a handful of round trips; and events wherever immediacy is not strictly required. Two interaction styles are deferred from this chapter. **GraphQL** gets its own treatment in **03 · Composition**, because GraphQL is fundamentally a -composition pattern — a gateway that fans out to REST and gRPC back-ends — and that +composition pattern: a gateway that fans out to REST and gRPC back-ends, and that context matters more than the wire format. **WebSockets** are covered in **Appendix C · WebSockets at Scale**, because long-lived, stateful sockets behave so differently on Kubernetes (pinned to a pod, fighting the HPA, needing a backplane) that @@ -40,7 +40,7 @@ mixing them into this chapter would bury the scaling concerns that define them. ## REST at the edge -This is the public surface — in our system, `order-service`. Four things teams +This is the public surface: in our system, `order-service`. Four things teams routinely get wrong, and the code gets right: resource-shaped URLs (nouns, not verbs), request and response contracts validated before bad input reaches your logic, correct status codes (**201** on create, not 200), and **cursor** @@ -221,7 +221,7 @@ automatically, which becomes the registered contract. ## gRPC for internal calls -Between internal services — `order-service` calling `inventory` — reach for gRPC: +Between internal services (`order-service` calling `inventory`), reach for gRPC: binary, multiplexed over HTTP/2, fast, and streaming-capable. The crucial point is that the **`.proto` is the contract**, and it lives in the registry exactly like OpenAPI. It is the same file for every language; each side generates typed stubs @@ -237,7 +237,7 @@ message ReserveRequest { string sku = 1; int32 quantity = 2; } message ReserveReply { bool reserved = 1; int32 remaining = 2; } ``` -The field numbers (`= 1`, `= 2`) are the wire contract, not cosmetic — **Appendix B** +The field numbers (`= 1`, `= 2`) are the wire contract, not cosmetic: **Appendix B** explains exactly why they can never be reused. Generate stubs with `protoc` or `buf`, and both sides share types. @@ -339,7 +339,7 @@ func (s *inventoryServer) ReserveStock( } ``` -There is no socket code, no serialisation, no HTTP/2 framing — the generated base class +There is no socket code, no serialisation, no HTTP/2 framing: the generated base class handles all of it. Your method receives a typed `ReserveRequest` and returns a typed `ReserveReply`, and every language fills in the same one `ReserveStock` contract. @@ -354,7 +354,7 @@ that were down catch up from the log later. {% include excalidraw.html file="02-sync-vs-async" alt="Top row: order makes a synchronous call to payment and blocks on it, so if payment is down the order fails. Bottom row: order emits an order.placed fact to Kafka, and consumers catch up from the log independently" - caption="Figure 2.2 — A blocking call couples availability; a published fact decouples it" %} + caption="Figure 2.2: A blocking call couples availability; a published fact decouples it" %} The rule: reach for async wherever immediacy isn't strictly required. A call you *must* wait for is a call whose availability you have inherited. @@ -362,7 +362,7 @@ The rule: reach for async wherever immediacy isn't strictly required. A call you ## Resilience as configuration, not per-call code Your app speaks plain HTTP or gRPC to localhost; the Istio sidecar (Envoy) -intercepts and adds mTLS, retries, timeouts, and telemetry **on the wire** — +intercepts and adds mTLS, retries, timeouts, and telemetry **on the wire**, uniformly, for every service, with zero application code. This ties straight back to the principles chapter: resilience and security become platform defaults, not per-service reimplementations. @@ -370,7 +370,7 @@ per-service reimplementations. {% include excalidraw.html file="02-mesh-handles-wire" alt="Two pods, each containing an app (a REST or gRPC server speaking plain localhost) beside an istio-proxy Envoy sidecar that handles mTLS, retries, timeouts, and traces. The two sidecars communicate over mutual TLS, encrypted and authenticated. The istiod control plane distributes policy, SPIFFE identity, and telemetry config to both pods." - caption="Figure 2.3 — The mesh moves cross-cutting concerns off your code: sidecars handle mTLS, retries, timeouts, and telemetry, configured by istiod" %} + caption="Figure 2.3: The mesh moves cross-cutting concerns off your code: sidecars handle mTLS, retries, timeouts, and telemetry, configured by istiod" %} Each pod runs your service next to an Envoy sidecar. Your code talks plain HTTP or gRPC to localhost; the sidecar intercepts every call, adds mutual TLS so traffic between pods @@ -397,14 +397,14 @@ spec: ``` The critical word is **bounded**: `attempts: 2`, not infinite. Unbounded -client-side retries are how a brief blip becomes a full outage — every caller +client-side retries are how a brief blip becomes a full outage: every caller piles onto the struggling service in a retry storm. Putting the policy in the mesh means one bounded, observable rule for every caller, instead of each team re-inventing (and mis-tuning) retries in code. ### Cross-check it yourself -Use plain tools. `curl` and Postman exercise the REST contract — confirm a create +Use plain tools. `curl` and Postman exercise the REST contract: confirm a create returns **201** and that a bad body returns **400** before your logic runs. `ghz` throws gRPC load at `inventory` so you can watch the Istio timeout and bounded retries behave under pressure rather than trusting they do. @@ -413,5 +413,5 @@ The code is in [`examples/02-communications/`](https://github.com/patterncatalys it; its `README.md` covers what it does and how to drive it. --- -*Verification status: verified — [`examples/02-communications/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/02-communications/) passes 10/10 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified: [`examples/02-communications/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/02-communications/) passes 10/10 checks in each of the six languages on Docker Engine (2026-10-13) (REST create and validation (201, 422), cursor pagination, order confirmation and rejection driven by the gRPC stock reservation, a GraphQL orders query, and the order.placed Kafka topic).* diff --git a/_docs/03-composition.md b/_docs/03-composition.md index 79aaac67..839a68c9 100644 --- a/_docs/03-composition.md +++ b/_docs/03-composition.md @@ -2,12 +2,12 @@ title: "Composition" order: 3 part: "Foundations & the system" -description: "Composing many back-ends behind one schema without coupling them — a GraphQL gateway that fans out to REST and gRPC, the resolver that makes it concrete, and the three places aggregation belongs." +description: "Composing many back-ends behind one schema without coupling them: a GraphQL gateway that fans out to REST and gRPC, the resolver that makes it concrete, and the three places aggregation belongs." duration: 18 minutes --- A client rarely wants what a single service holds. It wants an order *and* its -live stock *and* its shipment status — data spread across services. The wrong fix +live stock *and* its shipment status: data spread across services. The wrong fix is to have `order-service` call `inventory` call `pricing` in a chain. The right fix is to push the aggregation **outward, to the edge**, and keep the services unaware of each other. @@ -16,7 +16,7 @@ unaware of each other. A GraphQL gateway presents **one schema** to clients and fans out to REST and gRPC back-ends, then joins and shapes the result. The client makes a single round trip -and asks for exactly the fields it needs — no more, no less. +and asks for exactly the fields it needs. ```graphql # the gateway's schema — one contract for every client @@ -49,13 +49,13 @@ the composition. {% include excalidraw.html file="03-composition" alt="A client sends one GraphQL query to the graphql-gateway; the gateway calls order-service over REST for the order fields and inventory over gRPC for the stock field, then joins and returns one result" - caption="Figure 3.1 — One query in; the gateway fans out to REST and gRPC and joins" %} + caption="Figure 3.1: One query in; the gateway fans out to REST and gRPC and joins" %} ## The resolver makes it concrete Federation stops being abstract at the resolver. The top-level `order` query calls `order-service` over REST; the `Order.stock` field resolves *on demand* by -calling `inventory` over gRPC. **Each field resolves against its owning service** — +calling `inventory` over gRPC. **Each field resolves against its owning service**: the gateway is only the composition layer. {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -194,31 +194,31 @@ func (r *Resolver) attachStock(ctx context.Context, orders []*model.Order) { The top-level `order` resolver does a REST call; the `stock` field resolver does a gRPC call, and only runs *when the client asks for `stock`*. That on-demand -property is the point of GraphQL — unrequested fields cost nothing. +property is the point of GraphQL: unrequested fields cost nothing. The trap to know is **N+1**: resolve `stock` naively for a list of 100 orders and -you fire 100 gRPC calls. Production gateways batch with a **DataLoader** — collect +you fire 100 gRPC calls. Production gateways batch with a **DataLoader**: collect the keys requested in one tick, make one batched back-end call. HotChocolate ships one; Strawberry, the JVM servers, and the others have equivalents. When you write a per-field resolver that calls another service, assume you will need batching. -## Three ways to compose — and where each belongs +## Three ways to compose, and where each belongs The resolver is one shape of a more general idea. There are three, and all -three push aggregation outward — toward the client as coupling needs loosen. +three push aggregation outward, toward the client as coupling needs loosen. {% include excalidraw.html file="03-three-ways" alt="Three columns with tags: API Gateway (routing, auth, rate-limit; no business logic; Istio ingress) tagged thin edge; Backend-for-Frontend (one BFF per client type; client-specific shaping; owns aggregation) tagged client-shaped; and GraphQL federation (schema stitched across domains that own subgraphs; gateway plans query) tagged read-optimised. Aggregation moves toward the client as coupling needs loosen." - caption="Figure 3.2 — Three places to compose: a thin API gateway, a client-shaped BFF, or a read-optimised GraphQL federation" %} + caption="Figure 3.2: Three places to compose: a thin API gateway, a client-shaped BFF, or a read-optimised GraphQL federation" %} -- **API gateway** — one composed schema for everyone. Simple, central, one contract; +- **API gateway**: one composed schema for everyone. Simple, central, one contract; it routes, authenticates, and rate-limits at the edge (our Istio ingress) but holds no business logic. -- **Backend-for-frontend (BFF)** — a tailored aggregate per client type, so the +- **Backend-for-frontend (BFF)**: a tailored aggregate per client type, so the web and mobile apps each get a surface shaped for them rather than a lowest-common-denominator one. The BFF owns the aggregation for its client. -- **GraphQL federation** — a distributed graph, where each service owns its slice +- **GraphQL federation**: a distributed graph, where each service owns its slice of one logical schema and a gateway plans and stitches the query across subgraphs. Pick by how much each client's needs diverge: a gateway when one shape serves everyone, @@ -226,46 +226,46 @@ a BFF when clients differ enough to deserve their own surface, and federation wh domains must present as one graph and reads dominate. The anti-pattern to name explicitly is the opposite of all three: -**service-to-service call chains**, where `order` calls `inventory` calls -`pricing` calls… — synchronous, deeply coupled, and impossible to reason about. It +**service-to-service call chains**, where `order` calls `inventory`, which calls +`pricing`, in a synchronous, deeply coupled chain that's impossible to reason about. It rebuilds a distributed monolith. Composition belongs at the edge, not buried in a chain of internal calls. ## Composition through the coupling lens -**Appendix G** lays out Khononov's three coupling dimensions — **strength**, -**distance**, and **volatility** — and a four-rung strength ladder: intrusive, +**Appendix G** lays out Khononov's three coupling dimensions (**strength**, +**distance**, and **volatility**) and a four-rung strength ladder: intrusive, functional, model, contract. Every composition choice in this chapter maps directly onto this model. The gateway keeps inter-service coupling at the **contract rung**: services expose schemas (OpenAPI, proto, GraphQL SDL), and the gateway composes *contracts*, never -internals. No service knows another service's database or in-memory model — exactly +internals. No service knows another service's database or in-memory model: exactly what "contract strength" means. By contrast, the call-chain anti-pattern drifts toward **functional coupling**: service A doesn't just consume a contract from service B, it -depends on B's *behavior* — its timing, its side effects, its availability — because +depends on B's *behavior* (its timing, its side effects, its availability) because every hop in the chain is a synchronous call whose failure propagates upward. **Distance** matters too. A gateway adds one hop between the client and the -back-end — a small increase in distance — but that hop *lowers strength*, because +back-end (a small increase in distance), but that hop *lowers strength*, because the services behind it don't know each other. A BFF increases distance further (client → BFF → services) but buys **client-specific shaping**, which means the coupling between the client and the back-end is shaped exactly to what that client needs, nothing more. Federation distributes the composition across subgraphs, each -owning its slice — the gateway plans the query, and each subgraph exposes only +owning its slice: the gateway plans the query, and each subgraph exposes only its contract. The general rule from the appendix applies: **the greater the distance between components, the further down the strength ladder the coupling must be.** Composition -at the edge naturally satisfies this — the gateway is far from the services (high +at the edge naturally satisfies this: the gateway is far from the services (high distance), and it couples only to their contracts (low strength). A call chain -violates it — services that are far apart couple at the functional level, which is +violates it: services that are far apart couple at the functional level, which is exactly why it becomes a distributed monolith. ### Cross-check it yourself The gateway is just HTTP. Send the same query with `curl` or Postman and request -`stock` in one call, then omit it in another — confirm the gateway only calls +`stock` in one call, then omit it in another. Confirm the gateway only calls `inventory` when `stock` is selected. Then request a *list* of orders with `stock` and watch the back-end call count: without batching it climbs with the list size; with a DataLoader it stays flat. That flat line is the N+1 fix working. @@ -274,5 +274,5 @@ The code is in [`examples/03-composition/`](https://github.com/patterncatalyst/c its `README.md` covers what it does and how to drive it. --- -*Verification status: verified — [`examples/03-composition/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/03-composition/) passes 8/8 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified; [`examples/03-composition/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/03-composition/) passes 8/8 checks in each of the six languages on Docker Engine (2026-10-13) (GraphQL gateway fan-out to the REST order API and the gRPC stock service, selective stock-field resolution, DataLoader batching of stock lookups into one gRPC call, single-order lookup).* diff --git a/_docs/04-data.md b/_docs/04-data.md index aeadf7f1..21b38b66 100644 --- a/_docs/04-data.md +++ b/_docs/04-data.md @@ -7,7 +7,7 @@ duration: 24 minutes --- A cloud-native service rarely owns its data in isolation. The moment an order is -placed, something else needs to know — payment wants to charge, inventory wants +placed, something else needs to know: payment wants to charge, inventory wants to reserve stock, shipping wants to plan a label. The naive way to make that happen is to write the order to the database and then publish an event to Kafka. Those are two separate writes to two separate systems, and nothing makes them @@ -23,35 +23,35 @@ The first rule of data in a microservice system is the one most often broken: **each service owns its data, and no other service touches its tables.** Peers reach an owned dataset only through the owning service's API, never by connecting to its database. Sharing a database may look like reuse, but it couples every -consumer to the owner's schema — a column rename becomes a cross-team migration, and +consumer to the owner's schema: a column rename becomes a cross-team migration, and two services' deploys can no longer move independently. A shared database is the single most common way a nominally microservice system collapses back into a distributed monolith. {% include excalidraw.html file="04-db-per-service" - alt="Four services — order, inventory, payment, shipping — each owning its data, each backed by its own PostgreSQL with a private schema run by CloudNativePG. A banner reads: polyglot persistence, pick the store per workload (relational, key-value, search, object)." - caption="Figure 4.1 — A database per service: private schemas, reached only through the owning API" %} + alt="Four services (order, inventory, payment, shipping), each owning its data, each backed by its own PostgreSQL with a private schema run by CloudNativePG. A banner reads: polyglot persistence, pick the store per workload (relational, key-value, search, object)." + caption="Figure 4.1: A database per service: private schemas, reached only through the owning API" %} -Ownership also frees each service to pick the store that fits its workload — +Ownership also frees each service to pick the store that fits its workload: **polyglot persistence**. Most of our services are relational on PostgreSQL (run in-cluster by CloudNativePG, no managed cloud), but nothing stops a search-heavy service from using an index or a session store from using a key-value store. The contract a service exposes is its API; the database behind it is a private implementation detail it is free to change. Everything else in this chapter assumes -this boundary — the outbox, CDC, and sagas all exist precisely because each service +this boundary: the outbox, CDC, and sagas all exist precisely because each service keeps its own store and they must still agree on what happened. ## Two models, one stream of truth -The first idea is **CQRS** — Command Query Responsibility Segregation — using +The first idea is **CQRS** (Command Query Responsibility Segregation), using *different* models for writes and reads. The write side optimises for invariants and consistency: a normalised schema in a transactional store. The read side optimises for the shape of the question being asked: a denormalised table, a search index, or a document store, whatever answers the query fastest. The two are linked by a stream of changes flowing from the write side to the read side. -The cost is eventual consistency — a small lag between the write committing and +The cost is eventual consistency: a small lag between the write committing and the read model catching up. The benefit is that each side can scale and evolve independently. Reach for it when reads and writes have different shapes and access patterns; skip it when a single store handles both happily, @@ -60,16 +60,16 @@ because the separation is not free. {% include excalidraw.html file="04-cqrs" alt="A client sends commands (state changes) to a command handler that validates, loads the aggregate, applies invariants, and writes to a normalized transactional write store. The handler emits to an Events/WAL stream (or the write store is CDC-tailed); a projection consumes those events, denormalizes, and updates a read store (wide tables, Elasticsearch, or Mongo) shaped for the query, which serves read-optimised queries back to the client." - caption="Figure 4.2 — CQRS: commands flow into the write model; events project into a read model shaped for queries" %} + caption="Figure 4.2: CQRS: commands flow into the write model; events project into a read model shaped for queries" %} -Concretely: a **command** is a request to change state — place an order, reserve -stock. The command handler validates it, loads the relevant aggregate, applies the +Concretely: a **command** is a request to change state, such as placing an order or +reserving stock. The command handler validates it, loads the relevant aggregate, applies the business invariants, and writes the new state to a normalised, transactional store. A -**query** is read-only and answers a question — list a customer's recent orders with +**query** is read-only and answers a question, such as listing a customer's recent orders with their shipment status. Rather than assemble that from the normalised write tables on every call, a *projection* consumes the stream of changes and maintains a read model already shaped like the answer. The arrow from write side to read side is exactly the -change stream of Change Data Capture — log tailing is the cleanest way to drive a +change stream of Change Data Capture; log tailing is the cleanest way to drive a projection. The price is the eventual-consistency window, usually milliseconds; the payoff is that a read-heavy surface can be denormalised and scaled without distorting the write model's integrity. The discipline is to resist applying CQRS everywhere: @@ -78,7 +78,7 @@ overhead. ## Capturing change without a second write -What carries a committed change to the read model — or to any other service — +What carries a committed change to the read model (or to any other service) without a fragile second write? **Change Data Capture (CDC).** The application writes ordinary SQL. The database commits, and that change lands in the write-ahead log that already exists for replication and crash recovery (the WAL @@ -92,17 +92,17 @@ event is a strictly downstream consequence of it. {% include excalidraw.html file="04-log-tailing" alt="An application issues ordinary INSERT/UPDATE SQL to a database (Postgres, MySQL, or SQL Server). Tables and rows commit into the write-ahead log (WAL), which is commit-ordered and durable. A Debezium connector tails the WAL through a logical replication slot, converts row deltas into events, and publishes them to Kafka topics (orders.public, orders.outbox, and others)." - caption="Figure 4.3 — Change Data Capture: Debezium tails the WAL and turns every committed row into an ordered event" %} + caption="Figure 4.3: Change Data Capture: Debezium tails the WAL and turns every committed row into an ordered event" %} -Two uses dominate. The first is the **outbox** — shipping integration events -safely. The second is **CQRS projections** — the change stream that drives a CQRS read model is most cleanly produced this way, so the read model is updated from the same +Two uses dominate. The first is the **outbox**: shipping integration events +safely. The second is **CQRS projections**: the change stream that drives a CQRS read model is most cleanly produced this way, so the read model is updated from the same committed log that everything else trusts. In both cases the property is identical: the commit is the only truth, and every event is a consequence of it, emitted in commit order, with no second write that can be lost. Debezium runs as an open-source Kafka Connect connector (Apache 2.0), so this is reproducible on the same plain Kubernetes the rest of the stack runs on. -> **WAL — Write-Ahead Log.** The database's commit-ordered, durable log. It +> **WAL: Write-Ahead Log.** The database's commit-ordered, durable log. It > already exists so the database can replicate and recover from a crash; > Debezium subscribes to it. @@ -124,11 +124,11 @@ guaranteed, ordered consequence of it. {% include excalidraw.html file="04-data-outbox" alt="The REST handler writes the order and an outbox row in one local transaction to Postgres; Debezium tails the write-ahead log and publishes the order.placed event to Kafka, which fans out to consumers" - caption="Figure 4.4 — One local transaction; CDC turns the committed outbox row into an ordered event" %} + caption="Figure 4.4: One local transaction; CDC turns the committed outbox row into an ordered event" %} ### The code, in your language -Pick your stack — the tab selection follows you across all chapters. The +Pick your stack: the tab selection follows you across all chapters. The shape is identical everywhere: open one local transaction, persist the state change, persist the outbox row in the *same* transaction, commit. No broker call sits on the request path. @@ -263,12 +263,12 @@ moves appear: 1. **Open one local transaction.** `@Transactional` in Spring and Quarkus, an explicit `BeginTransactionAsync` in .NET, an `async with db.transaction()` in Python, `pg_.begin()` in C++. Everything that follows commits together or not - at all. This is the whole trick — there is exactly one commit, to one + at all. This is the whole trick: there is exactly one commit, to one database, so there is no distributed transaction to coordinate. 2. **Persist the state change.** Save the order. Nothing about this line knows an event is coming. 3. **Persist the outbox row in the same transaction.** This is the line people - are tempted to "optimise" into a Kafka call — don't. Writing a row to an + are tempted to "optimise" into a Kafka call. Don't. Writing a row to an `outbox` table keeps the event inside the transaction's atomicity. Quarkus can hide the row behind `event.fire(...)` with `debezium-quarkus-outbox`, which brings its own table layout; the example writes a plain Panache `OutboxEvent` @@ -285,15 +285,15 @@ on, and C++ leans on RAII so a thrown exception unwinds the transaction without `finally`. The pattern is the same in all five. > **At-least-once, so be idempotent.** Because the event is delivered after the -> commit and may be redelivered if the relay restarts, consumers must dedupe — by -> the message key or an idempotency key. "Exactly-once" in practice is +> commit and may be redelivered if the relay restarts, consumers must dedupe, +> by the message key or an idempotency key. "Exactly-once" in practice is > at-least-once plus idempotent consumers. We return to this in **05 · > Event-Driven**. ## Consistency without distributed transactions: the saga The outbox makes a *single* service's state-and-event atomic. But a business -transaction often spans several services — placing an order must take payment, reserve +transaction often spans several services: placing an order must take payment, reserve stock, and book shipping, each owning its own database. There is no shared transaction across those services, and a two-phase commit across the network is exactly the distributed coupling this book avoids. The answer is the **saga**: model the workflow @@ -302,16 +302,16 @@ event that triggers the next. {% include excalidraw.html file="04-saga" - alt="A happy-path chain: order-service (order created) triggers payment-service (payment taken), which triggers inventory-service (stock reserved), which triggers shipping-service (shipment booked) — each local commit emits the event that triggers the next. On failure, payment fails and emits order.cancelled, which drives order-service to compensate by cancelling. No two-phase commit; failures trigger compensating actions, not rollbacks." - caption="Figure 4.5 — A saga: local transactions chained by events, with compensations instead of rollbacks" %} + alt="A happy-path chain: order-service (order created) triggers payment-service (payment taken), which triggers inventory-service (stock reserved), which triggers shipping-service (shipment booked). Each local commit emits the event that triggers the next. On failure, payment fails and emits order.cancelled, which drives order-service to compensate by cancelling. No two-phase commit; failures trigger compensating actions, not rollbacks." + caption="Figure 4.5: A saga: local transactions chained by events, with compensations instead of rollbacks" %} -When a step fails, there is nothing to roll back — earlier steps already committed in +When a step fails, there is nothing to roll back: earlier steps already committed in their own services. Instead the saga runs **compensating actions**: if payment fails after the order was created, the order service does not un-commit; it performs a new local transaction that cancels the order and emits `order.cancelled`. Each forward step therefore needs a defined compensation, and because events are delivered at-least-once, every step and every compensation must be idempotent. Each local step -is itself made atomic by the outbox pattern — the saga is the multi-service +is itself made atomic by the outbox pattern: the saga is the multi-service layer built on top of single-service atomicity. This is only the shape of the pattern; the choreography-versus-orchestration trade-off, how a saga holds its state, and how a later step gets an earlier step's data are taken up in depth in **Appendix D @@ -319,7 +319,7 @@ and how a later step gets an earlier step's data are taken up in depth in **Appe ### Cross-check it yourself -You do not have to trust that the commit and the event stay in step — you can +You do not have to trust that the commit and the event stay in step: you can watch it. With the example running, stop the Kafka broker, place an order, and confirm two things independently: the row is present in the database (`SELECT … FROM orders`) and the outbox row is sitting unsent. Bring the broker @@ -331,5 +331,5 @@ The code is in [`examples/04-data/`](https://github.com/patterncatalyst/cloud-na `README.md` covers what it does and how to drive it. --- -*Verification status: verified — [`examples/04-data/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/04-data/) passes 8/8 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified; [`examples/04-data/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/04-data/) passes 8/8 checks in each of the six languages on Docker Engine (2026-10-13) (transactional write of order and outbox row, Debezium outbox connector running, relay of the outbox event to the order.placed Kafka topic with its payload).* diff --git a/_docs/05-event-driven.md b/_docs/05-event-driven.md index 390ce81a..c2e30f2d 100644 --- a/_docs/05-event-driven.md +++ b/_docs/05-event-driven.md @@ -2,20 +2,20 @@ title: "Event-Driven" order: 5 part: "Foundations & the system" -description: "Decoupled facts that fan out — the event backbone, producing and consuming with commit-after-side-effect, schemas as the enforced contract, and the difference between event sourcing and event streaming." +description: "Decoupled facts that fan out: the event backbone, producing and consuming with commit-after-side-effect, schemas as the enforced contract, and the difference between event sourcing and event streaming." duration: 72 minutes --- The Data chapter ended with `order.placed` leaving the database through the outbox. This chapter is about what that fact does once it's on the wire. An event -is a statement that something happened, published to whoever cares — and the +is a statement that something happened, published to whoever cares, and the producer cares about *none* of them. ## One fact, many reactions `order.placed` fans out to `payment`, `shipping`, and `notification`, and the -producer names none of them. That is the power move: you can add a new consumer — -a fraud-check service — **without touching or redeploying the producer.** Contrast +producer names none of them. That is the power move: you can add a new consumer +(a fraud-check service) **without touching or redeploying the producer.** Contrast the synchronous world, where adding a dependency means changing the caller. This is the open/closed principle at the architecture level, with Kafka (run by Strimzi on Kubernetes) as the backbone. @@ -23,9 +23,9 @@ on Kubernetes) as the backbone. {% include excalidraw.html file="05-event-backbone" alt="order-service emits order.placed to a Kafka topic; payment, shipping, and notification each consume it independently, and a later fraud-check service is added as a new consumer without changing the producer" - caption="Figure 5.1 — One published fact; consumers added without touching the producer" %} + caption="Figure 5.1: One published fact; consumers added without touching the producer" %} -## Core benefits — and the costs they carry +## Core benefits and the costs they carry Event-driven design buys three things, and each one comes paired with a cost you then have to design for. The benefits are why @@ -34,26 +34,26 @@ you at 3 a.m. | Benefit | What it gives you | The cost that shadows it | |---|---|---| -| **Loose coupling** | producers don't know their consumers — add a consumer without touching the producer | **eventual consistency**: readers see the effect *after* the event propagates, not in the producer's transaction | +| **Loose coupling** | producers don't know their consumers: add a consumer without touching the producer | **eventual consistency**: readers see the effect *after* the event propagates, not in the producer's transaction | | **Real-time responsiveness** | reactions fire as facts happen, not on a poll interval | **debugging complexity**: one request's story is now spread across services and a log | | **High scalability** | partitioned consumers scale out horizontally; a slow consumer can't back-pressure the producer | **ordering & duplicates**: order holds only *within* a partition, and delivery is at-least-once | Each cost has a standard answer, and all three already appeared in this book: -- **Eventual consistency** — the write side commits, the event ships, consumers catch +- **Eventual consistency**: the write side commits, the event ships, consumers catch up a moment later. Design reads to tolerate that lag, or read from the write model when you need read-your-writes; never assume a consumer is up to date the instant the producer commits. This is the same trade the outbox and CQRS sections make. -- **Debugging complexity** — a single user action becomes a chain of events across +- **Debugging complexity**: a single user action becomes a chain of events across several services, so "what happened to order A-1001?" is no longer one log to grep. The fix is to thread a correlation / trace id through every event envelope and lean on distributed tracing (**Observability**); without it, you are reconstructing the story from five terminals. -- **Ordering & duplicates** — Kafka orders events only within a partition, so key by +- **Ordering & duplicates**: Kafka orders events only within a partition, so key by the entity (the order id) to keep one entity's events in order; there is no global order across partitions. And at-least-once delivery means the same event can arrive - twice — a producer retry, a consumer rebalance — so every handler must be + twice (a producer retry, a consumer rebalance), so every handler must be **idempotent**, deduplicating by event id or folding the work into an upsert. The throughline: the benefits are real, but each one *moves* a hard problem rather than @@ -62,7 +62,7 @@ removing it. Design for the cost on the same day you reach for the benefit. ## Producing and consuming Two things matter more than the framework. First, **commit after the side-effect, -not before** — do the work, then acknowledge. Second, because you commit after, a +not before**: do the work, then acknowledge. Second, because you commit after, a crash mid-handler means redelivery, so delivery is **at-least-once** and your handlers **must be idempotent** (dedupe by message key or an idempotency key). @@ -236,15 +236,15 @@ func consume(ctx context.Context, cfg Settings) error { ### How the code works The producer is trivial and that's the point: send a **keyed** record to a topic -and walk away — it names no consumer. The key (the order id) decides the partition, +and walk away; it names no consumer. The key (the order id) decides the partition, which preserves per-order ordering. The consumer is where the care lives. Every tab disables auto-commit and acknowledges *after* `send_email` returns. Commit before the side-effect and a -crash loses the work silently; commit after and a crash merely *redelivers* — which +crash loses the work silently; commit after and a crash merely *redelivers*, which is why every handler must be idempotent. "Exactly-once" is a marketing term for -"at-least-once plus a dedupe you wrote." The frameworks differ in surface — -`Acknowledgment`, `msg.ack()`, consumer-return, `commitSync()` — but the rule is +"at-least-once plus a dedupe you wrote." The frameworks differ in surface +(`Acknowledgment`, `msg.ack()`, consumer-return, `commitSync()`), but the rule is identical. ## Schemas are the contract @@ -257,14 +257,14 @@ in-cluster. {% include excalidraw.html file="05-schemas-registry" alt="An Apicurio Registry at top holds versioned schemas with FORWARD/BACKWARD compatibility. The order-service producer serialises with a schema-id and sends id+bytes through Kafka to a consumer that resolves the id and deserialises; both resolve the schema by id from the registry. A CI publish gate rejects incompatible schemas before merge." - caption="Figure 5.2 — Apicurio holds versioned schemas; producer and consumer resolve by schema-id on the hot path, and a CI gate rejects incompatible changes before merge" %} + caption="Figure 5.2: Apicurio holds versioned schemas; producer and consumer resolve by schema-id on the hot path, and a CI gate rejects incompatible changes before merge" %} The mechanics matter. The producer serialises a record and prefixes it with a small **schema-id**; only the id plus the encoded bytes travel on the wire, never the schema itself. The consumer reads the id, fetches the matching schema from the registry (cached after the first lookup), and deserialises. Because the registry enforces a -compatibility policy — **FORWARD** so old consumers tolerate new producers, or -**BACKWARD** so new consumers tolerate old data — a change that would break the other +compatibility policy (**FORWARD** so old consumers tolerate new producers, or +**BACKWARD** so new consumers tolerate old data), a change that would break the other side is refused. Best of all, that check runs in CI at publish time: a breaking schema is **rejected before it merges**, not discovered in production when an anonymous consumer falls over. That enforcement is what makes "add a consumer safely" @@ -273,7 +273,7 @@ safe. ## Event sourcing vs. event streaming A vocabulary fix most teams need before going further. **Event sourcing** means -events *are* the database — there is no current-state row anywhere; current state +events *are* the database: there is no current-state row anywhere; current state is computed by folding the entire event history. Crediting an account isn't `UPDATE balance SET balance = balance + 100`; it's `INSERT INTO events ('credit', 100)`, and reading the balance means summing every event for that account. @@ -284,11 +284,11 @@ itself the valuable asset (ledgers, audit). {% include excalidraw.html file="05-sourcing-vs-streaming" - alt="Two columns. Event sourcing: events are the state — aggregate state is a fold of events from t0, the event log is the database, current state is rebuilt by replaying; fit is audit-heavy, financial, temporal; tools Marten, EventStoreDB, Axon. Event streaming: events are the transport — current state lives in a database, events are facts about changes, consumers react and propagate; fit is integration, fan-out, decoupling; tools Kafka, Pulsar, NATS JetStream." - caption="Figure 5.3 — Event sourcing makes events the state; event streaming uses events as transport — and full CQRS often combines both" %} + alt="Two columns. Event sourcing: events are the state: aggregate state is a fold of events from t0, the event log is the database, current state is rebuilt by replaying; fit is audit-heavy, financial, temporal; tools Marten, EventStoreDB, Axon. Event streaming: events are the transport: current state lives in a database, events are facts about changes, consumers react and propagate; fit is integration, fan-out, decoupling; tools Kafka, Pulsar, NATS JetStream." + caption="Figure 5.3: Event sourcing makes events the state; event streaming uses events as transport, and full CQRS often combines both" %} The two are not rivals and not mutually exclusive. A common full-CQRS design is -event-sourced on the write side — the log *is* the system of record — while the read +event-sourced on the write side (the log *is* the system of record) while the read side is a set of streaming projections fed from that same log. The question that disambiguates them in any given design is simple: is the **event itself** the source of truth (sourcing), or is it a **fact propagated** from a store that still holds the @@ -303,15 +303,15 @@ open-source tools by the job they do rather than by reputation. {% include excalidraw.html file="05-tools-landscape" alt="Three categories. Substrate (the durable log): Kafka and Pulsar. Compute (transform and aggregate): Flink, with alternatives Kafka Streams and Spark Structured Streaming. Movement (data in and out): NiFi and Debezium. Arrows show substrate feeding compute feeding movement; a typical pipeline runs source DB to Debezium/NiFi to Kafka/Pulsar to Flink to sink." - caption="Figure 5.4 — Three jobs: substrate (the durable log), compute (transform & aggregate), and movement (data in & out)" %} + caption="Figure 5.4: Three jobs: substrate (the durable log), compute (transform & aggregate), and movement (data in & out)" %} -Three categories cover almost everything. **Substrate** is the durable log itself — +Three categories cover almost everything. **Substrate** is the durable log itself: **Kafka** (the de facto event spine: partitioned, ordered, offset-tracked) or **Pulsar** (pub/sub plus queue plus log, with tiered storage and a Kafka-protocol -shim). **Compute** transforms and aggregates the streams — **Flink** is the heavyweight +shim). **Compute** transforms and aggregates the streams: **Flink** is the heavyweight stateful engine with windowed aggregations and exactly-once checkpointing, sitting on top of Kafka or Pulsar; lighter alternatives are Kafka Streams (JVM-only) and Spark -Structured Streaming. **Movement** gets data in and out — **NiFi** for visual ETL with +Structured Streaming. **Movement** gets data in and out: **NiFi** for visual ETL with hundreds of connectors and back-pressure, and **Debezium** for CDC, tailing the database commit log (the same mechanism the Data chapter builds the outbox on). A typical pipeline threads all three: source DB → Debezium or NiFi → Kafka or Pulsar → @@ -322,36 +322,36 @@ plain Kubernetes as everything else. ## Choosing the substrate: Kafka vs Pulsar vs AMQP The tools landscape split the work into substrate, compute, and movement; the -**substrate** — the durable broker the whole system leans on — is the most consequential +**substrate**, the durable broker the whole system leans on, is the most consequential choice. Three common answers, with different models rather than cosmetic differences: | | **Apache Kafka** | **Apache Pulsar** | **AMQP** (e.g. RabbitMQ) | |---|---|---|---| | **Model** | partitioned, append-only **log**; consumers track offsets | a log too, but split into **segments** with serving brokers separate from storage | **exchanges route to queues**; a delivered message is removed | -| **Replay** | native — rewind the offset | native — rewind, like Kafka | not built in; a consumed message is gone unless you kept a copy | +| **Replay** | native: rewind the offset | native: rewind, like Kafka | not built in; a consumed message is gone unless you kept a copy | | **Ordering** | per partition (per key) | per partition (per key) | per queue | | **Scaling** | partitions; a rebalance moves consumers | brokers and storage scale **independently**; cold segments can tier to object storage | queues; sharding is largely manual | -| **Multi-tenancy / geo** | add-on tooling | **first-class** — tenants, namespaces, and geo-replication built in | per-vhost; federation/shovel for geo | +| **Multi-tenancy / geo** | add-on tooling | **first-class**: tenants, namespaces, and geo-replication built in | per-vhost; federation/shovel for geo | | **Latency / throughput** | low latency, very high throughput | low latency, high throughput | very low latency at modest volume; throughput falls as queues deepen | | **Best when** | high-throughput streaming where the durable log *is* the source of truth | you want Kafka-style streaming **plus** native multi-tenancy, geo, or storage/compute separation | classic task queues, RPC-style work distribution, and rich routing at lower volumes | The dividing line is **log versus queue**. Kafka and Pulsar are *logs*: messages persist, independent consumer groups read at their own offsets, and replay is just -rewinding — which is exactly why either can be the source of truth this chapter builds +rewinding, which is exactly why either can be the source of truth this chapter builds on. AMQP is a *queue broker*: a message is delivered and removed, routing is rich (exchanges, bindings, topic patterns), and it excels at work distribution and -request/reply — but "replay last week" means you stood up a separate store, because the +request/reply, but "replay last week" means you stood up a separate store, because the broker didn't keep the messages. Pulsar's pitch over Kafka is mostly **operational**: because serving and storage are separate layers, you can scale them independently and tier cold data to object storage, and multi-tenancy and geo-replication are built in rather than bolted on. The price is -a larger surface to run — more components and moving parts — which only pays off for +a larger surface to run (more components and moving parts), which only pays off for teams that need those properties. -For this book's running system — `order.placed` as a replayable fact that several -services consume independently — a **log** is the right substrate, and Kafka is the +For this book's running system, `order.placed` as a replayable fact that several +services consume independently, a **log** is the right substrate, and Kafka is the sensible default. Reach for **Pulsar** when multi-tenancy, geo, or storage/compute separation are hard requirements, and for **AMQP** when the job is a *task queue* with complex routing rather than an event log you replay. @@ -360,13 +360,13 @@ queue* with complex routing rather than an event log you replay. The comparison table is the decision; the architecture is *why* each answer behaves the way it does. Three different internal designs produce three different sets of -advantages — and knowing the design is what lets you build services that lean on a +advantages, and knowing the design is what lets you build services that lean on a broker's strengths instead of fighting them. -### Apache Kafka — the partitioned, replicated log +### Apache Kafka: the partitioned, replicated log Kafka is a cluster of brokers serving topics split into **partitions**, and the -partition — not the topic — is the unit of everything. Each partition is an ordered, +partition, not the topic, is the unit of everything. Each partition is an ordered, append-only log: producers append (keyed by entity, so one entity's events land on one partition and stay ordered), and each partition is replicated to a set of in-sync replicas so a broker failure loses nothing. Consumers join a **consumer group**, the @@ -375,13 +375,13 @@ offset**, reading at its own position into the log. {% include excalidraw.html file="05-kafka-architecture" - alt="Producers, keyed by entity, append to a Kafka cluster drawn as a partitioned, replicated log with Partition 0, 1, and 2, each a leader plus in-sync replicas. Two consumer groups read from the cluster — a payment group that tracks its own offsets and an analytics group with independent offsets that can replay. A note: records are retained by time or size, not deleted on read; partitions give parallelism, replicas give durability, and groups read independently." - caption="Figure 5.5 — Kafka architecture: a partitioned, replicated log read by independent consumer groups" %} + alt="Producers, keyed by entity, append to a Kafka cluster drawn as a partitioned, replicated log with Partition 0, 1, and 2, each a leader plus in-sync replicas. Two consumer groups read from the cluster: a payment group that tracks its own offsets and an analytics group with independent offsets that can replay. A note: records are retained by time or size, not deleted on read; partitions give parallelism, replicas give durability, and groups read independently." + caption="Figure 5.5: Kafka architecture: a partitioned, replicated log read by independent consumer groups" %} Its advantages follow from that design. Sequential appends and zero-copy reads give high **throughput**; partitions give linear **horizontal scale**; replication gives **durability**; and because the log is retained rather than consumed, producers -and consumers are **decoupled in time** — a consumer can be down for an hour or added +and consumers are **decoupled in time**: a consumer can be down for an hour or added years later and still get the full story. You build services against it the way this chapter's producer/consumer code already does: key events by the entity, make consumers idempotent, and commit the offset *after* the side effect. @@ -393,24 +393,24 @@ The mental-model correction most people need: **Kafka is a log, not a queue.** {% include excalidraw.html file="05-log-truth" alt="A partitioned append-only log with offsets 0 through 9, head at the latest. Three consumers track independent offsets: a payment consumer at offset 9 (live), an analytics replay reset to offset 0, and a new consumer reading history from the middle with no producer impact." - caption="Figure 5.6 — A partitioned, append-only log: each consumer tracks its own offset, so replay is just rewinding" %} + caption="Figure 5.6: A partitioned, append-only log: each consumer tracks its own offset, so replay is just rewinding" %} -Messages aren't deleted when read — they're retained by time or size, and each +Messages aren't deleted when read; they're retained by time or size, and each consumer group tracks **its own offset** into the partition. That one fact has large consequences. A live consumer sits at the head, processing new records as they arrive. An analytics job can reset to offset zero and replay the entire history. A brand-new consumer can join later and read all of history to build its state, with **no impact -on the producer or on existing consumers** — nobody is draining a queue, so nobody +on the producer or on existing consumers**; nobody is draining a queue, so nobody contends for the messages. Replay is moving an offset backward. That is the foundation for event sourcing, for rebuilding a stream processor's state after a crash, and for the windowed computation taken up in **06 · Stream Processing**. Both Kafka (run by Strimzi in our cluster) and Pulsar give you this durable, replayable, -partitioned log — not a queue you drain. +partitioned log, not a queue you drain. -### Apache Pulsar — serving split from storage +### Apache Pulsar: serving split from storage Pulsar reaches the same durable-log outcome with a two-layer architecture. A tier of -**stateless brokers** handles serving — connections, routing, subscriptions — while a +**stateless brokers** handles serving: connections, routing, subscriptions, while a separate **Apache BookKeeper** cluster (the "bookies") owns durable storage, writing the log as **segments** rather than fixed partitions. Because brokers hold no data they fail over in seconds (another broker just adopts the topic), and because storage is its @@ -419,24 +419,24 @@ keep them. {% include excalidraw.html file="05-pulsar-architecture" - alt="Producers publish to a layer of stateless Pulsar brokers, which write segments to an Apache BookKeeper cluster of bookies; BookKeeper offloads cold segments to tiered object storage. Consumers read from the brokers using subscription modes — exclusive, shared, failover, or key_shared. A note: serving and storage scale independently, stateless brokers fail over fast, and a tenant-to-namespace hierarchy gives native multi-tenancy and geo-replication." - caption="Figure 5.7 — Pulsar architecture: stateless brokers over BookKeeper storage, with tiered offload" %} + alt="Producers publish to a layer of stateless Pulsar brokers, which write segments to an Apache BookKeeper cluster of bookies; BookKeeper offloads cold segments to tiered object storage. Consumers read from the brokers using subscription modes: exclusive, shared, failover, or key_shared. A note: serving and storage scale independently, stateless brokers fail over fast, and a tenant-to-namespace hierarchy gives native multi-tenancy and geo-replication." + caption="Figure 5.7: Pulsar architecture: stateless brokers over BookKeeper storage, with tiered offload" %} The advantages are exactly that split made useful. Serving and storage **scale independently**, so a spike in consumers doesn't force you to grow storage. -**Multi-tenancy is first-class** — a tenant → namespace → topic hierarchy with -per-tenant isolation and quotas — and **geo-replication** is built in. And one +**Multi-tenancy is first-class**, a tenant → namespace → topic hierarchy with +per-tenant isolation and quotas, and **geo-replication** is built in. And one substrate covers both jobs: its **subscription modes** (exclusive, shared, failover, key-shared) yield a competing-consumer *queue* and a streaming *log* from the same -topic. The price is operational surface — brokers plus BookKeeper plus a metadata -store — which pays off only when you need those properties. +topic. The price is operational surface (brokers plus BookKeeper plus a metadata +store), which pays off only when you need those properties. -### AMQP / RabbitMQ — routing through exchanges +### AMQP / RabbitMQ: routing through exchanges AMQP brokers are built around **routing**, not retention. A producer publishes to an **exchange**, never straight to a queue; the exchange type (direct, topic, fanout, headers) plus **bindings** decide which **queues** get a copy. Consumers pull from a -queue, **acknowledge** each message, and an acked message is **removed** — the broker's +queue, **acknowledge** each message, and an acked message is **removed**: the broker's job is to hand each message to a willing worker, not to keep history. Rejected or expired messages flow to a **dead-letter** queue, and queues carry priorities, TTLs, and overflow policies. @@ -444,11 +444,11 @@ and overflow policies. {% include excalidraw.html file="05-amqp-architecture" alt="A producer publishes with a routing key to an exchange (direct, topic, or fanout). Bindings route copies to a queue: orders bound by key and a queue: audit bound by pattern; each queue delivers to competing consumers that acknowledge, after which the message is removed. A dead-letter queue receives nacked, expired, or overflowed messages. A note: routing lives in the exchange and an acked message is removed with no replay, which is why AMQP excels at work distribution, RPC, and rich routing." - caption="Figure 5.8 — AMQP architecture: an exchange routes through bindings to queues that competing consumers drain" %} + caption="Figure 5.8: AMQP architecture: an exchange routes through bindings to queues that competing consumers drain" %} -Its strengths are the mirror image of a log's. Routing is **rich and dynamic** — fan a +Its strengths are the mirror image of a log's. Routing is **rich and dynamic**: fan a message to many queues, or route by key or pattern, without the producer knowing its -consumers — and per-message **ack / nack / requeue** makes it excellent for **work +consumers, and per-message **ack / nack / requeue** makes it excellent for **work distribution and request/reply**, where each task goes to exactly one worker and is retried on failure. The trade-offs are the flip side: there is **no replay** (a consumed message is gone unless you kept a copy), and throughput **degrades as queues @@ -457,29 +457,29 @@ designed to shed. Reach for it when the job is a task queue with complex routing, not an event log you rewind. For an event backbone the system treats as its memory, the log model is the one that -wins — which is the idea this chapter builds on. +wins, which is the idea this chapter builds on. ## Building event-driven microservices: consume, process, produce An event-driven microservice is small and shaped the same way every time: it **consumes** from one or more input streams, **processes** each event (often against some local state), and **produces** to output streams. The broker is the data plane -between services — there are no direct service-to-service calls on the hot path — and +between services; there are no direct service-to-service calls on the hot path, and each service owns its own state rather than reaching into anyone else's. {% include excalidraw.html file="05-edm-anatomy" - alt="An input stream, order.placed, feeds an event-driven microservice drawn as three steps in a row — consume (poll and deserialize), process (map or aggregate), produce (serialize and send) — which emits to an output stream, order.validated. A dashed state store hangs off the process step, marked stateful only, with a read/write arrow. A note: consume to process to produce; the broker is the data plane and the service owns its state." - caption="Figure 5.9 — The event-driven microservice: consume, process, produce, with state only when the work needs it" %} + alt="An input stream, order.placed, feeds an event-driven microservice drawn as three steps in a row, consume (poll and deserialize), process (map or aggregate), produce (serialize and send), which emits to an output stream, order.validated. A dashed state store hangs off the process step, marked stateful only, with a read/write arrow. A note: consume to process to produce; the broker is the data plane and the service owns its state." + caption="Figure 5.9: The event-driven microservice: consume, process, produce, with state only when the work needs it" %} -Processing comes in two flavours. **Stateless** processing — a map, a filter, an -enrichment — needs no memory of past events: each event is handled on its own, which -makes the service trivial to scale and to restart. **Stateful** processing — -aggregations, joins, running totals, deduplication — needs a **state store** the +Processing comes in two flavours. **Stateless** processing (a map, a filter, an +enrichment) needs no memory of past events: each event is handled on its own, which +makes the service trivial to scale and to restart. **Stateful** processing +(aggregations, joins, running totals, deduplication) needs a **state store** the service maintains and can rebuild from the log, a richer topic with its own correctness rules. Most services are mostly stateless with a few stateful steps. -The core loop — consume, transform, re-emit — is the same in every stack. Here a +The core loop (consume, transform, re-emit) is the same in every stack. Here a validator consumes `order.placed`, derives `order.validated`, and produces it, holding no state: @@ -545,13 +545,13 @@ fetches.EachRecord(func(r *kgo.Record) { ``` Keep the produce idempotent and commit the consumer offset *after* the produce -succeeds — the commit-after-side-effect rule again — so a crash between producing and +succeeds, the commit-after-side-effect rule again, so a crash between producing and committing replays cleanly instead of dropping the derived event. ## The sidecar pattern: business logic on one side, the broker on the other -The consume/produce machinery — serialization, schema resolution, offset management, -retries, tracing — is identical across services and has nothing to do with any one +The consume/produce machinery (serialization, schema resolution, offset management, +retries, tracing) is identical across services and has nothing to do with any one service's business logic. The **sidecar pattern** factors it out: a small companion process in the same pod owns all broker I/O, and the business microservice talks to it over a simple local API (here, gRPC on `localhost`). The service links **no broker @@ -560,14 +560,14 @@ client at all**. {% include excalidraw.html file="05-sidecar" alt="Two pods, service A in Go and service B in Python. In each, a business-logic component with no broker client talks over local gRPC to a sidecar that handles broker I/O. Both sidecars connect to a single Kafka cluster (topics, offsets, schema) to consume and produce. A note: the sidecar owns serde, schema, offsets, retries, and tracing, so business logic stays pure and polyglot." - caption="Figure 5.10 — The sidecar pattern: a companion process owns broker I/O so the service stays pure and language-agnostic" %} + caption="Figure 5.10: The sidecar pattern: a companion process owns broker I/O so the service stays pure and language-agnostic" %} -Three things fall out of that split. **Polyglot** — the business service can be in any +Three things fall out of that split. **Polyglot**: the business service can be in any language, because the sidecar speaks the broker and the service only speaks a local gRPC contract (which is why this book's six languages integrate identically). -**Cross-cutting concerns are centralized** — schema, serde, retry, dead-lettering, and +**Cross-cutting concerns are centralized**: schema, serde, retry, dead-lettering, and tracing live in one audited place instead of being re-implemented per service. And -**business logic stays pure** — testable with no broker, since publishing is just a +**business logic stays pure**: testable with no broker, since publishing is just a local call. The cost is a second process per pod and one extra local hop, usually a fair price for the decoupling. Publishing through the sidecar carries no Kafka types in the service: @@ -643,11 +643,11 @@ specifics. ## Eventification: turning data into streams An event-driven system runs on streams, but most of an organization's data starts life -trapped inside the database of the service that owns it — data on the *inside*, useless +trapped inside the database of the service that owns it: data on the *inside*, useless to anyone else. **Data liberation** is the practice of getting that data out as an event stream the rest of the system can build on. The cleanest mechanism is **change data capture (CDC)**: a connector like Debezium tails the database's commit log and emits an -event for every row change — the same log-tailing the Data chapter's outbox relies on. +event for every row change, the same log-tailing the Data chapter's outbox relies on. When you own the writer, the **outbox** is the alternative: write the event in the same transaction as the state change, then relay it. Either way you have **eventified** a table into a stream. @@ -655,32 +655,32 @@ table into a stream. {% include excalidraw.html file="05-eventification" alt="A source database with an orders table feeds a CDC connector (Debezium) that tails the commit log and turns each row change into an event on the topic order.changed, which consumers read to build their own views. Notes: table-to-stream (eventify) and stream-to-table (materialize) are two views of the same data with the log as the bridge; the outbox, writing the event in the same transaction, is the alternative when you own the writer." - caption="Figure 5.11 — Eventification: change data capture liberates a table into an event stream" %} + caption="Figure 5.11: Eventification: change data capture liberates a table into an event stream" %} The idea underneath is the **table–stream duality**. A stream of changes folded forward -gives you a table — its current state — and a table's change history *is* a stream. They +gives you a table, its current state, and a table's change history *is* a stream. They are two views of the same data, and the log is the bridge between them. That duality is what the next pattern depends on. ## Event-carried state transfer and local views -Once data is liberated, events come in two styles. An **event notification** is thin — -"order A-1001 was placed" — and tells a consumer only that something happened; to act on +Once data is liberated, events come in two styles. An **event notification** is thin: +"order A-1001 was placed", and tells a consumer only that something happened; to act on it, the consumer must **call back** to the source for the details, which re-introduces the synchronous coupling events were meant to remove. **Event-carried -state transfer (ECST)** instead puts enough state *in the event* — the order's customer, -total, and status — so the consumer needs no call-back at all. +state transfer (ECST)** instead puts enough state *in the event* (the order's customer, +total, and status) so the consumer needs no call-back at all. {% include excalidraw.html file="05-ecst-vs-notification" - alt="Two rows. Event notification: a source service sends a thin OrderPlaced with just an id to a consumer, which must make a synchronous GET /orders/{id} call back to the source API to get the details. Event-carried state transfer: the source service sends a fat OrderPlaced carrying the full state, and the consumer folds it into a local view via upsert, with no call-back. A note: carry enough state and the consumer never calls back — reads go local and the source decouples." - caption="Figure 5.12 — Event notification forces a call-back; event-carried state transfer lets the consumer keep a local view" %} + alt="Two rows. Event notification: a source service sends a thin OrderPlaced with just an id to a consumer, which must make a synchronous GET /orders/{id} call back to the source API to get the details. Event-carried state transfer: the source service sends a fat OrderPlaced carrying the full state, and the consumer folds it into a local view via upsert, with no call-back. A note: carry enough state and the consumer never calls back, reads go local and the source decouples." + caption="Figure 5.12: Event notification forces a call-back; event-carried state transfer lets the consumer keep a local view" %} With ECST the consumer **folds** events into its own **local, denormalized materialized -view** — a read model shaped exactly for its queries, kept current by the stream and +view**: a read model shaped exactly for its queries, kept current by the stream and rebuildable by replay. Reads become local and fast, and a momentary outage of the source no longer stalls the consumer. The cost is **denormalization**: the same facts now live -in several services' views, kept eventually consistent by the stream — the +in several services' views, kept eventually consistent by the stream: the trade an event-driven system makes, spending storage and duplication to buy decoupling and autonomy. The consumer side is a fold into an upsert; the event alone is enough to build the row: @@ -740,24 +740,24 @@ fetches.EachRecord(func(r *kgo.Record) { ## Stateful microservices: the state store and its changelog -Stateless processing handles each event alone; **stateful** processing — running totals, -counts, joins, deduplication — has to remember things between events. The cloud-native +Stateless processing handles each event alone; **stateful** processing (running totals, +counts, joins, deduplication) has to remember things between events. The cloud-native way to remember is a **local state store**: an embedded key-value store, co-located with the partition the instance owns, that the processor reads and writes on the hot path. -There is no network round-trip to a shared database, so it is fast — but it is *local*, +There is no network round-trip to a shared database, so it is fast, but it is *local*, so it has to survive the instance dying. That is what the **changelog** is for. Every write to the state store is also appended to a **compacted changelog topic** in the broker, which makes the store durable and **rebuildable**: when an instance crashes or a rebalance moves its partition to another instance, the new owner replays the changelog to restore the store before it -resumes. State lives locally for speed and in the log for safety — the table–stream +resumes. State lives locally for speed and in the log for safety: the table–stream duality doing real work. {% include excalidraw.html file="05-state-store-changelog" alt="An order.placed topic feeds a stateful instance whose process step reads and writes a local state store called customer-totals. Every write is mirrored to a compacted, durable changelog topic. When a rebalance creates a new instance, it restores the store by replaying the changelog. A note: every write also goes to the changelog, so on failure or rebalance you replay it to rebuild the store and state is never lost." - caption="Figure 5.13 — A stateful instance keeps a local state store and mirrors every write to a compacted changelog it can replay to recover" %} + caption="Figure 5.13: A stateful instance keeps a local state store and mirrors every write to a compacted changelog it can replay to recover" %} Maintaining a per-customer running total looks different in each ecosystem, because the embedded-store-plus-changelog machinery is exactly what the JVM stream-processing @@ -837,7 +837,7 @@ func aggregate(ctx goka.Context, msg any) { On the ecosystem side, Kafka Streams (Spring, Quarkus), its .NET port Streamiz, and Python's faust-streaming give you the state store and its changelog as first-class `KTable` / `Table` abstractions; Go's **goka** does the same with a group table; and in C++, with no such -framework, you keep the store and mirror writes to a compacted topic by hand — more code, +framework, you keep the store and mirror writes to a compacted topic by hand: more code, the identical pattern. ## State that doesn't depend on event order @@ -847,24 +847,24 @@ rebalance can redeliver. So a stateful handler has to stay correct when events a **out of order or more than once**. The discipline is to make state updates independent of arrival order: -- **Version or timestamp the events** and apply **last-writer-wins** — keep an update only +- **Version or timestamp the events** and apply **last-writer-wins**: keep an update only if its version is newer than what you hold, so a late `v1` can't clobber a `v2` already applied. -- Prefer **commutative** updates — adding to a counter, unioning a set — where order +- Prefer **commutative** updates (adding to a counter, unioning a set) where order doesn't matter. - Make every handler **idempotent**, so a duplicate is a no-op (dedupe by event id, or fold into an upsert). -For stateful **joins**, the two sides must be **co-partitioned** — same key, same -partition count — so matching keys land on the same instance and its state store can see +For stateful **joins**, the two sides must be **co-partitioned** (same key, same +partition count) so matching keys land on the same instance and its state store can see both sides at once. {% include excalidraw.html file="05-order-independent-state" - alt="Three status events arrive out of order — v2 PACKED first, v1 PAID second, v3 SHIPPED third. A handler that keeps the highest version (last-writer-wins) feeds a state that converges to v3 SHIPPED, correct in any order. A note: version-keyed or commutative updates plus idempotency on duplicates make state correct regardless of arrival order." - caption="Figure 5.14 — Last-writer-wins by version makes state converge correctly no matter the arrival order" %} + alt="Three status events arrive out of order: v2 PACKED first, v1 PAID second, v3 SHIPPED third. A handler that keeps the highest version (last-writer-wins) feeds a state that converges to v3 SHIPPED, correct in any order. A note: version-keyed or commutative updates plus idempotency on duplicates make state correct regardless of arrival order." + caption="Figure 5.14: Last-writer-wins by version makes state converge correctly no matter the arrival order" %} -The same guarded upsert handles both out-of-order and duplicate status events — apply one +The same guarded upsert handles both out-of-order and duplicate status events, apply one only if its version beats what is already stored: {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -926,10 +926,10 @@ if !ok || e.Version > cur.Version { // highest version wins A stream processor sees two clocks. **Processing time** is when the consumer happens to handle an event; **event time** is when the thing happened, carried *in* the -event. They drift apart — an event can be produced at 10:00, sit in a partition, and be -processed at 10:05 — and they drift by different amounts after a backlog or a replay. The +event. They drift apart: an event can be produced at 10:00, sit in a partition, and be +processed at 10:05, and they drift by different amounts after a backlog or a replay. The rule that keeps results stable is to compute on **event time, never the wall clock**. Do -that and reprocessing the same input — after a bug fix, or to rebuild a consumer — is +that and reprocessing the same input (after a bug fix, or to rebuild a consumer) is **deterministic**: it yields the same output it did the first time. Anchor a computation to processing time and a replay tomorrow produces different windows than it did today. @@ -937,16 +937,16 @@ Event time forces a decision about **late and out-of-order events**, since the n delivers neither in order nor on time. A **watermark** is the processor's running estimate that "event time has reached T; expect nothing older." It lets time-based computation know when it is safe to act: windows up to T can close, and an event older -than T that shows up afterwards is **late** — held within a grace period, or routed to a +than T that shows up afterwards is **late**: held within a grace period, or routed to a side output rather than silently corrupting a closed result. {% include excalidraw.html file="05-event-time-watermark" alt="Events arrive in processing-time order with event-times 12, then 10 (out of order), then 9 (late). A handler orders them by event time with a watermark at T equals 11. Windows up to 11 close into a deterministic result; the event with event-time 9, which is older than T, is late and goes to a side output. A note: key computation off event time, not the wall clock, so replaying the log yields the same answer." - caption="Figure 5.15 — Event time plus a watermark: close windows by event time and handle late events explicitly" %} + caption="Figure 5.15: Event time plus a watermark: close windows by event time and handle late events explicitly" %} -The deep mechanics of windows and watermarks — tumbling, sliding, and session windows, -and how a stateful engine like Flink advances them — are the subject of **06 · Stream +The deep mechanics of windows and watermarks (tumbling, sliding, and session windows, +and how a stateful engine like Flink advances them) are the subject of **06 · Stream Processing**; here the point is the discipline that makes an event-driven system replayable: deterministic, event-time computation. @@ -961,8 +961,8 @@ skipped and none is double-counted. {% include excalidraw.html file="05-checkpoint-recovery" - alt="A processor consumes from an input log, holding local state and an offset, and periodically snapshots state and offset together to a checkpoint store. On a crash, recovery restores from the checkpoint, seeks the input to the saved offset, and resumes the processor by replaying forward. A note: snapshot state and input offset together so on failure you restore and replay from there — and rewind to reprocess and rebuild." - caption="Figure 5.16 — A checkpoint snapshots state and offset as a unit, so recovery restores both and replays the gap" %} + alt="A processor consumes from an input log, holding local state and an offset, and periodically snapshots state and offset together to a checkpoint store. On a crash, recovery restores from the checkpoint, seeks the input to the saved offset, and resumes the processor by replaying forward. A note: snapshot state and input offset together so on failure you restore and replay from there, and rewind to reprocess and rebuild." + caption="Figure 5.16: A checkpoint snapshots state and offset as a unit, so recovery restores both and replays the gap" %} The last piece is making the *output* survive recovery too. Replaying after a crash re-emits events, so either every downstream consumer is idempotent (the last-writer-wins @@ -1027,8 +1027,8 @@ cl.EndTransaction(ctx, kgo.TryCommit) // commit output ``` Reprocessing is the same machinery, triggered by choice rather than by a crash. Because the log retains history and -the processor keys off event time, you can **rewind the offsets** — to a checkpoint, or -all the way to zero — and reprocess: to fix a bug in the logic, to rebuild a consumer's +the processor keys off event time, you can **rewind the offsets** (to a checkpoint, or +all the way to zero) and reprocess: to fix a bug in the logic, to rebuild a consumer's state from scratch, or to populate a brand-new view. Replay is not only a recovery path; it is how an event-driven system evolves without migrations. @@ -1040,43 +1040,43 @@ them. **Choreography** has no coordinator. Each service reacts to an event and emits its own: `OrderPlaced` makes payment charge and emit `PaymentTaken`, which makes inventory reserve -and emit `StockReserved`, on down the line. It is maximally decoupled — services know only -the events, not each other — and it scales naturally. The cost is that the end-to-end flow +and emit `StockReserved`, on down the line. It is maximally decoupled: services know only +the events, not each other, and it scales naturally. The cost is that the end-to-end flow is **emergent**: no single place describes the whole process, so understanding or changing it means tracing events across services. **Orchestration** adds a coordinator that owns the workflow explicitly. The orchestrator issues **commands** ("charge this order"), waits for replies, and decides the next step. -The flow is now legible and centrally monitored — one component knows the whole story — but +The flow is now legible and centrally monitored: one component knows the whole story, but the orchestrator is a **coupling point** and one more thing to keep available. {% include excalidraw.html file="05-choreography-orchestration" - alt="Top: choreography — order-service emits OrderPlaced to payment-service, which emits PaymentTaken to inventory-service, which emits StockReserved to shipping-service, a reacting chain with no coordinator. Bottom: orchestration — a central orchestrator that drives the workflow exchanges command and reply messages with payment, inventory, and shipping services. A note: choreography's flow is emergent and decoupled; orchestration's flow is explicit but the coordinator is a coupling point." - caption="Figure 5.17 — Choreography reacts to events with no coordinator; orchestration centralizes the flow in one" %} + alt="Top: choreography, order-service emits OrderPlaced to payment-service, which emits PaymentTaken to inventory-service, which emits StockReserved to shipping-service, a reacting chain with no coordinator. Bottom: orchestration, a central orchestrator that drives the workflow exchanges command and reply messages with payment, inventory, and shipping services. A note: choreography's flow is emergent and decoupled; orchestration's flow is explicit but the coordinator is a coupling point." + caption="Figure 5.17: Choreography reacts to events with no coordinator; orchestration centralizes the flow in one" %} Neither is universally right. Choreography suits simple, stable flows where decoupling matters most; orchestration pays off when a process is complex, needs visibility, or -changes often. Many systems mix them — choreography between bounded contexts, orchestration +changes often. Many systems mix them: choreography between bounded contexts, orchestration within one. ## The saga: a transaction across services The hard part of any multi-service flow is failure: payment succeeds, then shipping can't -be booked — and there is no distributed lock to roll the whole thing back, because each +be booked, and there is no distributed lock to roll the whole thing back, because each service owns its own data. A **saga** is the event-driven answer. It models the process as a sequence of **local transactions**, each in one service, each emitting an event that -triggers the next. If a step fails, the saga runs **compensating actions** — semantic undos -like "refund the payment" and "release the stock" — in reverse, returning the system to a +triggers the next. If a step fails, the saga runs **compensating actions** (semantic undos +like "refund the payment" and "release the stock") in reverse, returning the system to a consistent state without ever holding a lock across services. {% include excalidraw.html file="05-saga-compensation" - alt="A saga runs T1 charge payment, then T2 reserve stock, then T3 book shipping, which fails. The failure triggers compensation in reverse: C2 release stock, then C1 refund payment. A note: each step is a local transaction; on a failure, run compensating actions in reverse — atomicity without a distributed lock, with the full treatment in Appendix D." - caption="Figure 5.18 — A saga: local transactions forward, compensating actions in reverse when a step fails" %} + alt="A saga runs T1 charge payment, then T2 reserve stock, then T3 book shipping, which fails. The failure triggers compensation in reverse: C2 release stock, then C1 refund payment. A note: each step is a local transaction; on a failure, run compensating actions in reverse: atomicity without a distributed lock, with the full treatment in Appendix D." + caption="Figure 5.18: A saga: local transactions forward, compensating actions in reverse when a step fails" %} A saga can be driven either way. A **choreographed saga** advances on events and triggers -compensations the same way — fully decentralized. An **orchestrated saga** lets the +compensations the same way: fully decentralized. An **orchestrated saga** lets the orchestrator drive both the forward steps and the compensations, which is easier to reason about when there are many steps. A choreographed step is just a consume-act-emit handler that emits "advance" on success and "compensate" on failure: @@ -1155,8 +1155,8 @@ if err := payments.Charge(o.ID, o.Total); err != nil { // local trans Sagas trade atomicity for availability: the system is **eventually** consistent, briefly passing through states where payment is taken but stock isn't yet reserved, which the -compensations resolve. The full mechanics — idempotent compensations, the semantic-lock and -pivot-transaction patterns, and what to do when a step can't be compensated — are the +compensations resolve. The full mechanics (idempotent compensations, the semantic-lock and +pivot-transaction patterns, and what to do when a step can't be compensated) are the subject of **Appendix D**. ## Integrating event-driven architecture into existing systems @@ -1169,18 +1169,18 @@ keeps serving everything else, and continue until the new side has grown around the old can be retired. Two primitives make that incremental migration safe. **Change data capture** liberates the -legacy system's data into the event backbone — the same data-liberation pattern from -earlier — so a new service can build its own view from legacy state without calling into the +legacy system's data into the event backbone (the same data-liberation pattern from +earlier) so a new service can build its own view from legacy state without calling into the monolith. And the **outbox** solves the **dual-write** problem at the seam: writing the business state to the database and publishing an event are two separate systems, and a crash -between them leaves the two inconsistent. The outbox makes them one — write the event into an +between them leaves the two inconsistent. The outbox makes them one: write the event into an `outbox` table *in the same database transaction* as the state change, and let a relay (or CDC on that table) publish it. One commit, no lost or phantom events: {% include excalidraw.html file="05-strangler-integration" alt="An edge router splits traffic: legacy paths go to the legacy monolith plus its database, and migrated paths go to new EDA services that grow over time. The legacy monolith's state is liberated to the event backbone (Kafka) via CDC plus the outbox, and the new services consume from the backbone. A note: the strangler migrates capability by capability behind the router, while CDC and the outbox feed the new side with no dual-write or big-bang cutover." - caption="Figure 5.19 — Strangling a monolith: route by capability, liberate state with CDC and the outbox, grow the new side" %} + caption="Figure 5.19: Strangling a monolith: route by capability, liberate state with CDC and the outbox, grow the new side" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -1266,10 +1266,10 @@ truth. That window is not a bug to hide; it is a property to design for. {% include excalidraw.html file="05-consistency-window" alt="A client writes to the source of truth, which commits and emits an event; the event is applied to a read model or view after a delay of delta-t. A read of the view during that delta-t window may be stale. A note: read-your-writes means routing a writer's own reads to the source, or waiting until the read model reaches the write's version." - caption="Figure 5.20 — The consistency window: a read model lags the source by Δt, which read-your-writes hides for the writer" %} + caption="Figure 5.20: The consistency window: a read model lags the source by Δt, which read-your-writes hides for the writer" %} -Decide where strong consistency is required — usually a -small core, like the payment charge itself — and let everything else converge, with a few +Decide where strong consistency is required (usually a +small core, like the payment charge itself) and let everything else converge, with a few standard tactics to keep it humane: | Tactic | What it does | @@ -1286,7 +1286,7 @@ history. Spend that trade where it pays, and avoid it in the few places it doesn ### Cross-check it yourself -Produce one `order.placed` and watch two independent consumer groups receive it — +Produce one `order.placed` and watch two independent consumer groups receive it: the fan-out is real, not round-robin. Then stop a consumer mid-handler (before it commits), restart it, and confirm the message is **redelivered**, not lost: that redelivery is the at-least-once guarantee, and the reason your handler dedupes. @@ -1297,5 +1297,5 @@ it; its `README.md` covers what it does and how to drive it. The same example, w the Strimzi operator, also runs on a Kubernetes cluster; Appendix R covers that deployment on minikube and Appendix S covers OpenShift Local. --- -*Verification status: verified — [`examples/05-event-driven/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/05-event-driven/) passes 10/10 checks in each of the six languages on Docker Engine (2026-10-13) and on minikube (2026-10-11, Appendix R), and 10/10 in five languages on OpenShift Local 4.22.14 (2026-10-11, Appendix S) +*Verification status: verified; [`examples/05-event-driven/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/05-event-driven/) passes 10/10 checks in each of the six languages on Docker Engine (2026-10-13) and on minikube (2026-10-11, Appendix R), and 10/10 in five languages on OpenShift Local 4.22.14 (2026-10-11, Appendix S) (event publication to order.placed, fan-out to two independent consumer groups (shipping and notification), and idempotent consumers under a redelivered duplicate event).* diff --git a/_docs/06-stream-processing.md b/_docs/06-stream-processing.md index a8fd4429..bd7b84ce 100644 --- a/_docs/06-stream-processing.md +++ b/_docs/06-stream-processing.md @@ -2,7 +2,7 @@ title: "Stream Processing" order: 6 part: "Foundations & the system" -description: "Deriving new streams from streams — stateful, changelog-backed windowed aggregation, and KEDA scaling workers on consumer lag rather than CPU, down to zero when the topic is quiet." +description: "Deriving new streams from streams: stateful, changelog-backed windowed aggregation, and KEDA scaling workers on consumer lag rather than CPU, down to zero when the topic is quiet." duration: 18 minutes --- @@ -20,17 +20,17 @@ The output of one topology is just another stream that other services consume. {% include excalidraw.html file="06-derive-streams" alt="A streaming topology: the orders and payments topics feed a join-and-window operator (keyed by orderId, 5-minute tumbling, with a local state store); its output feeds an aggregate operator (count, sum, rate per merchant); that emits a derived revenue topic, which a Grafana dashboard reads." - caption="Figure 6.1 — A streaming app is a topology of sources, processors, and sinks; each output is just another stream" %} + caption="Figure 6.1: A streaming app is a topology of sources, processors, and sinks; each output is just another stream" %} Reading the topology left to right makes the pattern concrete: two input topics (`orders` and `payments`) are joined and bucketed into five-minute tumbling windows keyed by order id; the windowed result is aggregated into per-merchant counts, sums, and rates; and that aggregate is published as a brand-new `revenue` topic that a -dashboard — or any other service — can consume. Nothing here calls another service +dashboard, or any other service, can consume. Nothing here calls another service synchronously; each stage only reads a stream and writes a stream. The word that matters is **stateful**. Unlike a simple consumer, these processors -keep local state — running totals, window buffers — and that state must survive +keep local state (running totals, window buffers), and that state must survive restarts. The fault-tolerance trick is a **changelog**: the local state store is backed by a compacted Kafka topic, so if the pod dies, the state rebuilds by replaying the changelog. That is fault-tolerant local state with no external @@ -38,8 +38,8 @@ database in the hot path. ## A windowed aggregation -Here is the canonical example — revenue per merchant, in tumbling five-minute -windows — in each ecosystem. The JVM stacks use Kafka Streams; .NET uses Streamiz +Here is the canonical example, revenue per merchant in tumbling five-minute +windows, in each ecosystem. The JVM stacks use Kafka Streams; .NET uses Streamiz (a native Kafka Streams port); Python uses faust-streaming; C++ keeps bounded state in-process and reaches for Flink only when the state outgrows a single worker. @@ -198,7 +198,7 @@ func aggregate(ctx goka.Context, msg any) { Every tab is the same topology: read `order.placed`, group by merchant, bucket into five-minute windows, fold each window into a running total, and write the result to a derived `revenue.by-merchant` stream. The JVM, .NET, Python and Go versions (Kafka Streams, Streamiz, faust-streaming, goka) get a -changelog-backed store built in — kill the pod and the window state rebuilds by +changelog-backed store built in: kill the pod and the window state rebuilds by replaying the changelog topic. The C++ version keeps the window map in process, which is fine while the state is bounded; once you need joins or state larger than a worker, that is the signal to move to a dedicated engine like Flink rather than @@ -210,18 +210,18 @@ The windowed aggregation has a silent assumption: what happens when the worker crashes mid-window, or when a message is delivered twice? Stream processing has three delivery guarantee levels, and picking the right one is a design decision: -**At-most-once** — process each message zero or one times. Commit the offset *before* +**At-most-once**: process each message zero or one times. Commit the offset *before* processing. If the worker crashes after committing but before processing, the message is lost. Fast but lossy; acceptable for metrics that tolerate sampling. -**At-least-once** — process each message one or more times. Commit the offset *after* +**At-least-once**: process each message one or more times. Commit the offset *after* processing. If the worker crashes after processing but before committing, the message is redelivered and processed again. This is the default in most Kafka consumers and -the approach this book prefers — paired with **idempotent handlers** (the same +the approach this book prefers; paired with **idempotent handlers** (the same discipline from the event-driven and workflows chapters), reprocessing is a no-op and the combination is effectively once. -**Exactly-once** — process each message exactly once, even across crashes. Kafka's +**Exactly-once**: process each message exactly once, even across crashes. Kafka's `processing.guarantee=exactly_once_v2` wraps the consume-transform-produce cycle in a transaction: the offset commit and the output record are written atomically, so a crash either rolls back both or commits both. The cost is higher latency (transaction @@ -243,40 +243,40 @@ where double-counting would be incorrect rather than merely redundant. ## Beyond tumbling windows -The revenue example uses a tumbling window — fixed-size, non-overlapping buckets. +The revenue example uses a tumbling window: fixed-size, non-overlapping buckets. Kafka Streams, faust-streaming, and Streamiz support three other window types, each suited to a different access pattern: {% include excalidraw.html file="06-window-types" alt="Three rows of window types. Tumbling: fixed 5-minute buckets with no overlap, so an event lands in exactly one window. Hopping (sliding): 5-minute windows that advance every 1 minute, overlapping, so an event lands in multiple windows, giving a smoothed view. Session: variable-length windows defined by an inactivity gap; no events for 10 minutes closes the window, suited to user-activity sessions." - caption="Figure 6.3 — Three window types: tumbling (fixed, no overlap), hopping (fixed, overlapping), and session (variable, gap-based)" %} + caption="Figure 6.3: Three window types: tumbling (fixed, no overlap), hopping (fixed, overlapping), and session (variable, gap-based)" %} -**Tumbling windows** — fixed-size, non-overlapping. Every event lands in exactly one +**Tumbling windows**: fixed-size, non-overlapping. Every event lands in exactly one window. The revenue-per-merchant aggregation uses this: a clean five-minute bucket, no overlap. -**Hopping (sliding) windows** — fixed-size, but the window advances by a smaller +**Hopping (sliding) windows**: fixed-size, but the window advances by a smaller interval. A five-minute window that advances every one minute produces overlapping windows, giving a **smoothed** view: each output reflects the last five minutes of data, updated every minute. Good for dashboards where a hard bucket boundary would cause a visible cliff. -**Session windows** — variable-length, defined by an **inactivity gap**. No events +**Session windows**: variable-length, defined by an **inactivity gap**. No events for the gap duration (say, 10 minutes) closes the window. This fits user-activity patterns: a user's browsing session is a cluster of events with quiet gaps between them, and the session window captures that shape without you picking an arbitrary bucket size. -**Late-arriving data** — an event whose timestamp falls inside an already-closed +**Late-arriving data**: an event whose timestamp falls inside an already-closed window. Kafka Streams handles this with a **grace period**: windows stay open for an additional duration after their nominal close, accepting late events. Events that arrive after the grace period are dropped (or routed to a dead-letter topic). -Flink extends this with **watermarks** — a system-wide notion of "how far behind -event-time we expect to be" — which is essential for multi-source joins where +Flink extends this with **watermarks**, a system-wide notion of "how far behind +event-time we expect to be", which is essential for multi-source joins where sources have different latencies. -**When to reach for Flink** — in-process stream processing (Kafka Streams, faust-streaming, +**When to reach for Flink**: in-process stream processing (Kafka Streams, faust-streaming, Streamiz, goka, the C++ loop in this chapter's example) works well for single-topic, bounded-state computations. Once you need **stream-to-stream joins** (correlate orders with payments by key and time), **state larger than a single worker's @@ -287,7 +287,7 @@ and watermarks at a scale that in-process libraries don't attempt. ## Scale on the signal that matters A stream processor's load is *backlog*, not CPU. The right scaling signal is -**consumer lag** — how far behind the log the workers are. KEDA reads that lag and +**consumer lag**: how far behind the log the workers are. KEDA reads that lag and scales workers up, and **down to zero** when there is no lag. For synchronous HTTP services the same KEDA closes the loop on request rate instead. One mechanism, two signals. @@ -295,7 +295,7 @@ One mechanism, two signals. {% include excalidraw.html file="06-scale-on-lag" alt="KEDA reads the consumer lag on the order.placed topic and scales the payment-consumer pods up when lag grows and down to zero when the topic is idle" - caption="Figure 6.2 — Scale on lag, not CPU — and to zero when the topic is quiet" %} + caption="Figure 6.2: Scale on lag, not CPU, and to zero when the topic is quiet" %} ```yaml # event-driven: scale consumers on Kafka lag @@ -321,10 +321,10 @@ spec: replicas: { min: 0, max: 30 } ``` -The `lagThreshold: "100"` is the tuning knob — how many messages of backlog a +The `lagThreshold: "100"` is the tuning knob: how many messages of backlog a single worker is allowed before KEDA adds another. `minReplicaCount: 0` is the cost lever: true scale-to-zero when the topic is quiet, springing back the moment lag -builds. Picking the signal — lag for events, request rate for HTTP — is the whole +builds. Picking the signal (lag for events, request rate for HTTP) is the whole point; scaling a backlog-bound consumer on CPU would miss the load entirely. ### Cross-check it yourself @@ -332,12 +332,12 @@ point; scaling a backlog-bound consumer on CPU would miss the load entirely. Make lag visible, then make it move. Pause the `payment-consumer`, publish a burst of `order.placed` with `hey` driving `order-service`, and watch the consumer-group lag climb. Resume it and watch KEDA add replicas until the lag drains, then settle -back toward zero. The replica count tracking the lag curve — not CPU — is the +back toward zero. The replica count tracking the lag curve, not CPU, is the behaviour to confirm. The code is in [`examples/06-stream-processing/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/06-stream-processing/). The run script there builds and runs it; its `README.md` covers what it does and how to drive it. --- -*Verification status: verified — [`examples/06-stream-processing/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/06-stream-processing/) passes 11/11 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified: [`examples/06-stream-processing/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/06-stream-processing/) passes 11/11 checks in each of the six languages on Docker Engine (2026-10-13) (per-merchant revenue aggregation onto a derived Kafka topic, the stream-processor consumer group, consumer lag that builds while the processor is paused and drains on resume, and the burst appearing in the revenue stream).* diff --git a/_docs/07-workflows-and-jobs.md b/_docs/07-workflows-and-jobs.md index 5e3a2499..ca94e8fb 100644 --- a/_docs/07-workflows-and-jobs.md +++ b/_docs/07-workflows-and-jobs.md @@ -2,59 +2,59 @@ title: "Workflows & Jobs" order: 7 part: "The operational platform" -description: "Coordinating multi-step processes — orchestration versus choreography and when each fits — and where non-request work lives on Kubernetes: Jobs, CronJobs, and KEDA-scaled queue workers." +description: "Coordinating multi-step processes: orchestration versus choreography and when each fits, and where non-request work lives on Kubernetes: Jobs, CronJobs, and KEDA-scaled queue workers." duration: 17 minutes --- Part 1 built the synchronous surface and the asynchronous backbone. Part 2 is the platform *around* the system. It opens with the question every multi-service process raises: when a single business action spans several services, who -coordinates it — and where does the work that isn't a request go? +coordinates it, and where does the work that isn't a request go? ## Orchestration vs. choreography There are two ways to coordinate a multi-step process, and they are mirror images. **Orchestration** puts a central coordinator in charge: it commands each service -in turn — reserve stock, take payment, book shipping — and owns the end-to-end +in turn (reserve stock, take payment, book shipping) and owns the end-to-end flow. It is straightforward to follow, observe, and audit. Its failure mode is the **god-service**: the coordinator accretes business logic until it becomes the monolith you were trying to avoid. **Choreography** has no central brain: each service reacts to the previous one's -event and emits its own. It is maximally decoupled — adding a step means adding a +event and emits its own. It is maximally decoupled: adding a step means adding a subscriber. Its failure mode is the opposite: the end-to-end flow is *emergent*, so nobody owns it and nobody can answer "where is order 42 stuck?". The deeper distinction is the direction of control. Orchestration moves **commands** -outward from a coordinator that holds the workflow state explicitly — which is why +outward from a coordinator that holds the workflow state explicitly, which is why you can query that state and answer where any instance is. Choreography moves **events** between peers that each hold only their own slice, so the workflow exists only as the sum of their reactions: cheaper to extend, harder to see. Neither is more "correct"; they trade *visibility and central ownership* against *coupling*, and most -large systems use both — orchestration for the few flows that need an auditable +large systems use both: orchestration for the few flows that need an auditable lifecycle, choreography for the many simple fan-outs around them. {% include excalidraw.html file="07-orchestration-vs-choreography" alt="Top: a saga orchestrator commands payment, shipping, and notification in turn. Bottom: payment, shipping, and notification react to each other's events in a chain with no central coordinator" - caption="Figure 7.1 — A central coordinator that owns the flow, versus services reacting to events" %} + caption="Figure 7.1: A central coordinator that owns the flow, versus services reacting to events" %} ## Choosing between the two The heuristic: - **Orchestration** for complex, long-lived business processes where you need - auditability and a clear owner — order fulfilment, loan approval. You want to be + auditability and a clear owner: order fulfilment, loan approval. You want to be able to ask where a given instance is, and get an answer. - **Choreography** for simple, additive reactions where decoupling matters more - than visibility — send a welcome email when a user signs up. Adding a reaction + than visibility: send a welcome email when a user signs up. Adding a reaction shouldn't require touching a coordinator. The trap is using choreography for a process that's complex, then discovering six months later that no single place describes what the process *is*. When the flow has compensations and a lifecycle, that lifecycle wants to live -somewhere explicit — which is exactly the persisted saga state machine in **Appendix D · Sagas**. +somewhere explicit, which is exactly the persisted saga state machine in **Appendix D · Sagas**. ## Where non-request work lives @@ -62,11 +62,11 @@ The other half of this chapter is a rule: **long, batch, or periodic work never belongs on the request path.** Doing heavy work inside an HTTP handler blocks the request and times out the client. Kubernetes gives you three homes for it: -- A **Job** runs a task to completion and exits — a data migration, a one-off +- A **Job** runs a task to completion and exits: a data migration, a one-off export. It retries on failure up to a `backoffLimit` and is the right home for anything that must run once and finish. -- A **CronJob** schedules Jobs on a cron expression — the reconciliation that runs - at 02:00 — and its `concurrencyPolicy` decides what happens when a run is still +- A **CronJob** schedules Jobs on a cron expression (the reconciliation that runs + at 02:00), and its `concurrencyPolicy` decides what happens when a run is still going as the next is due (skip it, replace it, or allow the overlap). - A **queue worker** continuously drains a topic and is scaled by KEDA on lag, exactly as in the stream-processing chapter; it scales to zero when the topic is @@ -75,7 +75,7 @@ request and times out the client. Kubernetes gives you three homes for it: {% include excalidraw.html file="07-jobs-cronjobs" alt="Three columns of non-request work on Kubernetes. Job: run-to-completion, data backfill, schema migration, retries with backoffLimit. CronJob: scheduled with a cron expression, nightly reconcile, report generation, concurrencyPolicy. Queue worker: long-lived consumer scaled by KEDA, image resize and email, off the request path." - caption="Figure 7.2 — Three homes for non-request work: run-once Jobs, scheduled CronJobs, and KEDA-scaled queue workers" %} + caption="Figure 7.2: Three homes for non-request work: run-once Jobs, scheduled CronJobs, and KEDA-scaled queue workers" %} ```yaml # A CronJob schedules a Job; the Job runs to completion and exits. @@ -102,14 +102,14 @@ is a CronJob, continuous is a scaled Deployment. ## The queue worker the YAML never shows -The CronJob and the ScaledObject are infra — they schedule and scale the work. But +The CronJob and the ScaledObject are infra: they schedule and scale the work. But the *worker itself* is application code: a loop that drains a topic, processes each message off the request path, and commits. {% include excalidraw.html file="07-queue-worker" alt="An HTTP request places an order and returns 201 immediately; the order-service publishes an order.placed fact to Kafka. A separate queue-worker pod (scaled by KEDA) drains the topic, processes the work (send email, resize image, generate report), commits the offset, and the heavy work never blocks the request." - caption="Figure 7.3 — The queue worker drains the topic off the request path; the request returns immediately" %} + caption="Figure 7.3: The queue worker drains the topic off the request path; the request returns immediately" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -215,28 +215,28 @@ for { The point: the HTTP handler publishes the fact and returns immediately. The worker picks it up, does the heavy processing, and commits. Kill the worker and the message waits on the topic; restart it and work resumes from the last -committed offset — which is exactly why idempotency matters. +committed offset, which is exactly why idempotency matters. ## Orchestration with explicit state When a multi-step process has compensations and a lifecycle, that lifecycle wants -to live somewhere explicit — a **state machine row** that you can query, audit, and +to live somewhere explicit: a **state machine row** that you can query, audit, and resume. The orchestrator advances through defined steps, commits each transition with the next command in one transaction (the same outbox discipline from the Data chapter), and is idempotent by construction: re-running a step that already committed is a no-op. -The code for this pattern — the DB-backed saga with row locks, idempotent advance, -and compensation in reverse — is in **Appendix D · Sagas**. What matters here is +The code for this pattern (the DB-backed saga with row locks, idempotent advance, +and compensation in reverse) is in **Appendix D · Sagas**. What matters here is that the state is *queryable*: you can always answer "where is order 42?" by reading the saga row, which is the visibility that choreography sacrifices. One discipline ties all three homes together: **idempotency**. A Job retries on failure, a CronJob can re-run or overlap, and a redelivered message re-invokes a -queue worker — so each must be safe to run more than once. Re-running last night's +queue worker, so each must be safe to run more than once. Re-running last night's reconciliation must not double-count; reprocessing a message must not charge a card twice. This is the same at-least-once-plus-idempotent rule from the event-driven -chapter, applied to scheduled and background work — design the task so a second run +chapter, applied to scheduled and background work: design the task so a second run is a no-op, and the platform's retries become a safety net instead of a hazard. ### Cross-check it yourself @@ -244,11 +244,11 @@ is a no-op, and the platform's retries become a safety net instead of a hazard. Make the boundary visible. Trigger a CronJob's Job manually with `kubectl create job --from=cronjob/nightly-reconcile run-now` and watch it run to completion and the pod enter `Completed`, not `Running`. Separately, push a heavy task through an -HTTP handler and watch the client time out — then move the same task behind a queue +HTTP handler and watch the client time out. Then move the same task behind a queue worker and watch the request return immediately while the work drains in the background. That contrast is the whole rule. --- -*Verification status: conceptual chapter with an illustrative Kubernetes manifest — +*Verification status: conceptual chapter with an illustrative Kubernetes manifest; no per-language runnable example. The coordination patterns it names are made concrete in the saga appendix.* diff --git a/_docs/08-api-management.md b/_docs/08-api-management.md index 1cdd1ad2..aa501f3b 100644 --- a/_docs/08-api-management.md +++ b/_docs/08-api-management.md @@ -2,11 +2,11 @@ title: "API Management" order: 8 part: "The operational platform" -description: "API management as four cooperating planes rather than one gateway product — runtime, contract, discovery, and signal — and versioning and evolution as a managed lifecycle, not an afterthought." +description: "API management as four cooperating planes rather than one gateway product (runtime, contract, discovery, and signal) and versioning and evolution as a managed lifecycle, not an afterthought." duration: 15 minutes --- -"API management" is sold as a product — one box that does gateways, keys, docs, +"API management" is sold as a product: one box that does gateways, keys, docs, and analytics. The cloud-native version is different: it is **four cooperating planes with distinct owners**, each of which is a section of this book. Seeing them as separate responsibilities is what keeps any one of them from becoming a @@ -17,21 +17,21 @@ monolith. {% include excalidraw.html file="08-four-planes" alt="Four planes side by side: the runtime plane (gateway and mesh), the contract plane (registry, section 9), the discovery plane (metadata, section 10), and the signal plane (observability, section 11)" - caption="Figure 8.1 — Four planes, each a section of the platform with its own owner" %} + caption="Figure 8.1: Four planes, each a section of the platform with its own owner" %} -- **Runtime plane** — the gateway and the Istio mesh. Traffic, authentication, +- **Runtime plane**: the gateway and the Istio mesh. Traffic, authentication, rate limiting, and resilience happen here, on the wire, as configuration. This is where a `429 Too Many Requests` with a `Retry-After` comes from, and it is policy, not application code. -- **Contract plane** — the registry (**09 · API Registry**). Every REST, gRPC, +- **Contract plane**: the registry (**09 · API Registry**). Every REST, gRPC, and event schema, versioned and gated in CI. -- **Discovery plane** — the catalog and metadata (**10 · API Metadata**). How a +- **Discovery plane**: the catalog and metadata (**10 · API Metadata**). How a human finds an API and decides whether to trust it. -- **Signal plane** — observability (**11 · Observability**). What is +- **Signal plane**: observability (**11 · Observability**). What is happening to the APIs in production. The point is the separation. A single "API gateway product" that tries to own all -four becomes exactly the god-service the workflows chapter warned about — runtime +four becomes exactly the god-service the workflows chapter warned about: runtime policy, contract storage, discovery UX, and telemetry pipelines have different owners and release cadences, and fusing them into one box recreates the monolith at the edge. Distinct planes have distinct owners and evolve independently; @@ -48,14 +48,14 @@ In this stack each plane maps to a specific tool with a specific owner: | **Discovery** | OpenMetadata | Data governance | Searchable catalog, lineage, ownership, PII tags | | **Signal** | LGTM (Loki, Grafana, Tempo, Mimir) | SRE / platform team | Logs, traces, metrics, dashboards, alerts | -The separation is enforced by ownership, not by good intentions — the platform team +The separation is enforced by ownership, not by good intentions: the platform team cannot break the contract plane because they don't own the registry; the API team cannot change the retry policy because they don't own the mesh config. ## The runtime plane in practice: rate limiting The cross-check section tells you to confirm the gateway returns `429` with a -`Retry-After` — here is the configuration that makes it happen. The rate limit is a +`Retry-After`: here is the configuration that makes it happen. The rate limit is a runtime-plane concern: it lives in the mesh, not in application code, and it applies uniformly to every caller. @@ -94,7 +94,7 @@ spec: ``` The `Retry-After: 1` header is what makes the 429 useful rather than merely -hostile — the client knows *when* to retry, not just that it was rejected. This is +hostile: the client knows *when* to retry, not just that it was rejected. This is policy, not code; changing the rate or the burst is a config commit, not a redeploy. ## Versioning and evolution are part of management @@ -108,41 +108,41 @@ written against last year's contract should still work against this year's servi without a single change. When you must break compatibility, you don't mutate the existing -contract — you stand up a new major version beside the old one. Carry the major in +contract: you stand up a new major version beside the old one. Carry the major in the path (`/v2`) or in the media type, run both side by side, publish a **sunset -date**, and migrate consumers off before retiring the old one. The mechanics — -four ways to version, additive evolution, deprecation headers — are the subject of +date**, and migrate consumers off before retiring the old one. The mechanics +(four ways to version, additive evolution, deprecation headers) are the subject of **Appendix B**. What makes this more than good intentions is that the registry enforces it -**mechanically**. Its compatibility rules — `BACKWARD`, `FORWARD`, `FULL` — are +**mechanically**. Its compatibility rules (`BACKWARD`, `FORWARD`, `FULL`) are checked at publish time, so a change that would break existing consumers is rejected before it ships. Every contract therefore moves through one managed lifecycle: {% include excalidraw.html file="08-contract-lifecycle" alt="A six-stage horizontal lifecycle: design, register, publish, consume, deprecate, retire. Each stage flows into the next; the last two transitions (deprecate, retire) are highlighted as the wind-down." - caption="Figure 8.2 — One managed contract lifecycle; each arrow is an owned transition, not an accident" %} + caption="Figure 8.2: One managed contract lifecycle; each arrow is an owned transition, not an accident" %} Each arrow is a managed transition with an owner, not an accident. A contract is *designed* against the domain, *registered* in the schema registry, *published* once -CI's compatibility check passes, *consumed* by services that code against it, and — -when a successor exists — *deprecated* with headers and a sunset date before it is +CI's compatibility check passes, *consumed* by services that code against it, and, +when a successor exists, *deprecated* with headers and a sunset date before it is finally *retired*. Skipping a step is where breakage comes from: publishing without the registry check ships an incompatible schema, and retiring without a deprecation window strands consumers who never got the signal to move. The registry and the -metadata catalog — the next two chapters — are what make the middle of this lifecycle +metadata catalog (the next two chapters) are what make the middle of this lifecycle real and observable. ### Cross-check it yourself The runtime plane is observable from the outside. Drive `order-service` past its limit with `hey` and confirm the gateway returns `429` with a `Retry-After` rather -than letting the load through — that response is the runtime plane doing its job as +than letting the load through: that response is the runtime plane doing its job as configuration. The contract-plane enforcement is demonstrated in the registry chapter, where a breaking change is rejected at publish time. --- -*Verification status: conceptual chapter — no per-language runnable code. The +*Verification status: conceptual chapter: no per-language runnable code. The runtime-plane and contract-plane behaviours it describes are exercised in the security, registry, and Appendix B chapters.* diff --git a/_docs/09-api-registry.md b/_docs/09-api-registry.md index 1d6386de..8cdfcf51 100644 --- a/_docs/09-api-registry.md +++ b/_docs/09-api-registry.md @@ -2,7 +2,7 @@ title: "API Registry" order: 9 part: "The operational platform" -description: "One authoritative, versioned home for every contract — OpenAPI, .proto, AsyncAPI, and event schemas — with Apicurio's compatibility rules gating breaking changes in CI before code ships." +description: "One authoritative, versioned home for every contract (OpenAPI, .proto, AsyncAPI, and event schemas) with Apicurio's compatibility rules gating breaking changes in CI before code ships." duration: 17 minutes --- @@ -15,7 +15,7 @@ ships." Every contract type lands in the same place: REST OpenAPI specs, GraphQL SDL, gRPC `.proto` files, and event schemas (Avro or JSON). In this stack the registry is -**Apicurio**. Each artifact carries versions and a **compatibility rule** — +**Apicurio**. Each artifact carries versions and a **compatibility rule**: `BACKWARD`, `FORWARD`, or `FULL`. Producers register; consumers fetch and generate stubs from the registered version, so no one hand-writes parsing and everyone is reading the same source of truth. The Kafka serde libraries do this automatically: @@ -25,16 +25,16 @@ it. {% include excalidraw.html file="09-one-registry" alt="REST design (OpenAPI), gRPC (.proto), and events (AsyncAPI) all register into one Apicurio Registry that holds OpenAPI, AsyncAPI, proto, Avro, and JSON Schema artifacts with versions and compatibility rules. The registry fans out to three consumers: a CI gate that blocks breaking changes, client codegen for typed stubs, and runtime resolution that serdes by schema-id." - caption="Figure 9.1 — One registry for every contract type, feeding the CI gate, code generation, and runtime serde" %} + caption="Figure 9.1: One registry for every contract type, feeding the CI gate, code generation, and runtime serde" %} -Read it design-first: every contract type — REST, gRPC, and events alike — is +Read it design-first: every contract type (REST, gRPC, and events alike) is registered *before* code depends on it, and three different consumers read from that one home. CI gates breaking changes against it, client builds generate typed stubs from it, and at runtime the Kafka serde resolves the exact schema by the id carried on each message. One authoritative source, three jobs. Because every artifact is just bytes addressed by group and id, you talk to the -registry over plain HTTP — the same `curl` you already use, from any language or a +registry over plain HTTP; the same `curl` you already use, from any language or a pipeline step: ```bash @@ -172,13 +172,13 @@ func publish(group, artifactID string, schema []byte) (int, error) { This is the guardrail that makes the whole contract story real. The CI pipeline posts the new schema version to the registry; if it violates the artifact's compatibility rule, Apicurio returns **`400 Bad Request`** and the step fails the -build, blocking the merge. The contract is enforced *before* code ships — not +build, blocking the merge. The contract is enforced *before* code ships, not discovered broken in production by a downstream team three days later. {% include excalidraw.html file="09-registry-gate" alt="A pull request with a schema change posts the new version to the Apicurio registry; the registry checks it against the compatibility rule and either allows merge and publish, or returns 400 and blocks the merge; consumers then fetch the published schema and generate stubs" - caption="Figure 9.2 — The registry checks compatibility at publish time and blocks the merge on a breaking change" %} + caption="Figure 9.2: The registry checks compatibility at publish time and blocks the merge on a breaking change" %} ```yaml # .github/workflows/contracts.yml — block breaking changes before merge @@ -207,23 +207,23 @@ The compatibility rules tell you *whether* a change is safe; a strategy tells yo *how* to evolve a schema over time without breaking consumers. Four practices, in order of increasing cost: -1. **Additive-only** — add optional fields with defaults; never remove or rename. This +1. **Additive-only**: add optional fields with defaults; never remove or rename. This is the cheapest evolution: old consumers ignore the new field, new consumers fall back to the default when reading old data. If every change is additive, the compatibility rule never fires. -2. **Deprecation workflow** — when a field must go, mark it `@deprecated` (GraphQL, +2. **Deprecation workflow**: when a field must go, mark it `@deprecated` (GraphQL, proto) or add a `deprecation` annotation in the schema. Publish the deprecation, monitor usage (field-level metrics, consumer-group offsets), and remove the field only once all consumers have migrated. This is the same add → deprecate → watch → retire cycle from **Appendix B**. -3. **Default values for new required fields** — if a new field must be present in all +3. **Default values for new required fields**: if a new field must be present in all messages, add it with a default so old data on the topic can still be deserialized. Without the default, every consumer that encounters a message written before the field existed will fail to deserialize it. -4. **Union types for polymorphism** — when a single topic carries multiple event shapes +4. **Union types for polymorphism**: when a single topic carries multiple event shapes (e.g. `OrderPlaced`, `OrderCancelled`), use Avro unions or protobuf `oneof` so the schema accommodates each variant. This keeps the topic type-safe without splitting it into per-event topics. @@ -235,7 +235,7 @@ how often you approach that boundary. Teams that evolve additively rarely trigge ## Validate before you publish Before registering a new version you can ask the registry to **dry-run** the -compatibility check — validate the schema without creating a version. This is useful +compatibility check: validate the schema without creating a version. This is useful in local development and pre-commit hooks, where you want fast feedback without polluting the registry with test versions. @@ -333,12 +333,12 @@ func isCompatible(group, artifactID string, schema []byte) (bool, error) { ### Cross-check it yourself Prove the gate bites. Register `order-placed.avsc`, then make a breaking -change — rename a required field — and `curl` the new version: you should get a -`400`, and the CI step should exit non-zero. Then make an additive change — a new -*optional* field — and confirm it returns `200` and publishes a new version. The +change (rename a required field) and `curl` the new version: you should get a +`400`, and the CI step should exit non-zero. Then make an additive change (a new +*optional* field) and confirm it returns `200` and publishes a new version. The breaking change failing the build and the additive change passing it is the entire point of the registry. --- -*Verification status: verified — [`examples/09-api-registry/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/09-api-registry/) passes 8/8 checks on Docker Engine (2026-10-13) +*Verification status: verified; [`examples/09-api-registry/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/09-api-registry/) passes 8/8 checks on Docker Engine (2026-10-13) (schema registration, BACKWARD compatibility rule, rejection of a breaking change with 400, acceptance of an additive change, versioned retrieval).* diff --git a/_docs/10-api-metadata.md b/_docs/10-api-metadata.md index df738624..9cb21c67 100644 --- a/_docs/10-api-metadata.md +++ b/_docs/10-api-metadata.md @@ -2,20 +2,20 @@ title: "API Metadata" order: 10 part: "The operational platform" -description: "The discovery and trust plane — a searchable catalog of every API, topic, and table with ownership, automated lineage, and quality signals, so you can answer 'if I change this field, who breaks?' before you change it." +description: "The discovery and trust plane: a searchable catalog of every API, topic, and table with ownership, automated lineage, and quality signals, so you can answer 'if I change this field, who breaks?' before you change it." duration: 14 minutes --- A contract that is enforced but undiscoverable is half useless. If a consumer can't find your API and decide whether to trust it, they'll build their own or -screen-scrape yours — and the boundary you worked to establish erodes. +screen-scrape yours, and the boundary you worked to establish erodes. Metadata is the **discovery and trust plane**: a searchable catalog of every API, topic, and table, with ownership, lineage, and quality signals attached. ## Discovery, lineage, and ownership -In this stack the catalog is **OpenMetadata**. It ingests from many sources — -OpenAPI specs, the Apicurio registry, Kafka topics, Postgres schemas — through +In this stack the catalog is **OpenMetadata**. It ingests from many sources +(OpenAPI specs, the Apicurio registry, Kafka topics, Postgres schemas) through *scheduled connectors*, and builds one searchable catalog that stays current without manual upkeep. That last point matters: a wiki is stale the day after it's written; an ingested catalog is not. @@ -23,18 +23,18 @@ written; an ingested catalog is not. {% include excalidraw.html file="10-metadata-catalog" alt="OpenMetadata ingests from OpenAPI specs, the Apicurio registry, Kafka topics, and Postgres schemas through scheduled connectors, building one searchable catalog that delivers discovery, trust, lineage, and ownership to consumers" - caption="Figure 10.1 — One catalog, fed by connectors, answering the change-safety question" %} + caption="Figure 10.1: One catalog, fed by connectors, answering the change-safety question" %} The real differentiator over a wiki is **automated lineage**. OpenMetadata knows -that *this* topic feeds *that* table feeds *this* API — column- and topic-level. So +that *this* topic feeds *that* table feeds *this* API, column- and topic-level. So it can answer the question that otherwise gets answered the hard way, in production: > if I change this field, who breaks? -— and answer it *before* you make the change, not after a downstream team's +And it answers that *before* you make the change, not after a downstream team's dashboard goes blank. OpenMetadata derives this graph automatically from what it -ingests — parsing schemas, topic configurations, and query history — so the lineage +ingests (parsing schemas, topic configurations, and query history), so the lineage stays accurate as the system changes, with no one drawing a dependency diagram by hand. @@ -45,15 +45,15 @@ Four concrete payoffs, each replacing a thing teams currently do by asking aroun {% include excalidraw.html file="10-catalog-payoffs" alt="Four columns of what the catalog gives every consumer. Discovery: search across APIs, topics, and tables, find the owning team and how to get access. Trust: published SLOs, data-quality results, freshness, and a criticality tier. Lineage: column- and topic-level impact analysis answering who breaks if I change this. Ownership: tags and PII classification, an accountable owner, and compliance-readiness." - caption="Figure 10.2 — Four payoffs the catalog gives every consumer: discovery, trust, lineage, and ownership" %} + caption="Figure 10.2: Four payoffs the catalog gives every consumer: discovery, trust, lineage, and ownership" %} -- **Discovery** — search across APIs, topics, and tables, and find the owning team +- **Discovery**: search across APIs, topics, and tables, and find the owning team and how to get access. No more hunting through Slack for who owns `inventory`. -- **Trust** — published SLOs, data-quality test results, freshness, and a +- **Trust**: published SLOs, data-quality test results, freshness, and a criticality tier tell you whether this thing is safe to depend on. -- **Lineage** — column- and topic-level impact analysis answers the change-safety +- **Lineage**: column- and topic-level impact analysis answers the change-safety question proactively, for both producers and consumers. -- **Ownership and governance** — tags, PII classifications, and an accountable +- **Ownership and governance**: tags, PII classifications, and an accountable owner on every asset. This is also exactly what a compliance team needs, so the catalog does double duty. @@ -208,7 +208,7 @@ func lineage(ctx context.Context, topicFQN string) (map[string]any, error) { ## Data contracts as code The catalog is most valuable when its metadata lives in Git alongside the schemas it -describes — **data contracts as code**. A data contract is a YAML or JSON document +describes: **data contracts as code**. A data contract is a YAML or JSON document committed next to the schema that declares the asset's owner, its SLO (freshness, completeness), its PII classification, and its criticality tier. The CI pipeline that gates schema changes also validates the contract, so ownership and quality expectations @@ -224,12 +224,12 @@ build on. The catalog's value shows up as a question answered in seconds instead of a day. With the stack running, search the catalog for the `order.placed` topic and -follow its lineage downstream — confirm you can see, without asking anyone, which +follow its lineage downstream: confirm you can see, without asking anyone, which tables and APIs consume it and which team owns each. Then check that an asset carries its owner and PII tags. Getting that impact list from the catalog rather than from a post-incident retro is the whole point. --- -*Verification status: conceptual chapter — no per-language runnable code. The +*Verification status: conceptual chapter: no per-language runnable code. The discovery, lineage, and trust capabilities it describes are exercised against a running OpenMetadata catalog in the example stack.* diff --git a/_docs/11-observability.md b/_docs/11-observability.md index 599248ff..4a584164 100644 --- a/_docs/11-observability.md +++ b/_docs/11-observability.md @@ -2,12 +2,12 @@ title: "Observability" order: 11 part: "The operational platform" -description: "Traces, metrics, and logs from OpenTelemetry, correlated by one trace id across the LGTM stack — auto-instrumentation, W3C context propagation, and head- vs. tail-based sampling." +description: "Traces, metrics, and logs from OpenTelemetry, correlated by one trace id across the LGTM stack: auto-instrumentation, W3C context propagation, and head- vs. tail-based sampling." duration: 28 minutes --- The signal plane. Once you have services, events, and a platform, the question is -what is happening in production — and the answer is three signals, +what is happening in production, and the answer is three signals, emitted by **OpenTelemetry** (vendor-neutral) and stored in an LGTM-style stack (Loki, Grafana, Tempo, Mimir). @@ -15,10 +15,10 @@ emitted by **OpenTelemetry** (vendor-neutral) and stored in an LGTM-style stack Each signal answers a different question: -- **Traces** — where did the time go, and which service failed, in *this one* +- **Traces**: where did the time go, and which service failed, in *this one* request? -- **Metrics** — what are the aggregate rates, errors, and latencies over time? -- **Logs** — what exactly happened inside this component? +- **Metrics**: what are the aggregate rates, errors, and latencies over time? +- **Logs**: what exactly happened inside this component? The power isn't any one signal; it's that they share a **trace id**, so you can pivot from a latency spike on a metric, to the slow trace behind it, to the logs of @@ -28,33 +28,33 @@ tools. {% include excalidraw.html file="11-three-signals" alt="One OpenTelemetry SDK and Collector instrumentation layer fans out to three stores: Traces (request across services, Tempo, viewed in Grafana), Metrics (RED/USE histograms, Prometheus to Mimir, alerts and SLOs), and Logs (structured, trace-linked, Loki, filtered by trace_id). A band notes Grafana as the single pane and Kiali as the service-mesh topology view." - caption="Figure 11.1 — Instrument once with OpenTelemetry; three signals land in the LGTM stack, correlated by trace_id" %} + caption="Figure 11.1: Instrument once with OpenTelemetry; three signals land in the LGTM stack, correlated by trace_id" %} It is worth widening the lens slightly: the classic three signals now travel with two -more that matter increasingly — **baggage**, the key-value context propagated along +more that matter increasingly: **baggage**, the key-value context propagated along the whole trace, and **profiles**, continuous CPU and memory sampling per service. OpenTelemetry is the one vendor-neutral SDK and wire format for all of them. {% include excalidraw.html file="11-signals-in-depth" alt="Five signal columns. Traces: request flow across services, latency per span, where did time go, Tempo/Jaeger. Metrics: aggregate rates, errors, latencies (RED/USE), what's the trend, Prometheus/Mimir. Logs: discrete timestamped events, what exactly happened, Loki/ELK. Baggage: key-value context carried along the trace, tenant_id downstream, W3C Baggage. Profiles: continuous CPU/memory sampling, which function burns CPU, Pyroscope/Parca. All correlated by trace_id, span_id, and resource attributes." - caption="Figure 11.2 — The signals in depth: each answers a different question, all correlated by trace_id" %} + caption="Figure 11.2: The signals in depth: each answers a different question, all correlated by trace_id" %} -What each answers differs — traces locate *where*, metrics show the *trend*, logs say +What each answers differs: traces locate *where*, metrics show the *trend*, logs say *what exactly*, baggage carries *who or which* downstream, profiles reveal *which -function* — but the power is the `trace_id` that lets you pivot from one to the next. +function*. But the power is the `trace_id` that lets you pivot from one to the next. ## Three kinds of instrumentation Keep three things distinct, because people conflate them: -- **Manual instrumentation** — you write the spans (`start_span`, +- **Manual instrumentation**: you write the spans (`start_span`, `set_attribute`). Needed for *business* operations the libraries can't know - about — `reserve_stock`, `score_fraud`. Most signal per line, most effort. -- **Auto-instrumentation** — a library or agent wraps known frameworks (HTTP, gRPC, + about, such as `reserve_stock`, `score_fraud`. Most signal per line, most effort. +- **Auto-instrumentation**: a library or agent wraps known frameworks (HTTP, gRPC, database drivers, Kafka) into spans with zero code change. Covers the plumbing, misses your domain. -- **Auto-injection** — the platform (the OpenTelemetry Operator) injects the +- **Auto-injection**: the platform (the OpenTelemetry Operator) injects the auto-instrumentation agent into the pod at admission, so you don't even add the dependency. @@ -196,10 +196,10 @@ func placeOrder(w http.ResponseWriter, r *http.Request) { Every tab does two things: turn on auto-instrumentation for the frameworks (one line of config, or a builder chain in .NET), and add **one** manual span around the -business operation — `reserve_stock`. The plumbing spans (HTTP in, gRPC out, Kafka +business operation: `reserve_stock`. The plumbing spans (HTTP in, gRPC out, Kafka publish) need no code; the domain span is the bit only you can write. The one exception is C++: there is no auto-instrumentation for Drogon or -`librdkafka`, so you add a request-scoped middleware span by hand — the +`librdkafka`, so you add a request-scoped middleware span by hand. The chapter says so rather than pretending the agent exists. ## The other two signals, in code @@ -207,7 +207,7 @@ chapter says so rather than pretending the agent exists. Auto-instrumentation creates the trace spans, but the **metric** and the **trace-correlated log** are a line you write at the business moment. Emit a counter (or histogram) for the operation, and log through the OTel-aware logger so the active -`trace_id` is stamped onto the line — that stamp is what lets you jump from a log +`trace_id` is stamped onto the line; that stamp is what lets you jump from a log straight to its trace: {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -306,7 +306,7 @@ three correlated signals from one place in the code. ## How correlation works "Correlation" sounds like backend magic; it isn't. The trace context travels -**with the request, in band, at every hop** — the W3C `traceparent` header, +**with the request, in band, at every hop**: the W3C `traceparent` header, formatted `version-traceid-parentspanid-flags`. The `trace_id` (16 bytes) is *constant* across every hop, and that constancy is literally what stitches the spans into one trace. @@ -314,20 +314,20 @@ spans into one trace. {% include excalidraw.html file="11-trace-propagation" alt="One trace id propagates via the W3C traceparent header from the order-service REST span, through the inventory gRPC span, onto the published order.placed Kafka event, and into the notification consumer span" - caption="Figure 11.3 — One trace id, carried in band across REST, gRPC, and Kafka" %} + caption="Figure 11.3: One trace id, carried in band across REST, gRPC, and Kafka" %} So a single `order.placed` request becomes one readable waterfall across five -services — edge REST, the gRPC `ReserveStock`, the published event, the consumer — +services (edge REST, the gRPC `ReserveStock`, the published event, the consumer), and you can point at the time axis and see exactly which span dominated the latency. {% include excalidraw.html file="11-trace-stitch" alt="A trace waterfall over a time axis: a long REST POST /orders span contains a nested gRPC ReserveStock span, then a publish order.placed span, then a consume and notify span. All share one trace_id, shown as 4af9c, carried onto every span, metric exemplar, and log line." - caption="Figure 11.4 — One trace_id stitches REST, gRPC, the published event, and the consumer into a single waterfall" %} + caption="Figure 11.4: One trace_id stitches REST, gRPC, the published event, and the consumer into a single waterfall" %} Context propagation also carries application context you choose, not just the trace -ids — that is **baggage**. Set a key once at the edge and it rides every downstream +ids: that is **baggage**. Set a key once at the edge and it rides every downstream hop in the W3C `baggage` header, so a value like `tenant.id` is available to every service and can be attached to their spans and logs without threading it through each call signature: @@ -386,29 +386,29 @@ func withTenant(ctx context.Context, tenantID string) context.Context { // any gRPC / Kafka call made with this ctx carries tenant.id downstream ``` -Use baggage sparingly — it is copied onto every span and every downstream request, so +Use baggage sparingly: it is copied onto every span and every downstream request, so a handful of small, high-value keys (tenant, request priority) is the right dose, not a grab-bag. -## Sampling — head vs. tail +## Sampling: head vs. tail At scale you can't store every trace, so you sample, and the only real question is *when* the keep/drop decision is made. {% include excalidraw.html file="11-sampling" - alt="Two rows. Head sampling: a request enters, the SDK decides now (keep 1 in N probabilistically) before the trace completes, and the Collector forwards only sampled traces to the backend — cheap and low overhead but blind to errors and latency at decision time. Tail sampling: a request enters, the SDK sends all spans with no decision, the Collector buffers the whole trace and then keeps errors, slow, and rare-route traces, forwarding a kept subset to the backend — keeps all errors and slow traces but costs buffering and CPU." - caption="Figure 11.5 — Head sampling decides early in the SDK; tail sampling decides late in the Collector, after the whole trace is seen" %} + alt="Two rows. Head sampling: a request enters, the SDK decides now (keep 1 in N probabilistically) before the trace completes, and the Collector forwards only sampled traces to the backend: cheap and low overhead but blind to errors and latency at decision time. Tail sampling: a request enters, the SDK sends all spans with no decision, the Collector buffers the whole trace and then keeps errors, slow, and rare-route traces, forwarding a kept subset to the backend: keeps all errors and slow traces but costs buffering and CPU." + caption="Figure 11.5: Head sampling decides early in the SDK; tail sampling decides late in the Collector, after the whole trace is seen" %} -**Head sampling** decides at the start, in the SDK, before the trace completes — +**Head sampling** decides at the start, in the SDK, before the trace completes: "keep 1 in N" (probabilistic) or "keep N/second" (rate-limiting). Cheap, but blind: it can't keep a trace *because* it errored, since it doesn't know yet. One -head policy is non-negotiable in a distributed system — **parent-based**: honour +head policy is non-negotiable in a distributed system, **parent-based**: honour the upstream's decision carried in the `traceparent` flags, so you don't sample at service A and drop at B and get broken half-traces. **Tail sampling** buffers the whole trace in the Collector and decides after it -finishes — so it can keep every error and every slow trace and sample the boring +finishes, so it can keep every error and every slow trace and sample the boring rest. More memory and latency at the Collector, far better signal. ```yaml @@ -428,13 +428,13 @@ processors: probabilistic: { sampling_percentage: 10 } ``` -The shape — keep errors, keep slow, sample the rest — is the one most teams want: +The shape (keep errors, keep slow, sample the rest) is the one most teams want: you never drop the traces you'd investigate, and you pay full storage only for the boring majority you sample down. ### Cross-check it yourself -Prove the correlation with the house tools — no heavy framework needed. Drive a +Prove the correlation with the house tools; no heavy framework needed. Drive a request through `order-service` with `curl`, grab the `trace_id`, and find that same id on the gRPC span, the Kafka event, and the consumer span in Tempo: one id, five services. Then push load with `hey` (HTTP) and `ghz` (gRPC) and watch the @@ -443,5 +443,5 @@ the rest. Seeing one request as one waterfall is the moment observability stops being three disconnected tools. --- -*Verification status: verified — [`examples/11-observability/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/11-observability/) passes 10/10 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified: [`examples/11-observability/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/11-observability/) passes 10/10 checks in each of the six languages on Docker Engine (2026-10-13) (trace propagation, metric export, structured log correlation, LGTM stack integration).* diff --git a/_docs/12-security.md b/_docs/12-security.md index a808d0e1..4649288f 100644 --- a/_docs/12-security.md +++ b/_docs/12-security.md @@ -2,12 +2,12 @@ title: "Security" order: 12 part: "Security & anti-patterns" -description: "Security as four concentric layers — cloud, cluster, container, code — and seven patterns that secure them, from the sidecar that reads trusted identity headers to policy-as-code enforced at admission." +description: "Security as four concentric layers (cloud, cluster, container, code) and seven patterns that secure them, from the sidecar that reads trusted identity headers to policy-as-code enforced at admission." duration: 30 minutes --- Security isn't a feature you bolt on; it's a property of every layer, and the -cloud-native way to reason about it is the CNCF's **4 C's** — concentric layers, +cloud-native way to reason about it is the CNCF's **4 C's**: concentric layers, each protecting what's inside it, so a breach of one is contained by the next. ## The 4 C's of cloud-native security @@ -15,17 +15,17 @@ each protecting what's inside it, so a breach of one is contained by the next. {% include excalidraw.html file="12-four-cs" alt="Four concentric layers: Cloud (IAM, VPC, KMS, audit) on the outside, then Cluster (RBAC, NetworkPolicy, Pod Security, admission), then Container (minimal image, non-root, read-only filesystem, signed), with Code (authz, input validation, secrets, dependencies) at the centre" - caption="Figure 12.1 — Cloud → Cluster → Container → Code, each layer protecting the one inside" %} + caption="Figure 12.1: Cloud → Cluster → Container → Code, each layer protecting the one inside" %} -- **Cloud** — the perimeter, identities, and keys: IAM with least privilege, VPC +- **Cloud**: the perimeter, identities, and keys: IAM with least privilege, VPC isolation, KMS, audit logging, data-residency. Owned by the platform/cloud team. Failures here look like "someone left a bucket open" or "an over-permissive role enabled lateral movement." -- **Cluster** — the Kubernetes layer: RBAC, `NetworkPolicy` (default-deny plus +- **Cluster**: the Kubernetes layer: RBAC, `NetworkPolicy` (default-deny plus targeted allows), Pod Security Standards (restricted), admission control. -- **Container** — a minimal image, non-root, read-only root filesystem, and a +- **Container**: a minimal image, non-root, read-only root filesystem, and a *signed* image so only trusted artifacts run. -- **Code** — your application: authorization, input validation, secret handling, +- **Code**: your application: authorization, input validation, secret handling, and keeping dependencies patched. The value of the model is that controls live at the right layer with the right @@ -35,34 +35,34 @@ owner, instead of every team trying to do everything in application code. These span the 4 C's, and you'll meet several again in the appendices: -- **Sidecar** — move cross-cutting security (mTLS, JWT, audit, rate-limiting) into a +- **Sidecar**: move cross-cutting security (mTLS, JWT, audit, rate-limiting) into a companion process. *(Cluster/container layer.)* -- **Valet key** — hand the client a short-lived signed token for *direct* access to +- **Valet key**: hand the client a short-lived signed token for *direct* access to a resource, so your app stops proxying large payloads and sheds attack surface. -- **Strangler fig** — encapsulate legacy auth behind a gateway and replace it piece +- **Strangler fig**: encapsulate legacy auth behind a gateway and replace it piece by piece. *(The modernisation pattern from Appendix K, applied to security.)* -- **Zero-trust / identity-aware proxy** — every request authenticated and +- **Zero-trust / identity-aware proxy**: every request authenticated and authorised; perimeter trust is dead. -- **Policy-as-code** — security gates declared in Git, enforced at admission and in +- **Policy-as-code**: security gates declared in Git, enforced at admission and in CI. -- **Bulkhead** — isolate resources (per-tenant pools) so one tenant's overload +- **Bulkhead**: isolate resources (per-tenant pools) so one tenant's overload can't sink the rest. *(Returns in Appendix M.)* -- **Claim-check** — for large payloads, publish a *reference* and let the consumer +- **Claim-check**: for large payloads, publish a *reference* and let the consumer fetch the body, keeping secrets and bulk out of the event stream. ## The sidecar reads trusted headers The sidecar pattern is the one to internalise, because the mesh chapters already put an Envoy sidecar in every pod. It terminates mTLS, validates JWTs, and enforces -policy — so the app receives only authenticated, authorised requests and reads the +policy, so the app receives only authenticated, authorised requests and reads the identity as **plain, trusted headers**. Two rules: the app must **reject** any request missing the sidecar-set identity, and it must **never re-validate** the JWT -signature — the sidecar already did, and re-doing it duplicates trust and drifts. +signature. The sidecar already did, and re-doing it duplicates trust and drifts. {% include excalidraw.html file="12-sidecar" - alt="Clients reach a Kubernetes pod whose app container holds business logic only — no TLS, auth, rate-limit, or audit code — talking over localhost to an Envoy/OPA sidecar that terminates mTLS, validates JWTs, rate-limits, audits, enforces policy, and emits metrics and traces before egress. The app trusts the sidecar's headers." - caption="Figure 12.2 — Security concerns move out of the app and into an Envoy/OPA sidecar in the same pod" %} + alt="Clients reach a Kubernetes pod whose app container holds business logic only (no TLS, auth, rate-limit, or audit code), talking over localhost to an Envoy/OPA sidecar that terminates mTLS, validates JWTs, rate-limits, audits, enforces policy, and emits metrics and traces before egress. The app trusts the sidecar's headers." + caption="Figure 12.2: Security concerns move out of the app and into an Envoy/OPA sidecar in the same pod" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -156,7 +156,7 @@ func trustSidecar(next http.Handler) http.Handler { } ``` -The `X-Forwarded-Client-Cert` carries the validated caller identity from mTLS — +The `X-Forwarded-Client-Cert` carries the validated caller identity from mTLS: typically a SPIFFE id like `spiffe://cluster.local/ns/orders/sa/order-service`. The app reads it as a value; it does no certificate work itself. Every tab is the same two moves: deny if the identity header is absent, and trust (never re-check) the @@ -164,7 +164,7 @@ claims the sidecar forwarded. ## The valet key hands out direct access -When a client needs a *large* resource — upload an attachment, download an export — +When a client needs a *large* resource (upload an attachment, download an export), proxying those bytes through your service is wasted work and extra attack surface. The **valet key** pattern hands the client a short-lived, scoped, signed token and lets it talk to the storage system directly. @@ -172,7 +172,7 @@ talk to the storage system directly. {% include excalidraw.html file="12-valet-key" alt="A client requests a key from the application, which mints a scoped, time-bound, operation-restricted valet key and returns a signed token. The client then PUTs or GETs directly against the resource (object store, file) using the token; the storage/KMS layer validates the token, permits the named operation, and rejects everything else. The app never proxies the bytes." - caption="Figure 12.3 — The app mints a scoped, time-bound token; the client accesses storage directly and the app never proxies the bytes" %} + caption="Figure 12.3: The app mints a scoped, time-bound token; the client accesses storage directly and the app never proxies the bytes" %} The application mints a key that is *scoped* (this one object), *time-bound* (minutes, not forever), and *operation-restricted* (PUT or GET, not both). The client uses it to @@ -340,7 +340,7 @@ accept signed URLs of the same shape: the method, the object key and the expiry ## Zero-trust: every request proves itself -The old perimeter model — VPN in, then trust everything on the intranet — fails the +The old perimeter model (VPN in, then trust everything on the intranet) fails the moment one host is compromised: the attacker is *inside*, and lateral movement is trivial. **Zero-trust** discards the perimeter. Every request, even service-to-service inside the cluster, must carry a verifiable identity and pass policy on *every* hop. @@ -348,7 +348,7 @@ inside the cluster, must carry a verifiable identity and pass policy on *every* {% include excalidraw.html file="12-zero-trust" alt="Top: the old perimeter model (VPN to a trusted intranet to any service), where getting in once means trusted forever and lateral movement is trivial. Bottom: the zero-trust model: user/device to an IAP/auth proxy (who, where, device, MFA, posture) to a policy engine making a per-request decision (OPA/Cedar) to a service that accepts only requests carrying a valid claim, repeated for every downstream call, with mTLS and SPIFFE identity inside the cluster." - caption="Figure 12.4 — Perimeter trust versus zero-trust: identity and policy are checked on every hop, not once at the edge" %} + caption="Figure 12.4: Perimeter trust versus zero-trust: identity and policy are checked on every hop, not once at the edge" %} In our stack this is layered: an identity-aware proxy at the edge checks who and what (user, device, MFA, posture), and inside the cluster every workload has a SPIFFE @@ -374,15 +374,15 @@ spec: ## Policy-as-code The other pattern worth showing in code is **policy-as-code**: security rules -declared in Git, enforced the same way at two points — `conftest`/Kyverno in CI +declared in Git, enforced the same way at two points: `conftest`/Kyverno in CI *pre-merge*, and OPA Gatekeeper or Kyverno at *admission* on the cluster. The same policy catches a violation before merge and blocks it again if it somehow reaches `kubectl apply`. {% include excalidraw.html file="12-policy-as-code" - alt="A developer commits to Git/CI (conftest, kyverno, pre-merge gate), then kubectl apply (or ArgoCD/Flux) sends objects to an admission controller (OPA Gatekeeper/Kyverno) that evaluates ConstraintTemplates against incoming objects and the API server accepts or rejects based on policy. Policy in Rego or Kyverno YAML — images must be signed, no privileged containers — is loaded into the admission controller." - caption="Figure 12.5 — Policy lives in Git and is enforced at admission; no human runs a checklist" %} + alt="A developer commits to Git/CI (conftest, kyverno, pre-merge gate), then kubectl apply (or ArgoCD/Flux) sends objects to an admission controller (OPA Gatekeeper/Kyverno) that evaluates ConstraintTemplates against incoming objects and the API server accepts or rejects based on policy. Policy in Rego or Kyverno YAML (images must be signed, no privileged containers) is loaded into the admission controller." + caption="Figure 12.5: Policy lives in Git and is enforced at admission; no human runs a checklist" %} ```yaml # OPA Gatekeeper ConstraintTemplate — reusable policy logic in Rego @@ -406,7 +406,7 @@ spec: ## Bulkhead: bound the blast radius -Borrowed from ship design — watertight compartments stop one breach from sinking the +Borrowed from ship design: watertight compartments stop one breach from sinking the whole hull. In a service, the **bulkhead** isolates resources so one tenant's overload can't starve the others. Share one connection pool and one thread group across every tenant and a single client's storm drains the pool for everyone; give each tenant its @@ -415,9 +415,9 @@ own bounded pool and the storm is contained to that tenant. {% include excalidraw.html file="12-bulkhead" alt="Without bulkheads, a shared pool and shared thread group serve customer A traffic, customer B traffic, background jobs, and scheduled work together, so one client's storm starves everyone. With bulkheads, customer A, customer B, and background each get their own bounded connection pool and thread group, so A's storm fails only A while B and background keep working and the blast radius is bounded." - caption="Figure 12.6 — Per-tenant pools bound the blast radius: one tenant's storm fails only that tenant" %} + caption="Figure 12.6: Per-tenant pools bound the blast radius: one tenant's storm fails only that tenant" %} -The implementation is a bounded resource per partition — a per-tenant connection pool, +The implementation is a bounded resource per partition: a per-tenant connection pool, a per-call thread group, or a semaphore that caps concurrency: {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -569,17 +569,17 @@ connection-pool limits do the same job for every service at once. When a message is large *or* sensitive, putting it on the topic is wrong twice: it bloats the broker, and the payload then lives in topic retention where it can leak. The -**claim-check** pattern publishes only a *reference* — a URI plus a signed token — and +**claim-check** pattern publishes only a *reference* (a URI plus a signed token) and stores the body in an encrypted object store the consumer fetches directly. {% include excalidraw.html file="12-claim-check" alt="A producer PUTs a large or sensitive payload to an encrypted object store/vault (ACLs per object, retention policy) and publishes only a small, routable claim (a URI plus signed token) to the Kafka message bus, which holds only the token with no PII in topic history. A consumer consumes the claim and then fetches the payload directly from the vault using the token." - caption="Figure 12.7 — The producer stores the payload and publishes only a claim; the consumer fetches the body out of band" %} + caption="Figure 12.7: The producer stores the payload and publishes only a claim; the consumer fetches the body out of band" %} The producer PUTs the payload to the vault and publishes the claim; the consumer reads the claim and fetches the payload with the token. The broker holds **only the claim**, -so PII never crosses Kafka's wire or sits in its retention — and because the token can +so PII never crosses Kafka's wire or sits in its retention, and because the token can expire and the store can revoke it, access stays time-bound and auditable. This is the security-angled cousin of the large-payload handling in the failure-modes appendix: @@ -729,29 +729,29 @@ func onClaim(ctx context.Context, r *kgo.Record) error { Every pattern is a trade, so the cost line matters as much as the benefit: - **Sidecar** when you want one uniform security policy across every workload - regardless of language — cost: an extra container per pod (a few MB, a little + regardless of language. Cost: an extra container per pod (a few MB, a little startup time). For most systems the uniformity is worth it. - **Valet key** when clients need direct access to a large resource (file upload, - image download) — cost: the storage system must validate the token. -- **Bulkhead** when one noisy tenant must not sink the others — cost: capacity is + image download). Cost: the storage system must validate the token. +- **Bulkhead** when one noisy tenant must not sink the others. Cost: capacity is partitioned, so you can't pool it. -- **Zero-trust** when a breach must stay contained — cost: identity and policy on every +- **Zero-trust** when a breach must stay contained. Cost: identity and policy on every hop, which the mesh largely absorbs for you. -- **Policy-as-code** when security rules must be enforced uniformly and reviewably — - cost: writing and maintaining the policies, paid back the first time CI catches a bad +- **Policy-as-code** when security rules must be enforced uniformly and reviewably. + Cost: writing and maintaining the policies, paid back the first time CI catches a bad change. -- **Claim-check** when payloads are large or sensitive — cost: a second fetch and a +- **Claim-check** when payloads are large or sensitive. Cost: a second fetch and a store to manage, in exchange for keeping bulk and PII off the broker. ### Cross-check it yourself Test the deny path, which is the one that matters. Call `order-service` *directly*, -bypassing the mesh, with no `X-Forwarded-Client-Cert`, and confirm you get `403` — +bypassing the mesh, with no `X-Forwarded-Client-Cert`, and confirm you get `403`: the app trusts only the sidecar. Then apply a Deployment with an unsigned image and confirm Gatekeeper rejects it at admission, and that the same policy fails the CI step pre-merge. A request without identity being refused, and an unsigned image being blocked twice, is the layered model working. --- -*Verification status: verified — [`examples/12-security/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/12-security/) passes 18/18 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified: [`examples/12-security/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/12-security/) passes 18/18 checks in each of the six languages on Docker Engine (2026-10-13) (sidecar identity trust (403 without X-Forwarded-Client-Cert, health-check bypass), propagation of the JWT subject claim, valet-key mint, verify and tamper rejection, per-tenant bulkhead state, a full tenant pool refusing the overflow with 429 while another tenant is still served, and conftest Rego policy evaluation of a good and a bad manifest. JWT signature validation, RBAC enforcement and rate limiting are not covered).* diff --git a/_docs/13-anti-patterns.md b/_docs/13-anti-patterns.md index 3f3a6465..0fd354a7 100644 --- a/_docs/13-anti-patterns.md +++ b/_docs/13-anti-patterns.md @@ -2,7 +2,7 @@ title: "Anti-Patterns" order: 13 part: "Security & anti-patterns" -description: "Nearly every cloud-native failure is coupling sneaking back in — the recurring anti-patterns that reintroduce it, the five heuristics that catch them in review, and the whole book compressed to five lines." +description: "Nearly every cloud-native failure is coupling sneaking back in: the recurring anti-patterns that reintroduce it, the five heuristics that catch them in review, and the whole book compressed to five lines." duration: 16 minutes --- @@ -14,47 +14,47 @@ ways that happens and gives you heuristics to catch them. ## How cloud-native API systems go wrong -Each of these is a callback to an earlier chapter — and each one reintroduces +Each of these is a callback to an earlier chapter, and each one reintroduces exactly the coupling the architecture worked to remove: {% include excalidraw.html file="13-antipatterns-grid" alt="A grid of nine anti-patterns, each re-coupling what the architecture worked to separate: distributed monolith (services deploy together), shared database (two services one schema), chatty N+1 APIs (a screen makes 40 calls), sync call chains (A to B to C to D, availability multiplies down), no contract or registry (breaking changes go unflagged), smart pipes and dumb endpoints (business logic creeps into the gateway), missing observability (debugging by guesswork), retry storms with no backoff (retries amplify an incident into an outage), and leaky internal models (DB rows exposed as the public API)." - caption="Figure 13.1 — The recurring anti-patterns; each one re-couples what the platform worked to separate" %} + caption="Figure 13.1: The recurring anti-patterns; each one re-couples what the platform worked to separate" %} -- **Distributed monolith** — services that must be deployed together. You paid all +- **Distributed monolith**: services that must be deployed together. You paid all the cost of distribution and kept all the coupling. -- **Shared database** — two services reading and writing one table. The table is +- **Shared database**: two services reading and writing one table. The table is now an un-versioned, un-owned contract between them. -- **Chatty call chains** — `order` calls `inventory` calls `pricing` calls…, deep +- **Chatty call chains**: `order` calls `inventory` calls `pricing` calls…, deep synchronous hops that couple availability and latency end to end. -- **No contract / leaky internal models** — exposing database rows as the public +- **No contract / leaky internal models**: exposing database rows as the public API, so every internal change is a breaking change. -- **Smart pipes, dumb endpoints** — business logic creeping into the gateway or an +- **Smart pipes, dumb endpoints**: business logic creeping into the gateway or an ESB, so the routing layer becomes the new monolith and no single team can change a flow safely. -- **Missing observability** — debugging distributed systems by guesswork. -- **Retry storms** — retries with no backoff or bound, turning a blip into an +- **Missing observability**: debugging distributed systems by guesswork. +- **Retry storms**: retries with no backoff or bound, turning a blip into an outage. If you keep one diagram for review, keep this pairing. {% include excalidraw.html file="13-antipattern-heuristics" - alt="Five anti-patterns on the left — distributed monolith, shared database, chatty call chains, no contract or leaky models, missing observability and retry storms — each paired with the review heuristic on the right that catches it" - caption="Figure 13.2 — Every anti-pattern is coupling returning; every heuristic is a test you can apply in a PR" %} + alt="Five anti-patterns on the left (distributed monolith, shared database, chatty call chains, no contract or leaky models, missing observability and retry storms), each paired with the review heuristic on the right that catches it" + caption="Figure 13.2: Every anti-pattern is coupling returning; every heuristic is a test you can apply in a PR" %} ## The heuristics that keep you out of them -Five tests to apply in design and code review — paste them into a PR template: +Five tests to apply in design and code review (paste them into a PR template): -1. **If two services must deploy together, they are one service** — merge them or +1. **If two services must deploy together, they are one service**: merge them or fix the contract. 2. **One writer per dataset.** Everyone else reads through the API or subscribes to its events. No shared tables. 3. **Push aggregation to a gateway or BFF**, and forbid synchronous call chains deeper than one hop. -4. **No contract in the registry, no merge** — and every retry has a timeout and a +4. **No contract in the registry, no merge**, and every retry has a timeout and a bound. 5. **If you can't trace it, you can't run it.** Observability is a launch requirement, not a follow-up. @@ -65,12 +65,12 @@ If you remember nothing else: 1. Cloud-native is a set of **properties**; the platform exists to make them the default. -2. **Match each interaction to its style** — REST, gRPC, GraphQL, or async — and +2. **Match each interaction to its style** (REST, gRPC, GraphQL, or async), and compose at the edge. 3. **Own your data**; share it as contracts and facts, never as shared tables. 4. Make the contract a **governed artifact**: a registry to enforce it, metadata to find it, observability to watch it. -5. Watch for **coupling sneaking back in** — every anti-pattern in this chapter is a face of +5. Watch for **coupling sneaking back in**: every anti-pattern in this chapter is a face of that one failure. ## Further reading @@ -81,10 +81,10 @@ coupling and stability patterns, are collected in the relevant IETF RFCs and drafts. That closes the thirteen core sections. The appendices that follow go deeper on the -questions that always come up — protocols, versioning, WebSockets, sagas, errors, -DDD, coupling, shutdown, routing, decomposition, caching, and failure modes — and +questions that always come up (protocols, versioning, WebSockets, sagas, errors, +DDD, coupling, shutdown, routing, decomposition, caching, and failure modes) and can be read on demand. --- -*Verification status: conceptual chapter — no runnable code. Its heuristics are the +*Verification status: conceptual chapter; no runnable code. Its heuristics are the review-time application of the patterns proven in the earlier chapters.* diff --git a/_docs/14-appendix-a-protocols.md b/_docs/14-appendix-a-protocols.md index 78abf84e..cd35847c 100644 --- a/_docs/14-appendix-a-protocols.md +++ b/_docs/14-appendix-a-protocols.md @@ -4,7 +4,7 @@ marker: "A" label: "Appendix A" order: 14 part: "Deep-dive appendices" -description: "The deep dive behind 'pick the protocol' — REST, gRPC, and GraphQL compared on three axes: the mental model each imposes, how each handles fetching and round trips, and what travels on the wire." +description: "The deep dive behind 'pick the protocol': REST, gRPC, and GraphQL compared on three axes: the mental model each imposes, how each handles fetching and round trips, and what travels on the wire." duration: 26 minutes --- @@ -17,28 +17,28 @@ handles **fetching and round trips**, and what **travels on the wire**. Start with the mental model, because it drives everything else: -- **REST is resource-oriented** — nouns and HTTP verbs; you transfer +- **REST is resource-oriented**: nouns and HTTP verbs; you transfer representations of state. -- **gRPC is procedure-oriented** — you call typed methods; it's behaviour, not +- **gRPC is procedure-oriented**: you call typed methods; it's behaviour, not state, and the `.proto` is the contract. -- **GraphQL is graph-oriented** — one typed schema the client traverses, picking +- **GraphQL is graph-oriented**: one typed schema the client traverses, picking exactly the fields it wants. {% include excalidraw.html file="14-three-paradigms" alt="Three columns for the same place_order intent. REST as resources: nouns plus HTTP verbs, POST to orders, GET an order by id, state transfer, server defines payload (resource-oriented). gRPC as procedures: typed method calls like Order.Place and Inventory.Reserve, behaviour not state, contract is the proto (RPC-oriented). GraphQL as a graph: one typed schema, the client picks fields and traverses relationships, contract is the SDL (query-oriented)." - caption="Figure A.1 — One intent, three mental models: resources, procedures, and a typed graph" %} + caption="Figure A.1: One intent, three mental models: resources, procedures, and a typed graph" %} ## The three axes, side by side | Axis | REST | gRPC | GraphQL | |---|---|---|---| | **Mental model** | Resources: nouns + HTTP verbs, transfer state | Procedures: call typed methods, `.proto` is the contract | Typed graph: client traverses one schema, picks fields | -| **Fetching** | Several round trips, or a bespoke aggregate; fixed payloads over/under-fetch | One typed binary call | One tailored query; client names fields — no over/under-fetch | -| **Wire** | HTTP/1.1 or /2, JSON text, request/response, HTTP-cacheable, browser-native — *max reach* | HTTP/2 only, protobuf binary, uni- and bidirectional streaming, no HTTP cache, needs grpc-web in a browser | HTTP POST to one URL, JSON, request/response + subscriptions, caching is app-level | +| **Fetching** | Several round trips, or a bespoke aggregate; fixed payloads over/under-fetch | One typed binary call | One tailored query; client names fields, no over/under-fetch | +| **Wire** | HTTP/1.1 or /2, JSON text, request/response, HTTP-cacheable, browser-native (*max reach*) | HTTP/2 only, protobuf binary, uni- and bidirectional streaming, no HTTP cache, needs grpc-web in a browser | HTTP POST to one URL, JSON, request/response + subscriptions, caching is app-level | The fetching axis is the one clients *feel*. With REST you often need several round -trips — order, then its items, then stock, then shipping — or you build a bespoke +trips, order, then its items, then stock, then shipping, or you build a bespoke aggregate endpoint and still get a fixed shape that over- or under-fetches. GraphQL collapses that into one query whose fields the client chooses; gRPC is a single typed binary call. @@ -46,7 +46,7 @@ typed binary call. {% include excalidraw.html file="14-protocol-roundtrips" alt="REST requires the client to make several round trips to order, items, and stock or shipping, with fixed payloads; GraphQL collapses the same data need into a single query to the gateway where the client names the fields" - caption="Figure A.2 — Many REST round trips versus one client-shaped GraphQL query" %} + caption="Figure A.2: Many REST round trips versus one client-shaped GraphQL query" %} The third axis is what literally travels between processes, and it is the one that decides reach and caching. REST rides ordinary HTTP, so browsers and shared caches @@ -56,22 +56,22 @@ encoding; GraphQL keeps HTTP reach but moves caching up into the application. {% include excalidraw.html file="14-the-wire" alt="Three columns comparing what travels on the wire. REST: transport HTTP/1.1 or 2, JSON text encoding, request and response interaction, cacheable via HTTP, browser-native reach. gRPC: HTTP/2 only, protobuf binary, unidirectional and bidirectional streaming, no HTTP caching, needs grpc-web in a browser. GraphQL: HTTP POST to one URL, JSON text, request and response plus subscriptions, app-level caching, browser-native reach." - caption="Figure A.3 — What travels on the wire, per protocol: transport, encoding, streaming, caching, and reach" %} + caption="Figure A.3: What travels on the wire, per protocol: transport, encoding, streaming, caching, and reach" %} ## How an RPC travels: stub to stub REST and GraphQL are things you can hand-write with a terminal and `curl`; gRPC is not, and that is the point. The `.proto` contract is compiled into a **client stub** and a **server skeleton**, and your code only ever calls a typed method. Everything -between the two stubs — serialising the request to protobuf bytes, framing them onto -an HTTP/2 stream, and reversing all of that on the far side — is generated. +between the two stubs (serialising the request to protobuf bytes, framing them onto +an HTTP/2 stream, and reversing all of that on the far side) is generated. {% include excalidraw.html file="14-rpc-stub-to-stub" alt="A client process and a server process. In the client, application code calls inventory.ReserveStock(req); a generated client stub marshals the request to protobuf. An amber arrow labelled HTTP/2 frame, protobuf crosses to the server, where a generated server skeleton unmarshals the bytes back into a request and invokes the handler reserve(sku, qty). A dashed return arrow carries the ReserveReply protobuf bytes back. One typed contract generates both stubs." - caption="Figure A.4 — How an RPC travels: a typed call in, protobuf on the wire, a typed call out" %} + caption="Figure A.4: How an RPC travels: a typed call in, protobuf on the wire, a typed call out" %} -From the caller's side it is a single typed method — no URL, no JSON, no status-code +From the caller's side it is a single typed method: no URL, no JSON, no status-code mapping. This stub call is the same `ReserveStock` invocation in all six stacks; each language gets a generated client from the one `.proto`. @@ -153,28 +153,28 @@ func reserve(ctx context.Context, client pb.InventoryClient, ## Streaming and bidirectional communication -That single typed call is *unary* — one request, one response. gRPC's real +That single typed call is *unary*: one request, one response. gRPC's real distinction from request/response REST is that the same `.proto` can declare three more shapes, just by marking a side `stream`: -- **Server-streaming** — one request, a stream of responses. Live order feeds, +- **Server-streaming**: one request, a stream of responses. Live order feeds, progress updates, tailing a changelog. -- **Client-streaming** — a stream of requests, one response. Bulk ingest, file +- **Client-streaming**: a stream of requests, one response. Bulk ingest, file upload, telemetry batches folded into a single ack. -- **Bidirectional** — both sides stream independently over one connection. Chat, +- **Bidirectional**: both sides stream independently over one connection. Chat, collaborative editing, long-lived sync. {% include excalidraw.html file="14-streaming-modes" alt="Four panels comparing RPC modes between a client and a server. Unary: one request, one response. Server-streaming: one request, then N responses (amber). Client-streaming: N requests (amber), then one response. Bidirectional: both directions stream N to N over one connection (amber, double-headed)." - caption="Figure A.5 — The four RPC modes: unary, server-streaming, client-streaming, and bidirectional" %} + caption="Figure A.5: The four RPC modes: unary, server-streaming, client-streaming, and bidirectional" %} The impact is architectural, not cosmetic. A stream is one long-lived HTTP/2 stream, -so ordering is preserved and there is no per-message connection cost — but the +so ordering is preserved and there is no per-message connection cost, but the connection is now stateful and pinned to one server instance, which the load balancer must respect (the WebSockets appendix makes the same point). Backpressure matters: a slow consumer must be allowed to slow the producer rather than buffer -without bound. Here is the server side of a server-streaming RPC in each stack — +without bound. Here is the server side of a server-streaming RPC in each stack; note how every one expresses "push many, then finish." {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -263,8 +263,8 @@ markedly cheaper than JSON over HTTP/1.1. | **Persistent channels** | one long-lived (TLS) connection, reused | no per-call handshake cost | | **Generated stubs** | (de)serialisation is compiled, not reflective | less CPU per call than runtime JSON mapping | -Two caveats. The win is largest where you'd expect it — chatty, -internal, east-west paths — and shrinks for occasional, large-payload calls where +Two caveats. The win is largest where you'd expect it: chatty, +internal, east-west paths, and it shrinks for occasional, large-payload calls where the body dominates. And none of it reaches a browser directly: a browser needs `grpc-web` and a proxy, which is why the public edge usually stays REST or GraphQL. @@ -278,8 +278,8 @@ the body dominates. And none of it reaches a browser directly: a browser needs - **A fact that many consumers react to → don't make it a call at all; emit an event.** -The closing point ties back to the whole book: forcing one protocol everywhere — -"we're a gRPC shop," "everything is GraphQL" — is *itself* an anti-pattern. Each +The closing point ties back to the whole book: forcing one protocol everywhere +("we're a gRPC shop," "everything is GraphQL") is *itself* an anti-pattern. Each protocol is strongest on a specific axis; a healthy system uses each where it fits. @@ -287,12 +287,12 @@ fits. Drawn from real review comments: -- **REST** — verbs in URLs (`/createOrder`), returning `200` with an error body, +- **REST**: verbs in URLs (`/createOrder`), returning `200` with an error body, offset pagination that breaks under concurrent writes, and throwing away free HTTP caching. -- **gRPC** — calling it straight from a browser without grpc-web, or treating it as +- **gRPC**: calling it straight from a browser without grpc-web, or treating it as "REST with protobuf" and never touching its streaming superpower. -- **GraphQL** — the N+1 resolver explosion (fix with DataLoader batching, as in +- **GraphQL**: the N+1 resolver explosion (fix with DataLoader batching, as in **Composition**), no query depth or complexity limits, and using it for simple single-resource fetches where REST would be plainer. diff --git a/_docs/15-appendix-b-versioning.md b/_docs/15-appendix-b-versioning.md index 6cd57293..5a96b835 100644 --- a/_docs/15-appendix-b-versioning.md +++ b/_docs/15-appendix-b-versioning.md @@ -4,32 +4,32 @@ marker: "B" label: "Appendix B" order: 15 part: "Deep-dive appendices" -description: "How to change an API without breaking everyone — the four ways to version REST, what counts as a breaking change in protobuf, GraphQL's evolve-don't-version stance, and the principles common to all three." +description: "How to change an API without breaking everyone: the four ways to version REST, what counts as a breaking change in protobuf, GraphQL's evolve-don't-version stance, and the principles common to all three." duration: 28 minutes --- The question every team fights about: how do we change this without breaking -everyone? The answer differs sharply by protocol — but the principles underneath +everyone? The answer differs sharply by protocol, but the principles underneath rhyme, so we end with those. ## Four ways to version a REST API Ordered coarse to fine: -- **URI path** (`/v2/orders`) — visible and trivial to route, but couples the +- **URI path** (`/v2/orders`): visible and trivial to route, but couples the version to the URL and breaks REST's idea that a URL identifies a *resource*, not a representation. Common precisely because it's obvious. -- **Query parameter** (`?version=2`) — easy to default, but muddies caches (the +- **Query parameter** (`?version=2`): easy to default, but muddies caches (the same URL now returns different shapes). -- **Custom header** (`X-API-Version: 2`) — keeps URLs clean, but is invisible and +- **Custom header** (`X-API-Version: 2`): keeps URLs clean, but is invisible and easy to forget. -- **Media type** (`Accept: application/vnd.acme.order.v2+json`) — keeps the URL +- **Media type** (`Accept: application/vnd.acme.order.v2+json`): keeps the URL stable and versions the *representation*, which is the most RESTful choice. {% include excalidraw.html file="15-rest-versioning" alt="Four ways to version a REST API, coarse to fine. URI path: a v2 orders path, visible and trivial to route, couples the version to the URL, most widely used. Query param: an orders path with a version query, easy to default, muddies caching keys, least recommended. Custom header: an X-API-Version header, keeps the URL clean but is invisible and easy to miss, needs discipline. Media type: an Accept header naming the v2 representation, content negotiation, the URL still identifies the resource, the RESTful way." - caption="Figure B.1 — Four ways to version REST, from coarse URI paths to RESTful media-type negotiation" %} + caption="Figure B.1: Four ways to version REST, from coarse URI paths to RESTful media-type negotiation" %} ```http # Media-type versioning: the URL stays the same; the version lives in Accept @@ -40,20 +40,20 @@ HTTP/1.1 200 OK Content-Type: application/vnd.acme.order.v2+json ``` -## Which REST scheme to start with — and what it costs HATEOAS and security +## Which REST scheme to start with, and what it costs HATEOAS and security For a new public or partner API, **start with URI-path versioning** (`/v1/orders`). It is the least elegant of the four but the most operable: routable, visible in logs and dashboards, cacheable without surprises, and impossible for a client to get wrong. Reach for **media-type versioning** when you specifically need -the URL to stay stable — which, as it happens, is exactly what HATEOAS needs. +the URL to stay stable, which, as it happens, is exactly what HATEOAS needs. **Impact on HATEOAS.** Hypermedia only works if a client can follow the links the server hands it without guessing a version, so the scheme decides how the version travels with those links: - **Media-type** keeps every link clean (`/orders/A-1001/items`) and lets the - version ride in `Accept` on each follow — the representation is versioned, the + version ride in `Accept` on each follow: the representation is versioned, the resource identity is not. This is the HATEOAS-friendly choice. - **URI-path** works too, but every emitted link must already carry the version (`/v2/orders/A-1001/items`); a client that follows links stays pinned to one @@ -61,18 +61,18 @@ travels with those links: - **Query-param** and **custom-header** versioning break link-following without any error: the version lives outside the URL, so a naively followed link drops it. -**Impact on security.** Each live version is attack surface you must keep patched — +**Impact on security.** Each live version is attack surface you must keep patched; the real cost of versioning is *sprawl*, not the scheme. Two scheme-specific notes: URI and query versions sit in URLs, so they surface in proxy logs, browser history, and `Referer` headers (more fingerprintable, but also routable by a WAF), while header and media-type versions hide there instead. And never let an older version skip a -check the newer one added — an auth scope, an input bound, a rate limit — or `/v1` +check the newer one added (an auth scope, an input bound, a rate limit), or `/v1` becomes the soft way in. Deprecate and retire on the lifecycle every protocol shares (Figure B.4): a version you forgot to turn off is a version you forgot to patch. ## gRPC: field numbers are the contract -protobuf's rules are precise because the wire format is **positional** — the field +protobuf's rules are precise because the wire format is **positional**: the field *number* is the contract, not the name or the order. Safe, wire-compatible changes: add a field with a *new* number, add an rpc method, add an enum value, rename a field (the number is what matters), mark fields deprecated, and **reserve** removed @@ -82,7 +82,7 @@ reusing a retired tag. {% include excalidraw.html file="15-grpc-breaking" alt="Two columns for protobuf changes. Safe and wire-compatible: add a field with a new number, add a new rpc method, add a value to an enum, rename a field since the number is unchanged, deprecate with the deprecated option, and widen with reserved for old tags. Breaking and rejected by the gate: change a field's number or type, reuse a removed field number, remove a field without reserving its tag, change an rpc between unary and streaming, rename or move a service or package, or change the request or response message type." - caption="Figure B.2 — protobuf: safe wire-compatible changes versus breaking ones the registry rejects" %} + caption="Figure B.2: protobuf: safe wire-compatible changes versus breaking ones the registry rejects" %} ```proto syntax = "proto3"; @@ -110,14 +110,14 @@ tag so no future change can accidentally reuse it and silently corrupt old reade ## GraphQL evolves, it doesn't version GraphQL takes a different stance entirely: there is no `/v2`. One schema evolves -continuously. Add new fields and types (additive is always safe — existing queries +continuously. Add new fields and types (additive is always safe: existing queries don't ask for the new field, so they're unaffected), mark superseded fields `@deprecated` with a reason, watch usage drop, then retire. {% include excalidraw.html file="15-graphql-evolution" - alt="A four-step GraphQL evolution flow with no version number. Add: a new field or type, additive and safe, clients opt in. Deprecated: mark the old field with a reason and replacement, still served. Watch usage: field-level metrics show who still queries it and drive migration. Retire: at zero usage, remove the field — the only breaking step." - caption="Figure B.3 — GraphQL evolves rather than versions: add, deprecate, watch usage, then retire" %} + alt="A four-step GraphQL evolution flow with no version number. Add: a new field or type, additive and safe, clients opt in. Deprecated: mark the old field with a reason and replacement, still served. Watch usage: field-level metrics show who still queries it and drive migration. Retire: at zero usage, remove the field, the only breaking step." + caption="Figure B.3: GraphQL evolves rather than versions: add, deprecate, watch usage, then retire" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -216,12 +216,12 @@ which is what lets you measure who's still on the old field before removing it. ## Versioning a Kafka API -An event stream is an API too — its **schema is the contract**, exactly like a -`.proto` or an OpenAPI document — and it evolves under the same pressure. The first +An event stream is an API too: its **schema is the contract**, exactly like a +`.proto` or an OpenAPI document, and it evolves under the same pressure. The first thing to separate is the two roles a Kafka message plays, which are easy to conflate: - The **key** is *identity*: it picks the partition and so fixes ordering for a given - entity. It is not a version slot — never encode `v2` into the key, or you scatter + entity. It is not a version slot: never encode `v2` into the key, or you scatter one entity's events across partitions and lose ordering. - The **version** is *schema*: it lives in the schema registry (Apicurio, from the API-management chapter) as a numbered version with a **compatibility rule**, and @@ -231,18 +231,18 @@ thing to separate is the two roles a Kafka message plays, which are easy to conf The default that keeps consumers working is **one topic with a BACKWARD-compatible, evolving schema**: add fields, never renumber or repurpose them, and old consumers ignore what they don't know while new consumers read the additions. Only a -breaking change earns a new major — and then you run two topics (or two schema +breaking change earns a new major, and then you run two topics (or two schema majors) side by side, dual-write through the migration, and retire the old once consumers have moved. {% include excalidraw.html file="15-kafka-versioning" - alt="A producer writes v2 events to one topic, order.placed, which holds an evolving schema. A schema registry over the topic enforces compatibility — v1 to v2 is add-only, BACKWARD. The topic delivers to two consumers: one built against v1 that ignores the added fields, and one built against v2 that uses them. The key is identity and partitioning, never the version; the version is the schema, carried in the registry or a header." - caption="Figure B.5 — Versioning a Kafka API: one evolving topic, a registry-enforced compatibility rule, old and new consumers side by side" %} + alt="A producer writes v2 events to one topic, order.placed, which holds an evolving schema. A schema registry over the topic enforces compatibility: v1 to v2 is add-only, BACKWARD. The topic delivers to two consumers: one built against v1 that ignores the added fields, and one built against v2 that uses them. The key is identity and partitioning, never the version; the version is the schema, carried in the registry or a header." + caption="Figure B.5: Versioning a Kafka API: one evolving topic, a registry-enforced compatibility rule, old and new consumers side by side" %} On the wire, the cleanest portable signal is a `schema.version` header the producer sets alongside the key; a consumer reads it and upcasts an old payload into the -current shape before handling it. The producer side looks like this — the key stays +current shape before handling it. The producer side looks like this: the key stays identity, the version is metadata: {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -306,35 +306,35 @@ cl.Produce(ctx, rec, nil) A consumer completes the picture: read the `schema.version` header, and for an older version run a small upcaster (`v1 → v2`: default the new fields) before dispatching, so the rest of the handler only ever sees the current shape. That is the same -tolerant-reader discipline the cross-cutting principles call for — strict in what you emit, +tolerant-reader discipline the cross-cutting principles call for: strict in what you emit, liberal in what you accept. ## Principles that cut across them all The protocol-specific rules are all faces of four ideas: -- **Additive change is safe; removal and mutation break people** — so design to +- **Additive change is safe; removal and mutation break people**, so design to *add*. -- **Make compatibility mechanical** — the registry's `BACKWARD`/`FORWARD`/`FULL` +- **Make compatibility mechanical**: the registry's `BACKWARD`/`FORWARD`/`FULL` rules checked in CI, never a reviewer remembering. -- **Never break without notice** — deprecate with a reason, a replacement, and a sunset +- **Never break without notice**: deprecate with a reason, a replacement, and a sunset date, and measure who's still on the old version. - **Run old and new side by side** until consumers have migrated, then retire. {% include excalidraw.html file="15-deprecation-lifecycle" alt="A four-step lifecycle: ADD (additive, always safe), DEPRECATE (with reason, replacement, and sunset), MEASURE (who is still using it), then RETIRE after migration" - caption="Figure B.4 — The same lifecycle for every protocol: add, deprecate, measure, retire" %} + caption="Figure B.4: The same lifecycle for every protocol: add, deprecate, measure, retire" %} ### Cross-check it yourself Prove the additive/breaking line with the registry from **API Registry**. Add an optional field to a schema and confirm the `BACKWARD` check passes; change a field's type or reuse a protobuf tag and confirm it fails with `400`. The mechanical check -catching the breaking change — instead of a reviewer's memory — is the principle +catching the breaking change, instead of a reviewer's memory, is the principle made real. --- -*Verification status: unverified — examples transcribed from the source decks; the +*Verification status: unverified: examples transcribed from the source decks; the REST, gRPC, and GraphQL compatibility behaviour and the Kafka `schema.version` header are exercised against the registry in the `examples/09-api-registry/` runner.* diff --git a/_docs/16-appendix-c-websockets.md b/_docs/16-appendix-c-websockets.md index 1a5d7bd0..59fa0263 100644 --- a/_docs/16-appendix-c-websockets.md +++ b/_docs/16-appendix-c-websockets.md @@ -4,16 +4,16 @@ marker: "C" label: "Appendix C" order: 16 part: "Deep-dive appendices" -description: "The protocol we left out of the main interaction styles, because it behaves so differently on Kubernetes — why long-lived sockets fight stateless scaling, and the pub/sub backplane plus resume-don't-restart posture that fix it." +description: "The protocol we left out of the main interaction styles, because it behaves so differently on Kubernetes: why long-lived sockets fight stateless scaling, and the pub/sub backplane plus resume-don't-restart posture that fix it." duration: 28 minutes --- WebSockets get their own appendix instead of a place in the main interaction-styles discussion because they behave so differently on Kubernetes. A WebSocket is long-lived, -stateful, and **pinned to a pod** — the opposite of the disposable, stateless +stateful, and **pinned to a pod**: the opposite of the disposable, stateless request the platform is built around. -## A WebSocket endpoint — and the problem hiding in it +## A WebSocket endpoint and the problem hiding in it Each of these is a perfectly ordinary WebSocket endpoint. Look at the connection registry in every one: it's **per-pod, not shared.** That single fact is the entire @@ -142,11 +142,11 @@ func wsHandler(w http.ResponseWriter, r *http.Request) { Once a connection is established through the load balancer, it is pinned to one pod for its whole lifetime. Two problems follow: -- **Scale-up** — the HPA or KEDA adds `ws-pod-3`, but it gets *zero* existing +- **Scale-up**: the HPA or KEDA adds `ws-pod-3`, but it gets *zero* existing connections. New pods only receive *new* connections, so load stays lopsided and the new capacity barely helps the pods already saturated. -- **Scale-down** — killing a pod drops *every* connection it held at once. The - clients must all reconnect simultaneously — a thundering herd you caused +- **Scale-down**: killing a pod drops *every* connection it held at once. The + clients must all reconnect simultaneously: a thundering herd you caused yourself. And because the registry is per-pod, a message for user X is undeliverable from any @@ -154,12 +154,12 @@ pod that doesn't happen to hold X's socket. {% include excalidraw.html file="16-ws-scaling-problem" - alt="Clients hold long-lived ws connections through a Service or load balancer that is sticky per connection. ws-pod-1 holds 120 connections and ws-pod-2 holds 118, but a newly added ws-pod-3 holds 0 — the scale-up problem, since a new pod gets no existing connections, only new ones. Killing a pod is the scale-down or rollout problem: it drops every connection it was holding at once." - caption="Figure C.1 — A WebSocket pins to a pod, so scale-up leaves new pods idle and scale-down drops every connection at once" %} + alt="Clients hold long-lived ws connections through a Service or load balancer that is sticky per connection. ws-pod-1 holds 120 connections and ws-pod-2 holds 118, but a newly added ws-pod-3 holds 0: the scale-up problem, since a new pod gets no existing connections, only new ones. Killing a pod is the scale-down or rollout problem: it drops every connection it was holding at once." + caption="Figure C.1: A WebSocket pins to a pod, so scale-up leaves new pods idle and scale-down drops every connection at once" %} ## Scaling out: a pub/sub backplane -The fix keeps the part that's fine — each pod still owns only its own sockets — and +The fix keeps the part that's fine (each pod still owns only its own sockets) and adds a shared **backplane** that every pod subscribes and publishes to: **Redis pub/sub** for low latency, or a **Kafka topic** when you want durability and replay. To send to user X, any pod publishes to the backplane; the pod that holds @@ -169,7 +169,7 @@ know where any client lives. {% include excalidraw.html file="16-ws-backplane" alt="Three WebSocket pods each own only their own sockets; all subscribe and publish to a shared Redis or Kafka backplane. A message for client X arrives at ws-pod-1, which publishes to the backplane; the backplane delivers it to ws-pod-2, which holds X's socket" - caption="Figure C.2 — Externalised fan-out: the backplane routes each message to whichever pod holds the client" %} + caption="Figure C.2: Externalised fan-out: the backplane routes each message to whichever pod holds the client" %} For the payload itself, frame messages as **binary protobuf** rather than JSON text: smaller and faster on a hot socket, and versioned by the same field-number @@ -180,23 +180,23 @@ rules as gRPC (see **Appendix B**). The cloud-native posture is to treat every connection as disposable and design for its loss: -- **Heartbeats** (ping/pong) detect a dead link in seconds — raw TCP can hang for +- **Heartbeats** (ping/pong) detect a dead link in seconds; raw TCP can hang for minutes before noticing. - On a drop, the client **reconnects with exponential backoff plus jitter**, so a mass disconnect doesn't become a synchronised reconnect storm. -- The client **resumes from the last acknowledged sequence number** — a monotonic - per-connection counter — so the server replays only what was missed instead of +- The client **resumes from the last acknowledged sequence number** (a monotonic + per-connection counter) so the server replays only what was missed instead of restarting the stream. Resume, don't restart. {% include excalidraw.html file="16-ws-resume" alt="A sequence between a client and a ws-pod. A ping/pong heartbeat detects a dead link; the connection is lost; the client reconnects with backoff and jitter; it resumes by sending its last acknowledged sequence number (1042); the pod replays only the missed events (1043 to 1050), which are idempotent; the stream is live again, at-least-once and deduplicated by sequence." - caption="Figure C.3 — Resume, don't restart: detect the drop, reconnect with backoff, and replay only what was missed from the last acked sequence" %} + caption="Figure C.3: Resume, don't restart: detect the drop, reconnect with backoff, and replay only what was missed from the last acked sequence" %} ## Binary framing with protobuf over WebSockets The backplane section said to frame messages as binary protobuf; here is what that -means on the wire. Every message is one versioned **Envelope** — a thin protobuf +means on the wire. Every message is one versioned **Envelope**: a thin protobuf wrapper carrying a `version`, the per-connection `seq` that resume already relies on, a `type` discriminator, and the actual event as nested `payload` bytes. It is sent as a WebSocket *binary* frame, not a text frame. @@ -213,14 +213,14 @@ message Envelope { {% include excalidraw.html file="16-ws-protobuf-envelope" - alt="A producer encodes to bytes; the bytes travel inside a WebSocket binary frame (opcode 0x2) made of four fields — version (uint32 = 2), seq (uint64 = 1043), type (the string OrderPlaced), and payload (protobuf bytes, highlighted); a consumer decodes and dispatches. The envelope is versioned by field number like gRPC: add fields, never renumber." - caption="Figure C.4 — A versioned protobuf Envelope carried in one binary WebSocket frame" %} + alt="A producer encodes to bytes; the bytes travel inside a WebSocket binary frame (opcode 0x2) made of four fields: version (uint32 = 2), seq (uint64 = 1043), type (the string OrderPlaced), and payload (protobuf bytes, highlighted); a consumer decodes and dispatches. The envelope is versioned by field number like gRPC: add fields, never renumber." + caption="Figure C.4: A versioned protobuf Envelope carried in one binary WebSocket frame" %} The wins are the same three gRPC gets from protobuf: the frame is **smaller** (no field names on the wire), **cheaper to parse** than re-parsing JSON on every hot -message, and **versioned by field number** — add a field, never renumber, exactly as +message, and **versioned by field number**: add a field, never renumber, exactly as in **Appendix B**. Encoding and sending the envelope looks like this in each stack; -the receive side is the mirror image — read the binary frame, parse the `Envelope`, +the receive side is the mirror image: read the binary frame, parse the `Envelope`, and dispatch on `type`. {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -297,7 +297,7 @@ func push(ctx context.Context, c *websocket.Conn, seq uint64, ev *pb.OrderPlaced The send side is half the story. The receive side is the mirror: read the binary frame, parse the `Envelope`, and dispatch on `type`. Every language follows -the same structure — read binary, deserialize, switch. +the same structure: read binary, deserialize, switch. {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -423,8 +423,8 @@ for { ## Failover without interruption Resume is the client-side mechanic; **failover** is what it buys you when a whole pod -goes away — a rollout, a scale-down, or a crash. The socket drops, the client -reconnects through the Service, and the load balancer lands it on *some other* pod — +goes away: a rollout, a scale-down, or a crash. The socket drops, the client +reconnects through the Service, and the load balancer lands it on *some other* pod, never the dead one. That only works because the backplane already decoupled delivery from placement: any pod can serve any client, so the new pod is as good as the old one. The client then resumes from its last acked `seq`, and the new pod replays only @@ -433,10 +433,10 @@ the missed range. The user sees a brief reconnect, not a lost stream. {% include excalidraw.html file="16-ws-failover" alt="A client that resumes at seq 1043 reconnects with backoff through the LB or Service. The dashed path to ws-pod-2, now terminated, is marked gone; the client lands instead on ws-pod-5, which holds the new socket. ws-pod-5 subscribes to the Redis or Kafka backplane and replays events with seq greater than 1042. The takeaway: backplane plus per-connection seq means a pod dies and the client loses nothing." - caption="Figure C.5 — Failover without interruption: reconnect lands on a live pod, which replays the gap from the backplane" %} + caption="Figure C.5: Failover without interruption: reconnect lands on a live pod, which replays the gap from the backplane" %} The one requirement is that the missed range be *reconstructable* from somewhere the -new pod can reach. A Kafka backplane with retention provides this directly — the new pod +new pod can reach. A Kafka backplane with retention provides this directly: the new pod seeks to `seq + 1` and replays. With Redis pub/sub, which has no history, pair it with a short server-side ring buffer (or a Redis stream) keyed by sequence so the gap can be re-sent. Either way the contract is at-least-once, deduplicated by `seq`. @@ -450,21 +450,21 @@ so performance work is mostly about keeping the *per-message* cost low and the | Lever | Why it matters | |---|---| | **Binary protobuf frames** | fewer bytes and a cheaper parse than JSON text on every hot message | -| **One upgrade, then raw frames** | no per-message HTTP request/response or header overhead — the handshake is paid once | +| **One upgrade, then raw frames** | no per-message HTTP request/response or header overhead: the handshake is paid once | | **Bounded send buffers (backpressure)** | a slow consumer slows the producer instead of growing memory without limit; drop or disconnect rather than OOM | | **Coalesce small messages** | batch a burst into one frame to cut syscalls and framing overhead | | **Tuned heartbeats** | ping often enough to detect a dead link in seconds, rarely enough not to wake every idle connection constantly | | **Per-pod connection caps** | memory-per-connection × cap sizes the pod; scale and trigger on connection count, not CPU | The throughline of the WebSocket patterns: most of these are the same levers -that make the system *scale* — bounded buffers and connection caps are what let the +that make the system *scale*: bounded buffers and connection caps are what let the HPA reason about a pod, and binary framing is what keeps the backplane cheap. ## Backpressure in practice The performance table lists bounded send buffers as a lever; here is what that looks like. The idea is simple: each connection gets a fixed-size outbound queue. When the -queue is full — because the client is reading slower than you are writing — you either +queue is full (because the client is reading slower than you are writing), you either **drop the message** (acceptable for live-updating dashboards where the next update replaces the stale one) or **disconnect the client** (preferable when every message must be delivered and the client is stuck): @@ -494,13 +494,13 @@ go func() { }() ``` -The same pattern applies in every language — a bounded queue per connection, drained +The same pattern applies in every language: a bounded queue per connection, drained by a dedicated writer goroutine/task. Without it, a single slow client grows an unbounded buffer until the pod OOMs. ## SignalR and managed backplanes -Everything in this appendix uses raw WebSockets — direct `ws.SendAsync` or +Everything in this appendix uses raw WebSockets: direct `ws.SendAsync` or `c.Write`, manual heartbeats, hand-built backplanes, explicit sequence tracking. The appendix stays on raw sockets because they are cross-language, and the patterns (backplane, resume, binary framing) are universal. @@ -509,14 +509,14 @@ resume, binary framing) are universal. (with automatic fallback to Server-Sent Events or long polling when WebSockets aren't available). It provides: -- **Hub abstraction** — strongly typed methods instead of raw message parsing; the +- **Hub abstraction**: strongly typed methods instead of raw message parsing; the server calls `Clients.All.OrderPlaced(order)` and the client receives a typed callback. -- **Automatic reconnection** — built-in reconnect with configurable backoff, no +- **Automatic reconnection**: built-in reconnect with configurable backoff, no client-side resume logic needed. -- **Connection groups** — server-side `Groups.AddToGroupAsync("tenant-A", connectionId)` +- **Connection groups**: server-side `Groups.AddToGroupAsync("tenant-A", connectionId)` replaces the per-pod registry + backplane fan-out with a single API call. -- **Backplane** — `AddStackExchangeRedis()` wires up a Redis backplane for +- **Backplane**: `AddStackExchangeRedis()` wires up a Redis backplane for scale-out. **When SignalR is the right choice**: .NET-only teams where every producer and consumer @@ -529,8 +529,8 @@ appendix, but with less code and built-in group semantics. examples), binary protobuf framing (SignalR uses its own MessagePack or JSON protocol), or when you need control over the exact wire format for interop with non-.NET clients. -A full walkthrough of SignalR in a Blazor Server app — including the Redis backplane, -connection groups, and a runnable example — is in **Appendix Q · Blazor Server + +A full walkthrough of SignalR in a Blazor Server app (including the Redis backplane, +connection groups, and a runnable example) is in **Appendix Q · Blazor Server + SignalR**. ## A cloud-native WebSocket checklist @@ -543,7 +543,7 @@ SignalR**. to reconnect, and give them time to land on another pod before the pod exits (which is exactly the graceful-shutdown discipline of **Appendix H**). - **Frame binary and versioned**: protobuf `Envelope`s (version + seq + type + - payload), not JSON text — and bound each connection's send buffer so a slow + payload), not JSON text, and bound each connection's send buffer so a slow client applies backpressure instead of exhausting the pod. ### Cross-check it yourself @@ -555,5 +555,5 @@ Deployment down by one and confirm the dropped clients reconnect with backoff an resume from their last sequence rather than losing the stream. --- -*Verification status: verified — [`examples/16-websockets/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/16-websockets/) passes 6/6 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified; [`examples/16-websockets/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/16-websockets/) passes 6/6 checks in each of the six languages on Docker Engine (2026-10-13) (WebSocket connect, cross-pod delivery and broadcast over the Redis pub/sub backplane, monotonic sequence numbers).* diff --git a/_docs/17-appendix-d-sagas.md b/_docs/17-appendix-d-sagas.md index 7a63705d..b7c3aaa0 100644 --- a/_docs/17-appendix-d-sagas.md +++ b/_docs/17-appendix-d-sagas.md @@ -4,29 +4,28 @@ marker: "D" label: "Appendix D" order: 17 part: "Deep-dive appendices" -description: "The deep mechanics behind the saga — where the orchestrator keeps state so it survives a crash, how compensation unwinds committed steps in reverse, and how to thread one step's output to the next without coupling the services." +description: "The deep mechanics behind the saga: where the orchestrator keeps state so it survives a crash, how compensation unwinds committed steps in reverse, and how to thread one step's output to the next without coupling the services." duration: 20 minutes --- The Data chapter drew the happy path of a saga. This appendix is the three hard parts that picture glosses over: **where the state lives**, **how compensation -unwinds a failure**, and **how step B gets data that only step A produced** — -without coupling B to A. +unwinds a failure**, and **how step B gets data that only step A produced**, without coupling B to A. ## Where the saga keeps its state The orchestrator must survive a crash mid-flow, so in-memory state is a -non-starter — a pod restart would strand every in-flight saga with no idea what to +non-starter: a pod restart would strand every in-flight saga with no idea what to compensate. The workhorse option is a **DB-backed state machine**: one row per saga instance holding its `status`, a `step_index`, and a `context` blob, with *you* owning the transitions. The state update and the next command commit -**together** in one transaction — the same outbox discipline from the Data chapter, +**together** in one transaction: the same outbox discipline from the Data chapter, applied to the saga's own progress. {% include excalidraw.html file="17-saga-state" alt="Four ways a saga can keep its state. In-memory: a dict or object only, lost on pod restart, not for production. DB state machine: one row per saga instance with status and context JSON, and you own the transitions. Event-sourced: append step events and rebuild by replay, with full audit history. Workflow engine: Temporal or Camunda, durable execution with retries and timers built in. Whatever the store, the state transition and the outbound command must commit together." - caption="Figure D.1 — Where saga state can live; the DB-backed state machine is the workhorse, but the commit-together rule is the same for all" %} + caption="Figure D.1: Where saga state can live; the DB-backed state machine is the workhorse, but the commit-together rule is the same for all" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -153,16 +152,16 @@ picks up exactly where it left off. ## Compensation unwinds a failed saga -There is no cross-service rollback — when `book_shipping` fails, the `charge_payment` +There is no cross-service rollback: when `book_shipping` fails, the `charge_payment` that already committed in the payment service is *real money*. So compensation -runs a defined **semantic inverse** for each step that completed — `charge → refund`, -`reserve → release` — **in reverse order**, and only for the steps that +runs a defined **semantic inverse** for each step that completed (`charge → refund`, +`reserve → release`), **in reverse order**, and only for the steps that finished. {% include excalidraw.html file="17-saga-compensation" - alt="Forward path: charge_payment then reserve_stock then book_shipping, which fails. On failure the saga compensates in reverse: release_stock then refund_payment — only the steps that completed, newest first" - caption="Figure D.2 — On failure, run each completed step's inverse, newest first" %} + alt="Forward path: charge_payment then reserve_stock then book_shipping, which fails. On failure the saga compensates in reverse: release_stock then refund_payment, only the steps that completed, newest first" + caption="Figure D.2: On failure, run each completed step's inverse, newest first" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -293,15 +292,15 @@ func compensate(ctx context.Context, pool *pgxpool.Pool, sagaID string) error { } ``` -Compensations themselves can fail, so each must be **idempotent and retryable** — a +Compensations themselves can fail, so each must be **idempotent and retryable**: a refund that times out gets retried until it sticks, and a saga stuck compensating is an alert, not a silent loss. ## When step B needs data only step A produced -A common source of bad designs: the orchestrator calls A, then B — but B needs a +A common source of bad designs: the orchestrator calls A, then B, but B needs a value only A produced (a `reservation_id`, a warehouse region). The tempting wrong -answer is to have **B call A directly** — which recreates the exact service-to-service +answer is to have **B call A directly**, which recreates the exact service-to-service coupling the architecture exists to avoid, and hides a dependency the orchestrator can't see. @@ -309,24 +308,24 @@ The right answer is already in the code: A's result goes into the saga's **`context` blob** (`s.context[step] = result`), and the orchestrator passes that context into B. The data flows *through the orchestrator*, the only component that legitimately knows the whole flow. B stays unaware of A, and because the context is -persisted with the saga state, it survives a crash too — resume rehydrates it for +persisted with the saga state, it survives a crash too: resume rehydrates it for free. {% include excalidraw.html file="17-saga-context" alt="The orchestrator owns the saga context. It calls service A, which returns a reservation_id and warehouse region; that output is persisted into the saga context (reservation_id r-77, region eu-1) written after A and read before B; the orchestrator then calls service B with the context, which supplies the reservation_id B needs. If the orchestrator crashes between persisting A's output and calling B, on restart it loads the saga context, sees A is done, and resumes at B without re-calling A." - caption="Figure D.3 — A's output flows to B through the orchestrator's persisted context — B never calls A, and a crash resumes cleanly" %} + caption="Figure D.3: A's output flows to B through the orchestrator's persisted context; B never calls A, and a crash resumes cleanly" %} ### Cross-check it yourself Force the unhappy path. Run the saga with `book_shipping` rigged to fail, and confirm the database shows `release_stock` then `refund_payment` executing in that -order — and that `cancel_shipping` does *not* run, because shipping never completed. +order, and that `cancel_shipping` does *not* run, because shipping never completed. Then kill the orchestrator pod mid-flow and restart it: the `RUNNING` saga should resume from its persisted `step_index` and `context`, not start over. Reverse-order inverses over only the completed steps, plus a clean resume, shows the saga works. --- -*Verification status: verified — [`examples/17-sagas/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/17-sagas/) passes 11/11 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified: [`examples/17-sagas/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/17-sagas/) passes 11/11 checks in each of the six languages on Docker Engine (2026-10-13) (orchestrated saga happy path, compensation in reverse order on failure, skipped compensation for steps that never completed, resume of a RUNNING saga after an orchestrator restart).* diff --git a/_docs/18-appendix-e-errors.md b/_docs/18-appendix-e-errors.md index f45c00d1..2b96f3d4 100644 --- a/_docs/18-appendix-e-errors.md +++ b/_docs/18-appendix-e-errors.md @@ -4,7 +4,7 @@ marker: "E" label: "Appendix E" order: 18 part: "Deep-dive appendices" -description: "One error contract — a stable code, a safe message, a trace id, a retryable flag, field details — mapped onto REST, gRPC, Kafka and GraphQL, plus the retry-storm controls that keep a single failure from becoming an outage." +description: "One error contract (a stable code, a safe message, a trace id, a retryable flag, field details) mapped onto REST, gRPC, Kafka and GraphQL, plus the retry-storm controls that keep a single failure from becoming an outage." duration: 24 minutes --- @@ -12,10 +12,10 @@ Cloud-native systems rarely fall apart on the happy path; they fall apart on the error path, because that path gets the least design attention and the least testing. This appendix fixes the error path with the principle the book applies throughout: **agree one contract, then encode it consistently.** Pick a -single error model — a stable machine code, a safe human message, a correlation id, -a retryable flag, and structured field details — and map *that same model* onto each +single error model (a stable machine code, a safe human message, a correlation id, +a retryable flag, and structured field details) and map *that same model* onto each protocol the system speaks: REST, gRPC, Kafka, and GraphQL. We close on the failure -mode that turns a brief wobble into a full outage — the retry storm — and the small +mode that turns a brief wobble into a full outage (the retry storm) and the small set of controls that prevent it. ## The five facts every error carries @@ -23,9 +23,9 @@ set of controls that prevent it. Before any protocol-specific encoding, decide what an error *is* in this system. Every error, everywhere, carries the same five facts: -- A **stable machine-readable code** clients branch on — `STOCK_UNAVAILABLE`, not a +- A **stable machine-readable code** clients branch on: `STOCK_UNAVAILABLE`, not a localized sentence that changes when someone edits the copy. -- A **human-readable message** that is safe to surface — no internals. +- A **human-readable message** that is safe to surface: no internals. - A **correlation / trace id** that ties the error back to the distributed trace from the Observability chapter, so one id pivots from the client's failure to the exact span that produced it. @@ -33,7 +33,7 @@ Every error, everywhere, carries the same five facts: off correctly instead of guessing. - Structured **field-level details** when the problem is about specific inputs. -The four protocols differ only in *where* those facts go on the wire — REST puts +The four protocols differ only in *where* those facts go on the wire: REST puts them in a `problem+json` body, gRPC in a status plus typed details, Kafka in record headers and a dead-letter record, GraphQL in an `errors[]` entry's `extensions`. Agree the model once as a shared contract, and never leak a stack trace, a SQL fragment, @@ -43,9 +43,9 @@ a coupling that lets a callee's internals constrain a caller. {% include excalidraw.html file="18-one-error-model" alt="One error model maps to four wire formats. Every error carries five facts: a stable machine code like STOCK_UNAVAILABLE, a human-readable message safe to show, a correlation or trace id, a retryable flag with a retry-after when known, and field-level details. Those same facts map onto REST as application/problem+json (RFC 9457), gRPC as a status code plus google.rpc.Status details, Kafka as headers plus a dead-letter record, and GraphQL as an errors array with extensions and a code." - caption="Figure E.1 — One error contract, four wire formats: agree the facts once, then map them per protocol" %} + caption="Figure E.1: One error contract, four wire formats (agree the facts once, then map them per protocol)" %} -## REST — status is the category, the body is the specifics +## REST: status is the category, the body is the specifics REST error handling works on two layers, and conflating them is the usual mistake. The **HTTP status class is the category**, and it is what drives client behaviour: a @@ -53,21 +53,21 @@ The **HTTP status class is the category**, and it is what drives client behaviou `503` say "busy" and must travel with a `Retry-After` header so callers back off; `5xx` says a server fault that an *idempotent* call may retry with backoff. The **body carries the specifics**: RFC 9457 Problem Details (media type -`application/problem+json`, which superseded RFC 7807) gives a standard envelope — -`type`, `title`, `status`, `detail`, `instance` — that you extend with the stable +`application/problem+json`, which superseded RFC 7807) gives a standard envelope +(`type`, `title`, `status`, `detail`, `instance`) that you extend with the stable `code` and the `traceId`. The discipline that makes this maintainable is **one mapping seam**: domain code raises a typed exception, and exactly one place turns it into the wire format, so ad-hoc error JSON never spreads across endpoints. Every version here -omits internals — no stack trace, no exception class name, no SQL — so nothing about the implementation leaks. The status +omits internals (no stack trace, no exception class name, no SQL), so nothing about the implementation leaks. The status matches the category (`409`, a conflict the client can act on), and the body carries the code and the trace id for correlation. {% include excalidraw.html file="18-rest-errors" alt="REST error handling in two layers. Choose the status class first: 4xx is a client error, fix the request and do not retry as-is; 429 or 503 means busy, back off and honour the Retry-After header; 5xx is a server error, retry idempotent calls with backoff. The body is an application/problem+json document (RFC 9457) carrying type, title, status 409, detail, instance, the machine code STOCK_UNAVAILABLE, and a traceId for correlation." - caption="Figure E.2 — REST errors: the HTTP status class is the category; an RFC 9457 problem+json body carries the specifics" %} + caption="Figure E.2: REST errors (the HTTP status class is the category; an RFC 9457 problem+json body carries the specifics)" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -234,31 +234,31 @@ func writeProblem(w http.ResponseWriter, r *http.Request, err error) { Each one registers a single handler family and lets the framework attach the `application/problem+json` content type. In a larger codebase you would key a -base-class handler on the error family — one `switch` on a `code` — so a new domain +base-class handler on the error family (one `switch` on a `code`), so a new domain error means a new case, not a new endpoint-local block of error JSON. -## gRPC — the canonical code *is* the contract +## gRPC: the canonical code *is* the contract gRPC ships a small, fixed set of canonical status codes, and **choosing the right one is the contract**, because clients and the Istio mesh decide retry behaviour straight from the code. `INVALID_ARGUMENT` and `NOT_FOUND` are client bugs: do not retry. `FAILED_PRECONDITION` and `ALREADY_EXISTS` are state issues: do not retry. `UNAVAILABLE` and `ABORTED` are transient: retry with backoff. `RESOURCE_EXHAUSTED` -means you are being rate-limited — honour the `RetryInfo` delay. `DEADLINE_EXCEEDED` +means you are being rate-limited: honour the `RetryInfo` delay. `DEADLINE_EXCEEDED` means the timeout needs tuning. The cardinal sin is collapsing everything into a generic `INTERNAL`, which strips automated clients of the one signal they use to decide whether and when to retry. Beyond the code, attach a `google.rpc.Status` in the trailers carrying the rich -details — `ErrorInfo` (a machine `reason` plus a `domain`) and `RetryInfo` (exactly -how long to wait) — so a gRPC caller receives the same five facts as a REST caller. +details: `ErrorInfo` (a machine `reason` plus a `domain`) and `RetryInfo` (exactly +how long to wait), so a gRPC caller receives the same five facts as a REST caller. `UNAVAILABLE` tells both the client and the mesh "transient, retryable," and the `RetryInfo` tells them precisely how long to back off. {% include excalidraw.html file="18-grpc-errors" alt="gRPC error handling. The canonical status code is the contract: INVALID_ARGUMENT and NOT_FOUND are client bugs, don't retry; FAILED_PRECONDITION and ALREADY_EXISTS are state issues, don't retry; UNAVAILABLE and ABORTED are transient, retry with backoff; RESOURCE_EXHAUSTED means honour the RetryInfo delay; DEADLINE_EXCEEDED means tune the timeout and maybe retry. A google.rpc.Status in the trailers carries the rich details: code UNAVAILABLE 14, a message, an ErrorInfo with reason and domain, and a RetryInfo delay." - caption="Figure E.3 — gRPC errors: a canonical status code drives retry behaviour, with google.rpc.Status details in the trailers" %} + caption="Figure E.3: gRPC errors (a canonical status code drives retry behaviour, with google.rpc.Status details in the trailers)" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -402,14 +402,14 @@ func (s *Inventory) ReserveStock( The shape is identical across stacks: build a `google.rpc.Status` with the canonical code, pack `ErrorInfo` and `RetryInfo` into its details, and hand it to the framework's "abort with rich status" call so it lands in the trailers. The client -library unpacks those details and the mesh reads the code — which is why getting the +library unpacks those details and the mesh reads the code, which is why getting the code right is what prevents both silently lost work and runaway retries. -## Kafka — move failures aside so a poison record can't block the partition +## Kafka: move failures aside so a poison record can't block the partition A Kafka consumer has a failure mode that request/response protocols do not: a single **poison record** can block an entire partition forever. Retry it in place and the -offset never advances, so every record queued behind it is stuck too — +offset never advances, so every record queued behind it is stuck too: head-of-line blocking at the log level. The fix is to **move the failure aside and keep the offset moving**: try a bounded number of in-place retries, then route the record to a delayed-retry topic (reprocessed after a delay, off the live path), and @@ -419,13 +419,13 @@ DLQ record carries the original payload plus error and trace headers. The frameworks differ in how much they do for you. Spring Kafka, SmallRye Reactive Messaging express the policy declaratively; with `Confluent.Kafka`, `aiokafka` and -`rdkafka++` you write the loop yourself — which makes the invariant explicit: +`rdkafka++` you write the loop yourself, which makes the invariant explicit: the commit happens on *every* branch. {% include excalidraw.html file="18-kafka-dlq" alt="A Kafka consumer reads from the orders topic main partition and processes each record. On success it acks and commits, the happy path. On failure it routes the record to a retry.5s delayed-retry topic, then a retry.30s topic, and after N tries to an orders.DLQ dead-letter queue for alerting and tooling. The offset is committed either way, so a poison record never blocks head-of-line processing." - caption="Figure E.4 — Kafka: route failures to delayed-retry topics then a DLQ, and commit the offset on every branch so the partition never stalls" %} + caption="Figure E.4: Kafka (route failures to delayed-retry topics then a DLQ, and commit the offset on every branch so the partition never stalls)" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -597,18 +597,18 @@ investigation. One more requirement makes the whole thing safe: `handle()` must **idempotent**, because at-least-once delivery means a retried record may have already applied in part. -## GraphQL — partial success, with `errors[]` +## GraphQL: partial success, with `errors[]` GraphQL breaks the one-request-one-status assumption that REST and gRPC rely on. A single query fans out across many resolvers, so the response is almost always `HTTP 200` and carries *both* a `data` object and an `errors` array. When one field -fails — say `stock` cannot reach inventory — **that field is set to `null`**, an entry +fails (say `stock` cannot reach inventory), **that field is set to `null`**, an entry is appended to `errors[]` with a message and the `path` to the failed field, and the shared contract rides in `extensions`: `code`, `retryable`, `traceId`. Every other field the client asked for still returns. The anti-pattern to name out loud is **blanket-failing** the whole operation (or -returning a `500`) when only one field broke — that discards data the client already +returning a `500`) when only one field broke: that discards data the client already successfully fetched. The right move is to raise from the *single field's* resolver and let the framework null just that field; partial success is a feature, and the client decides whether to retry only the part that failed. @@ -616,7 +616,7 @@ client decides whether to retry only the part that failed. {% include excalidraw.html file="18-graphql-errors" alt="A GraphQL partial-success response. The HTTP 200 response carries a data object where the order has id 1037 and total 59.90 but stock is null because that field failed. Alongside it an errors array entry names the failure: message inventory unavailable, the path to order then stock, and an extensions object with code UNAVAILABLE, retryable true, and a traceId. Every other field still returns." - caption="Figure E.5 — GraphQL: one HTTP 200, the failed field nulled, and an errors[] entry carrying the machine code in extensions" %} + caption="Figure E.5: GraphQL (one HTTP 200, the failed field nulled, and an errors[] entry carrying the machine code in extensions)" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -787,28 +787,28 @@ func (r *orderResolver) Stock(ctx context.Context, obj *Order) (*int, error) { The pattern is the same everywhere: raise (or `co_return`-throw) from the single field's resolver, and let the framework catch it, null *that* field, and append the `errors[]` entry with your `extensions`. Resisting the urge to let the exception -bubble to the top is the whole discipline — the client sees the order, the total, a +bubble to the top is the whole discipline: the client sees the order, the total, a `null` stock, reads `extensions.code`, and retries just `stock`. -## How a retry storm forms — and how to stop it +## How a retry storm forms and how to stop it Everything so far makes retryability explicit, which is exactly why this last section matters: **retries that are not bounded are how a brief blip becomes a full outage.** The loop is a positive feedback loop. A service slows; calls start timing out; clients -retry — often three times — which *triples* the load on the already-struggling +retry (often three times), which *triples* the load on the already-struggling service; which causes more timeouts; which causes more retries. Naive retry logic is the amplifier. {% include excalidraw.html file="18-retry-storm" - alt="Top: a feedback loop between clients and a struggling order service — clients retry three times, tripling load, which produces more timeouts, which produces more retries. Bottom: six controls that break the loop — timeout, bounded retries, backoff plus jitter, circuit breaker, retry budget, and shed load." - caption="Figure E.6 — Unbounded retries amplify a struggling service; each control removes one multiplier from the loop" %} + alt="Top: a feedback loop between clients and a struggling order service: clients retry three times, tripling load, which produces more timeouts, which produces more retries. Bottom: six controls that break the loop: timeout, bounded retries, backoff plus jitter, circuit breaker, retry budget, and shed load." + caption="Figure E.6: Unbounded retries amplify a struggling service; each control removes one multiplier from the loop" %} The controls that break the loop are the lower band of the figure, and they compose: - **Timeout** every call, so a slow dependency cannot pin a caller's thread or connection indefinitely. -- **Bound the retries** to a small number, and only for *idempotent* calls — retrying +- **Bound the retries** to a small number, and only for *idempotent* calls: retrying a non-idempotent write risks doubling the side effect. - **Backoff with jitter** so a thundering herd of clients does not retry in lockstep and re-synchronise the spike. @@ -819,8 +819,8 @@ The controls that break the loop are the lower band of the figure, and they comp - **Shed load early** with `429` / `503` (or `UNAVAILABLE`) rather than queueing work you have no capacity to finish. -The Istio configuration from the Communications chapter gives several of these — the -timeout, the bounded retry policy, the outlier-detection circuit breaker — at the mesh +The Istio configuration from the Communications chapter gives several of these (the +timeout, the bounded retry policy, the outlier-detection circuit breaker) at the mesh layer, so they apply uniformly without every service re-implementing them. ## One contract, four encodings @@ -828,12 +828,12 @@ layer, so they apply uniformly without every service re-implementing them. The cross-cutting checklist that ties the appendix together: the same five facts ride every protocol, expressed as `problem+json`, a `google.rpc.Status`, a GraphQL `extensions` object, or a Kafka DLQ header. Let each protocol carry the *category* -natively — HTTP status, gRPC code, a GraphQL `errors[]` entry, a Kafka DLQ-versus-retry -routing decision — instead of inventing a parallel scheme. Make retryability explicit +natively (HTTP status, gRPC code, a GraphQL `errors[]` entry, a Kafka DLQ-versus-retry +routing decision) instead of inventing a parallel scheme. Make retryability explicit and machine-readable so clients and the mesh back off correctly rather than guessing. Never leak internals onto the wire; map them to a code and keep the detail in the -server-side log, joined back by the trace id. And bound every retry — timeout, small -maximum, backoff plus jitter, circuit breaker, retry budget — while keeping every +server-side log, joined back by the trace id. And bound every retry (timeout, small +maximum, backoff plus jitter, circuit breaker, retry budget) while keeping every handler idempotent. Get those right and the failure paths become as boring as the happy ones. @@ -842,16 +842,16 @@ happy ones. Prove the contract holds on the wire, in two protocols. For REST, trigger the stock conflict and inspect the response with curl: `curl -i` should show `HTTP/1.1 409`, a `Content-Type: application/problem+json` header, and a body whose `code` is -`STOCK_UNAVAILABLE` and whose `traceId` is present — paste that trace id into Grafana +`STOCK_UNAVAILABLE` and whose `traceId` is present: paste that trace id into Grafana Tempo and confirm it lands on the span that raised the error. For gRPC, call the method with `grpcurl` against a downed inventory and confirm the status is `UNAVAILABLE` (not `INTERNAL`) and that the `RetryInfo` detail carries the two-second delay. Then exercise the storm controls directly: point `hey` at the REST endpoint (or `ghz` at the gRPC one) with the dependency forced to fail, and watch the circuit -breaker trip — total downstream calls should *plateau* rather than climb with offered +breaker trip: total downstream calls should *plateau* rather than climb with offered load. A flat downstream-call line under rising pressure is the retry budget and breaker doing their job; a climbing one means a control is missing. --- -*Verification status: verified — [`examples/18-errors/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/18-errors/) passes 13/13 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified. [`examples/18-errors/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/18-errors/) passes 13/13 checks in each of the six languages on Docker Engine (2026-10-13) (RFC 9457 problem details over REST for validation (422), stock conflict (409) and upstream unavailable (503), with error codes, traceId, retryable and retryAfter fields).* diff --git a/_docs/19-appendix-f-ddd-hexagonal.md b/_docs/19-appendix-f-ddd-hexagonal.md index 99cfed01..14178b1e 100644 --- a/_docs/19-appendix-f-ddd-hexagonal.md +++ b/_docs/19-appendix-f-ddd-hexagonal.md @@ -4,7 +4,7 @@ marker: "F" label: "Appendix F" order: 19 part: "Deep-dive appendices" -description: "Where service boundaries and API contracts come from — strategic DDD to place the lines, tactical DDD to model inside them, and hexagonal architecture to keep REST, gRPC, GraphQL and Kafka out of the domain." +description: "Where service boundaries and API contracts come from: strategic DDD to place the lines, tactical DDD to model inside them, and hexagonal architecture to keep REST, gRPC, GraphQL and Kafka out of the domain." duration: 22 minutes --- @@ -20,23 +20,23 @@ polish rescues the result. ## Boundaries are an API question An API is the contract between two **bounded contexts**, which means the hardest API -questions — where to split a system, what a contract should contain, what stays -private — are boundary questions wearing API clothing, and DDD is the discipline for +questions (where to split a system, what a contract should contain, what stays +private) are boundary questions wearing API clothing, and DDD is the discipline for boundaries. It splits cleanly in two. **Strategic DDD** decides *where the lines go*: subdomains (core, supporting, generic) and the bounded contexts that realise them, each with its own model and its own ubiquitous language. **Tactical DDD** models *what lives inside a line*: aggregates as consistency boundaries, plus entities, value -objects, and domain events — the things that become your API resources and your Kafka +objects, and domain events: the things that become your API resources and your Kafka payloads. Hexagonal architecture then keeps the protocols from leaking into the business logic. The single idea worth carrying out of this appendix: boundaries dominate outcomes far more than protocol polish does. -## Strategic DDD — placing the boundaries +## Strategic DDD: placing the boundaries Classify the business by **value and rate of change**. A *core* subdomain is the -competitive edge — complex, volatile, where the strongest people belong. A +competitive edge: complex, volatile, where the strongest people belong. A *supporting* subdomain is necessary but not differentiating, so keep its design simple. A -*generic* subdomain is a solved problem you adopt or buy rather than invent — auth, +*generic* subdomain is a solved problem you adopt or buy rather than invent: auth, notifications. In the running system, order and pricing are core; inventory and shipping are supporting; auth and notification are generic. The classification is not academic: it tells you where to spend design effort and, later in this book's coupling @@ -45,11 +45,11 @@ discussion, it predicts which parts change most. Each subdomain becomes one or more **bounded contexts**, and a bounded context *is* the natural unit for a service and its API: the scope within which one model and one ubiquitous language stay consistent. The clarifying example is a single word that -refuses to mean one thing — an "order" in fulfillment (a pick-and-pack job) is not an +refuses to mean one thing: an "order" in fulfillment (a pick-and-pack job) is not an "order" in billing (a set of charge lines). Forcing one shared canonical model across the whole company is how you get a schema nobody can change; instead each context exposes *its* model through an API and translates at the edge. The context's published -API is its only public surface — schema, storage, and internal types stay private, +API is its only public surface: schema, storage, and internal types stay private, which is the "no leaky internal models" rule from the anti-patterns chapter restated in DDD terms. And by Conway's law, a boundary that does not match a team's ownership generates constant cross-team coordination, so put the lines where a team can own a @@ -58,59 +58,59 @@ context end to end. {% include excalidraw.html file="19-strategic-ddd" alt="Strategic DDD classifies the business into three kinds of subdomain by value and rate of change. Core: your competitive edge, complex and volatile, your best people (order and pricing). Supporting: necessary but not a differentiator, kept simple (inventory and shipping). Generic: a solved problem you buy or adopt off the shelf (auth and notification). Each subdomain becomes one or more bounded contexts, a model boundary with its own ubiquitous language, and the API is that boundary: one context's order is not another's, and the contract translates between models rather than sharing internals." - caption="Figure F.1 — Subdomains classified by value and volatility; each becomes a bounded context whose API is the boundary" %} + caption="Figure F.1: Subdomains classified by value and volatility; each becomes a bounded context whose API is the boundary" %} -## Context mapping — how contexts integrate +## Context mapping: how contexts integrate Context mapping catalogues the ways two contexts relate, and three of those patterns are the ones an API implements: - An **Open-Host Service** is a context that, because many others integrate with it, - publishes a stable, documented API as its public protocol — that is your REST, + publishes a stable, documented API as its public protocol: that is your REST, gRPC, or GraphQL surface. -- A **Published Language** is the shared, versioned schema that API speaks — OpenAPI, - `.proto`, Avro — which is exactly what lives in the schema registry from the API +- A **Published Language** is the shared, versioned schema that API speaks (OpenAPI, + `.proto`, Avro), which is exactly what lives in the schema registry from the API Registry chapter. - An **Anticorruption Layer** is the translation you build when *consuming* another context, so their concepts never leak into your domain. -The ACL has a cheap opposite worth naming: **Conformist**, where you accept the +The ACL has a cheap opposite: **Conformist**, where you accept the upstream model wholesale and inherit its every change. That is the right call when the -upstream is clean and stable, and the wrong one when it is messy or volatile — there, +upstream is clean and stable, and the wrong one when it is messy or volatile. There, an ACL localises the blast radius of their changes to a single translation layer. The remaining map entries are trade-offs you choose rarely: a Shared Kernel means two contexts share a model and therefore change in lockstep (high coupling, use sparingly); a Partnership means they co-evolve in coordination; Separate Ways means do not integrate at all. The payoff of this section is recognition: the registry-backed contract you already have is a Published Language, and the gateway or consumer-edge translation you -already do is an Anticorruption Layer — now they have names and a theory behind them. +already do is an Anticorruption Layer; now they have names and a theory behind them. {% include excalidraw.html file="19-context-mapping" alt="The eight context-mapping patterns for how two bounded contexts integrate. The three API-relevant ones are starred: Open-Host Service, publish a stable API for all comers; Published Language, a shared schema in OpenAPI, proto, or Avro; and Anticorruption Layer, translate a foreign model at your edge. The others are Conformist, accept the upstream model as-is; Customer and Supplier, upstream serves a negotiated downstream; Shared Kernel, share a model with high coupling used sparingly; Partnership, two contexts evolve together; and Separate Ways, don't integrate at all." - caption="Figure F.2 — The context-mapping patterns; the starred three are the ones an API implements" %} + caption="Figure F.2: The context-mapping patterns; the starred three are the ones an API implements" %} -## Hexagonal architecture — protocols are adapters +## Hexagonal architecture: protocols are adapters Hexagonal architecture (ports and adapters) is the structural answer to "keep the protocols out of the domain," and it is the architectural payoff of the whole protocols discussion. The **domain core** sits in the middle holding entities, -aggregates, and domain logic, and it depends on nothing — no framework, no I/O. It +aggregates, and domain logic, and it depends on nothing: no framework, no I/O. It declares **ports**: inbound ports (what it can be asked to do) and outbound ports -(what it needs from the world). **Driving adapters** — REST, gRPC, GraphQL — translate -an incoming protocol into a call on an inbound port. **Driven adapters** — a Postgres -repository, a Kafka publisher — implement the outbound ports. Every dependency arrow +(what it needs from the world). **Driving adapters** (REST, gRPC, GraphQL) translate +an incoming protocol into a call on an inbound port. **Driven adapters** (a Postgres +repository, a Kafka publisher) implement the outbound ports. Every dependency arrow points inward: adapters depend on the domain, never the reverse. {% include excalidraw.html file="19-hexagonal-ports" alt="A central domain core that depends on nothing, declaring ports. On the left, three driving adapters (REST, gRPC, GraphQL) call inward through an inbound port. On the right, two driven adapters (a Postgres repository and a Kafka publisher) implement outbound ports. All dependency arrows point inward toward the core." - caption="Figure F.3 — The domain core is isolated; the four protocols are interchangeable adapters around it" %} + caption="Figure F.3: The domain core is isolated; the four protocols are interchangeable adapters around it" %} The consequence is the one this book cares about: the four protocols are *interchangeable adapters*. You can add gRPC alongside REST, or a Kafka consumer, or swap Postgres for another store, without touching a line of domain logic. The code -makes the structure literal — the domain declares plain interfaces for what it needs, +makes the structure literal: the domain declares plain interfaces for what it needs, the application service is pure, and the REST handler is a three-line translation from protocol to a domain call. @@ -291,12 +291,12 @@ writeJSON(w, http.StatusCreated, order) ``` Read the three blocks the same way in every language. The **ports** are plain -interfaces the domain declares for what it needs — a repository and an event publisher -— and the domain imports nothing else. The application service `PlaceOrder` is **pure**: +interfaces the domain declares for what it needs (a repository and an event publisher), +and the domain imports nothing else. The application service `PlaceOrder` is **pure**: it creates the aggregate (which enforces its own invariants), saves through the outbound port, publishes a domain event through the outbound port, and never references the web framework, the database driver, or the Kafka client. The **driving adapter** is -trivial — it turns a request body into a command and calls the use case; a gRPC or +trivial: it turns a request body into a command and calls the use case; a gRPC or GraphQL adapter would be the same handful of lines against the same inbound call. The outbound ports are satisfied by **driven adapters** (a Postgres repository, a Kafka publisher) injected at startup. That injection is dependency inversion in service of @@ -306,7 +306,7 @@ the stable thing they all depend on. The example proves it with a second driving adapter in every language: a CLI, `cli-place-order `, installed on the `PATH` of each order-service image. It builds a command, calls the same `PlaceOrder` use case against the same Postgres, and -prints a `CLI_ORDER_CREATED` line — no HTTP server, and no change to any domain file. +prints a `CLI_ORDER_CREATED` line, with no HTTP server and no change to any domain file. Run it inside the running container with `docker exec cndp-order-service cli-place-order bolt-x 5`; the order is then readable through the REST adapter. @@ -314,15 +314,15 @@ the order is then readable through the REST adapter. Tactical DDD is where the modelling inside a context ties three earlier parts of the book together. An **aggregate** is a cluster of objects treated as one consistency -unit — the boundary inside which invariants hold and a single transaction commits — and +unit: the boundary inside which invariants hold and a single transaction commits, and it usually maps to one API resource. Because it is the transaction boundary, it is also the **outbox boundary** from the Data chapter: the state change and the domain event commit together, atomically. **Domain events** (`OrderPlaced`) are first-class in DDD, -and they are precisely the payloads that become Kafka events other contexts react to — +and they are precisely the payloads that become Kafka events other contexts react to: choreography between bounded contexts, as in the Event-Driven chapter. The rule that keeps this tractable is to keep aggregates small: one aggregate per transaction, -reference other aggregates by id rather than nesting them, and let events connect them -— which is exactly why consistency *across* aggregates needs a saga, the subject of the +reference other aggregates by id rather than nesting them, and let events connect them. +That is exactly why consistency *across* aggregates needs a saga, the subject of the Saga appendix. ## How DDD maps to the stack @@ -343,16 +343,16 @@ treatment is Vlad Khononov's *Learning Domain-Driven Design* (O'Reilly, 2021). ### Cross-check it yourself Test the isolation claim directly, because it is the one that pays the rent. In the -domain package — `domain/` in Python or C++, the application service and its port -interfaces in the JVM and .NET versions — grep for framework imports: there should be +domain package (`domain/` in Python or C++, the application service and its port +interfaces in the JVM and .NET versions): grep for framework imports: there should be no `fastapi`, no `org.springframework.web`, no `Microsoft.AspNetCore`, no `drogon`/`grpc`, no database driver. The application service should import only its own domain types and the ports it declares. Then prove replaceability by construction: add a second driving adapter (a gRPC endpoint, or a Kafka consumer) that calls the *same* -`PlaceOrder` use case, and confirm the domain files do not change in that diff — only a +`PlaceOrder` use case, and confirm the domain files do not change in that diff; only a new adapter file appears. A clean domain import list plus a no-domain-change diff when you bolt on a protocol is hexagonal architecture holding. --- -*Verification status: verified — [`examples/19-ddd-hexagonal/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/19-ddd-hexagonal/) passes 17/17 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified: [`examples/19-ddd-hexagonal/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/19-ddd-hexagonal/) passes 17/17 checks in each of the six languages on Docker Engine (2026-10-13) (source scans proving the domain layer imports no framework or persistence types, REST and CLI adapters driving the same use case, input validation, OrderPlaced event emission).* diff --git a/_docs/20-appendix-g-coupling.md b/_docs/20-appendix-g-coupling.md index d373f19f..3871684c 100644 --- a/_docs/20-appendix-g-coupling.md +++ b/_docs/20-appendix-g-coupling.md @@ -4,29 +4,29 @@ marker: "G" label: "Appendix G" order: 20 part: "Deep-dive appendices" -description: "Coupling has three dimensions — integration strength, distance, and volatility — and it only hurts when all three are high at once, which is why looser is not automatically better and why splitting a system can make coupling worse." +description: "Coupling has three dimensions (integration strength, distance, and volatility) and it only hurts when all three are high at once, which is why looser is not automatically better and why splitting a system can make coupling worse." duration: 30 minutes --- This book has used "coupled" and "decoupled" throughout as if loose were always the goal. This appendix sharpens that, using Vlad Khononov's *Balancing Coupling* model. -The reframe is the whole point: coupling is *necessary* — a system with zero coupling -also has zero behaviour — so the skill is to balance it, not eliminate it. The model +The reframe is the whole point: coupling is *necessary*, and a system with zero coupling +also has zero behaviour, so the skill is to balance it, not eliminate it. The model gives precise language for why a shared database is bad but a registry-backed API contract is fine, and it re-diagnoses every anti-pattern from the anti-patterns chapter as one underlying imbalance. -## Coupling is necessary — balance it, don't eliminate it +## Coupling is necessary: balance it, don't eliminate it Components must share *some* knowledge to collaborate, so "loosely coupled" is too -blunt a target. Khononov's model evaluates the **connection between two components** — -not the components themselves — along three dimensions: integration strength, distance, +blunt a target. Khononov's model evaluates the **connection between two components**, +not the components themselves, along three dimensions: integration strength, distance, and volatility. The headline is a single sentence: a coupling is only painful when strength, distance, *and* volatility are all high at once, so reducing any one of the three drops the pain. That sentence is also why "just split it into microservices" can backfire. Splitting -strongly- and volatile-coupled code across a network does not remove the coupling — it +strongly- and volatile-coupled code across a network does not remove the coupling: it adds *distance* to it, pushing all three dimensions high simultaneously. The result is a distributed monolith: the same tangle as before, now with network calls, partial failures, and multi-team deploys layered on top. Distance is something you pay for; you @@ -34,45 +34,45 @@ only want to pay it where the coupling across that distance is weak. ## The three dimensions -**Integration strength** is how much knowledge two parts share — the more they share, +**Integration strength** is how much knowledge two parts share: the more they share, the more likely a change in one forces a change in the other. **Distance** is how far -apart they sit — same method, class, module, service, or system — and it sets the cost +apart they sit (same method, class, module, service, or system) and it sets the cost of any change that has to cross the boundary. **Volatility** is how often the coupled thing changes; it is the dimension you cannot observe directly, but the DDD -appendix already gave the tool to predict it — a subdomain's type tells you its +appendix already gave the tool to predict it: a subdomain's type tells you its change-rate, with core subdomains volatile and generic ones stable. The design rule that falls out of this is Khononov's, rendered as a slogan worth keeping: **modularity = strength XOR distance.** You want exactly one of strength and -distance, never both. Strong-and-local is a cohesive module — healthy. Weak-and-distant +distance, never both. Strong-and-local is a cohesive module, healthy. Weak-and-distant is healthy microservices integrating through contracts. Strong-and-distant is the trap. -A strong, distant coupling on something that *never* changes still costs nothing — which +A strong, distant coupling on something that *never* changes still costs nothing, which is why all three dimensions have to be weighed together rather than chasing "loose coupling" in the abstract. {% include excalidraw.html file="20-three-dimensions" - alt="Khononov's balanced-coupling model has three dimensions, multiplied together. Integration strength: how much knowledge two parts share, from intrusive to contract. Distance: how far apart they sit, from method to class to module to service to system. Volatility: how often the coupled part changes, core versus generic subdomain. The cost of a coupling is that strong plus distant equals global complexity, a distributed monolith — but it only bites when the code is volatile." - caption="Figure G.1 — Coupling has three dimensions; a coupling only hurts when strength, distance, and volatility are all high at once" %} + alt="Khononov's balanced-coupling model has three dimensions, multiplied together. Integration strength: how much knowledge two parts share, from intrusive to contract. Distance: how far apart they sit, from method to class to module to service to system. Volatility: how often the coupled part changes, core versus generic subdomain. The cost of a coupling is that strong plus distant equals global complexity, a distributed monolith, but it only bites when the code is volatile." + caption="Figure G.1: Coupling has three dimensions; a coupling only hurts when strength, distance, and volatility are all high at once" %} ## Integration strength is a ladder Strength is not binary; it is a ladder with four rungs, worst to best: -- **Intrusive** — one component reaches into another's private internals. The canonical +- **Intrusive**: one component reaches into another's private internals. The canonical case is two services sharing one database: either can break the other invisibly, because neither knows what the other depends on. This is the shared-database anti-pattern, and its strength is both maximal and hidden. -- **Functional** — two parts share business logic or rules and must change together for +- **Functional**: two parts share business logic or rules and must change together for functional reasons. -- **Model** — they share a data model or DTOs, typically a shared library of types. -- **Contract** — they integrate only through an explicit, stable, *versioned* contract. +- **Model**: they share a data model or DTOs, typically a shared library of types. +- **Contract**: they integrate only through an explicit, stable, *versioned* contract. That is exactly the REST, gRPC, GraphQL, and Kafka schemas governed by the registry. Lower on the ladder means less shared knowledge and looser coupling, and the rule that makes the ladder actionable is: **the greater the distance, the further down this ladder you must be.** Inside a single bounded context, functional or model coupling is -fine — a shared library among code that already releases together costs little. Across +fine: a shared library among code that already releases together costs little. Across independently-deployed services, only contract coupling is appropriate, because it is the loosest strength that still lets services collaborate; reaching for shared state across that distance is how the distributed monolith forms. @@ -80,21 +80,21 @@ across that distance is how the distributed monolith forms. {% include excalidraw.html file="20-strength-ladder" alt="Integration strength as a four-rung ladder, worst to best, each with its example in our system. Intrusive coupling reaches into another's private internals, such as a shared database or scraping internals. Functional coupling shares business logic and rules, such as duplicated or co-owned domain logic. Model coupling shares a data model or DTOs, such as a shared library of types. Contract coupling integrates via an explicit, stable contract: REST, gRPC, or GraphQL plus a registry schema. Across services, aim for contract coupling." - caption="Figure G.2 — The integration-strength ladder; across services, aim for the bottom rung — contract coupling" %} + caption="Figure G.2: The integration-strength ladder. Across services, aim for the bottom rung: contract coupling" %} ## Strength × distance, and volatility as the trigger Put strength and distance on two axes and the design space resolves into four -quadrants — only one of which is dangerous. +quadrants, only one of which is dangerous. {% include excalidraw.html file="20-coupling-quadrant" - alt="A two-by-two of integration strength (vertical) against distance (horizontal). Strong and close is a cohesive module (healthy); strong and distant is the distributed monolith (the trap); weak and close is needless indirection; weak and distant is healthy microservices. A note states that volatility is the trigger — even the trap quadrant is harmless on code that never changes." - caption="Figure G.3 — Balance means having strength or distance, not both; volatility decides whether an imbalance bites" %} + alt="A two-by-two of integration strength (vertical) against distance (horizontal). Strong and close is a cohesive module (healthy); strong and distant is the distributed monolith (the trap); weak and close is needless indirection; weak and distant is healthy microservices. A note states that volatility is the trigger: even the trap quadrant is harmless on code that never changes." + caption="Figure G.3: Balance means having strength or distance, not both; volatility decides whether an imbalance bites" %} The two healthy quadrants are a cohesive module (strong, close) and contract-integrated services (weak, distant). The bottom-left, weak-and-close, is merely needless -indirection — you have split things that could be merged. The top-right, +indirection: you have split things that could be merged. The top-right, strong-and-distant, is the distributed monolith, where one change ripples across services and teams. @@ -103,15 +103,15 @@ What turns that danger quadrant from a diagram into an outage is the third dimen edit; the same change spanning two services and two teams is a slow, multi-deploy project. **Volatility is the trigger**: a strong, distant coupling on code that never changes is harmless, while the same coupling on hot core-domain code is where slow -delivery and outages originate. So you predict volatility from the domain — keep the +delivery and outages originate. So you predict volatility from the domain: keep the couplings around volatile core subdomains weak and local, and spend your tolerance for stronger or more distant coupling on stable, generic areas. Khononov's full balance -condition: **balance = (strength XOR distance) OR low volatility** — route any +condition: **balance = (strength XOR distance) OR low volatility**: route any unavoidable strong-and-distant coupling to the code that rarely changes. ## The classical coupling vocabulary, mapped to the model -Khononov's three dimensions are not the only vocabulary for coupling — decades of +Khononov's three dimensions are not the only vocabulary for coupling: decades of practitioner writing named the same forces from different angles. They are not rivals; each is a finer lens on **strength** or **distance**, and mapping them onto the model keeps you making decisions instead of collecting terms. @@ -119,16 +119,16 @@ keeps you making decisions instead of collecting terms. | Term | What it names | Where it lands in the model | |---|---|---| | **Module coupling** (content → common → control → stamp → data) | the classic structured-design ladder, content worst, data best | integration **strength**, at the code level | -| **Connascence** (static + dynamic) | the specific way two parts must change together | **strength**, finer-grained — with *locality* = distance | -| **Semantic coupling** | sharing the *meaning* of a concept (what an "order status" is) | strength — shared functional knowledge | +| **Connascence** (static + dynamic) | the specific way two parts must change together | **strength**, finer-grained (with *locality* = distance) | +| **Semantic coupling** | sharing the *meaning* of a concept (what an "order status" is) | strength: shared functional knowledge | | **Temporal coupling** | one step must happen before another in time | a runtime sequencing dependency | -| **Lifecycle coupling** | parts must be built, released, or scaled together | **distance** collapsing — the distributed-monolith smell | -| **Runtime coupling** | at runtime, one part's availability or latency depends on another | operational distance — synchronous call chains (**Appendix M**) | +| **Lifecycle coupling** | parts must be built, released, or scaled together | **distance** collapsing: the distributed-monolith smell | +| **Runtime coupling** | at runtime, one part's availability or latency depends on another | operational distance: synchronous call chains (**Appendix M**) | | **Afferent / efferent** (Ca / Ce) | who depends on you (Ca) versus whom you depend on (Ce) | a *measurable* proxy for fan-in, fan-out, and distance | **Connascence** is worth drawing out because it is the most precise of these. It comes -in two families — *static* forms visible in the code and *dynamic* forms that surface -only at runtime — each ordered weakest to strongest, and it carries one rule that is +in two families (*static* forms visible in the code and *dynamic* forms that surface +only at runtime), each ordered weakest to strongest, and it carries one rule that is the three-dimension model in miniature: the stronger the connascence, the more *local* it must stay. Connascence of position across a service boundary (relying on field order) is fragile; the same thing inside one function is fine. Protobuf field numbers @@ -138,33 +138,33 @@ renumber a field. {% include excalidraw.html file="20-connascence" alt="Connascence in two families, each weaker at the top and stronger at the bottom. Static connascence, visible in the code: Name (agree on an identifier), Type (agree on a type or shape), Meaning (a shared convention or magic value), Position (the same order of arguments or fields), Algorithm (the same algorithm on both sides). Dynamic connascence, only at runtime: Execution order (A must run before B), Timing (race-sensitive), Value (values must stay in agreement), Identity (the same instance or entity). The rule: the stronger the connascence between two parts, the more local they must stay." - caption="Figure G.4 — Connascence: static and dynamic forms, weakest to strongest — keep stronger forms more local" %} + caption="Figure G.4: Connascence (static and dynamic forms, weakest to strongest): keep stronger forms more local" %} The **afferent / efferent** entry is the one you can measure. Afferent coupling (Ca) counts who depends on a module; efferent (Ce) counts what it depends on. Their ratio gives an *instability* score, `I = Ce / (Ca + Ce)`, running from 0 (nothing -depends outward — maximally stable) to 1 (depends on everything, depended on by nothing -— maximally unstable). The guidance that falls out is the stable-abstractions idea: +depends outward, maximally stable) to 1 (depends on everything, depended on by nothing, +maximally unstable). The guidance that falls out is the stable-abstractions idea: things many others depend on (low instability, high Ca) should change rarely and expose -abstractions, not concretions — which is just "keep strong, distant coupling on stable +abstractions, not concretions, which is just "keep strong, distant coupling on stable code," restated as a number you can compute from an import graph. ## Choosing a coupling strategy with Cynefin How much coupling is "right" depends on how well you understand the domain, and the Cynefin framework names the four situations you might be in. The mistake it guards -against is applying a mature-domain answer — standardise everything on tidy contracts — +against is applying a mature-domain answer (standardise everything on tidy contracts) to a domain you don't yet understand, where the boundaries themselves are still guesses. {% include excalidraw.html file="20-cynefin-coupling" - alt="The four Cynefin domains, each with its coupling posture. Complex (probe, sense, respond): keep coupling loose and reversible — favour events and contracts, and don't split prematurely. Complicated (sense, analyse, respond): design boundaries with DDD context mapping and invest in the right contracts. Chaotic (act, sense, respond): stabilise first with strong, local coupling, even a monolith, then refactor as it clarifies. Clear (sense, categorise, respond): standardise on contract coupling and codify the known-good with registry-checked schemas." - caption="Figure G.5 — Cynefin and coupling: match the posture to how well you understand the domain" %} + alt="The four Cynefin domains, each with its coupling posture. Complex (probe, sense, respond): keep coupling loose and reversible: favour events and contracts, and don't split prematurely. Complicated (sense, analyse, respond): design boundaries with DDD context mapping and invest in the right contracts. Chaotic (act, sense, respond): stabilise first with strong, local coupling, even a monolith, then refactor as it clarifies. Clear (sense, categorise, respond): standardise on contract coupling and codify the known-good with registry-checked schemas." + caption="Figure G.5: Cynefin and coupling, match the posture to how well you understand the domain" %} The throughline of the coupling model: in a **complex** domain you keep strength low and reversibility high because you *will* move the boundaries; once the domain becomes **clear**, you can afford to standardise on the loosest workable strength -— contract coupling — and freeze it. The anti-pattern is treating an early, complex +(contract coupling) and freeze it. The anti-pattern is treating an early, complex domain as if it were clear, splitting it into services with contracts you then break repeatedly. That is the model's "don't split prematurely" lesson dressed as a sense-making framework. @@ -181,13 +181,13 @@ sits. Read through this lens, every protocol decision in the book was secretly a coupling decision, and the answer was always the same: choose **contract coupling**, the loosest strength that still works across service distance. The registry is what keeps that -contract coupling stable — its compatibility rules stop a producer from silently +contract coupling stable: its compatibility rules stop a producer from silently *raising* the integration strength on its consumers, which is precisely what a breaking change is. There is a finer-grained classical lens on the same idea: **connascence** (Page-Jones), which names the specific ways two pieces of code must change together. API versioning is -managing connascence of name and position in your schemas — and protobuf field numbers +managing connascence of name and position in your schemas. And protobuf field numbers are connascence of position made explicit and held stable, which is why you never renumber a field. The model also re-diagnoses the anti-patterns chapter as a single principle rather than a list of separate rules: a shared database is intrusive strength @@ -199,7 +199,7 @@ only carry a contract. The design-review checklist, drawn straight from the model: -- Don't chase zero coupling — chase **balance**. Strong coupling is fine when it is +- Don't chase zero coupling: chase **balance**. Strong coupling is fine when it is local and stable. - As distance grows, drive strength **down**: across services, integrate only through versioned contracts, never shared state. @@ -208,7 +208,7 @@ The design-review checklist, drawn straight from the model: - Use DDD subdomain types to **predict volatility**, and put your weakest couplings around the parts that change most. - Name the **connascence** before you move code: stronger forms (position, algorithm, - value, identity) must stay local — weaken them to a contract before they cross a + value, identity) must stay local: weaken them to a contract before they cross a service boundary. - Match coupling to **certainty**: in an unclear, complex domain keep it loose and reversible; standardise on contracts only once the domain is clear. @@ -223,20 +223,20 @@ Khononov's `coupling.dev` core-concepts series, Meilir Page-Jones's connascence This appendix has no code to run, so cross-check it against your own system in a design review. Pick one real cross-service integration and score its connection on all three -dimensions: rate the **strength** by naming the rung — does the consumer touch a shared +dimensions: rate the **strength** by naming the rung: does the consumer touch a shared table (intrusive), import a shared model (model), or speak only a registered schema -(contract)? Then name the strongest **connascence** that crosses the boundary — if it is -position, algorithm, value, or identity, that is your first thing to weaken. Rate the **distance** — same module, or two services owned by two teams? -Rate the **volatility** — is the thing being shared in a core subdomain that changes +(contract)? Then name the strongest **connascence** that crosses the boundary: if it is +position, algorithm, value, or identity, that is your first thing to weaken. Rate the **distance**: same module, or two services owned by two teams? +Rate the **volatility**: is the thing being shared in a core subdomain that changes weekly, or a generic one that has not moved in a year? A connection that scores high on all three is your distributed-monolith risk, and the model tells you which lever is cheapest to pull: usually drive strength down to a contract, since distance and volatility are often fixed by the org chart and the domain. If your worst-scoring integration is already contract-coupled and sits on stable code, the model says to leave -it alone — which is the appendix's real lesson: looser is not automatically better. +it alone, which is the appendix's real lesson: looser is not automatically better. --- -*Verification status: unverified — this is a conceptual appendix with no runnable +*Verification status: unverified. This is a conceptual appendix with no runnable example; its claims are model-level (Khononov's *Balancing Coupling*) rather than code that executes. There is no `examples/` runner for it; verification here means a design review confirms the three-dimension scoring matches how changes propagate in a diff --git a/_docs/21-appendix-h-shutdown.md b/_docs/21-appendix-h-shutdown.md index a2745c99..d7814b17 100644 --- a/_docs/21-appendix-h-shutdown.md +++ b/_docs/21-appendix-h-shutdown.md @@ -4,14 +4,14 @@ marker: "H" label: "Appendix H" order: 21 part: "Deep-dive appendices" -description: "Shutting a pod down without dropping work — a short protocol between the app and Kubernetes: fail readiness first, cover the deregistration race, drain HTTP and consumers inside the grace budget, and stay idempotent for the inevitable hard kill." +description: "Shutting a pod down without dropping work, a short protocol between the app and Kubernetes: fail readiness first, cover the deregistration race, drain HTTP and consumers inside the grace budget, and stay idempotent for the inevitable hard kill." duration: 21 minutes --- This appendix is the operational flip side of factor IX, disposability, from the Cloud-Native Principles chapter. Pods are cattle: every deploy, scale-down, autoscale event, or reschedule kills and replaces them, often many times a day. An ungraceful -exit drops in-flight requests, abandons unacked messages, and leaves work half-done — +exit drops in-flight requests, abandons unacked messages, and leaves work half-done: the scatter of `5xx` blips users notice during an otherwise-healthy rolling deploy. Graceful shutdown is not a single setting; it is a short protocol the app and the platform perform together, on a clock: stop taking new work, finish what you @@ -20,8 +20,8 @@ accepted, drain consumers and sockets, then exit before the grace period runs ou ## Disposability is a promise you have to keep The promise is the second half of factor IX: a process should start fast *and* shut -down cleanly on demand. Cleanly means a definite sequence — stop accepting new work, -finish what you already accepted, release resources, then exit — and the platform hands +down cleanly on demand. Cleanly means a definite sequence: stop accepting new work, +finish what you already accepted, release resources, then exit, and the platform hands you a fixed budget to do it in. Everything in this appendix is that sequence, on both sides of the app/Kubernetes boundary. @@ -30,33 +30,33 @@ app/Kubernetes boundary. When Kubernetes terminates a pod it sends `SIGTERM` to PID 1, starts the `terminationGracePeriodSeconds` clock, and will `SIGKILL` the process if it overruns. The subtlety that catches teams out is that removing the pod from the Service endpoints -is **asynchronous** — kube-proxy, the kubelet, and the mesh all have to catch up — so +is **asynchronous**: kube-proxy, the kubelet, and the mesh all have to catch up, so for a moment *after* `SIGTERM`, traffic can still arrive at a pod that is already shutting down. {% include excalidraw.html file="21-shutdown-sequence" alt="Two lanes on a left-to-right timeline. Kubernetes: SIGTERM to PID 1, grace clock starts (default 30s), endpoint removal is asynchronous, SIGKILL if you overrun. Your pod, triggered by SIGTERM: flip readiness DOWN, preStop sleep about 5 seconds, drain HTTP plus Kafka and sockets and pools, then exit cleanly before the grace period ends. A note says readiness DOWN plus preStop keep new traffic off the dying pod during the async-removal race, and liveness must stay UP throughout." - caption="Figure H.1 — Shutdown is a timed two-lane protocol; readiness and preStop cover the deregistration race" %} + caption="Figure H.1: Shutdown is a timed two-lane protocol; readiness and preStop cover the deregistration race" %} The pod's side of the contract is five rules: -- `SIGTERM` is the signal — Kubernetes sends it to PID 1, then waits +- `SIGTERM` is the signal: Kubernetes sends it to PID 1, then waits `terminationGracePeriodSeconds` (default 30s) before `SIGKILL`. - **Readiness DOWN** is how you stop *new* traffic: the pod leaves the Service endpoints but keeps running to finish in-flight work. Do this first. -- There is a **race** — endpoint removal is asynchronous — so a short `preStop` sleep +- There is a **race**: endpoint removal is asynchronous, so a short `preStop` sleep covers the lag while the rest of the cluster stops routing to you. - **Liveness must stay UP** during the drain, or the kubelet restarts a pod that is winding down. This is the distinction people get backwards: you *want* readiness to fail, but you must *not* let liveness fail. -- Size the grace period **larger than your longest acceptable in-flight operation** — +- Size the grace period **larger than your longest acceptable in-flight operation**: too short cuts work off mid-flight, too long stalls every deploy. ## Graceful means something different per workload The trap is assuming "graceful shutdown" is one thing. In practice only HTTP request -draining tends to be automatic — the server stops accepting and finishes what is in +draining tends to be automatic: the server stops accepting and finishes what is in flight. Everything else you wire up yourself in the shutdown hook. **Kafka consumers** stop polling, finish the records already in hand, commit offsets, and leave the group so the rebalance is clean. **WebSockets** (from the WebSockets appendix) cannot be @@ -68,8 +68,8 @@ Underneath all of it is the safety net: because a hard `SIGKILL` is always possi {% include excalidraw.html file="21-per-workload" - alt="Graceful shutdown means something different per workload, in four columns. HTTP REST/gRPC: stop accepting new, finish in-flight requests bounded by a timeout — the one that's automatic. Kafka consumer: stop polling, finish current records, commit offsets, leave the group. WebSocket: send a close or go-away, clients reconnect elsewhere, lean on the per-connection sequence and backplane. Jobs and scheduler: don't start new work, let running jobs finish or checkpoint progress, resume after restart. Underneath: make every handler idempotent, because a hard SIGKILL is always possible." - caption="Figure H.2 — Only HTTP drains automatically; consumers, sockets, and jobs each need their own shutdown logic, all backed by idempotency" %} + alt="Graceful shutdown means something different per workload, in four columns. HTTP REST/gRPC: stop accepting new, finish in-flight requests bounded by a timeout: the one that's automatic. Kafka consumer: stop polling, finish current records, commit offsets, leave the group. WebSocket: send a close or go-away, clients reconnect elsewhere, lean on the per-connection sequence and backplane. Jobs and scheduler: don't start new work, let running jobs finish or checkpoint progress, resume after restart. Underneath: make every handler idempotent, because a hard SIGKILL is always possible." + caption="Figure H.2: Only HTTP drains automatically; consumers, sockets, and jobs each need their own shutdown logic, all backed by idempotency" %} ## The application side @@ -257,7 +257,7 @@ HTTP, then drain everything else. ## The Kubernetes side The platform half of the contract is the same whatever language the service is written -in — three knobs that have to agree with the application's timing. +in: three knobs that have to agree with the application's timing. ```yaml # deployment.yaml — give the pod time to deregister and drain @@ -279,8 +279,8 @@ spec: `terminationGracePeriodSeconds` is the total budget, and it must exceed `preStop` plus your worst-case drain plus a buffer. The readiness probe's `periodSeconds × failureThreshold` -sets how fast Kubernetes stops routing after you report `DOWN` — roughly ten -seconds here — which your `preStop` sleep should cover. Liveness points at a *separate* +sets how fast Kubernetes stops routing after you report `DOWN` (roughly ten +seconds here), which your `preStop` sleep should cover. Liveness points at a *separate* `/healthz` that stays up throughout the drain, so the kubelet does not restart a pod that is cleanly winding down. The `preStop` sleep looks crude, but it is the pragmatic fix for the asynchronous-deregistration race; align its duration with the probe timing rather than @@ -288,15 +288,15 @@ picking an arbitrary number. ## Beyond HTTP: messages, sockets, jobs -The shutdown hook is where the non-HTTP drains live, and the Kafka one is worth doing -carefully because it connects to at-least-once delivery from the Event-Driven chapter: -stop polling, finish the records in hand, **commit offsets** on the way out to minimise -redelivery, and **leave the group** so the rebalance is fast. WebSockets cannot be -finished, so graceful there means handing the client off — send a close/go-away and let +The shutdown hook is where the non-HTTP drains live. The Kafka drain connects to +at-least-once delivery from the Event-Driven chapter: stop polling, finish the records +in hand, **commit offsets** on the way out to minimise redelivery, and **leave the +group** so the rebalance is fast. WebSockets cannot be finished, so graceful there +means handing the client off: send a close/go-away and let it reconnect to another pod, exactly what the WebSockets appendix's reconnection design enables. Jobs and schedulers stop starting new work and checkpoint so they resume after restart. And the philosophical anchor of the whole appendix: you *cannot* guarantee a -graceful exit, because the kubelet will sometimes `SIGKILL` you — so idempotency is what +graceful exit, because the kubelet will sometimes `SIGKILL` you, so idempotency is what makes the system safe, and graceful shutdown just makes the unsafe window rare. ## Graceful shutdown checklist @@ -307,10 +307,10 @@ Read it as an ordered runbook: stop accepting. - Add a short `preStop` sleep to cover asynchronous endpoint removal; keep **liveness UP** throughout the drain. -- Drain in-flight HTTP, commit consumer offsets, close sockets and pools — all bounded by +- Drain in-flight HTTP, commit consumer offsets, close sockets and pools, all bounded by `terminationGracePeriodSeconds`. - Make the work **idempotent** so the inevitable occasional `SIGKILL` is survivable. -- **Test it under a real `SIGTERM` during a rolling deploy** — not just a happy-path +- **Test it under a real `SIGTERM` during a rolling deploy**, not just a happy-path integration test. References: the Quarkus lifecycle guide and Markus Eisele's "Quarkus Graceful Shutdown" @@ -320,16 +320,16 @@ References: the Quarkus lifecycle guide and Markus Eisele's "Quarkus Graceful Sh The one test teams skip is the only one that proves the recipe: a real `SIGTERM` mid-request during a rollout. Put steady traffic on the service with `hey` (a modest concurrency for a -minute or two), and while it runs, trigger a rolling restart — `kubectl rollout restart +minute or two), and while it runs, trigger a rolling restart: `kubectl rollout restart deployment/order-service`. The pass condition is simple and binary: `hey`'s summary shows **zero non-2xx responses** across the restart. If you see a cluster of `5xx` or connection -resets timed to the restart, walk the protocol backwards — the usual culprit is readiness +resets timed to the restart, walk the protocol backwards: the usual culprit is readiness not flipping early enough, or a `preStop`/grace-period that is shorter than the real drain. For the Kafka path, restart a consumer pod mid-stream and confirm from the broker that the -group rebalanced quickly and offsets advanced — no long stall, no large redelivery spike. +group rebalanced quickly and offsets advanced: no long stall, no large redelivery spike. A clean rollout under live load, not a green integration test, is evidence that graceful shutdown works. --- -*Verification status: verified — [`examples/21-graceful-shutdown/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/21-graceful-shutdown/) passes 17/17 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified: [`examples/21-graceful-shutdown/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/21-graceful-shutdown/) passes 17/17 checks in each of the six languages on Docker Engine (2026-10-13) (one SIGTERM, readiness flip to 503, the drain log lines, exit code 0 inside the grace period, a `docker stop` timing check, restart recovery, and orders persisting across the restart. In-flight request draining is not checked by the script).* diff --git a/_docs/22-appendix-i-l7-routing.md b/_docs/22-appendix-i-l7-routing.md index 0429089d..5104cf58 100644 --- a/_docs/22-appendix-i-l7-routing.md +++ b/_docs/22-appendix-i-l7-routing.md @@ -4,12 +4,12 @@ marker: "I" label: "Appendix I" order: 22 part: "Deep-dive appendices" -description: "Layer-7 routing as a four-layer stack of software — edge, gateway, mesh, and in-app — that decides on every HTTP and gRPC envelope, north/south and east/west, including rule-driven routing pushed inside the service." +description: "Layer-7 routing as a four-layer stack of software (edge, gateway, mesh, and in-app) that decides on every HTTP and gRPC envelope, north/south and east/west, including rule-driven routing pushed inside the service." duration: 32 minutes --- This is the longest appendix in the book, large enough to be a talk of its own. The -reframe it argues for: L7 routing in 2026 is not an appliance in a rack — it is layered +reframe it argues for: L7 routing in 2026 is not an appliance in a rack: it is layered software running inside Kubernetes that makes a decision on every HTTP and gRPC envelope crossing the network, both north/south (clients into the system) and east/west (service to service). The chapter walks what each layer can see and decide, shows real config for @@ -17,7 +17,7 @@ the mesh, covers TLS, sticky sessions, traffic steering, and the latency budget, goes *inside* the service for rule-driven routing, and finally circles back to east/west, where most of the decisions happen. -## L4 versus L7 — what each layer can decide on +## L4 versus L7: what each layer can decide on A Layer-4 load balancer sees *connections*: IPs, ports, and the TCP 5-tuple. It can pick a backend and hash-balance, but it cannot read URLs, headers, cookies, methods, or gRPC @@ -31,7 +31,7 @@ else (richer, slightly slower). {% include excalidraw.html file="22-l4-vs-l7" alt="Two columns. L4 transport sees source and destination IP and port, TCP/UDP protocol, and TLS SNI hostname only; it can pick a backend by IP or port and load-balance by the 5-tuple, but cannot read URLs or headers or route per user or cookie. L7 application sees the host header, URL path, HTTP method, query params, request and response headers, cookies, JWT claims, and the gRPC service.method; it can route by path, header, or cookie, branch by header to a canary, version, or user, and inject or rewrite headers and rate-limit." - caption="Figure I.1 — L4 sees connections (IPs, ports, the 5-tuple); L7 sees the whole HTTP/gRPC conversation and can route on any of it" %} + caption="Figure I.1: L4 sees connections (IPs, ports, the 5-tuple); L7 sees the whole HTTP/gRPC conversation and can route on any of it" %} ## The four-layer stack @@ -41,26 +41,26 @@ decision. A request is a chain of independent L7 decisions across them. {% include excalidraw.html file="22-l7-layers" alt="North/south: a client passes through edge/ingress (TLS, host/path), then the API gateway (auth, rate limit), then a mesh sidecar (mTLS, retries), then the service (in-app rules). East/west: service A and service B talk through mesh sidecars that add mTLS, outlier detection, and locality routing. Each layer makes a distinct decision and emits a trace span." - caption="Figure I.2 — L7 routing is a four-layer stack; the mesh applies the same routing to every internal call, not just at the edge" %} + caption="Figure I.2: L7 routing is a four-layer stack; the mesh applies the same routing to every internal call, not just at the edge" %} The **edge / ingress** layer (HAProxy, Nginx, Envoy, Traefik) is where TLS terminates and coarse host/path routing happens. The **API gateway** (Kong, an Istio gateway, -Spring Cloud Gateway) handles product-level concerns — authentication, rate limiting, +Spring Cloud Gateway) handles product-level concerns: authentication, rate limiting, transformation. The **service mesh** (Istio with Envoy sidecars, or Linkerd) governs east/west traffic, giving every internal call mTLS, retries, outlier detection, and locality-aware load balancing. And when a decision depends on business rules the platform cannot know, routing pushes into the **app** itself. Each box can refuse, transform, or -branch the request, and each emits a trace span — so the path from the Observability +branch the request, and each emits a trace span, so the path from the Observability chapter, viewed through the L7 lens, is observable end to end. {% include excalidraw.html file="22-l7-journey" alt="A request's L7 path across five components: Client (request), Edge LB (TLS, host and path), API Gateway (auth, rate-limit), Mesh sidecar (mTLS, retry, locality), and the Service (in-app routing). Each box can refuse, transform, or branch the request and emits a trace span for the next layer." - caption="Figure I.3 — A request's path: Client → Edge LB → API Gateway → Mesh sidecar → Service, each making its own L7 decision" %} + caption="Figure I.3: A request's path: Client → Edge LB → API Gateway → Mesh sidecar → Service, each making its own L7 decision" %} ## Content-based routing -HAProxy and Nginx are the workhorses of the edge — mature, fast, predictable. HAProxy +HAProxy and Nginx are the workhorses of the edge: mature, fast, predictable. HAProxy reads the envelope with ACLs (`path_beg` for URL prefixes, `hdr` for headers, `content-type` for the body kind) and `use_backend` dispatches on them; Nginx uses `location` blocks and the `map` directive to turn a header value into an upstream pool. @@ -87,15 +87,15 @@ spec: - route: [{ destination: { host: orders, subset: v1 } }] # default ``` -Three match rules — by URI prefix (which is how you route on a gRPC `service.method`, since -gRPC paths look like `/package.Service/Method`), by header, and by cookie — each sending to +Three match rules: by URI prefix (which is how you route on a gRPC `service.method`, since +gRPC paths look like `/package.Service/Method`), by header, and by cookie, each sending to a different subset, with a default route at the bottom that is *critical*: without it, unmatched requests have nowhere to go. The CRD lives in Git, gets reviewed like any other change, and Istio reconfigures every sidecar within seconds. ## TLS termination -TLS at the edge comes in three modes. **Terminate-and-decrypt** is the common case — it +TLS at the edge comes in three modes. **Terminate-and-decrypt** is the common case: it lets the router do everything L7. **Terminate-and-re-encrypt** is the most secure default in a mesh, since mTLS inside is automatic. **Passthrough** is for traffic you must not see (client mTLS for regulated flows): route by SNI only, decrypt nothing. cert-manager is the @@ -106,21 +106,21 @@ amortise it with session resumption, TLS 1.3, and HTTP/2 multiplexing. ## Sticky sessions There are three affinity modes, and the default is the one you want. **No affinity** sends -any request to any pod — even load, and client state lives elsewhere (Redis, the JWT, the +any request to any pod: even load, and client state lives elsewhere (Redis, the JWT, the database). **Cookie-based** stickiness is the fallback when something stateful must stay on -one pod: the balancer sets a cookie and routes subsequent requests back to the same pod — +one pod: the balancer sets a cookie and routes subsequent requests back to the same pod, but that ties the session to one pod's lifetime, so a restart loses it. **IP-hash** needs no cookie but is fragile behind NAT and mobile networks, where many users share an address and load skews. The real cost is that stickiness fights elastic scaling, fast restarts, and -graceful shutdown — the entire cloud-native delivery model. Treat it as a choice made +graceful shutdown: the entire cloud-native delivery model. Treat it as a choice made for a known constraint, never a default, and prefer to externalise state (Redis for session data, JWT claims for identity, the backplane from the WebSockets appendix for fan-out) so the balancer can return to no-affinity. {% include excalidraw.html file="22-sticky-sessions" - alt="Three affinity modes. No affinity: any pod gets any request, load is even, client state lives elsewhere — the preferred default. Cookie-based: the LB sets a session cookie so the client returns to the same pod, works behind any L7 LB, but ties the session to the pod's life. IP-hash: hash the client IP to a pod, no cookies needed, but NAT and mobile NAT break it and load can skew badly. The cost of stickiness is uneven load, painful pod restarts, broken scale-to-zero, and every restart re-balancing all sessions." - caption="Figure I.4 — Affinity modes: no-affinity is the default; cookie and IP-hash stickiness both fight elastic scaling" %} + alt="Three affinity modes. No affinity: any pod gets any request, load is even, client state lives elsewhere: the preferred default. Cookie-based: the LB sets a session cookie so the client returns to the same pod, works behind any L7 LB, but ties the session to the pod's life. IP-hash: hash the client IP to a pod, no cookies needed, but NAT and mobile NAT break it and load can skew badly. The cost of stickiness is uneven load, painful pod restarts, broken scale-to-zero, and every restart re-balancing all sessions." + caption="Figure I.4: Affinity modes: no-affinity is the default; cookie and IP-hash stickiness both fight elastic scaling" %} ## Intelligent traffic steering @@ -157,12 +157,12 @@ spec: labels: { version: v2 } ``` -Ramping the canary is changing the weights in Git — typically 10, 25, 50, 100 percent with -metric gates between steps — and a controller like Flagger or Argo Rollouts flips the weights +Ramping the canary is changing the weights in Git (typically 10, 25, 50, 100 percent with +metric gates between steps), and a controller like Flagger or Argo Rollouts flips the weights back in seconds if error rate or latency on `v2` crosses a threshold. Header-based steering adds surgical precision: an upstream identity proxy sets a header for opted-in users and the `VirtualService` routes *those* users to `v2`, on real production data with zero blast radius. -Payload modification is the cheap cousin — header inject/strip and URL rewrites are nearly +Payload modification is the cheap cousin: header inject/strip and URL rewrites are nearly free because the router is already parsing the line and headers (this is how the W3C `traceparent` header propagates, and how a gateway validates a JWT once and injects trusted claims downstream). Body transformation is the expensive one: it forces the router to buffer @@ -172,25 +172,25 @@ can do it in the headers, do it in the headers. {% include excalidraw.html file="22-traffic-steering" alt="Six release shapes built from the same L7 primitive. Canary: 90% v1, 10% v2, watch metrics and ramp. Blue/Green: all v1 to all v2, flip atomically with fast rollback. A/B test: a cohort maps to a variant, stable per user, measure. Header-based: an x-internal header routes to v2 for dark launches and betas. Geo/latency: EU users to EU pods for lower RTT and data residency. Shadow: copy traffic to v2 to test under load with no user impact." - caption="Figure I.5 — Six familiar release shapes, all the same weighted/header L7 rule with different deploy-and-rollback discipline" %} + caption="Figure I.5: Six familiar release shapes, all the same weighted/header L7 rule with different deploy-and-rollback discipline" %} ## Routing inside the app by business rules There is a threshold where routing leaves the network and enters the service. If the decision fits in a few YAML lines and depends only on the envelope, do it at the mesh. If it depends on the *payload*, requires a decision table that domain experts edit, or involves many -intersecting conditions — think routing a healthcare message by patient class and encounter -type — push it into a rule engine inside the app, behind the mesh, not in a network CRD. The +intersecting conditions (think routing a healthcare message by patient class and encounter +type), push it into a rule engine inside the app, behind the mesh, not in a network CRD. The discipline is "data, not code": the routing logic lives in an externalised ruleset the domain experts own, separate from the dispatcher, and changes without redeploying the service. {% include excalidraw.html file="22-business-rule-routing" alt="Routing by business rules inside the service. Inputs from REST, WebSocket, or Kafka reach an in-service router that enriches the order with context, evaluates an externalised ruleset in a rule engine (Drools, NRules or durable-rules, all forward chaining; Lua in C++ and Go), and dynamically dispatches to a list of destinations. The router fans out to service A on a Kafka topic, service B over HTTP, and a DLQ/audit topic. The rule engine sits beside the router and is consulted per request." - caption="Figure I.6 — When routing depends on a business rule, the service's router evaluates an externalised ruleset and dispatches to services A, B, or a DLQ" %} + caption="Figure I.6: When routing depends on a business rule, the service's router evaluates an externalised ruleset and dispatches to services A, B, or a DLQ" %} -Each ecosystem composes this from its own rule engine — a Rete forward-chaining engine in -most cases — feeding a small dispatcher. First the **ruleset**, read as a set of "when … then …" +Each ecosystem composes this from its own rule engine (a Rete forward-chaining engine in +most cases) feeding a small dispatcher. First the **ruleset**, read as a set of "when … then …" statements that set a list of destinations: {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -578,7 +578,7 @@ func (s *Server) route(w http.ResponseWriter, r *http.Request) { The shapes converge: a forward-chaining rule engine (Drools, NRules, durable-rules, or Lua) evaluates the enriched fact and produces a `where_to` list, and a small dispatcher fans the input out to each destination. The pattern is the classic content-based router from the -enterprise-integration literature — the value is that the *decision* is data a domain expert +enterprise-integration literature: the value is that the *decision* is data a domain expert edits, not code a developer redeploys. ## The cost of L7 hops @@ -587,20 +587,20 @@ Every L7 capability costs time, and it is a budget you spend before your service TLS termination is sub-2ms, path/header routing sub-1ms, JWT validation a few ms, a WAF a few more, each mesh sidecar about a millisecond, a rate-limit Redis lookup a couple of ms. Body inspection is the outlier at roughly 5–50ms depending on size. The working heuristic is to -stay under about 10ms of total L7 on hot paths — and with four layers (edge, gateway, sidecar +stay under about 10ms of total L7 on hot paths, and with four layers (edge, gateway, sidecar out, sidecar in) that means every layer must justify its latency. The common pitfalls follow from this: too many hops nobody measured until p99 spiked; expensive regex matches; sticky sessions fighting the platform; body inspection blowing the budget; per-request auth lookups cascading under load; and timeout misalignment, which is the gateway-to-mesh-to-service version of the -retry storm from the Error Handling appendix. The fix is identical — bound retries, align +retry storm from the Error Handling appendix. The fix is identical: bound retries, align timeouts top-down, and give the deepest layer the strictest budget. {% include excalidraw.html file="22-hop-cost" alt="A latency budget per L7 stage: TLS termination 0.5 to 2 ms, L7 path/header routing under 1 ms, Auth/JWT validation 1 to 5 ms, WAF inspection 2 to 5 ms, mesh sidecar 0.5 to 2 ms per hop, rate-limit Redis lookup 1 to 3 ms, body/payload inspection 5 to 50 ms, and in-app rule evaluation in a rule engine under 1 ms cached. The rule of thumb is to stay under about 10 ms of total L7 on hot paths, so every hop must pay for itself." - caption="Figure I.7 — Each L7 stage adds latency; stay under ~10 ms of total L7 on hot paths, so every hop must pay for itself" %} + caption="Figure I.7: Each L7 stage adds latency; stay under ~10 ms of total L7 on hot paths, so every hop must pay for itself" %} -## East / west — L7 between services +## East/west: L7 between services Most L7 decisions in a microservices system are not at the edge; they happen east/west, every time one service calls another. The mesh's sidecar pair handles that traffic with no app code: @@ -612,7 +612,7 @@ internal host: {% include excalidraw.html file="22-east-west" alt="East/west L7 between services. Service A talks to service B through a sidecar pair (Envoy or Linkerd) over mTLS, HTTP/2, and gRPC. What the sidecars do per call with no app code: mTLS, retries with backoff and jitter, outlier detection, locality-aware load balancing, header-based routing and canary weights, rich traces and per-call metrics, circuit breakers, and rate limits." - caption="Figure I.8 — East/west L7: a sidecar pair gives every service-to-service call mTLS, retries, locality, and traces with no app code" %} + caption="Figure I.8: East/west L7: a sidecar pair gives every service-to-service call mTLS, retries, locality, and traces with no app code" %} ```yaml apiVersion: networking.istio.io/v1 @@ -644,13 +644,13 @@ spec: - { destination: { host: payments, subset: v2 }, weight: 5 } ``` -That last block is a canary release of a backend service no external client knows exists — +That last block is a canary release of a backend service no external client knows exists: something you cannot do through an edge load balancer. gRPC east/west has a few sharp -edges worth naming: route on the gRPC URI prefix for per-method canaries; prefer least-request +edges: route on the gRPC URI prefix for per-method canaries; prefer least-request load balancing over round-robin, because gRPC connections are long-lived and round-robining connections (rather than requests) can starve backends; be careful retrying streaming RPCs, since replaying a half-finished stream redoes half the work; and remember gRPC returns its real -status in HTTP trailers, so outlier detection must read the trailer — a `200` with a non-OK +status in HTTP trailers, so outlier detection must read the trailer: a `200` with a non-OK gRPC status is a failure, and a naive check would call a failing service healthy. ## When to use which layer @@ -659,7 +659,7 @@ Each layer has a job it is good at, and pushing work to the wrong one is where c comes from. Edge/ingress for the things every client crosses (TLS, host/path). API gateway for product-shaped policy (auth, rate limits, plans). Mesh for east/west and everything internal. In-app for business rules. The antipattern this prevents is the slow growth of baroque -`VirtualService` configs that encode rules domain experts need to maintain — if it is a +`VirtualService` configs that encode rules domain experts need to maintain: if it is a business rule, it belongs in a rule engine inside a service, not a network CRD. And the organisational failure to avoid is the cluster that ends up running three ingress controllers because three teams each picked one: choose one ingress, one gateway, one mesh, manage routing @@ -667,19 +667,19 @@ through GitOps, watch certificate rotation, and load-test the failure paths. ### Cross-check it yourself -Prove a routing decision twice — once at the mesh, once in the app. For the mesh: apply the +Prove a routing decision twice: once at the mesh, once in the app. For the mesh: apply the canary `VirtualService`/`DestinationRule`, drive a few hundred requests with `hey`, and confirm -the split lands near the configured weight — roughly 10% of responses carrying the `v2` build +the split lands near the configured weight: roughly 10% of responses carrying the `v2` build marker (a response header or version field), the rest `v1`. Bump the weight in Git, re-apply, and watch the ratio move within seconds. Then flip a request's `x-internal` header and confirm -it goes to `v2` every time regardless of weight — header match beats weight. For the in-app +it goes to `v2` every time regardless of weight: header match beats weight. For the in-app router: post one VIP order over the configured threshold and one ordinary order, and confirm from the broker that the VIP order landed on `orders.priority` (and triggered the `notify-vip` -HTTP call) while the ordinary one fell through to `orders.default` — then change only the +HTTP call) while the ordinary one fell through to `orders.default`; then change only the ruleset, not the dispatcher, and confirm the routing changes without a redeploy. Weights that track the config and a ruleset that can move routing without a code change are the two halves of this appendix working. --- -*Verification status: verified — [`examples/22-l7-routing/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/22-l7-routing/) passes 15/15 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified; [`examples/22-l7-routing/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/22-l7-routing/) passes 15/15 checks in each of the six languages on Docker Engine (2026-10-13) (Envoy weighted traffic split, header-based routing override, in-app rule-driven routing, and changing routing rules without a redeploy. Canary promotion and circuit breaking are not covered).* diff --git a/_docs/23-appendix-j-dotnet-toolkit.md b/_docs/23-appendix-j-dotnet-toolkit.md index 10238993..94ad2cfc 100644 --- a/_docs/23-appendix-j-dotnet-toolkit.md +++ b/_docs/23-appendix-j-dotnet-toolkit.md @@ -10,7 +10,7 @@ duration: 18 minutes > **This appendix is .NET-specific.** It exists only in the .NET edition of the source > material; the Spring Boot, Quarkus, Python, and C++ editions skip it. Every other -> chapter in this book is multi-language — this one is the reference a team moving from +> chapter in this book is multi-language: this one is the reference a team moving from > Windows and .NET Framework keeps open in another window while they build. This is the reference appendix: a consolidated bill of materials for the application @@ -18,12 +18,12 @@ side, the same for the platform side, an expanded migration table with a one-lin rationale per row, a look at what .NET lacks, and a short recipe to start a new service. It is built for the .NET-developers-moving-from-Windows audience, and its through-line is that the library choices and the platform choices are entirely -separable — the cluster does not know or care that the Pods happen to run .NET. +separable: the cluster does not know or care that the Pods happen to run .NET. ## The pinned .NET 10 toolkit Every library that appears anywhere in the .NET edition, grouped by concern. The license -column is the one to read closely: every entry is MIT, Apache 2.0, BSD or (for Redis) the AGPLv3 option — no +column is the one to read closely: every entry is MIT, Apache 2.0, BSD or (for Redis) the AGPLv3 option, no commercial-only dependencies, and nothing that requires a specific cloud. | Concern | Library | Purpose | License | NuGet | @@ -41,7 +41,7 @@ commercial-only dependencies, and nothing that requires a specific cloud. | Event store | Marten | Document + event sourcing on Postgres | MIT | Marten | | Messaging | Confluent.Kafka 2.16.0 | Kafka client: producers, consumer groups, explicit offset commits | Apache 2.0 | Confluent.Kafka | | Streaming | Streamiz.Kafka.Net 1.8.2 | .NET port of Kafka Streams | MIT | Streamiz.Kafka.Net | -| Workflow | Temporalio | Temporal .NET SDK — durable execution | MIT | Temporalio | +| Workflow | Temporalio | Temporal .NET SDK: durable execution | MIT | Temporalio | | Workflow | Stateless | In-process state machine library | Apache 2.0 | Stateless | | Rules | NRules | Rete-algorithm rules engine | MIT | NRules.Fluent | | Rules · DSL | NRules.Language | External DSL for non-developer rule editing | MIT | NRules.Language | @@ -58,18 +58,18 @@ commercial-only dependencies, and nothing that requires a specific cloud. | Scheduling | DistributedLock | Distributed mutex over Postgres / Redis / SQL Server | MIT | DistributedLock.* | `Microsoft.AspNetCore.OpenApi` and SignalR are built -into ASP.NET Core 10 — no extra package. **Confluent.Kafka** is the Kafka client the examples use directly: a bus abstraction such as MassTransit would hide the consumer-group, offset-commit, and partition semantics the chapters teach, and MassTransit v9 moved to a commercial license. +into ASP.NET Core 10, no extra package. **Confluent.Kafka** is the Kafka client the examples use directly: a bus abstraction such as MassTransit would hide the consumer-group, offset-commit, and partition semantics the chapters teach, and MassTransit v9 moved to a commercial license. **Streamiz.Kafka.Net** is a .NET port of Kafka Streams, which means windowed aggregations without standing up a Flink cluster. **Temporalio** (the Temporal .NET SDK) pairs with the Temporal Server in the platform stack for durable workflows; **Stateless** fills the lightweight in-process -state-machine slot; **NRules** is the Rete rules engine (a .NET-only library — it has no JVM +state-machine slot; **NRules** is the Rete rules engine (a .NET-only library with no JVM or C++ equivalent). And **Wolverine** is the modern OSS mediator to reach for now that MediatR has moved to a commercial license. ## The pinned cloud-native platform stack The other half of the bill of materials: the cluster-side components the Pods talk *to*. -None of them know the workload is .NET — the same stack serves Spring Boot, Quarkus, or +None of them know the workload is .NET: the same stack serves Spring Boot, Quarkus, or Python identically, which is the entire point. If you already run this platform under a Java or Python application, .NET services drop in alongside without changing anything under the application layer. @@ -103,18 +103,18 @@ stack on which .NET 10 is one workload type. ## From Windows / .NET 4.x to Kubernetes / .NET 10 -This is the expanded version of the bridge from the Cloud-Native Principles chapter — +This is the expanded version of the bridge from the Cloud-Native Principles chapter: twenty-two rows, four columns. Read the rightmost column, "what changes most," because that is where the architectural shift lives. Almost every row preserves the *shape* of the pattern: you still have request handlers, background workers, scheduled jobs, distributed locks, health checks, configuration sources, and telemetry. What changes is who -owns the substrate — IIS owned the host, Kubernetes does now; the substrate becomes +owns the substrate: IIS owned the host, Kubernetes does now; the substrate becomes open-source and cluster-native. {% include excalidraw.html file="23-strangler-fig" - alt="The strangler-fig migration approach. Clients hit one URL through a proxy routing layer — step zero is adding the proxy first. The proxy sends a widening 1% to 100% slice of traffic to a new service (the extracted slice) while everything else defaults to the shrinking monolith. Identify, move, redirect one bounded asset at a time, with the system live throughout; redirect is a proxy config change, not a client release, so every step is reversible." - caption="Figure J.1 — Migrate incrementally with a strangler-fig proxy: redirect a widening slice to the new service while the monolith shrinks, system live throughout" %} + alt="The strangler-fig migration approach. Clients hit one URL through a proxy routing layer: step zero is adding the proxy first. The proxy sends a widening 1% to 100% slice of traffic to a new service (the extracted slice) while everything else defaults to the shrinking monolith. Identify, move, redirect one bounded asset at a time, with the system live throughout; redirect is a proxy config change, not a client release, so every step is reversible." + caption="Figure J.1: Migrate incrementally with a strangler-fig proxy: redirect a widening slice to the new service while the monolith shrinks, system live throughout" %} | Concern | Coming from (Windows / .NET 4.x) | Going to (.NET 10 / Kubernetes) | What changes most | |---|---|---|---| @@ -142,9 +142,9 @@ open-source and cluster-native. | Health probes | ASP.NET app warmup, monitoring agents | M.E.Diagnostics.HealthChecks → /healthz/live + ready | Kubernetes is the monitoring agent | Only two rows are conceptual rewrites rather than substrate swaps. **COM+ / MSDTC -distributed transactions become sagas** — you write compensations explicitly because there +distributed transactions become sagas**: you write compensations explicitly because there is no two-phase commit across services. And **BizTalk orchestration becomes a Temporal -workflow or an NRules routing pipeline** — orchestration is now C# code in your +workflow or an NRules routing pipeline**: orchestration is now C# code in your repository, not a separately-authored artifact in a separate runtime. Everything else is the platform owning more and your code owning less. @@ -159,7 +159,7 @@ in each case there is a workable .NET path or an open-source polyglot path. from the enrichment step. NRules covers the rule-*engine* slot but not the DMN authoring experience. - **No Apache Camel-scale connector ecosystem.** Nothing in .NET matches Camel's 300-plus component connectors. Routing code over plain Kafka producers covers the EIP patterns well, but for - long-tail integration — HL7 over MLLP, AS2, EDI, mainframe gateways — you may still run a + long-tail integration (HL7 over MLLP, AS2, EDI, mainframe gateways), you may still run a Camel sidecar in a JVM Pod (Apache Camel, Apache 2.0) next to the .NET services. - **Thinner stream processing.** Streamiz.Kafka.Net is excellent but is a smaller project than Kafka Streams. If stream processing is central, Flink remains the heavyweight option, @@ -174,8 +174,8 @@ in each case there is a workable .NET path or an open-source polyglot path. the modern MIT-licensed equivalent, and a better design, for new projects. The .NET cloud-native story is solid for well over 95% of line-of-business -and SaaS workloads. The remaining few percent — integration platforms with heavy mainframe, -EDI, or HL7 needs, or central heavy stream processing — is where you mix runtimes or accept a +and SaaS workloads. The remaining few percent (integration platforms with heavy mainframe, +EDI, or HL7 needs, or central heavy stream processing) is where you mix runtimes or accept a different trade-off. ## The five-line recipe for a new service @@ -202,7 +202,7 @@ settle data ownership, then pick a workflow tier, then let the cluster do the he Six libraries get you most of the way: **ASP.NET Core, EF Core, Confluent.Kafka, Temporalio, OpenTelemetry, and Polly.** Add NRules, Streamiz, HotChocolate, or -SignalR per workload — only when the use case earns it. +SignalR per workload, only when the use case earns it. ### Cross-check it yourself @@ -210,7 +210,7 @@ This appendix is reference material, so the check is that the recipe stands up. Scaffold the six-library service and confirm the day-one wiring end to end: `dotnet new web`, add the six packages, map `/healthz/live` and `/healthz/ready`, and start it. Hit both health endpoints with curl and confirm `200`s with the expected probe payloads. Then issue one real -request and confirm a trace appears in Tempo — paste the trace id from the response (or the +request and confirm a trace appears in Tempo: paste the trace id from the response (or the log line) into Grafana and verify the span carries the service name and the `ActivitySource`-based attributes. A new service that answers both probes and emits a correlated trace on its first request has the spine of every pattern in this book working diff --git a/_docs/24-appendix-k-monolith-to-microservices.md b/_docs/24-appendix-k-monolith-to-microservices.md index 864144df..b7c6344b 100644 --- a/_docs/24-appendix-k-monolith-to-microservices.md +++ b/_docs/24-appendix-k-monolith-to-microservices.md @@ -4,7 +4,7 @@ marker: "K" label: "Appendix K" order: 24 part: "Deep-dive appendices" -description: "Decomposing a monolith with the system still running — the modular monolith as a frequent destination, the strangler-fig family (identify, move, redirect), content-based routing for fine-grained cutover, and the decorating collaborator for legacy you can't touch." +description: "Decomposing a monolith with the system still running: the modular monolith as a frequent destination, the strangler-fig family (identify, move, redirect), content-based routing for fine-grained cutover, and the decorating collaborator for legacy you can't touch." duration: 28 minutes --- @@ -20,45 +20,45 @@ coordinating a release. ## A monolith is a unit of deployment -Define the term before fixing it. A monolith is not a code-organisation problem — it is a +Define the term before fixing it. A monolith is not a code-organisation problem: it is a *deployment* property. A clean codebase with clear modules, layers, and tests is still a monolith if it ships as one binary on one heartbeat. That property has real costs: a change anywhere means redeploying everything, a bug in reporting can take down payments, and one scaling unit has to serve different workloads. But the costs do not make monoliths *wrong*. For a new product, a small team, or anything that fits in one team's head, a monolith is often the simplest and fastest path to value, and premature decomposition is its -own anti-pattern. The real question is never "monolith or microservices?" in the abstract — +own anti-pattern. The real question is never "monolith or microservices?" in the abstract: it is "when does *this* monolith start hurting more than it helps?", and that is when these patterns earn their keep. {% include excalidraw.html file="24-monolith-unit" alt="A monolith as a unit of deployment. Web/mobile and other systems call into a single deployable that contains every module (auth, orders, inventory, catalog, payments, shipping, reporting, notifications, admin, pricing, returns, search), all sharing one database that every module reads and writes, giving schema-level coupling everywhere. A change anywhere means redeploying everything, a bug in reporting can take down payments, and one scaling unit serves different workloads. It is often the simplest, fastest path to value when the system fits in one team's head." - caption="Figure K.1 — A monolith is a deployment property: one binary, one heartbeat, one shared database — every module ships and scales together" %} + caption="Figure K.1: A monolith is a deployment property: one binary, one heartbeat, one shared database; every module ships and scales together" %} -## The modular monolith — often the destination +## The modular monolith: often the destination The modular monolith is the intermediate pattern that is frequently the right *destination*, not merely a stepping stone. It is still one deployable, but with strict in-process module boundaries: each module exposes a clear public API (`PlaceOrder`, `ReserveStock`, -`ChargeCard`) and keeps its internals private. The defining discipline is *enforcement* — not +`ChargeCard`) and keeps its internals private. The defining discipline is *enforcement*: not a gentlemen's agreement but build tooling that fails CI when one module reaches into another's internals (import-linter or modulegraph rules in Python; module-system or -build-dependency rules on the JVM and .NET). You get most of the benefit of microservices — -clear boundaries, replaceable modules, independent reasoning — at a fraction of the +build-dependency rules on the JVM and .NET). You get most of the benefit of microservices +(clear boundaries, replaceable modules, independent reasoning) at a fraction of the operational cost: one deployment, one log stream, one debugger session. And if you later need to split, it is a small refactor (rename the public API, move the files, ship) not a rewrite. {% include excalidraw.html file="24-modular-monolith" - alt="A modular monolith, still a single deployable, holding an Orders module (public API PlaceOrder and GetOrder, private internals guarded by the build tool), an Inventory module (public API ReserveStock and CheckAvailability, no shared types), and a Payments module (public API ChargeCard and Refund, its own data schema). They share one database but with separate schemas — orders, inventory, payments — and no cross-schema joins, enforced. Modules talk only through their public API, enforced by linters and build rules rather than a gentlemen's agreement." - caption="Figure K.2 — The modular monolith: one deployable, strict in-process boundaries enforced by build tooling, separate schemas" %} + alt="A modular monolith, still a single deployable, holding an Orders module (public API PlaceOrder and GetOrder, private internals guarded by the build tool), an Inventory module (public API ReserveStock and CheckAvailability, no shared types), and a Payments module (public API ChargeCard and Refund, its own data schema). They share one database but with separate schemas (orders, inventory, payments) and no cross-schema joins, enforced. Modules talk only through their public API, enforced by linters and build rules rather than a gentlemen's agreement." + caption="Figure K.2: The modular monolith: one deployable, strict in-process boundaries enforced by build tooling, separate schemas" %} The hard part of decomposition is the *database*, and the modular monolith lets you do it while still one process. -The hardest coupling to see is in the data — schema dependencies, cross-table joins, and +The hardest coupling to see is in the data: schema dependencies, cross-table joins, and reporting queries that are invisible until you try to move them. So split the data *before* you split the process: give each module its own store, its own connection pool, and its own migration history, and replace cross-module joins with a service call or an event @@ -66,29 +66,29 @@ subscription. That is exactly the discipline microservices demand, but with one target instead of many. Start with the highest-coupling module, and tackle the database the reporting layer joins across early, because it is the second-hardest. Once the data is decomposed, promoting a module to a separate service is a change of deployment topology, not a -data-migration project — and many teams discover at this point that the modular monolith with +data-migration project, and many teams discover at this point that the modular monolith with decomposed data is already enough. {% include excalidraw.html file="24-decomposed-databases" - alt="A modular monolith that is still one process, but with decomposed databases. The Orders, Inventory, and Payments modules each own a separate database: Orders DB on its own Postgres with its own schema and connection pool, Inventory DB on its own Postgres or Mongo, and Payments DB on its own Postgres, encrypted at rest behind a PII boundary. Cross-module data is now async and event-driven — the same discipline as microservices without the operational cost — so promoting one module to a process is a deployment change, not a data-migration project." - caption="Figure K.3 — Decompose the data while still one process: each module gets its own store, so promotion later is a deployment change, not a migration" %} + alt="A modular monolith that is still one process, but with decomposed databases. The Orders, Inventory, and Payments modules each own a separate database: Orders DB on its own Postgres with its own schema and connection pool, Inventory DB on its own Postgres or Mongo, and Payments DB on its own Postgres, encrypted at rest behind a PII boundary. Cross-module data is now async and event-driven (the same discipline as microservices without the operational cost), so promoting one module to a process is a deployment change, not a data-migration project." + caption="Figure K.3: Decompose the data while still one process: each module gets its own store, so promotion later is a deployment change, not a migration" %} ## Identify → move → redirect When you do extract a service, the unit of work is Martin Fowler's strangler-fig pattern, named for the vine that grows around its host tree until the host is gone and the vine stands -on its own. Three steps, repeated per asset. **Identify** a bounded asset — not "the orders +on its own. Three steps, repeated per asset. **Identify** a bounded asset: not "the orders subsystem" but "order creation" or "order status query" specifically; the smaller and more bounded, the safer the move. **Move** by building the new service alongside the monolith, replicating the behaviour, and testing it in parallel without production traffic. **Redirect** -callers to the new service. Each cycle is independently reversible — if the new service is +callers to the new service. Each cycle is independently reversible: if the new service is wrong, redirect back and try again. {% include excalidraw.html file="23-strangler-fig" - alt="Clients hit one URL at a proxy. The proxy routes a widening slice of traffic (1% to 100%) to a new service holding the extracted asset, and everything else to a shrinking monolith. A note says the proxy is step zero, added first, and that redirect is a proxy config change — widen to roll forward, narrow to roll back — so every step is reversible." - caption="Figure K.4 — The strangler-fig cutover: a proxy redirects a widening slice to the new service while the monolith shrinks; every step is reversible" %} + alt="Clients hit one URL at a proxy. The proxy routes a widening slice of traffic (1% to 100%) to a new service holding the extracted asset, and everything else to a shrinking monolith. A note says the proxy is step zero, added first, and that redirect is a proxy config change (widen to roll forward, narrow to roll back), so every step is reversible." + caption="Figure K.4: The strangler-fig cutover: a proxy redirects a widening slice to the new service while the monolith shrinks; every step is reversible" %} ## The proxy is step zero @@ -96,7 +96,7 @@ The precondition for incremental migration is a routing layer, introduced *befor anything. On day zero, clients hit the proxy and the proxy forwards everything to the monolith; behaviour is unchanged. But now you hold the lever: the proxy is where you later route 1% of `/orders/*` to the extracted service, then 5%, then 100%. Without this step, every cutover -needs a coordinated client release; with it, every cutover is a proxy config change — and the +needs a coordinated client release; with it, every cutover is a proxy config change, and the proxy is also the right place for the TLS, auth, rate-limiting, and observability you wanted anyway. The strangler-fig literature calls this *event interception*: you intercept the request before it reaches the legacy system, giving yourself a place to redirect later. @@ -104,26 +104,26 @@ before it reaches the legacy system, giving yourself a place to redirect later. {% include excalidraw.html file="24-proxy-step-zero" alt="Step zero of every migration: introduce a routing layer before extracting anything. Clients hit a proxy or reverse proxy (nginx, Envoy, HAProxy, or YARP) through which all traffic now flows, and the proxy forwards everything to the monolith, which still owns everything and doesn't know about the proxy. New services like an extracted Orders.svc and a future Inventory.svc are soon to come behind the proxy's future routes. There is zero behaviour change for clients on day one, but the lever to route traffic to extracted services is now in place, and the proxy is also where TLS, auth, rate-limiting, and observability consolidate." - caption="Figure K.5 — The proxy is step zero: add the routing layer first, forwarding everything to the monolith, so later cutovers are a config change not a client release" %} + caption="Figure K.5: The proxy is step zero: add the routing layer first, forwarding everything to the monolith, so later cutovers are a config change not a client release" %} -Once the proxy is in place it pays off by *redirecting* a slice — by path +Once the proxy is in place it pays off by *redirecting* a slice: by path (`/orders/*` → new service), by header (`X-Tenant: acme` → new service), by percentage, or by feature flag. The new service handles its slice and the monolith handles the rest, while clients still see one URL, one auth boundary, one TLS cert. Roll forward by widening the new-service -route; roll back by narrowing it — and if the new service fails completely, the monolith picks +route; roll back by narrowing it, and if the new service fails completely, the monolith picks up, which is the entire point. Instrument both paths so you can verify the new service produces the same outcomes as the monolith before you widen. This is the same routing machinery as the L7 -Routing appendix — `nginx` server blocks, Envoy route configs, or an Istio `VirtualService` — +Routing appendix (`nginx` server blocks, Envoy route configs, or an Istio `VirtualService`) used for migration instead of canary release. -## The shared database — a deliberate intermediate state +## The shared database: a deliberate intermediate state Sometimes the data split is too risky to do at the same time as the process split. Then you let the new service and the monolith share the database for a while: both read and write the same tables, and the new service owns the *API* but not the *data*. This is useful for extracting reads first (the new service is a nicer API over a read replica) and as a stepping stone when the data migration is its own project. But it is a transition state, never a -destination — every schema change now needs both apps to agree, and that coordination tax grows +destination: every schema change now needs both apps to agree, and that coordination tax grows with every team that owns part of the same database. Plan its dissolution from the start: the next step is to give the new service its own store, via CDC out of the monolith's database, dual-writing during a brief sync window, or event-sourcing the data across. A shared-database @@ -131,13 +131,13 @@ strangler with no exit plan becomes the new monolith. {% include excalidraw.html file="24-shared-database" - alt="The strangler fig with a shared database as a deliberate intermediate state: process boundary first, data boundary later. Clients hit a proxy that routes to a new Orders service, which reads and writes the same DB and owns the API but not the data, and to the monolith, which still reads and writes the same DB. Both point at a Shared DB whose tables both apps know, with no schema-change isolation. Two apps writing the same tables is a coordination tax — schema changes need both sides to agree — so the next step is to give the new service its own store." - caption="Figure K.6 — The shared-database strangler: split the process first and the data later; a deliberate intermediate state, never a destination" %} + alt="The strangler fig with a shared database as a deliberate intermediate state: process boundary first, data boundary later. Clients hit a proxy that routes to a new Orders service, which reads and writes the same DB and owns the API but not the data, and to the monolith, which still reads and writes the same DB. Both point at a Shared DB whose tables both apps know, with no schema-change isolation. Two apps writing the same tables is a coordination tax (schema changes need both sides to agree), so the next step is to give the new service its own store." + caption="Figure K.6: The shared-database strangler: split the process first and the data later; a deliberate intermediate state, never a destination" %} ## Content-based routing during migration Path-based proxying cuts over a whole endpoint at once. Content-based routing is finer: it reads -the request *body* and decides by payload — tenant, region, feature flag, user segment — so you +the request *body* and decides by payload (tenant, region, feature flag, user segment), so you can cut over one tenant at a time, watch it under real traffic, then widen. The URL is identical across both backends; only the content differs. @@ -342,27 +342,27 @@ func forward(w http.ResponseWriter, r *http.Request, c *http.Client, upstream st The shape is identical everywhere: read the payload, pick an upstream by content rather than URL, and forward. Two implementation notes matter. The body is consumed when you read it, so you -must hand the same bytes through to the upstream — the Python `forward` and the C++ `req->body()` +must hand the same bytes through to the upstream: the Python `forward` and the C++ `req->body()` both forward the buffered body unchanged. And the HTTP client is reused across requests so its connection pool stays warm. The only real cost is the per-request body parse; over a migration measured in days -it is irrelevant next to the risk it controls, but over months it adds up — which is why, in +it is irrelevant next to the risk it controls, but over months it adds up, which is why, in production, the same logic often moves into a sidecar's Lua/WASM filter or an `EnvoyFilter`. ## The decorating collaborator The proxy patterns own *routing*; the decorating collaborator owns *behaviour*. The new service -sits in front of a legacy API and adds cross-cutting capability — caching, audit, validation, -enrichment, fan-out to Kafka — while the legacy service stays unaware it has been wrapped. +sits in front of a legacy API and adds cross-cutting capability (caching, audit, validation, +enrichment, fan-out to Kafka) while the legacy service stays unaware it has been wrapped. Clients talk to the decorator, which delegates the underlying work to the legacy service but contributes everything else. It is a real microservice (its own data, deploy, and scaling), and over time it can absorb more behaviour until the legacy service can be retired without clients -noticing. This is the pattern for legacy you *cannot* modify — a vendor product, a fragile system -touching which is forbidden — and for adding capabilities the original never had. +noticing. This is the pattern for legacy you *cannot* modify (a vendor product, a fragile system +touching which is forbidden) and for adding capabilities the original never had. {% include excalidraw.html file="24-decorating-collaborator" - alt="The decorating collaborator: the new service wraps the old, adds capability, then becomes the API. Clients talk to a decorating service that sits in front of the legacy API and adds caching, audit, validation, rate-limiting, fan-out to Kafka, and metrics, while the legacy service stays unaware it has been wrapped and remains the source of truth. The decorator also emits events to Kafka as a new capability. Clients now talk to the decorator; the legacy service keeps running and is eventually replaced behind the decorator's API — unlike a plain proxy, the decorator owns behaviour, not just routing." - caption="Figure K.7 — The decorating collaborator: a real service wraps the legacy API, adds behaviour, and absorbs it over time — for legacy you cannot modify" %} + alt="The decorating collaborator: the new service wraps the old, adds capability, then becomes the API. Clients talk to a decorating service that sits in front of the legacy API and adds caching, audit, validation, rate-limiting, fan-out to Kafka, and metrics, while the legacy service stays unaware it has been wrapped and remains the source of truth. The decorator also emits events to Kafka as a new capability. Clients now talk to the decorator; the legacy service keeps running and is eventually replaced behind the decorator's API: unlike a plain proxy, the decorator owns behaviour, not just routing." + caption="Figure K.7: The decorating collaborator: a real service wraps the legacy API, adds behaviour, and absorbs it over time, for legacy you cannot modify" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -548,24 +548,24 @@ func (d *Decorator) placeOrder(w http.ResponseWriter, r *http.Request) { } ``` -The two paths tell the whole story. The `GET` adds a Redis cache with a short TTL — a capability -the legacy service never had — falling back to the legacy call on a miss and populating the cache +The two paths tell the whole story. The `GET` adds a Redis cache with a short TTL (a capability +the legacy service never had), falling back to the legacy call on a miss and populating the cache on the way out. The `POST` delegates the write to the legacy service (still the source of truth), *then* emits a Kafka event so downstream systems can subscribe to a new event stream, *then* invalidates the cache. The legacy service has no idea it has been wrapped, and clients have no -idea they are talking to a decorator. Over time you move more behaviour in — validation, -idempotency keys, dedup, multi-source reads — and when you are ready to retire the legacy service +idea they are talking to a decorator. Over time you move more behaviour in (validation, +idempotency keys, dedup, multi-source reads) and when you are ready to retire the legacy service you implement storage inside the decorator (or a clean service it calls) and stop forwarding, while the client-facing API never changes. ## Choosing the right migration pattern -The decision rules, in priority order. A **modular monolith is frequently a fine destination** — +The decision rules, in priority order. A **modular monolith is frequently a fine destination**: microservices have real operational cost, and not every system warrants paying it. **Split the database early**, while still one process, because that is where projects get stuck for years and -where being wrong is cheapest to redo. **No proxy, no incremental migration** — without the +where being wrong is cheapest to redo. **No proxy, no incremental migration**: without the routing layer every cutover is a coordinated client release. A **shared-database strangler is a -means, never an end** — plan its dissolution or it becomes the new monolith. Reach for +means, never an end**: plan its dissolution or it becomes the new monolith. Reach for **content-based routing** when you need fine-grained, tenant-at-a-time progress, and the **decorating collaborator** when you need to add capability to code you cannot change. The architectural take-home is the property named at the start: design every step to be a reversible @@ -578,13 +578,13 @@ Prove the two properties this appendix rests on: reversibility and equivalence. content router in front of a stub "monolith" and a stub "new service" that tag their responses, and drive traffic with `hey`. Send a batch with `tenant: acme` and confirm every one lands on the new service; send the rest and confirm they land on the monolith; then flip the rule (or widen the -percentage) and watch the split move — and flip it *back* and confirm traffic returns to the +percentage) and watch the split move, and flip it *back* and confirm traffic returns to the monolith with no client change. That round-trip is reversibility. For the decorator, hit the `GET` twice and confirm the second response is served from cache (the legacy backend sees one call, not -two — check its access log), then `POST` once and confirm both that the legacy service recorded the +two: check its access log), then `POST` once and confirm both that the legacy service recorded the write *and* that an `order.placed` event landed on Kafka. Same client-visible API, new behaviour -underneath, legacy untouched — that is the decorator doing its job. +underneath, legacy untouched: that is the decorator doing its job. --- -*Verification status: verified — [`examples/24-monolith-to-microservices/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/24-monolith-to-microservices/) passes 19/19 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified. [`examples/24-monolith-to-microservices/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/24-monolith-to-microservices/) passes 19/19 checks in each of the six languages on Docker Engine (2026-10-13) (strangler fig router with per-tenant routing, reversible cutover, a decorating collaborator adding caching and Kafka event emission, and an untouched legacy service).* diff --git a/_docs/25-appendix-l-caching.md b/_docs/25-appendix-l-caching.md index 15236368..d6f517b4 100644 --- a/_docs/25-appendix-l-caching.md +++ b/_docs/25-appendix-l-caching.md @@ -4,13 +4,13 @@ marker: "L" label: "Appendix L" order: 25 part: "Deep-dive appendices" -description: "Six standard caching patterns — cache-aside, read-through, write-through, write-around, write-back, and refresh-ahead — each with its consistency story, its failure mode, and when to reach for it; plus the anti-patterns and a decision guide." +description: "Six standard caching patterns (cache-aside, read-through, write-through, write-around, write-back, and refresh-ahead), each with its consistency story, its failure mode, and when to reach for it; plus the anti-patterns and a decision guide." duration: 24 minutes --- Caching is one of the cheapest big performance wins, and the easiest to get -subtly wrong. The six patterns in this appendix all use the same pieces — an application, a cache -(Redis here), and a database — and differ in exactly two things: *who does the work* and +subtly wrong. The six patterns in this appendix all use the same pieces: an application, a cache +(Redis here), and a database. They differ in exactly two things: *who does the work* and *what consistency story you get*. The discipline is to pick the pattern that matches the consistency you need and the read/write profile you have, not the one that sounds best. Every pattern here also adds an invalidation story and a failure-mode @@ -18,23 +18,23 @@ story, so the appendix ends on the anti-patterns and a decision guide for mixing ## Five things a cache earns you -A cache is worth its complexity for five concrete reasons. **Application performance** — a +A cache is worth its complexity for five concrete reasons. **Application performance**: a cache hit is microseconds where a database round trip is milliseconds, so the hot path gets -roughly 100× faster on hits. **Backend load** — caching shields the database from repeated +roughly 100× faster on hits. **Backend load**: caching shields the database from repeated reads of unchanged data, offloading 80–95% of queries on read-heavy workloads with a sensible -TTL. **Throughput** — the same Pod fleet serves far more requests because it spends less time -waiting on the database. **Predictable performance** — p99 latency stops tracking database +TTL. **Throughput**: the same Pod fleet serves far more requests because it spends less time +waiting on the database. **Predictable performance**: p99 latency stops tracking database load, because the hot path no longer contends on the same lock-bound resource as everyone -else. And **database cost** — fewer queries means smaller or fewer DB instances, a smaller +else. And **database cost**: fewer queries means smaller or fewer DB instances, a smaller IOPS budget, and less replication bandwidth; the cache tier is almost always cheaper than the DB tier it offloads. The trade-off, in one line: every cache adds a consistency story, an -invalidation story, and a failure-mode story — so pick the pattern that matches what you need. +invalidation story, and a failure-mode story, so pick the pattern that matches what you need. -## Cache-aside (look-aside) — the default +## Cache-aside (look-aside): the default Cache-aside is where everyone starts and the right choice for most reads: the application is in charge. Try the cache; on a miss, read the database, populate the cache on the way back, -and return. It is explicit, debuggable, and has no magic — the cache is just a key-value store +and return. It is explicit, debuggable, and has no magic: the cache is just a key-value store and the app decides what to cache and when. The costs are that every read carries the if/else, a miss costs two round trips, and the logic repeats across services unless you extract it. The TTL is the consistency knob: longer means fewer DB hits and more stale-read risk. The one @@ -43,7 +43,7 @@ discipline that keeps it correct is invalidating on writes. {% include excalidraw.html file="24-cache-aside" alt="A client calls the application, which reads the cache (GET), falls back to the database with a SELECT on a miss, then populates the cache with SETEX before returning. The application coordinates every step." - caption="Figure L.1 — Cache-aside: the application coordinates; a miss costs two round trips" %} + caption="Figure L.1: Cache-aside: the application coordinates; a miss costs two round trips" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -238,7 +238,7 @@ func (s *Server) updateProduct(w http.ResponseWriter, r *http.Request) { } ``` -## Read-through — the cache fetches on miss +## Read-through: the cache fetches on miss Read-through moves the database call from the application into the cache layer: the app calls `cache.get()` and gets the value back, not knowing or caring whether the cache had it or had to @@ -251,7 +251,7 @@ declaratively (Spring's `@Cacheable`, Quarkus's `@CacheResult` and `Cache.get`, {% include excalidraw.html file="24-read-through" alt="A client calls the application, which calls cache.get on a single interface. On a miss the cache itself runs a loader against the database. The application never sees the database directly." - caption="Figure L.2 — Read-through: the cache library owns the miss; the app sees one interface" %} + caption="Figure L.2: Read-through: the cache library owns the miss; the app sees one interface" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -421,12 +421,12 @@ var products = ReadThrough[Product]{ } ``` -## Write-through — cache and DB written synchronously +## Write-through: cache and DB written synchronously When stale reads after a write are not acceptable, write-through updates both stores on the write path: write the database, then write the cache, before the response returns. A read immediately after a write sees the new value, and the cache is never stale for the writer. The costs are that -write latency is the sum of both stores, and there is no two-phase commit — so the discipline is to +write latency is the sum of both stores, and there is no two-phase commit, so the discipline is to write the DB first (the source of truth), write the cache second with a `SET` rather than an invalidate, and pair it with a short TTL so any window of inconsistency self-heals. The platform frameworks express it as `@CachePut` (Spring) or a direct `SetAsync` (FusionCache). @@ -434,7 +434,7 @@ frameworks express it as `@CachePut` (Spring) or a direct `SetAsync` (FusionCach {% include excalidraw.html file="24-write-through" alt="A client sends a write to the application, which writes the database first as the source of truth, then writes (SET) the cache second, before responding. Both stores are updated on the write path." - caption="Figure L.3 — Write-through: DB first, cache second (SET, not invalidate); never stale, pays both latencies" %} + caption="Figure L.3: Write-through: DB first, cache second (SET, not invalidate); never stale, pays both latencies" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -565,9 +565,9 @@ func (s *Server) updateProduct(w http.ResponseWriter, r *http.Request) { } ``` -## Write-around — writes skip the cache +## Write-around: writes skip the cache -When data is written far more often than it is read — audit logs, event records, telemetry — caching +When data is written far more often than it is read (audit logs, event records, telemetry), caching every write is wasteful, because most entries expire before anyone reads them. Write-around writes the database and leaves the cache alone; the next reader pays the miss and populates it. The cache then holds only things being read, with no pollution by write-only data. The cost is that @@ -578,7 +578,7 @@ workload. {% include excalidraw.html file="24-write-around" alt="A client sends a write to the application, which writes only the database; the cache is untouched. A later read populates the cache lazily." - caption="Figure L.4 — Write-around: writes go to the DB only; the next read populates the cache" %} + caption="Figure L.4: Write-around: writes go to the DB only; the next read populates the cache" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -734,16 +734,16 @@ func (s *Server) getEvent(w http.ResponseWriter, r *http.Request) { } ``` -## Write-back (write-behind) — fast writes, eventual durability +## Write-back (write-behind): fast writes, eventual durability Write-back is the fastest-write pattern and the one with the most caveats. The client writes to the -cache and gets an immediate ack — the cache is the source of truth for that window — while a +cache and gets an immediate ack (the cache is the source of truth for that window) while a background flusher drains the cache to the database, batching writes for efficiency. Write latency equals cache latency (microseconds), which is enormous throughput on write-heavy workloads. The cost is durability: a cache crash before the flusher catches up loses unflushed writes, so the cache must be configured for persistence (Redis AOF or RDB) and you must have thought about what "lost the last few seconds of writes" means for the business. Reserve it for cases where write throughput is the -constraint *and* eventual durability is acceptable — real-time game state, telemetry with a +constraint *and* eventual durability is acceptable: real-time game state, telemetry with a downstream aggregator, write-heavy buffering ahead of a slower store. The API reflects the contract: a write answers `202 Accepted` because the value is queued, not yet @@ -754,7 +754,7 @@ value still in the cache, so an outage never drops a write silently. {% include excalidraw.html file="24-write-back" alt="A client writes to the application, which writes to the cache and acks immediately. A background flusher drains dirty keys from the cache to the database asynchronously in batches." - caption="Figure L.5 — Write-back: ack at cache speed; a background flusher persists later — a crash before flush loses unpersisted writes" %} + caption="Figure L.5: Write-back: ack at cache speed; a background flusher persists later; a crash before flush loses unpersisted writes" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -923,20 +923,20 @@ func flusher(ctx context.Context) { // 2. drain the dirty set to the D } ``` -## Refresh-ahead — keep hot keys warm +## Refresh-ahead: keep hot keys warm Plain cache-aside has a latency cliff: while a key is cached reads are fast, but the instant its TTL expires the next reader pays the slow DB fetch. For homepage data, top products, or frequently-queried entities that spike is user-visible. Refresh-ahead pre-empts it: a background task tracks a set of hot keys and refreshes their values from the database *before* the TTL fires, so users always read warm cache. The cost is extra DB load for keys that might not be requested again, plus the need to identify -which keys are hot — typically by touching a Redis sorted set on every read. Use it where miss latency +which keys are hot: typically by touching a Redis sorted set on every read. Use it where miss latency on a small hot set matters more than DB throughput. {% include excalidraw.html file="24-refresh-ahead" alt="A client reads through the application from a cache that is always warm. A background refresher tracks hot keys and fetches them from the database before their TTL expires, refreshing the cache so users never hit a cold miss." - caption="Figure L.6 — Refresh-ahead: a background task re-warms hot keys before TTL, eliminating user-visible cold misses" %} + caption="Figure L.6: Refresh-ahead: a background task re-warms hot keys before TTL, eliminating user-visible cold misses" %} {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -1178,7 +1178,7 @@ func (s *Server) refresher(ctx context.Context) { ## When not to cache, and what to watch for -Caching is a complexity tax, only worth paying when reads hurt — the cheapest cache is no +Caching is a complexity tax, only worth paying when reads hurt: the cheapest cache is no cache, so if reads are already fast, don't add one. The failure modes to design against: - **Don't cache rarely-read data.** Caching write-once-read-never data thrashes the cache and gains @@ -1192,38 +1192,38 @@ cache, so if reads are already fast, don't add one. The failure modes to design lock in the loader (only one fetches, the rest wait), with refresh-ahead, or by jittering TTLs slightly per entry so similar keys don't expire together. - **Watch cache-aside or write-around without invalidation.** The write goes to the DB but a stale - cached value can persist for the full TTL — the most common stale-data bug in production. Invalidate + cached value can persist for the full TTL: the most common stale-data bug in production. Invalidate on every write, or use a shorter TTL. - **Never treat the cache as the source of truth.** A cache outage must mean slower reads, not data - loss, so always have a DB fallback path — and test it by stopping Redis in staging. + loss, so always have a DB fallback path, and test it by stopping Redis in staging. ## Choosing the right caching pattern The patterns are not mutually exclusive; matching them to read/write profiles is the engineering work. -**Cache-aside** is the default for most reads — explicit control, reads dominating writes. +**Cache-aside** is the default for most reads: explicit control, reads dominating writes. **Read-through** is the same semantics with cleaner code when you have a loader-capable library. **Write-through** trades write latency for zero stale-read risk, and fits read-heavy data with infrequent but consistency-critical writes (financial, healthcare, regulatory). **Write-around** suits -append-heavy workloads — telemetry, events, logs — where re-reads are rare. **Write-back** is the niche +append-heavy workloads (telemetry, events, logs) where re-reads are rare. **Write-back** is the niche extreme-write-throughput pattern, valid only when eventual durability is acceptable and the cache is configured for persistence. **Refresh-ahead** is the hot-key pattern, eliminating user-visible misses on a small identified hot set at the cost of extra DB load on it. Most production systems run cache-aside for the bulk of traffic, layer refresh-ahead on the hot keys they have identified, and use write-through -only where stale reads are forbidden — a mix, not a single rule. +only where stale reads are forbidden: a mix, not a single rule. ### Cross-check it yourself Prove the two properties that matter: the cache offloads the database, and a cache outage degrades rather than breaks. Put read load on the cache-aside endpoint with `hey` and watch the database's query counter (Postgres `pg_stat_statements`, or the query log): after the first miss per -key the DB call count should flatten while throughput climbs, and p99 should drop sharply — that is the +key the DB call count should flatten while throughput climbs, and p99 should drop sharply: that is the offload working. Then, with load still running, **stop Redis**. The correct behaviour is that latency -rises (every read becomes a DB read) but requests still return `200` — if you instead see `5xx`, the +rises (every read becomes a DB read) but requests still return `200`; if you instead see `5xx`, the cache is wired as a source of truth rather than an optimisation, which is the one caching mistake you cannot ship. Finally, for write-through, write a key and immediately read it back: the read must return the new value with no DB round trip; for write-around, write then read and confirm the first read is a miss (the DB log shows the `SELECT`) while the second is a hit. --- -*Verification status: verified — [`examples/25-caching/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/25-caching/) passes 29/29 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified; [`examples/25-caching/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/25-caching/) passes 29/29 checks in each of the six languages on Docker Engine (2026-10-13) (cache-aside with invalidation on update, read-through, write-through, write-around, write-back with 202 and a flush retry across a Postgres outage, refresh-ahead, hot-set tracking, and degradation with Redis stopped and restarted).* diff --git a/_docs/26-appendix-m-failure-modes.md b/_docs/26-appendix-m-failure-modes.md index 4c5148c9..8f1dcb1c 100644 --- a/_docs/26-appendix-m-failure-modes.md +++ b/_docs/26-appendix-m-failure-modes.md @@ -4,7 +4,7 @@ marker: "M" label: "Appendix M" order: 26 part: "Deep-dive appendices" -description: "The synthesis appendix — a taxonomy of how distributed systems fail (partition, split-brain, partial, gray, amplification, cascading) and the defensive toolkit mapped to each: timeouts and deadlines, retry with backoff and jitter, circuit breakers, bulkheads, load shedding, quorum, and the mesh." +description: "The synthesis appendix: a taxonomy of how distributed systems fail (partition, split-brain, partial, gray, amplification, cascading) and the defensive toolkit mapped to each: timeouts and deadlines, retry with backoff and jitter, circuit breakers, bulkheads, load shedding, quorum, and the mesh." duration: 24 minutes --- @@ -12,15 +12,15 @@ This is the synthesis appendix. It pulls together error handling, graceful shutd observability, and the resilience patterns scattered across the book and points them at the specific ways distributed systems fail. If you keep one thing from it, keep the matrix at the end, which maps each failure mode to the patterns that defend it. The reframe that -organises everything: we are not trying to *prevent* failure — across a network that is -impossible — we are trying to *bound* it and recover. That shifts every question from "how +organises everything: we are not trying to *prevent* failure (across a network that is +impossible), we are trying to *bound* it and recover. That shifts every question from "how do we stop failures" to "how do we contain and survive them." ## Distributed systems fail differently The single most important conceptual shift for a developer moving from monoliths is **partial failure**. In a single process, a function call returns or throws, synchronously, -in-process. Across a network there is a whole spectrum in between — including the two worst +in-process. Across a network there is a whole spectrum in between, including the two worst cases: the call that *hangs* indefinitely, and the call that *succeeded on the server but whose response was lost*, leaving the client unsure whether retrying is safe. Two further properties make distributed failure hard: failures are **correlated** (one failure causes @@ -35,92 +35,92 @@ underneath them. {% include excalidraw.html file="25-failure-taxonomy" alt="Three families feed into a distributed system: NETWORK (partition, split-brain: alive but cannot talk), COMPONENT (partial, gray failure: some fail, some lie), and LOAD (amplification, cascading: a failure feeds on itself). The defensive toolkit answers them: timeouts, retries, circuit breakers, bulkheads, load shedding, fallback, quorum, Istio, graceful shutdown, observability." - caption="Figure M.1 — Three families of distributed failure, over the defensive toolkit that addresses them" %} + caption="Figure M.1: Three families of distributed failure, over the defensive toolkit that addresses them" %} -**Network** failures are partition and split-brain — nodes are alive but cannot talk. -**Component** failures are partial and gray — some dependencies fail outright, some degrade +**Network** failures are partition and split-brain: nodes are alive but cannot talk. +**Component** failures are partial and gray: some dependencies fail outright, some degrade and are perceived differently by different observers. **Load** failures are amplification and -cascading — a failure feeds on itself or spreads to neighbours. It restates the maxim from the -first chapter: all management is risk management — you manage the *distribution* of failures, +cascading: a failure feeds on itself or spreads to neighbours. It restates the maxim from the +first chapter: all management is risk management; you manage the *distribution* of failures, not their existence. ## Network partition A partition is **not a crash**. Every node is healthy and running; the network between them is -what failed — which is exactly what makes partitions so confusing to debug, since each +what failed, which is exactly what makes partitions so confusing to debug, since each component looks fine in isolation. {% include excalidraw.html file="25-network-partition" alt="Three nodes on the left form a majority partition A; two nodes on the right form a minority partition B; the link between them is down. Note: every node is healthy and only the network failed, and CAP forces a choice between consistency and availability while partitioned." - caption="Figure M.2 — A network partition: healthy nodes that cannot reach each other" %} + caption="Figure M.2: A network partition: healthy nodes that cannot reach each other" %} This is where CAP becomes concrete: while partitioned you must choose to stay **consistent** (the minority side refuses writes, sacrificing availability) or **available** (both sides accept writes, sacrificing consistency and reconciling later). Kubernetes itself uses etcd, -which chooses consistency — the minority side of a control-plane partition stops accepting +which chooses consistency: the minority side of a control-plane partition stops accepting writes. Address it with quorum/consensus, idempotent writes (so a retried write after the heal is safe), and a design that tolerates partitions and reconciles. ## Split-brain Split-brain is the catastrophic outcome of a partition in a system that elects a leader. Both -sides lose contact, each assumes the other died, each promotes itself — and now two primaries +sides lose contact, each assumes the other died, each promotes itself, and now two primaries accept conflicting writes, leaving divergent state with no clean merge on heal. {% include excalidraw.html file="25-split-brain" alt="A partition where each side has self-elected a leader that accepts writes, producing two primaries with conflicting writes. The cure noted is quorum: an action needs a majority, so a minority partition cannot act." - caption="Figure M.3 — Split-brain: a partition elects two leaders; quorum is the only real cure" %} + caption="Figure M.3: Split-brain: a partition elects two leaders; quorum is the only real cure" %} The prevention is **quorum**: an action requires a majority (N/2 + 1) to agree, so a minority -partition mathematically cannot act — it knows it is the minority and steps down. This is why +partition mathematically cannot act: it knows it is the minority and steps down. This is why cluster sizes are odd (3, 5, 7): to always have a clear majority. Fencing tokens (monotonically increasing IDs that let a resource reject a stale leader's writes) and leases are complementary, and etcd/Raft give the Kubernetes control plane this property out of the -box. As the matrix will show, quorum is the *only* answer to split-brain — no amount of +box. As the matrix will show, quorum is the *only* answer to split-brain: no amount of retrying or circuit-breaking helps when the problem is two leaders. ## Partial failure This is the slide for developers moving from monoliths. In-process, the order and inventory -services are function calls in one process — if one is "down," the process is down. +services are function calls in one process: if one is "down," the process is down. Distributed, the order service calls inventory (healthy), payment (down), and shipping (slow), and must make a **per-dependency decision** for each. {% include excalidraw.html file="25-partial-failure" alt="An order service makes one decision per dependency: use the result from a healthy inventory service, fall back or fail for a down payment service, and time out then fall back for a slow shipping service." - caption="Figure M.4 — Partial failure: each dependency needs its own use / fall-back / time-out decision" %} + caption="Figure M.4: Partial failure: each dependency needs its own use / fall-back / time-out decision" %} -The mistake behind most production incidents is treating a remote call like a local one — no -timeout, no fallback, no circuit breaker — so a single down dependency hangs the caller. The +The mistake behind most production incidents is treating a remote call like a local one (no +timeout, no fallback, no circuit breaker), so a single down dependency hangs the caller. The next several patterns are precisely the per-dependency defences: timeouts, circuit breakers, fallbacks, and bulkheads. ## Gray failure Gray failure is the mode that defeats naive health checking. The classic shape: a node is slow -or dropping a fraction of requests, but its liveness probe — a trivial `GET /healthz` — still +or dropping a fraction of requests, but its liveness probe (a trivial `GET /healthz`) still returns `200`, so Kubernetes keeps it in rotation and the load balancer keeps sending it traffic. {% include excalidraw.html file="25-gray-failure" - alt="A pod that is slow or dropping 5% of requests has a liveness probe that still returns 200, so the probe sees it as healthy, while real client traffic sees errors and latency — differential observability." - caption="Figure M.5 — Gray failure: the probe sees healthy, the client sees errors" %} + alt="A pod that is slow or dropping 5% of requests has a liveness probe that still returns 200, so the probe sees it as healthy, while real client traffic sees errors and latency: differential observability." + caption="Figure M.5: Gray failure: the probe sees healthy, the client sees errors" %} -This is **differential observability** — the observer that *decides* (the probe) sees something +This is **differential observability**: the observer that *decides* (the probe) sees something different from the observer that *matters* (the client). The fixes are all about observing from the right vantage point: deep health checks that exercise real dependencies, RED metrics -measured on actual traffic, SLO burn-rate alerts, and — critically — Istio outlier detection, +measured on actual traffic, SLO burn-rate alerts, and, critically, Istio outlier detection, which ejects a host from the pool when it returns consecutive errors, regardless of what the host's own probe claims. ## Amplification loops and retry storms -Amplification is a positive feedback loop in which the system's own defensive behaviour — -retrying — becomes the attack. Two effects compound: retries **multiply** across layers (a +Amplification is a positive feedback loop in which the system's own defensive behaviour, +retrying, becomes the attack. Two effects compound: retries **multiply** across layers (a gateway that retries 3×, calling a service that retries 3×, calling another that retries 3×, turns one request into 27 at the bottom), and they **synchronise** (every client that failed at the same moment retries at the same moment). @@ -128,9 +128,9 @@ the same moment retries at the same moment). {% include excalidraw.html file="25-amplification" alt="One client request hits a gateway that retries three times, then a service that retries three times, then a service that retries three times, producing 27 requests at the bottom of the stack." - caption="Figure M.6 — Amplification: retries multiply across layers and synchronise into a thundering herd" %} + caption="Figure M.6: Amplification: retries multiply across layers and synchronise into a thundering herd" %} -The result can be **metastable** — the system stays collapsed even after the original trigger is +The result can be **metastable**: the system stays collapsed even after the original trigger is gone, because the retry load alone sustains the failure. Breaking it requires backoff with jitter (de-synchronise), a retry budget (cap retries to a small percentage of live traffic), retrying at exactly *one* layer, and a circuit breaker. This is the same dynamic as the @@ -146,8 +146,8 @@ the user-facing edge. {% include excalidraw.html file="25-cascading" - alt="A slow database blocks Service C's threads, which exhausts its resources so it fails for all callers, which fills Service B's pool, which takes down the edge — the failure propagates up the graph." - caption="Figure M.7 — Cascading failure: one slow dependency exhausts resources up the whole call graph" %} + alt="A slow database blocks Service C's threads, which exhausts its resources so it fails for all callers, which fills Service B's pool, which takes down the edge: the failure propagates up the graph." + caption="Figure M.7: Cascading failure: one slow dependency exhausts resources up the whole call graph" %} The amplifiers are synchronous call chains and unbounded resources (unbounded queues, unbounded thread pools, no timeouts). Each defensive pattern breaks one link: timeouts stop threads @@ -157,7 +157,7 @@ edge before it enters the system. ## The defensive toolkit -**No pattern in the toolkit is sufficient alone** — several interact, and production +**No pattern in the toolkit is sufficient alone**: several interact, and production resilience is their composition. Retry without a circuit breaker amplifies; a circuit breaker without a fallback just fails faster; a bulkhead without a timeout still lets threads block within its pool. The patterns here each get a diagram and per-language code; the platform-level pieces (Istio, @@ -165,7 +165,7 @@ shutdown, observability) sit underneath the application-level ones. ## Timeouts and deadline propagation -Two ideas. First, **every remote call must have a timeout** — the default in many HTTP clients +Two ideas. First, **every remote call must have a timeout**: the default in many HTTP clients is infinite, which is exactly how one slow dependency hangs a whole service. Second, **deadline propagation**: the original deadline travels with the request, and each hop subtracts the time it used and passes the remainder downstream; when too little budget remains to plausibly finish, @@ -174,7 +174,7 @@ the downstream fails immediately rather than starting work that will be thrown a {% include excalidraw.html file="25-timeouts-deadline" alt="A 1000ms deadline at the edge travels through Service A (uses 200ms, 800 left), Service B (uses 300ms, 500 left), to the database, which decides whether enough budget remains to start." - caption="Figure M.8 — Deadline propagation: the budget travels and shrinks; too little left means fail now" %} + caption="Figure M.8: Deadline propagation: the budget travels and shrinks; too little left means fail now" %} gRPC does this natively with deadlines; for HTTP you carry the remaining budget in a header convention like `X-Deadline-Ms` and honour it at each hop. @@ -325,19 +325,19 @@ code, body, err := callBackend(ctx, map[string]string{"X-Deadline-Ms": strconv.I ## Retry with exponential backoff and jitter -Retry is the most abused resilience pattern, because the naive version — retry immediately, a -fixed number of times, at every layer — is exactly how the amplification storm forms. The +Retry is the most abused resilience pattern, because the naive version (retry immediately, a +fixed number of times, at every layer) is exactly how the amplification storm forms. The correct version has four properties: **exponential backoff** (100ms, 200ms, 400ms), **jitter** (randomness so clients don't synchronise), a **cap** on both attempts and total time, and a **retry budget** (retries may never exceed, say, 10% of live traffic). {% include excalidraw.html file="25-retry-backoff" - alt="Three attempts with growing, jittered waits between them — about 100ms plus jitter, then about 200ms plus jitter — capped at three attempts." - caption="Figure M.9 — Retry done right: exponential backoff, jitter, a hard cap, idempotent calls only" %} + alt="Three attempts with growing, jittered waits between them (about 100ms plus jitter, then about 200ms plus jitter), capped at three attempts." + caption="Figure M.9: Retry done right: exponential backoff, jitter, a hard cap, idempotent calls only" %} -It applies to **idempotent operations only** — retrying a non-idempotent `POST` without an -idempotency key can double-charge a customer — and it is **always paired with a circuit +It applies to **idempotent operations only**: retrying a non-idempotent `POST` without an +idempotency key can double-charge a customer. It is also **always paired with a circuit breaker**, so once a dependency is clearly down you stop retrying entirely. {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -475,7 +475,7 @@ allows a few trial calls; if they succeed it closes, if they fail it re-opens. {% include excalidraw.html file="25-circuit-breaker" alt="A three-state machine: CLOSED (calls pass, count failures) trips to OPEN (fail fast, don't call) when failures exceed the threshold; after a reset timeout OPEN moves to HALF-OPEN (a few trial calls); a successful trial returns to CLOSED, a failed trial returns to OPEN." - caption="Figure M.10 — The circuit breaker: closed, open, half-open — one breaker per dependency, paired with a fallback" %} + caption="Figure M.10: The circuit breaker: closed, open, half-open; one breaker per dependency, paired with a fallback" %} Two details are crucial: **one breaker per dependency** (never a single global breaker), and **pair it with a fallback** so an open breaker degrades gracefully rather than just erroring @@ -640,13 +640,13 @@ func handleBreaker(w http.ResponseWriter, r *http.Request) { A bulkhead bounds concurrency so one saturated dependency cannot consume all of a service's threads or connections. It got its full per-language treatment in the Security chapter, so the point to re-state here is the interaction: **a bulkhead bounds how many, a timeout bounds how -long** — concurrency-limiting alone does not help if the bounded calls block forever, so the two +long**: concurrency-limiting alone does not help if the bounded calls block forever, so the two must be combined. The mesh provides bulkheading at the connection-pool level, covered next in this appendix. ## Load shedding and graceful degradation -Two related patterns engineers tend to skip. **Load shedding** feels like giving up — -choosing to reject requests — but it is the difference between degrading and collapsing. The +Two related patterns engineers tend to skip. **Load shedding** feels like giving up +(choosing to reject requests), but it is the difference between degrading and collapsing. The arithmetic is simple: at 120% of capacity, accepting everything makes *everything* slow and timing out (0% goodput); shedding the excess 20% means the 80% you accept is served well. Shed by priority (drop health-check-style or low-value traffic first, protect revenue-critical paths), @@ -655,12 +655,12 @@ and let adaptive concurrency limiters shed automatically based on observed laten **Graceful degradation** is the product-level decision about what to return when you cannot return everything: serve a stale cache, omit the personalised section, queue the write for later, show a default recommendation. A product page that can't reach the reviews service should still -show the product — just without reviews. Together with the breaker these form a system: the +show the product, just without reviews. Together with the breaker these form a system: the breaker decides to *stop calling*, the fallback decides *what to return instead*, and load -shedding decides *what not to accept* in the first place — and none of them can be done +shedding decides *what not to accept* in the first place, and none of them can be done intelligently without the observability from the Observability chapter to measure saturation. -## Istio — resilience at the mesh layer +## Istio: resilience at the mesh layer The service mesh delivers a large slice of this toolkit without code in every service, in every language. @@ -668,11 +668,11 @@ language. {% include excalidraw.html file="25-istio-resilience" alt="An Istio Envoy sidecar wraps service-to-service calls: a VirtualService configures timeout and retries with backoff and jitter; a DestinationRule configures connectionPool (mesh bulkhead) and outlierDetection, which ejects a bad host from the callee pool." - caption="Figure M.11 — The mesh supplies timeouts, retries, connection-pool bulkheads, and gray-failure ejection" %} + caption="Figure M.11: The mesh supplies timeouts, retries, connection-pool bulkheads, and gray-failure ejection" %} A `VirtualService` configures per-route timeouts and retries (Envoy adds backoff and jitter automatically); a `DestinationRule` configures `connectionPool` limits (mesh-level bulkheading) -and `outlierDetection` — the mesh's answer to gray failure, ejecting a host that returns +and `outlierDetection`: the mesh's answer to gray failure, ejecting a host that returns consecutive errors regardless of what its own probe claims. ```yaml @@ -706,7 +706,7 @@ spec: ``` The discipline that prevents amplification: **don't double up**. If Istio is retrying, do not -*also* retry in the app — decide where each concern lives. Push uniform concerns (timeouts, +*also* retry in the app: decide where each concern lives. Push uniform concerns (timeouts, outlier detection, mesh bulkheads) to Istio; keep business-aware concerns (fallbacks, idempotency keys) in the app. @@ -716,10 +716,10 @@ Two threads from elsewhere in the book close the loop. **Graceful shutdown** (th Shutdown appendix) is what prevents dropped requests during a rolling deploy: on `SIGTERM`, flip readiness to draining so the platform stops sending new work, cover the deregistration lag, drain in-flight work under a bounded timeout, then exit. **Observing failure** (the Observability -chapter) is what makes these modes visible at all — a default CPU/memory dashboard will not show +chapter) is what makes these modes visible at all: a default CPU/memory dashboard will not show a gray failure or an amplification loop. Measure RED per *dependency* (so partial failures are visible), compare health-check status to real-traffic SLOs (so gray failures are visible), and -watch request-rate versus user-action-rate (so amplification is visible) — then pivot from the +watch request-rate versus user-action-rate (so amplification is visible), then pivot from the metric to the trace to the logs by trace id when something breaks. ## Which pattern addresses which failure mode @@ -736,10 +736,10 @@ to see what a pattern buys you across modes. | Amplification | | ✓ | ✓ | | ✓ | | | | | Cascading | ✓ | | ✓ | ✓ | ✓ | | | ✓ | -Two observations to state aloud. **Quorum/consensus is the only real answer to split-brain** — no +Two observations to state aloud. **Quorum/consensus is the only real answer to split-brain**: no amount of retrying or breaking helps when the problem is two leaders; you need majority rule. And almost every other cell is about assembling a **portfolio**: there is no single pattern that -handles partial, gray, amplification, and cascading failures — you layer timeouts, +handles partial, gray, amplification, and cascading failures: you layer timeouts, retries-with-budget, breakers, bulkheads, shedding, and fallbacks, and you observe the result. ## Take-aways and references @@ -760,7 +760,7 @@ throttling), the Istio traffic-management docs, and the metastable-failures rese Reproduce the two dynamics that this appendix turns on. **Cascading and the breaker:** put steady load on a service with `hey` (or `ghz` for gRPC) while you force its downstream dependency to be -slow, and watch the downstream call count — without a breaker it climbs and the caller's threads +slow, and watch the downstream call count: without a breaker it climbs and the caller's threads fill (cascading); with a per-dependency breaker configured, the count *plateaus* as the breaker trips and the caller returns its fallback fast. **Deadline propagation:** issue a request with a small deadline header to a chain of two services and confirm from the logs that the *second* hop @@ -772,5 +772,5 @@ This appendix touches difficult operational territory; if you are diagnosing a l Google SRE cascading-failures chapter is the fastest way to a shared vocabulary with your team. --- -*Verification status: verified — [`examples/26-failure-modes/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/26-failure-modes/) passes 22/22 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified; [`examples/26-failure-modes/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/26-failure-modes/) passes 22/22 checks in each of the six languages on Docker Engine (2026-10-13) (timeout, retry exhaustion, circuit breaker closed, open and recovered with fallback, deadline-budget propagation, bulkhead limit, a full bulkhead refusing the overflow with 429 and admitting again).* diff --git a/_docs/27-appendix-n-feature-flags.md b/_docs/27-appendix-n-feature-flags.md index f06975e2..b77099fb 100644 --- a/_docs/27-appendix-n-feature-flags.md +++ b/_docs/27-appendix-n-feature-flags.md @@ -9,8 +9,8 @@ duration: 22 minutes --- A feature flag is the runtime switch that separates *shipping code* from *releasing a -feature*. Once an artifact is in production, a flag decides — per request, per user, per -cohort — whether the new path is taken. That one decoupling is the spine of progressive +feature*. Once an artifact is in production, a flag decides (per request, per user, per +cohort) whether the new path is taken. That one decoupling is the spine of progressive delivery: you deploy continuously and dark, then release gradually and reversibly, with no redeploy on the critical path of either the rollout or the rollback. This appendix covers the flag types, how to evaluate flags in a vendor-neutral, open-source way with OpenFeature and @@ -25,9 +25,9 @@ are different events**, and a flag is what splits them. {% include excalidraw.html file="26-deploy-release" alt="A pipeline: build and deploy with the flag off, then release to 1% with the flag on, then ramp the percentage, then GA at 100%, then retire the flag. The deploy is decoupled from the release; every release and rollback is a config change, not a new deploy." - caption="Figure N.1 — Deploy ships the code dark; release is a runtime flag flip — and so is rollback" %} + caption="Figure N.1: Deploy ships the code dark; release is a runtime flag flip, and so is rollback" %} -The artifact ships to production with the new path turned off — a non-event, observable only in +The artifact ships to production with the new path turned off, a non-event, observable only in the deploy log. Releasing is then a runtime flag flip: on for 1%, then a wider cohort, then everyone, each step reversible by narrowing the flag rather than rolling back a deployment. This is what lets a team merge to main many times a day yet release on its own cadence, and it is the @@ -36,12 +36,12 @@ mechanism behind every canary and every safe cutover in the book. ## Four kinds of flag "Feature flag" covers four different tools, and conflating them is the root of most -flag messes. They differ along two axes — how long the flag lives, and who decides to flip it. +flag messes. They differ along two axes: how long the flag lives, and who decides to flip it. {% include excalidraw.html file="26-flag-types" alt="A two-by-two of flag types by lifetime and owner: release flags (short-lived, dev-owned, ship dark and remove); experiment/A-B flags (short-lived, data-owned, split variants and measure); ops/kill-switch flags (long-lived, SRE-owned, disable a feature or dependency fast); permission/entitlement flags (long-lived, product-owned, gate by plan or tier)." - caption="Figure N.2 — Four flag types by lifetime and owner — each wants different defaults and lifecycle" %} + caption="Figure N.2: Four flag types by lifetime and owner, each wanting different defaults and lifecycle" %} **Release flags** are temporary and developer-owned: they exist to ship new code dark, ramp it, and then be removed. **Experiment (A/B) flags** are temporary and data-owned: they split traffic @@ -56,7 +56,7 @@ feature stays available unless someone kills it). ## Evaluating a flag: OpenFeature and flagd Evaluation should not couple your code to a vendor. **OpenFeature** is the CNCF vendor-neutral -API standard — one SDK surface across languages — and **flagd** is its lightweight open-source +API standard (one SDK surface across languages), and **flagd** is its lightweight open-source backend that evaluates flags from a file, ConfigMap, or synced source, in-cluster. Together they keep the whole feature-flag story open-source and running on plain Kubernetes, with no managed SaaS, consistent with the open-source stack used throughout the book; swapping flagd for another provider later is a @@ -65,10 +65,10 @@ configuration change, not a code change. {% include excalidraw.html file="26-evaluation-architecture" alt="A request carrying user, plan, and region reaches the application, which calls getBooleanValue(flag, default, context) through the OpenFeature SDK; the flagd provider evaluates locally in-cluster against a flag source (ConfigMap, file, or sync). The default is returned if the provider is unreachable, so a flag-system outage never takes the app down, and the SDK is vendor-neutral." - caption="Figure N.3 — OpenFeature SDK + flagd: local evaluation, vendor-neutral, fail-safe to the default" %} + caption="Figure N.3: OpenFeature SDK + flagd: local evaluation, vendor-neutral, fail-safe to the default" %} The two properties that matter most: evaluation returns the **default** if the provider is -unreachable — a flag-system outage must never take the application down — and the SDK is +unreachable (a flag-system outage must never take the application down), and the SDK is **vendor-neutral**, so the app calls `getBooleanValue(flag, default, context)` and never names the backend. The evaluation *context* carries the targeting inputs (a stable targeting key plus attributes like plan and region) that the rules act on. @@ -256,26 +256,26 @@ and a `fractional` operator for percentage rollouts. This single definition does three things at once: enterprise-plan users always get the new checkout (an *entitlement*), everyone else is split 25/75 (a *rollout*), and a second flag is turned off entirely (a *kill switch*, via `state`). The `fractional` operator buckets on the -`targetingKey`, which is what gives **sticky assignment** — the same user hashes to the same +`targetingKey`, which is what gives **sticky assignment**: the same user hashes to the same bucket every time. {% include excalidraw.html file="26-progressive-rollout" alt="A stable userId is hashed into a bucket 0 to 99; the flag returns the new variant if the bucket is below the rollout threshold (5, then 25, then 100) and the old variant otherwise. The same user always lands in the same bucket, so assignment is consistent across repeat visits, and widening or narrowing the threshold ramps or rolls back instantly without a deploy." - caption="Figure N.4 — Percentage rollout via sticky bucketing: consistent per-user assignment, instant ramp and rollback" %} + caption="Figure N.4: Percentage rollout via sticky bucketing: consistent per-user assignment, instant ramp and rollback" %} -Sticky bucketing matters for two reasons. For a **rollout**, it prevents flicker — a user who +Sticky bucketing matters for two reasons. For a **rollout**, it prevents flicker: a user who got the new checkout on their first visit keeps getting it, rather than bouncing between variants -on refresh. For an **experiment**, it is a correctness requirement — a user must stay in one arm +on refresh. For an **experiment**, it is a correctness requirement: a user must stay in one arm of the test for the measurement to mean anything. Ramping the rollout is editing the `25` upward; -rolling back is editing it down — both take effect in the running fleet within flagd's sync +rolling back is editing it down, and both take effect in the running fleet within flagd's sync interval, with no redeploy. ## The kill switch A kill switch is the operational flag you reach for during an incident: wrap a risky or expensive dependency in a flag so it can be disabled instantly without a deploy. It is the human-driven -complement to the circuit breaker from the Failure Modes appendix — the breaker +complement to the circuit breaker from the Failure Modes appendix: the breaker trips automatically on errors; the kill switch is flipped by a person. {% include codetabs.html langs="Spring Boot|Quarkus|.NET|Python|C++|Go" %} @@ -350,7 +350,7 @@ this book. The same decoupling expresses several of them. {% include excalidraw.html file="26-flags-control-plane" alt="A flag control plane (flagd) sits at the centre with four spokes: a kill-switch disables a dependency (pairing with the circuit breaker), a canary combines a flag with L7 routing, a strangler cutover routes a percentage to a new service, and an experiment picks a sticky-bucketed variant." - caption="Figure N.5 — A flag is a runtime control surface — the same lever behind canaries, cutovers, kill-switches, and experiments" %} + caption="Figure N.5: A flag is a runtime control surface, the same lever behind canaries, cutovers, kill-switches, and experiments" %} A **canary** is a percentage flag deciding which build a request exercises, often paired with the L7 routing from the routing appendix. A **strangler-fig cutover** is a flag (or the proxy's @@ -368,7 +368,7 @@ is dead code, a hidden conditional, and a live risk surface that nobody remember {% include excalidraw.html file="26-flag-lifecycle" alt="A lifecycle: create the flag default off, roll it out from 0 to 100 percent with targeting, reach GA fully on, then remove the flag and its dead code. Every release flag is born with a removal ticket because a stale flag is dead code, a hidden branch, and a live risk surface." - caption="Figure N.6 — The flag lifecycle: a release flag is born with a removal ticket" %} + caption="Figure N.6: The flag lifecycle: a release flag is born with a removal ticket" %} The discipline that keeps it healthy: give every *release* flag an owner and an expiry from the moment it is created, and treat removing it as part of finishing the feature, not optional @@ -383,7 +383,7 @@ depend on (a test suite that only exercises the default path will not catch a bu flag), keep the number of *simultaneously interacting* flags small, and default flags off in test environments unless a test sets them explicitly. Observability ties flags back to the Observability chapter: record the resolved variant as a span attribute and a metric label, so a -latency or error change can be attributed to a specific flag and cohort — "the new checkout's p99 +latency or error change can be attributed to a specific flag and cohort: "the new checkout's p99 is worse for the 25% on `on`" is only visible if the variant is on the telemetry. The recurring anti-patterns to avoid are using flags as a general configuration store (they are for *release control*, not for tuning timeouts), letting release flags become permanent, stacking many flags @@ -408,14 +408,14 @@ the same OpenFeature SDK. Prove the three properties this appendix rests on, against a running flagd. **Decoupling:** with the service running, edit the `new-checkout` flag in flagd's source from `off` to a 25% fractional -rule and confirm the running fleet changes behaviour within the sync interval — no redeploy, and +rule and confirm the running fleet changes behaviour within the sync interval, with no redeploy, and narrowing it back is your rollback. **Sticky assignment:** call the endpoint repeatedly with the same `X-User` and confirm it always resolves the same variant, then sweep many distinct users and confirm the split lands near 25%. **Fail-safe:** stop flagd entirely and confirm every evaluation -returns the coded default and the service keeps serving `200`s — if it instead errors, a flag +returns the coded default and the service keeps serving `200`s. If it instead errors, a flag lookup is on a critical path without a default, which is the one feature-flag mistake you cannot ship. --- -*Verification status: verified — [`examples/27-feature-flags/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/27-feature-flags/) passes 14/14 checks in each of the six languages on Docker Engine (2026-10-13) +*Verification status: verified: [`examples/27-feature-flags/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/27-feature-flags/) passes 14/14 checks in each of the six languages on Docker Engine (2026-10-13) (plan-based targeting rules, sticky assignment and percentage rollout, kill switch, simple boolean flag, flag listing, and safe defaults while flagd is down).* diff --git a/_docs/28-appendix-o-newman.md b/_docs/28-appendix-o-newman.md index 9d2221f2..524e27d1 100644 --- a/_docs/28-appendix-o-newman.md +++ b/_docs/28-appendix-o-newman.md @@ -4,14 +4,14 @@ marker: "O" label: "Appendix O" order: 28 part: "Deep-dive appendices" -description: "Turning Postman collections into an automated API test gate with Newman: the anatomy of a collection, writing assertions, running one collection against all six implementations, data-driven tests, schema and contract validation, and wiring it into CI on plain Docker Engine — no new test framework to learn." +description: "Turning Postman collections into an automated API test gate with Newman: the anatomy of a collection, writing assertions, running one collection against all six implementations, data-driven tests, schema and contract validation, and wiring it into CI on plain Docker Engine: no new test framework to learn." duration: 20 minutes --- Postman is already one of the tools this book reaches for to exercise an API by hand. **Newman** is its command-line runner: it executes the same Postman collection headlessly, which turns the requests your team already pokes at interactively into an automated test gate you can run on -every change. The payoff is that the assertions live in one place — the collection — and the same +every change. The payoff is that the assertions live in one place (the collection), and the same file serves three audiences: a developer clicking through Postman, a reviewer reading the tests, and a CI pipeline failing the build when an assertion breaks. @@ -22,8 +22,8 @@ without the GUI and emits machine-readable results. {% include excalidraw.html file="27-newman-run" - alt="A collection.json (requests + tests) and an environment.json (base_url, tokens) feed into 'newman run', the headless CLI, which produces reporters — cli for humans, junit.xml for CI, html to share — feeding a CI gate that passes or fails. The same collection the team runs by hand becomes an automated gate with no new test framework." - caption="Figure O.1 — Newman runs a Postman collection headlessly and emits CI-readable results" %} + alt="A collection.json (requests + tests) and an environment.json (base_url, tokens) feed into 'newman run', the headless CLI, which produces reporters (cli for humans, junit.xml for CI, html to share), feeding a CI gate that passes or fails. The same collection the team runs by hand becomes an automated gate with no new test framework." + caption="Figure O.1: Newman runs a Postman collection headlessly and emits CI-readable results" %} There is no new test framework to learn: the tests are the JavaScript assertions already supported inside Postman, and Newman is the bridge from "works when I click it" to "verified on @@ -38,7 +38,7 @@ parts that change between runs. {% include excalidraw.html file="27-collection-anatomy" alt="A collection contains folders that group requests by resource; each request has a pre-request script, the HTTP call, and tests; the tests are JavaScript using pm.test and pm.expect to assert status, schema, and response time; environment variables like base_url and token are resolved at run time." - caption="Figure O.2 — A collection is folders of requests, each with assertions; the environment supplies the variables" %} + caption="Figure O.2: A collection is folders of requests, each with assertions; the environment supplies the variables" %} The tests are small JavaScript blocks using Postman's `pm` API. A request that creates an order might assert the status, the response time, and the shape of the body: @@ -65,8 +65,8 @@ might assert the status, the response time, and the shape of the body: }{% endraw %} ``` -That last line stores the new id in a variable so a later request — fetch, cancel, or pay for the -order — can chain off it. Chaining requests through variables is how a collection models a whole +That last line stores the new id in a variable so a later request (fetch, cancel, or pay for the +order) can chain off it. Chaining requests through variables is how a collection models a whole flow, not just isolated calls. ## One collection, six backends @@ -77,8 +77,8 @@ implementation in this book. {% include excalidraw.html file="27-black-box" - alt="One collection running under Newman tests the same HTTP API (/orders, /payments) against six order-service implementations — Spring Boot, Quarkus, .NET, Python, C++, and Go. Because Newman tests the API as a black box, one collection validates every implementation; the contract is the spec, not the code." - caption="Figure O.3 — One collection is the conformance suite for all six implementations" %} + alt="One collection running under Newman tests the same HTTP API (/orders, /payments) against six order-service implementations: Spring Boot, Quarkus, .NET, Python, C++, and Go. Because Newman tests the API as a black box, one collection validates every implementation; the contract is the spec, not the code." + caption="Figure O.3: One collection is the conformance suite for all six implementations" %} The collection *is* the API specification made executable. Point it at whichever implementation is running and the assertions are identical. Bringing the service @@ -149,8 +149,8 @@ This is how you cover boundary values, bad input, and edge cases without duplica {% include excalidraw.html file="27-data-driven" - alt="A collection of one request set, combined with an iteration-data.csv of rows (id and qty values including a zero and a large number), runs once per row under newman, producing N result sets — one per case." - caption="Figure O.4 — Data-driven runs: one request set, many inputs via --iteration-data" %} + alt="A collection of one request set, combined with an iteration-data.csv of rows (id and qty values including a zero and a large number), runs once per row under newman, producing N result sets (one per case)." + caption="Figure O.4: Data-driven runs: one request set, many inputs via --iteration-data" %} ```bash {% raw %}newman run orders.postman_collection.json -e local.postman_environment.json \ @@ -167,7 +167,7 @@ becomes a table of positive and negative cases. ## Contract and schema validation -Beyond status codes, a test can validate the response *shape* against a JSON Schema — which is how +Beyond status codes, a test can validate the response *shape* against a JSON Schema, which is how you turn the metadata and error-format work from earlier appendices into an enforced contract. A `Problem Details` error body or an order representation can be checked field by field. @@ -187,19 +187,19 @@ you turn the metadata and error-format work from earlier appendices into an enfo ``` Asserting against the schema rather than exact values keeps the test stable as data changes while -still catching a contract break — a removed field, a wrong type, a renamed property — the moment +still catching a contract break (a removed field, a wrong type, a renamed property) the moment it appears. ## Newman in CI The runner exits non-zero when any assertion fails, which is all a pipeline needs to gate a merge. -The services come up on Docker Engine exactly as they do locally — no managed cloud — and Newman runs +The services come up on Docker Engine exactly as they do locally (no managed cloud), and Newman runs against them. {% include excalidraw.html file="27-ci-gate" alt="A pipeline: pull request, build image, docker compose up of the order/payment/inventory services, newman run of the collection, then a gate where pass merges and fail blocks. The services run on Docker Engine in CI as they do locally, and a failed assertion fails the build." - caption="Figure O.5 — Newman as a merge gate: bring the stack up on Docker, run the collection, fail the build on a broken contract" %} + caption="Figure O.5: Newman as a merge gate: bring the stack up on Docker, run the collection, fail the build on a broken contract" %} ```yaml # .github/workflows/api-tests.yml (any runner that has Docker Engine + Node works) @@ -227,13 +227,13 @@ boot and fail spuriously. ## Where Newman fits -Newman is not a replacement for unit tests — it is the layer that sits on top of them. It owns the +Newman is not a replacement for unit tests: it is the layer that sits on top of them. It owns the API/contract tier: real HTTP against a running service, fast enough to run on every pull request. {% include excalidraw.html file="27-test-pyramid" alt="A test pyramid: a wide base of unit tests (many, fast, cheap), then service/integration tests, then API/contract tests where Newman runs, then a narrow top of E2E tests (few, slow, brittle). Newman owns the API/contract tier and complements unit tests rather than replacing them." - caption="Figure O.6 — Newman owns the API/contract tier — above unit tests, below full end-to-end" %} + caption="Figure O.6: Newman owns the API/contract tier (above unit tests, below full end-to-end)" %} Keep the base of fast unit tests broad, use Newman for the contract and integration behaviour that only shows up over real HTTP, and reserve a thin layer of full end-to-end tests for the few @@ -248,7 +248,7 @@ a `403` when it is present but unauthorised, a `409` or `422` on a business-rule `429` carrying `RateLimit` headers when a client exceeds its budget. Confirm idempotency by sending the same `Idempotency-Key` twice and asserting the second response matches the first rather than creating a duplicate. These tie the testing appendix back to the error-format, security, -idempotency, and rate-limiting material — a contract is only enforced if its error behaviour is +idempotency, and rate-limiting material: a contract is only enforced if its error behaviour is tested, not just its success behaviour. ## Take-aways and references @@ -258,7 +258,7 @@ headlessly so the same assertions guard every change. Because Newman tests over collection is the conformance suite for all six implementations. Validate response *shape* against a schema, not just status codes; drive edge cases from data rather than duplicated requests; and assert the failure paths as carefully as the happy path. Wire it into CI on plain Docker Engine so the -build fails on a broken contract, and keep Newman in its lane — the API/contract tier, on top of unit +build fails on a broken contract, and keep Newman in its lane: the API/contract tier, on top of unit tests and under a thin end-to-end layer. The canonical references are the Postman documentation and the Newman command-line documentation (install, reporters, and the `--iteration-data` and `--bail` flags), and the Postman Learning Center's guides on writing tests with the `pm` API and on @@ -268,13 +268,13 @@ JSON-schema validation. Prove the gate end to end against the running stack. Bring the services up with `docker compose up -d --build --wait`, then `newman run` the orders collection with `--reporters cli,junit` and confirm a green run and a -`newman.xml` you could hand to CI. Now break the contract — have the service return -`200` instead of `201` on create, or drop the `id` field — and confirm Newman exits non-zero and +`newman.xml` you could hand to CI. Now break the contract (have the service return +`200` instead of `201` on create, or drop the `id` field) and confirm Newman exits non-zero and the failing assertion names the exact expectation. Finally, add a negative row to your `--iteration-data` (a zero-quantity order expecting `422`) and confirm it passes against a correct service and fails against one that wrongly accepts it. A suite that goes red on a real contract break, and only then, is the whole value of this appendix. --- -*Verification status: verified — [`examples/28-newman/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/28-newman/) passes 11/11 checks on Docker Engine (2026-10-13) +*Verification status: verified; [`examples/28-newman/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/28-newman/) passes 11/11 checks on Docker Engine (2026-10-13) (a Newman collection run with status-code and state assertions (201, 422, 404, 204, cancel) and JUnit XML export).* diff --git a/_docs/29-appendix-p-dotnet-on-linux.md b/_docs/29-appendix-p-dotnet-on-linux.md index 5226f303..b342ad00 100644 --- a/_docs/29-appendix-p-dotnet-on-linux.md +++ b/_docs/29-appendix-p-dotnet-on-linux.md @@ -10,38 +10,38 @@ duration: 15 minutes > **This appendix is .NET-specific.** It exists for developers who know C# but have > only ever run it on Windows with Visual Studio. Every other chapter is -> multi-language — this one answers the questions that come up when the deployment +> multi-language; this one answers the questions that come up when the deployment > target changes from IIS on Windows Server to a container on Linux. The move from Windows to Linux changes _where_ .NET runs, not _what_ it is. The same language, the same compiler, the same runtime, the same NuGet packages. What disappears is the Windows-only substrate: IIS, the Windows registry, COM interop, Windows Services. What replaces them is the same thing that replaced them for Java and Python -years ago — a container runtime, a process supervisor (Kubernetes), and a +years ago: a container runtime, a process supervisor (Kubernetes), and a standards-based web server (Kestrel). This appendix walks through four layers, from source code to running container: -1. **C# and .NET** — the language vs. the platform, and why the distinction matters. -2. **CIL and the CoreCLR runtime** — what `dotnet build` produces and how it executes. -3. **Kestrel** — the web server built into ASP.NET Core. -4. **Containers** — multi-stage builds, Red Hat UBI images, and what ships to production. +1. **C# and .NET**: the language vs. the platform, and why the distinction matters. +2. **CIL and the CoreCLR runtime**: what `dotnet build` produces and how it executes. +3. **Kestrel**: the web server built into ASP.NET Core. +4. **Containers**: multi-stage builds, Red Hat UBI images, and what ships to production. ## C# is a language. .NET is a platform. The distinction matters because conflating them causes architectural confusion. **C#** is -a statically-typed, garbage-collected language — one of several that target .NET. F# and +a statically-typed, garbage-collected language, one of several that target .NET. F# and VB.NET are the others. All three compile to the same intermediate representation and run on the same runtime. **.NET** (since .NET 5, unified; previously split into .NET Framework, .NET Core, and -Mono) is the platform: a runtime (CoreCLR), a base class library (BCL — `System.*`, +Mono) is the platform: a runtime (CoreCLR), a base class library (BCL: `System.*`, `Microsoft.Extensions.*`), a compiler toolchain (Roslyn for C#, the F# compiler), a -package manager (NuGet), and a CLI (`dotnet`). The platform is cross-platform — the +package manager (NuGet), and a CLI (`dotnet`). The platform is cross-platform: the same `dotnet publish` output runs on Linux x86-64, Linux ARM64, macOS, and Windows without recompilation, because the output is not native machine code. -The practical consequence: when this book says ".NET 10," it means the platform — the +The practical consequence: when this book says ".NET 10," it means the platform: the runtime, the BCL, and the ASP.NET Core framework that sits on top. When it says "C#," it means the language syntax. A `.csproj` file targets a .NET version (`net10.0`), not a C# version, because the platform version determines what APIs are available. @@ -49,35 +49,35 @@ version, because the platform version determines what APIs are available. ## CIL: what `dotnet build` produces When Roslyn compiles a C# source file, the output is not x86 machine code. It is -**Common Intermediate Language** (CIL, historically called MSIL or just IL) — a +**Common Intermediate Language** (CIL, historically called MSIL or just IL): a CPU-agnostic bytecode stored in `.dll` files. A `.dll` built on macOS runs unmodified on Linux because the bytes inside are CIL, not native instructions. {% include excalidraw.html file="29-dotnet-compilation" alt="The .NET compilation pipeline. C# source is compiled by Roslyn into CIL bytecode stored in DLL files. At runtime, CoreCLR loads the CIL and RyuJIT compiles it method-at-a-time into native x86-64 code. NativeAOT is an alternative that compiles ahead of time." - caption="Figure P.1 — From C# source to native code: Roslyn emits CIL bytecode, RyuJIT compiles to native at runtime" %} + caption="Figure P.1: From C# source to native code (Roslyn emits CIL bytecode, RyuJIT compiles to native at runtime)" %} -The runtime — **CoreCLR** — loads these CIL assemblies and compiles them to native code +The runtime (**CoreCLR**) loads these CIL assemblies and compiles them to native code on the fly using **RyuJIT**, the just-in-time compiler. RyuJIT compiles one method at a time, the first time it is called. After that first call, the method runs at native -speed — there is no interpreter step and no re-compilation on subsequent calls. +speed: there is no interpreter step and no re-compilation on subsequent calls. CoreCLR on Linux is `libcoreclr.so`, a shared library. It provides: -- **JIT compilation** (RyuJIT) — CIL to native x86-64 or ARM64. -- **Garbage collection** — a generational, concurrent GC that runs on its own threads. +- **JIT compilation** (RyuJIT): CIL to native x86-64 or ARM64. +- **Garbage collection**: a generational, concurrent GC that runs on its own threads. Server GC mode (the default in ASP.NET Core) allocates one heap per CPU core. -- **Type system** — reflection, generics, interface dispatch. -- **Threading** — maps .NET threads to pthreads; `Task` and `async/await` run on the +- **Type system**: reflection, generics, interface dispatch. +- **Threading**: maps .NET threads to pthreads; `Task` and `async/await` run on the thread pool. -- **Exception handling** — uses platform-specific unwinding (libunwind on Linux). -- **P/Invoke** — call native C libraries directly (`libpq`, `librdkafka`, `openssl`). +- **Exception handling**: uses platform-specific unwinding (libunwind on Linux). +- **P/Invoke**: call native C libraries directly (`libpq`, `librdkafka`, `openssl`). ### NativeAOT: the ahead-of-time alternative .NET also offers **NativeAOT** compilation, which runs Roslyn and the AOT compiler at -build time to produce a single native binary — no JIT, no `libcoreclr.so`, no `.dll` +build time to produce a single native binary: no JIT, no `libcoreclr.so`, no `.dll` files at runtime. The trade-off is real: NativeAOT binaries start faster and use less memory, but they cannot use runtime code generation (no `Reflection.Emit`, limited reflection). For the examples in this book, the JIT path is the default because it @@ -86,7 +86,7 @@ supports the full library ecosystem without restrictions. ### What the runtime is _not_ CoreCLR is not a virtual machine in the JVM sense. There is no bytecode interpreter -running continuously — CIL is compiled to native code before it executes. After JIT +running continuously: CIL is compiled to native code before it executes. After JIT compilation, a .NET method is indistinguishable from a method compiled by GCC or Clang. The runtime is still present (for GC, exception handling, and type loading), but the hot path is native machine code. @@ -98,27 +98,27 @@ Services). IIS managed the process lifecycle, handled TLS, parsed HTTP, and forw requests to the .NET application via a proprietary hosting model. On Linux (and on modern Windows), **Kestrel** replaces all of that. Kestrel is the web -server built into ASP.NET Core — it is not a separate process, not a reverse proxy, and +server built into ASP.NET Core: it is not a separate process, not a reverse proxy, and not optional. When a .NET web application starts, Kestrel binds to a port and handles HTTP directly. {% include excalidraw.html file="29-kestrel-pipeline" alt="Kestrel request processing pipeline. Kestrel handles TLS via OpenSSL, parses HTTP using Span-based parsers, passes requests through the ASP.NET Core middleware pipeline (routing, auth, CORS, diagnostics), and dispatches to application endpoints (minimal APIs, gRPC, GraphQL, WebSockets, health checks)." - caption="Figure P.2 — Kestrel processes requests end to end: TLS, HTTP parsing, middleware, endpoint dispatch" %} + caption="Figure P.2: Kestrel processes requests end to end (TLS, HTTP parsing, middleware, endpoint dispatch)" %} Key things Kestrel does: -- **Listens on TCP sockets** — using `epoll` on Linux (or `io_uring` on newer kernels). +- **Listens on TCP sockets**: using `epoll` on Linux (or `io_uring` on newer kernels). No external web server needed. -- **Handles TLS** — via OpenSSL on Linux. In Kubernetes, TLS is typically terminated by +- **Handles TLS**: via OpenSSL on Linux. In Kubernetes, TLS is typically terminated by the service mesh (Istio) or ingress controller, so Kestrel often runs plain HTTP inside the Pod. -- **Parses HTTP/1.1, HTTP/2, and HTTP/3** — using zero-allocation `Span` parsers. +- **Parses HTTP/1.1, HTTP/2, and HTTP/3**: using zero-allocation `Span` parsers. gRPC requires HTTP/2, which Kestrel supports natively. -- **Runs the middleware pipeline** — each request passes through a configurable chain: +- **Runs the middleware pipeline**: each request passes through a configurable chain: routing, authentication, CORS, diagnostics, exception handling. -- **Dispatches to endpoints** — minimal APIs, gRPC services, GraphQL servers, +- **Dispatches to endpoints**: minimal APIs, gRPC services, GraphQL servers, WebSocket handlers, and health check endpoints all share the same Kestrel host. ### Why not nginx or Apache in front? @@ -127,7 +127,7 @@ In production Kubernetes, the answer is usually: you do not need them. The servi handles TLS termination, load balancing, and retries. Kestrel handles everything else. Adding nginx as a sidecar adds latency, memory, and operational complexity for features Kestrel already provides. The one exception is when you need nginx-specific features -like rate limiting at the edge — in that case, it sits in front as an ingress controller, +like rate limiting at the edge. In that case, it sits in front as an ingress controller, not as a per-Pod sidecar. ### Kestrel configuration in practice @@ -147,19 +147,19 @@ builder.WebHost.ConfigureKestrel(k => ``` The `+` binds to all interfaces (0.0.0.0). Port 8080 is the convention for non-root -containers — ports below 1024 require root, which containers should not run as. +containers: ports below 1024 require root, which containers should not run as. ## Containers: what ships to production A .NET application in a container is a Linux process. There is no VM, no app pool, no -IIS — just `dotnet YourApp.dll` running as PID 1 (or behind a container init process +IIS: just `dotnet YourApp.dll` running as PID 1 (or behind a container init process like `tini`). The container image must contain the .NET runtime and your application's published DLLs; it does not need the SDK, Roslyn, NuGet, or MSBuild. {% include excalidraw.html file="29-container-layers" alt="Multi-stage container build. The build stage uses the UBI9 .NET 10 SDK image to restore NuGet packages and compile the application. Only the published output is copied to the runtime stage, which uses the smaller UBI9 ASP.NET runtime image. The SDK never ships." - caption="Figure P.3 — Multi-stage build: the SDK compiles, only the runtime and app DLLs ship" %} + caption="Figure P.3: Multi-stage build (the SDK compiles, only the runtime and app DLLs ship)" %} ### Multi-stage Containerfiles @@ -202,11 +202,11 @@ code, and intermediate build artifacts are all discarded. The final image is typ The examples use **Red Hat Universal Base Images** (UBI) instead of the `mcr.microsoft.com/dotnet/*` images from Microsoft. UBI images are: -- **Freely redistributable** — no subscription required to pull, run, or ship them. -- **Based on RHEL 9** — same packages, same security patches, same lifecycle. -- **Supported by Red Hat** — when running on RHEL, UBI images get full +- **Freely redistributable**: no subscription required to pull, run, or ship them. +- **Based on RHEL 9**: same packages, same security patches, same lifecycle. +- **Supported by Red Hat**: when running on RHEL, UBI images get full commercial support. -- **Minimal by default** — `ubi9-minimal` variants use microdnf instead of yum, +- **Minimal by default**: `ubi9-minimal` variants use microdnf instead of yum, producing smaller images. The .NET UBI images (`ubi9/dotnet-100` for SDK, `ubi9/dotnet-100-aspnet` for runtime) @@ -219,10 +219,10 @@ Microsoft's releases but the build, base OS layer and support lifecycle are Red Layer by layer, from bottom to top: -1. **UBI9 base** (~70 MB) — glibc, openssl, ca-certificates, tzdata, libicu. -2. **ASP.NET Core runtime** (~100 MB) — `libcoreclr.so`, `libclrjit.so`, the BCL +1. **UBI9 base** (~70 MB): glibc, openssl, ca-certificates, tzdata, libicu. +2. **ASP.NET Core runtime** (~100 MB): `libcoreclr.so`, `libclrjit.so`, the BCL assemblies (`System.*.dll`), and the ASP.NET Core shared framework. -3. **Your application** (~5–80 MB) — your compiled `.dll` files, third-party NuGet +3. **Your application** (~5–80 MB): your compiled `.dll` files, third-party NuGet package DLLs, appsettings.json. Things that are _not_ in the final image: the SDK, MSBuild, Roslyn, NuGet, your source @@ -232,13 +232,13 @@ code, `obj/` and `bin/` intermediates, test projects. When Kubernetes starts the container, `dotnet OrderService.dll` runs: -1. CoreCLR initializes — loads `libcoreclr.so`, starts the GC, initializes the thread +1. CoreCLR initializes: loads `libcoreclr.so`, starts the GC, initializes the thread pool. -2. The host builder runs — registers services (DI), configures logging, sets up +2. The host builder runs: registers services (DI), configures logging, sets up OpenTelemetry. -3. Kestrel binds to port 8080 — begins accepting HTTP connections. -4. The application is ready — Kubernetes starts sending liveness and readiness probes. -5. On shutdown, Kubernetes sends **SIGTERM** — the .NET host catches it and begins +3. Kestrel binds to port 8080: begins accepting HTTP connections. +4. The application is ready: Kubernetes starts sending liveness and readiness probes. +5. On shutdown, Kubernetes sends **SIGTERM**: the .NET host catches it and begins graceful shutdown: stops accepting new requests, drains in-flight requests, flushes telemetry, disposes services. @@ -248,19 +248,19 @@ no w3wp.exe, no application domain. The container _is_ the process boundary. ### Cross-check it yourself Verify the compilation pipeline by inspecting a published application's output. Run -`dotnet publish -c Release` on any example and list the output directory — you will see +`dotnet publish -c Release` on any example and list the output directory: you will see `.dll` files (CIL bytecode, not native code), a `.deps.json` (dependency graph), and a `.runtimeconfig.json` (runtime version selector). Open one of the `.dll` files with -`dotnet-ildasm` or `ILSpy` and you will see CIL opcodes, not x86 instructions — proof +`dotnet-ildasm` or `ILSpy` and you will see CIL opcodes, not x86 instructions: proof that the JIT does its work at runtime, not at build time. Then build a container image and inspect its layers. Run -`docker image inspect | jq '.[0].RootFS.Layers | length'` — you should see the UBI +`docker image inspect | jq '.[0].RootFS.Layers | length'`: you should see the UBI base layer, the runtime layer, and your application layer. Compare the image size to the SDK image: the runtime image should be roughly 4x smaller. --- -*Verification status: unverified — the three diagrams (P.1 compilation pipeline, P.2 +*Verification status: unverified. The three diagrams (P.1 compilation pipeline, P.2 Kestrel pipeline, P.3 container layers) are structurally correct based on the .NET 10 architecture, and the Containerfile pattern matches the verified examples in this book. The UBI image sizes (~780 MB SDK, ~170 MB runtime) are approximate and will vary by .NET diff --git a/_docs/29-glossary.md b/_docs/29-glossary.md index 307352c6..f620dc35 100644 --- a/_docs/29-glossary.md +++ b/_docs/29-glossary.md @@ -4,12 +4,12 @@ marker: "★" label: "Glossary" order: 99 part: "Deep-dive appendices" -description: "Every acronym used across the book, expanded and placed in context — grouped by where it shows up: the wire, the data layer, events, the platform, security, observability, and the domain." +description: "Every acronym used across the book, expanded and placed in context, grouped by where it shows up: the wire, the data layer, events, the platform, security, observability, and the domain." --- Every acronym the book leans on, expanded and put in context. They are grouped by the -layer they belong to — the wire, the data store, messaging, the platform, security, -observability, and the domain — so a term is easy to find by the part of the system it +layer they belong to (the wire, the data store, messaging, the platform, security, +observability, and the domain), so a term is easy to find by the part of the system it describes. Where an acronym means something specific *in this book*, the third column says what, because a few of them (`ACL` among them) carry a different meaning here than the one you might expect. @@ -18,17 +18,17 @@ might expect. | Acronym | Expands to | In this book | |---|---|---| -| API | Application Programming Interface | The contract a service exposes — the whole subject of the book | +| API | Application Programming Interface | The contract a service exposes: the whole subject of the book | | REST | Representational State Transfer | Resource-oriented HTTP; the public, cacheable, browser-native default | | gRPC | gRPC Remote Procedure Call | Procedure-oriented, HTTP/2, protobuf binary; the internal-call default | | RPC | Remote Procedure Call | Calling a typed method on another process as if it were local | | HTTP, HTTP/1.1, HTTP/2 | HyperText Transfer Protocol | The /1.1 and /2 suffixes are protocol versions; gRPC requires /2 | | TLS | Transport Layer Security | Encryption on the wire; an L7 router usually terminates it | -| mTLS | mutual TLS | Both sides present certificates — workload-to-workload identity in the mesh | +| mTLS | mutual TLS | Both sides present certificates: workload-to-workload identity in the mesh | | SNI | Server Name Indication | The hostname an L4 balancer can route on without terminating TLS | | TCP | Transmission Control Protocol | The L4 connection an L7 router rides on top of | | JSON | JavaScript Object Notation | Text encoding for REST and GraphQL payloads | -| SDL | Schema Definition Language | GraphQL's typed schema — its published contract | +| SDL | Schema Definition Language | GraphQL's typed schema: its published contract | | URL / URI | Uniform Resource Locator / Identifier | The address of a resource; a URI identifies, a URL also locates | | DTO | Data Transfer Object | A shape passed across a boundary; sharing one is *model coupling* | | BFF | Backend for Frontend | An API tailored to one client, often a GraphQL gateway | @@ -52,7 +52,7 @@ might expect. | PII | Personally Identifiable Information | Data that defines a privacy and encryption boundary | | TTL | Time To Live | How long a cached entry or record stays valid before expiry | | MSDTC | Microsoft Distributed Transaction Coordinator | The two-phase-commit coordinator that sagas replace across services | -| N+1 | the N+1 query problem | One query per row instead of one batched query — fixed with batching | +| N+1 | the N+1 query problem | One query per row instead of one batched query, fixed with batching | ## Events, messaging, and streaming @@ -72,7 +72,7 @@ might expect. | KEDA | Kubernetes Event-Driven Autoscaling | Scales on queue depth and other event sources, including to zero | | CRD | Custom Resource Definition | How an operator (or Istio) extends the Kubernetes API | | CNCF | Cloud Native Computing Foundation | Home of Kubernetes, OpenTelemetry, OpenFeature, and the rest of the stack | -| OSS | Open-Source Software | The whole platform is OSS — no managed-cloud lock-in | +| OSS | Open-Source Software | The whole platform is OSS: no managed-cloud lock-in | | YAML | YAML Ain't Markup Language | The configuration format for Kubernetes and Istio objects | | LB | Load Balancer | An L4 or L7 component spreading traffic across pods | | L4 / L7 | Layer 4 / Layer 7 (OSI) | Transport-level versus application-level routing | @@ -117,7 +117,7 @@ might expect. | RED | Rate, Errors, Duration | The request-centric monitoring method | | USE | Utilization, Saturation, Errors | The resource-centric monitoring method | | RTT | Round-Trip Time | Network latency; what locality-aware routing minimises | -| p50 / p99 | 50th / 99th percentile | Latency percentiles — p99 is where L7 hop overhead shows up | +| p50 / p99 | 50th / 99th percentile | Latency percentiles: p99 is where L7 hop overhead shows up | | MDC | Mapped Diagnostic Context | Per-request fields attached to every log line, e.g. the trace id | | SRE | Site Reliability Engineering | The discipline behind SLOs, error budgets, and these methods | | ETW | Event Tracing for Windows | The legacy telemetry source replaced by OpenTelemetry | @@ -127,8 +127,8 @@ might expect. | Acronym | Expands to | In this book | |---|---|---| | DDD | Domain-Driven Design | Bounded contexts and ubiquitous language drive where API boundaries go | -| ACL | Anticorruption Layer | **In DDD**, a translation layer keeping a foreign model out of yours — *not* an access-control list | -| XOR | exclusive OR | The coupling rule "modularity = strength XOR distance" — have one, not both | +| ACL | Anticorruption Layer | **In DDD**, a translation layer keeping a foreign model out of yours, *not* an access-control list | +| XOR | exclusive OR | The coupling rule "modularity = strength XOR distance": have one, not both | | MVC | Model-View-Controller | The legacy web UI pattern migrated from | | UI | User Interface | The client surface, e.g. Blazor in the .NET stack | | AI | Artificial Intelligence | The agent context the opening chapter frames the book around | @@ -144,6 +144,6 @@ might expect. | MIT, BSD, Apache 2.0, AGPL | open-source software licenses | The licenses on the pinned toolkit and platform components | --- -*Verification status: compiled from the book's own usage — every entry appears in the +*Verification status: compiled from the book's own usage: every entry appears in the chapters or appendices. Expansions follow each technology's canonical naming; where a term is overloaded (`ACL`), the meaning used in this book is called out explicitly.* diff --git a/_docs/30-appendix-q-blazor-signalr.md b/_docs/30-appendix-q-blazor-signalr.md index a6c9ea82..6b9cc369 100644 --- a/_docs/30-appendix-q-blazor-signalr.md +++ b/_docs/30-appendix-q-blazor-signalr.md @@ -4,30 +4,30 @@ marker: "Q" label: "Appendix Q" order: 30 part: "Deep-dive appendices" -description: "Real-time server-rendered UI with Blazor Server over a SignalR circuit — how the persistent connection works, the Redis backplane for scale-out, and when to choose SignalR over raw WebSockets." +description: "Real-time server-rendered UI with Blazor Server over a SignalR circuit: how the persistent connection works, the Redis backplane for scale-out, and when to choose SignalR over raw WebSockets." duration: 22 minutes --- -Everything in **Appendix C** used raw WebSockets — manual binary frames, hand-built +Everything in **Appendix C** used raw WebSockets: manual binary frames, hand-built backplanes, explicit sequence tracking. That is the right choice for a polyglot system. But when the entire stack is .NET, **SignalR** abstracts all of that away, and **Blazor Server** builds a full server-rendered UI on top of it. This appendix -covers both — what they are, how they scale, and when to reach for them instead of +covers both: what they are, how they scale, and when to reach for them instead of raw sockets. ## What Blazor Server is Blazor Server is a server-rendered component model where the browser runs a thin JavaScript runtime (the SignalR client) and all C# logic stays server-side. The -browser and the server maintain a **persistent SignalR circuit** — a single -WebSocket per connected client — and the server sends DOM diffs over that circuit +browser and the server maintain a **persistent SignalR circuit** (a single +WebSocket per connected client), and the server sends DOM diffs over that circuit whenever the UI changes. The browser applies the diff; the user sees a live, interactive page without downloading a heavy client-side framework. {% include excalidraw.html file="30-blazor-circuit" alt="A browser running the Blazor JS runtime connects to the ASP.NET Core server via a SignalR circuit (WebSocket). The server holds the component tree, handles events (button click, form submit), re-renders the affected components, computes the DOM diff, and sends it over the circuit. The browser applies the diff. All C# logic stays server-side." - caption="Figure Q.1 — Blazor Server: a persistent SignalR circuit carries DOM diffs, events flow up, diffs flow down" %} + caption="Figure Q.1: Blazor Server: a persistent SignalR circuit carries DOM diffs, events flow up, diffs flow down" %} The trade-off is latency: every click is a round-trip to the server, so users on high-latency connections notice a delay. For internal dashboards, admin panels, and @@ -37,7 +37,7 @@ SPA instead. ## SignalR under the hood -SignalR is ASP.NET Core's real-time abstraction. At its core it is a **Hub** — a +SignalR is ASP.NET Core's real-time abstraction. At its core it is a **Hub**: a class with methods the client can call, and methods the server can push to connected clients. The transport is a WebSocket by default, with automatic fallback to Server-Sent Events or long polling when WebSockets aren't available. @@ -60,13 +60,13 @@ public class OrderHub : Hub Three features distinguish SignalR from raw WebSockets: -- **Strongly typed methods** — the server calls `Clients.All.SendAsync("OrderPlaced", order)` +- **Strongly typed methods**: the server calls `Clients.All.SendAsync("OrderPlaced", order)` and the client receives a typed callback, not a raw byte buffer. No manual serialization, no type discriminators, no `switch` on message type. -- **Connection groups** — `Groups.AddToGroupAsync` replaces the per-pod registry +- **Connection groups**: `Groups.AddToGroupAsync` replaces the per-pod registry and backplane fan-out from Appendix C. One API call adds a client to a logical group; SignalR handles the routing. -- **Automatic reconnection** — the client SDK reconnects with configurable backoff +- **Automatic reconnection**: the client SDK reconnects with configurable backoff and restores group memberships automatically. ## Scaling: the same problem, a built-in solution @@ -78,9 +78,9 @@ it, scale-up leaves new pods idle, and scale-down drops active circuits. {% include excalidraw.html file="30-signalr-backplane" alt="Three Blazor Server pods behind a Kubernetes Service / load balancer, each holding its own set of SignalR circuits. All three subscribe and publish to a shared Redis backplane (StackExchange.Redis). A message sent to a group on pod-1 is published to Redis; Redis delivers it to pod-2 and pod-3, which push it to their local circuits in that group." - caption="Figure Q.2 — SignalR's Redis backplane: the same pub/sub externalisation as Appendix C, built into the framework" %} + caption="Figure Q.2: SignalR's Redis backplane: the same pub/sub externalisation as Appendix C, built into the framework" %} -The solution is the same — a shared backplane — but SignalR builds it in: +The solution is the same, a shared backplane, but SignalR builds it in: ```csharp // Program.cs — wire the Redis backplane for multi-pod scale-out @@ -147,7 +147,7 @@ connected dashboard. ``` The component subscribes to the hub on mount, inserts each new order at the top of -the list, and calls `StateHasChanged` to trigger a re-render — which sends a DOM +the list, and calls `StateHasChanged` to trigger a re-render, which sends a DOM diff (one new ``) over the circuit. No polling, no manual WebSocket code, no binary framing. @@ -170,7 +170,7 @@ server-side, avoiding a separate frontend framework entirely. ### Cross-check it yourself Run the Blazor Server example, open the dashboard in two browser tabs, and place an -order via `curl`. Both tabs should update simultaneously — confirming the SignalR hub +order via `curl`. Both tabs should update simultaneously, confirming the SignalR hub push and the Redis backplane fan-out. Then stop and restart one of the Blazor Server pods and confirm the circuit reconnects automatically without losing the dashboard state. @@ -179,4 +179,4 @@ The code is in [`examples/30-blazor-signalr/`](https://github.com/patterncatalys runs it; its `README.md` covers what it does and how to drive it. --- -*Verification status: verified — [`examples/30-blazor-signalr/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/30-blazor-signalr/) passes 10/10 checks on Docker Engine (2026-10-13): health, order placement and listing, the dashboard page, the SignalR negotiate endpoint, a second dashboard instance, Redis backplane delivery in both directions between the two instances (a SignalR client on one receives an order posted to the other), orders listed across instances, and backplane delivery to a restarted instance (the script asserts the container's start time changed and it returned healthy). Blazor circuit reconnection is the framework default and is not covered by the script.* +*Verification status: verified: [`examples/30-blazor-signalr/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/30-blazor-signalr/) passes 10/10 checks on Docker Engine (2026-10-13): health, order placement and listing, the dashboard page, the SignalR negotiate endpoint, a second dashboard instance, Redis backplane delivery in both directions between the two instances (a SignalR client on one receives an order posted to the other), orders listed across instances, and backplane delivery to a restarted instance (the script asserts the container's start time changed and it returned healthy). Blazor circuit reconnection is the framework default and is not covered by the script.* diff --git a/_docs/31-appendix-r-minikube.md b/_docs/31-appendix-r-minikube.md index 7e9db769..dd735200 100644 --- a/_docs/31-appendix-r-minikube.md +++ b/_docs/31-appendix-r-minikube.md @@ -44,7 +44,7 @@ idle, and a drop looks like a broken service. Because the ports are fixed at cre {% include excalidraw.html file="31-minikube-topology" alt="The host runs Docker Engine. Inside the node container cndp-k8s there are two namespaces: strimzi holds the Strimzi operator, and cndp-05 holds order-service, the shipping and notification consumers, a Kafka cluster, Postgres, the LGTM stack and the Postgres Secret. The host reaches order-service on 127.0.0.1:30080 and Grafana on 127.0.0.1:30300, and runs kubectl exec against the Kafka and Postgres pods." - caption="Figure R.1 — The node container holds two namespaces; the host reaches two loopback NodePorts and uses kubectl exec for the Kafka and Postgres checks" %} + caption="Figure R.1: The node container holds two namespaces; the host reaches two loopback NodePorts and uses kubectl exec for the Kafka and Postgres checks" %} ## From compose to Kubernetes @@ -175,7 +175,7 @@ Tags are always `v1`; nothing uses `latest`. {% include excalidraw.html file="31-minikube-image-flow" alt="A Containerfile feeds docker build, which uses the host BuildKit cache and produces an image named localhost/cndp-05/language-service:v1 in the Docker image store. minikube image load copies it into containerd in the node container under the same name, and the pod runs it with imagePullPolicy Never." - caption="Figure R.2 — Images are built on the host and loaded into the node; pods never pull them" %} + caption="Figure R.2: Images are built on the host and loaded into the node; pods never pull them" %} ## Run it @@ -274,4 +274,4 @@ consumer group resumes from its last committed offset. The manifests and scripts are in [`examples/05-event-driven/k8s/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/05-event-driven/k8s/). --- -*Verification status: verified — [`examples/05-event-driven/k8s/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/05-event-driven/k8s/) passes 10/10 checks in each of the six languages on minikube v1.39.0 (Docker driver, containerd, Kubernetes v1.36.5, Strimzi 1.2.0, 2026-10-13, on the code after the S13 and S14 refactors), using the same `verify.sh` as compose with `RUNTIME=kubernetes`. No pod restarted or was OOM-killed under the manifests' memory limits. The consumer scale-down and scale-up in the cross-check was run by hand and is not part of the script.* +*Verification status: verified: [`examples/05-event-driven/k8s/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/05-event-driven/k8s/) passes 10/10 checks in each of the six languages on minikube v1.39.0 (Docker driver, containerd, Kubernetes v1.36.5, Strimzi 1.2.0, 2026-10-13, on the code after the S13 and S14 refactors), using the same `verify.sh` as compose with `RUNTIME=kubernetes`. No pod restarted or was OOM-killed under the manifests' memory limits. The consumer scale-down and scale-up in the cross-check was run by hand and is not part of the script.* diff --git a/_docs/32-appendix-s-openshift-local.md b/_docs/32-appendix-s-openshift-local.md index ed3184b7..b657be2e 100644 --- a/_docs/32-appendix-s-openshift-local.md +++ b/_docs/32-appendix-s-openshift-local.md @@ -58,7 +58,7 @@ web console login, run `crc console --credentials` yourself. {% include excalidraw.html file="32-openshift-topology" alt="The host runs curl and verify.sh, a browser and oc with the crc-admin context. Inside OpenShift Local there are two namespaces. cndp-amq-streams holds an OLM Subscription with Manual approval and a pinned CSV, and the AMQ Streams operator, which manages the Kafka cluster. cndp-05 holds order-service behind an edge-TLS Route, the shipping and notification consumers, Kafka 4.2.0, Postgres 18 from the sclorg image, the LGTM stack behind a Grafana Route and the Postgres Secret." - caption="Figure S.1 — The operator lives in its own namespace and manages Kafka in cndp-05; the host reaches order-service and Grafana through Routes and uses oc exec for the Kafka and Postgres checks" %} + caption="Figure S.1: The operator lives in its own namespace and manages Kafka in cndp-05; the host reaches order-service and Grafana through Routes and uses oc exec for the Kafka and Postgres checks" %} ## Installing an operator with OLM @@ -271,7 +271,7 @@ from a directory the user owns. {% include excalidraw.html file="32-openshift-image-flow" alt="A Containerfile feeds docker build, which uses the host BuildKit cache and produces an image named localhost/cndp-05/language-service:v1 in the Docker image store. skopeo copy reads it from the docker daemon and pushes it over HTTPS to the internal registry, where it becomes an ImageStream tag v1. The pod pulls it with imagePullPolicy Always. The credential is a ten-minute token for the builder service account, held in a shell variable and passed through a pipe." - caption="Figure S.2 — skopeo pushes from the Docker image store to the internal registry; the credential never touches a file or an argument list" %} + caption="Figure S.2: skopeo pushes from the Docker image store to the internal registry; the credential never touches a file or an argument list" %} The registry needs a credential, and the design keeps it out of every place a secret leaks to: @@ -489,4 +489,4 @@ on the recorded run; read the ReplicaSet's events if you try it. The manifests and scripts are in [`examples/05-event-driven/openshift/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/05-event-driven/openshift/). --- -*Verification status: verified — [`examples/05-event-driven/openshift/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/05-event-driven/openshift/) passes 10/10 checks in each of the five languages on OpenShift Local 4.22.14 (crc 2.64.0, Kubernetes v1.35.6, AMQ Streams `amqstreams.v3.2.1-14`, Kafka 4.2.0, 2026-10-13, on the code after the S13 and S14 refactors), using the same `verify.sh` as compose with `RUNTIME=kubernetes` over HTTPS. Every pod ran under `restricted-v2`, Tempo held a trace with all three services for each language, and teardown left the cluster empty and stopped. The SCC rejection of a fixed root user ID was not exercised.* +*Verification status: verified: [`examples/05-event-driven/openshift/`](https://github.com/patterncatalyst/cloud-native-design-patterns/tree/main/examples/05-event-driven/openshift/) passes 10/10 checks in each of the five languages on OpenShift Local 4.22.14 (crc 2.64.0, Kubernetes v1.35.6, AMQ Streams `amqstreams.v3.2.1-14`, Kafka 4.2.0, 2026-10-13, on the code after the S13 and S14 refactors), using the same `verify.sh` as compose with `RUNTIME=kubernetes` over HTTPS. Every pod ran under `restricted-v2`, Tempo held a trace with all three services for each language, and teardown left the cluster empty and stopped. The SCC rejection of a fixed root user ID was not exercised.*