Skip to content

Commit bdfcd32

Browse files
fix(capstone): published loopback NodePorts; minikube on Docker Engine + containerd (#10)
* chore(plans): plan published loopback NodePorts replacing tunnels * chore(plans): keep host ports; map 127.0.0.1:host:nodePort * chore(plans): no Docker Desktop assumption; isolation rule * ci: add forbidden-syntax gate for tunnels and non-loopback ports * feat(capstone): endpoints lib and show-endpoints for published NodePorts * demo(capstone): group A demos use published NodePorts * demo(capstone): group B demos and walkthrough use published NodePorts * feat(capstone): publish NodePorts on 127.0.0.1 at profile creation * chore(capstone): setup scripts and config text describe published NodePorts * chore(capstone): remove stale nested 17-capstone copy * chore(capstone): delete SSH tunnel machinery * docs: describe published NodePorts; add DRA-017; fix deck trace notes * chore(plans): archive executed nodeports plan * docs(deck): correct stale KEDA HTTP add-on status in speaker notes * chore: untrack bytecode and ignore __pycache__ * fix(capstone): address validation findings for published NodePorts * fix(capstone): second round of NodePort validation fixes * fix(capstone): CRI-O + crun runtime; reset node FORWARD DROP; registry addon at start * docs: CRI-O + crun runtime (DRA-018); lessons on node FORWARD DROP * fix(capstone): local-path default StorageClass so non-root pods can write volumes * docs: record local-path StorageClass in DRA-018 and lessons * chore(plans): plan Docker Engine + containerd migration for the capstone * feat(capstone): docker-backed endpoint helpers and image pipeline lib * feat(capstone): capstone/<svc>:v1 images with pullPolicy Never * feat(capstone): build-image uses docker build + minikube image load * chore(scripts): Docker Engine prereq audit; podman only as optional CRC tooling * demo(capstone): demos check capstone/<svc> images; build + load * feat(capstone): profile on Docker Engine + containerd; cluster scripts use image load * docs: Docker Engine setup; podman only for the optional CRC appendix * docs: DRA-019 Docker Engine + containerd; example README and diagram * docs: onboarding and chapters on Docker Engine; rootless lessons kept as history * ci: scan 5 keeps podman/rootless/CRI-O scoped to the CRC appendix * fix(capstone): set SERVICES after images.sh is sourced * docs(lessons): summarize why minikube moved off rootless podman * fix(capstone): accept LoadBalancer Services (they allocate nodePorts) in endpoint checks * fix(capstone): build-image waits for the Deployments it restarts * fix(services): gRPC clients use the native DNS resolver (c-ares took ~3 s) * fix(capstone): build-image waits for replaced pods to finish terminating * docs: record the live verification on Docker Engine + containerd * fix(capstone): validation round — Fedora/RHEL hosts, gate coverage, build-image robustness, lesson tense * fix(capstone): clearer VM-engine note * chore(plans): archive executed docker migration plan
1 parent c0dc264 commit bdfcd32

122 files changed

Lines changed: 2349 additions & 2214 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/checks.yml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
name: checks
2+
3+
on:
4+
pull_request:
5+
push:
6+
branches: [main]
7+
8+
permissions:
9+
contents: read
10+
11+
jobs:
12+
checks:
13+
runs-on: ubuntu-latest
14+
steps:
15+
- name: Check out repo
16+
uses: actions/checkout@v5
17+
18+
- name: Forbidden syntax
19+
run: bash scripts/forbidden-syntax.sh
20+
21+
- name: Shell syntax check
22+
run: |
23+
find examples/lgtm-datamesh/scripts examples/lgtm-datamesh/demos scripts \
24+
-name '*.sh' -type f -print0 | xargs -0 -n1 bash -n

‎.github/workflows/pages.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,9 @@ jobs:
2727
- name: Check out repo
2828
uses: actions/checkout@v5
2929

30+
- name: Forbidden syntax
31+
run: bash scripts/forbidden-syntax.sh
32+
3033
- name: Set up Ruby
3134
uses: ruby/setup-ruby@v1
3235
with:

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,7 @@ Gemfile.lock
1010
# Node.js
1111
node_modules/
1212
package-lock.json
13+
14+
# Python bytecode
15+
__pycache__/
16+
*.pyc

‎Gemfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,5 +13,5 @@ group :jekyll_plugins do
1313
gem "jekyll-seo-tag" # provides {% seo %} — used in _layouts/default.html
1414
end
1515

16-
# Windows and macOS sometimes need this explicitly
16+
# Local `jekyll serve` on Ruby 3+ needs webrick explicitly
1717
gem "webrick", "~> 1.8"

‎PRD.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -113,14 +113,14 @@ forks.
113113
- A reader who runs the example tree gets a working data mesh on their
114114
laptop: five services, contracts, catalog, canary, autoscaling,
115115
observability, all green within ~30 minutes of bootstrap on the
116-
verified configuration.
116+
target configuration.
117117
- A reader can demonstrate the cross-product behaviours that make a
118118
collection of services a *mesh*: a single GraphQL query producing a
119119
trace tree across three products, a contract canary shifting live
120120
traffic by weight without application code changes, a Kafka-lag
121121
scaler waking and quiescing an event consumer.
122-
- The runnable examples build and run end to end on a Fedora-on-rootless-podman
123-
minikube setup with one bootstrap command. CAP-047 of the historical
122+
- The runnable examples build and run end to end on a Fedora 44 + Docker Engine + minikube
123+
(docker driver, containerd) setup with one bootstrap command. CAP-047 of the historical
124124
decision log documents the one deferred component (the KEDA HTTP
125125
add-on's interceptor on the gateway scale-from-zero path) and the
126126
worked-around demo path that keeps the walkthrough 5-of-5 green.
@@ -147,11 +147,11 @@ forks.
147147
Snowflake vs. BigQuery vs. Databricks, dbt vs. SQLMesh, or warehouse
148148
shape. Those choices are downstream of the mesh; this reference stops
149149
at the boundary.
150-
- **Coverage of every Kubernetes runtime.** The verified configuration
150+
- **Coverage of every Kubernetes runtime.** The target configuration
151151
is minikube on Fedora 44. The reference is portable in principle to
152152
OpenShift (where the implementation deck pairs it explicitly), EKS,
153153
GKE, AKS, and vanilla K8s, but the bootstrap and chart values are
154-
written for the verified runtime. Adapting to other runtimes is left
154+
written for the target runtime. Adapting to other runtimes is left
155155
to readers as a known and welcome contribution path.
156156
- **Building a generic "data mesh framework."** This is a reference
157157
implementation, not a framework. There's no abstraction layer
@@ -263,7 +263,7 @@ diagrams that have changed.
263263
### Verification metrics (project-controlled)
264264

265265
- The presenter walkthrough runs 5-of-5 green on the verified runtime
266-
(Fedora 44 + rootless podman + minikube), as documented in the
266+
(Fedora 44 + Docker Engine + minikube (docker driver, containerd)), as documented in the
267267
reconciliation file at `_plans/reconciliation.md`.
268268
- All internal links resolve (`scripts/check-cross-references.sh` exits 0).
269269
- All `{{ }}` Liquid expressions in `_plans/*.md` are properly fenced
@@ -293,7 +293,7 @@ day-to-day decisions, but are useful as a long-range health check:
293293
- **Kubernetes is the substrate model.** Anything that contradicts
294294
upstream Kubernetes idioms (operators, CRDs, RBAC, service mesh) is
295295
out of scope.
296-
- **The verified runtime is minikube on Fedora 44 with rootless podman.**
296+
- **The verified runtime is Fedora 44 + Docker Engine + minikube (docker driver, containerd)** (2026-10-09, engine provided by Docker Desktop; see `_plans/reconciliation.md`).
297297
Other runtimes are documented as portable in principle; verifying
298298
them is a contribution path.
299299
- **Vendor-neutral language.** Where multiple tools could fit a slot

‎README.md‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -81,8 +81,8 @@ end to end, verified order-independent (see
8181
|---|---|---|
8282
| Services | Python 3.12, FastAPI, SQLAlchemy + Alembic, Poetry | Five domain services + GraphQL gateway; migrations via init-containers |
8383
| Protocols | REST (OpenAPI), gRPC (Protobuf/buf), GraphQL (SDL), Kafka (Avro) | The deliberate protocol mix: cross-product APIs, hot paths, read surface, async spine |
84-
| Packaging | Helm (umbrella chart + per-service subcharts), podman | One chart tree for the whole system; images built on the host, pushed to the in-cluster registry |
85-
| Substrate | Kubernetes v1.32 on minikube (rootless podman driver, containerd) | The whole system on one 24 GB profile |
84+
| Packaging | Helm (umbrella chart + per-service subcharts), Docker Engine | One chart tree for the whole system; images built on the host and loaded into the minikube node |
85+
| Substrate | Kubernetes v1.32 on minikube (Docker driver, containerd + runc) | The whole system on one 24 GB profile |
8686
| Database | CloudNativePG operator, PostgreSQL | One shared cluster, schema-per-service ownership |
8787
| Events | Strimzi operator, Apache Kafka | The asynchronous spine (`order-placed` topic) |
8888
| Contracts | Apicurio Registry | OpenAPI + Protobuf + GraphQL SDL (discovery) and Avro (runtime, Confluent-compatible API) |
@@ -130,9 +130,11 @@ cd examples/lgtm-datamesh
130130
./demos/walkthrough.sh # the five-act presenter walkthrough
131131
```
132132

133-
The verified configuration is Fedora with rootless podman, 64 GB RAM
133+
The verified configuration is Fedora 44 with Docker Engine, 64 GB RAM
134134
(24 GB for the minikube profile), and minikube ≥ 1.36.
135135

136+
An optional OpenShift Local (CRC) appendix uses podman. <!-- forbidden-ok -->
137+
136138
## Origin
137139

138140
This repo is a fork of the §17 capstone from

‎_docs/02-kubernetes-substrate.md‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,17 @@ a laptop but also concentrates failure modes that a real multi-node cluster woul
100100
spread out. Where those single-node realities bite — resource ceilings, node-level
101101
decay, the operational care a long-lived single-node cluster needs — they're collected
102102
as operational gotchas rather than mixed into the conceptual material here, because
103-
they're particular to this deployment choice rather than to data mesh as a pattern.
103+
they're particular to this deployment choice rather than to data mesh as a pattern. The
104+
profile is a Docker container (the docker driver) running containerd with the runc
105+
runtime, sized by `MINIKUBE_MEMORY`, `MINIKUBE_CPUS` and `MINIKUBE_DISK`.
106+
107+
Host access follows the same single-node logic. The capstone's UIs and service
108+
endpoints are NodePorts published on `127.0.0.1` when the minikube profile is created
109+
(`--ports=127.0.0.1:<hostPort>:<nodePort>`), never SSH tunnels or port-forwarding. <!-- forbidden-ok -->
110+
The ports are fixed at creation, so changing the set means recreating the profile with
111+
`./scripts/setup-capstone-profile.sh --replace`. `--replace` deletes and recreates the cluster, so run `./scripts/bootstrap-capstone.sh` again afterwards. Run the workshop in isolation: shut down
112+
CRC, other minikube profiles, and other workloads first, so the host ports are free.
113+
`./scripts/show-endpoints.sh` prints what is published and reachable.
104114

105115
Next, the services themselves: what a data product looks like in this build, the
106116
order-service template the others follow, and how each one is packaged and shipped.

‎_docs/03-services-and-data-products.md‎

Lines changed: 22 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -83,37 +83,29 @@ start rather than retrofitting it.
8383
## The one part that fights back: getting images to the kubelet
8484

8585
This is worth its own section because it's the single part of the capstone that
86-
reliably trips people up, and it's a direct consequence of a deliberate choice made
87-
back in §3: the capstone uses the **rootless-podman driver with the containerd
88-
runtime**, because that's the most realistic local mirror of how Kubernetes runs in
89-
production. The cost of that realism is that getting a locally-built image to the
90-
kubelet is not as simple as you'd expect.
91-
92-
The intuitive approaches are unreliable on this driver. `minikube image build` can
93-
exit successfully without the image actually landing in the profile's containerd
94-
store, so the pod then fails to pull. `minikube image load` can report "image not
95-
found" for an image that's plainly present, because the lookup goes through the
96-
rootless podman socket in a way that doesn't resolve.
97-
98-
The reliable answer the capstone standardizes on is **minikube's built-in registry
99-
addon**: build on the host with podman, push to the registry, and let deployments pull
100-
from it like any ordinary image. The one detail that catches everyone is that the
101-
registry has *two addresses*. With the podman driver the host-side port is not 5000 —
102-
minikube assigns one (something like `127.0.0.1:41685`) and tells you when you enable
103-
the addon — while *inside* the cluster the kubelet reaches the same registry at
104-
`localhost:5000`. So you **push** from the host to `127.0.0.1:<assigned-port>` and the
105-
cluster **pulls** from `localhost:5000`. The build script discovers the host port
106-
automatically; the charts pull from the in-cluster address.
107-
108-
One environment variable makes or breaks all of this: `MINIKUBE_ROOTLESS=true`. If
109-
it's not set in your shell, minikube routes host operations through `sudo podman`,
110-
which can't see your rootless container — producing a spread of failures that look
111-
like a broken cluster but aren't. The capstone scripts both persist it and export it
112-
at the top of every script; if you run minikube commands by hand, export it first.
113-
114-
None of this is unique to the capstone — it's inherent to the rootless driver — but
86+
reliably trips people up. The profile is a Docker container (the docker driver) running
87+
containerd, and that container has its own image store, separate from the Docker Engine
88+
on your host. An image you build locally is invisible to the kubelet until you put it
89+
there, and a pod that can't find its image fails in ways that look like a broken
90+
cluster.
91+
92+
The capstone's answer is deliberately simple: **build with Docker Engine, then load the
93+
image into the profile with `minikube image load`**. There is no registry to run and no
94+
addresses to reconcile, and it works the same on a native engine and on a VM-based one
95+
(Docker Desktop, for example). That portability is the reason for the choice:
96+
pushing to a registry on the host's loopback only works when the engine's daemon shares
97+
the host's network, which a VM-based engine's does not.
98+
99+
Images are tagged `capstone/<svc>:v1` and the charts set `imagePullPolicy: Never`. The
100+
policy matters in both directions. If an image was never loaded, the pod fails at once
101+
with `ErrImageNeverPull` instead of quietly trying Docker Hub for a name that exists
102+
nowhere. And because `:v1` is a mutable tag, a rebuild does not change what running pods
103+
use, so `scripts/build-image.sh` restarts the Deployments that use the image it just
104+
loaded.
105+
106+
None of this is unique to the capstone — any local cluster has some version of the gap between the host's images and the node's — but
115107
the capstone is where it bites, because it's the first place you build and deploy your
116-
*own* images at scale. Get the registry workflow right once here, and every service
108+
*own* images at scale. Get the image workflow right once here, and every service
117109
afterward is the same three commands. (The deeper operational sharp edges of running
118110
all this on a single node are collected as gotchas, separate from this build-level
119111
friction.)

‎_docs/06-progressive-delivery-mtls.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,8 +37,8 @@ A couple of honest simplifications worth naming. v1 and v2 here run the *same im
3737
with an environment toggle and a different `version` label, rather than two images from
3838
two commits — that keeps the demo focused on the traffic-management mechanism rather
3939
than an image pipeline, and the Istio mechanics are identical either way. And traffic
40-
enters through the ingress gateway (reached over an SSH tunnel to its NodePort, under
41-
the rootless-podman setup from §11) rather than from a meshed client, so no client needs
40+
enters through the ingress gateway (reached on its NodePort, published to
41+
127.0.0.1:8088 when the profile is created, DRA-017) rather than from a meshed client, so no client needs
4242
to join the mesh for the canary to work.
4343

4444
This was verified end to end: with the split set to 90/10, a run of 100 requests landed

‎_docs/11-running-on-openshift-crc.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ duration: 40 min
77

88
[Chapter 3]({{ '/docs/02-kubernetes-substrate/' | relative_url }}) built the case for
99
Kubernetes as the mesh's substrate and brought the capstone up on minikube via Helm —
10-
NodePorts, a stable SSH tunnel, a local image registry. That's a real deployment, and
10+
NodePorts published to loopback, a local image registry. That's a real deployment, and
1111
everything in the reading set up to the [summary]({{ '/docs/10-summary/' | relative_url }})
1212
assumes it. But "deploy it to Kubernetes" and "deploy it to OpenShift" are not the same
1313
sentence. OpenShift adds opinions on top of plain Kubernetes — Security Context
@@ -45,6 +45,8 @@ documenting the OpenShift-native counterpart.
4545
You don't need a paid subscription or a cloud account. **OpenShift Local** (CRC —
4646
"CodeReady Containers") runs a real, single-node OpenShift cluster in a local VM.
4747

48+
This appendix uses podman (and `oc`) on the host to build and push to the integrated registry; the minikube path uses Docker Engine. Podman is needed only here. <!-- forbidden-ok -->
49+
4850
1. **Create a free Red Hat Developer account.** Go to
4951
[developers.redhat.com](https://developers.redhat.com), register or log in. It's free
5052
and gives you access to OpenShift Local and the pull secret.
@@ -168,12 +170,13 @@ datamesh-infra`, which is worth knowing when debugging a pod stuck in
168170
`CreateContainerConfigError` with an SCC-related event — it's the same RoleBinding under
169171
the hood.
170172

171-
### 2. Routes, not port-forward/NodePort
173+
### 2. Routes, not published NodePorts
172174

173175
The minikube path this capstone uses elsewhere in the reading set exposes services via
174-
NodePort, reached over a stable SSH tunnel because minikube has no cluster-native router.
175-
OpenShift has one built in: a `Route` object hands a Service a real external hostname,
176-
served by the cluster's router, with no tunnel to keep alive. The chart defines two —
176+
NodePorts published to `127.0.0.1` when the profile is created, because minikube has no
177+
cluster-native router. OpenShift has one built in: a `Route` object hands a Service a
178+
real external hostname, served by the cluster's router, with nothing to publish on the
179+
host. The chart defines two —
177180
`templates/route.yaml` ranges over the same `services:` map and emits a Route for any
178181
service with `route: true` set (currently just `graphql-gateway`), and `apicurio.yaml`
179182
wires its own Route directly since Apicurio isn't part of that map:

0 commit comments

Comments
 (0)