Skip to content

Commit 764aa3f

Browse files
fix(lgtm-datamesh): demos survive KEDA scale-to-zero of gateway and consumer
bootstrap-capstone.sh applies the KEDA scalers (gateway HTTPScaledObject, notification ScaledObject), but several demos were written before that and assume both services always have a live pod: - walkthrough act 1 and smoke-discovery port-forward straight to the graphql-gateway Service (the CAP-046 interceptor bypass), which has no pod when KEDA has it at zero — and nothing on this path ever wakes it. Wake it explicitly (scale to 1 + rollout wait) when availableReplicas is 0; KEDA re-adopts it afterwards. Relax the walkthrough preflight from 'Deployment available' to 'Deployment exists' to match. - smoke-avro and smoke-notifications port-forward notification-service and poll /received. A port-forward pins the pod it attached to; helm sets replicas=1, KEDA reconciles to 0 within its 5s poll, and the tested event then wakes a NEW pod — so the smoke polls a dead tunnel while '|| echo []' masks every failure as 'not consumed yet' (the event was in fact persisted mid-window; confirmed in Postgres). Re-attach the port-forward when curl fails, and widen the poll window from ~60s to ~180s to cover lag-poll + pod start + group join. - smoke-notifications failure dump: show the Deployment and ScaledObject before the pod list, so an empty list reads as scale-to-zero rather than a missing deploy. Verified live with both scalers applied and both services at zero: smoke-discovery, smoke-avro, smoke-notifications, and walkthrough --only trace all pass, with the wake/re-attach paths exercised. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent b320e20 commit 764aa3f

4 files changed

Lines changed: 65 additions & 10 deletions

File tree

‎examples/lgtm-datamesh/demos/smoke-avro.sh‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -122,16 +122,28 @@ printf ' ✓ schema registered\n'
122122

123123
# ── 7b. assert the event was consumed (proves Avro decode via registry) ───────
124124
step "Polling notification-service /received for the decoded order.placed event"
125+
# ~180s window, not ~60s: when the bootstrap-applied KEDA ScaledObject has
126+
# notification-service scaled to zero, this very event is what wakes it —
127+
# KEDA's kafka lag poll + pod start + consumer-group join must fit here.
125128
seen=0
126-
for i in $(seq 1 30); do
127-
RECV="$(curl -fsS "http://127.0.0.1:${LOCAL_NOTIF}/received" 2>/dev/null || echo '[]')"
129+
for i in $(seq 1 90); do
130+
if ! RECV="$(curl -fsS "http://127.0.0.1:${LOCAL_NOTIF}/received" 2>/dev/null)"; then
131+
# A port-forward pins the pod it attached to at start. With the KEDA
132+
# ScaledObject applied, that pod can be replaced or scaled away at any
133+
# moment (helm sets replicas=1, KEDA reconciles to 0, the tested event
134+
# wakes a NEW pod) — leaving us polling a dead tunnel forever. Re-attach
135+
# to the Service and treat this attempt as "not yet".
136+
kill "$PF_N" 2>/dev/null
137+
kubectl port-forward -n "$NS" service/notification-service "${LOCAL_NOTIF}:80" >/dev/null 2>&1 & PF_N=$!
138+
RECV='[]'
139+
fi
128140
if printf '%s' "$RECV" | python3 -c "import sys,json; d=json.load(sys.stdin); sys.exit(0 if any(e.get('order_id')=='$ORDER_ID' and e.get('event_type')=='order.placed' and e.get('item_sku')=='WIDGET-001' for e in d) else 1)" 2>/dev/null; then
129141
printf ' ✓ notification decoded order.placed for %s (after ~%ds)\n' "$ORDER_ID" "$((i*2))"
130142
seen=1; break
131143
fi
132144
sleep 2
133145
done
134-
(( seen )) || fail "notification did not consume/decode the Avro event within ~60s"
146+
(( seen )) || fail "notification did not consume/decode the Avro event within ~180s"
135147

136148
printf '\n✓ SUCCESS — order.placed flows as registered Avro: schema in Apicurio + event decoded by the consumer via the registry\n'
137149

‎examples/lgtm-datamesh/demos/smoke-discovery.sh‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,17 @@ for svc in "${DEPLOY[@]}"; do
8989
kubectl rollout status "deployment/${svc}" -n "$NS" --timeout=120s || fail "${svc} rollout failed"
9090
done
9191

92+
# The SDL fetch port-forwards straight to the gateway Service, but the
93+
# bootstrap-applied KEDA HTTPScaledObject scales the gateway to zero when idle
94+
# (rollout status on a 0-replica deployment succeeds instantly, so the loop
95+
# above doesn't catch it). Wake it by hand; KEDA re-adopts it afterwards.
96+
gw_avail="$(kubectl get deploy graphql-gateway -n "$NS" -o jsonpath='{.status.availableReplicas}' 2>/dev/null)"
97+
if ! [[ "$gw_avail" =~ ^[1-9][0-9]*$ ]]; then
98+
step "graphql-gateway is scaled to zero — waking it for the SDL fetch"
99+
kubectl scale deploy graphql-gateway -n "$NS" --replicas=1 >/dev/null 2>&1
100+
kubectl rollout status deployment/graphql-gateway -n "$NS" --timeout=120s || fail "graphql-gateway did not wake"
101+
fi
102+
92103
# ── port-forwards ─────────────────────────────────────────────────────────────
93104
step "Port-forwards: order(${LOCAL_ORDER}) gateway(${LOCAL_GW}) apicurio(${LOCAL_APIC})"
94105
kubectl port-forward -n "$NS" service/order-service "${LOCAL_ORDER}:80" >/dev/null 2>&1 & PF_O=$!

‎examples/lgtm-datamesh/demos/smoke-notifications.sh‎

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@ step() { printf '\n==> %s\n' "$1"; }
3535
fail() {
3636
printf '\nFAILED: %s\n' "$1" >&2
3737
printf '\n--- notification-service pods (incl. init container) ---\n' >&2
38+
# Deployment + ScaledObject first: an empty pod list here usually means
39+
# KEDA has it scaled to zero, not that the deploy is missing.
40+
kubectl get deploy notification-service -n "$NS" 2>&1 || true
41+
kubectl get scaledobject -n "$NS" 2>&1 || true
3842
kubectl get pods -n "$NS" -l "app.kubernetes.io/name=notification-service" -o wide 2>&1 || true
3943
kubectl logs -n "$NS" -l "app.kubernetes.io/name=notification-service" -c migrate --tail=40 2>&1 || true
4044
kubectl logs -n "$NS" -l "app.kubernetes.io/name=notification-service" --tail=30 2>&1 || true
@@ -46,7 +50,16 @@ fail() {
4650
check_received() {
4751
# $1 = order_id to look for in /received
4852
local recv
49-
recv="$(curl -fsS "http://127.0.0.1:${LOCAL_NOTIF}/received" 2>/dev/null || echo '[]')"
53+
if ! recv="$(curl -fsS "http://127.0.0.1:${LOCAL_NOTIF}/received" 2>/dev/null)"; then
54+
# A port-forward pins the pod it attached to at start. With the KEDA
55+
# ScaledObject applied, that pod can be replaced or scaled away at any
56+
# moment (helm sets replicas=1, KEDA reconciles to 0, the tested event
57+
# wakes a NEW pod) — leaving us polling a dead tunnel forever. Re-attach
58+
# to the Service and report "not yet" for this attempt.
59+
kill "$PF_N" 2>/dev/null
60+
kubectl port-forward -n "$NS" service/notification-service "${LOCAL_NOTIF}:80" >/dev/null 2>&1 & PF_N=$!
61+
return 1
62+
fi
5063
printf '%s' "$recv" | python3 -c "import sys,json; d=json.load(sys.stdin); sys.exit(0 if any(e.get('order_id')=='$1' for e in d) else 1)" 2>/dev/null
5164
}
5265

@@ -120,12 +133,15 @@ ORDER_ID="$(curl -fsS -X POST "http://127.0.0.1:${LOCAL_ORDER}/orders" -H 'Conte
120133
printf ' order id=%s\n' "$ORDER_ID"
121134

122135
step "Waiting for the notification to be persisted (/received, DB-backed)"
136+
# ~180s window, not ~60s: when the bootstrap-applied KEDA ScaledObject has
137+
# notification-service scaled to zero, this very event is what wakes it —
138+
# KEDA's kafka lag poll + pod start + consumer-group join must fit here.
123139
seen=0
124-
for i in $(seq 1 30); do
140+
for i in $(seq 1 90); do
125141
if check_received "$ORDER_ID"; then printf ' ✓ persisted after ~%ds\n' "$((i*2))"; seen=1; break; fi
126142
sleep 2
127143
done
128-
(( seen )) || fail "notification was not persisted within ~60s"
144+
(( seen )) || fail "notification was not persisted within ~180s"
129145

130146
# ── durability: restart notification, the row must still be there ─────────────
131147
step "Restarting notification-service to prove durability (in-memory would lose it)"

‎examples/lgtm-datamesh/demos/walkthrough.sh‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -208,10 +208,13 @@ preflight() {
208208
if want_act trace; then
209209
# trace bypasses the interceptor and port-forwards directly to the
210210
# graphql-gateway Service — see CAP-046. Confirm the Deployment + Service
211-
# are both there so the port-forward has something to attach to.
212-
check "graphql-gateway Deployment available" \
213-
"kubectl -n $NS get deploy graphql-gateway -o jsonpath='{.status.availableReplicas}' 2>/dev/null | grep -qE '^[1-9][0-9]*$'" \
214-
"kubectl -n $NS rollout status deploy/graphql-gateway (or re-run bootstrap-capstone.sh)"
211+
# are both there so the port-forward has something to attach to. Zero
212+
# available replicas is fine here: bootstrap applies the KEDA
213+
# HTTPScaledObject, which scales the gateway to zero when idle, and the
214+
# trace act wakes it before port-forwarding.
215+
check "graphql-gateway Deployment exists" \
216+
"kubectl -n $NS get deploy graphql-gateway >/dev/null 2>&1" \
217+
"helm upgrade --install graphql-gateway charts/capstone/charts/graphql-gateway -n $NS (or re-run bootstrap-capstone.sh)"
215218
check "graphql-gateway Service exists" \
216219
"kubectl -n $NS get svc graphql-gateway >/dev/null 2>&1" \
217220
"kubectl -n $NS get svc graphql-gateway (or re-run bootstrap-capstone.sh)"
@@ -287,6 +290,19 @@ trace_act() {
287290
local pf_pid=0 result=0
288291
local query='{"query":"{ order(id: \"trace-probe\") { id itemSku quantity stock { sku quantityOnHand available } } }"}'
289292

293+
# The direct port-forward needs a live pod behind the Service, but the KEDA
294+
# HTTPScaledObject (applied by bootstrap) scales the gateway to zero when
295+
# idle — and only interceptor traffic wakes it, which is exactly the path
296+
# this act bypasses (CAP-046). Wake it by hand; KEDA reconciles afterwards.
297+
local avail
298+
avail="$(kubectl -n "$NS" get deploy graphql-gateway -o jsonpath='{.status.availableReplicas}' 2>/dev/null)"
299+
if ! [[ "$avail" =~ ^[1-9][0-9]*$ ]]; then
300+
info "graphql-gateway is scaled to zero — waking it (KEDA re-adopts it after the act)"
301+
kubectl -n "$NS" scale deploy graphql-gateway --replicas=1 >/dev/null 2>&1
302+
kubectl -n "$NS" rollout status deploy/graphql-gateway --timeout=120s >/dev/null 2>&1 \
303+
|| { narrate "✗ graphql-gateway did not become Ready after wake"; return 1; }
304+
fi
305+
290306
info "starting port-forward: graphql-gateway Service (capstone) → 127.0.0.1:8080"
291307
info " (bypasses keda-add-ons-http-interceptor-proxy — see CAP-046)"
292308
kubectl port-forward -n "$NS" svc/graphql-gateway 8080:80 >/dev/null 2>&1 &

0 commit comments

Comments
 (0)