demo_qa: live situation kit and plugin (accepted on a live run)

demo/PRESENTER.md

@@ -0,0 +1,45 @@
+# demo_qa presenter sheet: manual test plan to Playwright
+
+**Goal:** turn the bank's 60-row manual regression plan into a Playwright suite that runs in CI on MySQL 8, live, in about 17 minutes. Every row is accounted for. Devin does not invent an expected result for MT-047 (negative transfer); it flags the money bug and holds the row for QA.
+
+## Links
+| What | Link |
+|---|---|
+| **Live session (already started with prompt 1)** | the session link your setup session sent you (the setup session starts it with prompt 1) |
+| The 60-row plan | https://github.com/COG-GTM/Springboot-BankApp/blob/demo/qa-live-before/qa/manual-test-plan.csv |
+| MT-047, the hard case (line 48) | https://github.com/COG-GTM/Springboot-BankApp/blob/demo/qa-live-before/qa/manual-test-plan.csv#L48 |
+| BEFORE and AFTER videos (kit PR #335 description, does not close on reseed) | https://github.com/COG-GTM/Springboot-BankApp/pull/335 |
+| Pre-built suite: 54 automated, CI green | https://github.com/COG-GTM/Springboot-BankApp/pull/328 |
+| Pre-built coverage matrix (MT-047 row) | https://github.com/COG-GTM/Springboot-BankApp/blob/devin/fs-testing-1791848695-manual-to-automated/qa/coverage-matrix.md |
+| Pre-built planted regression: CI red on MT-040 only, 53 pass | https://github.com/COG-GTM/Springboot-BankApp/pull/329 |
+
+## Beats
+| Time | Beat | Use |
+|---|---|---|
+| 0:00 | "60 rows, run by hand every release. There is no `e2e/` folder." Open the plan. | The 60-row plan |
+| 0:01 | Optional: show the app working before any automation, and the AFTER clip of the 54-test suite running. | BEFORE / AFTER videos on #335 |
+| 0:02 | Open the live session. Prompt 1 is already in it as the first message; show it to the room. | Live session |
+| 0:04 | Devin reads the plan and the app, brings up MySQL 8 and the app. Point at its plan. | Live session |
+| 0:05 to ~0:17 | Read MT-047 aloud: "System handles it appropriately." Ask the room what a tester would assert. Check back every few minutes. | MT-047 link |
+| ~0:17 | Live PR lands against `demo/qa-live-before`. Open the coverage matrix: all 60 rows have a status. | Live PR (from the session). Running late: #328 |
+| ~0:18 | **Wow moment:** the MT-047 row. "A minus 50 transfer moves money the wrong way, the sender gains 50. Devin did not write a test that blesses that, and did not make up a rule. It held the row and flagged the bug for QA." | Live PR matrix. Pre-built: coverage matrix link |
+| ~0:20 to ~0:26 | Optional prompt 2 in the same session. CI goes red on exactly MT-040, the other 53 stay green. Devin asks keep the rule or drop the change; answer "drop it". | Live stacked PR. Short on time: #329 |
+
+## Prompt 1 (already sent to the live session as its first message)
+```
+Repo: COG-GTM/Springboot-BankApp, branch demo/qa-live-before. Branch off it and open your PR against demo/qa-live-before.
+
+Here is our 60-row manual regression plan (qa/manual-test-plan.csv). Automate every row you responsibly can as Playwright tests that run in CI against MySQL 8, publish a coverage matrix that accounts for all 60 rows, and tell me which rows you could not automate and why. If a row's expected result is not testable, do not invent one.
+```
+Expect: about 54 automated, about 5 not automatable with a reason each, MT-047 held. Counts can move by a row or two; all 60 accounted for and MT-047 held must always be true.
+
+## Prompt 2 (optional, same session, after prompt 1's PR is up)
+```
+On a new branch off your PR branch, change AccountService so a customer cannot transfer their full balance (< 0 to <= 0), and open a PR against your PR branch. Tell me which plan rows CI flags.
+```
+Expect: CI red about 2.5 min after the PR opens, on MT-040 only.
+
+## If the live run stalls
+Show #328 (suite, green) and #329 (planted regression, red on MT-040 only). To run the finished state locally: `git checkout devin/fs-testing-1791848695-manual-to-automated && demo/setup-demo.sh`.
+
+Honesty notes: the app is the open-source Springboot BankApp; the plan was written for this demo.

demo/PROMPTS.md

@@ -0,0 +1,33 @@
+# demo_qa prompts (live Devin)
+
+Presenter sheet with goal, links, beats and which PR to use: `demo/PRESENTER.md`.
+
+Before the meeting: run `demo/seed-situation.sh` from this kit branch and wait for `READY`.
+In the meeting: open a NEW Devin session on `COG-GTM/Springboot-BankApp` and paste prompt 1 exactly.
+
+## Prompt 1: manual plan to automated suite (live, about 17 min wall time measured)
+
+```
+Repo: COG-GTM/Springboot-BankApp, branch demo/qa-live-before. Branch off it and open your PR against demo/qa-live-before.
+
+Here is our 60-row manual regression plan (qa/manual-test-plan.csv). Automate every row you responsibly can as Playwright tests that run in CI against MySQL 8, publish a coverage matrix that accounts for all 60 rows, and tell me which rows you could not automate and why. If a row's expected result is not testable, do not invent one.
+```
+
+Expected back:
+- A PR against `demo/qa-live-before` with `e2e/` (Playwright, TypeScript), a GitHub Actions workflow running against a MySQL 8 service, and a coverage matrix accounting for all 60 rows.
+- Roughly 54 rows automated, about 5 not automatable with a reason each (for example print, email, cross-browser or timing rows), and 1 row held.
+- The hard case: `MT-047` (negative transfer). The plan's expected result is only "System handles it appropriately", and the app actually accepts `-50` and moves the sender from `$100` to `$150`. Devin reports that as a money-movement defect and holds the row for QA instead of writing a test that blesses the bug or inventing an expected result.
+
+Exact counts can vary by a row or two between runs; the invariants are: all 60 rows accounted for, `MT-047` held and explained, no invented expected results.
+
+## Prompt 2: planted regression (optional, run in the same session after prompt 1 finishes)
+
+```
+On a new branch off your PR branch, change AccountService so a customer cannot transfer their full balance (< 0 to <= 0), and open a PR against your PR branch. Tell me which plan rows CI flags.
+```
+
+Expected back: CI on that PR fails on exactly the full-balance transfer case (`MT-040` in the plan), and Devin says so.
+
+## If the live run stalls
+
+Switch to the pre-built fallback: `git checkout devin/fs-testing-1791848695-manual-to-automated && demo/setup-demo.sh`, then show PR #328 (suite) and PR #329 (planted regression).

demo/TALK-TRACK-LIVE.md

@@ -0,0 +1,19 @@
+# demo_qa live talk track
+
+Situation: the bank has a 60-row manual regression plan run by hand every release. No automation exists. The presenter asks Devin, live, to automate it.
+
+| Time | Beat | Live or pre-built |
+|---|---|---|
+| 0:00 | Show `qa/manual-test-plan.csv` on `demo/qa-live-before`. "60 rows, run by hand every release. There is no `e2e/` folder." | Live (GitHub) |
+| 0:02 | Open a new Devin session, paste prompt 1 from `PROMPTS.md`. | Live |
+| 0:04 | Devin reads the plan and the app, runs `demo/setup-demo.sh`, brings MySQL 8 and the app up. Point at the plan Devin writes. | Live |
+| 0:05 to ~0:17 | While Devin works, open `qa/manual-test-plan.csv` and read MT-047 aloud: "System handles it appropriately." Ask the room what a tester would assert. Check back every few minutes. | Live session running |
+| ~0:17 | Live PR lands (measured 16 min 32 s from paste) against `demo/qa-live-before`. Open the coverage matrix: every one of 60 rows has a status. | Live |
+| ~0:18 | Wow moment: the `MT-047` row. "The plan only says the system handles a negative transfer appropriately. Devin tried it: a minus 50 transfer moves money the wrong way, the sender gains 50. Devin did not write a test that blesses that, and did not make up a new rule. Devin held the row and flagged the money bug for QA." | Live |
+| ~0:20 to ~0:26 | Optional prompt 2 (about 5 min when CI is not flaky; CI itself takes about 2.5 min): planted regression on a stacked PR. CI goes red on exactly MT-040, the full-balance transfer, and the other 53 rows stay green. Hard case: Devin does not edit MT-040 to make CI pass. Devin asks whether to keep the new rule (and update MT-040) or drop the change. Answer "drop it". If time is short, show PR #329 (pre-built, red on MT-040 only). Live-run PRs like #332 and #336 close when the seed is reset. | Live or pre-built |
+
+Hard case: `MT-047`. Devin must hold and escalate, not invent an expected result.
+
+Fallback: if the live session stalls or the network misbehaves, `git checkout devin/fs-testing-1791848695-manual-to-automated && demo/setup-demo.sh` gives the finished state in about 5 minutes, and PRs #328 and #329 show the same outcome.
+
+Honesty notes: the app is the open-source Springboot BankApp; the manual plan was written for this demo. Exact automated/not-automatable counts can differ by a row or two between live runs.

demo/live/AGENTS.md

@@ -0,0 +1,8 @@
+# Agent notes for this repository
+
+- Stack: Spring Boot 3 / Java 17, Thymeleaf, MySQL 8. App listens on http://localhost:8080.
+- First time on a machine, run `demo/setup-demo.sh`. It installs JDK 17, Node 20, pulls MySQL 8 (Docker Hub, then `mirror.gcr.io` fallback), warms Maven and Playwright Chromium, and starts the app. It ends with a `READY` banner or a `SETUP FAILED: <step>` line.
+- `demo/start-demo.sh` resets to an empty `bankappdb` and restarts the app (`--build` rebuilds the jar offline first).
+- Maven Central rate-limits cloud VMs. Always build with `./mvnw -s demo/maven-settings.xml ...` (GCS mirror). CI should do the same.
+- Docker Hub may rate-limit. Use `mirror.gcr.io/library/mysql:8.0` where you can (including CI service images).
+- Start long-running processes with `setsid`, not plain `nohup`.

demo/live/setup-demo.sh

@@ -0,0 +1,86 @@
+#!/usr/bin/env bash
+# demo_qa BEFORE-state setup. Installed into the seed branch as demo/setup-demo.sh.
+# Prepares a machine so Devin can build, run and Playwright-test the app with no network surprises.
+set -eEuo pipefail
+
+REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+STARTED_AT="$(date +%s)"
+CURRENT_STEP="initializing"
+PLAYWRIGHT_VERSION="1.52.0"
+
+trap 'status=$?; printf "SETUP FAILED: %s (exit %s)\n" "$CURRENT_STEP" "$status" >&2; exit "$status"' ERR
+cd "$REPO_ROOT"
+
+step() { CURRENT_STEP="$1"; printf '==> %s\n' "$1"; }
+fail() { CURRENT_STEP="$1"; false; }
+
+ensure_mysql_image() {
+  if docker pull mysql:8.0; then return 0; fi
+  printf 'Docker Hub pull failed, trying mirror.gcr.io/library/mysql:8.0\n'
+  if docker pull mirror.gcr.io/library/mysql:8.0; then docker tag mirror.gcr.io/library/mysql:8.0 mysql:8.0; return 0; fi
+  docker image inspect mysql:8.0 >/dev/null 2>&1 && { printf 'Using cached mysql:8.0 image.\n'; return 0; }
+  printf 'FAILED: could not get mysql:8.0 from Docker Hub or mirror.gcr.io\n' >&2
+  return 1
+}
+
+jdk_major() {
+  local java_v javac_v
+  command -v java >/dev/null 2>&1 && command -v javac >/dev/null 2>&1 || { echo 0; return; }
+  java_v="$(java -version 2>&1 | sed -n '1s/.*version "\([0-9]*\).*/\1/p')"
+  javac_v="$(javac -version 2>&1 | sed -n 's/^javac \([0-9]*\).*/\1/p')"
+  java_v="${java_v:-0}"; javac_v="${javac_v:-0}"
+  echo $(( java_v < javac_v ? java_v : javac_v ))
+}
+
+step "Plan present, automation absent"
+[[ -f qa/manual-test-plan.csv ]] || fail "qa/manual-test-plan.csv missing"
+rows="$(tail -n +2 qa/manual-test-plan.csv | grep -c '^MT-')"
+[[ "$rows" == "60" ]] || fail "expected 60 manual rows, found $rows"
+
+step "JDK 17"
+if (( $(jdk_major) < 17 )); then sudo apt-get install -y openjdk-17-jdk-headless; fi
+(( $(jdk_major) >= 17 )) || fail "JDK 17 verification (need java and javac)"
+
+step "Node 20"
+node_major=0
+command -v node >/dev/null 2>&1 && node_major="$(node -p 'process.versions.node.split(".")[0]')"
+if [[ "$node_major" != "20" ]]; then
+  curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
+  sudo apt-get install -y nodejs
+fi
+[[ "$(node -p 'process.versions.node.split(".")[0]')" == "20" ]] || fail "Node 20 verification"
+
+step "Docker and MySQL 8 image"
+if ! command -v docker >/dev/null 2>&1; then
+  sudo apt-get install -y docker.io
+  sudo systemctl start docker || sudo service docker start
+fi
+docker info >/dev/null 2>&1 || fail "Docker daemon unavailable"
+ensure_mysql_image
+
+step "Maven dependencies (GCS mirror, Maven Central rate-limits cloud VMs)"
+export MVNW_REPOURL=https://maven-central.storage-download.googleapis.com/maven2
+./mvnw -B -q -s demo/maven-settings.xml dependency:go-offline
+./mvnw -B -q -s demo/maven-settings.xml -DskipTests package
+[[ -f target/bankapp-0.0.1-SNAPSHOT.jar ]] || fail "Maven package verification"
+
+step "Playwright ${PLAYWRIGHT_VERSION} Chromium and OS deps (warm cache)"
+npm cache add "@playwright/test@${PLAYWRIGHT_VERSION}" typescript@5 @types/node@20 >/dev/null
+npx -y "playwright@${PLAYWRIGHT_VERSION}" install --with-deps chromium
+
+step "App reset (empty bankappdb, app on :8080)"
+demo/start-demo.sh
+curl -sf -o /dev/null http://localhost:8080/login || fail "app not answering on :8080"
+
+step "Seed is still the BEFORE state"
+[[ ! -e e2e ]] || fail "e2e/ exists; this is not the BEFORE state, rerun demo/seed-situation.sh"
+
+branch="$(git rev-parse --abbrev-ref HEAD)"
+printf 'Setup completed in %ss.\n' "$(( $(date +%s) - STARTED_AT ))"
+printf '%s\n' \
+  '==================================================' \
+  'READY: demo_qa (manual QA plan, no automation yet)' \
+  "Branch:   ${branch}" \
+  'App:      http://localhost:8080 (reset any time: demo/start-demo.sh)' \
+  'Plan:     qa/manual-test-plan.csv (60 rows)' \
+  '=================================================='

demo/seed-situation.sh

@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+# demo_qa: (re)create the BEFORE branch a presenter hands to a live Devin session.
+# BEFORE = upstream app + the 60-row manual plan, with no e2e/, no CI workflow, no coverage matrix.
+# Safe to rerun: force-resets ONLY the seed branch and closes PRs that targeted it.
+set -eEuo pipefail
+
+REPO="${DEMO_QA_REPO:-COG-GTM/Springboot-BankApp}"
+SEED_BRANCH="${DEMO_QA_SEED_BRANCH:-demo/qa-live-before}"
+BASE_SHA=305826d755587d70eb24a7076eaf0797743548b3   # upstream app, before any automation
+PLAN_SHA=75d6e9b82f61a28e7b4c8939bcd0df075790dea4   # commit that introduced qa/manual-test-plan.csv
+FALLBACK_BRANCH=devin/fs-testing-1791848695-manual-to-automated
+KIT_ROOT="$(cd "$(dirname "$0")/.." && pwd)"
+CURRENT_STEP="initializing"
+WORKTREE=""
+
+trap 'status=$?; printf "SEED FAILED: %s (exit %s)\n" "$CURRENT_STEP" "$status" >&2; exit "$status"' ERR
+trap '[[ -n "$WORKTREE" ]] && git -C "$KIT_ROOT" worktree remove --force "$WORKTREE" >/dev/null 2>&1; true' EXIT
+step() { CURRENT_STEP="$1"; printf '==> %s\n' "$1"; }
+fail() { CURRENT_STEP="$1"; false; }
+
+case "$SEED_BRANCH" in main|master|devin/fs-testing-*) fail "refusing to reset protected branch $SEED_BRANCH";; esac
+
+step "Tools"
+for t in git gh; do command -v "$t" >/dev/null || fail "$t not installed"; done
+gh auth status >/dev/null 2>&1 || fail "gh not authenticated"
+
+step "Fetch $REPO"
+git -C "$KIT_ROOT" fetch -q origin
+git -C "$KIT_ROOT" cat-file -e "$BASE_SHA^{commit}" || fail "base commit $BASE_SHA not found"
+git -C "$KIT_ROOT" cat-file -e "$PLAN_SHA^{commit}" || fail "plan commit $PLAN_SHA not found"
+
+step "Close PRs and delete branches from earlier live runs against $SEED_BRANCH"
+while read -r num head; do
+  [[ -z "$num" ]] && continue
+  gh pr close "$num" -R "$REPO" --comment "Closed by demo/seed-situation.sh: demo_qa seed branch was reset." >/dev/null
+  printf '    closed PR #%s (%s)\n' "$num" "$head"
+  case "$head" in devin/*) git -C "$KIT_ROOT" push -q origin --delete "$head" 2>/dev/null && printf '    deleted branch %s\n' "$head" || true;; esac
+done < <(gh pr list -R "$REPO" --base "$SEED_BRANCH" --state open --json number,headRefName --jq '.[] | "\(.number) \(.headRefName)"')
+
+step "Build BEFORE tree from $BASE_SHA"
+WORKTREE="$(mktemp -d)/seed"
+git -C "$KIT_ROOT" worktree add -q --detach "$WORKTREE" "$BASE_SHA"
+cd "$WORKTREE"
+mkdir -p qa demo
+git show "$PLAN_SHA:qa/manual-test-plan.csv" > qa/manual-test-plan.csv
+cp "$KIT_ROOT/demo/live/setup-demo.sh" demo/setup-demo.sh
+cp "$KIT_ROOT/demo/start-demo.sh" "$KIT_ROOT/demo/maven-settings.xml" demo/
+cp "$KIT_ROOT/demo/live/AGENTS.md" AGENTS.md
+cp "$KIT_ROOT/mvnw" "$KIT_ROOT/mvnw.cmd" .
+cp "$KIT_ROOT/.gitignore" .gitignore
+chmod +x demo/setup-demo.sh demo/start-demo.sh mvnw
+git add -A
+git -c commit.gpgsign=false commit -q -m "chore: seed demo_qa BEFORE state (60-row manual plan, no automation)"
+
+step "Verify BEFORE state"
+rows="$(tail -n +2 qa/manual-test-plan.csv | grep -c '^MT-')"
+[[ "$rows" == "60" ]] || fail "expected 60 manual rows, found $rows"
+grep -q '^MT-047,' qa/manual-test-plan.csv || fail "hard case MT-047 missing from plan"
+for p in e2e .github/workflows/e2e.yml qa/coverage-matrix.md qa/not-automated.json; do
+  [[ ! -e "$p" ]] || fail "AFTER artifact present in seed: $p"
+done
+
+step "Push $SEED_BRANCH"
+git push -q --force origin "HEAD:refs/heads/$SEED_BRANCH"
+remote_sha="$(git ls-remote origin "refs/heads/$SEED_BRANCH" | cut -f1)"
+[[ "$remote_sha" == "$(git rev-parse HEAD)" ]] || fail "remote $SEED_BRANCH does not match seed commit"
+
+printf '%s\n' \
+  '==================================================================' \
+  'READY: demo_qa seeded (BEFORE state, Devin has not run yet)' \
+  "Seed branch:  https://github.com/$REPO/tree/$SEED_BRANCH (${remote_sha:0:7})" \
+  "Prompt:       demo/PROMPTS.md, prompt 1 (open a NEW Devin session on $REPO)" \
+  "Fallback:     git checkout $FALLBACK_BRANCH && demo/setup-demo.sh" \
+  "              pre-built PRs: https://github.com/$REPO/pull/328 (suite), /pull/329 (planted regression)" \
+  '=================================================================='

plugins/demo_qa/plugin.json

@@ -0,0 +1,6 @@
+{
+  "name": "demo_qa",
+  "version": "1.0.0",
+  "description": "FS Golden Demo Bank: set up the live QA situation (a 60-row manual regression plan with no automation) for a presenter to automate with Devin live.",
+  "author": { "name": "Cognition GTM" }
+}

plugins/demo_qa/skills/demo_qa/SKILL.md

@@ -0,0 +1,25 @@
+---
+name: demo_qa
+description: Set up the live demo_qa situation (Springboot-BankApp with a 60-row manual test plan and no automated tests) and hand the presenter the exact prompts.
+---
+
+# demo_qa: live situation setup
+
+Use when a presenter asks to set up, reset, or prepare demo_qa (manual regression plan to Playwright, FS Golden Demo Bank).
+
+The presenter, not this setup, runs Devin live. Never pre-run the automation, never open a test PR against the seed branch, never merge.
+
+## Steps
+1. Clone `COG-GTM/Springboot-BankApp` and check out the kit branch `devin/1792723253-demo-qa-live`.
+2. Run `demo/seed-situation.sh` and wait for `READY`. It force-resets ONLY `demo/qa-live-before` to the BEFORE state (the app, `qa/manual-test-plan.csv` with 60 rows, no `e2e/`, no CI test workflow) and closes PRs that targeted it. If the script prints anything other than `READY` at the end, stop and report the log.
+3. Optional machine check: on `demo/qa-live-before`, run `demo/setup-demo.sh` and wait for `READY` (MySQL 8 plus the app). The live Devin session runs the same script itself.
+4. Print for the presenter, verbatim from `demo/PROMPTS.md` on the kit branch: the seed branch URL `https://github.com/COG-GTM/Springboot-BankApp/tree/demo/qa-live-before`, prompt 1, optional prompt 2, the expected outcome, and the fallback. Point to `demo/TALK-TRACK-LIVE.md` for beats.
+
+## What the presenter should see live (measured 16 min 32 s)
+- A PR against `demo/qa-live-before`: about 54 rows automated as Playwright tests, green in CI against MySQL 8, and a coverage matrix accounting for all 60 rows.
+- About 6 rows held manual, each with a reason (brand signoff, physical printer, JAWS, moderated session, missing notification feature).
+- Hard case: MT-047 ("System handles it appropriately"). Devin does not invent an expected result; Devin shows that a negative transfer moves money the wrong way and hands the row to QA.
+- Optional prompt 2: a one-character regression on a stacked PR; CI flags exactly MT-040 (full-balance transfer), 53 other rows stay green, and Devin asks whether to keep the rule or drop the change instead of editing the plan. Measured live: PR #336, CI red about 2.5 min after the PR opened.
+
+## Fallback
+`git checkout devin/fs-testing-1791848695-manual-to-automated && demo/setup-demo.sh`, then PRs #328 and #329.