diff --git a/demo/PRESENTER.md b/demo/PRESENTER.md new file mode 100644 index 00000000..5e35ce15 --- /dev/null +++ b/demo/PRESENTER.md @@ -0,0 +1,45 @@ +# demo_qa presenter sheet: manual test plan to Playwright + +**Goal:** turn the bank's 60-row manual regression plan into a Playwright suite that runs in CI on MySQL 8, live, in about 17 minutes. Every row is accounted for. Devin does not invent an expected result for MT-047 (negative transfer); it flags the money bug and holds the row for QA. + +## Links +| What | Link | +|---|---| +| **Live session (already started with prompt 1)** | the session link your setup session sent you (the setup session starts it with prompt 1) | +| The 60-row plan | https://github.com/COG-GTM/Springboot-BankApp/blob/demo/qa-live-before/qa/manual-test-plan.csv | +| MT-047, the hard case (line 48) | https://github.com/COG-GTM/Springboot-BankApp/blob/demo/qa-live-before/qa/manual-test-plan.csv#L48 | +| BEFORE and AFTER videos (kit PR #335 description, does not close on reseed) | https://github.com/COG-GTM/Springboot-BankApp/pull/335 | +| Pre-built suite: 54 automated, CI green | https://github.com/COG-GTM/Springboot-BankApp/pull/328 | +| Pre-built coverage matrix (MT-047 row) | https://github.com/COG-GTM/Springboot-BankApp/blob/devin/fs-testing-1791848695-manual-to-automated/qa/coverage-matrix.md | +| Pre-built planted regression: CI red on MT-040 only, 53 pass | https://github.com/COG-GTM/Springboot-BankApp/pull/329 | + +## Beats +| Time | Beat | Use | +|---|---|---| +| 0:00 | "60 rows, run by hand every release. There is no `e2e/` folder." Open the plan. | The 60-row plan | +| 0:01 | Optional: show the app working before any automation, and the AFTER clip of the 54-test suite running. | BEFORE / AFTER videos on #335 | +| 0:02 | Open the live session. Prompt 1 is already in it as the first message; show it to the room. | Live session | +| 0:04 | Devin reads the plan and the app, brings up MySQL 8 and the app. Point at its plan. | Live session | +| 0:05 to ~0:17 | Read MT-047 aloud: "System handles it appropriately." Ask the room what a tester would assert. Check back every few minutes. | MT-047 link | +| ~0:17 | Live PR lands against `demo/qa-live-before`. Open the coverage matrix: all 60 rows have a status. | Live PR (from the session). Running late: #328 | +| ~0:18 | **Wow moment:** the MT-047 row. "A minus 50 transfer moves money the wrong way, the sender gains 50. Devin did not write a test that blesses that, and did not make up a rule. It held the row and flagged the bug for QA." | Live PR matrix. Pre-built: coverage matrix link | +| ~0:20 to ~0:26 | Optional prompt 2 in the same session. CI goes red on exactly MT-040, the other 53 stay green. Devin asks keep the rule or drop the change; answer "drop it". | Live stacked PR. Short on time: #329 | + +## Prompt 1 (already sent to the live session as its first message) +``` +Repo: COG-GTM/Springboot-BankApp, branch demo/qa-live-before. Branch off it and open your PR against demo/qa-live-before. + +Here is our 60-row manual regression plan (qa/manual-test-plan.csv). Automate every row you responsibly can as Playwright tests that run in CI against MySQL 8, publish a coverage matrix that accounts for all 60 rows, and tell me which rows you could not automate and why. If a row's expected result is not testable, do not invent one. +``` +Expect: about 54 automated, about 5 not automatable with a reason each, MT-047 held. Counts can move by a row or two; all 60 accounted for and MT-047 held must always be true. + +## Prompt 2 (optional, same session, after prompt 1's PR is up) +``` +On a new branch off your PR branch, change AccountService so a customer cannot transfer their full balance (< 0 to <= 0), and open a PR against your PR branch. Tell me which plan rows CI flags. +``` +Expect: CI red about 2.5 min after the PR opens, on MT-040 only. + +## If the live run stalls +Show #328 (suite, green) and #329 (planted regression, red on MT-040 only). To run the finished state locally: `git checkout devin/fs-testing-1791848695-manual-to-automated && demo/setup-demo.sh`. + +Honesty notes: the app is the open-source Springboot BankApp; the plan was written for this demo. diff --git a/demo/PROMPTS.md b/demo/PROMPTS.md new file mode 100644 index 00000000..bcd7d717 --- /dev/null +++ b/demo/PROMPTS.md @@ -0,0 +1,33 @@ +# demo_qa prompts (live Devin) + +Presenter sheet with goal, links, beats and which PR to use: `demo/PRESENTER.md`. + +Before the meeting: run `demo/seed-situation.sh` from this kit branch and wait for `READY`. +In the meeting: open a NEW Devin session on `COG-GTM/Springboot-BankApp` and paste prompt 1 exactly. + +## Prompt 1: manual plan to automated suite (live, about 17 min wall time measured) + +``` +Repo: COG-GTM/Springboot-BankApp, branch demo/qa-live-before. Branch off it and open your PR against demo/qa-live-before. + +Here is our 60-row manual regression plan (qa/manual-test-plan.csv). Automate every row you responsibly can as Playwright tests that run in CI against MySQL 8, publish a coverage matrix that accounts for all 60 rows, and tell me which rows you could not automate and why. If a row's expected result is not testable, do not invent one. +``` + +Expected back: +- A PR against `demo/qa-live-before` with `e2e/` (Playwright, TypeScript), a GitHub Actions workflow running against a MySQL 8 service, and a coverage matrix accounting for all 60 rows. +- Roughly 54 rows automated, about 5 not automatable with a reason each (for example print, email, cross-browser or timing rows), and 1 row held. +- The hard case: `MT-047` (negative transfer). The plan's expected result is only "System handles it appropriately", and the app actually accepts `-50` and moves the sender from `$100` to `$150`. Devin reports that as a money-movement defect and holds the row for QA instead of writing a test that blesses the bug or inventing an expected result. + +Exact counts can vary by a row or two between runs; the invariants are: all 60 rows accounted for, `MT-047` held and explained, no invented expected results. + +## Prompt 2: planted regression (optional, run in the same session after prompt 1 finishes) + +``` +On a new branch off your PR branch, change AccountService so a customer cannot transfer their full balance (< 0 to <= 0), and open a PR against your PR branch. Tell me which plan rows CI flags. +``` + +Expected back: CI on that PR fails on exactly the full-balance transfer case (`MT-040` in the plan), and Devin says so. + +## If the live run stalls + +Switch to the pre-built fallback: `git checkout devin/fs-testing-1791848695-manual-to-automated && demo/setup-demo.sh`, then show PR #328 (suite) and PR #329 (planted regression). diff --git a/demo/TALK-TRACK-LIVE.md b/demo/TALK-TRACK-LIVE.md new file mode 100644 index 00000000..54f06252 --- /dev/null +++ b/demo/TALK-TRACK-LIVE.md @@ -0,0 +1,19 @@ +# demo_qa live talk track + +Situation: the bank has a 60-row manual regression plan run by hand every release. No automation exists. The presenter asks Devin, live, to automate it. + +| Time | Beat | Live or pre-built | +|---|---|---| +| 0:00 | Show `qa/manual-test-plan.csv` on `demo/qa-live-before`. "60 rows, run by hand every release. There is no `e2e/` folder." | Live (GitHub) | +| 0:02 | Open a new Devin session, paste prompt 1 from `PROMPTS.md`. | Live | +| 0:04 | Devin reads the plan and the app, runs `demo/setup-demo.sh`, brings MySQL 8 and the app up. Point at the plan Devin writes. | Live | +| 0:05 to ~0:17 | While Devin works, open `qa/manual-test-plan.csv` and read MT-047 aloud: "System handles it appropriately." Ask the room what a tester would assert. Check back every few minutes. | Live session running | +| ~0:17 | Live PR lands (measured 16 min 32 s from paste) against `demo/qa-live-before`. Open the coverage matrix: every one of 60 rows has a status. | Live | +| ~0:18 | Wow moment: the `MT-047` row. "The plan only says the system handles a negative transfer appropriately. Devin tried it: a minus 50 transfer moves money the wrong way, the sender gains 50. Devin did not write a test that blesses that, and did not make up a new rule. Devin held the row and flagged the money bug for QA." | Live | +| ~0:20 to ~0:26 | Optional prompt 2 (about 5 min when CI is not flaky; CI itself takes about 2.5 min): planted regression on a stacked PR. CI goes red on exactly MT-040, the full-balance transfer, and the other 53 rows stay green. Hard case: Devin does not edit MT-040 to make CI pass. Devin asks whether to keep the new rule (and update MT-040) or drop the change. Answer "drop it". If time is short, show PR #329 (pre-built, red on MT-040 only). Live-run PRs like #332 and #336 close when the seed is reset. | Live or pre-built | + +Hard case: `MT-047`. Devin must hold and escalate, not invent an expected result. + +Fallback: if the live session stalls or the network misbehaves, `git checkout devin/fs-testing-1791848695-manual-to-automated && demo/setup-demo.sh` gives the finished state in about 5 minutes, and PRs #328 and #329 show the same outcome. + +Honesty notes: the app is the open-source Springboot BankApp; the manual plan was written for this demo. Exact automated/not-automatable counts can differ by a row or two between live runs. diff --git a/demo/live/AGENTS.md b/demo/live/AGENTS.md new file mode 100644 index 00000000..436d2695 --- /dev/null +++ b/demo/live/AGENTS.md @@ -0,0 +1,8 @@ +# Agent notes for this repository + +- Stack: Spring Boot 3 / Java 17, Thymeleaf, MySQL 8. App listens on http://localhost:8080. +- First time on a machine, run `demo/setup-demo.sh`. It installs JDK 17, Node 20, pulls MySQL 8 (Docker Hub, then `mirror.gcr.io` fallback), warms Maven and Playwright Chromium, and starts the app. It ends with a `READY` banner or a `SETUP FAILED: ` line. +- `demo/start-demo.sh` resets to an empty `bankappdb` and restarts the app (`--build` rebuilds the jar offline first). +- Maven Central rate-limits cloud VMs. Always build with `./mvnw -s demo/maven-settings.xml ...` (GCS mirror). CI should do the same. +- Docker Hub may rate-limit. Use `mirror.gcr.io/library/mysql:8.0` where you can (including CI service images). +- Start long-running processes with `setsid`, not plain `nohup`. diff --git a/demo/live/setup-demo.sh b/demo/live/setup-demo.sh new file mode 100755 index 00000000..9585ccc0 --- /dev/null +++ b/demo/live/setup-demo.sh @@ -0,0 +1,86 @@ +#!/usr/bin/env bash +# demo_qa BEFORE-state setup. Installed into the seed branch as demo/setup-demo.sh. +# Prepares a machine so Devin can build, run and Playwright-test the app with no network surprises. +set -eEuo pipefail + +REPO_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +STARTED_AT="$(date +%s)" +CURRENT_STEP="initializing" +PLAYWRIGHT_VERSION="1.52.0" + +trap 'status=$?; printf "SETUP FAILED: %s (exit %s)\n" "$CURRENT_STEP" "$status" >&2; exit "$status"' ERR +cd "$REPO_ROOT" + +step() { CURRENT_STEP="$1"; printf '==> %s\n' "$1"; } +fail() { CURRENT_STEP="$1"; false; } + +ensure_mysql_image() { + if docker pull mysql:8.0; then return 0; fi + printf 'Docker Hub pull failed, trying mirror.gcr.io/library/mysql:8.0\n' + if docker pull mirror.gcr.io/library/mysql:8.0; then docker tag mirror.gcr.io/library/mysql:8.0 mysql:8.0; return 0; fi + docker image inspect mysql:8.0 >/dev/null 2>&1 && { printf 'Using cached mysql:8.0 image.\n'; return 0; } + printf 'FAILED: could not get mysql:8.0 from Docker Hub or mirror.gcr.io\n' >&2 + return 1 +} + +jdk_major() { + local java_v javac_v + command -v java >/dev/null 2>&1 && command -v javac >/dev/null 2>&1 || { echo 0; return; } + java_v="$(java -version 2>&1 | sed -n '1s/.*version "\([0-9]*\).*/\1/p')" + javac_v="$(javac -version 2>&1 | sed -n 's/^javac \([0-9]*\).*/\1/p')" + java_v="${java_v:-0}"; javac_v="${javac_v:-0}" + echo $(( java_v < javac_v ? java_v : javac_v )) +} + +step "Plan present, automation absent" +[[ -f qa/manual-test-plan.csv ]] || fail "qa/manual-test-plan.csv missing" +rows="$(tail -n +2 qa/manual-test-plan.csv | grep -c '^MT-')" +[[ "$rows" == "60" ]] || fail "expected 60 manual rows, found $rows" + +step "JDK 17" +if (( $(jdk_major) < 17 )); then sudo apt-get install -y openjdk-17-jdk-headless; fi +(( $(jdk_major) >= 17 )) || fail "JDK 17 verification (need java and javac)" + +step "Node 20" +node_major=0 +command -v node >/dev/null 2>&1 && node_major="$(node -p 'process.versions.node.split(".")[0]')" +if [[ "$node_major" != "20" ]]; then + curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash - + sudo apt-get install -y nodejs +fi +[[ "$(node -p 'process.versions.node.split(".")[0]')" == "20" ]] || fail "Node 20 verification" + +step "Docker and MySQL 8 image" +if ! command -v docker >/dev/null 2>&1; then + sudo apt-get install -y docker.io + sudo systemctl start docker || sudo service docker start +fi +docker info >/dev/null 2>&1 || fail "Docker daemon unavailable" +ensure_mysql_image + +step "Maven dependencies (GCS mirror, Maven Central rate-limits cloud VMs)" +export MVNW_REPOURL=https://maven-central.storage-download.googleapis.com/maven2 +./mvnw -B -q -s demo/maven-settings.xml dependency:go-offline +./mvnw -B -q -s demo/maven-settings.xml -DskipTests package +[[ -f target/bankapp-0.0.1-SNAPSHOT.jar ]] || fail "Maven package verification" + +step "Playwright ${PLAYWRIGHT_VERSION} Chromium and OS deps (warm cache)" +npm cache add "@playwright/test@${PLAYWRIGHT_VERSION}" typescript@5 @types/node@20 >/dev/null +npx -y "playwright@${PLAYWRIGHT_VERSION}" install --with-deps chromium + +step "App reset (empty bankappdb, app on :8080)" +demo/start-demo.sh +curl -sf -o /dev/null http://localhost:8080/login || fail "app not answering on :8080" + +step "Seed is still the BEFORE state" +[[ ! -e e2e ]] || fail "e2e/ exists; this is not the BEFORE state, rerun demo/seed-situation.sh" + +branch="$(git rev-parse --abbrev-ref HEAD)" +printf 'Setup completed in %ss.\n' "$(( $(date +%s) - STARTED_AT ))" +printf '%s\n' \ + '==================================================' \ + 'READY: demo_qa (manual QA plan, no automation yet)' \ + "Branch: ${branch}" \ + 'App: http://localhost:8080 (reset any time: demo/start-demo.sh)' \ + 'Plan: qa/manual-test-plan.csv (60 rows)' \ + '==================================================' diff --git a/demo/seed-situation.sh b/demo/seed-situation.sh new file mode 100755 index 00000000..ce5231fd --- /dev/null +++ b/demo/seed-situation.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# demo_qa: (re)create the BEFORE branch a presenter hands to a live Devin session. +# BEFORE = upstream app + the 60-row manual plan, with no e2e/, no CI workflow, no coverage matrix. +# Safe to rerun: force-resets ONLY the seed branch and closes PRs that targeted it. +set -eEuo pipefail + +REPO="${DEMO_QA_REPO:-COG-GTM/Springboot-BankApp}" +SEED_BRANCH="${DEMO_QA_SEED_BRANCH:-demo/qa-live-before}" +BASE_SHA=305826d755587d70eb24a7076eaf0797743548b3 # upstream app, before any automation +PLAN_SHA=75d6e9b82f61a28e7b4c8939bcd0df075790dea4 # commit that introduced qa/manual-test-plan.csv +FALLBACK_BRANCH=devin/fs-testing-1791848695-manual-to-automated +KIT_ROOT="$(cd "$(dirname "$0")/.." && pwd)" +CURRENT_STEP="initializing" +WORKTREE="" + +trap 'status=$?; printf "SEED FAILED: %s (exit %s)\n" "$CURRENT_STEP" "$status" >&2; exit "$status"' ERR +trap '[[ -n "$WORKTREE" ]] && git -C "$KIT_ROOT" worktree remove --force "$WORKTREE" >/dev/null 2>&1; true' EXIT +step() { CURRENT_STEP="$1"; printf '==> %s\n' "$1"; } +fail() { CURRENT_STEP="$1"; false; } + +case "$SEED_BRANCH" in main|master|devin/fs-testing-*) fail "refusing to reset protected branch $SEED_BRANCH";; esac + +step "Tools" +for t in git gh; do command -v "$t" >/dev/null || fail "$t not installed"; done +gh auth status >/dev/null 2>&1 || fail "gh not authenticated" + +step "Fetch $REPO" +git -C "$KIT_ROOT" fetch -q origin +git -C "$KIT_ROOT" cat-file -e "$BASE_SHA^{commit}" || fail "base commit $BASE_SHA not found" +git -C "$KIT_ROOT" cat-file -e "$PLAN_SHA^{commit}" || fail "plan commit $PLAN_SHA not found" + +step "Close PRs and delete branches from earlier live runs against $SEED_BRANCH" +while read -r num head; do + [[ -z "$num" ]] && continue + gh pr close "$num" -R "$REPO" --comment "Closed by demo/seed-situation.sh: demo_qa seed branch was reset." >/dev/null + printf ' closed PR #%s (%s)\n' "$num" "$head" + case "$head" in devin/*) git -C "$KIT_ROOT" push -q origin --delete "$head" 2>/dev/null && printf ' deleted branch %s\n' "$head" || true;; esac +done < <(gh pr list -R "$REPO" --base "$SEED_BRANCH" --state open --json number,headRefName --jq '.[] | "\(.number) \(.headRefName)"') + +step "Build BEFORE tree from $BASE_SHA" +WORKTREE="$(mktemp -d)/seed" +git -C "$KIT_ROOT" worktree add -q --detach "$WORKTREE" "$BASE_SHA" +cd "$WORKTREE" +mkdir -p qa demo +git show "$PLAN_SHA:qa/manual-test-plan.csv" > qa/manual-test-plan.csv +cp "$KIT_ROOT/demo/live/setup-demo.sh" demo/setup-demo.sh +cp "$KIT_ROOT/demo/start-demo.sh" "$KIT_ROOT/demo/maven-settings.xml" demo/ +cp "$KIT_ROOT/demo/live/AGENTS.md" AGENTS.md +cp "$KIT_ROOT/mvnw" "$KIT_ROOT/mvnw.cmd" . +cp "$KIT_ROOT/.gitignore" .gitignore +chmod +x demo/setup-demo.sh demo/start-demo.sh mvnw +git add -A +git -c commit.gpgsign=false commit -q -m "chore: seed demo_qa BEFORE state (60-row manual plan, no automation)" + +step "Verify BEFORE state" +rows="$(tail -n +2 qa/manual-test-plan.csv | grep -c '^MT-')" +[[ "$rows" == "60" ]] || fail "expected 60 manual rows, found $rows" +grep -q '^MT-047,' qa/manual-test-plan.csv || fail "hard case MT-047 missing from plan" +for p in e2e .github/workflows/e2e.yml qa/coverage-matrix.md qa/not-automated.json; do + [[ ! -e "$p" ]] || fail "AFTER artifact present in seed: $p" +done + +step "Push $SEED_BRANCH" +git push -q --force origin "HEAD:refs/heads/$SEED_BRANCH" +remote_sha="$(git ls-remote origin "refs/heads/$SEED_BRANCH" | cut -f1)" +[[ "$remote_sha" == "$(git rev-parse HEAD)" ]] || fail "remote $SEED_BRANCH does not match seed commit" + +printf '%s\n' \ + '==================================================================' \ + 'READY: demo_qa seeded (BEFORE state, Devin has not run yet)' \ + "Seed branch: https://github.com/$REPO/tree/$SEED_BRANCH (${remote_sha:0:7})" \ + "Prompt: demo/PROMPTS.md, prompt 1 (open a NEW Devin session on $REPO)" \ + "Fallback: git checkout $FALLBACK_BRANCH && demo/setup-demo.sh" \ + " pre-built PRs: https://github.com/$REPO/pull/328 (suite), /pull/329 (planted regression)" \ + '==================================================================' diff --git a/plugins/demo_qa/plugin.json b/plugins/demo_qa/plugin.json new file mode 100644 index 00000000..337b780c --- /dev/null +++ b/plugins/demo_qa/plugin.json @@ -0,0 +1,6 @@ +{ + "name": "demo_qa", + "version": "1.0.0", + "description": "FS Golden Demo Bank: set up the live QA situation (a 60-row manual regression plan with no automation) for a presenter to automate with Devin live.", + "author": { "name": "Cognition GTM" } +} diff --git a/plugins/demo_qa/skills/demo_qa/SKILL.md b/plugins/demo_qa/skills/demo_qa/SKILL.md new file mode 100644 index 00000000..1326a564 --- /dev/null +++ b/plugins/demo_qa/skills/demo_qa/SKILL.md @@ -0,0 +1,25 @@ +--- +name: demo_qa +description: Set up the live demo_qa situation (Springboot-BankApp with a 60-row manual test plan and no automated tests) and hand the presenter the exact prompts. +--- + +# demo_qa: live situation setup + +Use when a presenter asks to set up, reset, or prepare demo_qa (manual regression plan to Playwright, FS Golden Demo Bank). + +The presenter, not this setup, runs Devin live. Never pre-run the automation, never open a test PR against the seed branch, never merge. + +## Steps +1. Clone `COG-GTM/Springboot-BankApp` and check out the kit branch `devin/1792723253-demo-qa-live`. +2. Run `demo/seed-situation.sh` and wait for `READY`. It force-resets ONLY `demo/qa-live-before` to the BEFORE state (the app, `qa/manual-test-plan.csv` with 60 rows, no `e2e/`, no CI test workflow) and closes PRs that targeted it. If the script prints anything other than `READY` at the end, stop and report the log. +3. Optional machine check: on `demo/qa-live-before`, run `demo/setup-demo.sh` and wait for `READY` (MySQL 8 plus the app). The live Devin session runs the same script itself. +4. Print for the presenter, verbatim from `demo/PROMPTS.md` on the kit branch: the seed branch URL `https://github.com/COG-GTM/Springboot-BankApp/tree/demo/qa-live-before`, prompt 1, optional prompt 2, the expected outcome, and the fallback. Point to `demo/TALK-TRACK-LIVE.md` for beats. + +## What the presenter should see live (measured 16 min 32 s) +- A PR against `demo/qa-live-before`: about 54 rows automated as Playwright tests, green in CI against MySQL 8, and a coverage matrix accounting for all 60 rows. +- About 6 rows held manual, each with a reason (brand signoff, physical printer, JAWS, moderated session, missing notification feature). +- Hard case: MT-047 ("System handles it appropriately"). Devin does not invent an expected result; Devin shows that a negative transfer moves money the wrong way and hands the row to QA. +- Optional prompt 2: a one-character regression on a stacked PR; CI flags exactly MT-040 (full-balance transfer), 53 other rows stay green, and Devin asks whether to keep the rule or drop the change instead of editing the plan. Measured live: PR #336, CI red about 2.5 min after the PR opened. + +## Fallback +`git checkout devin/fs-testing-1791848695-manual-to-automated && demo/setup-demo.sh`, then PRs #328 and #329.