Repository navigation
demo_qa: live situation kit and plugin (accepted on a live run) - #335
devin-ai-integration[bot] wants to merge 8 commits into
Conversation
…lk track) Co-Authored-By: Garret Andreine <garret.andreine@cognition.ai>
Co-Authored-By: Garret Andreine <garret.andreine@cognition.ai>
…rgets Devin's PR branch Co-Authored-By: Garret Andreine <garret.andreine@cognition.ai>
Co-Authored-By: Garret Andreine <garret.andreine@cognition.ai>
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| gh pr close "$num" -R "$REPO" --comment "Closed by demo/seed-situation.sh: demo_qa seed branch was reset." >/dev/null | ||
| printf ' closed PR #%s (%s)\n' "$num" "$head" | ||
| case "$head" in devin/*) git -C "$KIT_ROOT" push -q origin --delete "$head" 2>/dev/null && printf ' deleted branch %s\n' "$head" || true;; esac | ||
| done < <(gh pr list -R "$REPO" --base "$SEED_BRANCH" --state open --json number,headRefName --jq '.[] | "\(.number) \(.headRefName)"') |
There was a problem hiding this comment.
🟡 Failed PR lookup leaves live PRs open
If gh pr list fails, the loop sees no PRs and git push still resets their base. Old live PRs remain open against the rewritten seed, and the script prints READY.
Learn more
The PR lookup runs inside process substitution, whose exit status does not become the status of the while loop. Authentication can pass before a later GitHub API request fails. In that case the loop reads no entries, and the script continues to rewrite the seed branch without closing its open PRs.
Example: GitHub authentication succeeds, but the gh pr list request returns an API error while PR #332 is open. The script pushes a new seed commit and prints READY; PR #332 remains open against that reset base.
Recommended fix: Capture gh pr list into a variable or temporary file and check its exit status before entering the loop or modifying remote state. Abort the seed on a listing failure.
Was this helpful? React with 👍 or 👎 to provide feedback.
| done | ||
|
|
||
| step "Push $SEED_BRANCH" | ||
| git push -q --force origin "HEAD:refs/heads/$SEED_BRANCH" |
There was a problem hiding this comment.
🟡 Repository override resets the wrong branch
When DEMO_QA_REPO differs from origin, PR cleanup targets the override but the force-push targets origin. The requested repository remains unseeded while another repository's branch is reset.
Learn more
The repository override controls gh pr list and gh pr close, but Git operations still use the checkout's origin. Those two destinations can differ when a presenter uses a fork or changes DEMO_QA_REPO. The script then reports the override's URL with the SHA from origin, so its READY banner does not describe the branch it created.
Example: With origin pointing at a fork and DEMO_QA_REPO=COG-GTM/Springboot-BankApp, the script closes PRs in COG-GTM but pushes demo/qa-live-before to the fork. The COG-GTM seed branch does not change.
Recommended fix: Resolve and validate a single remote corresponding to DEMO_QA_REPO before any PR cleanup. Use that same destination for fetch, push, and SHA verification, or reject mismatched configurations before changing remote state.
Was this helpful? React with 👍 or 👎 to provide feedback.
| step() { CURRENT_STEP="$1"; printf '==> %s\n' "$1"; } | ||
| fail() { CURRENT_STEP="$1"; false; } | ||
|
|
||
| case "$SEED_BRANCH" in main|master|devin/fs-testing-*) fail "refusing to reset protected branch $SEED_BRANCH";; esac |
There was a problem hiding this comment.
🟥 Branch override can overwrite active branches
With DEMO_QA_SEED_BRANCH set to DevOps, the script accepts the branch and force-pushes the demo seed. That replaces the active branch's tip with the demo state.
Was this helpful? React with 👍 or 👎 to provide feedback.
…top) Co-Authored-By: Garret Andreine <garret.andreine@cognition.ai>
…fallback at #329 Co-Authored-By: Garret Andreine <garret.andreine@cognition.ai>
|
demo_qa BEFORE state: app proof (reseeded 2026-10-23, Seed and The presenter sheet with links, beats and which PR to use for each beat is in |
…TER videos on #335 Co-Authored-By: Garret Andreine <garret.andreine@cognition.ai>
Co-Authored-By: Garret Andreine <garret.andreine@cognition.ai>
Presenter evidence (default links, this PR is never closed by a reseed)
Presenter sheet:
demo/PRESENTER.mdhas the goal, links, beats and which PR to use for each beat.BEFORE:
demo/qa-live-before@2594624on MySQL 8.demo_senderlogs in, the dashboard loads and the Transfer Money form opens. Noe2e/and no CI test workflow.AFTER: the generated suite from #328 (
devin/fs-testing-1791848695-manual-to-automated) runs headed against the same app. 54 passed and 0 skipped (35.7 s).qa/coverage-matrix.mdlists 60 of 60 rows, and MT-047 is "Needs QA clarification" with no invented expected result.Pre-built PRs: #328 (suite, CI green) and #329 (planted regression, CI red on exactly MT-040 with 53 passing).
Summary
Live-situation kit for demo_qa.
demo/seed-situation.shforce-resets onlydemo/qa-live-beforeto BEFORE: upstream305826dplus the 60-rowqa/manual-test-plan.csv, the setup scripts andAGENTS.md. It verifies there is noe2e/, no.github/workflows/e2e.ymland no coverage matrix, then printsREADY. The setup session starts the live Devin session with prompt 1 fromdemo/PROMPTS.md.Live acceptance (fresh session, prompt 1 pasted verbatim): #332 against
demo/qa-live-beforein 16 min 32 s. 54 of 60 rows automated as Playwright, green in CI on MySQL 8.0.46. Coverage matrix accounts for all 60 rows. 6 rows held manual with reasons (MT-021, 048, 055, 059, 060). Hard case MT-047 held: Devin did not invent an expected result and reported that a -50 transfer moves money the wrong way.Changes on top of #331 (closed):
plugins/demo_qa/: rootplugin.jsonandskills/demo_qa/SKILL.md. Runs the seed and prints the seed URL, prompts, expected outcome and fallback.PROMPTS.md/TALK-TRACK-LIVE.md: measured timing (about 17 min, not 35 to 50). Prompt 2 (planted regression) now branches off and targets Devin's PR branch. The BEFORE branch has no CI workflow, so a PR against it could not go red on MT-040. The prompt 2 fallback is bug: tighten transfer balance check (planted regression, do not merge) #329.PRESENTER.md: one-page presenter sheet with goal, links, per-beat PR and both prompts.Note: running the seed again closes open PRs against
demo/qa-live-before(including the live run's PR) by design. This PR targets a different base and stays open.Devin-Org: engineering
Link to Devin session: https://devin-gtm.devinenterprise.com/sessions/c698a982bc30414d928e1c22609cbefb
Open in Devin Desktop: https://devin-gtm.devinenterprise.com/desktop/session/c698a982bc30414d928e1c22609cbefb?variant=devin
Requested by: @GarretAnd
Link to Devin session: https://devin-gtm.devinenterprise.com/sessions/169d53b29adb4dee81a4de0f3daa05ca
Open in Devin Desktop: https://devin-gtm.devinenterprise.com/desktop/session/169d53b29adb4dee81a4de0f3daa05ca?variant=devin
Requested by: @achalc
Devin Review