[devin-ai-integration]

Summary

Automates qa/manual-test-plan.csv as Playwright tests (e2e/) and runs them in GitHub Actions against a MySQL 8 service container. 54 of 60 rows are automated; 6 are not, and each of the 6 has a reason in qa/not-automated.csv.

• Each test title starts with its plan ID (test('MT-031 Overdraw is rejected', ...)). Each test creates its own uniquely named customers, so tests share no state, need no DB reset, and run in parallel. Rows that say "After MT-036" repeat the MT-036 setup instead of relying on test order.
• e2e/scripts/coverage.mjs builds qa/coverage-matrix.md from the plan, the spec titles and qa/not-automated.csv. --check fails CI in these cases: a plan row has no test and no reason, a row has both, an ID isn't in the plan, or the committed matrix is out of date. --results results.json writes per-row pass/fail to the job summary.
• CI (.github/workflows/e2e.yml): mirror.gcr.io/library/mysql:8.0 service, ./mvnw -s demo/maven-settings.xml verify (so contextLoads also runs against MySQL), start jar, npx playwright test on Chromium. Report, traces, results.json and app log are uploaded as artifacts.

Assumptions where the plan was not specific (also noted as comments in the specs):

• MT-042/MT-046: the sender deposits 100 first. transferAmount checks funds before it looks up the recipient, so with a $0 balance the error would be "Insufficient funds" instead of the "Recipient account not found" this row is testing.
• MT-058: the plan doesn't name the bank, so the expected name is read from the page's .navbar-brand text ("Wells Fargo").
• MT-025/026/034/043/044: "not submitted" means the field's validity flag is set and no POST to the endpoint goes out.

Not automated (expected result is missing or untestable; no expected result was invented):

Row	Why
MT-021	Expected result is a brand-team sign-off, and the brand guide isn't in the repo
MT-047	"System handles it appropriately" is undefined. Observed: a -50 transfer is accepted and moves money from the recipient to the sender (sender $50.00, recipient $-50.00)
MT-048	The app has no SMS/email feature. Testing needs a real phone or inbox
MT-055	Needs a physical branch printer and a person to judge legibility
MT-059	Needs JAWS on Windows, and "announced clearly" is a listening judgement
MT-060	Moderated usability session with a person

Also observed (not tested, no expected result in the plan): /images/wells-fargo-logo.png returns 302 to /login for logged-out visitors, so the logo doesn't load on the login and register pages.

Note: demo/setup-demo.sh checks that e2e/ is absent ("BEFORE state"), so that check fails on this branch by design.

Local run against MySQL 8: 54 passed in about 20s. Two deliberate mutants (removing required, wrong expected balance) both failed, so the assertions do catch breakage.

Link to Devin session: https://devin-gtm.devinenterprise.com/sessions/0c42a18bdfdb467e87909110e8ca065e
Open in Devin Desktop: https://devin-gtm.devinenterprise.com/desktop/session/0c42a18bdfdb467e87909110e8ca065e?variant=devin
Requested by: @achalc

---

Devin Review

Status	Commit
⚪ Not started	—

Run Devin Review

Enable auto-review to review your pull requests automatically.

---

[devin-ai-integration]

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

• Disable automatic comment, CI, and merge conflict monitoring

[devin-ai-integration]

Devin Review found 2 potential issues.

[devin-ai-integration]

🟡 Disabled tests retain false coverage

When a plan test becomes test.skip or is commented out, scanSpecs still counts its ID as automated. The coverage check passes even though Playwright no longer tests that row.

Learn more

The coverage check uses scanSpecs to decide whether each manual-plan row has an executable Playwright test. It searches raw source, so test.skip('MT-022 ...') and a commented-out test('MT-022 ...') both register as coverage. The coverage gate only checks whether that ID was found, so it passes after the test is disabled.

Example: A developer skips MT-022 during debugging and commits the change. Playwright does not run the deposit test, but npm run coverage:check still reports the row as automated.

Recommended fix: Discover runnable tests through Playwright's test listing or another structured source instead of raw source matching. Exclude skipped and fixme tests, and verify that each plan ID has at least one executable test.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟥 Untrusted pull requests expose database credentials

On external pull requests, checkout runs contributor code with SPRING_DATASOURCE_PASSWORD in the job environment. A malicious install script can read and exfiltrate that credential.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

Closed by demo/seed-situation.sh: demo_qa seed branch was reset.