Skip to content

test: automate 54/60 manual QA rows with Playwright on MySQL 8 + coverage matrix - #338

Closed
devin-ai-integration[bot] wants to merge 1 commit into
demo/qa-live-beforefrom
devin/1792763138-qa-playwright-automation
Closed

devin-ai-integration[bot] wants to merge 1 commit into
demo/qa-live-beforefrom
devin/1792763138-qa-playwright-automation

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Summary

Automates qa/manual-test-plan.csv as Playwright tests (e2e/) that run in CI against the real app on a MySQL 8 service container (.github/workflows/e2e.yml). 54 of 60 rows automated; the 6 that aren't are listed with reasons in qa/coverage-matrix.md, which covers all 60 rows.

  • Each test title starts with its plan ID (test('MT-031 Overdraw is rejected', ...)). e2e/scripts/check-coverage.mjs runs in CI and fails if a plan row is missing from the matrix, an Automated row has no matching test (or more than one), or a Not automated row has no reason.
  • Each test registers its own unique customers, so there are no shared fixtures and tests can run in parallel. Local run: 54 passed in 17.7s against MySQL 8 (demo/start-demo.sh).
  • "Form is not submitted" rows (MT-025/026/034/043/044) record outgoing POST requests and assert there were none, plus check the HTML5 validity flag. The balance check alone isn't enough to prove it.
  • Dashboard panels are Bootstrap collapses loaded from a CDN, so openPanel waits for jQuery.fn.collapse before clicking. Without that wait the tests race in CI.
  • CI builds with ./mvnw -s demo/maven-settings.xml (GCS mirror) and uses mirror.gcr.io/library/mysql:8.0, as AGENTS.md requires. It also asserts that the server version is 8.*.

Not automated (6):

ID Why
MT-021 The expected result is a brand-team sign-off against a brand guide that isn't in the repo, so there's no pass condition a machine can check.
MT-047 The expected result is "System handles it appropriately", which isn't a defined behaviour, so I didn't invent one. What happens today: a -50 transfer is accepted and takes $50 from the recipient (sender $100 → $150, recipient $0 → $-50). A -50 deposit is also accepted. This needs a product decision.
MT-048 The app has no SMS or email capability, so there's nothing to observe.
MT-055 Needs a physical branch printer and a person to judge the printout.
MT-059 Needs JAWS on Windows and a person to judge whether announcements are clear. An axe scan wouldn't be equivalent.
MT-060 Moderated usability session with a real first-time customer.

Test precondition I added: MT-042 deposits 100 first. transferAmount checks the balance before it looks up the recipient, so a $0 sender would get "Insufficient funds" instead of "Recipient account not found".

Link to Devin session: https://devin-gtm.devinenterprise.com/sessions/5408d659a20a42ebb3c4cd2c8b086de9
Open in Devin Desktop: https://devin-gtm.devinenterprise.com/desktop/session/5408d659a20a42ebb3c4cd2c8b086de9?variant=devin
Requested by: @GarretAnd


Devin Review

…d coverage matrix

Co-Authored-By: Garret Andreine <garret.andreine@cognition.ai>
@devin-ai-integration

Copy link
Copy Markdown
Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@devin-ai-integration

Copy link
Copy Markdown
Author

Closed by demo/seed-situation.sh: demo_qa seed branch was reset.

@devin-ai-integration
devin-ai-integration Bot deleted the devin/1792763138-qa-playwright-automation branch October 26, 2026 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant