Skip to content

TEST Focus frontend E2E checks on user-visible failures - #3535

Open
Richard Lundeen (richlundeen) wants to merge 3 commits into
microsoft:mainfrom
richlundeen:richlundeen-reliable-ci-tests
Open

Richard Lundeen (richlundeen) wants to merge 3 commits into
microsoft:mainfrom
richlundeen:richlundeen-reliable-ci-tests

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Description

Focus frontend E2E checks on user-visible failures instead of cosmetic layout details.

The converter test in the failed job failed on a 1.125-pixel width difference, then on programmatic scrolling. Replace its frame sampler with keyboard scrolling and selection at the same three screen sizes.

Also remove exact-position comparisons, one-pixel reset-button alignment, desktop compactness checks, and arbitrary popup/card/input size limits from related suites. Check filter reset and mobile drawer behavior instead. Keep content containment, overflow, scrolling, focus, and minimum touch-target checks.

No production code or CI failure rules change. The removed cosmetic checks are not replaced with weaker pixel tolerances or more retries.

Tests and Documentation

Removed the README addition. Added two short E2E rules to the frontend test instructions.

Commands below ran from frontend, with the worktree's Python environment selected by PYRIT_PYTHON and UV_NO_SYNC=1:

  • npx playwright test registry.spec.ts --project=mock --grep "last converter" --retries=0 --repeat-each=20 --workers=1 --reporter=list - passed: 60/60, with no retries. Used a dedicated Vite server on port 3107. This local run had no backend; the CI-mode check below also covers backend startup.
  • npx playwright test chat.spec.ts config.spec.ts datasets.spec.ts labels-operation-picker.spec.ts registry.spec.ts touch-targets.spec.ts --project=mock --grep "mobile conversations|Responsive Target Registry|dataset catalog layout|wraps a long harm category inside the example table|stacks detail labels|keeps the list on screen|keeps metadata inputs|Converter Registry|Mobile touch targets" --retries=0 --workers=1 --reporter=list - passed: 30/30. Used CI=true, E2E_SEEDED_MODE=true, FAIL_ON_SKIPPED_TESTS=true, frontend port 3110, and backend port 18110. Refreshed the development compatibility stamp before starting the backend and frontend.
  • npm test -- --runInBand --runTestsByPath src\components\Registry\CreateConverterDialog.test.tsx - passed: 53/53.
  • npx eslint e2e\accessibility.spec.ts e2e\chat.spec.ts e2e\config.spec.ts e2e\datasets.spec.ts e2e\labels-operation-picker.spec.ts e2e\registry.spec.ts e2e\touch-targets.spec.ts --max-warnings 0 - passed.
  • npm run type-check - passed.
  • npx playwright test --list --reporter=list - passed: 326 tests discovered.

From the repository root, uv run --no-sync pre-commit run --all-files - passed, including all hooks.

JupyText was not run. No Python or notebook code changed.

Check keyboard scrolling and selection at the existing viewports instead of sampling popup animation frames or forcing scroll position.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove cosmetic position and size checks while keeping containment, scrolling, focus, actions, and minimum touch targets. Drop README test details and keep concise guidance in the test instructions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@richlundeen Richard Lundeen (richlundeen) changed the title TEST Replace flaky converter popup geometry checks TEST Focus frontend E2E checks on user-visible failures Oct 25, 2026
const lastOption = listbox.getByRole("option").last();
await lastOption.scrollIntoViewIfNeeded();
await expect(lastOption).not.toBeInViewport();
await trigger.press("End");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep the converter popup's viewport-containment check

Could we keep a whole-popup viewport check alongside the new keyboard flow? The deleted checks caught menus opening off-screen, which is a usability issue rather than a cosmetic size difference.

I tried a browser-only CSS mutation, [role="listbox"] { translate: 200px 0 !important; }, without changing the app or these tests. All three replacement cases still passed. At the 390px viewport, the popup opened from x=241 to x=549, leaving 159px, including its scrollbar, off-screen. The automated click scrolled it into view later, so successful selection did not catch the bad initial placement. Restoring the old viewport-bound check rejected this mutation, while the unmodified app passed.

A check such as await expect(listbox).toBeInViewport({ ratio: 1 }) after opening would preserve this coverage without restoring the frame sampler or exact anchor-width comparisons.

Check the whole popup before keyboard navigation or clicks can scroll it into view. Keep the behavior-based flow without frame sampling or exact geometry checks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants