Skip to content

Test Suite: no leftovers from older suites, and tests that pass on every device - #370

Merged
kloptops merged 2 commits into
PortsMaster:mainfrom
kloptops:fix-test-leftovers
Nov 2, 2026
Merged

kloptops merged 2 commits into
PortsMaster:mainfrom
kloptops:fix-test-leftovers

Conversation

@kloptops

@kloptops kloptops commented Nov 2, 2026

Copy link
Copy Markdown
Member

I ran the PortMaster Test Suite on seven devices: muOS (RG35XX-H), AmberELEC (RG351MP), ROCKNIX (Retroid Pocket 6, MANGMI Air X), Knulli (TrimUI Smart Pro), dArkOS (RG353V) and spruce (Miyoo Flip). PortMaster passed on all of them, but every run still failed because of problems in the tests and the suite. This fixes those, so a clean device now reports a clean run. No PortMaster code changes.

Changes

Test Suite (tools/test_suite, do_release.sh)

  • Installing the suite over an older one kept the older one's files, because installing a port only adds and replaces files. A test the new build no longer has (test_i18n_snapshot) still ran, and failed. The build now lists every file in tree/ and site/ in pmtest/files.txt. When it starts, the suite removes anything else there (suite.prune_stale); without the list it removes nothing.

Tests that failed only on devices (outside a git checkout)

  • test_get_path_fs_testing assumed a checkout (HM_TESTING); it now sets it.
  • test_gui_get_config_broken uses the GUI fixture but wasn't marked sdl, so it ran where there's no dummy video driver (muOS, Knulli).
  • On a device, the language, theme and color scheme tests write .pugwash-reboot into tools/PortMaster. conftest now creates that folder, as an install has it.
  • test_theme_files expected the Noto fonts, which the suite leaves out. It now checks for them only where PortMaster has them. A new test covers listing them from NotoSans.tar.xz.
  • The fuzzer on AmberELEC: that Python can't recognize NaN (math.isnan(nan) is False), so json.dumps writes a bare nan that json.loads refuses. The fuzzer's own fake downloads round-trip through JSON, so they failed there. It now generates NaN and infinity only where they survive a round trip. PortMaster's code was never involved.
  • test_stop_gives_up_waiting (theme server) left its start thread running. That thread later woke after monkeypatch had put the real Worker back, and failed inside another test's thread (seen on the RG353V). The test now waits for it.
  • The fuzzer imports harbourmaster.cli at the top. Importing it opens its log in HM_TOOLS_DIR/PortMaster, which now exists, and inside a fuzz case that counted as a write outside the sandbox.

Testing

  • Full suite: 2377 passed, 2 xfailed, on Python 3.7 and 3.11. The affected tests also pass on 3.12, 3.13 and 3.14. CI's flake8 lint is clean.
  • tests/test_build.py checks that files.txt lists exactly the files in tree/ and site/.
  • tests/test_test_suite.py covers pruning: leftovers removed, listed files kept, nothing outside tree/ and site/ touched, and nothing removed without the list.
  • The device-only fixes were checked on muOS and Knulli. The fuzzer, theme-server and font fixes come from failures seen on AmberELEC, dArkOS and all seven devices respectively. The Test Suite hasn't been run on the devices with this branch.

kloptops and others added 2 commits November 2, 2026 10:24
Found running the PortMaster Test Suite on muOS (Jacaranda, RG35XX-H):
1340 passed, 2 failed; GUI tests 405 passed, 4 failed.

- The language, theme and colour scheme tests: outside a checkout the
  scenes touch HM_TOOLS_DIR/PortMaster/.pugwash-reboot, and the test
  session's tools dir had no PortMaster folder. conftest makes it, as
  an install has it.
- test_get_path_fs_testing assumed a checkout (HM_TESTING); it sets it.
- test_gui_get_config_broken uses the gui fixture but wasn't marked
  sdl, so it ran in the non-SDL pass (muOS has no dummy video driver).

All six pass on the device now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NxABmR1rUUgi4iicVLZchY
…on every device

From the Test Suite on seven devices (muOS, AmberELEC, ROCKNIX x2, Knulli,
dArkOS, spruce):

- pmtest: the build lists tree/ and site/ in pmtest/files.txt and the suite
  removes anything else there when it starts. Installing over an older suite
  kept its files: a dropped test (test_i18n_snapshot) still ran and failed.
- test_theme_files expected the Noto fonts, which the Test Suite leaves out;
  it only does where PortMaster has them, and a new test covers the archive.
- The fuzzer only makes NaN/infinity where JSON round-trips them: AmberELEC's
  Python can't recognise NaN, so json.dumps wrote a bare `nan` and the
  fuzzer's own fake downloads failed (PortMaster code wasn't involved).
- test_stop_gives_up_waiting waits for its start thread: left running, it woke
  after monkeypatch had put the real Worker back and failed in a later test
  (seen on the RG353V).
- The fuzzer imports harbourmaster.cli up front: on import it opens its log in
  HM_TOOLS_DIR/PortMaster, which conftest now makes (the device test fixes,
  cherry-picked here), and inside a fuzz case that counted as a write outside
  the sandbox.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NxABmR1rUUgi4iicVLZchY
@kloptops
kloptops merged commit 9ed2928 into PortsMaster:main Nov 2, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant