Repository navigation
Test Suite: no leftovers from older suites, and tests that pass on every device - #370
Merged
Merged
Conversation
Found running the PortMaster Test Suite on muOS (Jacaranda, RG35XX-H): 1340 passed, 2 failed; GUI tests 405 passed, 4 failed. - The language, theme and colour scheme tests: outside a checkout the scenes touch HM_TOOLS_DIR/PortMaster/.pugwash-reboot, and the test session's tools dir had no PortMaster folder. conftest makes it, as an install has it. - test_get_path_fs_testing assumed a checkout (HM_TESTING); it sets it. - test_gui_get_config_broken uses the gui fixture but wasn't marked sdl, so it ran in the non-SDL pass (muOS has no dummy video driver). All six pass on the device now. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NxABmR1rUUgi4iicVLZchY
…on every device From the Test Suite on seven devices (muOS, AmberELEC, ROCKNIX x2, Knulli, dArkOS, spruce): - pmtest: the build lists tree/ and site/ in pmtest/files.txt and the suite removes anything else there when it starts. Installing over an older suite kept its files: a dropped test (test_i18n_snapshot) still ran and failed. - test_theme_files expected the Noto fonts, which the Test Suite leaves out; it only does where PortMaster has them, and a new test covers the archive. - The fuzzer only makes NaN/infinity where JSON round-trips them: AmberELEC's Python can't recognise NaN, so json.dumps wrote a bare `nan` and the fuzzer's own fake downloads failed (PortMaster code wasn't involved). - test_stop_gives_up_waiting waits for its start thread: left running, it woke after monkeypatch had put the real Worker back and failed in a later test (seen on the RG353V). - The fuzzer imports harbourmaster.cli up front: on import it opens its log in HM_TOOLS_DIR/PortMaster, which conftest now makes (the device test fixes, cherry-picked here), and inside a fuzz case that counted as a write outside the sandbox. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NxABmR1rUUgi4iicVLZchY
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I ran the PortMaster Test Suite on seven devices: muOS (RG35XX-H), AmberELEC (RG351MP), ROCKNIX (Retroid Pocket 6, MANGMI Air X), Knulli (TrimUI Smart Pro), dArkOS (RG353V) and spruce (Miyoo Flip). PortMaster passed on all of them, but every run still failed because of problems in the tests and the suite. This fixes those, so a clean device now reports a clean run. No PortMaster code changes.
Changes
Test Suite (
tools/test_suite,do_release.sh)test_i18n_snapshot) still ran, and failed. The build now lists every file intree/andsite/inpmtest/files.txt. When it starts, the suite removes anything else there (suite.prune_stale); without the list it removes nothing.Tests that failed only on devices (outside a git checkout)
test_get_path_fs_testingassumed a checkout (HM_TESTING); it now sets it.test_gui_get_config_brokenuses the GUI fixture but wasn't markedsdl, so it ran where there's no dummy video driver (muOS, Knulli)..pugwash-rebootintotools/PortMaster.conftestnow creates that folder, as an install has it.test_theme_filesexpected the Noto fonts, which the suite leaves out. It now checks for them only where PortMaster has them. A new test covers listing them fromNotoSans.tar.xz.math.isnan(nan)is False), sojson.dumpswrites a barenanthatjson.loadsrefuses. The fuzzer's own fake downloads round-trip through JSON, so they failed there. It now generates NaN and infinity only where they survive a round trip. PortMaster's code was never involved.test_stop_gives_up_waiting(theme server) left its start thread running. That thread later woke aftermonkeypatchhad put the realWorkerback, and failed inside another test's thread (seen on the RG353V). The test now waits for it.harbourmaster.cliat the top. Importing it opens its log inHM_TOOLS_DIR/PortMaster, which now exists, and inside a fuzz case that counted as a write outside the sandbox.Testing
tests/test_build.pychecks thatfiles.txtlists exactly the files intree/andsite/.tests/test_test_suite.pycovers pruning: leftovers removed, listed files kept, nothing outsidetree/andsite/touched, and nothing removed without the list.