Known failures - #3042
Closed
Milosz Wasilewski (mwasilew) wants to merge 4 commits into
Closed
Known failures#3042Milosz Wasilewski (mwasilew) wants to merge 4 commits into
Milosz Wasilewski (mwasilew) wants to merge 4 commits into
Conversation
Some tests are known to fail on a given device and their failure is accepted for the time being, usually because the fix belongs to another component and is already tracked in an issue. Until now the test job summary reported them like any other failure, so every run showed the same red cells and the reader had to remember which of them were already known. A genuine regression was easy to miss among them. Add a list of known failures per tested build variant under .github/known-failures, keyed by LAVA device type, where "*" stands for every device of the variant. Each entry carries the test name and a comment naming the issue the failure is tracked in, so that the list stays reviewable and does not turn into a pile of unexplained names. Apply the list when the summary is rendered: a listed failure is reported as a known failure and counted like a pass, and a test that passes although it is listed is reported as an unexpected pass and counted as a failure. The second half is what keeps the lists honest, as an entry that is no longer needed makes the summary fail until it is removed. The applied entries are listed with their comments in a section of their own below the results table. The list is picked by name from the distro and its suffix, so a new build variant gets its own list without any further wiring. Take that list from the pull request under test rather than from the base branch. The test chain runs on workflow_run, so its own checkout is the base branch and the list found there is the one already merged, which would leave a pull request that fixes a listed failure reported against a stale list until it lands. Check the lists out a second time from the branch or fork the pull request is built from, sparsely and read only as data, everything the chain runs still coming from the base branch. A run with no pull request behind it, and a fork that is no longer reachable, keep using the list of the checkout. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
The test job summary is not the only place a failure is reported. The LAVA JUnit files are published as the "Test Results" check, which fails on any failed test case, so an accepted failure kept that check red no matter what the summary said. The check is the one that gates a pull request, which made the lists of little use on their own. Rewrite the result files before they are published. A failure that is on the list of its build variant becomes a skipped test case, JUnit having no better way to express an expected failure, so it no longer fails the check while staying visible in the report. A test case that passes although it is listed becomes a failure, matching what the summary reports and asking for the entry to be removed. The comment of the entry is appended to the message of the rewritten test case, so the issue is one click away from the report. A result file is mapped to a variant and a device through its name, the one lava-test-plans builds. The lists come from the branch or fork the pull request is built from, like the ones the summary applies; both checkouts this needs have to happen before the artifacts are downloaded, as they would otherwise wipe them from the workspace. The lists are only read at the end of a test run, hours after a change to them was pushed, and an entry naming a variant or a device that is not tested is not an error at that point, it is simply never applied. Such a list looks like it suppresses a failure while the check stays red. Validate the lists on every change to them instead, against the build variants and devices the test workflow really tests. That last check belongs to the branch the lists live on, not to the lists themselves, so it is not one to make of a list coming from a pull request: such a list may well add or drop a variant. Add --syntax-only for that case, checking that the lists are well formed and no more, and fall back to the lists of the base branch when they are not - a list that does not even parse would otherwise abort the run publishing the results. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
The repository documents how to build an image but not what happens to it afterwards. Which build variants are tested on hardware, on which devices, and where to look for the results is spread over the workflow files, and the known failures lists that were just added have no entry point other than the README next to them. Add TESTING.md, describing when the tests run, the three build variants that are tested and the devices each of them covers, the boot and pre-merge stages, and the three places the results are reported. Give the known failures a section of their own, from adding an entry to what it changes in each report and why an entry that is no longer needed makes the check fail. Link it from the README. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
The result cells of the summary table linked to the resource_uri the LAVA API reports for a test case. That is the API endpoint of the test case itself: following it returns a JSON document restating the result that was just read from the table, over plain http, and leaves the reader to find the job and the failure in it by hand. The job log renders every line as an element with an "L<n>" id and offers a link to it, and the API reports the log line a test case was recorded on. Build the link from those instead, so that a result in the table leads straight to the line the test reported it on, with the output of the test around it. A boot job has no test case to point at, so link it to its log rather than to its results page, the log being what a boot failure has to be read from anyway. Assisted-by: Claude Code:claude-opus-5 Signed-off-by: Milosz Wasilewski <milosz.wasilewski@oss.qualcomm.com>
Milosz Wasilewski (mwasilew)
force-pushed
the
known-failures
branch
from
August 31, 2026 16:42
c1cf765 to
7f73080
Compare
Test Results 68 files 319 suites 6h 2m 16s ⏱️ For more details on these failures, see this check. Results for commit 7f73080. |
Contributor
|
I guess the main question from my side is if the known failures are just taken from the master branch or if they come from the PR merged tree. We really need the latter case, otherwise it will be a nightmare to handle them. |
Contributor
Author
I assume you're trying to cover the situation when PR fixes a bug. In such case the list of known issues should also be updated to keep the CI "green". I don't think it's the case with this PR. This approach makes sense. Let me try to implement it. |
Contributor
Author
|
This PR is obsolete because of #3137 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In attempt to make failed test signal significant, this series adds support for "known issues" file. The file is kept in the repository that uses test-distro.yml reusable workflow. Know issues are evaluated at the reporting stage. Raw results can still be retrieved from LAVA logs.