ci: add testimage - #1900
Open
Jose Quaresma (quaresmajose) wants to merge 7 commits into
Open
ci: add testimage#1900Jose Quaresma (quaresmajose) wants to merge 7 commits into
Jose Quaresma (quaresmajose) wants to merge 7 commits into
Conversation
Jose Quaresma (quaresmajose)
force-pushed
the
ptest
branch
2 times, most recently
from
April 8, 2026 15:00
1666dab to
eae42e9
Compare
Jose Quaresma (quaresmajose)
requested review from
Anuj Mittal (anujm1),
Koen Kooi (koenkooi) and
Viswanath Kraleti (vkraleti)
as code owners
April 8, 2026 15:00
Jose Quaresma (quaresmajose)
marked this pull request as draft
April 8, 2026 15:01
Contributor
|
ptests won't work with sota, and it is even removed from distro features. |
Member
Author
I forgot about that, I'll remove it from sota. |
Test run workflowTest jobs for commit eae42e9024cf8ad4c3269a261c94d0fb22c470fe
All jobs summary
|
Test Results 68 files - 13 319 suites - 107 6h 36m 31s ⏱️ - 47m 14s For more details on these failures, see this check. Results for commit 24d3413. ± Comparison against base commit c23d167. This pull request removes 16 tests.This pull request skips 1 and un-skips 1 tests.♻️ This comment has been updated with latest results. |
Jose Quaresma (quaresmajose)
force-pushed
the
ptest
branch
from
April 8, 2026 18:11
eae42e9 to
8456571
Compare
Jose Quaresma (quaresmajose)
marked this pull request as ready for review
April 8, 2026 18:11
Test run workflowTest jobs for commit 8456571519bc81ae843635979eea289831218b49
All jobs summary
|
Contributor
|
I'm not sure we should add ptests by default in every image, maybe in only one variant? What is the image size implication here? |
Contributor
|
As I said in #1903:
|
Member
Author
I'll measure that and add it to the commit. |
Member
Author
So how do you think we should go about installing the tests on the target? |
Member
Author
Here are the results for Before this PR: After this PR: |
Jose Quaresma (quaresmajose)
force-pushed
the
ptest
branch
from
April 9, 2026 15:41
8456571 to
7ea1046
Compare
Jose Quaresma (quaresmajose)
marked this pull request as draft
April 9, 2026 16:02
Merged
Contributor
This is a massive difference, we should discuss this more before pushing any changes here. |
Member
Author
|
Yes, I agree. |
Dmitry Baryshkov (lumag)
added a commit
that referenced
this pull request
Apr 10, 2026
Jose Quaresma (quaresmajose)
force-pushed
the
ptest
branch
from
April 10, 2026 13:58
7ea1046 to
3ac172e
Compare
Member
Author
For now it runs the default tests definded upstream in oe-core, The "SKIPPED" tests are due to missing packages in the image. I would prefer to start with the default definition and then adapt it to our needs. |
Jose Quaresma (quaresmajose)
force-pushed
the
ptest
branch
from
September 8, 2026 15:48
906cd47 to
441f994
Compare
| header: | ||
| version: 14 | ||
| includes: | ||
| - ci/genericarm64.yml |
Contributor
There was a problem hiding this comment.
Name it genericarm64-testimage or similar, since it is specific to this machine.
Member
Author
There was a problem hiding this comment.
I updated to ci/genericarm64-testimage.yml but maybe it would be better to just be ci/genericarm64.yml, because otherwise it gets too big.
| - testimage | ||
| distro: | ||
| - name: nodistro | ||
| yamlfile: "" |
Contributor
There was a problem hiding this comment.
Should probably take 'profile' into account, and decide if to run on PR, nightly or weekly.
Member
Author
There was a problem hiding this comment.
Yes, I think the nightly is sufficient for now and will allow us to better assess the situation.
Member
Author
There was a problem hiding this comment.
'profile' has not yet been implemented.
Jose Quaresma (quaresmajose)
force-pushed
the
ptest
branch
from
September 11, 2026 15:16
441f994 to
e594798
Compare
Member
Author
|
I removed the PTEST; because of this, the time required will be significantly reduced. I'll update the execution time later. |
Jose Quaresma (quaresmajose)
force-pushed
the
ptest
branch
from
September 11, 2026 15:28
e594798 to
8a49744
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+192
to
+215
| testimage-run: | ||
| needs: [testimage, github-cache-check, compile-env] | ||
| if: | | ||
| github.repository_owner == 'qualcomm-linux' && | ||
| (inputs.skip_if_github_cached != true || needs.github-cache-check.outputs.cache_hit != 'true') | ||
| strategy: | ||
| fail-fast: false | ||
| matrix: | ||
| machine: | ||
| - genericarm64-testimage-run | ||
| distro: | ||
| - name: nodistro | ||
| yamlfile: "" | ||
| uses: ./.github/workflows/compile.yml | ||
| with: | ||
| run_on: arm64-metal | ||
| name: ${{ matrix.machine }}/${{ matrix.distro.name }} | ||
| world: false | ||
| machine: ${{matrix.machine}} | ||
| distro_yaml: ${{matrix.distro.yamlfile}} | ||
| distro_name: ${{matrix.distro.name}} | ||
| kernel_yaml: '' | ||
| kernel_dirname: '' | ||
| cache_dir: ${{ needs.compile-env.outputs.CACHE_DIR }} |
Member
Author
The full time of the testimage-run job takes less then 15 minutes. https://github.com/qualcomm-linux/meta-qcom/actions/runs/34616349986/job/103353948519?pr=1900 |
Jose Quaresma (quaresmajose)
force-pushed
the
ptest
branch
2 times, most recently
from
September 16, 2026 14:59
83634ba to
24d3413
Compare
| distro: | ||
| - name: nodistro | ||
| yamlfile: "" | ||
| uses: ./.github/workflows/compile.yml |
| distro: | ||
| - name: nodistro | ||
| yamlfile: "" | ||
| uses: ./.github/workflows/compile.yml |
Jose Quaresma (quaresmajose)
force-pushed
the
ptest
branch
from
October 2, 2026 21:06
24d3413 to
4dcfe58
Compare
We have a special interest in the meta-yocto-bsp layer that supplies the genericarm64 machine. Signed-off-by: Jose Quaresma <jose.quaresma@oss.qualcomm.com>
Add kas fragment support for the genericarm64 machine which is provided in the meta-yocto-bsp layer from the meta-yocto. The genericarm64 machine definition in the Yocto Project is a standardized 64-bit ARM architecture target designed to boot on Arm SystemReady compliant hardware. By utilizing UEFI for boot and mainline Linux configurations, it eliminates the need for hardware-specific Board Support Packages (BSPs) across compliant devices. We still need to drop 'cmdline' and 'qcomflash' that come from ci/base.yml. Signed-off-by: Jose Quaresma <jose.quaresma@oss.qualcomm.com>
There are scenarios where we want to choose the machine that will execute the workflow. We have 3 build machines available: [1] amd64: default used [2] arm64: used to build testimage [3] arm64+kvm: used to run testimage The [3] arm64+kvm has higher costs. So we use the [2] arm64 to build everything first and after that, the [3] arm64+kvm to run the tests. Signed-off-by: Jose Quaresma <jose.quaresma@oss.qualcomm.com>
Add kas fragment support to build oe-core testimage. The testimage is provided in the class testimage.bbclass which handles loading the tests and starting the qemu image. More information on how to use this class can be found in [1] and [2]. To run the standard testimage tests, the dependencies in CORE_IMAGE_EXTRA_INSTALL were added. [1] https://wiki.yoctoproject.org/wiki/Image_tests [2] https://docs.yoctoproject.org/test-manual/runtime-testing.html#performing-automated-runtime-testing Signed-off-by: Jose Quaresma <jose.quaresma@oss.qualcomm.com>
Add ci job to build testimage on nodistro. The qcom-distro is not ready for testimage and several adaptations need to be made. The rationale behind this step is to produce the artifacts to be used in the target image. This allows them to be used quickly when needed because they can be rapidly restored from the sstate-cache. Signed-off-by: Jose Quaresma <jose.quaresma@oss.qualcomm.com>
This kas fragment is a complement to b/ci/testimage.yml Initially, the two were together, but this way we can have the runtime configuration separate from the build configuration. This, in turn, allows for a more streamlined build where only the native components are needed to run the testimage. More information on how to use the testimage class in [1][2][3] and [4]. [1] https://wiki.yoctoproject.org/wiki/Image_tests [2] https://docs.yoctoproject.org/test-manual/runtime-testing.html#performing-automated-runtime-testing [3] https://wiki.yoctoproject.org/wiki/Ptest [4] https://docs.yoctoproject.org/test-manual/ptest.html Signed-off-by: Jose Quaresma <jose.quaresma@oss.qualcomm.com>
Add testimage-run ci job to build and run testimage on nodistro. The qcom-distro is not ready for testimage and several adaptations need to be made. This job is performed on a arm64-metal VM machine because it provides KVM to allows for much more efficient image emulation. This job uses artifacts produced in testimage ci job for the target architecture, but the native part has to be produced since the host is not the same as the one used in on the testimage ci job. Signed-off-by: Jose Quaresma <jose.quaresma@oss.qualcomm.com>
Jose Quaresma (quaresmajose)
force-pushed
the
ptest
branch
from
October 2, 2026 21:09
4dcfe58 to
9ec37b5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The build system has the ability to run a series of automated tests for qemu images.
All the tests are actually commands run on the target system over ssh.
The tests themselves are written in Python, making use of the unittest module.
The class that enables this is testimage.bbclass (which handles loading the tests and starting the qemu image)
For now it runs the default tests definded upstream in oe-core, The "SKIPPED" tests are due to missing packages in the image. I would prefer to start with the default definition and then adapt it to our needs.
Depends on #2688