Conversation
|
First CI run: the wheels build, the wheel can't run — needs a decision before any re-runRun 35480766550, job 105998045327, 01:12–13:36 UTC (12h24m). The build is not the problem. Exit 132 = 128+4 = SIGILL. None of the usual suspects apply: no The failing point is exact — in the cp312 smoke check, Root cause: the RISC-V CPU plugin requires RVV 1.0, this fleet does not have itDiagnosed from the uploaded artifact rather than by spending another runner slot. Of the 20 libraries in the wheel, It holds 64,554 vector (OP-V) instructions, 34,526 of them Upstream's gate is a runtime probe ( Static evidence can't separate "the probe's SIGILL recovery fails" from "a static initializer inside one of the RVV-compiled TUs runs at Why the triage didn't catch this: upstream's Second, independent blocker: the bundled oneTBB is a T-Head vendor buildFound the same way. The prebuilt oneTBB that It works here only because the runner fleet is itself T-Head (which also explains gotcha 272's otherwise-odd "HWCAP says V but Why I have not re-triggeredThere is no off switch: And an RVV-off build still couldn't ship while the T-Head TBB blob is in the wheel. A shippable wheel needs both an RVV patch and a replacement for the prebuilt TBB (source-built oneTBB, or At this cost that's a call worth making deliberately rather than by pushing a speculative re-run. The options as I see them:
No code pushed to this branch; PR left open and draft status untouched. Findings recorded on |
Two confirmed riscv64 blockers from CI run 35480766550 (gotchas 448/449), targeting the generic rv64gc baseline this fleet actually runs: - cmake/dependencies.cmake's ov_download_tbb() fetches a prebuilt oneTBB for riscv64 (oneapi-tbb-2022.3.0-lin-riscv-release.tgz) that turns out to be a T-Head Xuantie toolchain build carrying vendor CUSTOM-0 opcodes (xtheadc) -- it "works" only because this runner fleet also happens to be T-Head silicon, not because it's a portable riscv64 build. oneAPI/Intel does not publish a riscv64 oneTBB release themselves; this asset is OpenVINO's own CI team cross-compiling the same source with (evidently) a T-Head-flavored default -march. Build oneTBB v2022.3.0 -- the exact version OpenVINO pins -- from its real upstream source (oneapi-src/oneTBB) with the manylinux image's own generic-rv64gc toolchain instead, and point OpenVINO at it with -DTBB_DIR, which ov_find_package_tbb() already honors ahead of ever calling ov_download_tbb(). No source patch needed for this half. - libopenvino_riscv_cpu_plugin.so unconditionally probes for RVV 1.0 by executing a v1.0 vsetivli under a SIGILL handler (nodes/kernels/riscv64/cpu_isa_traits.cpp); on this fleet's RVV 0.7.1 T-Head hardware the probe doesn't recover, it SIGILLs the whole process during ov.Core()/device enumeration. The CPU plugin already has a complete non-RVV fallback for riscv64 built in and already exercised by this exact mayiuse(gv) gate -- scalar/generic node executors, and Transformations::MainSnippets() already turns Snippets/Subgraph tokenization off entirely when is_supported_isa() is false -- so hardcoding the two probes to report "unsupported" (patches/openvino/ 2026.3.1/0001-...) routes this target through that existing path unconditionally instead of ever attempting RVV 1.0 codegen. Adds a python-wheels self-checkout and `git apply` step (this workflow previously had nowhere to source patches from), following the build-tensorstore.yml checkout-into-subdir pattern.
Run 35995279535 (with patch 0001 applied and compiled) still died with
SIGILL at exactly the same point as run 35480766550: ov.get_version()
printed, print(core.available_devices) did not (podman run -t, so stdout
is line-buffered and the missing line is real). So the RVV 1.0 probe was
not the crash.
The crash is a static initializer of libopenvino_riscv_cpu_plugin.so,
run by the dynamic loader when available_devices dlopens the CPU plugin:
.init_array entry 17 points at plugin offset 0x34926a, whose first
instruction is `vsetivli zero, 8, e8, mf2, ta, ma` (a GCC auto-vectorized
zero-fill of globals, followed by vmv.v.i/vse8.v). vsetivli and fractional
LMUL do not exist in RVV 0.7.1, and no probe or HWCAP check runs before
it. It is the only one of the plugin's 26 initializers with vector code.
Reproduced from the CI artifact under QEMU: QEMU_CPU=rv64,v=false dies
with SIGILL right after ov.Core() and before available_devices prints,
si_addr = plugin base + 0x34926a; with v=true,vext_spec=v1.0 the whole
smoke test passes (which is why upstream's QEMU-with-V CI is green).
Where the vector code comes from: oneDNN's cmake/platform.cmake, for
DNNL_TARGET_ARCH=RV64, appends -march=rv64gcv to all of oneDNN's C/C++
flags whenever the compiler can build RVV intrinsics (the CI log says
"Using RV64 march flag: -march=rv64gcv"). There is no runtime dispatch
behind it, and oneDNN is linked statically into the plugin. The plugin's
own sources and every other library in the wheel are built rv64gc; the
riscv64 JIT emitters generate vector code at run time only.
Pre-seed CAN_COMPILE_RVV_INTRINSICS=OFF so oneDNN takes its existing
non-RVV branch: -march=rv64gc, DNNL_RISCV_USE_RVV_INTRINSICS off, the
rvv_* sources excluded, and every reference to them already compiled
out by #ifdef. Checked by configuring oneDNN 2026.3.1's submodule with
it in the manylinux_riscv64 image ("Using RV64 march flag:
-march=rv64gc", 177 of 180 TUs left), and by compiling the guarded TUs
(gemm.cpp, cpu_matmul_list.cpp, cpu_pooling_list.cpp): they build, with
zero vector instructions and no V in Tag_RISCV_arch.
Patch 0002 closes the one remaining unguarded vector instruction on the
compile_model() path: mayiuse() constructs Xbyak_riscv::CPU for every
query, and its constructor runs a JIT-generated `csrr a0, vlenb` with no
SIGILL handler whenever AT_HWCAP has V. The gv case now checks the
(hardcoded) RVV 1.0 probe first and only g builds the CPU object. It is
defensive -- not yet seen crashing by itself -- and compiles with
-Wall -Wextra -Werror in the manylinux_riscv64 image.
Run 36145686942 was cancelled at the 24h job limit, but not by the build: the portable-oneTBB/no-RVV build finished in 12h20m (14:14 to 02:34 UTC), all four wheels were built and uploaded, and the cp312/cp313/cp314 legs each passed the smoke check and 7019 passed / 61 skipped in about three minutes. The earlier SIGILL is gone. The cp314t leg passed the smoke check too (Core(), ['CPU'], a relu compiled and run on CPU), then hung in pytest right after test_async_infer_request.py::test_start_async[True] from 02:49 until the cancellation at 14:11, 11h22m without output. The next test, test_start_async[False], sets a Python callback that runs on an inference worker thread while the main thread waits. The free-threaded build (ENABLE_GIL_PYTHON_API=OFF) is the only one that hangs there. Keep building and smoke-testing cp314t, since upstream ships it, but run the pytest suite only on the GIL interpreters. Also give the test step its own 60-minute timeout. Upstream's job_python_api_tests.yml caps its run at 30, and without a limit here a hang holds the runner until the 24h job limit.
This branch has not been deployed
1 participant
openvino2026.3.1Compiles the OpenVINO runtime, its IR/ONNX/TensorFlow/TF-Lite/PyTorch/Paddle frontends and the RISC-V CPU plugin - vendored oneDNN at
DNNL_TARGET_ARCH=RV64plus the RVV JIT emitters built onxbyak_riscv- then the Python API once per interpreter. Upstream publishes no riscv64 wheel.Mirrors upstream's
manylinux_2_28.yml, the recipe behind the published wheels, with the riscv64 cmake options oflinux_riscv.yml. The wheel's platform tag, the RVV CPU plugin and the prebuilt riscv64 oneTBB all come from branches upstream already carries.Differs from upstream
cmakeinstalled - vendored snappy and xbyak_riscv predate CMake 4's floorTesting
pyopenvinosuite narrowed totest_runtime/test_graph/test_transformations- the rest need the template plugin that only the tests component buildsLicense: Wheel bundles oneTBB (Apache-2.0), fetched by upstream's own riscv64 branch.