Skip to content

tools/testing: Host tests for the kernel failure paths - #128

Closed
brkosebay wants to merge 56 commits into
OrchardLinux:mainfrom
brkosebay:j71t/kernel-safety-tests
Closed

brkosebay wants to merge 56 commits into
OrchardLinux:mainfrom
brkosebay:j71t/kernel-safety-tests

Conversation

@brkosebay

Copy link
Copy Markdown
Contributor

Part 23 of 23 of the iPad 5 (J71t) series in this repository; merge after #127.

Problem

The cold-reset and recovery paths only run when something has already gone wrong on the device, so they are hard to exercise on hardware.

Change

Add host tests that run the real kernel entry points of the applied tree with MMIO and OS mocks: the ANS/PCIe/NVMe cold-reset paths, the ANS stop and S8003 port handoff on restart, the GPU handover failures, and the USB hold and manual-host recovery. Run them against an applied kernel tree with APPLIED_KERNEL_SOURCE=<tree> python3 -m unittest discover -s tools/testing.

Testing

  • Hardware: not exercised on the device.
  • 2026-11-12: no further hardware test of this change beyond the above. The top of this series applies to the same kernel tree as the release build tested on J71t that day (20+ warm restarts without a freeze).
  • Host: the Linux test suite passes at this branch's head, and the kernel and m1n1 patch series apply on apple-16k and apple-4k. At the top of the series the applied kernel tree is identical to the one in the built release image.
  • Other boards that share these drivers or compatibles have not been tested.

Dependencies

Depends on #127.

This branch is stacked on the PRs it depends on and carries their commits; the change of this PR is its last commit, "tools/testing: Host tests for the kernel failure paths".

🤖 Generated with Claude Code

Add the A9 (S8003) G5P render driver used by the merged PowerVR/G5P Mesa
winsys as kernel patches, enable it on J71t in the device tree, set
CONFIG_DRM_APPLE_G5P=y for apple-16k, and add an install-time recipe that
extracts the original META LDR (apple/agx-g5p-c0.ldr) from the device's own
iOS kernelcache into /lib/firmware. No firmware is committed.

The driver provides TYPED_V3 rendering, GEM buffers, VM/arena mappings,
PRIME import/export, syncobj, bounded retirement and recovery. It does not
expose the old compute or fixed-fixture ABI.

Hardware: the exact kernel this derives from (brkosebay/linux a9/gpu-pr1
407b7bf2cbbe) booted on a J71t with the merged Mesa, rendered labwc on the
GPU with KMS scanout at 60 Hz, and passed the G5P device rungs with zero GPU
events. This Orchard adaptation itself has not yet been built by orchard or
run on hardware.

Assisted-by: LLM
kernel/configs/apple-16k/config:
CONFIG_DRM_APPLE_G5P=y
These options provide g5p and its Kconfig dependencies on both page sizes.

Assisted-by: LLM
kernel/configs/apple-16k/config:
CONFIG_APPLE_AOP_GYRO=m
CONFIG_APPLE_AOP_PRESSURE=m
These options provide aop and its Kconfig dependencies on both page sizes.

Assisted-by: LLM
kernel/configs/apple-16k/config:
# CONFIG_NVME_APPLE_ANS_EARLY_USB is not set
These options provide warm-storage and its Kconfig dependencies on both page sizes.

kernel/configs/apple-4k/config:
# CONFIG_NVME_APPLE_ANS_EARLY_USB is not set
These options provide warm-storage and its Kconfig dependencies on both page sizes.

Assisted-by: LLM
kernel/configs/apple-16k/config:
# CONFIG_NVME_APPLE_ANS_EARLY_USB is not set
These options provide warm-storage and its Kconfig dependencies on both page sizes.

kernel/configs/apple-4k/config:
# CONFIG_NVME_APPLE_ANS_EARLY_USB is not set
These options provide warm-storage and its Kconfig dependencies on both page sizes.

Assisted-by: LLM
kernel/configs/apple-16k/config:
CONFIG_APPLE_AOP_GYRO=m
CONFIG_APPLE_AOP_PRESSURE=m
These options provide aop and its Kconfig dependencies on both page sizes.

Assisted-by: LLM
kernel/configs/apple-16k/config:
# CONFIG_NVME_APPLE_ANS_EARLY_USB is not set
These options provide warm-storage and its Kconfig dependencies on both page sizes.

kernel/configs/apple-4k/config:
# CONFIG_NVME_APPLE_ANS_EARLY_USB is not set
These options provide warm-storage and its Kconfig dependencies on both page sizes.

Assisted-by: LLM
Bring this branch onto current main. The merge keeps the other side's
tree; the next commit carries the reworked version of this branch's
change.
A7-A9 have no EL2 and boot without any PSCI firmware, so Linux on an
iPad 5 could not bring secondary CPUs down and up again, and a kexec had
nowhere to park them. CPU hotplug, kexec and a clean poweroff all need a
PSCI implementation that stays resident after m1n1 hands over.

Carry m1n1 patches that keep a small EL3 PSCI implementation resident:
CPU_ON/CPU_OFF with a firmware rendezvous (EL1 WFI/WFE trap to EL3),
SYSTEM_OFF, and a guarded SYSTEM_RESET that boots the default entry
again. The loader marks retained hardware in the device tree after a
PSCI re-entry. The resident PSCI is on by default on these SoCs; the
shipped boot framing keeps psci_reset=0, so SYSTEM_RESET stays off
unless a loader enables it. Adds boot/m1n1/psci-loader and its tests.

Assisted-by: LLM
Bring this branch onto current main and the series it depends on. The
merge keeps the other side's tree; the next commit carries the reworked
version of this branch's change.
The A9 AOP hosts the accelerometer, gyroscope, pressure and
ambient-light sensors. Its firmware needs boot arguments and per-device
calibration that m1n1 did not pass on S8003, and a reloaded AOP (after
kexec or sleep) lost its sensor firmware state, so the sensors did not
come back.

m1n1 passes the A9 AOP boot arguments and sensor calibration. The kernel
selects a certified AOP reload path through device-tree match data
(bindings, DT, mailbox, pmdomain, IIO and AOP driver patches), keeps the
AOP power domain and firmware state for that reload, and refuses a
suspend that would interrupt the firmware's boot. The J71t firmware
recipe gains the AOP pieces, the gyroscope and pressure drivers are
enabled, and J71t's accelerometer and gyroscope move from missing to
working, and its light sensor from untested to working.

Assisted-by: LLM
Bring this branch onto current main and the series it depends on. The
merge keeps the other side's tree; the next commit carries the reworked
version of this branch's change.
The TriStar charger subscribed to the PMU's RTC event 16 for source
identification. An acknowledgement there could take the RTC's wake
event, and the polled cover inputs had no driver enabled.

Give RTC event 16 to the RTC alone and keep the PMU and RTC event split
(DT, RTC and USB patches), so one consumer's acknowledgement cannot take
the other's wake event. Enable KEYBOARD_GPIO_POLLED for the polled cover
inputs.

Assisted-by: LLM
Bring this branch onto current main and the series it depends on. The
merge keeps the other side's tree; the next commit carries the reworked
version of this branch's change.
On J71t the NVMe controller (ANS) sits behind the H9P PCIe port, and
resetting that port loses the controller until a cold boot. A kexec
therefore has to hand over a stopped but powered ANS, with its DART
translations retained, or the next kernel loses its root filesystem. The
first version of this handoff panicked when a step failed, which with
panic=0 is an indefinite hang.

Keep the certified ANS kexec handoff: DT and bindings describe it, the
DART, H9P and NVMe drivers stop queues and translations in order and the
next kernel picks them up, and core/printk export what the modules need.
Every failure on the outgoing side now ends in a native cold reset
through apple_cold_reset() instead of a panic, with explicit returns
after it. The port is handed over powered, with PERST# asserted and the
link stopped, and the incoming kernel brings it up with the cold-boot
sequence (port power cycle, NVMe mode, port enable, then link training).
Retraining the powered port by PERST# alone, as iOS never does, left the
root port in Detect.Quiet and cost a timeout on every warm boot.

Assisted-by: LLM
Bring this branch onto current main and the series it depends on. The
merge keeps the other side's tree; the next commit carries the reworked
version of this branch's change.
After a kexec the next kernel found the S8000 GPIO controller with
interrupt modes armed by the previous kernel, the PIO SPI controller
with queued transfers and an asserted chip select, and the touch
controller still running. An orphaned GPIO interrupt fires forever and
touch did not come back.

Retire inherited GPIO interrupt modes before the chained parent
interrupts are enabled, drain and idle the S8000 PIO SPI controller on
shutdown and at probe, and stop the Apple Z2 touch controller before its
SPI parent shuts down.

Assisted-by: LLM
Bring this branch onto current main and the series it depends on. The
merge keeps the other side's tree; the next commit carries the reworked
version of this branch's change.
A Mac grants an iPad more than the standard USB current, but the charger
took the gadget's reported budget unchecked and the composite gadget
could draw the extra current while the bus was suspended.

Cap and log the USB host budget in the TriStar charger, draw no extra
Apple current while suspended, and let the initramfs and orchard USB
helpers opt J71t into the Mac current profile. Documented in
docs/bring-up/charging/j71t-mac-current.md, with tests.

Assisted-by: LLM
Bring this branch onto current main and the series it depends on. The
merge keeps the other side's tree; the next commit carries the reworked
version of this branch's change.
The USB serial number changed between installed boots and RAM sessions,
so the host saw a new device every boot and could not keep its SSH
known_hosts entry or serial console name.

Share one BOARD/hashed-ID serial format between installed boots and RAM
sessions (usb-identity), and get the per-boot SSH identity through the
existing ACM console. Documented in docs/device-protocol.md, with tests.

Assisted-by: LLM
Bring this branch onto current main and the series it depends on. The
merge keeps the other side's tree; the next commit carries the reworked
version of this branch's change.
A warm restart through kexec needs the next kernel to see the pristine
device tree from boot, the pstore region, the display state and the
power domains that must not be cut while the next kernel starts.

DT, OF, DRM, pmdomain and pstore patches retain the board-selected warm
handoff state, so probe and shutdown keep what the next kernel needs:
the pristine boot device tree, the pstore region, the display state and
the Apple pmgr warm hold. The Orchard service file learns the retained
state.

Assisted-by: LLM
On a tethered J71t boot a restart must be a native cold reset, but the
watchdog's restart handler ran below PSCI's, and its restart guard
dereferenced a parent that probe never set.

Describe J71t's tethered boot in DT. On such a board the watchdog
restart handler registers above PSCI and performs the native cold reset;
in the warm and soft modes it leaves the watchdog alone and returns
success, so PSCI's restart runs. Probe sets the parent before
registering, and the restart callback tolerates a missing one. Other
devices keep the current priority.

Assisted-by: LLM
Suspend on J71t lost the NVMe controller (its port must keep power and
refclk), the DART translations, the USB gadget link and Wi-Fi after
resume, and drivers could only refuse sleep after the filesystems had
been synced.

Add a driver admission step and a precheck notifier before sync to the
PM core, with a suspend_ready report. Pair suspend to idle with retained
ANS power in the H9P driver, stop and rebuild the ANS DART and NVMe
controller around sleep, keep the dwc2 gadget connected, stop the VXD
firmware, and keep Wi-Fi down when its link is lost. The PMIC GPIO
driver refuses sleep when it cannot program the wake mask, the charger
uses J71t's plug event as a wake source, and the J71t DT enables paired
sleep, the s2idle gadget and charger wake. On restart and poweroff, NVMe
stops the Apple ANS as it does before a kexec, and the H9P driver hands
the S8003 ports off in the same powered state; these two patches sit
here because they build on this topic's H9P and NVMe sleep code.

Assisted-by: LLM
Opening the Smart Cover should wake the iPad, but the PMIC GPIO driver
had no notion of wake-only events, and arming the cover events while the
cover was absent aborted every suspend.

Describe wake-only events in the apple,d2365-pmic-gpio binding, let the
driver wake from sleep on the listed GPIO events, and list the Smart
Cover events in the J71t DT.

Assisted-by: LLM
Bring this branch onto current main and the series it depends on. The
merge keeps the other side's tree; the next commit carries the reworked
version of this branch's change.
The A9 GPU (PowerVR-derived G5P) had no kernel driver, so the desktop
rendered in software.

Add the apple-g5p DRM driver: bindings and J71t DT, the render/compute
UAPI, the firmware transport and recovery core, buffer VMs with cache
visibility, typed render and compute submission, and the GEM exporter's
CPU cache contract. The firmware recipe extracts the G5P firmware from
the device's own IPSW (src/orchard/apple/agx_g5p.py), and DRM_APPLE_G5P
is enabled for apple-16k.

Assisted-by: LLM
Each render job paid a fixed host cost (serial submission, polling after
events, uninterruptible VM binds), which capped frame rates well below
the GPU's capacity.

Account per-job GPU and host time, reduce serial job overhead, make VM
binds killable and tune post-event polling, record joint latency,
pipeline two render jobs deep, honour fragment-only waits, start the
next job early during the 3D wait, gate TA PM records on dirty state,
and default to the measured pipeline configuration.

Assisted-by: LLM
The GPU blocked every system sleep, and an idle GPU kept its power on
because the firmware was never allowed to park itself.

Let the firmware self-park after bring-up and ring a parked firmware
without the GPC settle wait. Implement a bounded retained system sleep:
register the reservation as a suspend admission, hold clients across it,
answer the readiness query, and refuse sleep with a named reason while
work, a kexec power hold or pmgr's gfx hold is active. pmdomain ends the
warm boot hold at sync_state, the gfx gate is cycled with auto-PM like
pmgr-pwrstate, and the other admission users name their refusals too.

Assisted-by: LLM
The VXD decoder only exposed H.264, so HEVC video was decoded on the
CPU.

Prepare the driver for a second coded format, add the HEVC
command-stream builder and the stateless HEVC decoder, enable it on the
S8000 and T8010 variants, and require a proven firmware stop before
kexec. Adds the video_decode_hevc feature (working on J71t, untested on
the other boards that have the driver, missing elsewhere).

Assisted-by: LLM
S8000/S8003 left the SoC and memory performance state at the boot
default, which bounds GPU throughput by memory bandwidth.

Describe the SoC performance state in the binding and DT, run the SoC
and memory at the high state while active, and restore the low state for
sleep.

Assisted-by: LLM
There was no way to measure memory bandwidth per port on A9, which is
needed to tell GPU work from memory stalls.

Add a binding, DT node and perf driver for the A9 AMC port monitor,
exposing CPU and GFX read/write byte counters, and enable it for
apple-16k.

Assisted-by: LLM
Bring this branch onto current main and the series it depends on. The
merge keeps the other side's tree; the next commit carries the reworked
version of this branch's change.
The device descriptors had no way to say whether a board supports a warm
restart, and J71t's feature states did not follow what has been seen on
the hardware.

Add the warm_restart feature (working on J71t, untested elsewhere),
update J71t's states, document the config READMEs, and resolve VXD
temporal direct references by timestamp so the high-profile H.264
streams decode correctly. The compatibility matrix is regenerated.

Assisted-by: LLM
The codec's playback mixer source selectors kept whatever value they
had, which left a silent or duplicated channel after a codec reset or
firmware use.

Assign mixer buses 0 and 1 to ASP RX slots 0 and 1 before enabling
playback, as iOS does.

Assisted-by: LLM
A warm restart could jump into the next kernel while the GPU still had
DMA in flight, or hang in an unbounded wait for the firmware to stop.
Two races could also lose the GPU: with two jobs in flight, the previous
job's 3D completion could make a new job's 3D phase look retired, so its
doorbell was never rung and the job was lost after a silent second; and
when the firmware's ALIVE overtook the GPU cores' power-on request, the
busy request was counted as a failed power-on and the boot fell back to
software rendering.

Bound and serialise the GPU handover, accept an already completed
handover, keep halt and poweroff working after a quiesce failure, and
flush the consoles before an emergency cold reset. Ring a render phase's
doorbell unless the phase's own stamp shows it retired, as Apple's TA
and 3D submit paths ring after every ring entry. When ALIVE overtakes a
power-on request that is still in flight, let the first-ALIVE handler
wait for the request's busy bit to clear within the power-on bound, as
Apple's power-state code does, before any GPU register write.

Assisted-by: LLM
The Lightning port only worked as a USB device, so USB sticks, keyboards
and other peripherals behind Apple's camera adapter did not work.

Add the S8000 USB role switch (binding, driver, DT with EHCI/OHCI),
switch the AUSB PHY between host and device, add the Lightning ID-bus
master and host mode for the camera adapter to the TriStar charger, and
bound a manual host selection to 60 seconds. Enables USB storage, UAS
and exFAT/UTF-8 as modules, adds the usb_host feature (working on J71t,
missing elsewhere) and its catalog rows. For devices behind the adapter,
the Apple configs also build USB Ethernet (RTL8152/8153, AX88179, ASIX
AX88772, CDC Ethernet/NCM/EEM, RNDIS) and USB audio as modules. The iPad
5 DT patch records the J71t validation.

Assisted-by: LLM
A transitional USB data hold stayed in place after it was no longer
needed, and a long unplug followed by a reconnect kept the old hold
deadline.

Release later USB holds, and restart the hold when the cable reconnects.
Run the charger poll on an unbound workqueue: it sleeps on I2C and the
HDQ gauge and keeps no per-CPU state, and the queue's missing
WQ_UNBOUND/WQ_PERCPU flag made alloc_workqueue() warn at every boot.
J71t's USB serial and USB networking move to working.

Assisted-by: LLM
The cold-reset and recovery paths only run when something has already
gone wrong on the device, so they are hard to exercise on hardware.

Add host tests that run the real kernel entry points of the applied tree
with MMIO and OS mocks: the ANS/PCIe/NVMe cold-reset paths, the ANS stop
and S8003 port handoff on restart, the GPU handover failures, and the
USB hold and manual-host recovery. Run them against an applied kernel
tree with APPLIED_KERNEL_SOURCE=<tree> python3 -m unittest discover -s
tools/testing.

Assisted-by: LLM
@henrikklee

Copy link
Copy Markdown
Contributor

Thanks for this. Closing as superseded: most of what these host tests exercise (the kexec handover paths and the earlier USB hold timer) didn't land in that form, and the tests run outside CI against an extra applied tree. When warm restart is rebuilt, we'd like its failure-path checks as KUnit tests inside the patches. The code is kept as tag contrib/brkosebay-j71t-kernel-safety-tests.

@henrikklee henrikklee closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants