Skip to content

accel/ane: h17: retry a failed resume after power cycling ANE_SYS - #1303

Draft
RyanTheTide wants to merge 1 commit into
integration/aurorafrom
ane4/h17-resume-recover
Draft

RyanTheTide wants to merge 1 commit into
integration/aurorafrom
ane4/h17-resume-recover

Conversation

@RyanTheTide

Copy link
Copy Markdown
Contributor

Summary

Since #1283, an ANE_SYS power-on that latches during system sleep (PS ane_sys: Failed to reach power state 0xf (requested 0x200f, now: 0x2b4f)) is left failed. The PMGR's PS-reset recovery cannot run in the noirq resume, where it hung the Neo. genpd carries on as if the domain were on, the H17 boot at PM_POST_SUSPEND fails, and the ANE is quarantined until the next boot ("reboot required").

That resume runs in process context after the thaw, where the PMGR may run the recovery. When the boot after sleep fails, ane_fw_resume_retry():

  1. puts the engine down;
  2. power cycles ANE_SYS with apple_pmgr_ps_cycle(), where a power-on that latches again is cleared by the PS reset (-EINVAL means genpd already turned the domain off, so the next power-on is a fresh one);
  3. cold boots once more.

The engine is quarantined only if that also fails. Nothing changes on a resume that succeeds.

Test (Neo J700, one-shot boots, NEOHANG's sleep loop)

Each boot runs 2 rounds of 6 systemd + 6 direct pm_test=platform suspends + 3 rtcwake s2idle (rtcwake armed), from a fresh boot. Silent, os_uuid pin on every entry.

kernel boots suspends hangs boots with an ANE_SYS latch recovered quarantined
control (integration 6ff31cb behaviour; test kernel ane4n, whose only change was in the H16 backend the Neo does not run) 6 180/180 0 4 (12/8/16/16 'Failed to reach' lines) 0 4 (ANE dead until reboot)
this PR (test kernel ane4n2 = 6ff31cb + 70b1c8b) 6 180/180 0 0 (no natural latch in 6 boots) - 0
  • So far this proves no regression: 0 hangs, every resume ok, MobileNet works after every loop, and the one-shot FLEET suite equals fleet44 (ane 16/16, ane_resume 15/0, CTXR 1696/0, 0 kerr, 0 AVC, 0 failed).
  • The recovery path itself has not run on hardware yet. No ane4n2 boot latched, while the control latched on 4 of 6 (NEOHANG measured 1 in 4-8 earlier).
  • This PR stays DRAFT until a boot with a natural latch shows H17 ANE: recovered after a failed resume and a working ANE afterwards. The next Neo slot runs more boots of the same loop.

Raw: traces/power/20261109-ane4-neo-latch (README; runs/summary.txt, ane-.txt per boot).

checkpatch --strict 0/0/0. KUnit: drivers/accel/ane boot 224/224, transport 224/224 (x86_64).

🤖 Generated with Claude Code

Since #1283, a latched ANE_SYS power-on during system sleep is left
failed: the PS reset recovery cannot run from the noirq resume, where it
hung the Neo. genpd treats the domain as on, the boot at PM_POST_SUSPEND
fails, and the engine stays quarantined until the next boot. That
happens on about one boot in four to eight.

The resume at PM_POST_SUSPEND runs in process context after the thaw,
where the PMGR can run the recovery. When the boot after sleep fails,
put the engine down, power cycle ANE_SYS with apple_pmgr_ps_cycle() (a
power-on that latches again is cleared there), and boot once more. The
engine is quarantined only if that fails too.

Signed-off-by: Ryan Murray <81541646+RyanTheTide@users.noreply.github.com>
Assisted-by: LLM
@RyanTheTide

Copy link
Copy Markdown
Contributor Author

Part 2 (Neo, 21:23-22:17, ane4n2 = this PR on 6ff31cb, same loop): the one-shot FLEET suite shows no hard regressions (ane 16/16, ane_resume 15/0, CTXR 1696/0, 0 AVC, 0 failed).

boot suspends ANE_SYS latch result
1-3 30/30 each 0 clean, ANE works after the loop
4, 5, 6, 7 hung after 20, 5, 13 and 21 0 sleep-entry hang
8 - - not run (stopped)

Each hang has the NEOHANG signature: the journal ends at PM: suspend entry (s2idle) or right after a resume, and the hw watchdog resets the Neo. In all four the previous resume was resume ok on the first try, with no Failed to reach line. This PR's retry path never ran (no power cycling ANE_SYS, no recovered), so these hangs are not from this change. Earlier today the same base had 0/6 (control) and 0/6 (this PR, run 1); the rate jumped after ~21:50. That points to a Neo environment change (dock / peripherals); NEOHANG has been told.

Status: no natural latch in 14 boots of this PR, so the recovery is still unproven on hardware. Kept as DRAFT. Next: an injected-failure build (ane4n3: a test param fails every Nth resume after the boot) to drive the retry path end to end, once the Neo is stable again.

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

J700 suspend A/B, 6ff31cb with and without this change (same loop: 6 systemd + 6 direct pm_test=platform suspends + 3 rtcwake s2idle per round, 2 rounds per boot, 4 interleaved boots each):

kernel boots hung boots boots with an ANE_SYS latch
6ff31cb 4 (8 incl. an earlier run) 0 (0) 2 (4)
6ff31cb + this PR 4 (18 incl. earlier runs) 1 (5) 0 (0)

The hung boot stops in the device suspend phases (the last line out is the freezer completing). The retry path added here did not run in any boot, hung or clean. Its only effect on the normal path is the binary layout of the ane module, and builds from different source paths also differ in kmalloc partition assignment (typed alloc tokens), so a control build of 6ff31cb from the same path is next before reading this as a code effect. Keeping this as a draft until then.

@RyanTheTide

Copy link
Copy Markdown
Contributor Author

J700 run on the production fleet64 kernel. Only the ane modules were rebuilt: fleet64 + this change + a test-only parameter that fails every Nth resume after the boot. The built-in kernel is unchanged.

  • The retry path now runs on hardware. With the injected failures: resume failed (-110); power cycling ANE_SYS and booting again, then recovered after a failed resume (twice), then resume ok on the following wakes. No quarantine.
  • Sleep: two of three runs with these modules ended in a reset at an rtc s2idle entry. One was the first s2idle with injection off; the other was the third s2idle of the injected round. The third run was 12/12 clean. The stock modules, same loop, same boot sequence: 12/12 clean.
  • Next: the same loop with an out-of-tree build of the unchanged fleet64 ane modules, to separate this change from the module build. Still draft.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant