Skip to content

T8140 sleep follow-ups: RTC wake on rtc0, SEP restart after a refusal, commit audit, USB recovery - #1333

Closed
DjDeveloperr wants to merge 8 commits into
fix/t8140-sleep-reviewfrom
fix/t8140-sleep-followups
Closed

DjDeveloperr wants to merge 8 commits into
fix/t8140-sleep-reviewfrom
fix/t8140-sleep-followups

Conversation

@DjDeveloperr

Copy link
Copy Markdown
Contributor

Follow-ups to #1313 (SHIP_PLAN §4 items #21, #14 and #22), four commits, one per item. The owner
split #22 in two: (a) replaces "module PMP removal refuses sleep" with an audit of the PMP/GFX/GFX1
commit steps, and (b) is USB sleep_dead. Built offline with the asahi profile (Rust on), with host
tests and KUnit. Nothing here has run on a Neo yet: the checks for the hardware worker are below.

#21 RTC: rtc0 is the PMU RTC (rtc: apple-pmu: …)

What breaks for a user. On the J700, rtc0 is the SMC RTC, which has no alarm. The PMU RTC that
can wake the machine is rtc1. So rtcwake (it defaults to rtc0), and anything else that writes
rtc0/wakealarm, arms nothing, and the sleep never ends by itself. Alarm-clock timers (systemd
WakeSystem=, CLOCK_*_ALARM) already reach rtc1 through the alarm timer. But the PMU RTC caps
alarms at one hour, so a longer timer wakes the machine early, and rtcwake -s 7200 on rtc1 fails
with ERANGE.

Fix.

  • The J700 aliases now put rtc0 = &pmu_rtc and rtc1 = &smc_rtc. The other Apple DTs keep the
    SMC RTC as rtc0.
  • rtc0 is also the RTC the kernel sets the system time from (RTC_HCTOSYS_DEVICE), and the one
    hwclock and timedated write. So on Baku the PMU RTC can now set the clock. It writes the same
    whole-second offset cell the SMC RTC writes, from the counter rounded to the nearest second, as the
    SMC RTC does.
  • Baku alarms can reach as far as the 32-bit comparator. The hour was a suspend-to-idle policy; on
    the J700 this alarm is the wake source of macOS's own deep sleeps (0f9f95f). The other PMUs
    keep the hour.
  • asahi.config builds RTC_DRV_APPLE_PMU in (=y).
  • tools/aurora-wake find the PMU RTC by name instead of assuming rtc1.

Distros: the PMU RTC has to be built in. As rtc0, a modular build means no rtc0 until the
module loads. Until then the kernel sets no time from any RTC, so early boot runs on systemd's
guess, and the clock jumps when the module loads. Fedora picks =y up from asahi.config. ALARM
(packages/linux-aurora/config) and NixOS (kernel/config) carry full configs with =m: both need
CONFIG_RTC_DRV_APPLE_PMU=y with the kernel pin that brings this in. Alias numbering doesn't depend
on load order (both nodes have aliases), so =m still gives rtc0/rtc1 as above, only later.

Rig note. rig/row/pmtimes.sh and rig/row/sleep20.sh hard-code rtc1. With this kernel that's
the SMC RTC, which has no wakealarm, so they'd arm nothing. soak.sh, aurora-sleeptest.sh and the
functional row scripts look the RTC up by name and are fine.

On-device check.

  • R1. cat /sys/class/rtc/rtc0/name starts with apple-pmu-rtc, rtc1 with macsmc-rtc.
    cat /sys/class/rtc/rtc0/hctosys is 1, /dev/rtc points at rtc0, and dmesg has apple-pmu-rtc … setting system clock to with the right date.
  • R2. rtcwake -m mem -s 30 (no -d) wakes by itself after about 30 s.
    hwclock -r, then hwclock -w, then hwclock -r and hwclock -r -f /dev/rtc1 agree within 1 s.
    timedatectl shows RTC time next to the system time.
  • R3, more than an hour. rtcwake -m mem -s 3900 wakes at about 65 min, not at 60.
    Optional: systemd-run --on-active=70min --timer-property=WakeSystem=true /bin/true, then
    systemctl suspend: it wakes at about 70 min, and the journal shows the timer firing.

#14 SEP: an enclave that gated in a refused sleep is started again (soc: apple: sep: …)

What breaks for a user. The sleep request goes out as the sleep's last refusal, and that step
waits for the enclave to gate. If it gives up and the enclave gates later, the sleep is refused, and
the enclave stayed "lost": Touch ID and SEP-backed keys failed until the next real sleep or a reboot.

Fix: macOS's sequence. In the 25G83 AppleSEPManager, an enclave that answered its sleep request
(SEP_ASLEEP) is started again the same way whether or not the system slept. The power-state change
back from 0 runs the provider's startCPU and then the wake (wakeSEPFast), and an aborted sleep goes
through the same code (sep-protocol-evidence/report.txt §D–E). Linux now does the same. Once the
refused sleep's enclave reads gated (DEV_DISABLE) and its domain reads off (actual 0), it's started
as after a real sleep: DEV_DISABLE cleared, the DART restored, the doorbell, then both wake messages
(0x0d, 0x10). This is macOS's measured wake order (W5 §5). It's marked lost only if that start
fails, and the next real sleep tries again. SEP stays a black box. Only the PMGR power word, its
sleep/wake messages and the driver's own state machine are involved. A gated enclave's mailbox is
never read: both waits read the PMGR word.

The reviewed hardening from sep-hardening/ is in the same commit, as the review passed it:

  • A last-refusal registration that fails for any reason but an unsupported platform now refuses the
    sleep with nothing sent. Before, it fell back to the early request.
  • A failed read of the domain word is an error, not an expired wait, and never leads to a mailbox
    probe. As the review asked, the commit message says what that changes: an enclave node without
    power-domains (older SoCs) stays lost after an aborted firmware suspend instead of being probed.
  • Fixed from the review: the final-read-after-deadline cases now have their own host controls
    (gate first seen at the last read counts; one read later doesn't; an error on that read is an
    error). The changed hunks are rustfmt-clean.

Kept as is: the 5 s answer and 5 s gate waits, which run with interrupts off. macOS waits up to 15 s
for the answer, but that's in a thread. With the restart, a late gate no longer costs the enclave.

Test hook. apple_sep.sleep_refuse_after_gate (bool, 0644, default N): the next deep sleep is
refused at its last step once the enclave has gated, then the flag clears itself. That's the case
above, on demand.

On-device check.

  • S1. echo 1 > /sys/module/apple_sep/parameters/sleep_refuse_after_gate, then
    echo mem > /sys/power/state. Pass: it returns at once with refused: the enclave didn't take the sleep request: -11, then the sleep was refused after the enclave gated: starting it again and
    the enclave is back from its sleep request, and no did not come back. The parameter reads N
    again. The enclave answers: whatever SEP check the soak already runs, or a read of /dev/hwrng if
    the SEP is rng_current. No key or keybag operations are needed. Then a normal M sleeps, wakes,
    and logs the enclave is back again.
  • S2. 10 ordinary deep sleeps: each logs the enclave is back once, and never starting it again.

#22a PMP/GFX/GFX1 past the point of no return (soc: apple: t8140-sleep: …)

Audit. After the enclave's tell, the last refusal, .enter runs coprocs_stop (void), the wake
arm and match (void), the USB down (its result is ignored, with bounded HPM retries) and
cpu_suspend into t8140_sleep_commit, which ends in the __noreturn WFI. The GFX/GFX1 commit
stop (asahi_g17_commit_stop, both instances) and the PMP's (apple_pmp_commit_stop_fn) only record
a failure, and the sleep goes on. Their run-bit clear can't fail. So at #1313's head nothing after
the point of no return returns an error or cancels.

What did break for a user. The early fallbacks. If the commit couldn't take its stop, the GPU
(down() at .suspend) and the PMP (apple_pmp_sleep() at .suspend_late) stopped the firmware
right away. Any later refusal then left the GPU or the PMP stopped until the next real sleep.

Fix. When this sleep's commit stops the registered coprocessors (the T8140 sleep driver drives
it: new apple_sleep_commit_stops_coprocs()), a GPU or PMP whose stop can't be registered refuses
the sleep before anything stops. Elsewhere the early stop is unchanged.

On-device check. None of its own. The refusal needs a failed registration, which a normal boot
doesn't produce. Covered by KUnit/host builds. In R/S/U rows, dmesg shows no no GPU stop at the commit and no no stop at the sleep's commit.

#22b USB sleep_dead (phy: apple: atc: …)

What breaks for a user. If the USB host's core didn't come back to D0 after a sleep in place, or
its domains didn't come on, USB stayed off until a reboot, and every later deep sleep was refused.
Worse, suspend-to-idle still went ahead, and xhci_suspend() reads PORTSC
(xhci_disable_hub_port_wake()) before it checks HW_ACCESSIBLE. On a core that may be unpowered,
that's an SError.

Fix: macOS's sequence.

  • macOS's T8140 raise from Sleep polls D0_ON for 200 ms, then only logs timed out waiting for D0_ON interrupt and carries on (0xfffffe000aca7f20, then b 0xfffffe000aca5e78, back into the normal
    raise).
  • A controller with a hardware error is taken Off at the next sleep (maxCapabilityForDomainState
    forces 0) and started cold afterwards (W15 §1, §6).

Linux can't carry on into a core that may be unpowered, so it does that Off and cold start at once,
touching nothing in the core:

  • The core is clamped in reset (pipe handler AON_GEN). ATC0_USB, ATC0_COMMON and ATC0_USB_AON go
    off, with the DARTs saved and the fabric bridge gated as at sleep. Then they come on again in
    macOS's order, with up to 3 tries for each domain.
  • The power controller comes back reset. That's the existing "always-on side lost power" case
    (APPLE_ATCPHY_SLEEP_POWER_LOST), so xHCI starts over as it does then.
  • If that still fails at the wake, the controller tries once more when the system is back, and
    again before every later sleep.
  • Only while the core may still be unpowered is a sleep refused, suspend-to-idle too. That closes
    the SError, and the log line says so.

On-device check.

  • U1, regression. 10 deep sleeps with the fixed hub and a USB keyboard, then 3 with
    mem_sleep = s2idle. Every wake is in place (no new … USB device number for the hub's
    devices, the same lsusb -t), and there's no power cycling it.
  • U2, opportunistic. The recovery runs only when a raise fails, which doesn't happen on
    demand (no /dev/mem fault injection on the Neo). If any row logs USB controller didn't come back: power cycling it, pass is USB controller back after a power cycle: the host starts over.
    The hub and keyboard then enumerate again, and the next sleep works.

Traces

migration/neo-continuation-20261108/sleep-followups/TRACE_REQUESTS.md (in the workspace): T1 is
macOS's SEP start after a sleep aborted past its power-down, T2 is macOS's ATC0 Off and cold start,
and T3 (optional) is how far ahead macOS arms the PMU alarm. None blocks this PR. Each one says
whether the static evidence holds.

Tests run offline

  • Build: ./linux-build/build kernel --profile asahi of drivers/rtc/ drivers/soc/apple/ drivers/usb/dwc3/ drivers/phy/apple/ drivers/gpu/drm/asahi/ apple/t8140-j700.dtb, exit 0.
    CONFIG_RUST=y, and sep.o, pmp.o and asahi_core.o were rebuilt; rtc-apple-pmu.o is now
    built in. No new warnings: the 11 Rust fetch_update deprecations are in GPU files this PR
    doesn't touch. The built DTB has rtc0 = …/pmic@e/rtc and rtc1 = …/smc@30c600000/rtc.
    checkpatch --strict is clean, apart from the MAINTAINERS note for a new test file.
  • Host suites, through tools/asahi/run-host-tests.py:
    • pmu-rtc 12/12: control plus 11 mutations, 3 of them new (Baku limit, rounding, negative offset);
    • sep-suspend-errors 8/8: new, with 13 controls and 7 mutations;
    • neo-stability 10/10, atc-usb3 2/2, display26 1/1 (its sleep.h stub gained the new function),
      t6030-pmp-supplier 2/2, t6030-clock 1/1, g15-display-order 1/1;
    • aurora-sep test_pm_domains and test_host_entropy pass.
  • KUnit, QEMU arm64: T8140 sleep: fixes from review #1313's stack config plus APPLE_SEP, APPLE_PMP and DRM_ASAHI.
    835 tests: 833 passed, 2 skipped (apple_g17_rx_util_min_test needs CONFIG_UCLAMP_TASK;
    j713_accepts_both_loaded_images is in fwid.rs, untouched). Among them: apple_sep_pm,
    apple_sep_*, asahi_g17_sleep, dwc3_apple_state, xhci_plat, phy-apple-atc-*,
    apple-pmgr-pwrstate and apple-dart.
  • Not covered offline: the restart, the USB power cycle and the GPU/PMP refusal themselves. They
    need hardware state, and the rows above are their checks.

🤖 Generated with Claude Code

On the J700 rtc0 is the SMC RTC, which has no alarm, and the PMU RTC that
wakes the machine is rtc1. rtcwake, and anything else that arms /dev/rtc0
or rtc0's wakealarm by default, can't arm a wake, so a sleep it starts
never ends by itself. Alarm-clock timers, such as systemd's WakeSystem=,
reach rtc1 already, but the PMU RTC limits alarms to an hour, so the alarm
timer cuts a longer one short and the machine wakes early.

Make the PMU RTC rtc0 through the aliases. The first RTC is also the one
the kernel sets the system time from, and the one hwclock and timedated
write, so:

- set Baku's whole-second offset, the cell the SMC RTC writes, the same
  way: from the counter rounded to the nearest second;
- build the driver in, as the SMC RTC is. Built as a module, rtc0 only
  appears once the module loads, and until then the time isn't set from
  any RTC;
- let Baku alarms reach as far as the comparator. The hour is a limit for
  suspend-to-idle, where nothing else is known to wake the system; on the
  J700 this alarm is the wake source of macOS's own deep sleeps.

The other PMUs keep the hour, and their clock is still set through the
SMC RTC. The wake test scripts find the PMU RTC by name instead of
assuming rtc1.

Signed-off-by: Diljit Singh <hey@djdev.me>
…ts request

A sleep refused after the enclave took its request, for instance when the
enclave gated only after the sleep's last step stopped waiting for it,
left the enclave marked lost: Touch ID and SEP-backed keys failed until
the next real sleep or a reboot.

macOS has no such state. AppleSEPManager starts an enclave that answered
its sleep request the same way whether the system slept or not: its power
state change back from 0 runs the provider's startCPU and the wake, and an
aborted sleep goes through it too. Do the same. Once the refused sleep's
enclave reads gated and its domain off, start it as a real wake does:
DEV_DISABLE cleared, DART restored, the doorbell, then both wake messages.
It's marked lost only if that fails. The mailbox of an enclave whose
domain is off is still never touched: the gate and the power state are
read in the PMGR word.

Also, as reviewed in sep-hardening:
- a last-refusal registration that fails for any reason but an
  unsupported platform refuses the sleep while nothing was sent, instead
  of falling back to the early request;
- a failed read of the domain word is an error, not an expired wait, and
  never leads to a mailbox probe. So an enclave node without power-domains
  (some older SoCs) stays lost after an aborted firmware suspend instead
  of being probed, as nothing says it's safe to.

A test-only parameter, sleep_refuse_after_gate, refuses the next deep
sleep after the enclave gated, so that the restart can be checked on
hardware.

Signed-off-by: Diljit Singh <hey@djdev.me>
…PU or PMP

On T8140 the GPU's two firmware instances and the PMP stop only at the
sleep's commit, after the last refusal, and only a real sleep restarts
them. Nothing from there on can fail or turn back: their last stops
record a failure and the sleep goes on, and the rest of the way to the
WFI returns nothing.

But where their stop couldn't be left to the commit, both drivers fell
back to stopping the firmware at .suspend or .suspend_late. Any refusal
after that left GFX and GFX1, or the PMP, stopped until the next real
sleep.

Refuse the sleep instead, before anything stops, when the sleep's commit
stops the registered coprocessors. Elsewhere the early stop is unchanged.

Signed-off-by: Diljit Singh <hey@djdev.me>
…OS does

A USB host whose core didn't come back from a sleep in place, or whose
domains didn't come on again, stayed off until a reboot, and every later
deep sleep was refused. Suspend-to-idle still went ahead, and its xHCI
suspend reads PORTSC on a core that may be unpowered: an SError.

macOS's raise from sleep only logs a missing D0_ON and carries on, and it
takes a controller with a hardware error Off, then starts it cold. Linux
can't carry on into a core that may be unpowered, so it does the Off and
the cold start at once: the core clamped in reset, ATC0_USB, ATC0_COMMON
and ATC0_USB_AON off, then on again, with up to three tries for each
domain. The power controller comes back reset, as after a sleep that took
its power, and the host starts over as it does then.

If that fails at the wake, the controller tries once more when the system
is back, and again before every later sleep. Only while the core may
still be unpowered is a sleep refused, suspend-to-idle too, with a log
line that says so.

Signed-off-by: Diljit Singh <hey@djdev.me>
Brings in the debug port's page select resend and integration/aurora
15f88e1. The conflicts are host tests: the PMU RTC mocks keep both the
follow-ups' nvmem write error and integration's control event latch,
and the runner keeps the follow-ups' RTC mutations and integration's
new perf test.

Signed-off-by: Diljit Singh <hey@djdev.me>
Brings in #1208's aop-bootargs host test fix. No conflicts.

Signed-off-by: Diljit Singh <hey@djdev.me>
Signed-off-by: Diljit Singh <hey@djdev.me>
Brings in integration/aurora ac11649 (#1311, PCIe multi power domain
support) through #1208. No conflicts.

Signed-off-by: Diljit Singh <hey@djdev.me>
@RyanTheTide

Copy link
Copy Markdown
Contributor

The diff is fine. The description quotes two macOS code addresses in the D0_ON timeout paragraph. Please describe the timeout behaviour there without the addresses.

@DjDeveloperr

Copy link
Copy Markdown
Contributor Author

Superseded by #1445 (merged as 2b05915); its changes are in integration/aurora. The RTC wake on rtc0 is in drivers/rtc/rtc-apple-pmu.c. The SEP restart after a refusal was redesigned in #1445: a refused sleep now unwinds before the SEP sleep request goes out at the commit (sep_pm_commit_* in drivers/soc/apple/pm_shim.c).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants