Skip to content

drm/asahi: g17: M5: report the performance state, its frequency and busy time - #1611

Open
erikwb wants to merge 3 commits into
integration/aurorafrom
j716c/gpu-m5-stats
Open

erikwb wants to merge 3 commits into
integration/aurorafrom
j716c/gpu-m5-stats

Conversation

@erikwb

@erikwb erikwb commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

The problem

agx_stats on the M5 reported utilisation only: pstate, the frequencies
and busy_ns read "unsupported", and a client's fdinfo had its engines' busy
time but not the clock. There was no way to tell whether the GPU ran at full
clock during a workload, or how long it had been busy.

The change

drm/asahi: g17: M5: report the performance state, its frequency and busy time
decodes two more kinds of record of the M5 firmware's statistics ring:

  • The performance state reports (tag 0x1b, time stamped by the interval
    record 0x19 before it) give pstate, and the board's performance state
    table gives achieved_frequency_hz for it. A report whose two state words
    differ, or that names a state outside the table, is not used. While the
    GPU is gated the state is stale, as before.
  • The utilisation records (tag 0x13, every 8 ms) carry their own timestamps,
    and busy_ns integrates the device utilisation over the window each record
    closes. A record after a gap of more than two periods only sets a new
    baseline, so an idle gap is never credited.
  • requested_frequency_hz, power_mw and temperature_raw stay
    unsupported. Other firmwares are unchanged.
  • Documentation/ABI/testing/sysfs-driver-asahi-agx-stats and
    Documentation/gpu/asahi-usage-stats.rst describe the M5 values.

rust: drm: file: print the engines' frequency in fdinfo lets a Rust driver
print drm-maxfreq-<engine> (Documentation/gpu/drm-usage-stats.rst) and
drm-curfreq-<engine>, the key drivers print for the frequency an engine
runs at, beside drm-engine-<engine>. Every engine gets the same pair,
which suits a device whose engines share one clock. The default prints
nothing.

drm/asahi: g17: M5: report the GPU frequency in fdinfo prints them on the
M5 for the vertex, fragment and compute engines, which one clock drives:
the current frequency while the firmware reports a state, and the highest
frequency in the board's table. An fdinfo read on the M5 now takes the device
mutex once to drain the statistics rings, as a read of agx_stats does.

Documentation/gpu/asahi-usage-stats.rst had drm-maxfreq-* left out on
purpose, because engine times must not be scaled by a frequency. This series
keeps drm-cycles-* absent and documents the two keys as the device's clock
at the time of the read, not as a weight for the engine times.

Testing

Hardware: M5 Max MacBook Pro (J716c).

  • Build: integration/aurora c868213 plus this branch, with the J716c
    configuration, at each commit. No warning in the files the branch touches.

  • KUnit, arm64 with 16K pages in QEMU:
    ./tools/testing/kunit/kunit.py run --arch=arm64 --kunitconfig=drivers/soc/apple/sep.kunitconfig:
    Ran 601 tests: passed: 599, skipped: 2, with three new tests in
    asahi_g17_telemetry (9 of 9).

  • Host tests, as CI runs them: python3 -I tools/asahi/run-host-tests.py:
    297/302 passed. The five that fail (apple-dcp, g15-pb-growth-tests
    and g15-stats-drain-tests, each of the last two also with --mutations)
    fail identically on integration/aurora c868213 on the build host,
    which lacks the Rust 1.93.1 toolchain the g15 tests ask for.
    Among them g17-telemetry-integration-tests with each of its eleven
    mutations, g17-telemetry-tests (9 unit tests, three new) and
    legacy-frequency-tests (2, one new).

  • scripts/checkpatch.pl: 0 errors, 0 warnings on the three commits.

  • On the J716c, with a Vulkan test program whose compute jobs are a
    single-thread loop of dependent integer operations (54.0 M iterations/s at
    1620 MHz), reading agx_stats while it runs:

    load result
    one 1.4 s job that fills the GPU, agx_stats read every 100 ms 12 of 12 reads: pstate 13, achieved_frequency_hz 1620000000, util_device_percent 100
    60 jobs of 20000 iterations, 30 ms apart, agx_stats read every 1 ms 48 jobs ran in about 1792 us (335 MHz), 12 in under 600 us. Of the reads that found a current state, 48 gave pstate 1, achieved_frequency_hz 338000000 and 11 gave pstate 13, achieved_frequency_hz 1620000000. The other reads found the GPU gated (pstate_status stale)
    three 300 ms jobs at full clock, 300 ms apart (two runs) busy_ns advanced by 871 ms and 863 ms for 905 ms and 919 ms of work, over 1.83 s and 1.84 s of wall time
  • On the J716c, the fdinfo of the test program during a 1.4 s job that fills
    the GPU has, for each of vertex, fragment and compute,
    drm-curfreq-<engine>: 1620000000 Hz and
    drm-maxfreq-<engine>: 1620000000 Hz. An idle client's fdinfo, with the
    GPU gated, has the drm-maxfreq lines only.

  • The J716c runs above were made with this series on the
    integration/aurora of 2026-11-09 (e7f996f plus the then-open J716c
    pull requests), kernel 7.3.0, Mesa aurora-wip with the open J716c pull
    requests.

  • On integration/aurora ea10c4d with this series (kernel
    7.3.0-ewb107+ on the J716c), the same program gave: 12 of 12 reads at
    full clock with pstate 13, achieved_frequency_hz 1620000000 and
    util_device_percent 100; 59 of the 60 short jobs at about 1792 us and
    one under 600 us, the reads that found a current state giving pstate 1,
    achieved_frequency_hz 338000000 (57) or pstate 13,
    achieved_frequency_hz 1620000000 (2); busy_ns advancing by 848 ms for
    918 ms of work over 1.85 s of wall time; and the fdinfo lines above.

  • On integration/aurora c868213 with this series, drm/asahi: GEM_RESIDENCY, keep an object's memory resident #1610 and drm/asahi: g17: per-queue watchdog allowance for compute commands #1752
    (kernel 7.3.0-ewb122+ on the J716c): 12 of 12 reads at full clock with
    pstate 13 and achieved_frequency_hz 1620000000 in each of three runs
    (util_device_percent 100 in the two runs after the desktop had
    started, 0 in the first run after boot); all 60 short jobs at
    about 1792 us, the reads that found a current state giving pstate 1,
    achieved_frequency_hz 338000000 (49) or pstate 13,
    achieved_frequency_hz 1620000000 (1); busy_ns advancing by 896 ms for
    932 ms of work over 1.85 s of wall time; the fdinfo lines above, and
    drm-maxfreq-<engine>: 1620000000 Hz only for the idle compositor. The
    kernel log had no Oops, BUG, WARNING or GPU recovery line.

Depends on

Nothing.

Not covered

  • M5 machines other than the J716c: the decode is the same for every board
    that runs the M5 command set, but only the J716c was measured.
  • Performance states other than 1 and 13: the test loads only produced the
    lowest and the highest.
  • requested_frequency_hz, power_mw and temperature_raw are not
    implemented.
  • No fdinfo consumer (gputop, nvtop) was run against the new keys; the
    machine has neither installed.

🤖 Generated with Claude Code

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at f7604d7, and run on an M5 Pro (J714s, T6050, G17S, firmware 26A428). Test kernel: integration/aurora 69a9fbb + d76f650 + 86708ce + #1608..#1611. This series' files are byte-identical to the PR head.

Verdict: approve.

Other GPUs (checked by reading):

  • Telemetry::m5() is chosen only for variant.m5_commands, and both new record arms are if self.m5. On the T8140 (G17P), the default telemetry ignores tags 0x19/0x1b, busy_status stays unsupported, and reports_pstate() still returns false.
  • The fdinfo keys need the board table (cfg.m5). Without it, frequency() returns None before it reads anything, so a non-M5 fdinfo read takes no device mutex.
  • The larger Readout (80 bytes) is filled only by read_g17; the legacy GPUs format through read_frequency. The C side's asserts match.
  • The J714 board table has the same 14 entries as J716c's (state 1 = 338 MHz, state 13 = 1620 MHz), so the decode applies unchanged.

On the M5 Pro (J714s):
agx_stats sampled every 100 ms, with fdinfo read during each phase. Baseline (integration/aurora 447b752 without the series, idle plus the same compute.heavy load): pstate, achieved_frequency_hz and busy_ns read unsupported throughout.

phase (gpubench, ~6.4 s) samples pstate / frequency util_device busy_ns advance
idle 32 stale 0 0
compute.heavy alu, depth 8 64 13 / 1620 MHz in 63, 1 stale at the start mean 98% +6345 ms in 6.38 s (99%)
idle 33 stale 0 0
compute.light, depth 1 (~160 us per submit) 64 13 / 1620 MHz in all 64 utilisation stale 0
idle 29 stale 0 0
render.heavy fill 1080p, depth 8 64 13 / 1620 MHz in 63, 1 stale at the start mean 98% +6360 ms in 6.38 s (100%)
idle 34 stale 0 0
  • fdinfo: under load every client has drm-curfreq-{vertex,fragment,compute}: 1620000000 Hz beside drm-maxfreq-*: 1620000000 Hz. With the GPU gated it has drm-maxfreq-* only. The drm-engine-* keys are still there beside them.
  • Other states: no state other than 13 appeared in these loads, so the board-table lookup for lower states was not exercised here.
  • No regression on the same boot: multi-queue timeline 214/214, synchronization2 600/421 NS, gpubench 41/41, 0 recoveries, 0 Oops/WARN.

One observation, not a regression: during compute.light at depth 1, the firmware delivered no tag-0x13 records for the whole 6.3 s. utilisation_status was stale and utilisation_samples 0, while the client's drm-engine-compute rose by ~45 ms. So busy_ns did not advance during that sparse load, although pstate 13 was reported. This is the existing utilisation path, and busy_ns inherits it. Worth one sentence in the ABI doc: busy_ns counts only windows the firmware reports, and sparse sub-millisecond work may go uncounted.

Non-blocking:

  1. Hard-coded top state. M5_MAX_PSTATE = 13 restates the table length. Deriving it from perf_states.len() - 1 would keep the two from drifting.
  2. Device mutex per fdinfo read. On the M5, each fdinfo read now takes the device mutex, which is the submission path's hot lock. That's fine for a top-like tool at 1-2 Hz. A monitor that reads every client's fdinfo in a tight loop would add contention, and the docs already say so.

Build shortcut, for the record: the M5 test kernel was built incrementally with include/linux/dma-mapping.h and include/drm/drm_gem_shmem_helper.h at their old mtimes, so files that only include them weren't recompiled. The hunks are a file-scope #define plus comment, and a same-type, same-position field rename; this head has no green CI compile to stand in for a clean build (below).

CI: kunit, the only full kernel build, is red on this head (https://github.com/aurora-silicon/linux/actions/runs/43125705476/job/130877105315). It failed compiling drivers/pmdomain/apple/pmp-report.c (get_cpu_idle_time undeclared), a file this series doesn't touch: integration/aurora's state when the check ran, fixed since by #1594 (8333109). So there is no green CI compile of this PR yet. A re-run reuses the original merge commit, so it needs a fresh run (a new push, or a re-trigger after the base update). host-tests, source and rust-build are green (rust-build compiles only apple-sep.o).

@RyanTheTide

Copy link
Copy Markdown
Contributor

@erikwb kunit fails on pmp-report.c:1704 (get_cpu_idle_time), which comes from the base this branch sits on, not from your change. Please rebase onto current integration/aurora so CI is green. One small thing: frequency() takes the last row of perf_states as the maximum; taking the max over the table, or a comment that the table is sorted, would make that safe for other boards.

…usy time

agx_stats on the M5 reported utilisation only: pstate, the frequencies
and busy_ns read "unsupported", so there was no way to tell whether the
GPU runs at full clock during a workload.

The M5 firmware reports its performance state in its statistics ring:
tag 0x1b names the state now in effect (words 1 and 3, an index into
the device tree's operating points, 1 = 338 MHz to 13 = 1620 MHz), and
tag 0x19 the interval that just ended (start in words 3-4, end in words
1-2, state in word 8). The decoder now takes the state from 0x1b, time
stamped by the end of the preceding 0x19, and the reader turns it into
achieved_frequency_hz through the board's performance state table. A
report whose two state words differ, or that names state 0 or a state
beyond the table, is not used; a gate makes the state stale, as before.

Its utilisation records (tag 0x13, every 8 ms) carry their own timer
timestamps. busy_ns now integrates the device utilisation over the
window each record closes. A record after a gap of more than two
periods (the GPU was gated or records were lost) or with an earlier
timestamp only sets a new baseline, as the legacy integrator does after
a power change, so an idle gap is never credited.

requested_frequency_hz, power_mw and temperature_raw stay unsupported.
Documentation/ABI/testing/sysfs-driver-asahi-agx-stats and
Documentation/gpu/asahi-usage-stats.rst describe the M5 values.

Checked on the J716c (M5 Max MacBook Pro) with a single-thread loop of
dependent integer operations, whose rate is proportional to the clock
(54.0 M iterations/s at 1620 MHz):
- one 1.4 s job that fills the GPU: every agx_stats read during it
  (12, 100 ms apart) gave pstate 13, achieved_frequency_hz 1620000000
  and 100 % device utilisation.
- 60 jobs of 20000 iterations, 30 ms apart: 48 ran in about 1792 us
  (335 MHz against the 338 MHz of state 1) and 12 in under 600 us. Of
  the agx_stats reads during the run (1 ms apart) that found a current
  state, 48 gave pstate 1 with achieved_frequency_hz 338000000 and 11
  gave 13 with 1620000000; the others found the GPU gated.
- three jobs of 300 ms at full clock, 300 ms apart, twice: busy_ns
  advanced by 871 and 863 ms for 905 and 919 ms of work, over 1.8 s
  (one 8 ms window is lost at each power-up).

Tests: the telemetry unit tests (g17-telemetry-tests.rs on the host, 9
passed, and KUnit asahi_g17_telemetry) cover the M5 decode from
firmware records, the busy windows, gates, gaps and other firmwares;
g17-telemetry-integration-tests.py passes and still fails each of its
mutations.

Developed with Claude Code.
Signed-off-by: Erik Bourget <erik@ebourget.net>
Documentation/gpu/drm-usage-stats.rst defines drm-maxfreq-<engine>, the
highest frequency of an engine, and drivers that know it also print the
frequency the engine runs at as drm-curfreq-<engine>. A Rust driver
could only print the busy time.

DriverFile::engine_frequency returns the current frequency, if the
driver knows it, and the highest, in Hz; show_fdinfo prints them for
every engine engine_stats emits. All engines get the same pair, which
suits a device whose engines share one clock. The default, None, prints
nothing, so drivers that do not implement it are unchanged.

Built for arm64; the next commit uses it.

Developed with Claude Code.
Signed-off-by: Erik Bourget <erik@ebourget.net>
Tools that read a client's fdinfo (gputop, nvtop) see its engine busy
time but not the clock the GPU runs at, which agx_stats now reports on
the M5.

Every engine of a client's fdinfo gets drm-curfreq-<engine>, the
achieved frequency of the performance state the firmware reports, while
it reports one (it does not while the GPU is gated), and
drm-maxfreq-<engine>, the highest frequency in the board's table (1620
MHz on the J716c). One clock drives the vertex, fragment and compute
engines. Each read drains the statistics rings under the device mutex,
as an agx_stats read does. GPUs without a board performance table (all
but the M5) print neither key.

Documentation/gpu/asahi-usage-stats.rst said that drm-maxfreq-* was
left out on purpose, because the engine times must not be scaled by a
frequency. That still holds and drm-cycles-* stays absent: the document
now says that the two keys show the device's clock at the time of the
read, and that an fdinfo read on the M5 takes the device mutex for
them.

On the J716c (M5 Max MacBook Pro), the fdinfo of a client running a
1.4 s job that fills the GPU reads, for each of vertex, fragment and
compute, drm-curfreq-<engine>: 1620000000 Hz and drm-maxfreq-<engine>:
1620000000 Hz; that of an idle client with the GPU gated has the
drm-maxfreq lines only.

Developed with Claude Code.
Signed-off-by: Erik Bourget <erik@ebourget.net>
@erikwb
erikwb force-pushed the j716c/gpu-m5-stats branch from f7604d7 to 17eca40 Compare November 13, 2026 03:41
@erikwb

erikwb commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. Rebased onto current integration/aurora (c868213), which has the idle-accounting change behind the kunit failure; head is now 17eca40.

frequency() now takes the highest frequency over the board's table instead of the last row, and the doc comment, asahi-usage-stats.rst and the commit message say "highest frequency in the board's table". The other two commits are unchanged.

Rechecked on the new base: each commit builds with the J716c config, KUnit 599 passed / 2 skipped, and a J716c boot with this series (plus #1610 and #1752) gives the same agx_stats and fdinfo results as before; the testing section has the numbers. Separately, g17/mod.rs still reads perf_states[13] for GET_PARAMS max_frequency_khz; that is outside this series, but I can send the same change there if you want it.

@RyanTheTide RyanTheTide left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebase and the max-over-table change look good, and CI is green. Before merge it needs a check that agx_stats and fdinfo are unchanged on a non-M5 G17 before merging. A separate patch for the perf_states[13] read in GET_PARAMS would be welcome.

@DjDeveloperr DjDeveloperr left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current-head readiness review on @DjDeveloperr’s behalf.

The rebase/max-over-table change is present. frequency() derives the maximum from the board table and returns before read_firmware()/the device lock when no M5 table exists. The default DriverFile hook emits no frequency keys, and M5 decoding remains gated by the variant flag. I found no new source blocker in those changes.

Verified actual green Host43360666695/job131673136521 and SEP43360666812 build/test steps. The hosted QEMU job131673138336 reports294 tests; it is not the author’s separately reported599-pass/2-skip run with the G17 telemetry suite. Keep those scopes distinct. The existing M5 Pro and J716c hardware reports retain their exact composed-kernel limits.

This is still held by Ryan’s5886924588 pre-merge requirement: agx_stats and fdinfo unchanged on a non-M5 G17, with current source/composition identity. Green checks or the M5 rows do not supply that result. The separate GET_PARAMS perf_states[13] cleanup remains nonblocking follow-up. No merge, new hardware run or global GPU/reclaim action performed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants