Repository navigation
drm/asahi: GEM_USERPTR, buffer objects over the caller's memory - #1609
Conversation
Object::new_external() creates a GEM object whose backing is a page array that the object's driver data holds (pages pinned from a process, say) instead of shmem pages. The object is a private GEM object, with no shmem file, so that no C helper can ever allocate pages for it. Its SG table is built and DMA-mapped once, at creation, under a Devres, and unmapped when the object is freed or the device unbound. The mapping never outlives the pages. The driver data is initialized first and the pages are mapped only once it holds them: a driver initializer that fails, and with that releases the pages it was given, leaves nothing mapped, and if the mapping fails the driver data is dropped with no mapping left. When the object is freed, the mapping field is dropped before the driver data. The page array is passed as a raw pointer, since it may be memory the driver data owns, and is only read once that data is initialized. The DMA mapping does no CPU cache maintenance on map or unmap: the pages are the owner's to use through cacheable CPU mappings, and the invalidation an unmap does for a non-coherent device would discard data written through them. The device must therefore reach the pages themselves: a bounce buffer, which an unsynchronized mapping never updates, will not do. The table is mapped with DMA_ATTR_SKIP_CPU_SYNC and DMA_ATTR_NO_BOUNCE, so the DMA layer itself fails the mapping where it would put other memory in a page's place: with EIO for pages it would bounce (those the device cannot address, all pages where bouncing is forced or the device needs unencrypted memory, and pages of an untrusted device whose IOMMU granule is larger than a page), and with EOPNOTSUPP for a device that is mapped by DMA operations of its own. Nothing is inferred from the DMA addresses, so a device with a bus offset, an address encoding or an IOMMU is served like any other. The constructor is unsafe, and its safety contract has two parts. One is the lifetime of the pages. The other is that the device accesses the pages coherently with the CPU caches. The two attributes give the device the pages themselves and leave the caches alone; they make no device coherent, the mapping succeeds for one that is not, and nothing here can check it. The caller has to know it of its hardware. Such objects get their own GEM vtable: no mmap or vm_ops, vmap, pin or get_sg_table, and an export that refuses (the core would otherwise fall back to its generic export). vmap() fails, madvise() is ignored, and the reclaim and purge helpers report the backing as never releasable. Tested: KUnit rust_drm_gem_shmem in QEMU (arm64, 16K pages), 18 of 18 passed, the six new tests included: an object over three pages has a table whose DMA addresses are those pages, no shmem file or pages, and frees without touching the pages' contents; vmap, drm_gem_vmap (EOPNOTSUPP), export (EINVAL), madvise, idle release and purge are refused; no pages, a foreign device or a write-combined config fail with EINVAL; a device whose bus sees every CPU address 1 TiB up (a dma_range_map) gets DMA addresses that are not the pages' physical addresses and its object is created; with a driver whose data owns the pages, no mapping exists at the moment they are released, whether the object is freed or the driver's initializer fails after taking them; and for a device whose bus reaches all memory but one page, an ordinary mapping of that page is a bounce buffer (its DMA address lies outside the page, and the memory there holds what the page held when it was mapped and does not follow a later store to the page), an object over pages the device reaches is created, and an object over a page it reaches and that one fails with EIO, with no mapping left when the driver data releases the pages. The two tests with a bus build the device's dma_range_map from struct bus_dma_region as C defines it: the bindings include linux/dma-direct.h, the DMA layer's own header, for KUnit builds only, and the abstraction uses nothing from it. Developed with Claude Code. Signed-off-by: Erik Bourget <erik@ebourget.net>
DRM_IOCTL_ASAHI_GEM_USERPTR creates a GEM object whose backing is a page-aligned range of the caller's address space, without a copy, for Vulkan's VK_EXT_external_memory_host. GET_PARAMS reports it with DRM_ASAHI_FEATURE_USERPTR (bit 37), on G17 GPUs. The range is pinned at creation with pin_user_pages_fast(FOLL_LONGTERM, plus FOLL_WRITE unless DRM_ASAHI_GEM_USERPTR_READ_ONLY), all of it or none, and stays pinned until the object is freed; unpinning marks the pages dirty if the GPU could write them. The pages count against RLIMIT_MEMLOCK through mm->pinned_vm unless the caller has CAP_IPC_LOCK, and their size is admitted against the client and device memory limits and counted as held for the object's life (not added to the process's RSS again: they already are its memory). The core mm handles the rest: fork gives the child copies of pinned anonymous pages, munmap leaves the pinned pages to the object, writable pins of read-only mappings and of write-tracked file mappings, and pins of device or PFN mappings, fail with EFAULT, and read-only pins of private mappings unshare the pages first. A read-only shared file mapping pins its page-cache pages, which stay resident while the object lives. User-memory objects are only ever mapped by user VM binds: a read-only object takes only GPU read-only binds (EACCES otherwise), since its pages may be a file the caller cannot write. Kernel and firmware mappings (timestamp buffers, kernel aliases) refuse them, as do MMAP_OFFSET, MADVISE, the VM status mirror (no vmap) and export. They are never idle-reclaim or purge candidates. The pages are mapped for the GPU without CPU cache maintenance, and the caller and the GPU use them at the same time. That rests on an assumption about the hardware, which neither the DMA attributes nor the driver establish: that the GPU's accesses through user VM binds are coherent with the CPU caches for normal cacheable memory. The driver relies on it already: objects created with DRM_ASAHI_GEM_WRITEBACK are mapped cacheable into their process and used by the GPU through user VM binds with the same page attributes, with no cache maintenance by the driver and no UAPI call for it, and a user-memory object is bound the same way. The assumption covers normal cacheable RAM only: pinning refuses device and PFN mappings but does not check the caching attributes of every mapping it accepts. It was measured on the J716c (below); other machines with a G17 GPU were not measured. Tested: KUnit in QEMU (arm64, 16K pages), 545 tests, 543 passed, 2 skipped (unrelated configuration), including the new asahi_gem_user_memory suite against a real test mm: pinning four anonymous pages counts four in pinned_vm and drops it again, and a store through the user mapping is read back from the pinned page; unaligned, empty and overflowing ranges fail, a range running into an unmapped page fails after pinning part of it and leaves pinned_vm unchanged; a read-only mapping refuses a writable pin and takes a read-only one; a pinned page keeps its contents after munmap. On the J716c (M5 Max MacBook Pro), with a Vulkan driver that implements VK_EXT_external_memory_host on this ioctl, dEQP-VK.memory.external_memory_host.* gives 3 passed, 8 not supported and none failed. There, a Vulkan program in which the CPU stores a new pattern to one imported range and a marker to another through its own mappings, a compute job copies the first to the second and the CPU compares, found no wrong word: 20000 iterations each with ranges of 16 KiB and 64 KiB, 3000 with 1 MiB and 30 with 64 MiB. Developed with Claude Code. Signed-off-by: Erik Bourget <erik@ebourget.net>
A kselftest harness program for the render node that exercises DRM_IOCTL_ASAHI_GEM_USERPTR without submitting GPU work: - anonymous memory: creation counts the pages in VmPin and the size in the client's drm-total-system, a read-write bind succeeds, MMAP_OFFSET and MADVISE fail with EINVAL, export and a timestamp BIND_OBJECT fail, munmap while bound keeps the pin until the handle closes, and closing it gives everything back; - bad ranges: unaligned address or size, zero size, unknown flags and a write-back flag fail with EINVAL; a range with a hole fails with EFAULT and leaves VmPin as it was, as does the CPU mapping of another buffer object (a PFN mapping), writable or read-only; - a read-only shared mapping of /usr/bin/sh: a writable import fails with EFAULT, a read-only one succeeds and takes only read binds (EACCES for a write bind); - RLIMIT_MEMLOCK: the test gives up CAP_IPC_LOCK first, so whoever runs it a request above the limit must fail with ENOMEM and leave VmPin as it was; a request within the limit succeeds, and a further one fails when the pages already pinned take it past the limit. The soft limit it sets is what the process has pinned plus 16 pages, or as many as the hard limit has room for (skipped below two); - after fork the child's write lands in its own copy. The driver keeps ranges of every VM for its own mappings, where a bind fails with EINVAL, and GET_PARAMS does not list them: the test binds at the first address, in steps of 4 GiB, at which an ordinary buffer object binds. The program is listed in the selftests' TARGETS. It skips where renderD128 is not this driver's node or lacks the feature, and where the caller has neither CAP_IPC_LOCK nor a RLIMIT_MEMLOCK hard limit with room for the four pages a test pins (the soft limit is raised to it if need be). Tested on the J716c (M5 Max MacBook Pro), built there with gcc against this tree's UAPI header, as an unprivileged user and as root: pass 5, fail 0, skip 0, also with RLIMIT_MEMLOCK at 128 KiB (soft and hard) and at 32 KiB soft, 128 KiB hard. With a hard limit of 48 KiB the unprivileged run skips all five and root passes five; with 16 KiB root passes four and skips memlock_limit. Developed with Claude Code. Signed-off-by: Erik Bourget <erik@ebourget.net>
DjDeveloperr
left a comment
There was a problem hiding this comment.
Reviewed at 7e2760f, and run on an M5 Pro (J714s, T6050, G17S). Test kernel: integration/aurora 69a9fbb + d76f650 + 86708ce (the two integration commits this branch's base has in gem.rs, file.rs, mmu.rs and shmem.rs) + #1608..#1611, picked with -x. All 37 files the series touches are byte-identical to the PR heads.
Verdict: approve.
Code (read in full):
- Shmem constructor. The ordering holds:
inneris initialized beforeexternal, andexternalis declared beforeinner, so a failed mapping drops the driver data with nothing mapped, and the free path unmaps before the pages are unpinned. free_callback. It still callsdrm_gem_shmem_release()on the private object. With the shmem fields zeroed, that only takes the resv and ends indrm_gem_object_release(), which handles the NULLfilp.- No shmem page allocation for external objects. The C helpers that would allocate shmem pages are guarded:
make_vmap,madvise,release_idle_pages_locked,purge_locked, both releasable tests, andsg_table(), which returns the creation-time table.has_sg_table_locked()reads false for these objects, but every reclaim walk checksidle_reclaim_candidate()first, and that excludes them. - Driver mappings. Every driver-made mapping goes through
held_sg_table(), which refuses user memory: kernel aliases, firmware objects, timestamps, the VM-status mirror and the T8152 path (that path is kernel-only as well). Only the two user VM_BIND paths takeheld_user_sg_table(). VM_BIND prots are only RW/RO/WO, so the read-only rule (RO only,EACCESotherwise) is complete. - RLIMIT_MEMLOCK. The accounting follows the usual long-term pinner pattern (add to
pinned_vm, check,CAP_IPC_LOCK, undo), and Drop uncounts all of the range on any failure path. - UAPI numbering. Ioctl 12 and feature bit 37 collide with nothing on integration/aurora or in any open PR that touches
asahi_drm.h(only this one and #1610 do).
Other G17 GPUs. supports_user_memory() is is::<g17::Gpu>(), which is true on the T8140 (G17P) as well as on T6050. The coherence assumption is the one DRM_ASAHI_GEM_WRITEBACK already makes, as the description says. On the J714s (G17S), the three external_memory_host CTS cases that run go through user-memory objects and pass. That is consistent with the assumption, not a proof of it. I did not run anything on a T8140. Existing paths change in only two places, and both are equivalent for non-user-memory objects:
held_sg_table()gained auser_memory.is_some()test;- the bind paths call
held_user_sg_table().
On the M5 Pro (J714s):
- Selftest
asahi-userptr, built against this branch's UAPI header: 5/5 pass (anonymous_memory_round_trip,bad_ranges,read_only_file_pages,memlock_limit,fork_keeps_the_parent_pages).- It passed unprivileged, under the session user's default RLIMIT_MEMLOCK of 8 MiB, and again as root.
- On a baseline kernel without the series, all 5 skip ("no DRM_ASAHI_FEATURE_USERPTR").
- Vulkan side, mesa #242 (on aurora-wip be147e36250b, built on the device):
dEQP-VK.memory.external_memory_host.*gave 3 passed, 8 not supported, 0 failed. That's the same split as your J716c run. No GPU fault or recovery. - No regression in ordinary GPU work on the same boot (both bind paths now go through
held_user_sg_table()):- dEQP-VK multi-queue timeline list: 214/214;
- synchronization2 sample: 600 pass, 421 not supported, the same counts as the default kernel's own validation;
- gpubench core + feature chunks: 41/41;
- 0 GPU recoveries.
Build shortcut, for the record: the M5 test kernel was built incrementally with include/linux/dma-mapping.h and include/drm/drm_gem_shmem_helper.h at their old mtimes, so files that only include them weren't recompiled. The hunks are a file-scope #define plus comment, and a same-type, same-position field rename; the compile proof is CI's, below.
Compile proof: this head's kunit job built a full arm64 kernel (DRM_ASAHI=y, Rust) with #1608 below it, and passed 257/257 (https://github.com/aurora-silicon/linux/actions/runs/43125697227/job/130877078058). source and rust-build are green too (rust-build compiles only apple-sep.o).
|
Regression rows, fleet kernel with #1608 and #1609 against the same fleet kernel, same boot sequence: J700 (T8140) and J616s (T6040), which both report DRM_ASAHI_FEATURE_USERPTR: asahi-userptr 5/5 as user and as root, again after two s2idle cycles; dEQP-VK.memory.external_memory_host.* on mesa#242 4 pass / 8 not supported / 0 fail; CTS subset (3852 cases) 2968 pass / 884 not supported / 0 fail, same as the base; s2idle 2/2. M2 (feature not reported): CTS 3155/697/0, same as the base, s2idle 2/2. glmark2 / vkmark means within run-to-run spread: J700 5203 vs 5228 and 8238 vs 8305, J616s 12448 vs 12689 and 12752 vs 12679, M2 6364 vs 6333 and 8591 vs 8519. No AVC, DART, GPU fault or SWIOTLB error on any boot. |
The problem
A Vulkan application that already holds its data in memory (a mapped file, a
buffer another library filled) can hand that memory to the driver with
VK_EXT_external_memory_host and have the GPU use it in place. The asahi UAPI
had no way to create a buffer object over memory the caller owns: every
object was shmem memory the driver allocated, so such data had to be copied
into a buffer object first, and for large read-only inputs that doubles the
memory they take.
The change
rust: drm: gem: shmem: objects over driver-supplied pagesaddsObject::new_external(): a GEM object whose backing is a page array that theobject's driver data holds. It is a private GEM object with no shmem file, its
SG table is built and DMA-mapped once at creation (without CPU cache
maintenance, since the pages stay the owner's to use through its own
mappings), and it has its own GEM vtable with no mmap, vmap, pin or export.
The reclaim and purge helpers report its backing as never releasable.
and the pages are mapped only once it holds them, so a driver initializer
that fails, and releases the pages it was given, leaves nothing mapped. When
the object is freed, the mapping goes before the driver data.
bounce buffer, which a mapping without CPU synchronization never updates,
will not do. The table is therefore mapped with
DMA_ATTR_SKIP_CPU_SYNCand
DMA_ATTR_NO_BOUNCE(dma-mapping: add DMA_ATTR_NO_BOUNCE #1608), and the DMAlayer itself fails the mapping where it would put other memory in a page's
place:
EIOfor pages it would bounce (the DMA layer's error for any pageit cannot map),
EOPNOTSUPPfor a device with DMA operations of its own.The constructor infers nothing from the DMA addresses and uses no
DMA-internal interface, so a device with a bus offset, an address encoding
or an IOMMU is served like any other.
lifetime of the pages, and that the device accesses them coherently with
the CPU caches. The two attributes give the device the pages themselves and
leave the caches alone; they make no device coherent, the mapping succeeds
for one that is not, and the constructor cannot check it. The caller has to
know it of its hardware.
How the driver meets that contract:
GPU-only entries of the VM's page tables at the pages' DMA addresses; the
mappings the driver makes for itself and for the firmware refuse it. Such
a bind gets the same entries (
PROT_GPU_SHARED_*, the same GPU memoryattribute) as a bind of an ordinary object: nothing in the choice of the
entry depends on whether the pages are shmem pages or the caller's.
caches is an assumption about the hardware. The DMA attributes do not
establish it and the driver does not check it. The driver relies on it
already for its ordinary objects: one created with
DRM_ASAHI_GEM_WRITEBACKis mapped cacheable into its process, whichstores to it and loads from it while the GPU uses it, with no cache
maintenance by the driver and no UAPI call for it. User-memory objects
rely on the same assumption, which is stated in the safety comment at the
call site, and on the J716c the coherence check under "Testing" supports
it.
PFN mappings, but it does not check the caching attributes of every
mapping it accepts, so nothing is claimed for memory that a process has
mapped in another way. The feature is offered on G17 GPUs only, and no
claim is made for machines with a G17 GPU other than the J716c.
dma-rangesand its DMA mask (42 bits) covers the machine's memory, so theDMA layer maps it directly and has no reason to bounce. Its device node is
not marked DMA coherent, which is why the mapping has to skip the DMA
layer's cache maintenance explicitly.
drm/asahi: GEM_USERPTR, buffer objects over the caller's memoryaddsDRM_IOCTL_ASAHI_GEM_USERPTRon G17 GPUs:creation, all of it or none, and stays pinned until the object is freed.
The pages count against the caller's RLIMIT_MEMLOCK (
mm->pinned_vm)unless it has CAP_IPC_LOCK, and their size against the client and device
GPU memory limits.
DRM_ASAHI_GEM_USERPTR_READ_ONLYthe range must be writable.With it any readable mapping is accepted and the object takes only GPU
read-only binds (
EACCESotherwise). Device and PFN mappings are refused.export, BIND_OBJECT and the VM status mirror refuse it, and it is never an
idle-reclaim or purge candidate.
selftests: drm/asahi: GEM_USERPTRadds a kselftest program for the rendernode (
tools/testing/selftests/drivers/gpu/asahi-userptr, listed in theselftests'
TARGETS). It skips whererenderD128is not this driver's node,and where the caller has neither CAP_IPC_LOCK nor a RLIMIT_MEMLOCK hard limit
with room for the four pages a test pins.
memlock_limittakes its sizesfrom the hard limit (16 pages, or as many as it has room for).
UAPI additions: ioctl
DRM_ASAHI_GEM_USERPTR(id 12, the next afterDRM_ASAHI_GEM_MADVISE),struct drm_asahi_gem_userptr, the object flagDRM_ASAHI_GEM_USERPTR_READ_ONLY(bit 2 ofenum drm_asahi_gem_flags) andthe feature
DRM_ASAHI_FEATURE_USERPTR, bit 37. Bit 35 is taken by#1610; bit 36 is left to a per-queue compute time limit that is not filed yet.
Testing
Hardware: M5 Max MacBook Pro (J716c).
Build:
integration/auroraea10c4d plus dma-mapping: add DMA_ATTR_NO_BOUNCE #1608plus this branch, with the J716c configuration (Image, modules, device
trees), at each commit. No new warning: the build has the same 31 Rust
warnings at each commit, none in code the branch adds.
KUnit, arm64 with 16K pages in QEMU:
./tools/testing/kunit/kunit.py run --arch=arm64 --kunitconfig=drivers/soc/apple/sep.kunitconfig:Ran 545 tests: passed: 543, skipped: 2, no warning in the kernel log.rust_drm_gem_shmempasses 18 of 18 and the newasahi_gem_user_memorysuite 5 of 5. Of the six new
rust_drm_gem_shmemtests:external_mapping_never_outlives_the_pagescreates objects with a driverwhose data owns the pages and whose initializer can fail after taking
them, and checks that no mapping exists when the pages are released;
external_mapping_follows_a_bus_offsetgives the device adma_range_mapthat puts every CPU address 1 TiB up on its bus. The DMAlayer hands out addresses that are not the pages' physical addresses,
and the object is created;
external_mapping_refuses_a_bounce_bufferis the test of the refusalthrough the constructor. Its device has a
dma_range_mapthat reachesall memory except one of the test's pages, so the DMA layer can serve it
that page only through SWIOTLB. The test first shows that it does: an
ordinary mapping of the page (no attribute) succeeds, its DMA address
lies outside the page, the memory there holds what the page held when it
was mapped, and a later store to the page does not show there. Then an
object over two pages the device reaches is created, and an object over
a page it reaches followed by that one fails with
EIO: the first pagewas mapped and unmapped again inside the DMA layer, and the driver data,
which had taken the pages, releases them with no mapping left. The test
is built for arm64, which always sets up SWIOTLB. It fails when the
constructor maps without
DMA_ATTR_NO_BOUNCE: the object is thencreated over the bounce buffer.
The two tests with a bus build the device's
dma_range_mapfromstruct bus_dma_regionas C defines it: the Rust bindings includelinux/dma-direct.h, the DMA layer's own header, for KUnit builds only(
CONFIG_KUNIT), and the abstraction uses nothing from it.Host tests, as CI runs them:
python3 -I tools/asahi/run-host-tests.py:219/224 passed. The five that fail (apple-dcp,g15-pb-growth-testsand
g15-stats-drain-tests, each of the last two also with--mutations)fail identically on
integration/auroraea10c4d on the build host,which lacks the Rust 1.93.1 toolchain the g15 tests ask for.
scripts/checkpatch.pl: 0 errors and 0 warnings on the two drivercommits; the selftest commit gets only the MAINTAINERS note for its new
files.
On the J716c, kernel 7.3.0-ewb111+, built from
integration/auroraea10c4d plus the two open J716c device tree pull requests and the
kernel branches of this set, with the kernel code of this branch and of
dma-mapping: add DMA_ATTR_NO_BOUNCE #1608 as it is here: the selftest built there with
gcc 16.1.1 against this branch's UAPI header, as an unprivileged user and
as root:
# Totals: pass:5 fail:0 xfail:0 xpass:0 skip:0 error:0(
anonymous_memory_round_trip,bad_ranges,read_only_file_pages,memlock_limit,fork_keeps_the_parent_pages).memlock_limitgives upCAP_IPC_LOCK before it asks for more than RLIMIT_MEMLOCK, so the request
has to fail for root as well. With RLIMIT_MEMLOCK at 128 KiB (soft and
hard, and 32 KiB soft with 128 KiB hard) both runs pass 5 again; with a
hard limit of 48 KiB the unprivileged run skips all five and root passes
five; with 16 KiB the unprivileged run skips all five again, and root
passes four and skips
memlock_limit.On 7.3.0-ewb111+, Vulkan CTS (commit 7ba03cd) with Mesa
aurora-wipplusthe honeykrisp change that implements VK_EXT_external_memory_host on this
ioctl:
dEQP-VK.memory.external_memory_host.*: 3 passed, 8 not supported(the image cases: no image format is offered over host memory), 0 failed.
On 7.3.0-ewb111+ with the same Mesa, a check that the GPU and the CPU see
the same data in such objects with no cache maintenance: a Vulkan program
(not part of the branch) imports two anonymous ranges of its own as
storage buffers; in each iteration the CPU stores a new pattern to the
first and a marker to the second through its own cacheable mappings, a
compute job copies the first to the second while the CPU polls for its
end, and the CPU compares.
A GPU read that missed the CPU's last store would show as the previous
pattern, a GPU store that the CPU does not see as the marker.
PASS: ... 0 wrong wordsfor ranges of 16 KiB and 64 KiB (20000iterations each; both ranges fit the CPU's first-level cache), 1 MiB (3000)
and 64 MiB (30). The process's
VmPinshows the imported pages pinned(32 kB, 128 kB, 2048 kB, 131072 kB).
What shows what: that a bounce buffer is refused is shown in QEMU, by the
KUnit test above through the constructor and by the suite of
#1608 in the DMA layer. The J716c runs show that the
GPU, which the DMA layer maps directly and never bounces, is accepted, and
the coherence check found no word in which the GPU and the CPU saw different
data: a measurement on one machine that supports the hardware assumption
above, not a proof of it.
Depends on
#1608: this branch starts from its branch and maps
every object with the attribute it adds. #1610 is stacked
on this branch. The Vulkan side is a separate pull request against
aurora-silicon/mesa.
Not covered
with
EOPNOTSUPP.hardware that the driver relies on and does not establish. It was measured
on the J716c only, with the program above and with anonymous memory;
other machines with a G17 GPU were not measured, and memory mapped with
caching attributes other than those of normal RAM was not tried and is
not covered by the assumption. The device node of the GPU is not marked
DMA coherent, and the constructor would map the pages of a device that is
not coherent just the same.
GPU reads and writes through such an object were exercised by the three
CTS cases and by the coherence check only.
never bounced. The refusals (
EIO,EOPNOTSUPP) were seen in QEMU only,and only the dma-direct one; see dma-mapping: add DMA_ATTR_NO_BOUNCE #1608 for what of
the attribute no test reaches.
a platform with memory encryption: the GPU has none of these. The bus
offset is exercised in KUnit through the DMA layer.
🤖 Generated with Claude Code