From 694a799a632826f80ca8e22653659e6684a736b0 Mon Sep 17 00:00:00 2001 From: Andrey Kolkov Date: Sun, 1 Nov 2026 03:13:12 +0300 Subject: [PATCH] fix(amd64): lay out System V arguments as libffi does A by-value struct over 72 bytes panicked with "index out of range [15]" on Linux, macOS and FreeBSD amd64: stack arguments went into a fixed 9-word area whose bound was checked after the write, and PrepareCallInterface counted a struct over 16 bytes as using no stack. webgpu.h's 96-byte WGPUAdapterInfo hit it. The argument layout now follows psABI 3.2.3 as libffi implements it (src/x86/ffi64.c): an argument goes in registers only if every eightbyte gets one, otherwise the whole argument goes on the stack and the registers left stay free. Before, a 9-16 byte struct with too few registers was split between a register and the stack, and the callee read wrong values. One allocator (layout_sysv.go) serves PrepareCallInterface, the call and the Unix callback decoder. syscallN reserves a stack area sized from the arguments, up to 64 KiB (MaxStackBytes); a larger argument list is an error at prepare time and, for a hand-built CallInterface, at call time. The Windows call path gets the same call-time check. Nested struct members are classified by their offsets, for arguments, results and callbacks. Document that a Go address must stay pointer-typed all the way to avalue, binding wrappers included (wgpu's vkMapMemory binding lost a mapping that way), and test that C can write through an out-pointer after a callback moved the goroutine stack. Tests: 96/64/40-byte structs (webgpu shapes), 24 stack words, the split cases in calls and callbacks, nested float structs, too many arguments. All but the 64- and 40-byte cases (which fit the old 9 words) fail on b9cdfd7. Fixes #112 --- CHANGELOG.md | 5 + README.md | 3 +- SECURITY.md | 26 +- docs/ARCHITECTURE.md | 24 +- ffi/call.go | 7 + ffi/callback.go | 357 ++++++++++--------- ffi/callback_struct_args_test.go | 52 +-- ffi/cif.go | 9 +- ffi/ffi.go | 21 +- ffi/ffi_test.go | 12 +- ffi/pointer_contract_e2e_test.go | 58 ++++ ffi/stack_layout_e2e_test.go | 438 ++++++++++++++++++++++++ ffi/testdata/structtest.c | 113 ++++++ internal/arch/amd64/call_unix.go | 221 +++++------- internal/arch/amd64/call_windows.go | 11 + internal/arch/amd64/classification.go | 50 +-- internal/arch/amd64/layout_sysv.go | 175 ++++++++++ internal/arch/amd64/struct_args_test.go | 109 ++---- internal/syscall/syscall_unix_amd64.go | 158 ++++----- internal/syscall/syscall_unix_amd64.s | 155 ++++----- 20 files changed, 1386 insertions(+), 618 deletions(-) create mode 100644 ffi/pointer_contract_e2e_test.go create mode 100644 ffi/stack_layout_e2e_test.go create mode 100644 internal/arch/amd64/layout_sysv.go diff --git a/CHANGELOG.md b/CHANGELOG.md index 82e228c..02efa9b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,9 +8,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Changed +- **Documented: a Go address must stay pointer-typed all the way to `avalue`.** The compiler keeps a `uintptr`'s target alive and unmoved only in the argument list of a call to a function implemented in assembly or marked `//go:uintptrescapes`; `CallFunction` and a binding's own wrapper functions are ordinary Go functions. A local whose address travels as a `uintptr` — for example through a wrapper parameter `ppData uintptr` — can stay on the goroutine stack, and when the stack moves during the call C writes into the old stack (found in wgpu's Vulkan `vkMapMemory` binding). The `CallFunctionContext` documentation and `SECURITY.md` now state the rule, the out-pointer pattern and when `runtime.Pinner` is needed, and that memory reachable from `rvalue` and `avalue` through pointer-typed values stays in place for the call. A new test lets C write through an out-pointer after a callback has moved the goroutine stack. - **Documentation brought in line with the code.** The `ffi` package documentation now has a compiling example (`CallFunction` returns the errno first) and states that a prepared `CallInterface` may be shared by goroutines; it no longer starts with a file comment from `callback.go` (three callback files carried `// Package ffi` comments). The `types` package has a package comment. README, `docs/ARCHITECTURE.md`, `docs/PERFORMANCE.md`, `CONTRIBUTING.md`, `SECURITY.md` and `ROADMAP.md` now describe the current state: the Linux/FreeBSD `CGO_ENABLED=1` link limitation with other cgo code and its workarounds ([#72](https://github.com/go-webgpu/goffi/issues/72)) — README previously suggested `CGO_ENABLED=1` for exactly that case; the FreeBSD `-gcflags` needed with `CGO_ENABLED=0`; Windows amd64 calls through `syscall.SyscallN` rather than goffi assembly; callback signature rules per platform (the README callback example panicked on Windows); `avalue` slices escape, so building one per call costs one allocation; default builds are dynamically linked, not static; the GitHub Flow contribution workflow (there is no `develop` branch), license headers, and lint on every target; supported versions and reporting in `SECURITY.md`; the v0.6.8 and v0.7.0 plans and the libvips stress-test target ([#71](https://github.com/go-webgpu/goffi/issues/71)) in `ROADMAP.md`. ### Fixed +- **System V amd64 (Linux, macOS, FreeBSD): a struct argument over 72 bytes panicked.** goffi copied stack arguments into a fixed area of 9 words and checked its size only after writing, so a by-value struct of more than 72 bytes (webgpu.h's 96-byte `WGPUAdapterInfo`), or several large structs together, ended in `index out of range [15] with length 15` inside `CallFunction`. `PrepareCallInterface` did not catch it either: it counted a struct over 16 bytes as using no registers and no stack. The stack area is now sized from the arguments, as libffi does, up to 64 KiB (`amd64.MaxStackBytes`; libffi has no limit, the bound keeps the copy well inside the system stack the call runs on). An argument list that needs more is an error at `PrepareCallInterface` and, for a `CallInterface` filled in by hand, at the call — never a panic. The same call-time check now guards the Windows amd64 call path. ([#112](https://github.com/go-webgpu/goffi/issues/112)) +- **System V amd64: a 9–16 byte struct argument was split between a register and the stack.** When fewer registers were left than the struct needs — a `{int64, int64}` after five integer arguments, a `{double, double}` after seven doubles, an `{int64, double}` after six integers — its first eightbyte went into the last register and the rest onto the stack, and the arguments after it were shifted. The ABI passes such an argument on the stack whole and leaves the register to later arguments (psABI §3.2.3; libffi `src/x86/ffi64.c`). The callee received wrong values without any error. Calls and Unix amd64 callbacks now follow the ABI; one argument layout (`internal/arch/amd64/layout_sysv.go`) serves `PrepareCallInterface`, the call and the callback decoder. ([#112](https://github.com/go-webgpu/goffi/issues/112)) +- **System V amd64: nested struct members were classified as INTEGER.** An eightbyte holding floating-point members of a nested struct — `struct { struct { float x, y; } a, b; }`, arrays of floats in callbacks — was passed in a general-purpose register instead of XMM, and such a struct result was read from RAX:RDX instead of XMM0:XMM1. Members are now classified by their offsets at any depth, as libffi does. ([#112](https://github.com/go-webgpu/goffi/issues/112)) +- **Unix amd64 callbacks: arguments past 114 stack words indexed past the frame.** `NewCallback` now counts the stack words of the signature and panics with a clear message past the same 64 KiB bound as calls. - **Unix amd64: crash when a goroutine exits locked to its thread after a call.** `syscallN` saved its argument pointer above its own frame, into its caller's frame on the g0 stack. A goroutine that made a goffi call and then exited without `runtime.UnlockOSThread` crashed the process with SIGSEGV when its thread exited. Linux, macOS and FreeBSD amd64 were affected; arm64 and Windows were not. ([#108](https://github.com/go-webgpu/goffi/issues/108), [#109](https://github.com/go-webgpu/goffi/pull/109), reported and fixed by @aardwulf89) ## [0.6.7] - 2026-10-18 diff --git a/README.md b/README.md index 5d47910..a469646 100644 --- a/README.md +++ b/README.md @@ -143,7 +143,8 @@ func main() { } // Call strlen — avalue elements are pointers TO argument values. - // Keep pointer arguments in unsafe.Pointer variables, never uintptr. + // Keep pointer arguments in unsafe.Pointer variables, never uintptr — + // in your binding wrappers' parameters too (see SECURITY.md). testStr := "Hello, goffi!\x00" strPtr := unsafe.Pointer(unsafe.StringData(testStr)) var length uint64 diff --git a/SECURITY.md b/SECURITY.md index ad4e463..d831ae4 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -78,10 +78,34 @@ p := unsafe.Pointer(&buf[0]) _, _ = ffi.CallFunction(&cif, fn, nil, []unsafe.Pointer{unsafe.Pointer(&p)}) runtime.KeepAlive(buf) +// ❌ BAD - a binding wrapper with a uintptr parameter. Writing the conversion +// in the caller's argument list does not help: only calls to functions +// implemented in assembly (or marked //go:uintptrescapes) keep a uintptr's +// target in place, and MapMemory is an ordinary Go function. mapped stays on +// the stack, the stack moves, and C writes the mapping into the old stack. +func MapMemory(cif *types.CallInterface, fn unsafe.Pointer, ppData uintptr) { + _, _ = ffi.CallFunction(cif, fn, nil, []unsafe.Pointer{unsafe.Pointer(&ppData)}) +} +var mapped unsafe.Pointer +MapMemory(&cif, fn, uintptr(unsafe.Pointer(&mapped))) + +// ✅ GOOD - out-pointer (C type void **pp): keep &mapped in an +// unsafe.Pointer slot and take the parameter as unsafe.Pointer. +func MapMemory(cif *types.CallInterface, fn unsafe.Pointer, ppData unsafe.Pointer) { + _, _ = ffi.CallFunction(cif, fn, nil, []unsafe.Pointer{unsafe.Pointer(&ppData)}) +} +var mapped unsafe.Pointer +MapMemory(&cif, fn, unsafe.Pointer(&mapped)) + // ✅ If C keeps the pointer after the call returns, or a callback reads it -// later, pin the memory (runtime.Pinner) or allocate it on the C side. +// later, or the Go address must go into a uintptr-typed C struct field, +// pin the memory (runtime.Pinner) or allocate it on the C side. ``` +goffi keeps memory reachable from `rvalue` and `avalue` through pointer-typed values alive and in place for the +call (it is heap memory: `avalue` escapes). A `uintptr` breaks that chain wherever it appears between the address +and `avalue`. + C strings must be NUL-terminated (`"hello\x00"`); Go strings are not. ### 2. Race Conditions diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index be61964..4328bf7 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -96,12 +96,12 @@ Hand-written assembly for System V AMD64 and AAPCS64. The function receives a st ```asm TEXT syscallN(SB), NOSPLIT|NOFRAME, $0 - // R11 = args struct pointer - // Load 6 GP registers: RDI, RSI, RDX, RCX, R8, R9 + // R12 = args struct pointer (callee-saved across the call) + // Reserve nstack words (rounded to 16 bytes) and copy the stack arguments // Load 8 SSE registers: XMM0-XMM7 - // Push stack-spill args if needed + // Load 6 GP registers: RDI, RSI, RDX, RCX, R8, R9 CALL R10 // call target function - // Save RAX (int return), RDX (second return), XMM0 (float return) + // Save RAX (int return), RDX (second return), XMM0/XMM1 (float returns) ``` **Win64** — no goffi assembly. `internal/arch/amd64/call_windows.go` packs the arguments into `uintptr` slots (a hidden result pointer first when the result needs one) and calls `syscall.SyscallN`, which loads RCX, RDX, R8, R9 and XMM0–XMM3 from those slots and provides the 32-byte shadow space. @@ -141,13 +141,11 @@ ABI rules for passing structs as arguments depend on size and platform: Per §3.2.3, each struct is classified by its eightbytes (8-byte chunks): -- **≤ 8 bytes**: single eightbyte. If all fields are float/double → SSE (XMM register). Otherwise → INTEGER (GP register). INTEGER wins over SSE within the same eightbyte (merge rule). -- **9-16 bytes**: two eightbytes, each classified independently. First 8 bytes → GP or XMM. Remaining bytes → GP or XMM. Both classifications use the same INTEGER-wins merge rule. -- **\> 16 bytes**: MEMORY class. Caller copies struct bytes onto the stack in 8-byte chunks. +- **≤ 8 bytes**: single eightbyte. If all fields in it are float/double → SSE (XMM register). Otherwise → INTEGER (GP register). INTEGER wins over SSE within the same eightbyte (merge rule). Fields of nested structs are classified by their offsets, at any depth. +- **9-16 bytes**: two eightbytes, each classified independently with the same rule. +- **\> 16 bytes**: MEMORY class. Caller copies struct bytes onto the stack in 8-byte words. -Implementation in `internal/arch/amd64/call_unix.go`, helpers in `classification.go`: -- `isStructAllFloats(t)` — returns true if all members are float/double -- `classifyEightbyte(t, startOff, endOff)` — per-eightbyte SSE classification with merge rule +`internal/arch/amd64/layout_sysv.go` assigns every argument its registers or stack words, as libffi does (`ffi_prep_cif_machdep` and `ffi_call_int` in `src/x86/ffi64.c`): an argument goes in registers only if every one of its eightbytes gets a register of its class; otherwise the whole argument goes on the stack and the registers left stay free for later arguments. `PrepareCallInterface`, the call (`call_unix.go`) and the callback decoder (`ffi/callback.go`) use the same allocator. The stack area is sized from the arguments, up to 64 KiB (`amd64.MaxStackBytes`); a call that needs more returns `ErrTooManyArguments` at prepare time, or an error at call time for a hand-built `CallInterface`. Eightbyte classification: `classifyEightbyte` in `classification.go`. ### Win64 (Windows AMD64) @@ -172,6 +170,7 @@ The ABI is verified against real C callers and callees: `TestMain` in `ffi/struc - byte structs of 1–9, 12, 15, 16, 17, 24 and 32 bytes, as argument and as result, including a nil result buffer where a hidden result pointer is needed (`struct_sizes_e2e_test.go`); - 9–16 byte results in each register pair (`struct_return_e2e_amd64_test.go`) and struct results of a hand-built `CallInterface` (`unprepared_cif_e2e_test.go`); - stack arguments, narrow and `int` integers, variadic calls, the one-value-per-argument rule (`stack_args_e2e_test.go`, `narrow_int_e2e_test.go`, `int_type_e2e_test.go`, `variadic_e2e_test.go`, `avalue_count_e2e_test.go`); +- the stack layout: by-value structs of 40, 64 and 96 bytes (webgpu.h shapes), 24 stack words, 9–16 byte structs that must go to the stack whole, nested float structs, too many arguments as an error, in calls and callbacks (`stack_layout_e2e_test.go`); an out-pointer written after the goroutine stack moved (`pointer_contract_e2e_test.go`); - callbacks called from C: float arguments and results, stack arguments, foreign threads (`callback_float_e2e_test.go`, `callback_stack_e2e_test.go`, `callback_cthread_test.go`). The test library is built with `$CC` (default `gcc`; the FreeBSD CI job uses the base system `cc`). The tests skip only on Android (test binaries run on the device), in a `goffi_static` build, or when there is no C compiler for the test's architecture; with `GOFFI_REQUIRE_TESTLIB=1`, which CI sets wherever such a compiler exists, a missing compiler fails the run as well. A library that fails to build or load always fails the run. CI runs them on Linux, macOS and Windows (amd64 and arm64) and on FreeBSD amd64; on windows-11-arm they skip, because the runner only has an x86-64 MinGW compiler. @@ -182,10 +181,11 @@ The test library is built with `$CC` (default `gcc`; the FreeBSD CI job uses the When C code calls a goffi callback with struct arguments, the callback dispatch (`callbackWrap` in `callback.go`) reconstructs struct values from CPU registers and stack using `reflect.Type`: -- **≤ 8 bytes**: single eightbyte in GP or XMM register. `isStructAllFloats()` determines classification (recursive, supports nested structs). -- **9-16 bytes**: two eightbytes, each classified independently via `classifyEightbyte()` using `reflect.StructField.Offset`. +- **≤ 16 bytes**: one or two eightbytes, each classified by `classifyEightbyte()` from the fields' offsets (nested structs and arrays included). If the registers left cannot take every eightbyte, the C caller put the whole struct on the stack, and the callback reads it from there. - **\> 16 bytes**: MEMORY class — C caller copies bytes onto stack. Callback reads consecutive stack slots directly. No assembly changes needed. +Registers and stack words are assigned by the allocator the call path uses (`internal/arch/amd64/layout_sysv.go`). `NewCallback` counts the signature's stack words and panics past 64 KiB. + Classification uses `reflect.Type` (not `types.TypeDescriptor`) since callback signatures are Go functions registered via `NewCallback()`. **Limitations**: callback struct args supported on AMD64 Unix only. ARM64 and Windows callbacks do not yet support struct arguments. diff --git a/ffi/call.go b/ffi/call.go index b6736fe..614f189 100644 --- a/ffi/call.go +++ b/ffi/call.go @@ -23,5 +23,12 @@ func executeFunction( // on Windows. The assembly trampoline's conditional (TESTQ/CBZ) skips the // errno capture when errnoFn is 0, so this is safe on all platforms. errnoFn := gosyscall.ErrnoFnAddr() + // The interface call makes rvalue and avalue escape ("leaking param" in + // -gcflags=-m), so everything reachable from them through pointer-typed + // values is heap memory that does not move while C uses it: the contract + // the CallFunctionContext documentation states. A change that lets the + // compiler see through this call (devirtualization, inlining Execute) + // must keep that escape, or the arch code would read stack addresses that + // a stack move invalidates. return arch.Registry.Caller.Execute(cif, fn, rvalue, avalue, errnoFn) } diff --git a/ffi/callback.go b/ffi/callback.go index 885cbeb..bb4768b 100644 --- a/ffi/callback.go +++ b/ffi/callback.go @@ -9,9 +9,12 @@ package ffi import ( "math" "reflect" + "strconv" "structs" "sync" "unsafe" + + archamd64 "github.com/go-webgpu/goffi/internal/arch/amd64" ) // maxCallbacks is the maximum number of concurrent callbacks supported. @@ -23,9 +26,10 @@ const maxCallbacks = 2000 // called from C code. Functions are stored as reflect.Value to enable dynamic // invocation with proper type checking and argument marshaling. var callbacks struct { - mu sync.Mutex // Protects funcs and count - funcs [maxCallbacks]reflect.Value // Registered callback functions - count int // Number of active callbacks + mu sync.Mutex // Protects funcs, stackWords and count + funcs [maxCallbacks]reflect.Value // Registered callback functions + stackWords [maxCallbacks]int // Stack words of each callback's arguments + count int // Number of active callbacks } // callbackArgs represents the argument block passed from assembly to callbackWrap. @@ -78,7 +82,7 @@ func NewCallback(fn any) uintptr { // Validate function signature typ := val.Type() - validateCallbackSignature(typ) + stackWords := validateCallbackSignature(typ) // Register callback in global registry callbacks.mu.Lock() @@ -90,6 +94,7 @@ func NewCallback(fn any) uintptr { idx := callbacks.count callbacks.funcs[idx] = val + callbacks.stackWords[idx] = stackWords callbacks.count++ // Return address to corresponding trampoline entry @@ -98,7 +103,8 @@ func NewCallback(fn any) uintptr { // validateCallbackSignature checks if a function type is valid for callbacks. // This function enforces the constraints required by the FFI calling convention. -func validateCallbackSignature(typ reflect.Type) { +// It returns how many stack words the arguments occupy. +func validateCallbackSignature(typ reflect.Type) int { // Validate input arguments numIn := typ.NumIn() for i := 0; i < numIn; i++ { @@ -133,6 +139,13 @@ func validateCallbackSignature(typ reflect.Type) { default: panic("ffi: callbacks can only return zero or one value") } + + words := callbackStackWords(typ) + if words > callbackMaxStackWords { + panic("ffi: callback arguments need " + strconv.Itoa(words) + " stack words, at most " + + strconv.Itoa(callbackMaxStackWords) + " are supported") + } + return words } // trampolineEntryAddr calculates the address of a specific trampoline entry. @@ -182,6 +195,7 @@ func callbackWrap(a *callbackArgs) { // Retrieve the registered callback function callbacks.mu.Lock() fn := callbacks.funcs[a.index] + stackWords := callbacks.stackWords[a.index] callbacks.mu.Unlock() typ := fn.Type() @@ -189,87 +203,43 @@ func callbackWrap(a *callbackArgs) { // Argument block layout (System V AMD64 ABI): // [XMM0-7: 64 bytes][RDI,RSI,RDX,RCX,R8,R9: 48 bytes][stack args...] - const ( - numFloatRegs = 8 // XMM0-XMM7 - numIntRegs = 6 // RDI, RSI, RDX, RCX, R8, R9 - ) - - // Cast args pointer to array for easy indexing - // Each register is 8 bytes (64-bit) - frame := (*[128]uintptr)(a.args) // Large enough for registers + reasonable stack args - - var floatIdx int // Current float register index (0-7) - var intIdx int // Current integer register index (0-5) - stackIdx := numFloatRegs + numIntRegs // Stack arguments start after registers - - // getFloatBits returns the raw 64 bits of the next SSE-class eightbyte: an - // XMM register while one is left, otherwise the next stack slot. A float - // occupies the low 32 bits (both in XMMn and in its 8-byte stack slot), a - // double all 64; callers decode accordingly. - getFloatBits := func() uint64 { - if floatIdx < numFloatRegs { - bits := frame[floatIdx] - floatIdx++ - return uint64(bits) - } - bits := frame[stackIdx] - stackIdx++ - return uint64(bits) - } - - getInt := func() uintptr { - var value uintptr - if intIdx < numIntRegs { - value = frame[numFloatRegs+intIdx] - intIdx++ - } else { - // All register slots are used: value is on the stack - value = frame[stackIdx] - stackIdx++ - } - return value - } - - // Write only some bytes to a struct to avoid overwrite. - writePartial := func(dest unsafe.Pointer, size uintptr, value uintptr) { - switch { - case size == 1: - *(*uint8)(dest) = uint8(value) - case size == 2: - *(*uint16)(dest) = uint16(value) - case size <= 4: - *(*uint32)(dest) = uint32(value) - default: - *(*uintptr)(dest) = value - } - } + // Arguments are assigned registers and stack words by the same allocator + // the call path uses (internal/arch/amd64/layout_sysv.go), so an argument + // that does not fit in the registers left is read from the stack whole. + // The frame covers exactly the stack words the signature declares + // (counted by validateCallbackSignature), which the C caller wrote. + frame := unsafe.Slice((*uintptr)(a.args), callbackStackBase+stackWords) + var alloc archamd64.SysVAllocator // Build argument slice for reflection Call args := make([]reflect.Value, numArgs) for i := 0; i < numArgs; i++ { argType := typ.In(i) + p := placeCallbackArg(&alloc, argType) + // word is the scalar argument's register or stack word. A float + // occupies the low 32 bits (both in XMMn and in its 8-byte stack + // slot), a double all 64. + word := func() uintptr { + if p.onStack { + return frame[p.first] + } + return frame[p.idx[0]] + } var val reflect.Value switch argType.Kind() { case reflect.Float32: // The caller passes a single-precision value, not a double: // reinterpret the low 32 bits, do not convert a float64. - val = reflect.ValueOf(math.Float32frombits(uint32(getFloatBits()))).Convert(argType) + val = reflect.ValueOf(math.Float32frombits(uint32(word()))).Convert(argType) case reflect.Float64: - val = reflect.ValueOf(math.Float64frombits(getFloatBits())).Convert(argType) + val = reflect.ValueOf(math.Float64frombits(uint64(word()))).Convert(argType) case reflect.Bool: // Bool comes from integer register - if intIdx < numIntRegs { - pos := numFloatRegs + intIdx - val = reflect.ValueOf(frame[pos] != 0) - intIdx++ - } else { - val = reflect.ValueOf(frame[stackIdx] != 0) - stackIdx++ - } + val = reflect.ValueOf(word() != 0) case reflect.Pointer: // Pointers come from integer registers. @@ -279,87 +249,38 @@ func callbackWrap(a *callbackArgs) { // 1. The pointer came from C code which is responsible for its lifetime // 2. We're only using it for the duration of this callback invocation // 3. reflect.NewAt creates a proper typed pointer from the address - if intIdx < numIntRegs { - pos := numFloatRegs + intIdx - // Double-indirection: reinterpret uintptr bits as pointer without - // triggering checkptr arithmetic check (go.dev/issue/58625). - ptr := *(*unsafe.Pointer)(unsafe.Pointer(&frame[pos])) - val = reflect.NewAt(argType.Elem(), ptr) - intIdx++ - } else { - ptr := *(*unsafe.Pointer)(unsafe.Pointer(&frame[stackIdx])) - val = reflect.NewAt(argType.Elem(), ptr) - stackIdx++ - } + w := word() + // Double-indirection: reinterpret uintptr bits as pointer without + // triggering checkptr arithmetic check (go.dev/issue/58625). + ptr := *(*unsafe.Pointer)(unsafe.Pointer(&w)) + val = reflect.NewAt(argType.Elem(), ptr) case reflect.UnsafePointer: - if intIdx < numIntRegs { - pos := numFloatRegs + intIdx - val = reflect.ValueOf(*(*unsafe.Pointer)(unsafe.Pointer(&frame[pos]))) - intIdx++ - } else { - val = reflect.ValueOf(*(*unsafe.Pointer)(unsafe.Pointer(&frame[stackIdx]))) - stackIdx++ - } + w := word() + val = reflect.ValueOf(*(*unsafe.Pointer)(unsafe.Pointer(&w))) case reflect.Struct: + // The struct's bytes as whole words: from its registers, one per + // eightbyte, when they fit, otherwise from its stack words. sz := argType.Size() - structData := make([]byte, max(sz, 8)) - var valPtr unsafe.Pointer - if sz > 0 { - valPtr = unsafe.Pointer(&structData[0]) - } - - switch { - case sz == 0: - // Zero-size struct: no fields to populate - - case sz <= 8: - // Single eightbyte: INTEGER if any member is not float/double, else SSE. - if isStructAllFloats(argType) { - *(*uint64)(valPtr) = getFloatBits() - } else { - writePartial(valPtr, sz, getInt()) - } - case sz <= 16: - // Two eightbytes: classify each independently. - // System V ABI §3.2.3: INTEGER wins over SSE within an eightbyte. - if classifyEightbyte(argType, 0, 8) { - *(*uint64)(valPtr) = getFloatBits() - } else { - *(*uintptr)(valPtr) = getInt() - } - remaining := sz - 8 - valPtr = unsafe.Add(valPtr, 8) - if classifyEightbyte(argType, 8, sz) { - *(*uint64)(valPtr) = getFloatBits() - } else { - writePartial(valPtr, remaining, getInt()) - } - default: - // MEMORY class (> 16 bytes): copy from the stack in 8-byte chunks. - // Per SysV ABI §3.2.3: MEMORY class structs bypass registers entirely. - nChunks := (sz + 7) / 8 - for k := range nChunks { - chunkPtr := unsafe.Add(valPtr, k*8) - chunk := frame[stackIdx] - bytesLeft := sz - k*8 - if bytesLeft >= 8 { - *(*uintptr)(chunkPtr) = chunk - } else { - writePartial(chunkPtr, bytesLeft, chunk) - } - stackIdx++ + words := make([]uintptr, (sz+7)/8) + if p.onStack { + copy(words, frame[p.first:p.first+len(words)]) + } else { + for k := range p.n { + words[k] = frame[p.idx[k]] } } val = reflect.New(argType) - valByteSlice := unsafe.Slice((*byte)(val.UnsafePointer()), sz) - copy(valByteSlice, structData) + if sz > 0 { + copy(unsafe.Slice((*byte)(val.UnsafePointer()), sz), + unsafe.Slice((*byte)(unsafe.Pointer(unsafe.SliceData(words))), sz)) + } val = val.Elem() default: // All other integer types (int, uint, int32, etc.) - value := getInt() + value := word() val = reflect.NewAt(argType, unsafe.Pointer(&value)).Elem() } @@ -405,44 +326,146 @@ func callbackWrap(a *callbackArgs) { var _callbackTrampoline byte var trampolineBaseAddr = uintptr(unsafe.Pointer(&_callbackTrampoline)) -// isStructAllFloats returns true if every member of a flat struct is float or double. -// Per System V AMD64 ABI §3.2.3: if any member in an eightbyte is INTEGER class, -// the entire eightbyte is classified as INTEGER (INTEGER wins over SSE). -func isStructAllFloats(structType reflect.Type) bool { - if structType.NumField() == 0 { - return false +// Layout of the callback argument block (callback_amd64.s): XMM0-XMM7, then +// RDI, RSI, RDX, RCX, R8, R9, then the caller's stack arguments. +const ( + callbackSSERegs = 8 // XMM0-XMM7 + callbackStackBase = callbackSSERegs + 6 // first stack word + // callbackMaxStackWords bounds the stack words a callback reads, as + // MaxStackBytes bounds them for a call. They lie in the C caller's frame, + // above the trampoline's saved registers; the C caller wrote every one + // the signature declares. + callbackMaxStackWords = archamd64.MaxStackBytes / 8 +) + +// callbackStructClasses classifies a struct argument like the call path does +// (internal/arch/amd64/layout_sysv.go): n eightbytes of the given classes, or +// memory for a struct over 16 bytes. +func callbackStructClasses(t reflect.Type) (sse [2]bool, n int, memory bool) { + sz := t.Size() + if sz > 16 { + return sse, 0, true + } + n = int((sz + 7) / 8) + for i := range n { + start := uintptr(i) * 8 + sse[i] = classifyEightbyte(t, start, min(start+8, sz)) } + return sse, n, false +} - for i := range structType.NumField() { - field := structType.Field(i) - if field.Type.Kind() == reflect.Struct { - if !isStructAllFloats(field.Type) { +// countClasses returns how many of the first n eightbytes are INTEGER and SSE. +func countClasses(sse [2]bool, n int) (ngpr, nsse int) { + for i := range n { + if sse[i] { + nsse++ + } else { + ngpr++ + } + } + return ngpr, nsse +} + +// classifyEightbyte reports whether the eightbyte [startOff, endOff) of a +// struct is SSE class: every scalar field that starts in it, at any nesting +// depth, is float32 or float64 (psABI §3.2.3: the classes of the fields in an +// eightbyte are merged and INTEGER wins). An eightbyte with no field is not SSE. +func classifyEightbyte(structType reflect.Type, startOff, endOff uintptr) bool { + hasField, allFloat := false, true + walkScalarFields(structType, 0, func(off uintptr, k reflect.Kind) bool { + if off < startOff || off >= endOff { + return true + } + hasField = true + if k != reflect.Float32 && k != reflect.Float64 { + allFloat = false + return false + } + return true + }) + return hasField && allFloat +} + +// walkScalarFields calls fn with the offset (from base) and kind of every +// non-struct field of t, nested structs and arrays included, until fn returns +// false. It returns false when fn stopped the walk. +func walkScalarFields(t reflect.Type, base uintptr, fn func(off uintptr, k reflect.Kind) bool) bool { + switch t.Kind() { + case reflect.Struct: + for i := range t.NumField() { + f := t.Field(i) + if !walkScalarFields(f.Type, base+f.Offset, fn) { return false } - } else if field.Type.Kind() != reflect.Float32 && field.Type.Kind() != reflect.Float64 { - return false } + return true + case reflect.Array: + elem := t.Elem() + for i := range t.Len() { + if !walkScalarFields(elem, base+uintptr(i)*elem.Size(), fn) { + return false + } + } + return true + default: + return fn(base, t.Kind()) } - return true } -// classifyEightbyte returns true if all struct fields whose offset falls within -// [startOff, endOff) are SSE types (float or double). -// Returns false if any field in the range is INTEGER class, or if no fields lie in the range. -// -// CAUTION: Does not currently support nested structs. -func classifyEightbyte(structType reflect.Type, startOff, endOff uintptr) bool { - allFloat := true - hasField := false - for i := range structType.NumField() { - field := structType.Field(i) - if field.Offset >= startOff && field.Offset < endOff { - hasField = true - if field.Type.Kind() != reflect.Float32 && field.Type.Kind() != reflect.Float64 { - allFloat = false - break +// callbackPlacement says where one callback argument is in the argument +// block: the frame index of each eightbyte in a register, or the frame index +// of its first stack word. +type callbackPlacement struct { + n int // eightbytes in registers + idx [2]int // frame index of each eightbyte in a register + onStack bool + first int // frame index of the first stack word when onStack +} + +// placeCallbackArg assigns the next argument, of type t, its registers or +// stack words with the System V rules of the call path +// (internal/arch/amd64/layout_sysv.go). +func placeCallbackArg(alloc *archamd64.SysVAllocator, t reflect.Type) callbackPlacement { + var ( + sse [2]bool + n = 1 + memory bool + size uintptr = 8 + align uintptr = 8 + ) + switch t.Kind() { + case reflect.Float32, reflect.Float64: + sse[0] = true + case reflect.Struct: + sse, n, memory = callbackStructClasses(t) + size, align = t.Size(), uintptr(t.Align()) + } + + var p callbackPlacement + if ngpr, nsse := countClasses(sse, n); !memory && alloc.Fits(ngpr, nsse) { + p.n = n + for k := range n { + if sse[k] { + p.idx[k] = alloc.SSE() + } else { + p.idx[k] = callbackSSERegs + alloc.GPR() } } + return p } - return hasField && allFloat + if size > 0 { + p.onStack = true + p.first = callbackStackBase + alloc.Stack(size, align) + } + return p +} + +// callbackStackWords returns how many stack words the arguments of a callback +// of type typ occupy. +func callbackStackWords(typ reflect.Type) int { + var alloc archamd64.SysVAllocator + for i := range typ.NumIn() { + placeCallbackArg(&alloc, typ.In(i)) + } + return int(alloc.StackBytes() / 8) } diff --git a/ffi/callback_struct_args_test.go b/ffi/callback_struct_args_test.go index 82c0201..eae8fc2 100644 --- a/ffi/callback_struct_args_test.go +++ b/ffi/callback_struct_args_test.go @@ -220,46 +220,56 @@ func TestCallbackStruct20B(t *testing.T) { testCallbackStruct(t, frame, expected) } -func TestIsStructAllFloats(t *testing.T) { +// The classes of a struct's eightbytes (true = SSE), nested structs and arrays +// included: psABI §3.2.3 classifies every scalar field by its offset, at any +// depth, and INTEGER wins within an eightbyte. +func TestCallbackStructClasses(t *testing.T) { + type pairF32 struct{ _, _ float32 } // only the layout matters tests := []struct { - name string - struct_ any - expected bool + name string + struct_ any + n int + sse [2]bool + memory bool }{ - {"empty", struct{}{}, false}, - {"float32", struct { - a float32 - }{}, true}, - {"float64", struct { - a float64 - }{}, true}, - {"two floats", struct { + {"empty", struct{}{}, 0, [2]bool{}, false}, + {"float32", struct{ a float32 }{}, 1, [2]bool{true}, false}, + {"float64", struct{ a float64 }{}, 1, [2]bool{true}, false}, + {"float32, float64", struct { a float32 b float64 - }{}, true}, - {"mixed", struct { + }{}, 2, [2]bool{true, true}, false}, + {"float32, int", struct { a float32 b int - }{}, false}, + }{}, 2, [2]bool{true, false}, false}, {"nested, float", struct { a float32 b struct{ a float32 } - }{}, true}, + }{}, 1, [2]bool{true}, false}, {"nested, mixed", struct { a float32 b struct { a float32 - b int + b int32 } - }{}, false}, + }{}, 2, [2]bool{true, false}, false}, + {"two nested float pairs", struct{ a, b pairF32 }{}, 2, [2]bool{true, true}, false}, + {"float array", struct{ a [4]float32 }{}, 2, [2]bool{true, true}, false}, + {"int then nested floats", struct { + a int64 + b pairF32 + }{}, 2, [2]bool{false, true}, false}, + {"over 16 bytes", struct{ a, b, c float64 }{}, 0, [2]bool{}, true}, } for _, test := range tests { t.Run(test.name, func(t *testing.T) { typ := reflect.ValueOf(test.struct_).Type() - isAllFLoats := isStructAllFloats(typ) - if isAllFLoats != test.expected { - t.Fatalf("expected %t, but is %t", test.expected, isAllFLoats) + sse, n, memory := callbackStructClasses(typ) + if n != test.n || sse != test.sse || memory != test.memory { + t.Fatalf("callbackStructClasses = (%v, %d, %t), want (%v, %d, %t)", + sse, n, memory, test.sse, test.n, test.memory) } }) } diff --git a/ffi/cif.go b/ffi/cif.go index 68fcecc..bf5649f 100644 --- a/ffi/cif.go +++ b/ffi/cif.go @@ -268,11 +268,10 @@ func maxSSERegisters(convention types.CallingConvention) int { } // maxStackSlots returns the maximum number of additional stack argument slots -// supported by the platform-specific syscall layer. +// supported by the platform-specific syscall layer. Only architectures +// without an arch.StackLayout use it: Windows amd64 (4 register slots + 9 +// stack slots, internal/arch/amd64/call_windows.go). System V amd64 and arm64 +// compute their exact stack size instead. func maxStackSlots(convention types.CallingConvention) int { - // Our syscall layer supports up to maxArgs (15) total args: - // Unix AMD64: 6 GP registers + 9 stack slots = 15 - // Unix ARM64: 8 GP registers + 7 stack slots = 15 - // Windows: syscall.SyscallN supports up to 15 args natively return 9 } diff --git a/ffi/ffi.go b/ffi/ffi.go index f5bbbee..0c1a2da 100644 --- a/ffi/ffi.go +++ b/ffi/ffi.go @@ -283,9 +283,24 @@ func PrepareVariadicCallInterface( // // Safety: // - All argument pointers must remain valid during the call -// - Hold pointer arguments in unsafe.Pointer variables, never uintptr: a -// uintptr hides the target from escape analysis, so a local it points to -// can stay on the goroutine stack and move before the C function runs +// - Hold pointer arguments in unsafe.Pointer (or *T) variables, never +// uintptr, all the way from where the address is taken to avalue. That +// includes the parameters of your own binding wrappers: a wrapper such as +// func MapMemory(..., ppData uintptr) loses the pointer even when the +// caller writes uintptr(unsafe.Pointer(&local)) in its argument list. The +// compiler keeps a uintptr's target alive and unmoved only in the argument +// list of a call to a function implemented in assembly or marked +// //go:uintptrescapes (package unsafe, rule (4)); CallFunction and your +// wrapper are ordinary Go functions. A local reachable only through a +// uintptr stays on the goroutine stack, the stack can move before the C +// function runs, and C then reads or writes the old address. +// - Memory reachable from rvalue and avalue through pointer-typed values is +// kept alive and in place for the call: CallFunction lets avalue escape, so +// such memory is allocated on the heap, which the GC does not move. +// - If C keeps a pointer to Go memory after the call returns, or reads it +// from a uintptr-typed C struct field or an out-parameter slot that must +// stay uintptr, pin it with runtime.Pinner for as long as C uses it, or +// use memory allocated by C // - Return value buffer must be large enough for the result type // - Use runtime.KeepAlive() if needed to prevent premature GC of arguments func CallFunctionContext( diff --git a/ffi/ffi_test.go b/ffi/ffi_test.go index 056c5f4..6f47e1e 100644 --- a/ffi/ffi_test.go +++ b/ffi/ffi_test.go @@ -917,11 +917,11 @@ func TestFloat32ArgEncoding(t *testing.T) { // // Regression test for TASK-020 / GAP-10. func TestOverflowDetection(t *testing.T) { - // Build a CIF with 20 pointer arguments — far beyond any platform's capacity. - // System V AMD64: 6 GP regs + 9 stack slots = 15 max GP args. + // Build a CIF with 16384 pointer arguments — beyond every platform's capacity. + // System V AMD64: 6 GP regs + stack words up to amd64.MaxStackBytes (8192 words). // ARM64: 8 GP regs + 7 stack slots = 15 max GP args. - // Windows: syscall.SyscallN supports up to 15 args. - const tooMany = 20 + // Windows: 4 register slots + 9 stack slots. + const tooMany = 1 << 14 argTypes := make([]*types.TypeDescriptor, tooMany) for k := range argTypes { @@ -938,9 +938,9 @@ func TestOverflowDetection(t *testing.T) { cif := &types.CallInterface{} err := PrepareCallInterface(cif, convention, types.VoidTypeDescriptor, argTypes) if err == nil { - t.Error("PrepareCallInterface with 20 args should return error, got nil") + t.Errorf("PrepareCallInterface with %d args should return error, got nil", tooMany) } else { - t.Logf("Correctly rejected 20 args: %v", err) + t.Logf("Correctly rejected %d args: %v", tooMany, err) } } diff --git a/ffi/pointer_contract_e2e_test.go b/ffi/pointer_contract_e2e_test.go new file mode 100644 index 0000000..513f88e --- /dev/null +++ b/ffi/pointer_contract_e2e_test.go @@ -0,0 +1,58 @@ +// SPDX-License-Identifier: MIT +// SPDX-FileCopyrightText: 2026 Andrey Kolkov and GoGPU Contributors + +//go:build ((linux && !android) || darwin || freebsd) && (amd64 || arm64) + +package ffi + +import ( + "testing" + "unsafe" + + "github.com/go-webgpu/goffi/types" +) + +// TestOutPointerSurvivesStackMove checks the contract of CallFunctionContext's +// documentation: memory reachable from avalue through pointer-typed values +// stays in place for the call. C calls back into Go, the callback grows (and +// so moves) the goroutine stack, then C writes through the out-pointer to the +// caller's local. Held as an unsafe.Pointer, the local is heap memory and +// receives the value. +// +// There is no negative control with a uintptr: then the local can stay on the +// stack and C writes into the old, freed stack. That corrupts memory instead +// of failing cleanly, so it cannot be a test. +func TestOutPointerSurvivesStackMove(t *testing.T) { + requireStructLib(t) + sym, err := GetSymbol(structTestLib, "call_then_store") + if err != nil { + t.Fatal(err) + } + var cif types.CallInterface + if err := PrepareCallInterface(&cif, types.DefaultCall, types.VoidTypeDescriptor, + []*types.TypeDescriptor{types.PointerTypeDescriptor, types.PointerTypeDescriptor, types.SInt64TypeDescriptor}, + ); err != nil { + t.Fatal(err) + } + cb := NewCallback(func() { + // About 2 MiB of stack growth, enough to force a copystack mid-call. + growStack(512) + }) + + done := make(chan int64, 1) + go func() { + // A fresh goroutine starts with a small stack, so the growth moves it. + var out int64 + outPtr := unsafe.Pointer(&out) + v := int64(0x5eed_cafe) + if _, err := CallFunction(&cif, sym, nil, + []unsafe.Pointer{unsafe.Pointer(&cb), unsafe.Pointer(&outPtr), unsafe.Pointer(&v)}); err != nil { + t.Error(err) + } + done <- out + }() + + if got := <-done; got != 0x5eed_cafe { + t.Fatalf("out = %#x after the call, want 0x5eedcafe (C wrote to a moved address)", got) + } +} diff --git a/ffi/stack_layout_e2e_test.go b/ffi/stack_layout_e2e_test.go new file mode 100644 index 0000000..7d89f22 --- /dev/null +++ b/ffi/stack_layout_e2e_test.go @@ -0,0 +1,438 @@ +// SPDX-License-Identifier: MIT +// SPDX-FileCopyrightText: 2026 Andrey Kolkov and GoGPU Contributors + +//go:build (linux || darwin || freebsd || windows) && (amd64 || arm64) + +package ffi + +import ( + "errors" + "runtime" + "testing" + "unsafe" + + "github.com/go-webgpu/goffi/types" +) + +// Stack-argument layout (TASK-098): by-value structs that need more stack than +// a fixed area holds, and arguments that must go to the stack whole because +// the registers left cannot take all of their eightbytes (psABI §3.2.3, libffi +// src/x86/ffi64.c). The C functions in testdata/structtest.c weight every +// field and argument, so a misplaced word changes the result. + +var ( + u64Type = types.UInt64TypeDescriptor + u32Type = types.UInt32TypeDescriptor + + // svType is webgpu.h's WGPUStringView: {data, length}, 16 bytes. + svType = &types.TypeDescriptor{ + Kind: types.StructType, Size: 16, Alignment: 8, + Members: []*types.TypeDescriptor{u64Type, u64Type}, + } + // info96Type is the shape of webgpu.h's WGPUAdapterInfo. + info96Type = &types.TypeDescriptor{ + Kind: types.StructType, Size: 96, Alignment: 8, + Members: []*types.TypeDescriptor{ + u64Type, svType, svType, svType, svType, + u32Type, u32Type, u32Type, u32Type, u32Type, u32Type, + }, + } + words8Type = &types.TypeDescriptor{ + Kind: types.StructType, Size: 64, Alignment: 8, + Members: []*types.TypeDescriptor{u64Type, u64Type, u64Type, u64Type, u64Type, u64Type, u64Type, u64Type}, + } + cbinfo40Type = &types.TypeDescriptor{ + Kind: types.StructType, Size: 40, Alignment: 8, + Members: []*types.TypeDescriptor{u64Type, u32Type, u64Type, u64Type, u64Type}, + } + pairI64Type = &types.TypeDescriptor{ + Kind: types.StructType, Size: 16, Alignment: 8, + Members: []*types.TypeDescriptor{types.SInt64TypeDescriptor, types.SInt64TypeDescriptor}, + } + pairF64Type = &types.TypeDescriptor{ + Kind: types.StructType, Size: 16, Alignment: 8, + Members: []*types.TypeDescriptor{types.DoubleTypeDescriptor, types.DoubleTypeDescriptor}, + } + mixedIntFloatType = &types.TypeDescriptor{ + Kind: types.StructType, Size: 16, Alignment: 8, + Members: []*types.TypeDescriptor{types.SInt64TypeDescriptor, types.DoubleTypeDescriptor}, + } +) + +type stringView struct{ Data, Length uint64 } + +type adapterInfo96 struct { + Next uint64 + Vendor, Architecture, Device, Description stringView + Backend, AdapterType, VendorID, DeviceID uint32 + SubgroupMin, SubgroupMax uint32 +} + +// sum mirrors info96_sum in structtest.c. +func (s *adapterInfo96) sum() uint64 { + return s.Next + 2*s.Vendor.Data + 3*s.Vendor.Length + 4*s.Architecture.Data + + 5*s.Architecture.Length + 6*s.Device.Data + 7*s.Device.Length + + 8*s.Description.Data + 9*s.Description.Length + 10*uint64(s.Backend) + + 11*uint64(s.AdapterType) + 12*uint64(s.VendorID) + 13*uint64(s.DeviceID) + + 14*uint64(s.SubgroupMin) + 15*uint64(s.SubgroupMax) +} + +// newAdapterInfo96 fills every field with a distinct value derived from base. +func newAdapterInfo96(base uint64) adapterInfo96 { + return adapterInfo96{ + Next: base + 1, + Vendor: stringView{base + 2, base + 3}, + Architecture: stringView{base + 4, base + 5}, + Device: stringView{base + 6, base + 7}, + Description: stringView{base + 8, base + 9}, + Backend: uint32(base + 10), AdapterType: uint32(base + 11), + VendorID: uint32(base + 12), DeviceID: uint32(base + 13), + SubgroupMin: uint32(base + 14), SubgroupMax: uint32(base + 15), + } +} + +func TestStackLayout_Struct96(t *testing.T) { + if unsafe.Sizeof(adapterInfo96{}) != 96 { + t.Fatalf("adapterInfo96 is %d bytes, want 96", unsafe.Sizeof(adapterInfo96{})) + } + s := newAdapterInfo96(100) + var got uint64 + if err := callStackFunc(t, "take_struct_96", u64Type, unsafe.Pointer(&got), + []*types.TypeDescriptor{info96Type}, []unsafe.Pointer{unsafe.Pointer(&s)}); err != nil { + t.Fatal(err) + } + if want := s.sum(); got != want { + t.Errorf("take_struct_96 = %d, want %d", got, want) + } +} + +func TestStackLayout_Struct64(t *testing.T) { + var s [8]uint64 + var want uint64 + for i := range s { + s[i] = uint64(i+1) * 1000 + want += uint64(i+1) * s[i] + } + var got uint64 + if err := callStackFunc(t, "take_struct_64", u64Type, unsafe.Pointer(&got), + []*types.TypeDescriptor{words8Type}, []unsafe.Pointer{unsafe.Pointer(&s)}); err != nil { + t.Fatal(err) + } + if got != want { + t.Errorf("take_struct_64 = %d, want %d", got, want) + } +} + +// The shape of wgpuBufferMapAsync: four integer arguments, then a 40-byte +// struct by value. +func TestStackLayout_MapAsyncShape(t *testing.T) { + type cbinfo40 struct { + Next uint64 + Mode uint32 + Callback uint64 + UD1, UD2 uint64 + } + buffer, mode, offset, size := uint64(3), uint32(5), uint64(7), uint64(11) + info := cbinfo40{Next: 13, Mode: 17, Callback: 19, UD1: 23, UD2: 29} + want := buffer + 2*uint64(mode) + 3*offset + 5*size + 7*info.Next + 11*uint64(info.Mode) + + 13*info.Callback + 17*info.UD1 + 19*info.UD2 + var got uint64 + if err := callStackFunc(t, "map_async_like", u64Type, unsafe.Pointer(&got), + []*types.TypeDescriptor{u64Type, u32Type, u64Type, u64Type, cbinfo40Type}, + []unsafe.Pointer{unsafe.Pointer(&buffer), unsafe.Pointer(&mode), unsafe.Pointer(&offset), + unsafe.Pointer(&size), unsafe.Pointer(&info)}); err != nil { + t.Fatal(err) + } + if got != want { + t.Errorf("map_async_like = %d, want %d", got, want) + } +} + +// Two 96-byte structs and two scalars: 24 stack words on SysV amd64. +func TestStackLayout_TwoStruct96(t *testing.T) { + a, b := int64(3), int64(5) + x, y := newAdapterInfo96(100), newAdapterInfo96(200) + want := uint64(a) + 1000*x.sum() + 7*uint64(b) + 100000*y.sum() + var got uint64 + if err := callStackFunc(t, "two_info96", u64Type, unsafe.Pointer(&got), + []*types.TypeDescriptor{types.SInt64TypeDescriptor, info96Type, types.SInt64TypeDescriptor, info96Type}, + []unsafe.Pointer{unsafe.Pointer(&a), unsafe.Pointer(&x), unsafe.Pointer(&b), unsafe.Pointer(&y)}); err != nil { + t.Fatal(err) + } + if got != want { + t.Errorf("two_info96 = %d, want %d", got, want) + } +} + +// Five int64 leave one GPR: the 16-byte struct goes on the stack whole and the +// int64 after it takes R9 (System V). On arm64 and Win64 the same signature +// has registers to spare or passes the struct by reference. +func TestStackLayout_SplitS16After5I(t *testing.T) { + var regs [5]int64 + args := make([]unsafe.Pointer, 0, 7) + var want int64 + for i := range regs { + regs[i] = int64(i + 1) + args = append(args, unsafe.Pointer(®s[i])) + want += int64(i+1) * regs[i] + } + type pairI64 struct{ A, B int64 } + s := pairI64{A: 42, B: 7} + last := int64(9) + args = append(args, unsafe.Pointer(&s), unsafe.Pointer(&last)) + want += 1000*s.A + 100*s.B + 100000*last + + argTypes := append(repeatType(types.SInt64TypeDescriptor, 5), pairI64Type, types.SInt64TypeDescriptor) + var got int64 + if err := callStackFunc(t, "split_s16_after_5i", types.SInt64TypeDescriptor, unsafe.Pointer(&got), + argTypes, args); err != nil { + t.Fatal(err) + } + if got != want { + t.Errorf("split_s16_after_5i = %d, want %d", got, want) + } +} + +// Seven doubles leave XMM7: {double, double} goes on the stack whole and the +// double after it takes XMM7 (System V). +func TestStackLayout_SplitD2After7D(t *testing.T) { + var regs [7]float64 + args := make([]unsafe.Pointer, 0, 9) + var want float64 + for i := range regs { + regs[i] = float64(i + 1) + args = append(args, unsafe.Pointer(®s[i])) + want += float64(i+1) * regs[i] + } + type pairF64 struct{ A, B float64 } + s := pairF64{A: 42, B: 7} + last := 9.0 + args = append(args, unsafe.Pointer(&s), unsafe.Pointer(&last)) + want += 1000*s.A + 100*s.B + 100000*last + + argTypes := append(repeatType(types.DoubleTypeDescriptor, 7), pairF64Type, types.DoubleTypeDescriptor) + var got float64 + if err := callStackFunc(t, "split_d2_after_7d", types.DoubleTypeDescriptor, unsafe.Pointer(&got), + argTypes, args); err != nil { + t.Fatal(err) + } + if got != want { + t.Errorf("split_d2_after_7d = %v, want %v", got, want) + } +} + +// Six int64 use every GPR: {int64, double} needs a GPR and an SSE register, so +// it goes on the stack whole and the double after it takes XMM0 (System V). +func TestStackLayout_SplitMixedAfter6I(t *testing.T) { + var regs [6]int64 + args := make([]unsafe.Pointer, 0, 8) + var wantInts int64 + for i := range regs { + regs[i] = int64(i + 1) + args = append(args, unsafe.Pointer(®s[i])) + wantInts += int64(i+1) * regs[i] + } + type mixed struct { + A int64 + B float64 + } + s := mixed{A: 42, B: 7} + last := 9.0 + args = append(args, unsafe.Pointer(&s), unsafe.Pointer(&last)) + want := float64(wantInts) + 1000*float64(s.A) + 100*s.B + 100000*last + + argTypes := append(repeatType(types.SInt64TypeDescriptor, 6), mixedIntFloatType, types.DoubleTypeDescriptor) + var got float64 + if err := callStackFunc(t, "split_mixed_after_6i", types.DoubleTypeDescriptor, unsafe.Pointer(&got), + argTypes, args); err != nil { + t.Fatal(err) + } + if got != want { + t.Errorf("split_mixed_after_6i = %v, want %v", got, want) + } +} + +// A CallInterface filled in by hand skips PrepareCallInterface, so the call +// itself must place every argument or return an error, never panic. +func TestStackLayout_HandBuiltCIF(t *testing.T) { + requireStructLib(t) + + t.Run("sixteen int64", func(t *testing.T) { + sym, err := GetSymbol(structTestLib, "stack_i64x16") + if err != nil { + t.Fatal(err) + } + var in [16]int64 + args := make([]unsafe.Pointer, len(in)) + var want int64 + for i := range in { + in[i] = int64(i + 1) + args[i] = unsafe.Pointer(&in[i]) + want += in[i] + } + want += 15 * in[15] + cif := types.CallInterface{ReturnType: types.SInt64TypeDescriptor, + ArgTypes: repeatType(types.SInt64TypeDescriptor, 16)} + var got int64 + if _, err := CallFunction(&cif, sym, unsafe.Pointer(&got), args); err != nil { + t.Logf("rejected: %v", err) + return + } + if got != want { + t.Errorf("stack_i64x16 = %d without an error, want %d or an error", got, want) + } + }) + + t.Run("96-byte struct", func(t *testing.T) { + sym, err := GetSymbol(structTestLib, "take_struct_96") + if err != nil { + t.Fatal(err) + } + s := newAdapterInfo96(300) + cif := types.CallInterface{ReturnType: u64Type, ArgTypes: []*types.TypeDescriptor{info96Type}} + var got uint64 + if _, err := CallFunction(&cif, sym, unsafe.Pointer(&got), []unsafe.Pointer{unsafe.Pointer(&s)}); err != nil { + t.Fatal(err) + } + if want := s.sum(); got != want { + t.Errorf("take_struct_96 = %d, want %d", got, want) + } + }) +} + +// Arguments past every platform's limit are an error at prepare time and, for +// a hand-built CallInterface, at call time. The function is never called. +func TestStackLayout_TooManyWordsIsAnError(t *testing.T) { + requireStructLib(t) + sym, err := GetSymbol(structTestLib, "stack_i64x16") + if err != nil { + t.Fatal(err) + } + const n = 1 << 14 // 16384 int64: far more stack than any call path allows + argTypes := repeatType(types.SInt64TypeDescriptor, n) + var v int64 + args := make([]unsafe.Pointer, n) + for i := range args { + args[i] = unsafe.Pointer(&v) + } + + var cif types.CallInterface + err = PrepareCallInterface(&cif, types.DefaultCall, types.SInt64TypeDescriptor, argTypes) + if !errors.Is(err, ErrTooManyArguments) { + t.Errorf("PrepareCallInterface with %d arguments: err = %v, want ErrTooManyArguments", n, err) + } + + hand := types.CallInterface{ReturnType: types.SInt64TypeDescriptor, ArgTypes: argTypes} + var got int64 + if _, err := CallFunction(&hand, sym, unsafe.Pointer(&got), args); err == nil { + t.Errorf("CallFunction with %d arguments on a hand-built CallInterface returned no error", n) + } +} + +// Callbacks decode the same layout: C passes the struct on the stack whole and +// the next argument in the register that was left. +func TestCallbackStackLayout(t *testing.T) { + if runtime.GOOS == "windows" || runtime.GOARCH == "arm64" { + t.Skip("callback struct args not supported on Windows/ARM64") + } + requireStructLib(t) + + callPtr := func(t *testing.T, name string, ret *types.TypeDescriptor, rvalue unsafe.Pointer, cb uintptr) { + t.Helper() + if err := callStackFunc(t, name, ret, rvalue, []*types.TypeDescriptor{types.PointerTypeDescriptor}, + []unsafe.Pointer{unsafe.Pointer(&cb)}); err != nil { + t.Fatal(err) + } + } + + t.Run("split s16 after 5 int64", func(t *testing.T) { + type pairI64 struct{ A, B int64 } + cb := NewCallback(func(r0, r1, r2, r3, r4 int64, s pairI64, last int64) int64 { + return r0 + 2*r1 + 3*r2 + 4*r3 + 5*r4 + 1000*s.A + 100*s.B + 100000*last + }) + want := int64(1+4+9+16+25) + 1000*42 + 100*7 + 100000*9 + var got int64 + callPtr(t, "cb_call_split_s16", types.SInt64TypeDescriptor, unsafe.Pointer(&got), cb) + if got != want { + t.Errorf("callback result = %d, want %d", got, want) + } + }) + + t.Run("split {double,double} after 7 doubles", func(t *testing.T) { + type pairF64 struct{ A, B float64 } + cb := NewCallback(func(d0, d1, d2, d3, d4, d5, d6 float64, s pairF64, last float64) float64 { + return d0 + 2*d1 + 3*d2 + 4*d3 + 5*d4 + 6*d5 + 7*d6 + 1000*s.A + 100*s.B + 100000*last + }) + want := float64(1+4+9+16+25+36+49) + 1000*42 + 100*7 + 100000*9 + var got float64 + callPtr(t, "cb_call_split_d2", types.DoubleTypeDescriptor, unsafe.Pointer(&got), cb) + if got != want { + t.Errorf("callback result = %v, want %v", got, want) + } + }) + + t.Run("96-byte struct", func(t *testing.T) { + cb := NewCallback(func(s adapterInfo96) uint64 { return s.sum() }) + want := newAdapterInfo96(0) + var got uint64 + callPtr(t, "cb_call_info96", u64Type, unsafe.Pointer(&got), cb) + if got != want.sum() { + t.Errorf("callback result = %d, want %d", got, want.sum()) + } + }) + + t.Run("nested float pairs", func(t *testing.T) { + cb := NewCallback(func(s quadF32) float32 { return s.A.X + 2*s.A.Y + 3*s.B.X + 4*s.B.Y }) + var got float32 + callPtr(t, "cb_call_nested_f32x4", types.FloatTypeDescriptor, unsafe.Pointer(&got), cb) + if want := float32(1 + 4 + 9 + 16); got != want { + t.Errorf("callback result = %v, want %v", got, want) + } + }) +} + +type pairF32 struct{ X, Y float32 } +type quadF32 struct{ A, B pairF32 } + +// quadF32Type describes struct quad_f32 {struct pair_f32 a, b;} with nested +// member descriptors, as a binding generator emits them. +func quadF32Type() *types.TypeDescriptor { + pair := &types.TypeDescriptor{ + Kind: types.StructType, Size: 8, Alignment: 4, + Members: []*types.TypeDescriptor{types.FloatTypeDescriptor, types.FloatTypeDescriptor}, + } + return &types.TypeDescriptor{ + Kind: types.StructType, Size: 16, Alignment: 4, + Members: []*types.TypeDescriptor{pair, pair}, + } +} + +// Nested structs are classified by their scalar members: two {float, float} +// pairs are two SSE eightbytes on System V amd64, not INTEGER. +func TestStackLayout_NestedFloatStruct(t *testing.T) { + t.Run("argument", func(t *testing.T) { + s := quadF32{A: pairF32{1.5, 2.5}, B: pairF32{3.5, 4.5}} + var got float32 + if err := callStackFunc(t, "take_nested_f32x4", types.FloatTypeDescriptor, unsafe.Pointer(&got), + []*types.TypeDescriptor{quadF32Type()}, []unsafe.Pointer{unsafe.Pointer(&s)}); err != nil { + t.Fatal(err) + } + if want := s.A.X + 2*s.A.Y + 3*s.B.X + 4*s.B.Y; got != want { + t.Errorf("take_nested_f32x4 = %v, want %v", got, want) + } + }) + + t.Run("result", func(t *testing.T) { + in := [4]float32{1.5, 2.5, 3.5, 4.5} + args := []unsafe.Pointer{unsafe.Pointer(&in[0]), unsafe.Pointer(&in[1]), + unsafe.Pointer(&in[2]), unsafe.Pointer(&in[3])} + var got quadF32 + if err := callStackFunc(t, "return_nested_f32x4", quadF32Type(), unsafe.Pointer(&got), + repeatType(types.FloatTypeDescriptor, 4), args); err != nil { + t.Fatal(err) + } + if want := (quadF32{A: pairF32{1.5, 2.5}, B: pairF32{3.5, 4.5}}); got != want { + t.Errorf("return_nested_f32x4 = %+v, want %+v", got, want) + } + }) +} diff --git a/ffi/testdata/structtest.c b/ffi/testdata/structtest.c index ac53e21..2fe0c71 100644 --- a/ffi/testdata/structtest.c +++ b/ffi/testdata/structtest.c @@ -340,3 +340,116 @@ struct three_f32 return_struct_3floats(float a, float b, float c) { struct f32_i32 { float a; int32_t b; }; struct f32_i32 return_struct_float_int32(float a, int32_t b) { struct f32_i32 s = {a, b}; return s; } + +// Stack-argument layout (TASK-098). All targets are 64-bit, so uint64_t +// stands in for pointers and size_t; C never dereferences these values. + +// {data, length}: webgpu.h's WGPUStringView, 16 bytes, two INTEGER eightbytes. +struct sv { uint64_t data; uint64_t length; }; + +// 96 bytes, the shape of webgpu.h's WGPUAdapterInfo. SysV amd64 copies it onto +// the stack (12 words, MEMORY class); arm64 and Win64 pass a pointer to a copy. +struct info96 { + uint64_t next; + struct sv vendor, architecture, device, description; + uint32_t backend, adapter_type, vendor_id, device_id, sub_min, sub_max; +}; +static uint64_t info96_sum(const struct info96 *s) { + return s->next + 2 * s->vendor.data + 3 * s->vendor.length + 4 * s->architecture.data + + 5 * s->architecture.length + 6 * s->device.data + 7 * s->device.length + + 8 * s->description.data + 9 * s->description.length + 10 * (uint64_t)s->backend + + 11 * (uint64_t)s->adapter_type + 12 * (uint64_t)s->vendor_id + 13 * (uint64_t)s->device_id + + 14 * (uint64_t)s->sub_min + 15 * (uint64_t)s->sub_max; +} +uint64_t take_struct_96(struct info96 s) { return info96_sum(&s); } + +// 64 bytes, as webgpu.h's WGPUSurfaceCapabilities: 8 stack words on SysV amd64. +struct words8 { uint64_t w[8]; }; +uint64_t take_struct_64(struct words8 s) { + uint64_t sum = 0; + for (int i = 0; i < 8; i++) sum += (uint64_t)(i + 1) * s.w[i]; + return sum; +} + +// 40 bytes, as webgpu.h's WGPUBufferMapCallbackInfo, after four integer +// arguments: the shape of wgpuBufferMapAsync. +struct cbinfo40 { uint64_t next; uint32_t mode; uint64_t callback, userdata1, userdata2; }; +uint64_t map_async_like(uint64_t buffer, uint32_t mode, uint64_t offset, uint64_t size, + struct cbinfo40 info) { + return buffer + 2 * (uint64_t)mode + 3 * offset + 5 * size + 7 * info.next + + 11 * (uint64_t)info.mode + 13 * info.callback + 17 * info.userdata1 + 19 * info.userdata2; +} + +// Two 96-byte structs and scalars: 24 stack words on SysV amd64. +uint64_t two_info96(int64_t a, struct info96 x, int64_t b, struct info96 y) { + return (uint64_t)a + 1000 * info96_sum(&x) + 7 * (uint64_t)b + 100000 * info96_sum(&y); +} + +// A 16-byte argument that does not fit in the registers left goes to the stack +// whole, and the registers stay free for later arguments (psABI §3.2.3; libffi +// src/x86/ffi64.c). Five int64 leave R9; s needs two GPRs, so s goes on the +// stack and last takes R9. +int64_t split_s16_after_5i(int64_t r0, int64_t r1, int64_t r2, int64_t r3, int64_t r4, + struct pair_i64 s, int64_t last) { + return r0 + 2 * r1 + 3 * r2 + 4 * r3 + 5 * r4 + 1000 * s.a + 100 * s.b + 100000 * last; +} + +// The same for SSE: seven doubles leave XMM7; {double, double} goes on the +// stack and last takes XMM7. +double split_d2_after_7d(double d0, double d1, double d2, double d3, double d4, double d5, + double d6, struct pair_f64 s, double last) { + return d0 + 2 * d1 + 3 * d2 + 4 * d3 + 5 * d4 + 6 * d5 + 7 * d6 + 1000 * s.a + 100 * s.b + + 100000 * last; +} + +// {int64, double} needs one GPR and one SSE register. Six int64 use every GPR, +// so s goes on the stack whole and last takes XMM0. +double split_mixed_after_6i(int64_t r0, int64_t r1, int64_t r2, int64_t r3, int64_t r4, + int64_t r5, struct mixed_int_float s, double last) { + return (double)(r0 + 2 * r1 + 3 * r2 + 4 * r3 + 5 * r4 + 6 * r5) + 1000 * (double)s.a + + 100 * s.b + 100000 * last; +} + +// Callbacks with the same shapes: C calls the Go callback. +int64_t cb_call_split_s16(int64_t (*f)(int64_t, int64_t, int64_t, int64_t, int64_t, + struct pair_i64, int64_t)) { + struct pair_i64 s = {.a = 42, .b = 7}; + return f(1, 2, 3, 4, 5, s, 9); +} +double cb_call_split_d2(double (*f)(double, double, double, double, double, double, double, + struct pair_f64, double)) { + struct pair_f64 s = {.a = 42, .b = 7}; + return f(1, 2, 3, 4, 5, 6, 7, s, 9); +} +uint64_t cb_call_info96(uint64_t (*f)(struct info96)) { + struct info96 s = { + .next = 1, .vendor = {2, 3}, .architecture = {4, 5}, .device = {6, 7}, + .description = {8, 9}, .backend = 10, .adapter_type = 11, .vendor_id = 12, + .device_id = 13, .sub_min = 14, .sub_max = 15, + }; + return f(s); +} + +// Nested structs are classified by their scalar members (psABI §3.2.3): two +// {float, float} pairs make two SSE eightbytes on SysV amd64 (XMM0, XMM1), a +// four-float HFA on arm64, a pointer to a copy on Win64. +struct quad_f32 { struct pair_f32 a, b; }; +float take_nested_f32x4(struct quad_f32 s) { + return s.a.x + 2 * s.a.y + 3 * s.b.x + 4 * s.b.y; +} +struct quad_f32 return_nested_f32x4(float a, float b, float c, float d) { + struct quad_f32 s = {{a, b}, {c, d}}; + return s; +} +float cb_call_nested_f32x4(float (*f)(struct quad_f32)) { + struct quad_f32 s = {{1, 2}, {3, 4}}; + return f(s); +} + +// An out-pointer written after a callback into Go (the vkMapMemory shape: the +// driver writes *ppData). The Go callback makes the goroutine stack move, so a +// Go local passed as out must not be on that stack. +void call_then_store(void (*cb)(void), int64_t *out, int64_t v) { + cb(); + *out = v; +} diff --git a/internal/arch/amd64/call_unix.go b/internal/arch/amd64/call_unix.go index 46dae94..1aced08 100644 --- a/internal/arch/amd64/call_unix.go +++ b/internal/arch/amd64/call_unix.go @@ -1,7 +1,7 @@ //go:build amd64 && (linux || darwin || freebsd) // Unix implementation using System V AMD64 ABI (Linux, macOS, FreeBSD, etc.) -// This implementation closely follows purego's proven approach but is OUR OWN code. +// The argument layout is layout_sysv.go; this file places the values. package amd64 @@ -23,9 +23,8 @@ func loadWord(p unsafe.Pointer, n uintptr) uintptr { return w } -// maxTotalArgs is the maximum number of GP + stack argument slots supported. -// Matches purego's maxArgs = 15. -const maxTotalArgs = 15 +// smallStackWords is how many stack words a call places without allocating. +const smallStackWords = 16 func (i *Implementation) Execute( cif *types.CallInterface, @@ -34,152 +33,79 @@ func (i *Implementation) Execute( avalue []unsafe.Pointer, errnoFn uintptr, ) (cerrno uintptr, err error) { - // System V AMD64 ABI: - // - GP registers: RDI, RSI, RDX, RCX, R8, R9 (6 registers, indices 0-5) - // - SSE registers: XMM0-XMM7 (8 registers) - // - Stack args: additional GP/integer args beyond register count - // - // In our syscallArgs layout: - // sysargs[0..5] -> RDI..R9 (6 GP registers) - // sysargs[6..14] -> stack slots (pushed before CALL) - var sysargs [maxTotalArgs]uintptr - var floats [8]uintptr - - numInts := 0 // GP register index (0-5 = registers, 6+ = stack) - numFloats := 0 // SSE register index (0-7) - numStack := 0 // stack slot index - - addInt := func(x uintptr) { - const maxGPRegs = 6 - if numInts < maxGPRegs { - sysargs[numInts] = x - numInts++ - } else { - // Overflow to stack: placed after the 6 register slots - sysargs[maxGPRegs+numStack] = x - numStack++ - } + // The stack area is sized before any value is placed, so an argument list + // that does not fit is an error, never a write past the area. A + // CallInterface filled in by hand reaches this point without the check in + // PrepareCallInterface. + stackBytes := sysvStackBytes(cif) + if stackBytes > MaxStackBytes { + return 0, fmt.Errorf("goffi: arguments need %d bytes of stack, the amd64 call path supports %d", + stackBytes, MaxStackBytes) } - - addStack := func(x uintptr) { - const maxGPRegs = 6 - sysargs[maxGPRegs+numStack] = x - numStack++ + var ( + gpr [sysvGPRs]uintptr + sse [sysvSSEs]float64 + small [smallStackWords]uintptr + alloc SysVAllocator + ) + var stack []uintptr + if n := int(stackBytes / 8); n > len(small) { + stack = make([]uintptr, n) + } else { + stack = small[:n] } - - addFloat := func(x uintptr) { - if numFloats < 8 { - floats[numFloats] = x - numFloats++ - } else { - // Float overflow to stack (each float occupies one 8-byte stack slot) - const maxGPRegs = 6 - sysargs[maxGPRegs+numStack] = x - numStack++ - } + setSSE := func(reg int, bits uintptr) { + *(*uintptr)(unsafe.Pointer(&sse[reg])) = bits } - // Detect sret: struct > 16 bytes requires hidden first argument in RDI. - // The caller's rvalue buffer is passed as the first integer argument and - // callee writes the return value directly into it. - sret := cif.ReturnType.Kind == types.StructType && cif.ReturnType.Size > 16 + // A struct result over 16 bytes is returned through caller memory whose + // address is the hidden first argument in RDI; the callee writes the + // result there directly. + sret := sysvRetInMemory(cif) if sret { - addInt(uintptr(rvalue)) + gpr[alloc.GPR()] = uintptr(rvalue) } - // Map arguments to registers or stack. ffi.CallFunctionContext has checked - // that avalue holds a value for every argument type. + // ffi.CallFunctionContext has checked that avalue holds a value for every + // argument type. for idx, argType := range cif.ArgTypes { - switch argType.Kind { - case types.FloatType: - // Use math.Float32bits to preserve exact bit pattern in XMM register. - // Widening float32 → float64 corrupts the lower 32 bits read by callee. - addFloat(uintptr(math.Float32bits(*(*float32)(avalue[idx])))) - case types.DoubleType: - addFloat(*(*uintptr)(avalue[idx])) - case types.PointerType: - addInt(*(*uintptr)(avalue[idx])) - // Narrow integers are sign- or zero-extended to the full register, as - // libffi does (src/x86/ffi64.c, ffi_call_int). The psABI leaves the upper - // bits unspecified, but clang-built callees rely on the caller extending - // char and short to 32 bits: Apple clang compiles - // `int64_t f(int8_t a) { return a; }` to `movslq %edi, %rax`. - case types.SInt8Type: - addInt(uintptr(int64(*(*int8)(avalue[idx])))) - case types.UInt8Type: - addInt(uintptr(*(*uint8)(avalue[idx]))) - case types.SInt16Type: - addInt(uintptr(int64(*(*int16)(avalue[idx])))) - case types.UInt16Type: - addInt(uintptr(*(*uint16)(avalue[idx]))) - case types.SInt32Type, types.IntType: - addInt(uintptr(int64(*(*int32)(avalue[idx])))) - case types.UInt32Type: - addInt(uintptr(*(*uint32)(avalue[idx]))) - case types.SInt64Type, types.UInt64Type: - addInt(uintptr(*(*uint64)(avalue[idx]))) - case types.StructType: - argPtr := avalue[idx] - sz := argType.Size - switch { - case sz == 0: - // Zero-size struct: pass nothing. - case sz <= 8: - // Single eightbyte: INTEGER if any member is not float/double, else SSE. - if isStructAllFloats(argType) { - addFloat(loadWord(argPtr, sz)) - } else { - addInt(loadWord(argPtr, sz)) - } - case sz <= 16: - // Two eightbytes: classify each independently. - // System V ABI §3.2.3: INTEGER wins over SSE within an eightbyte. - if classifyEightbyte(argType, 0, 8) { - addFloat(loadWord(argPtr, 8)) - } else { - addInt(loadWord(argPtr, 8)) + val := avalue[idx] + p := alloc.place(argType) + + if argType.Kind == types.StructType { + if p.onStack { + sz := argType.Size + for off := uintptr(0); off < sz; off += 8 { + stack[p.word+int(off/8)] = loadWord(unsafe.Add(val, off), min(8, sz-off)) } - second := loadWord(unsafe.Add(argPtr, 8), sz-8) - if classifyEightbyte(argType, 8, sz) { - addFloat(second) + continue + } + for k := range p.n { + off := uintptr(k) * 8 + w := loadWord(unsafe.Add(val, off), min(8, argType.Size-off)) + if p.sse[k] { + setSSE(p.reg[k], w) } else { - addInt(second) - } - default: - // MEMORY class (> 16 bytes): copy onto stack in 8-byte chunks. - // Per SysV ABI §3.2.3: MEMORY class structs bypass registers entirely. - for off := uintptr(0); off < sz; off += 8 { - addStack(loadWord(unsafe.Add(argPtr, off), min(8, sz-off))) + gpr[p.reg[k]] = w } } - default: - // For unknown/composite types, pass as pointer-to-value - addInt(uintptr(avalue[idx])) + continue } - } - - // Validate we haven't exceeded platform maximum - if numStack > maxTotalArgs-6 { - return 0, fmt.Errorf("goffi: %d stack arguments exceed platform limit of %d", numStack, maxTotalArgs-6) - } - - // Build GP register array (first 6 slots) - var gpr [6]uintptr - copy(gpr[:], sysargs[:6]) - // Build SSE array as float64 bit-patterns - var sse [8]float64 - for k := range floats { - sse[k] = *(*float64)(unsafe.Pointer(&floats[k])) + w := scalarWord(argType, val) + switch { + case p.onStack: + stack[p.word] = w + case p.sse[0]: + setSSE(p.reg[0], w) + default: + gpr[p.reg[0]] = w + } } - // Build stack slots (sysargs[6..14]) - var stackArgs [9]uintptr - copy(stackArgs[:], sysargs[6:]) - // Call via syscall; errnoFn is non-zero on Unix, 0 on Windows. // When errnoFn is 0, the assembly skips errno capture (TESTQ/JZ). - ret, r2, fret, fret2, capturedErrno := gosyscall.CallNFloatErrno(uintptr(fn), gpr, sse, stackArgs, numStack, errnoFn) + ret, r2, fret, fret2, capturedErrno := gosyscall.CallNFloatErrno(uintptr(fn), gpr, sse, stack, errnoFn) runtime.KeepAlive(avalue) runtime.KeepAlive(rvalue) @@ -199,3 +125,34 @@ func (i *Implementation) Execute( return capturedErrno, i.handleReturn(cif, rvalue, retVal, uint64(r2), fret, fret2) } + +// scalarWord returns the register or stack word for a non-struct argument. +// Narrow integers are sign- or zero-extended to the full register, as libffi +// does (src/x86/ffi64.c, ffi_call_int). The psABI leaves the upper bits +// unspecified, but clang-built callees rely on the caller extending char and +// short to 32 bits: Apple clang compiles `int64_t f(int8_t a) { return a; }` +// to `movslq %edi, %rax`. A float keeps its exact 32-bit pattern in the low +// bits: widening it to float64 would corrupt what the callee reads. +func scalarWord(t *types.TypeDescriptor, p unsafe.Pointer) uintptr { + switch t.Kind { + case types.FloatType: + return uintptr(math.Float32bits(*(*float32)(p))) + case types.DoubleType, types.PointerType, types.SInt64Type, types.UInt64Type: + return *(*uintptr)(p) + case types.SInt8Type: + return uintptr(int64(*(*int8)(p))) + case types.UInt8Type: + return uintptr(*(*uint8)(p)) + case types.SInt16Type: + return uintptr(int64(*(*int16)(p))) + case types.UInt16Type: + return uintptr(*(*uint16)(p)) + case types.SInt32Type, types.IntType: + return uintptr(int64(*(*int32)(p))) + case types.UInt32Type: + return uintptr(*(*uint32)(p)) + default: + // For unknown/composite types, pass as pointer-to-value + return uintptr(p) + } +} diff --git a/internal/arch/amd64/call_windows.go b/internal/arch/amd64/call_windows.go index 49c33d0..b566acd 100644 --- a/internal/arch/amd64/call_windows.go +++ b/internal/arch/amd64/call_windows.go @@ -3,6 +3,7 @@ package amd64 import ( + "fmt" "math" "runtime" "syscall" @@ -11,6 +12,10 @@ import ( "github.com/go-webgpu/goffi/types" ) +// maxWindowsSlots is the number of argument slots (RCX, RDX, R8, R9 and 9 +// stack slots) the Windows call path accepts. +const maxWindowsSlots = 4 + 9 + // Execute implements arch.FunctionCaller for Windows AMD64. // errnoFn is always 0 on Windows (ErrnoFnAddr returns 0); cerrno is always 0. func (i *Implementation) Execute( @@ -34,6 +39,12 @@ func (i *Implementation) Execute( if sret { first = 1 } + // PrepareCallInterface allows 4 register slots and 9 stack slots + // (ffi/cif.go maxStackSlots); a CallInterface filled in by hand skips that + // check, so it is repeated here. + if n := first + len(cif.ArgTypes); n > maxWindowsSlots { + return 0, fmt.Errorf("goffi: %d argument slots exceed the Windows call path limit of %d", n, maxWindowsSlots) + } args := make([]uintptr, first+len(cif.ArgTypes)) if sret { args[0] = uintptr(rvalue) diff --git a/internal/arch/amd64/classification.go b/internal/arch/amd64/classification.go index b33b981..743e11c 100644 --- a/internal/arch/amd64/classification.go +++ b/internal/arch/amd64/classification.go @@ -70,28 +70,34 @@ func classifyReturnAMD64(t *types.TypeDescriptor, abi types.CallingConvention) i } } -// isStructAllFloats returns true if every member of a flat struct is float or double. -// Per System V AMD64 ABI §3.2.3: if any member in an eightbyte is INTEGER class, -// the entire eightbyte is classified as INTEGER (INTEGER wins over SSE). -func isStructAllFloats(t *types.TypeDescriptor) bool { - if len(t.Members) == 0 { - return false - } - for _, m := range t.Members { +// classifyEightbyte reports whether the eightbyte [startOff, endOff) of struct +// t is SSE class: every scalar member that starts in it, at any nesting depth, +// is a float or double (System V AMD64 ABI §3.2.3: the classes of all fields +// in an eightbyte are merged and INTEGER wins; libffi classify_argument walks +// nested structs the same way). An eightbyte with no member is not SSE. +func classifyEightbyte(t *types.TypeDescriptor, startOff, endOff uintptr) bool { + hasField, allFloat := false, true + walkScalarMembers(t, 0, func(off uintptr, m *types.TypeDescriptor) bool { + if off < startOff || off >= endOff { + return true + } + hasField = true if m.Kind != types.FloatType && m.Kind != types.DoubleType { + allFloat = false return false } - } - return true + return true + }) + return hasField && allFloat } -// classifyEightbyte returns true if all struct fields whose offset falls within -// [startOff, endOff) are SSE types (float or double). -// Returns false if any field in the range is INTEGER class, or if no fields lie in the range. -func classifyEightbyte(t *types.TypeDescriptor, startOff, endOff uintptr) bool { +// walkScalarMembers calls fn with the offset (from base) of every non-struct +// member of struct t, nested structs included, in layout order, until fn +// returns false. Offsets follow each member's alignment, as +// PrepareCallInterface lays the struct out. It returns false when fn stopped +// the walk. +func walkScalarMembers(t *types.TypeDescriptor, base uintptr, fn func(off uintptr, m *types.TypeDescriptor) bool) bool { var offset uintptr - allFloat := true - hasField := false for _, m := range t.Members { if m == nil { continue @@ -99,16 +105,16 @@ func classifyEightbyte(t *types.TypeDescriptor, startOff, endOff uintptr) bool { if m.Alignment > 0 { offset = (offset + m.Alignment - 1) &^ (m.Alignment - 1) } - if offset >= startOff && offset < endOff { - hasField = true - if m.Kind != types.FloatType && m.Kind != types.DoubleType { - allFloat = false - break + if m.Kind == types.StructType { + if !walkScalarMembers(m, base+offset, fn) { + return false } + } else if !fn(base+offset, m) { + return false } offset += m.Size } - return hasField && allFloat + return true } // classifyArgumentAMD64 for x86_64 diff --git a/internal/arch/amd64/layout_sysv.go b/internal/arch/amd64/layout_sysv.go new file mode 100644 index 0000000..de8a4f8 --- /dev/null +++ b/internal/arch/amd64/layout_sysv.go @@ -0,0 +1,175 @@ +// SPDX-License-Identifier: MIT +// SPDX-FileCopyrightText: 2026 Andrey Kolkov and GoGPU Contributors + +//go:build amd64 && (linux || darwin || freebsd) + +package amd64 + +import ( + "github.com/go-webgpu/goffi/types" +) + +// Argument layout for the System V AMD64 psABI (§3.2.3, "Parameter Passing"), +// as libffi implements it (src/x86/ffi64.c: ffi_prep_cif_machdep sizes the +// stack, ffi_call_int places the values): +// +// - Integers and pointers take one INTEGER eightbyte, floats and doubles +// one SSE eightbyte. A struct of at most 16 bytes is split into +// eightbytes, each INTEGER or SSE by its members (INTEGER wins); a larger +// struct is MEMORY. +// - If every eightbyte of an argument gets a register of its class +// (RDI, RSI, RDX, RCX, R8, R9 / XMM0-XMM7), the argument goes in +// registers. Otherwise the whole argument goes on the stack, and the +// registers that were left stay free for later arguments. +// - Stack arguments are aligned to at least 8 bytes and take whole +// 8-byte words, in argument order. + +const ( + sysvGPRs = 6 // RDI, RSI, RDX, RCX, R8, R9 + sysvSSEs = 8 // XMM0-XMM7 + + // MaxStackBytes bounds the stack area for one C call. The words are + // copied onto the system stack (g0) the call runs on, which is the OS + // thread's stack: 512 KiB for a secondary thread on macOS, 8 MiB by + // default on Linux. libffi has no bound (it uses alloca); 64 KiB leaves + // that stack to the callee, and no real C signature comes close (8192 + // words, e.g. 680 structs of 96 bytes). + MaxStackBytes = 64 << 10 +) + +// SysVAllocator hands out registers and stack words to arguments in the order +// they are passed. The ffi package uses it to decode callback arguments with +// the same rules the call path places them. +type SysVAllocator struct { + gpr, sse int + stack uintptr // stack bytes used so far +} + +// Fits reports whether ngpr integer and nsse SSE registers are still free. +func (a *SysVAllocator) Fits(ngpr, nsse int) bool { + return a.gpr+ngpr <= sysvGPRs && a.sse+nsse <= sysvSSEs +} + +// GPR takes the next integer register and returns its index (0 = RDI). +func (a *SysVAllocator) GPR() int { + r := a.gpr + a.gpr++ + return r +} + +// SSE takes the next SSE register and returns its index (0 = XMM0). +func (a *SysVAllocator) SSE() int { + r := a.sse + a.sse++ + return r +} + +// Stack reserves size bytes, aligned to at least 8, and returns the index of +// the first 8-byte word. +func (a *SysVAllocator) Stack(size, align uintptr) int { + if align < 8 { + align = 8 + } + a.stack = (a.stack + align - 1) &^ (align - 1) + word := int(a.stack / 8) + a.stack += (size + 7) &^ 7 + return word +} + +// StackBytes returns the stack bytes reserved so far (a multiple of 8). +func (a *SysVAllocator) StackBytes() uintptr { + return a.stack +} + +// sysvPlacement says where one argument goes. +type sysvPlacement struct { + n int // eightbytes passed in registers (0 when on the stack) + sse [2]bool // class of each eightbyte: SSE (true) or INTEGER + reg [2]int // register index of each eightbyte + onStack bool + word int // first stack word when onStack +} + +// sysvClasses classifies t: n eightbytes with the given classes, or memory +// when t is passed on the stack whatever registers are free. A zero-size +// struct has no eightbytes and is not passed at all. +func sysvClasses(t *types.TypeDescriptor) (sse [2]bool, n int, memory bool) { + switch t.Kind { + case types.FloatType, types.DoubleType: + return [2]bool{true, false}, 1, false + case types.StructType: + if t.Size > 16 { + return sse, 0, true + } + n = int((t.Size + 7) / 8) + for i := range n { + start := uintptr(i) * 8 + sse[i] = classifyEightbyte(t, start, min(start+8, t.Size)) + } + return sse, n, false + default: + return sse, 1, false + } +} + +// place assigns t its registers or stack words. +func (a *SysVAllocator) place(t *types.TypeDescriptor) sysvPlacement { + sse, n, memory := sysvClasses(t) + var p sysvPlacement + if !memory { + ngpr, nsse := 0, 0 + for i := range n { + if sse[i] { + nsse++ + } else { + ngpr++ + } + } + if a.Fits(ngpr, nsse) { + p.n, p.sse = n, sse + for i := range n { + if sse[i] { + p.reg[i] = a.SSE() + } else { + p.reg[i] = a.GPR() + } + } + return p + } + } + size := t.Size + if t.Kind != types.StructType { + size = 8 + } + if size == 0 { + return p + } + p.onStack = true + p.word = a.Stack(size, t.Alignment) + return p +} + +// sysvRetInMemory reports whether the result of cif is returned through a +// hidden pointer, which takes RDI before any argument. +func sysvRetInMemory(cif *types.CallInterface) bool { + return cif.ReturnType.Kind == types.StructType && cif.ReturnType.Size > 16 +} + +// sysvStackBytes returns the stack bytes the arguments of cif need. +func sysvStackBytes(cif *types.CallInterface) uintptr { + var a SysVAllocator + if sysvRetInMemory(cif) { + a.GPR() + } + for _, t := range cif.ArgTypes { + a.place(t) + } + return a.StackBytes() +} + +// StackArgBytes reports how many bytes of stack the arguments of cif need and +// how many the call path supports. PrepareCallInterface uses it to reject +// calls that do not fit. +func (i *Implementation) StackArgBytes(cif *types.CallInterface) (need, limit uintptr, err error) { + return sysvStackBytes(cif), MaxStackBytes, nil +} diff --git a/internal/arch/amd64/struct_args_test.go b/internal/arch/amd64/struct_args_test.go index 81e96b0..8c94592 100644 --- a/internal/arch/amd64/struct_args_test.go +++ b/internal/arch/amd64/struct_args_test.go @@ -9,80 +9,12 @@ import ( "github.com/go-webgpu/goffi/types" ) -// --- isStructAllFloats --- - -func TestIsStructAllFloats(t *testing.T) { - tests := []struct { - name string - typ *types.TypeDescriptor - want bool - }{ - { - "empty members", - &types.TypeDescriptor{Kind: types.StructType}, - false, - }, - { - "single float", - &types.TypeDescriptor{Kind: types.StructType, Members: []*types.TypeDescriptor{ - types.FloatTypeDescriptor, - }}, - true, - }, - { - "single double", - &types.TypeDescriptor{Kind: types.StructType, Members: []*types.TypeDescriptor{ - types.DoubleTypeDescriptor, - }}, - true, - }, - { - "two floats", - &types.TypeDescriptor{Kind: types.StructType, Members: []*types.TypeDescriptor{ - types.FloatTypeDescriptor, - types.FloatTypeDescriptor, - }}, - true, - }, - { - "float and int", - &types.TypeDescriptor{Kind: types.StructType, Members: []*types.TypeDescriptor{ - types.FloatTypeDescriptor, - types.UInt32TypeDescriptor, - }}, - false, - }, - { - "all ints", - &types.TypeDescriptor{Kind: types.StructType, Members: []*types.TypeDescriptor{ - types.UInt32TypeDescriptor, - types.UInt32TypeDescriptor, - }}, - false, - }, - { - "double and uint64", - &types.TypeDescriptor{Kind: types.StructType, Members: []*types.TypeDescriptor{ - types.DoubleTypeDescriptor, - types.UInt64TypeDescriptor, - }}, - false, - }, - { - "pointer only", - &types.TypeDescriptor{Kind: types.StructType, Members: []*types.TypeDescriptor{ - types.PointerTypeDescriptor, - }}, - false, - }, - } - for _, tt := range tests { - t.Run(tt.name, func(t *testing.T) { - got := isStructAllFloats(tt.typ) - if got != tt.want { - t.Errorf("isStructAllFloats() = %v, want %v", got, tt.want) - } - }) +// floatPairTypeAligned returns a {float32, float32} descriptor with its size +// and alignment set, as PrepareCallInterface leaves a nested member. +func floatPairTypeAligned() *types.TypeDescriptor { + return &types.TypeDescriptor{ + Size: 8, Alignment: 4, Kind: types.StructType, + Members: []*types.TypeDescriptor{types.FloatTypeDescriptor, types.FloatTypeDescriptor}, } } @@ -122,6 +54,26 @@ func TestClassifyEightbyte(t *testing.T) { }, } + // {{float32, float32}, {float32, float32}}: nested members are classified + // by their offsets, so both eightbytes are SSE (psABI §3.2.3). + nestedFloatsType := &types.TypeDescriptor{ + Size: 16, Alignment: 4, Kind: types.StructType, + Members: []*types.TypeDescriptor{floatPairTypeAligned(), floatPairTypeAligned()}, + } + // {int64, {float32, float32}}: INTEGER, then a nested SSE eightbyte. + intThenNestedType := &types.TypeDescriptor{ + Size: 16, Alignment: 8, Kind: types.StructType, + Members: []*types.TypeDescriptor{types.SInt64TypeDescriptor, floatPairTypeAligned()}, + } + // {float32, {float32, int32}}: the nested int32 lands in the second eightbyte. + nestedMixedType := &types.TypeDescriptor{ + Size: 12, Alignment: 4, Kind: types.StructType, + Members: []*types.TypeDescriptor{types.FloatTypeDescriptor, { + Size: 8, Alignment: 4, Kind: types.StructType, + Members: []*types.TypeDescriptor{types.FloatTypeDescriptor, types.SInt32TypeDescriptor}, + }}, + } + tests := []struct { name string typ *types.TypeDescriptor @@ -129,6 +81,15 @@ func TestClassifyEightbyte(t *testing.T) { endOff uintptr wantSSE bool }{ + {"empty members", &types.TypeDescriptor{Kind: types.StructType}, 0, 8, false}, + {"pointer only", &types.TypeDescriptor{Size: 8, Alignment: 8, Kind: types.StructType, + Members: []*types.TypeDescriptor{types.PointerTypeDescriptor}}, 0, 8, false}, + {"nested floats [0,8)", nestedFloatsType, 0, 8, true}, + {"nested floats [8,16)", nestedFloatsType, 8, 16, true}, + {"int then nested floats [0,8)", intThenNestedType, 0, 8, false}, + {"int then nested floats [8,16)", intThenNestedType, 8, 16, true}, + {"nested mixed [0,8)", nestedMixedType, 0, 8, true}, + {"nested mixed [8,12)", nestedMixedType, 8, 12, false}, {"int pair [0,8)", intPairType, 0, 8, false}, {"float pair [0,8)", floatPairType, 0, 8, true}, {"mixed float+int [0,8) INTEGER wins", mixedType, 0, 8, false}, diff --git a/internal/syscall/syscall_unix_amd64.go b/internal/syscall/syscall_unix_amd64.go index fc651c0..a229b6d 100644 --- a/internal/syscall/syscall_unix_amd64.go +++ b/internal/syscall/syscall_unix_amd64.go @@ -14,31 +14,40 @@ import ( func runtime_cgocall(fn uintptr, arg unsafe.Pointer) int32 // syscallArgsPool recycles the per-call argument/return block. The block must -// live in non-moving memory (see CallNFloat): pooling keeps it heap-resident and -// reused, so the move-safe path costs no per-call allocation in steady state. +// live in non-moving memory (see CallNFloatErrno): pooling keeps it +// heap-resident and reused, so the move-safe path costs no per-call allocation +// in steady state. var syscallArgsPool = sync.Pool{New: func() any { return new(syscallArgs) }} // syscallArgs matches the layout expected by syscallN assembly. -// Supports up to 15 total arguments (6 GP registers + 9 stack slots), -// matching purego's syscall15Args layout. // // Layout (offsets in bytes): // // fn: 0 -// a1-a15: 8-128 (6 GP registers + 9 stack slots) -// f1-f8: 128-192 (XMM0-XMM7 as bit patterns) -// r1: 192 (RAX return) -// r2: 200 (RDX return, used for 9-16 byte struct returns) -// errno: 208 (captured C errno value; 0 if errnoFn == 0) -// errnoFn: 216 (address of __errno_location/__error; 0 = skip errno capture) +// a1-a6: 8-48 (RDI, RSI, RDX, RCX, R8, R9) +// stack: 56 (address of the first stack word) +// nstack: 64 (number of 8-byte stack words) +// f1-f8: 72-128 (XMM0-XMM7 as bit patterns) +// r1: 136 (RAX return) +// r2: 144 (RDX return, used for 9-16 byte struct returns) +// errno: 152 (captured C errno value; 0 if errnoFn == 0) +// errnoFn: 160 (address of __errno_location/__error; 0 = skip errno capture) type syscallArgs struct { - _ structs.HostLayout - fn uintptr - a1, a2, a3, a4, a5, a6, a7, a8, a9, a10, a11, a12, a13, a14, a15 uintptr - f1, f2, f3, f4, f5, f6, f7, f8 uintptr - r1, r2 uintptr - errno uintptr - errnoFn uintptr + _ structs.HostLayout + fn uintptr + a1, a2, a3, a4, a5, a6 uintptr + stack uintptr + nstack uintptr + f1, f2, f3, f4 uintptr + f5, f6, f7, f8 uintptr + r1, r2 uintptr + errno uintptr + errnoFn uintptr + + // words backs stack: the stack arguments are copied here, into the pooled + // heap block, so syscallN reads them from memory that does not move. It + // is not part of the layout syscallN sees. + words []uintptr } // syscallN is implemented in syscall_unix_amd64.s @@ -49,91 +58,59 @@ func syscallN(args unsafe.Pointer) // syscallNABI0 is the ABI0 entry point for syscallN var syscallNABI0 uintptr -// CallNFloat calls a C function with up to 6 integer register arguments, -// 8 SSE arguments, and 9 stack-spill arguments (15 total). +// CallNFloat is CallNFloatErrno without errno capture. +func CallNFloat(fn uintptr, gpr [6]uintptr, sse [8]float64, stack []uintptr) (r1 uintptr, r2 uintptr, f1 float64, f2 float64) { + r1, r2, f1, f2, _ = CallNFloatErrno(fn, gpr, sse, stack, 0) + return +} + +// CallNFloatErrno calls a C function with 6 integer register arguments, 8 SSE +// register arguments and len(stack) 8-byte stack words, and captures the C +// errno value set by the called function. +// +// gpr: RDI, RSI, RDX, RCX, R8, R9 +// sse: XMM0-XMM7 as float64 bit patterns +// stack: the stack words in order: stack[0] is at the stack pointer at the +// call. syscallN reserves exactly this many words on the system stack, so the +// caller bounds len(stack) (amd64.MaxStackBytes). // -// gpr: first 6 GP register values (RDI, RSI, RDX, RCX, R8, R9) -// sse: 8 SSE register values (XMM0-XMM7) as float64 bit patterns -// stackArgs: additional arguments to push onto the stack before CALL -// numStack: how many entries in stackArgs are valid (0-9) +// The errno function address (errnoFn) must be the address of __errno_location +// (Linux/FreeBSD) or __error (macOS), obtained from ErrnoFnAddr(). When errnoFn +// is 0, errno capture is skipped and the returned errno value is always 0. The +// errno is read inside the assembly trampoline immediately after the C function +// returns, before the Go runtime can migrate the goroutine to a different OS +// thread. This is the only safe window for errno capture. // // Returns: // - r1: RAX integer return value // - r2: RDX second integer return value (9-16 byte struct returns) // - f1: XMM0 float return value (bit pattern) // - f2: XMM1 second float return value — for {SSE, SSE} 9-16B struct returns (e.g. NSPoint) -func CallNFloat(fn uintptr, gpr [6]uintptr, sse [8]float64, stackArgs [9]uintptr, numStack int) (r1 uintptr, r2 uintptr, f1 float64, f2 float64) { - // args holds both the call arguments and the C function's return values. - // syscallN runs on g0 and writes the return values back into args after the - // call returns. That call can run a callback that re-enters Go and grows this - // goroutine's stack, which moves it. If args lived on the stack it would move - // too, syscallN's write would land at the old address, and the first such call - // would lose its return value. So args has to live in non-moving memory: we - // take it from the heap-backed pool. The heap does not move on today's Go - // (non-compacting GC, confirmed through Go 1.27). If a compacting GC ever - // lands, add runtime.Pinner here. +func CallNFloatErrno(fn uintptr, gpr [6]uintptr, sse [8]float64, stack []uintptr, errnoFn uintptr) (r1 uintptr, r2 uintptr, f1 float64, f2 float64, cerrno uintptr) { + // args holds the call arguments, the stack words and the C function's + // return values. syscallN runs on g0 and writes the return values back + // into args after the call returns. That call can run a callback that + // re-enters Go and grows this goroutine's stack, which moves it. If args + // lived on the stack it would move too, syscallN's write would land at the + // old address, and the first such call would lose its return value. So + // args has to live in non-moving memory: we take it from the heap-backed + // pool. The heap does not move on today's Go (non-compacting GC, confirmed + // through Go 1.27). If a compacting GC ever lands, add runtime.Pinner here. args := syscallArgsPool.Get().(*syscallArgs) defer syscallArgsPool.Put(args) - *args = syscallArgs{ - fn: fn, - a1: gpr[0], a2: gpr[1], a3: gpr[2], - a4: gpr[3], a5: gpr[4], a6: gpr[5], - // Stack spill slots: a7-a15 map to stackArgs[0]-stackArgs[8] - a7: stackArgs[0], - a8: stackArgs[1], - a9: stackArgs[2], - a10: stackArgs[3], - a11: stackArgs[4], - a12: stackArgs[5], - a13: stackArgs[6], - a14: stackArgs[7], - a15: stackArgs[8], - // SSE arguments as bit patterns - f1: *(*uintptr)(unsafe.Pointer(&sse[0])), - f2: *(*uintptr)(unsafe.Pointer(&sse[1])), - f3: *(*uintptr)(unsafe.Pointer(&sse[2])), - f4: *(*uintptr)(unsafe.Pointer(&sse[3])), - f5: *(*uintptr)(unsafe.Pointer(&sse[4])), - f6: *(*uintptr)(unsafe.Pointer(&sse[5])), - f7: *(*uintptr)(unsafe.Pointer(&sse[6])), - f8: *(*uintptr)(unsafe.Pointer(&sse[7])), - } - _ = numStack // numStack is informational; assembly always pushes all 9 slots - - runtime_cgocall(syscallNABI0, unsafe.Pointer(args)) - r1 = args.r1 - r2 = args.r2 - f1 = *(*float64)(unsafe.Pointer(&args.f1)) - f2 = *(*float64)(unsafe.Pointer(&args.f2)) - return -} + words := args.words + if cap(words) < len(stack) { + words = make([]uintptr, len(stack)) + } + words = words[:len(stack)] + copy(words, stack) -// CallNFloatErrno is like CallNFloat but also captures the C errno value set -// by the called function. The errno function address (errnoFn) must be the -// address of __errno_location (Linux/FreeBSD) or __error (macOS), obtained -// from ErrnoFnAddr(). When errnoFn is 0, errno capture is skipped and the -// returned errno value is always 0. -// -// The errno is read inside the assembly trampoline immediately after the C -// function returns, before the Go runtime can migrate the goroutine to a -// different OS thread. This is the only safe window for errno capture. -func CallNFloatErrno(fn uintptr, gpr [6]uintptr, sse [8]float64, stackArgs [9]uintptr, numStack int, errnoFn uintptr) (r1 uintptr, r2 uintptr, f1 float64, f2 float64, cerrno uintptr) { - args := syscallArgsPool.Get().(*syscallArgs) - defer syscallArgsPool.Put(args) *args = syscallArgs{ fn: fn, a1: gpr[0], a2: gpr[1], a3: gpr[2], a4: gpr[3], a5: gpr[4], a6: gpr[5], - a7: stackArgs[0], - a8: stackArgs[1], - a9: stackArgs[2], - a10: stackArgs[3], - a11: stackArgs[4], - a12: stackArgs[5], - a13: stackArgs[6], - a14: stackArgs[7], - a15: stackArgs[8], + nstack: uintptr(len(words)), f1: *(*uintptr)(unsafe.Pointer(&sse[0])), f2: *(*uintptr)(unsafe.Pointer(&sse[1])), f3: *(*uintptr)(unsafe.Pointer(&sse[2])), @@ -143,8 +120,13 @@ func CallNFloatErrno(fn uintptr, gpr [6]uintptr, sse [8]float64, stackArgs [9]ui f7: *(*uintptr)(unsafe.Pointer(&sse[6])), f8: *(*uintptr)(unsafe.Pointer(&sse[7])), errnoFn: errnoFn, + words: words, + } + if len(words) > 0 { + // The address of heap memory that args.words keeps alive and that the + // GC does not move; syscallN copies the words before the C call. + args.stack = uintptr(unsafe.Pointer(unsafe.SliceData(words))) } - _ = numStack runtime_cgocall(syscallNABI0, unsafe.Pointer(args)) diff --git a/internal/syscall/syscall_unix_amd64.s b/internal/syscall/syscall_unix_amd64.s index ad3f7f8..9e26bd0 100644 --- a/internal/syscall/syscall_unix_amd64.s +++ b/internal/syscall/syscall_unix_amd64.s @@ -3,9 +3,9 @@ #include "textflag.h" #include "abi_amd64.h" -// syscallN calls a C function with up to 6 integer, 8 float, and 9 stack arguments. +// syscallN calls a C function with 6 integer register arguments, 8 SSE +// register arguments and a variable number of 8-byte stack words. // System V AMD64 ABI calling convention (identical on Linux, macOS, FreeBSD). -// This implementation follows purego's syscall15X pattern. // // syscallN takes a pointer to syscallArgs struct: // struct { @@ -16,117 +16,100 @@ // a4 uintptr // offset 32 (RCX) // a5 uintptr // offset 40 (R8) // a6 uintptr // offset 48 (R9) -// a7 uintptr // offset 56 (stack[0]) -// a8 uintptr // offset 64 (stack[1]) -// a9 uintptr // offset 72 (stack[2]) -// a10 uintptr // offset 80 (stack[3]) -// a11 uintptr // offset 88 (stack[4]) -// a12 uintptr // offset 96 (stack[5]) -// a13 uintptr // offset 104 (stack[6]) -// a14 uintptr // offset 112 (stack[7]) -// a15 uintptr // offset 120 (stack[8]) -// f1 uintptr // offset 128 (XMM0 bit pattern) -// f2 uintptr // offset 136 (XMM1) -// f3 uintptr // offset 144 (XMM2) -// f4 uintptr // offset 152 (XMM3) -// f5 uintptr // offset 160 (XMM4) -// f6 uintptr // offset 168 (XMM5) -// f7 uintptr // offset 176 (XMM6) -// f8 uintptr // offset 184 (XMM7) -// r1 uintptr // offset 192 (RAX return) -// r2 uintptr // offset 200 (RDX return, 9-16 byte struct) -// errno uintptr // offset 208 (captured C errno; 0 if errnoFn == 0) -// errnoFn uintptr // offset 216 (address of __errno_location/__error; 0 = skip) +// stack uintptr // offset 56 (address of the first stack word) +// nstack uintptr // offset 64 (number of stack words) +// f1 uintptr // offset 72 (XMM0 bit pattern) +// f2 uintptr // offset 80 (XMM1) +// f3 uintptr // offset 88 (XMM2) +// f4 uintptr // offset 96 (XMM3) +// f5 uintptr // offset 104 (XMM4) +// f6 uintptr // offset 112 (XMM5) +// f7 uintptr // offset 120 (XMM6) +// f8 uintptr // offset 128 (XMM7) +// r1 uintptr // offset 136 (RAX return) +// r2 uintptr // offset 144 (RDX return, 9-16 byte struct) +// errno uintptr // offset 152 (captured C errno; 0 if errnoFn == 0) +// errnoFn uintptr // offset 160 (address of __errno_location/__error; 0 = skip) // } // // syscallN must be called on the g0 stack with runtime.cgocall. // -// Stack frame layout (STACK_SIZE = 80): -// SP+0 .. SP+71 : 9 stack-spill slots for a7-a15 (9 * 8 = 72 bytes) -// SP+72 .. SP+79 : saved args pointer (PTR_ADDRESS = 72) -// [then BP push/pop outside STACK_SIZE] +// Stack frame: BP is pushed, then nstack words rounded up to 16 bytes are +// reserved below it and the stack words are copied there, stack[0] at SP. +// asmcgocall aligns SP to 16 bytes before its CALL, so after PUSHQ BP the SP +// is 16-byte aligned again and stays so at our CALL, as the ABI requires. The +// args pointer is kept in R12, which is callee-saved, across the C call; SP +// is restored from BP. The caller bounds nstack (amd64.MaxStackBytes). GLOBL ·syscallNABI0(SB), NOPTR|RODATA, $8 DATA ·syscallNABI0(SB)/8, $syscallN(SB) TEXT syscallN(SB), NOSPLIT|NOFRAME, $0 PUSHQ BP MOVQ SP, BP - SUBQ $STACK_SIZE, SP - MOVQ DI, PTR_ADDRESS(SP) // save the pointer - MOVQ DI, R11 // R11 = args pointer + MOVQ DI, R12 // R12 = args pointer (callee-saved, survives the C call) - // Load float arguments into XMM0-XMM7 (offsets 128-184) - MOVQ 128(R11), X0 // f1 - MOVQ 136(R11), X1 // f2 - MOVQ 144(R11), X2 // f3 - MOVQ 152(R11), X3 // f4 - MOVQ 160(R11), X4 // f5 - MOVQ 168(R11), X5 // f6 - MOVQ 176(R11), X6 // f7 - MOVQ 184(R11), X7 // f8 + // Reserve round16(nstack*8) bytes and copy the stack words, stack[0] at SP. + MOVQ 64(R12), CX // CX = nstack + MOVQ CX, AX + SHLQ $3, AX + ADDQ $15, AX + ANDQ $~15, AX + SUBQ AX, SP + MOVQ 56(R12), SI // SI = &stack[0] + XORQ DX, DX - // Push stack-spill arguments a7-a15 onto the stack (offsets 56-120) - // System V AMD64 ABI: args 7+ are pushed right-to-left onto the stack, - // but since we build the stack growing downward with explicit offsets, - // we push a7 at SP+0, a8 at SP+8, etc. - MOVQ 56(R11), R12 - MOVQ R12, 0(SP) // push a7 - MOVQ 64(R11), R12 - MOVQ R12, 8(SP) // push a8 - MOVQ 72(R11), R12 - MOVQ R12, 16(SP) // push a9 - MOVQ 80(R11), R12 - MOVQ R12, 24(SP) // push a10 - MOVQ 88(R11), R12 - MOVQ R12, 32(SP) // push a11 - MOVQ 96(R11), R12 - MOVQ R12, 40(SP) // push a12 - MOVQ 104(R11), R12 - MOVQ R12, 48(SP) // push a13 - MOVQ 112(R11), R12 - MOVQ R12, 56(SP) // push a14 - MOVQ 120(R11), R12 - MOVQ R12, 64(SP) // push a15 +copy_loop: + CMPQ DX, CX + JGE copy_done + MOVQ (SI)(DX*8), AX + MOVQ AX, (SP)(DX*8) + INCQ DX + JMP copy_loop + +copy_done: + // Load float arguments into XMM0-XMM7 (offsets 72-128) + MOVQ 72(R12), X0 // f1 + MOVQ 80(R12), X1 // f2 + MOVQ 88(R12), X2 // f3 + MOVQ 96(R12), X3 // f4 + MOVQ 104(R12), X4 // f5 + MOVQ 112(R12), X5 // f6 + MOVQ 120(R12), X6 // f7 + MOVQ 128(R12), X7 // f8 // Load integer arguments into GP registers (System V AMD64 ABI, offsets 8-48) - MOVQ 8(R11), DI // a1 -> RDI - MOVQ 16(R11), SI // a2 -> RSI - MOVQ 24(R11), DX // a3 -> RDX - MOVQ 32(R11), CX // a4 -> RCX - MOVQ 40(R11), R8 // a5 -> R8 - MOVQ 48(R11), R9 // a6 -> R9 + MOVQ 8(R12), DI // a1 -> RDI + MOVQ 16(R12), SI // a2 -> RSI + MOVQ 24(R12), DX // a3 -> RDX + MOVQ 32(R12), CX // a4 -> RCX + MOVQ 40(R12), R8 // a5 -> R8 + MOVQ 48(R12), R9 // a6 -> R9 // For vararg functions: AL = number of float args in XMM registers (set 0 = safe default) XORL AX, AX // Load function pointer and call (offset 0) - MOVQ 0(R11), R10 + MOVQ 0(R12), R10 CALL R10 - // Restore args pointer and save C function return values. - // DI was clobbered by the CALL (it held a1/RDI before the call). - MOVQ PTR_ADDRESS(SP), DI - MOVQ AX, 192(DI) // r1: integer return in RAX - MOVQ DX, 200(DI) // r2: second integer return in RDX (9-16 byte structs) - MOVQ X0, 128(DI) // f1: float return in XMM0 - MOVQ X1, 136(DI) // f2: XMM1 — second SSE return for 9-16B all-float struct returns + // Save C function return values; R12 still holds the args pointer. + MOVQ AX, 136(R12) // r1: integer return in RAX + MOVQ DX, 144(R12) // r2: second integer return in RDX (9-16 byte structs) + MOVQ X0, 72(R12) // f1: float return in XMM0 + MOVQ X1, 80(R12) // f2: XMM1 — second SSE return for 9-16B all-float struct returns - // Errno capture (conditional): only when errnoFn (offset 216) is non-zero. + // Errno capture (conditional): only when errnoFn (offset 160) is non-zero. // Safe window: we are still on g0, same OS thread as the C call. - // R12 and R13 are callee-saved under the System V AMD64 ABI, so they - // survive the CALL R13 to __errno_location/__error below. - MOVQ 216(DI), R13 // R13 = errnoFn address + MOVQ 160(R12), R13 // R13 = errnoFn address TESTQ R13, R13 - JZ errno_done - MOVQ DI, R12 // R12 = save args pointer across the errno call - CALL R13 // __errno_location()/__error() → RAX = &errno (int*) - MOVL (AX), AX // AX = *(&errno) as int32, zero-extended to 64 bits - MOVQ AX, 208(R12) // args->errno = captured errno value + JZ errno_done + CALL R13 // __errno_location()/__error() → RAX = &errno (int*) + MOVL (AX), AX // AX = *(&errno) as int32, zero-extended to 64 bits + MOVQ AX, 152(R12) // args->errno = captured errno value errno_done: // Restore stack and return - XORL AX, AX // no error (ignored by runtime.cgocall) - ADDQ $STACK_SIZE, SP + XORL AX, AX // no error (ignored by runtime.cgocall) MOVQ BP, SP POPQ BP RET