Feature/stack interleave - #415
Merged
Merged
Conversation
The kernel ABI gives each hardware thread a contiguous 8 KB stack
(sp = STACK_BASE - hartid << 13), so the same frame slot across a
warp's threads sits one stack apart: NUM_THREADS separate lines, all
in the same dcache bank. Every spill, callee-saved save/restore and
runtime-indexed local turned into NT serialized same-bank misses.
The LSU AGU now remaps every address in the stack window so that,
within each group of NUM_THREADS stacks, offset {thread, word, byte}
is stored as {word ^ group, thread, byte} (word = XLEN bytes). A
warp-wide access to one frame slot fills a single line, the layout
NVIDIA local memory and AMD scratch use. The XOR skew keeps equal
slots of different warps out of the same cache set, since a group
spans a power of two. The map is a bijection that depends on the
address alone, so cross-thread stack pointers still resolve and
software, sp and frame offsets are unchanged.
- VX_CFG_LSU_STACK_INTERLEAVE_ENABLE (default on) in VX_config.toml.
- RTL: VX_lsu_agu applies the remap; window constants live in
VX_gpu_pkg (core count = clusters * sockets * socket size).
- SimX: StackInterleave (stack_interleave.h) applied in
LsuUnit::compute_addrs and the debug module's memory accesses.
- TCU metadata is fetched as a linear warp-wide tile and must not
live on a thread stack; RTL and SimX both assert this.
- Stack accesses wider than a word are rejected (static FLEN<=XLEN
in RTL, runtime check in SimX).
- New parity case parity-raycast-nt16 (16 warps x 16 threads).
raycast, 16 warps x 16 threads, rtlsim: 560672 -> 368919 cycles
(1.52x); simx 381214 (3.3% parity gap). Edge configs (NT=1,
SOCKET_SIZE > NUM_CORES, 2 cores, RV64 NT4/NT16) pass on both drivers
with matching instruction counts.
Gates (rtlsim, xlen 32): all 30 model_parity cases pass
(model_parity-wgmma-dxa gap 5.17% -> 1.22%, so
its known_issue is retired). perf_gate:
24 cases improved beyond the 2% ratchet and their baselines are
regenerated with --update-baselines, e.g. softmax-nt16 3584289 ->
1274529, om-nt16 539979 -> 192077, raster-nt16 75760 -> 14959,
copy-nt16 10781 -> 2098, raycast-nt16 734368 -> 567659.
wgmma-fp16-ss (128x128x128) moves 758510 -> 841657 (+11%) and its
baseline is regenerated too. It is not a cost of the remap: the
kernel touches the stack once per warp. Its RTL schedule is bimodal,
~5.9k cycles per k-step (scheduler idle 39%) or ~6.5k (idle 45%),
and any small timing change can flip it. Sweeping K at 128x128 on
master vs this change: k=32 202416/199829, 64 389817/388047,
96 574918/575776, 128 757675/841657, 160 944889/940841,
192 1253345/1130145, 256 1641902/1658985. Master lands in the slow
mode at k=192 and 256; this change moves the cliff to k=128 and
out of k=192; over the sweep the two are even (geomean -0.1%).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
raytracing:perf_gate-rt_raycast-nt16 segfaulted in reset, before any kernel ran, on master too. GCC 13.3 turns the generated per-word copy of a wide signal (the divider's VlWide<55> shift-register stage) into memcpy and inlines it with movaps, but the destination sits at an 8-byte-aligned offset of the 64-byte-aligned root class. -O0 builds and other configs happen to avoid it. Build every Verilator model with -fno-tree-loop-distribute-patterns (rtlsim, xrtsim, opaesim, avedsim). rt_raycast-nt16 now passes its gate (52488 cycles vs baseline 51652). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
blackbox.sh rejects --rebuild; it re-makes the driver on every run and make rebuilds it on any source or flag change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Vortex gives each hardware thread a contiguous 8 KB stack, so the same stack slot across a warp's threads lands in NUM_THREADS separate cache lines, all in the same dcache bank. Every register spill, callee-saved save/restore and runtime-indexed local becomes NT serialized same-bank misses. With RISC-V's 32 registers, kernels spill a lot, so this cost is significant.
This PR interleaves the per-thread stacks by word in the LSU address generator. Within each group of NUM_THREADS stacks, offset
{thread, word, byte}is stored as{word ^ group, thread, byte}. A warp-wide access to one stack slot then fills a single line; NVIDIA local memory and AMD scratch lay out their stacks the same way.sp, frame offsets and cross-thread stack pointers are unchanged.Commits
VX_CFG_LSU_STACK_INTERLEAVE_ENABLE, default on.VX_lsu_agu.sv; SimX is instack_interleave.h, kept in lockstep with the RTL.parity-raycast-nt16.rt_raycast-nt16segfaulted during reset, on master too.memcpy. It then inlines that copy with aligned 16-byte stores (movaps) to an address that is only 8-byte aligned.-fno-tree-loop-distribute-patterns.--rebuildoption.AGENTS.mdanddocs/debugging.mddocumented it, butblackbox.shrejects it.Performance
raycast, 16 warps × 16 threads:
24 perf-gate cases improved beyond the 2% ratchet. Their baselines were regenerated with
--update-baselines. Examples (rtlsim cycles):model_parity-wgmma-dxa's SimX/RTL gap closed from ~6% to 1.2%, so itsknown_issueis retired.wgmma-fp16-ss(+11%, baseline regenerated). This is not a cost of the remap: the kernel touches the stack once per warp. Its RTL schedule is bimodal, either ~5.9k cycles per k-step (39% scheduler idle) or ~6.5k (45% idle), and small timing changes flip it.K sweep at 128×128, rtlsim cycles:
Master is in the slow mode at K=192 and 256. This PR moves the slow point to K=128 and out of K=192. The geomean across the sweep is −0.1%. The bimodality exists on master and deserves its own investigation.
Test plan
pytest ci -m "model_parity or perf_gate"(rtlsim, xlen 32): 79 cases, 76 passed, 1 passed after itsknown_issuewas retired. The other 2 are pre-existing DXA xfails.rt_raycast-nt16runs again: 52,488 cycles against a baseline of 51,652.