Skip to content

Feature/stack interleave - #415

Merged
tinebp merged 4 commits into
masterfrom
feature/stack-interleave
Sep 20, 2026
Merged

tinebp merged 4 commits into
masterfrom
feature/stack-interleave

Conversation

@tinebp

@tinebp tinebp commented Sep 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Vortex gives each hardware thread a contiguous 8 KB stack, so the same stack slot across a warp's threads lands in NUM_THREADS separate cache lines, all in the same dcache bank. Every register spill, callee-saved save/restore and runtime-indexed local becomes NT serialized same-bank misses. With RISC-V's 32 registers, kernels spill a lot, so this cost is significant.

This PR interleaves the per-thread stacks by word in the LSU address generator. Within each group of NUM_THREADS stacks, offset {thread, word, byte} is stored as {word ^ group, thread, byte}. A warp-wide access to one stack slot then fills a single line; NVIDIA local memory and AMD scratch lay out their stacks the same way.

  • The XOR skew stops equal slots of different warps from aliasing into one cache set.
  • The map is a bijection that depends only on the address, so software, sp, frame offsets and cross-thread stack pointers are unchanged.

Commits

  1. lsu: interleave per-thread stacks across a warp's threads
    • VX_CFG_LSU_STACK_INTERLEAVE_ENABLE, default on.
    • RTL is in VX_lsu_agu.sv; SimX is in stack_interleave.h, kept in lockstep with the RTL.
    • TCU metadata inside the stack window is asserted against on both sides.
    • New parity case parity-raycast-nt16.
  2. sim: stop GCC 13 miscompiling Verilator models. rt_raycast-nt16 segfaulted during reset, on master too.
    • GCC 13.3 converts the generated per-word copy of a wide signal into memcpy. It then inlines that copy with aligned 16-byte stores (movaps) to an address that is only 8-byte aligned.
    • The fix builds all Verilator models with -fno-tree-loop-distribute-patterns.
  3. docs: drop the nonexistent blackbox --rebuild option. AGENTS.md and docs/debugging.md documented it, but blackbox.sh rejects it.

Performance

raycast, 16 warps × 16 threads:

Driver Before (cycles) After (cycles) Speedup
rtlsim 560,672 368,919 1.52x
simx 632,453 381,214 1.66x

24 perf-gate cases improved beyond the 2% ratchet. Their baselines were regenerated with --update-baselines. Examples (rtlsim cycles):

Case Before After
softmax-nt16 3,584,289 1,274,529
om-nt16 539,979 192,077
raster-nt16 75,760 14,959
dxa copy-nt16 10,781 2,098
raycast-nt16 734,368 567,659

model_parity-wgmma-dxa's SimX/RTL gap closed from ~6% to 1.2%, so its known_issue is retired.

wgmma-fp16-ss (+11%, baseline regenerated). This is not a cost of the remap: the kernel touches the stack once per warp. Its RTL schedule is bimodal, either ~5.9k cycles per k-step (39% scheduler idle) or ~6.5k (45% idle), and small timing changes flip it.

K sweep at 128×128, rtlsim cycles:

K master this PR
32 202,416 199,829
64 389,817 388,047
96 574,918 575,776
128 757,675 841,657
160 944,889 940,841
192 1,253,345 1,130,145
256 1,641,902 1,658,985

Master is in the slow mode at K=192 and 256. This PR moves the slow point to K=128 and out of K=192. The geomean across the sweep is −0.1%. The bimodality exists on master and deserves its own investigation.

Test plan

  • pytest ci -m "model_parity or perf_gate" (rtlsim, xlen 32): 79 cases, 76 passed, 1 passed after its known_issue was retired. The other 2 are pre-existing DXA xfails.
  • Edge configs (raycast, both drivers, matching instruction counts): NT=1, SOCKET_SIZE=2 with 1 core, 2 cores + L2, RV64 NT=4, RV64 NT=16.
  • rt_raycast-nt16 runs again: 52,488 cycles against a baseline of 51,652.

tinebp and others added 4 commits September 19, 2026 05:56
The kernel ABI gives each hardware thread a contiguous 8 KB stack
(sp = STACK_BASE - hartid << 13), so the same frame slot across a
warp's threads sits one stack apart: NUM_THREADS separate lines, all
in the same dcache bank. Every spill, callee-saved save/restore and
runtime-indexed local turned into NT serialized same-bank misses.

The LSU AGU now remaps every address in the stack window so that,
within each group of NUM_THREADS stacks, offset {thread, word, byte}
is stored as {word ^ group, thread, byte} (word = XLEN bytes). A
warp-wide access to one frame slot fills a single line, the layout
NVIDIA local memory and AMD scratch use. The XOR skew keeps equal
slots of different warps out of the same cache set, since a group
spans a power of two. The map is a bijection that depends on the
address alone, so cross-thread stack pointers still resolve and
software, sp and frame offsets are unchanged.

- VX_CFG_LSU_STACK_INTERLEAVE_ENABLE (default on) in VX_config.toml.
- RTL: VX_lsu_agu applies the remap; window constants live in
  VX_gpu_pkg (core count = clusters * sockets * socket size).
- SimX: StackInterleave (stack_interleave.h) applied in
  LsuUnit::compute_addrs and the debug module's memory accesses.
- TCU metadata is fetched as a linear warp-wide tile and must not
  live on a thread stack; RTL and SimX both assert this.
- Stack accesses wider than a word are rejected (static FLEN<=XLEN
  in RTL, runtime check in SimX).
- New parity case parity-raycast-nt16 (16 warps x 16 threads).

raycast, 16 warps x 16 threads, rtlsim: 560672 -> 368919 cycles
(1.52x); simx 381214 (3.3% parity gap). Edge configs (NT=1,
SOCKET_SIZE > NUM_CORES, 2 cores, RV64 NT4/NT16) pass on both drivers
with matching instruction counts.

Gates (rtlsim, xlen 32): all 30 model_parity cases pass
(model_parity-wgmma-dxa gap 5.17% -> 1.22%, so
its known_issue is retired). perf_gate:
24 cases improved beyond the 2% ratchet and their baselines are
regenerated with --update-baselines, e.g. softmax-nt16 3584289 ->
1274529, om-nt16 539979 -> 192077, raster-nt16 75760 -> 14959,
copy-nt16 10781 -> 2098, raycast-nt16 734368 -> 567659.

wgmma-fp16-ss (128x128x128) moves 758510 -> 841657 (+11%) and its
baseline is regenerated too. It is not a cost of the remap: the
kernel touches the stack once per warp. Its RTL schedule is bimodal,
~5.9k cycles per k-step (scheduler idle 39%) or ~6.5k (idle 45%),
and any small timing change can flip it. Sweeping K at 128x128 on
master vs this change: k=32 202416/199829, 64 389817/388047,
96 574918/575776, 128 757675/841657, 160 944889/940841,
192 1253345/1130145, 256 1641902/1658985. Master lands in the slow
mode at k=192 and 256; this change moves the cliff to k=128 and
out of k=192; over the sweep the two are even (geomean -0.1%).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
raytracing:perf_gate-rt_raycast-nt16 segfaulted in reset, before any
kernel ran, on master too. GCC 13.3 turns the generated per-word copy
of a wide signal (the divider's VlWide<55> shift-register stage) into
memcpy and inlines it with movaps, but the destination sits at an
8-byte-aligned offset of the 64-byte-aligned root class. -O0 builds
and other configs happen to avoid it.

Build every Verilator model with -fno-tree-loop-distribute-patterns
(rtlsim, xrtsim, opaesim, avedsim). rt_raycast-nt16 now passes its
gate (52488 cycles vs baseline 51652).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
blackbox.sh rejects --rebuild; it re-makes the driver on every run and
make rebuilds it on any source or flag change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@tinebp
tinebp merged commit 0bef53d into master Sep 20, 2026
2 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant