Skip to content

sim: fix TCU tile-buffer response matching past 65536 requests - #414

Open
eyoon1131 wants to merge 1 commit into
vortexgpgpu:masterfrom
eyoon1131:fix/tcu-tbuf-tag-wrap
Open

eyoon1131 wants to merge 1 commit into
vortexgpgpu:masterfrom
eyoon1131:fix/tcu-tbuf-tag-wrap

Conversation

@eyoon1131

@eyoon1131 eyoon1131 commented Sep 18, 2026

Copy link
Copy Markdown

Bug

TcuTbuf keys its in-flight request map with the unmasked tag counter,
while the response carries the tag masked to kSubTagMask (0xFFFF).

On the request side (sim/simx/tcu/tcu_tbuf.cpp):

uint32_t sub_tag = buf.next_tag_++;   // unmasked, unbounded
uint32_t tag = pack_tag(s, sub_tag);  // masked to 16 bits on the wire
buf.inflight_[sub_tag] = addr;        // key stored UNMASKED

and on the way back:

uint32_t sub_tag = unpack_sub_tag(r.tag);  // returns MASKED
auto it = buf.inflight_.find(sub_tag);     // MASKED lookup
if (it != buf.inflight_.end()) { ... }     // silently no-ops on a miss

The first 65,536 requests on a source work because the two values agree.
Request 65,536 is stored under key 65536, but its response comes back
carrying 0, so the lookup misses and the branch falls through:
resident_ never receives the line and the inflight_ entry is never
erased. The TCU waits forever for data that cannot arrive.

The failure is silent with no error or abort. rtlsim is unaffected since its
tags are sized properly, so this is a simx/rtlsim divergence. It is
latent for any long-running simx workload with enough TCU traffic.

Fix

Mask at generation, so the key and the wire tag are the same value:

uint32_t sub_tag = (buf.next_tag_++) & kSubTagMask;

Cycle counts below the threshold are unchanged, so the fix is a no-op
for any workload that already completed.

Reproduction

On commit 7a8d94266:

./ci/blackbox.sh --driver=simx --app=sgemm_tcu_wg --args="-m 384 -n 384 -k 384"
Matrix Size Before After
-m256 -n256 -k256 PASS 57,255,723 cycles PASS 57,255,723 cycles
-m384 -n384 -k384 no completion PASS 193,335,491 cycles

384³ is the first tested size at which the shared B tile buffer crosses 65,536
requests. 512³ hangs as well. Note that 256³ completes 57M cycles without
trouble, so the threshold is a count of tile-buffer requests rather than
run length.

The tile buffer keyed its in-flight map with the unmasked tag counter
while the response carried the tag masked to kSubTagMask (0xFFFF):

    uint32_t sub_tag = buf.next_tag_++;   // unmasked, unbounded
    uint32_t tag = pack_tag(s, sub_tag);  // masked on the wire
    buf.inflight_[sub_tag] = addr;        // key stored unmasked

and on the way back:

    uint32_t sub_tag = unpack_sub_tag(r.tag);  // masked
    auto it = buf.inflight_.find(sub_tag);     // masked lookup
    if (it != buf.inflight_.end()) { ... }     // no-ops on a miss

The first 65536 requests on a source work because the two values agree.
Request 65536 is stored under key 65536 but its response comes back
carrying 0, so the lookup misses and the branch silently falls through:
resident_ never receives the line and the inflight_ entry is never
erased. The TCU waits forever for data that cannot arrive.

Mask at generation so the key and the wire tag are the same value.

rtlsim is unaffected, its tags being properly sized, so this
also showed up as a simx/rtlsim divergence.

The trigger is a count of tile-buffer requests, not a run length, so it
can be reached by sgemm_tcu_wg at input 384^3. Crosses 65536 requests
on the shared B tile buffer and hangs, while 256^3 stays under it and
completes 57M cycles.

Cycle counts below the threshold are unchanged, so the fix is a no-op
for any workload that was previously unaffected.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant