Conversation
The tile buffer keyed its in-flight map with the unmasked tag counter
while the response carried the tag masked to kSubTagMask (0xFFFF):
uint32_t sub_tag = buf.next_tag_++; // unmasked, unbounded
uint32_t tag = pack_tag(s, sub_tag); // masked on the wire
buf.inflight_[sub_tag] = addr; // key stored unmasked
and on the way back:
uint32_t sub_tag = unpack_sub_tag(r.tag); // masked
auto it = buf.inflight_.find(sub_tag); // masked lookup
if (it != buf.inflight_.end()) { ... } // no-ops on a miss
The first 65536 requests on a source work because the two values agree.
Request 65536 is stored under key 65536 but its response comes back
carrying 0, so the lookup misses and the branch silently falls through:
resident_ never receives the line and the inflight_ entry is never
erased. The TCU waits forever for data that cannot arrive.
Mask at generation so the key and the wire tag are the same value.
rtlsim is unaffected, its tags being properly sized, so this
also showed up as a simx/rtlsim divergence.
The trigger is a count of tile-buffer requests, not a run length, so it
can be reached by sgemm_tcu_wg at input 384^3. Crosses 65536 requests
on the shared B tile buffer and hangs, while 256^3 stays under it and
completes 57M cycles.
Cycle counts below the threshold are unchanged, so the fix is a no-op
for any workload that was previously unaffected.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
TcuTbufkeys its in-flight request map with the unmasked tag counter,while the response carries the tag masked to
kSubTagMask(0xFFFF).On the request side (
sim/simx/tcu/tcu_tbuf.cpp):and on the way back:
The first 65,536 requests on a source work because the two values agree.
Request 65,536 is stored under key
65536, but its response comes backcarrying
0, so the lookup misses and the branch falls through:resident_never receives the line and theinflight_entry is nevererased. The TCU waits forever for data that cannot arrive.
The failure is silent with no error or abort.
rtlsimis unaffected since itstags are sized properly, so this is a
simx/rtlsimdivergence. It islatent for any long-running
simxworkload with enough TCU traffic.Fix
Mask at generation, so the key and the wire tag are the same value:
Cycle counts below the threshold are unchanged, so the fix is a no-op
for any workload that already completed.
Reproduction
On commit
7a8d94266:./ci/blackbox.sh --driver=simx --app=sgemm_tcu_wg --args="-m 384 -n 384 -k 384"384³ is the first tested size at which the shared B tile buffer crosses 65,536
requests. 512³ hangs as well. Note that 256³ completes 57M cycles without
trouble, so the threshold is a count of tile-buffer requests rather than
run length.