Cross-Chiplet Cache Coherence Engine
SystemVerilog UCIe/FDI MESI Protocol CDC Yosys
A directory-based MESI coherence bridge at the UCIe FDI flit boundary, connecting a Compute Die (1.0 GHz) and a Memory Die (1.1 GHz) running on asynchronous clock domains. The design enforces cache coherence at the hardware level across a disaggregated 2-chiplet system — handling cross-die read misses, write upgrades, snoop invalidations, and evictions through a pipelined FSM-based transaction engine.
graph TB
subgraph CDA ["Compute Die — clk_a (1.0 GHz)"]
CPU["CPU / Cache Agent"]
TXN["coh_txn_buf\n16-entry MSHR · per-entry FSM"]
SNP["coh_snoop_ctrl\nMESI decision FSM"]
end
subgraph CDB ["Memory Die — clk_b (1.1 GHz)"]
CDC["coh_cdc_bridge\nGray FIFO + 2-FF sync"]
DIR["coh_directory\n256-entry MESI directory"]
MEM["Memory Controller"]
end
CPU -->|"request"| TXN
TXN -->|"allocated txn"| SNP
SNP <-->|"UCIe FDI boundary · 128-bit flit"| CDC
CDC <-->|"coherence messages"| DIR
DIR <-->|"data / ack"| MEM
| Module | Role | Key Parameters |
|---|---|---|
cohesion_pkg.sv |
Shared types, enums, and packed structs | — |
cohesion_engine.sv |
Top-level wrapper, connects all submodules | — |
coh_cdc_bridge.sv |
Clock domain crossing — Gray-coded async FIFO + 2-FF synchronizers | FLIT_WIDTH=128, FIFO_DEPTH=8 |
coh_directory.sv |
256-entry MESI directory with per-chiplet presence bits and owner tracking | DIR_ENTRIES=256, ADDR_WIDTH=48 |
coh_txn_buf.sv |
16-entry in-flight request tracker with per-entry FSM and timeout counter | TB_ENTRIES=16, TIMEOUT_CYCLES=512 |
coh_snoop_ctrl.sv |
MESI state transition logic and snoop response controller | — |
stateDiagram-v2
direction LR
[*] --> INVALID
INVALID --> SHARED : Read miss\n(other sharers exist)
INVALID --> EXCLUSIVE : Read miss\n(sole requester)
INVALID --> MODIFIED : Write miss
SHARED --> INVALID : Invalidation from directory
SHARED --> MODIFIED : Write upgrade\n(invalidate all sharers)
EXCLUSIVE --> MODIFIED : Write hit\n(silent — no bus transaction)
EXCLUSIVE --> SHARED : Snoop read hit
MODIFIED --> SHARED : Snoop read hit\n(writeback to memory)
MODIFIED --> INVALID : Eviction / writeback
Each of the 16 transaction buffer entries runs its own instance of this FSM, allowing up to 16 in-flight cross-die requests simultaneously.
stateDiagram-v2
direction LR
[*] --> IDLE
IDLE --> LOOKUP_DIR : Request allocated to entry
LOOKUP_DIR --> SNOOP_SENT : Directory lookup complete
SNOOP_SENT --> WAIT_RESP : Snoop dispatched to peer
WAIT_RESP --> GRANT : Response received
GRANT --> COMPLETE : Grant issued to requester
COMPLETE --> IDLE : Entry freed
WAIT_RESP --> IDLE : Timeout — deadlock guard
All 6 directed tests pass under mismatched clock domains (1.0 GHz / 1.1 GHz).
| # | Test Scenario | Result | Notes |
|---|---|---|---|
| 1 | Cold read miss | ✅ PASS | 35-cycle cross-die grant latency @ 1.0 GHz |
| 2 | Write hit on Exclusive — silent upgrade | ✅ PASS | No snoop required |
| 3 | Write miss, new address → MODIFIED | ✅ PASS | — |
| 4 | Read from second address → EXCLUSIVE | ✅ PASS | — |
| 5 | Back-to-back reads, same address | ✅ PASS | No pipeline hang |
| 6 | 8 in-flight transactions pipelined | ✅ PASS | ~14 cycles/txn throughput |
MESI invariant checker: violation_count = 0 across all tests. Simulation terminates at $finish at 446,500 ps.
req_valid asserts HIGH → 35-cycle cross-die coherence handshake → grant_valid asserts with grant_state = 2 (EXCLUSIVE).
The 35-cycle latency breaks down as: ~3–4 cycles per FIFO crossing direction (Gray-coded pointer synchronization), with the remainder in FSM pipeline stages (LOOKUP_DIR → SNOOP_SENT → WAIT_RESP → GRANT).
Synthesized with Yosys 0.66, OSS CAD Suite. No PDK — Yosys internal cell library only. Timing analysis requires PDK-specific cell mapping and was not performed.
| Module | Cells | Notes |
|---|---|---|
coh_directory |
52,336 | 256 × 48-bit register file + 256-comparator unrolled tag-match tree |
coh_txn_buf |
7,415 | 16 entries × 6-state per-entry FSM |
async_fifo |
2,127 | 128-bit × 8-deep Gray-coded FIFO |
coh_snoop_ctrl |
571 | MESI decision logic |
coh_cdc_bridge |
1 | Thin wrapper |
ff2_sync |
2 | 2 DFFs per instance |
| Total | 64,612 | Full design hierarchy |
The directory dominates at 81% of total cells. The implementation uses a fully unrolled O(N=256) combinational scan — Yosys instantiates 256 parallel 42-bit tag comparators feeding a priority encoder. Of the 52,336 cells, 12,288 are $_DFFE_ flip-flops (the 256 × 48-bit storage); the remaining ~40K are the comparator tree. A hashed or set-associative directory structure would reduce this to roughly 3–5K cells. The unrolled approach was retained to keep the coherence protocol logic simple and directly verifiable.
The design crosses two independent PLL domains at the UCIe FDI boundary.
Multi-bit data path — Gray-coded async FIFO (async_fifo). Gray-code encoding ensures only one bit changes per pointer increment, which eliminates the multi-bit metastability problem that would occur if the binary read/write pointers were synchronized directly.
Unidirectional control signals — 2-FF synchronizer chain (ff2_sync) on every signal crossing the boundary. This provides at minimum two MTBF stages before any combinational logic consumes the signal.
There are no combinational paths between clock domains — every cross-domain signal is registered before use.
cohesion-engine/
├── rtl/ # Synthesizable RTL
│ ├── cohesion_pkg.sv
│ ├── cohesion_engine.sv
│ ├── coh_cdc_bridge.sv
│ ├── coh_directory.sv
│ ├── coh_txn_buf.sv
│ └── coh_snoop_ctrl.sv
├── tb/ # Directed testbenches
│ ├── tb_cohesion_engine.sv
│ ├── tb_coh_directory.sv
│ ├── tb_coh_txn_buf.sv
│ ├── tb_cdc_bridge.sv
│ └── tb_pkg_test.sv
├── synth/ # Synthesis artifacts
│ ├── synth_cohesion.ys
│ ├── synth_report.txt
│ └── cohesion_netlist.v
├── docs/
│ └── waveform_test1_cold_read.png
├── run_tools.bat
└── README.md
# Run from run_tools.bat (OSS CAD Suite environment)
iverilog -g2012 -I rtl -o sim/cohesion_sim.vvp \
rtl/cohesion_pkg.sv rtl/coh_cdc_bridge.sv rtl/coh_directory.sv \
rtl/coh_txn_buf.sv rtl/coh_snoop_ctrl.sv rtl/cohesion_engine.sv \
tb/tb_cohesion_engine.sv
vvp sim/cohesion_sim.vvpyosys synth/synth_cohesion.ys 2>&1 | tee synth/synth_report.txt| Tool | Version | Purpose |
|---|---|---|
| Icarus Verilog | 14.0 | RTL simulation |
| Yosys | 0.66+4 | Logic synthesis |
| GTKWave | — | Waveform viewing |
| OSS CAD Suite | — | Toolchain environment (Windows) |
B.E. ECE · M.S. Ramaiah Institute of Technology, Bangalore · 2023–2027