diff --git a/CHANGELOG.md b/CHANGELOG.md index e73a77b8d..453777eae 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -114,6 +114,75 @@ hashes are given where a change is easiest to understand by reading the commit. the cost follows what the guest wrote. Reflinking is unchanged. On Windows the copy is marked sparse first. +### R10000 TLB widths + +- **Full 44-bit user addresses.** On the R10000 (Octane, IP28, and an O2 set to + `r10000`) EntryHi, the TLB match and the user-segment check keep VA bits 43:40; + they stopped at bit 39 before, so an address above 1 TiB took an Address Error. + The R4400 and R5000 are unchanged. +- **FrameMask (CP0 register 21)** on the R10000 clears the EntryLo bits it + names, 33:18, on every TLB write, so a kernel that keeps software flags there + gets the physical address it meant. The register read as zero and ignored + writes before. The R4400 and R5000 have none. +- **EntryLo** on the R10000 keeps its 28-bit PFN (bits 33:6, a 40-bit physical + address) and the uncached attribute in 63:62; it kept 30 bits before. The + attribute is stored and read back, not sent to the bus. +- **TLB writes on the R10000** store zeros in the PFN bits below the page size, + so TLBR reads them back as zero, as the part does; on 16 KB pages that is + EntryLo bits 7:6. Translation is unchanged. The PROM's `ide` `tlb` test + failed 320 compares on this. The manual documents it for the R10000; the + R12000, R14000 and R16000 do the same as a model choice. The R4400 and + R5000 still keep the whole PFN. +- **TLB refill vector on the R10000** follows the address space that missed: + a user address takes the XTLB vector only with UX set, xsseg with SX, and + kernel addresses with KX. A 64-bit kernel touching a 32-bit user address now + refills through 0x000, as the part does, not 0x080. The R4400 and R5000 still + choose by the current mode. + +### Refused uncached stores on the R10000 family + +- On the R10000 family (Octane, IP28, O2), an uncached store that nothing answers now + completes instead of raising a Data Bus Error, as the processor does. iris prints + "Unsupported uncached write ... (posted, completes)" for the first four such + stores and then at each power of two. Loads, instruction fetches, the R4400 and + the R5000 are unchanged. + +### Partial LWR on the R10000 + +- On the R10000 (Octane, IP28, and an O2 with the R10000), an `LWR` that loads fewer + than four bytes now keeps bits 63:32 of the destination instead of sign-extending, + as the reference results for the part show. R4400 and R5000 machines are unchanged. + The JIT matches the interpreter, and the code-cache key changes only for the + machines that have the behaviour. + +### R10000 cache operations + +- **Index Load Data and Index Store Data** on the R10000's primary caches keep their + data (they read back zero before). The secondary cache's data slot is the address + shifted right by 3 instead of 4: one slot per doubleword. +- **Secondary-cache tags** on the R10000 family read the reserved bits (TagLo 9, 12, 13 and + TagHi 30:4) as zero; the MRU bit and the other tag fields are kept. + +### R12000, R14000 and R16000 + +- `[machine] cpu` and `--cpu` take `r12000`, `r14000`, `r16000` (revision 2.2, + the 800 MHz part) and `r16000_700` (revision 2.1, the 700 MHz part). The Octane + takes the R10000, R12000 and R14000 (defaulting to 195, 360 and 600 MHz), the + O2 the R5000, R10000 and R12000 (400 MHz for the R12000), the IP28 the R10000. + The R16000 has no machine yet; `iris-bench` is its only user. CP0 Config now + reports the part's own secondary-cache size, where every part read 1 MB, and + each R16000 presents its own reset word. + +### SGI Tezro (IP53 board, IP35 software platform) + +- Added a headless Tezro profile with the 700 MHz R16000 (`r16000_700`, the default; `r16000` selects the 800 MHz part), 4 MiB cache, 4 GiB RAM, a PIC with PXBOW, and the IO9's IOC4, ISP12160 and BCM5701; see the [Tezro guide](docs/tezro-ip53.md). +- A Tezro can fit a VPro V12 at widget 12 with `[graphics] board = "v12"`, on the VPro model the Octane and the Fuel use. +- A Tezro fits one, two or four processors, four by default, and two or four bring a second PIC at widget 11 on the PXBOW's link 11. Every fitted processor runs, each on its own host thread, so a default four-processor Tezro runs four. +- A fresh flash store for a two-processor Tezro is seeded with the environment records that mark the empty B and D slots disabled, as QEMU's boot tests seed them; this is a model policy, not a factory image. +- The Tezro's PIC fails a configuration read of an empty slot as a master timeout, so the PROM no longer records two phantom devices per empty slot. +- The Tezro returns the V12's graphics credits to PI0 along the node's graphics map, settles owed credits on a map write, and stops PI0's graphics slot below its bias as the Bedrock does. +- The Tezro's PROM stops while copying itself to memory unless the R10000 cache index operations are present; see the [Tezro guide](docs/tezro-ip53.md). + ### `IRIS_GUI_GL` works in iris-gui - The GUI's GL capture path (`IRIS_GUI_GL=1`, or `[debug] gui_gl_capture`) never @@ -203,6 +272,35 @@ hashes are given where a change is easiest to understand by reading the commit. - Memory past 4 GB is as fast as the rest: such a machine gets a 16 GB direct window, and the JIT compiles code anywhere in it. - A store-conditional on memory outside the window is now atomic with two CPUs. +### SGI Fuel (IP34 board, IP35 software platform) + +- **The Fuel takes a VPro board**: `[graphics] board = "v12"` or `"v10"` + fits the Octane's VPro model at crossbar widget 13. IRIX 6.5.30 with the + VPro driver brings the X login screen up on it. + +- Added a headless Fuel profile with the 500 MHz R14000, 512 MiB RAM, + Bedrock, dual-head XBridge, IOC3 networking and two-channel ISP12160 SCSI. + The normal interpreter boot reaches the IRIX 6.5.30 root shell with PROM + 6.210. See the [Fuel guide](docs/fuel-ip34.md). +- **jitv2 compiles uncached code from the uncached view.** The PROM runs from the + reset window uncached; once memory covers that window the cached view returns + RAM there, and the compile pool used to compile RAM's bytes in place of the + PROM's. An uncached fetch now keys a code page of its own and is compiled and + invalidated through the view the CPU fetched it by. Other machines are + unchanged. +- Marking an ECC doubleword or a directory line in Bedrock's memory no longer + rebuilds the whole memory map. The PROM's memory test marks and lifts a few + hundred of them thousands of times. A mark now changes only its own page, + and a lifted mark's page stays on Bedrock's checking path until a batch + returns it. + +### PCI configuration + +- A PCI function can describe its own writable header fields and + write-one-to-clear status bits. Byte and halfword configuration writes + through a Bridge carry their byte enables, so such a function keeps the + status events a narrow write did not select. No Octane function has + write-one-to-clear status bits, so the Octane behaves as before. ### VNC, and screen and pointer commands @@ -513,6 +611,9 @@ hashes are given where a change is easiest to understand by reading the commit. ### SGI O2 (IP32) +- **The O2's memory alias holds for pages with high PTE bits set.** A mapped page + whose address had a bit set above bit 31 missed the alias of RAM at + 0x8000_0000 and reached the PCI window instead. - **The O2 runs on the shadow cache from reset.** The full cache model, and the switch to and from it during POST and reboots, are gone (`src/ip32_handover.rs`), and with them a branch on every cached load, store diff --git a/FEATURES.md b/FEATURES.md index 487c73b05..b11499d7a 100644 --- a/FEATURES.md +++ b/FEATURES.md @@ -170,15 +170,53 @@ the first `[section]`, because TOML tables continue until the next header. | Key | Default | Values and behavior | |---|---|---| -| `machine.profile` | `indy_ip24` | indy_ip24, indigo2_ip22, indigo2_ip28, o2_ip32 (configured by `[o2]`, see [the O2 guide](docs/o2-ip32.md)), octane_ip30 (configured by `[octane]`, see [the Octane guide](docs/octane-ip30.md)). | -| `machine.cpu` | `r4400` | r4400, r5000, r10000; CPU/ISA choice is independent of build features. An O2 takes r5000 or r10000 (an r4400 is refused when the machine is built); the Octane is r10000 only. | -| `machine.cpus` | `1` | Processors fitted. 2 only on the Octane; see [its guide](docs/Octane/multiprocessor.md). | +| `machine.profile` | `indy_ip24` | indy_ip24, indigo2_ip22, indigo2_ip28, o2_ip32 (configured by `[o2]`, see [the O2 guide](docs/o2-ip32.md)), octane_ip30 (configured by `[octane]`, see [the Octane guide](docs/octane-ip30.md)), fuel_ip34 (see [the Fuel guide](docs/fuel-ip34.md)), tezro_ip53 (configured by `[tezro]`, see [the Tezro guide](docs/tezro-ip53.md)). | +| `machine.cpu` | `r4400`; a Fuel with no `cpu` runs `r14000` and a Tezro `r16000_700` | r4400, r5000, r10000, r12000 (Octane, O2), r14000 (Octane, Fuel), r16000 (800 MHz, Tezro), r16000_700 (700 MHz, Tezro). CPU/ISA choice is independent of build features. A Fuel always runs its one R14000: set `r14000` (`r10000`, the family, is also accepted); any other value is refused. A Tezro runs an R16000 with a 4 MB secondary cache: `r16000_700` (the 700 MHz part, the default) or `r16000` (the 800 MHz part, four processors only); `r10000`, the family, takes the 700 MHz part. Any other value is refused, and so is the 800 MHz part with an 8 MB secondary cache (QEMU's `r16000-800mhz-8mb-rev22`), which has no model. The Tezro's clock follows the part. | +| `machine.cpus` | `1`; `4` on the Tezro | Processors fitted. 2 on the Octane; see [its guide](docs/Octane/multiprocessor.md). 1, 2 or 4 on the Tezro, each run on its own host thread; one selects the one-processor board with one PIC, two or four the board with two PICs. | + +### [fuel] + +Fuel-specific settings apply when `machine.profile = "fuel_ip34"`. + +| Key | Default | Values and behavior | +|---|---|---| +| `flash` | unset | Persistent 16 MiB boot-flash file. Unset, the flash is volatile. | +| `console` | `l1` | `l1`, `ioc3a` or `ioc3b`. Selects the guest console connection. | +| `l1_port` | `8882` | Host TCP port for the L1 console. Used when `console = "l1"`. | +| `l1_log` | unset | Optional serial log path for L1 output when an IOC3 port is selected as console. | +| `l1_revision` | `1.48.1` | L1 firmware revision as `major.minor.patch`, decimal. The guest reads it with L1 request `0x0005`. | +| `brick_serial` | `EMUN00` | L1 brick serial, at most 37 bytes. The guest reads it with L1 request `0x0004`. The default is a marked synthetic serial, not a real one. | + +The machine uses one 500 MHz R14000 and fixed 512 MiB RAM in two 256 MiB +banks. Its IOC3 has no keyboard or mouse ports, so it runs on a serial +console. A `[fuel]` table is written to a saved config only when one of +these keys is set. The sample configuration and supported PROM inputs are in the +[Fuel guide](docs/fuel-ip34.md). + +### [tezro] + +Tezro-specific settings apply when `machine.profile = "tezro_ip53"`. + +| Key | Default | Values and behavior | +|---|---|---| +| `flash` | unset | Persistent 16 MiB boot-flash file. Unset, the flash is volatile. | +| `console` | `l1` | `l1`, `ioc4_a`, `ioc4_b`, `ioc4_c` or `ioc4_d`. Selects the guest console connection. | +| `l1_port` | `8882` | Host TCP port for the L1 console. | +| `l1_log` | unset | Optional serial log path for L1 output when an IOC4 port is selected as console. | +| `serial_port_c`, `serial_port_d` | `8883`, `8884` | Host TCP ports of IOC4 ports 2 and 3; ports 0 and 1 use `serial_port_a` and `serial_port_b`. | + +The machine runs the 700 MHz R16000 (`r16000_700`, the default) or the 800 MHz +one (`r16000`), each with a 4 MiB secondary cache. `banks` +takes 512 or 1024 MiB banks from bank 0; left unset, it is the 4 GiB layout of +1024, 1024 and four 512 MiB banks. A `[tezro]` table is written to a saved +config only when one of these keys is set. See the +[Tezro guide](docs/tezro-ip53.md), which lists its limits. ### [graphics] | Key | Default | Values and behavior | |---|---|---| -| `graphics.board` | `newport`; `solidimpact` on the Octane | newport, xz, extreme, solidimpact, highimpact, maximpact, v6, v8, v10, v12, none. Extreme is Indigo2 only. The Octane takes only `solidimpact` (SI), `highimpact` (SI with texture option), `maximpact` (MXI), `v6`, `v8`, `v10`, `v12` (the VPro boards, also `vpro:v12` and so on; need an IRIX install with the VPro driver, IRIX 6.5.10 or later and PROM 4.0, or 4.5 for the V10 and V12, which shipped only in the Octane2 and so give the machine the Octane2's Crossbow 1.4 and Bridge revision) or `none` (no board: the machine runs on its serial console, with no window). | +| `graphics.board` | `newport`; `solidimpact` on the Octane | newport, xz, extreme, solidimpact, highimpact, maximpact, v6, v8, v10, v12, none. Extreme is Indigo2 only. The Octane takes only `solidimpact` (SI), `highimpact` (SI with texture option), `maximpact` (MXI), `v6`, `v8`, `v10`, `v12` (the VPro boards, also `vpro:v12` and so on; need an IRIX install with the VPro driver, IRIX 6.5.10 or later and PROM 4.0, or 4.5 for the V10 and V12, which shipped only in the Octane2 and so give the machine the Octane2's Crossbow 1.4 and Bridge revision) or `none` (no board: the machine runs on its serial console, with no window). The Fuel takes `v12` or `v10` in its graphics slot, or `none`, its default. The Tezro takes `v12` at widget 12, or `none`, its default. | | `graphics.heads` | `1` | 1 or 2; dual-head requires Newport. | | `graphics.resolution` | `guest` | guest, 1024x768, 1280x960, 1280x1024; presets require Newport. | @@ -209,7 +247,7 @@ An explicit non-disk kind takes precedence over the legacy cdrom flag. | Key | Default | Values and behavior | |---|---|---| -| `network.mode` | `nat` | nat, pcap, stream or off (no Ethernet; O2 and Octane only). Stream needs `network.stream`. PCAP needs build support and host capture permissions. PCAP works on every Ethernet; stream on every onboard one, not on a DaynaPort. | +| `network.mode` | `nat` | nat, pcap, stream or off (no Ethernet; O2, Octane and Fuel only). Stream needs `network.stream`. PCAP needs build support and host capture permissions. PCAP works on every Ethernet; stream on every onboard one, not on a DaynaPort. | | `network.stream` | `unset` | `tcp:IP:PORT` or `unix:PATH`: the server of the private segment for `mode = "stream"` (QEMU `-netdev stream` framing). See NETWORKING.md. | | `network.pcap_interface` | `auto-pick when unset` | Interface name or numeric list index; PCAP only. | | `network.nfs_pcap_ip` | `unset` | Virtual LAN IP for the built-in NFS responder in PCAP mode. | @@ -373,8 +411,8 @@ IP22/IP24 four-bank IRIX 6.5 behavior remains separate from IP28's verified two-bank 1 GB layout. Extreme requires IP22. All non-Newport boards require `heads=1` and `resolution="guest"`. IP28 IRIX needs IMPACT; its PROM has an embedded fallback. -The config validator permits SCSI controller 1 only on IP22, the O2 and the -Octane, the machines with two controllers. +The config validator permits SCSI controller 1 only on IP22, the O2, Octane +and Fuel. These machines have two controllers. Standalone CLI options update the loaded config where supplied; boolean enable switches do not provide a general way to clear every saved true setting. @@ -427,7 +465,7 @@ iris-gui does not parse this standalone option set; use its named-machine UI. | `--list-net-interfaces` | Print the host network interfaces libpcap can bridge onto, then exit. Requires a build with --features pcap. | | `--no-scsi-deferred-int` | Disable deferred SCSI status interrupts (default: enabled for OpenBSD/NetBSD compatibility). | | `--cpu` | Emulated CPU: `r4400` (default), `r5000`, or `r10000`. Overrides `[machine] cpu`. | -| `--graphics` | Graphics board: `newport` (default), `xz`, `extreme`, `solidimpact`, `highimpact`, `maximpact` (or `impact:solid`, `impact:high`, `impact:max`), `v6`, `v8`, `v10`, `v12` (`vpro:v12` and so on, Octane only). | +| `--graphics` | Graphics board: `newport` (default), `xz`, `extreme`, `solidimpact`, `highimpact`, `maximpact` (or `impact:solid`, `impact:high`, `impact:max`), `v6`, `v8`, `v10`, `v12` (`vpro:v12` and so on; Octane, and `v10`/`v12` on the Fuel). | | `--gdb-port` | Enable GDB stub on the given TCP port (e.g. --gdb-port 1234). Connect with: target remote localhost: | | `--monitor-port` | Monitor console port on 127.0.0.1 (default 8888). Give each iris running at once its own. | | `--vnc [ADDR]` | Serve the screen, keyboard and mouse over VNC on ADDR (`host:port` or a port; default 127.0.0.1:5900). Sets `vnc.listen`. | diff --git a/HACKING.md b/HACKING.md index e62f2ecae..253e4114c 100644 --- a/HACKING.md +++ b/HACKING.md @@ -16,12 +16,19 @@ Other documents: ## 1. What IRIS is -IRIS emulates five SGI workstations: the Indy (IP24), the Indigo2 (IP22), the -Indigo2 IMPACT (IP28), the O2 (IP32) and the Octane (IP30). The CPU is an -R4400, R5000 or R10000, chosen at runtime (the Octane takes the R10000 only). -Graphics is Newport (REX3), GR2 (XZ/Extreme) or IMPACT on the Indy and -Indigo2, GBE on the O2 and Solid IMPACT on the Octane. It boots the real PROMs -and IRIX 5.3 and 6.5 to a desktop with networking. +IRIS emulates the Indy (IP24), Indigo2 (IP22), Indigo2 IMPACT (IP28), O2 +(IP32), Octane (IP30), Fuel (IP34 board, IP35 software platform) and Tezro +(IP53 board, IP35 software platform). The CPU is an R4400, R5000, R10000, +R12000, R14000 or R16000 (two parts), chosen at runtime (the IP28 takes the +R10000; the Octane the R10000, R12000 or R14000; the O2 the R5000, R10000 or +R12000; the Fuel its 500 MHz R14000; the Tezro its 700 or 800 MHz R16000). Graphics is Newport +(REX3), GR2 (XZ/Extreme) or IMPACT on the Indy and Indigo2, GBE on the O2, and +Solid IMPACT on the Octane. Fuel has no graphics board and runs headless on +its serial console. It does have a PS/2 keyboard and mouse on its IOC3, fed +from the host's input and the default as in the Fuel's manual, and `[fuel] +usb_input = true` adds a USB keyboard and mouse on its OHCI controller; +`ps2_input = false` turns PS/2 off. The graphical machines boot their real +PROMs and IRIX 5.3 or 6.5 to a desktop with networking. It is **not cycle-accurate**, anywhere. IRIX is interrupt-driven and does not care how many cycles an instruction takes, so IRIS models behaviour @@ -82,21 +89,32 @@ image. A CD-ROM's node is a `ChangerNode`: the queue of discs and the open active one, with the `Removable` trait (present, changed, eject, load) separate from `BlockNode` so that removable media which is not a block device, a tape, can use it too. The WD33C93A (Indy, Indigo2), the Adaptec -7880 (O2) and the ISP1040 (Octane) each make an empty bus of their own kind -(`new_bus`); the machine fills it from config and gives it to the controller's -constructor, so a controller exists with its targets and keeps only its -registers and DMA. What a `BlockNode` must guarantee about durable writes, -and why a disk's `write_cache` setting matters, is in the "Disk write cache" -section of [STORAGE.md](STORAGE.md) and the `block_node.rs` module docs: read -them before writing a backend or changing a flush. `src/scsi_monitor.rs` implements the monitor's `scsi` and `cow` +7880 (O2) and ISP1040 (Octane) each make an empty bus of their own kind +(`new_bus`). Fuel uses an ISP12160 with two SCSI buses behind one PCI +function. The machine fills each bus from config and gives it to the controller +constructor, so the controller keeps its registers and DMA. What a `BlockNode` +must guarantee about durable writes, and why a disk's `write_cache` setting +matters, is in the "Disk write cache" section of [STORAGE.md](STORAGE.md) and +the `block_node.rs` module docs: read them before writing a backend or changing +a flush. `src/scsi_monitor.rs` implements the monitor's `scsi` and `cow` subcommands and the run-time disc changes once, over the bus a controller hands -out through `ScsiTargets::scsi_bus`; chip-specific subcommands (`regs`; the -WD's `wdt`, `defer`, `debug`) stay with the chip, and `DiscChanger` is -implemented once for every `ScsiTargets`. Configuration is shared the same +out through `ScsiTargets::scsi_bus`. Chip-specific subcommands stay with the +chip, including `regs` and the WD's `wdt`, `defer` and `debug`. `DiscChanger` +is implemented once for every `ScsiTargets`. Configuration is shared the same way: `scsi::attach_from_config` walks the `[scsi.N]` tables for every machine, where the key is a slot (controller × 16 + id, see `config::scsi_slot`), and the machine supplies only a route from controller number to bus. +`ScsiTarget::bus_reset` defaults to no action. Disk targets record reset as +unit attention. Only the ISP12160 calls it; the other controllers keep +their earlier behaviour of not reporting resets to targets. +The ISP12160 defers target resets and releases its controller state lock +before walking the bus. It calls targets while holding the bus lock, so a +target callback must not take that lock again. + +Flash command engines share `FlashBacking` file storage while retaining +their own command state. + ## 2. Data path and endianness **Word-transparent architecture.** Host `u32`/`u64` values are bit @@ -165,11 +183,15 @@ screenshots), `CpuDevice` (what `Machine` holds for the CPU). Serial lines have two halves. A UART model owns the registers: the Indy's `Z85c30`, and `Ns16550` (`src/dev/ns16550.rs`), the 16550A core that the O2's `MaceUart` (lane decode, interrupt line, monitor) and the Octane's IOC3 wrap. +The IOC3 and IOC4 serial ring-DMA engine is one module, `ioc_serial_dma.rs`; each +chip implements its `Host` trait (its 16550s, interrupt bits and bus master). The host end is a `SerialBackend` from `src/dev/serial_host.rs`, which the machine constructors pick with `console_backends`: the CI backend, a TCP console, or a log tee. A UART's transmit is instant and its receive side pulls from the backend with back-pressure (`pump`), so nothing in the model -needs a thread of its own beyond the service loop's call. +needs a thread of its own beyond the service loop's call. A backend that can +be full overrides `try_send_byte`: the 16550 then holds the byte, reads busy, +and the service loop's `poll_tx` retries it for two seconds. ### The physical address map @@ -300,9 +322,14 @@ Every device that has real work to do runs on its own thread: | `VNC-Grab`, `VNC-Pointer`, `VNC-Client`/`VNC-Read` | with `--vnc`: frame diffing, pointer steering, one pair per viewer | | `machine-events` | power-off and reset requests from devices | | `ISP1040-0`, `ISP1040-1` | Octane SCSI request rings and disk transfers | -| `IOC3-RX-A`, `IOC3-RX-B`, `IOC3-NVRAM` | Octane serial input per port; the DS1685 NVRAM file writer (IOC3's other deadlines are `hptimer` one-shots) | +| `ISP12160-0` | Fuel SCSI request ring for both ISP12160 channels | +| `IOC3-RX-A`, `IOC3-RX-B`, `IOC3-NVRAM` | Octane and Fuel IOC3 serial input and NVRAM writer (IOC3's other deadlines are `hptimer` one-shots) | +| `l1-serial` | Fuel and Tezro L1 console UART worker. The Fuel's other serial ports are IOC3 A and B | +| `IOC4-RX-0` to `IOC4-RX-3` | Tezro IOC4 serial input | +| `bcm5701-rx`, `bcm5701-net` | Tezro BCM5701 receive pump and network backend | | `ip30-eth-rx`, `ip30-eth-net` | Octane ethernet receive pump; the network backend | | `odyssey-work`, `odyssey-raster`, `odyssey-display` | VPro: the front end decodes the queued command-FIFO stores (the processor's stores only queue), the raster unit draws what the front end sets up, and the display copies the planes out, counts the retrace and presents the window at 60 Hz. No board-wide lock: the front end owns its state, an access that needs the drawing is answered busy until it is done, and the display reads the planes while they are drawn | +| `ioc3-eth-rx`, `ioc3-eth-net` | Fuel IOC3 Ethernet receive pump and network backend | Threads that have nothing to do park instead of spinning: the graphics consumers when their FIFO is empty, and the CPU thread in `idle-pause` builds @@ -371,9 +398,13 @@ consumer thread never takes `rec`. handler sets the PC itself, including delay slots. - The CPU model is the `CpuModel` trait (`MIPS4`, `PRID`, `FIR`, `TLB_ENTRIES`, ...), implemented by each cache model. The executor is - monomorphised over it, so all three CPUs are in every binary with no + monomorphised over it, so every CPU is in every binary with no per-model branch on the hot path. `Machine::new` picks one from `[machine] cpu`. +- Every uncached SB/SH goes to the bus through `BusDevice::write_narrow_from` + with the whole source register: the byte or halfword is the low bits, the + other lanes are undefined by hardware, and only Bedrock's ECC view reads + them. - With `opcodefusion` (implied by `lightning`), the interpreter executes common pairs (branch + NOP, LUI + ORI/ADDIU, address + load/store) as one dispatch. @@ -558,6 +589,22 @@ code into host code with Cranelift. The concepts: pfn map publishes a slot after its claim, and inline stores bump generations atomically. Compiled code is freed only by a full flush, and every flush parks all CPUs first (`jitv2_flush_from_here`, `run_leader_flush`). +- **Views of a frame.** On most machines a physical frame has one set of bytes, + so a page is keyed by frame alone. The IP35 bus decodes an uncached fetch + (attribute 0) differently from a cached one: the reset PROM answers the + uncached view at 0x1fc00000 whatever RAM sits behind the cached view there. + `MipsExecutor::enable_jitv2_uncached_views` makes an uncached fetch key + `frame | UNCACHED_VIEW_PFN` (`jitv2::code_key`). Such a page is read, and its + generation counter taken, through `UNCACHED_VIEW_ADDR` on the compile bus + (`Ip35Bus::code_view`), which reaches the same device the CPU's uncached fetch + reached. A write bumps the counter of the device that holds the bytes it changed: + a cached store to RAM over the reset window leaves the PROM's page alone, a flash + command on the PROM invalidates it, and where both views reach the same RAM + they share its counter. Frames past the end of the machine's memory and uncached + attributes 1 to 3 still run in the interpreter. The view bit (bit 22) is above + every frame of every machine (the largest physical window is 16 GiB), so a + machine that does not opt in keeps its true frame numbers whatever its memory; + one that does has a frame map of the view bit plus its memory's frames. - Compiled pages can optionally be cached on disk across runs (`[jitv2] cache`, [docs/jitv2-persistent-cache.md](docs/jitv2-persistent-cache.md)). @@ -581,6 +628,23 @@ introspection commands are in [debug.md](debug.md#jit-v2). Crossbow and the Bridge with IOC3 and two ISP1040 SCSI controllers, an R10000, and IMPACT graphics (see the [Octane guide](docs/octane-ip30.md) for the boards). +- **IP34 board, IP35 software platform (Fuel)**: one 500 MHz R14000, 512 MiB + in two 256 MiB banks, Bedrock, two XBridge heads, IOC3 and a two-channel + ISP12160. It runs headless on the L1 or IOC3 serial console. The IOC3 uses + the shared `net::Link` for NAT or PCAP. The writable Bedrock flash and + serial-flash devices share `FlashBacking` storage, with separate command + handling. A VPro board (the Octane's Odyssey model) can sit at crossbar + widget 13; its packets and DMA go through the XBridge and Bedrock's inbound + path, and the PI graphics registers are stored but count no credits. + Snapshots are unsupported. See the [Fuel guide](docs/fuel-ip34.md). +- **IP53 board, IP35 software platform (Tezro)**: its own board + (`src/ip53_machine.rs`) on the Bedrock node it shares with the Fuel + (`ip35::node`): one, two or four R16000s (700 MHz by default), each run on its own host + thread, memory from 512 and 1024 MiB banks (4 GiB by default), and a PIC + whose PXBOW is widget 0, carrying the IO9's IOC4, ISP12160 and BCM5701; + the two- and four-processor board adds a second PIC at widget 11. It runs + headless on the L1 or an IOC4 serial console. See the + [Tezro guide](docs/tezro-ip53.md). The Indy and Indigo2 profiles each have an embedded PROM fallback; `src/platform.rs` and `src/machine.rs` wire their differences, and diff --git a/HELP.md b/HELP.md index 8b2494b9c..1fba781cb 100644 --- a/HELP.md +++ b/HELP.md @@ -1,7 +1,9 @@ -# IRIS — SGI Indy and Indigo2 emulator +# IRIS — SGI workstation emulator The complete core/GUI feature and configuration inventory is in -[FEATURES.md](FEATURES.md). This guide explains everyday use; the monitor's full command reference is in [debug.md](debug.md). +[FEATURES.md](FEATURES.md). For the Fuel, see the [Fuel guide](docs/fuel-ip34.md); for the Tezro, the [Tezro guide](docs/tezro-ip53.md). +This guide explains everyday use. The monitor's full command reference is in +[debug.md](debug.md). ## Quick start @@ -620,7 +622,7 @@ scsi_deferred_int = true # needed by OpenBSD/NetBSD; see --no-scsi-deferr [machine] profile = "indy_ip24" # "indigo2_ip22" (--ip22), "indigo2_ip28", or "o2_ip32" (an [o2] section: docs/o2-ip32.md); --machine NAME picks any -cpu = "r4400" # "r5000" or "r10000" (--cpu); use R10000 on IP28 +cpu = "r4400" # "r5000", "r10000", "r12000", "r14000", "r16000" or "r16000_700" (--cpu); IP28: r10000, Octane: r10000/r12000/r14000, O2: r5000/r10000/r12000, Tezro: r16000_700 (default)/r16000/r10000 # cpus = 2 # Octane only: a second processor (docs/Octane/multiprocessor.md) [graphics] @@ -787,7 +789,7 @@ Configuration --machine Machine to emulate (a [machine] profile name); `help` lists them --ip22 Same as --machine indigo2_ip22 --graphics newport, xz, extreme, solidimpact, highimpact, maximpact, v6, v8, v10, v12 - --cpu r4400 (default), r5000, or r10000 + --cpu r4400 (default), r5000, r10000, r12000, r14000, r16000 or r16000_700 --bank0..--bank3 RAM bank sizes (0/8/16/32/64/128; IP28: also 256/512) --clock-fixed-mhz CP0 Count frequency (IP22/IP24: 33; IP28: 97.5) diff --git a/bench/README.md b/bench/README.md index ab7380fb0..f347de119 100644 --- a/bench/README.md +++ b/bench/README.md @@ -328,6 +328,11 @@ for each cell, because a cell's features live in its binary. | `r5000-jitv2` | `jitv2` | `r5000` | | `r4400-lightning` | `lightning` | `r4400` | | `r4400-jitv2-lightning` | `jitv2,lightning` | `r4400` | +| `r10000-interp`, `r12000-interp`, `r14000-interp`, `r16000-interp`, `r16000-700-interp` | (default) | `r10000`, `r12000`, `r14000`, `r16000`, `r16000_700` | + +The R10000 family always runs the shadow caches, so these share the default +build. No shipped machine takes the R16000 yet; the Indy and Indigo2 +(non-IP28) profiles build any part, and the cell relies on that. Each build is copied to `bench/build/iris-` before the next one starts — the next `cargo build` overwrites `target/release/iris`, and a matrix that races @@ -397,7 +402,7 @@ device on real hardware, so results come back over serial with CP0 Count as the time base; the header says so. **On which SGI machines?** The suite identifies any MIPS CPU by name from PRId -— R4000, R4400, R4600, R4700, R5000, R8000, R10000, R12000, R14000, RM5200, +— R4000, R4400, R4600, R4700, R5000, R8000, R10000, R12000, R14000, R16000, RM5200, RM7000 — and an implementation it has no name for prints as `MIPS-imp-0xNN` rather than refusing. That is safe to do because **nothing here is CPU-specific**: the kernels are ordinary compiled MIPS III, and `golden.h` is diff --git a/bench/harness/main.c b/bench/harness/main.c index 75adb4bf3..db21a1b12 100644 --- a/bench/harness/main.c +++ b/bench/harness/main.c @@ -222,7 +222,9 @@ static void run_one(const struct bench *b) * The CPU by name, from PRId alone — so this is right on any MIPS machine, * not only on the two the emulator models. The revision split on imp 0x04 is * the standard one: an R4000 and an R4400 report the same implementation and - * are distinguished by major revision. + * are distinguished by major revision, and so are an R14000 and an R16000, + * which share implementation 0x0f (Linux's cpu-probe.c splits them above major + * revision 2). * * An imp we do not have a name for still runs; it just prints as "MIPS imp * 0xNN", which is more use to whoever is holding that machine than "unknown". @@ -239,7 +241,7 @@ static const char *cpu_name(void) case IMP_R10000: return "R10000"; case IMP_R4300: return "R4300"; case IMP_R12000: return "R12000"; - case IMP_R14000: return "R14000"; + case IMP_R14000: return PRID_REV_MAJOR(cpu_prid) > 2 ? "R16000" : "R14000"; case IMP_R8000: return "R8000"; case IMP_R4600: return "R4600"; case IMP_R4700: return "R4700"; diff --git a/bench/prebuilt/PROVENANCE b/bench/prebuilt/PROVENANCE index 235efddb8..7c17e5a1e 100644 --- a/bench/prebuilt/PROVENANCE +++ b/bench/prebuilt/PROVENANCE @@ -4,6 +4,6 @@ # CI compares it and fails if the image was not refreshed alongside a # source change. The ELF's own hash is informational only — two correct # builds of identical source differ, because -g records the build path. -sources sha256 b2aa6db09ad6f42b68754e4199c66e0802e8cbc9f3c886e6a2c92b1f3502bf67 -image sha256 21dc4eade5b3326afcf77199b405767a56775168c0682e65c9ba5134253b857a 308876 bytes +sources sha256 b4ac6a3acf1fcfc617cee7153d901e01b364e20df8ec6b97565ea863435218e7 +image sha256 8976472764da1a05af725c4494a15523c8266e4d71c1cee43a682a7379679170 308960 bytes inputs 32 files diff --git a/bench/prebuilt/irisbench.elf b/bench/prebuilt/irisbench.elf index 12916e07b..bcf00a61e 100755 Binary files a/bench/prebuilt/irisbench.elf and b/bench/prebuilt/irisbench.elf differ diff --git a/clanker/INDEX.md b/clanker/INDEX.md index 3c9f1ab18..0f5bbc960 100644 --- a/clanker/INDEX.md +++ b/clanker/INDEX.md @@ -43,6 +43,7 @@ CPU architecture: CP0, TLB and its fast paths, caches, FPU, LL/SC, CPU models. B | [r5000-taglo-is-the-r4400-pstate.md](cpu/r5000-taglo-is-the-r4400-pstate.md) | rule | R5000 L1 TagLo uses the R4400 PState encoding | | [r5k-l1i-cache-bugs.md](cpu/r5k-l1i-cache-bugs.md) | open | R5000 secondary cache: no working configuration currently exists | | [sc-succeeds-on-llbit-alone.md](cpu/sc-succeeds-on-llbit-alone.md) | rule | `SC` succeeds on the LLbit alone — `LLAddr` is not a gate | +| [staged-secondary-cache-lines.md](cpu/staged-secondary-cache-lines.md) | rule | Staged secondary-cache lines hit until an explicit op ends them; the page fast paths must be told | | [tcache-discards-dirty-data-semantics.md](cpu/tcache-discards-dirty-data-semantics.md) | rule | tcache makes L1-D write-through, so Hit_Invalidate cannot discard | | [tcache-l2-instr-coherency.md](cpu/tcache-l2-instr-coherency.md) | rule | tcache: L2 decoded-instruction coherency, and how it was broken | | [tlb-c-field-uses-the-models-table.md](cpu/tlb-c-field-uses-the-models-table.md) | rule | The TLB decodes EntryLo C with the CPU model's table; C=4 read as uncached sent every Octane user access to the bus | diff --git a/clanker/cpu/staged-secondary-cache-lines.md b/clanker/cpu/staged-secondary-cache-lines.md new file mode 100644 index 000000000..007e888c6 --- /dev/null +++ b/clanker/cpu/staged-secondary-cache-lines.md @@ -0,0 +1,100 @@ +# Staged cache lines: a hit until an explicit operation ends them + +The R10000-family shadow cache (`src/cpu/mips_cache_shadow.rs`) serves a cached +load, store or fetch from a secondary line the guest staged with Index Store Tag +(S) and Index Store Data (S): a valid state in TagLo[11:10] and a tag equal to +PA[39:18] of the access. A guest that never stages a line pays one load and a +not-taken branch per access. + +## The limit: no replacement + +An ordinary cached miss never fills the shadow, so it never evicts. A staged +line stays until Index Writeback Invalidate, Hit Invalidate, Hit Writeback +Invalidate, an Index Store Tag that overwrites it, or power-on. A guest that +reuses the memory without invalidating reads the stale line and its stores stay +in the line, where hardware would have evicted it, writing a dirty one back to +the tag's address. + +Staged lines left at the login prompt, `l2 probe 0` in the monitor, CPU 0: +Octane (IP30 PROM, IRIX 6.5.30) 0, Fuel (PROM 6.210, IRIX 6.5.30, jitv2) 0, +O2 R10000 (IRIX 6.5.30) 0. The shipped PROMs invalidate what they stage, so the +`staged_any()` branch is not taken once IRIX runs. + +## Primary instruction-cache lines + +Index Store Tag (I) and Index Store Data (I) stage a primary I line the same +way: tag TagLo[31:8] = PA[35:12], TagLo[6] valid, TagHi[3:0] = PA[39:36]; index +VA[13:6], way VA[0], one word per Index Store Data at VA[13:2], 16 per block. A +cached fetch checks both ways at its VA[13:6] against its PA, before the +secondary cache. Index Invalidate (I), Hit Invalidate (I), an Index Store Tag +with the valid bit clear, and power-on end the line. Loads are not served by it. +Model choices: the compare on a fetch is inferred and a hit does not touch LRU. + +The stored word is the 36-bit predecoded instruction plus even parity, not an +instruction in TagLo. A fetch hit decodes it with a layout inferred from +the words a shipped PROM stages, for sampled opcodes only (`l1i_decode`, `L1I_CLASSES`). `addiu $4,$0,2` is `0x3_4420_0004`. +- A word of a class not in the table is refused: reported once on stderr, and the + fetch is treated as a miss and reads the secondary cache or memory. Chosen over + a Cache Error because memory holds the instruction a word was built from in all + but a few stale entries, so a miss executes the right instruction for nearly + every staged word, where an error stops a guest the hardware runs. It can hide + a stale entry of an unsampled class, which is why it is reported. +- The class table is the 11 sampled classes (0x00 nop, 0x08 dsll/dsll32, + 0x18 or, 0x1a addi/addiu/ori/lui, 0x1e daddiu, 0x35 cache, 0x36 ld, 0x3f mtc0, + 0x40 jal, 0x41 jr/j, 0x71 REGIMM), all inferred. Class, p[28:26] and, for the + SPECIAL classes, the function field identify the instruction; a known class + with anything else is refused too. The Fuel PROM boots with no refusals. +- A cached fetch that misses the primary I cache and hits a staged secondary + line refills a 64-byte primary block from it (the only refill; ordinary + fetches are not tracked). The victim is an Invalid way before a Valid one, and + way 0 when both are valid: a model choice, the manual states no victim rule. The + words are stored in the raw encoding with even parity, the architectural word + carried in place of hardware's predecoded form. The block is tagged with the + fetch's PA, valid, so Index Load Tag/Data (I) return it and a readback stored + verbatim into another line executes. Invalidating a secondary block + invalidates the primary I lines inside it. +- Parity is checked on a fetch hit: even over the 36 bits and ECC[0]. A bad word + posts a Cache Error (CacheErr.ED) unless Status.DE or ERL is set. +- Index Load Data (I) returns the stored word and parity as stored, so a readback + stored back verbatim round-trips and executes. A refill is not modelled: a line + the guest never staged reads back as the shadow's contents, not as the + predecoded memory word. iris never refills a primary line, so there is no architectural word to carry; the readback of a staged word is the stored word. A word with p[35:32] = 0xf is this model's own encoding + of a raw instruction in p[31:0] (`L1I_RAW`), so a word outside the sampled set + can round-trip. Model choice; the hardware contents are unknown. + +The primary data cache is not served: its dirty lines write back to the secondary +cache and the state-modifier field has no model. + +## Why the fast paths must be told + +Translations are cached per page. The interpreter's and the JIT's inline loads +and stores reach RAM through the micro-TLB and ppmem's window without calling the +cache, and the JIT compiles code from memory. So: + +- `nutlb_fill` declines a page that holds a staged line (`MipsCache::page_staged`); +- `jitv2_track_pcp` leaves such a page to the interpreter; +- a CACHE instruction that changes the set of staged pages calls + `nanotlb_invalidate` (`take_staged_changed`), because entries made before it + do not know; +- the decode cache reads such a page word by word (`DecodeCache::set_bus_only`). + +Dropping any of the first three fails `a_compiled_load_hits_a_staged_line_not_the_window`, +`a_compiled_store_to_a_staged_line_stays_in_the_line` or +`a_staged_line_is_fetched_as_code_instead_of_the_compiled_page`; those run under +`jitv2` and under `lightning,jitv2`. + +## Inferred or absent + +- The TagLo word of a staged doubleword lands at the lower address + (`STAGED_TAGLO_AT_LOWER_ADDRESS`). Inferred from software, not measured. +- A store to a staged line makes it DirtyExclusive. NEC U10278EJ4V0UM p.246 + counts such requests; the manual does not state the transition. +- Every CACHE operand is translated first (exceptions: TLB Refill/Invalid, privilege Address Error, never TLB Modified). The secondary set, way and quadword come from the operand's physical address (`set_cache_op_pa`), the primaries' from its virtual address. An operand on an uncached page is undefined on the R10000 and goes ahead as if cached. +- Only the R10000 is documented. The R12000, R14000 and R16000 use the same + tag layout here, which is an inference. +- Not modelled: the predecode of unsampled classes, refills, primary data cache staged hits, ECC and check bits, the + virtual-index purge, the BadVAddr/Context/EntryHi values after a CACHE-raised TLB exception, Status.CH, the + tag rewrite on invalidate (only the state clears), coherence with other CPUs + and DMA. +- Staged lines are saved in snapshots (`save_cache_state`). The rest of the + shadow arrays, which only diagnostics read, are not. diff --git a/clanker/odyssey/gotchas.md b/clanker/odyssey/gotchas.md index 826d3672f..7d7db173b 100644 --- a/clanker/odyssey/gotchas.md +++ b/clanker/odyssey/gotchas.md @@ -40,7 +40,8 @@ Status: rule (2026-10-30). timing measured to better than that is noise. - **Model choices to remember.** The FIFO's depth is unknown in any unit. - The model clamps the level at 32,768 words, whatever the stored `0x6bX` + The model keeps 16,384 stores for + credits and clamps the level at 32,768 words, whatever the stored `0x6bX` values say, and uses a 16-word low-water threshold; `0x6b2` as the status-bit-15 threshold is a model choice too. The level is in words, a doubleword store counting two. The tests pin the model, not the diff --git a/clanker/odyssey/threads-and-locks.md b/clanker/odyssey/threads-and-locks.md index 0e2fc8c08..cd9485d26 100644 --- a/clanker/odyssey/threads-and-locks.md +++ b/clanker/odyssey/threads-and-locks.md @@ -37,7 +37,7 @@ apply to BUZZ). Nothing models more than one raster unit. - **Command FIFO** (`fifo.rs`, CPU to `odyssey-work`): the processor's stores to the two command-FIFO ports and the context-switch stores, each - as (offset, size, value). The level the guest sees + as (offset, size, value) with its credit state. The level the guest sees is in words, against the model's 32,768. - **Raster queue** (`raster_unit.rs`, `odyssey-work` to `odyssey-raster`): jobs, each owning what its raster work reads, captured when the front @@ -116,8 +116,8 @@ other, as MGRAS's producers take its `submit` lock (below). drawing. A processor access to a host-interface register does not take it. - **`HostInterface`** is behind the front lock, a register file's own lock held for a few instructions: identity, control, the interrupt bank, - the routing, the swap bank and the FIFO status, plus the interrupt state. - A front access (those registers) takes only it + the routing, the swap bank and the FIFO status, plus the interrupt state + and the credit return. A front access (those registers) takes only it and never waits for a draw. It is a leaf; the raster thread takes it for a fence's interrupt, and the display takes it to latch swaps. - **The raster unit's state** (`RasterCtx`: the stipple phase and the rows @@ -197,15 +197,15 @@ the room check and the push, is refused, and its store retries the push behind t (`BoardLocks::write_command`), not around it. The ring is deeper than the board's FIFO, so a second producer's overshoot of the level by a store is absorbed. The consumer (the worker, or a draining access with the worker stopped) looks at up to 32 stores, runs them, and takes the ones it ran out together -(`CommandFifo::take_run`, `retire`): the level and the store count -move once a run, not once a store, because each store costs the +(`CommandFifo::take_run`, `retire`): the level, the store count and the +credits move once a run, not once a store, because each store costs the worker little and the processor reads those words. A store that has run but not retired still counts in the level, so the level never reads below what the -board has left to take. A push counts its store (level, store count) -before the ring shows it and takes the counts back off if the ring +board has left to take. A push counts its store (level, store count, counted +stores) before the ring shows it and takes the counts back off if the ring refuses it: the worker can take a store and retire it the instant it lands, before the push returns, and a retire must find every store it takes already -counted or the level goes below zero. +counted or the level goes below zero. While a counted store waits a run is one store. While the worker runs every command store queues, so decoding and drawing never run on the processor's thread. @@ -225,6 +225,13 @@ A caller other than the bus, with the board's thread stopped, still runs the queue and then its store. The flow-control target word `+0x1074` is stored, not routed, so a store to it goes in at once even while the FIFO holds work. +Counted stores (machines whose processor interface counts graphics credits; +the Octane has none and the calls are no-ops) are credited as they enter the +FIFO's first 16,384 places (the model's depth, a model choice), counted in +stores: at the store when there is room, +or when the stores ahead have run, with the node holding the credit as owed +meanwhile. + ## The FIFO ordering rules These are guest-visible, and hold across both queues; any change to either diff --git a/debug.md b/debug.md index 3f2dfb5fd..d75d0875c 100644 --- a/debug.md +++ b/debug.md @@ -367,6 +367,13 @@ widget's link, refused Crossbow accesses, and accesses to physical addresses nothing is mapped at, with the PC; a device's own refusal is not listed). The `phys` commands work on the Octane as well. +On the Fuel the one controller is a two-channel ISP12160: `scsi` (or +`scsi0`) and `cow` act on channel 0, `scsi1` and `cow1` on channel 1, and +ids run 0 to 15 on each (`[scsi.17]` is id 1 on channel 1). `scsi notes` +shows the whole chip's command total and anything odd the driver did, and +has no `scsi1` form; `regs`, `wdt`, `defer`, `debug` and `dayna` are not +there. + Queue position `[0]` is the mounted disc, `[1]` the next on eject. A disc change reaches IRIX as a Unit Attention ("medium changed") on its next TEST UNIT READY; IRIX's `mediad` may still need a manual remount @@ -522,7 +529,7 @@ a forced termination; a normal stop saves them automatically. |---|---| | `phys mem [n]` / `mm` | Dump physical memory | | `phys dis [n]` / `md` | Disassemble physical memory | -| `phys map` | The decode table: pages by kind (untouched, filled but not yet mapped, shared, private), memory in use, slots faulted in, and the table as runs of devices. Every machine on `Physical` has the `phys` commands (Indy, Indigo2, O2, Octane) | +| `phys map` | The decode table: pages by kind (untouched, filled but not yet mapped, shared, private), memory in use, slots faulted in, and the table as runs of devices. Every machine on `Physical` has the `phys` commands (Indy, Indigo2, O2, Octane, Fuel) | | `phys trace ` / `trace ` | Log physical bus accesses | | `phys error ` | Stop on bus errors | | `phys hole ` | Stop on accesses to unmapped addresses | @@ -541,9 +548,11 @@ Per-module logging; all of it needs a developer build to produce output. | `log file ` | Send a module's log to a file | Modules: `net hpc3 seeq hal2 mc rex3 mips ioc scsi pdma vino dcb vc2 cmap -xmap bt445 scc ps2 rtc eeprom l1i l1d l2c`. Mask categories: `pdma` — +xmap bt445 scc ps2 rtc eeprom l1i l1d l2c bedrock l1 usb`. Mask categories: `pdma` — `hal enet scsi`; `mips` — `insn tlb mem fpu cp0`; `l1i`/`l1d`/`l2c` — `hit miss op`; every module also takes `on`/`all`, `off`/`none` or a hex mask. +`bedrock` covers the Fuel's devices on its Bedrock hub, `l1` its system +controller and `usb` its USB host controller. `IRIS_DEBUG_LOG` or `[debug] debug_log` set the same thing at startup. ### Performance and locks diff --git a/docs/fuel-ip34.md b/docs/fuel-ip34.md new file mode 100644 index 000000000..90d6d677e --- /dev/null +++ b/docs/fuel-ip34.md @@ -0,0 +1,103 @@ +# SGI Fuel (IP34 board, IP35 software platform) + +Fuel is the IP34 board running the IP35 software platform. The emulated +configuration has one 500 MHz R14000 and 512 MiB of RAM in two fixed 256 MiB +banks. It includes Bedrock, two XBridge heads, an IOC3, and an ISP12160 with +two SCSI channels. The machine runs headless on its L1 or IOC3 serial console. + +## Configuration + +Copy [`fuel.toml.example`](../fuel.toml.example) to `fuel.toml`, provide the +PROM and disk files named there, then run `iris --config fuel.toml` from the +repository root. The sample selects `machine.profile = "fuel_ip34"` and +`machine.cpu = "r10000"`. The machine always runs its one R14000; `cpu` +must name the R10000 family, and any other value is refused. The sample +sets `banks = [256, 256]`, `headless = true`, `no_audio = true`, serial +ports 18880 and 18881, and `graphics.board = "none"`. The `[fuel]` settings +select the flash backing file, console, and optional L1 port and log. The +console options are `l1`, `ioc3a` and `ioc3b`. The Fuel's IOC3 has no +keyboard or mouse ports, so the console is always serial. + +The L1 answers firmware revision requests with 1.48.1 and brick serial +requests with the synthetic `EMUN00`; both are model choices, not values read +from a Fuel. `[fuel] l1_revision = "major.minor.patch"` and +`[fuel] brick_serial = "..."` replace them. Its system serial request returns +the machine's Ethernet address (`[network] mac`) as twelve uppercase +hexadecimal digits, the text the PROM converts back into that address. +IRIX's `numastatd` reads the revision at startup; without one it logs +`SNDRV_GET_ELSC_VERSION: I/O error`. + +The PROM decoder accepts a raw payload or a supported version-six, +single-segment stored or gzip container. The sample attaches a raw IRIX disk +at SCSI ID 1 and writes guest changes to its overlay. `[network] mode` defaults +to `nat`. `off` disconnects the link. `pcap` uses the shared `net::Link` backend +with a PCAP-enabled build and host capture support. The example sets `off`. +Set it to `nat` for the built-in gateway. + +## Observed boot and limits + +With the normal interpreter configuration, PROM 6.210 reports +`ALL DIAGS PASSED` and IRIX 6.5.30 has reached the root shell. The guest +reports a 500 MHz R14000 with 512 MiB, detects `ef0`, and passes the observed +NAT networking checks. During initial boot, IRIX's file indexer can keep the +guest busy while the serial console shows little activity. + +The PROM uses the R10000-family primary data cache as temporary memory before +it configures DRAM: it tags cache lines for the 32 KiB window +`0x100000`-`0x107fff` and installs them without a fill. The memory controller +models this from the cache's tag stores. PROM 6.210 stores, in order, test +patterns, an invalidate sweep, a valid tag (state 0xc0 or 0xc1) for each of the +window's eight pages, then an invalidate sweep. A valid tag for a page of the +window starts a private zero-filled page that answers the window on every path +(the cached path, uncached accesses and DMA), and keeps that 64 MiB region out +of the direct window; the first invalid tag ends it, and the window is memory +again. Power-on clears the page. This is the same mechanism the O2 uses for its +POST stack. Two things about it are approximated, not traced: the end is the +first invalid tag of the final sweep, not the last, which is right only if the +PROM does not use its stack between them; and the page answers uncached and DMA +accesses too, where the hardware has nothing behind the window to answer them. +A different PROM revision could tag another window or in another order. + +The ordinary interpreter configuration is the one with recorded Fuel boot +evidence. `lightning` and experimental `jitv2` builds are available, but no +Fuel guest boot with either is claimed here. + +## Graphics + +`[graphics] board = "v12"` (or `"v10"`) fits a VPro board in the graphics +slot, crossbar widget 13: a Fuel IRIX inventory lists its board under +xtalk/13, and Fuels have been reported with both the V12 and the V10. It is +the Octane's VPro model. Its interrupt packets cross the XBridge to Bedrock +like the PCI heads', its DMA enters Bedrock's inbound path as widget 13, and +a link reset of port 13 resets it. The processor interfaces' graphics +flow-control registers keep what IRIX writes but count no stores, so the +credit-starvation interrupt never fires. + +With PROM 6.210 the graphics diagnostic finds the board and passes. An IRIX +6.5.30 disk whose kernel includes the VPro driver lists `Graphics board: V12` +in `hinv` and brings the X login screen up on it. A disk installed for a +machine without graphics has no VPro driver in its kernel and runs as before. + +## Input + +The IOC3 carries a PS/2 keyboard and mouse, fed from host input; they are the +default, as in the Fuel's manual. `[fuel] usb_input = true` also plugs a USB +keyboard and mouse into the OHCI controller, and `ps2_input = false` leaves the +PS/2 pair out. Machine snapshots are unsupported. + +## Source basis + +The literal `0x53474972756c6573` is named `IIO_PROTECT_OVRRD_KEY` in public +Linux revision `db2ddb87143519e20a95aa36c60b36107b736a58`, +`arch/mips/include/asm/sn/sn0/hubio.h`. This establishes the literal's public +provenance, not the mapping of each Fuel register offset that uses it. Those +offsets remain approximate. The board composition port follows the project's +BSD-licensed QEMU SGI model in `hw/mips/sgi/sn1-node.c`. That source is a BSD-licensed implementation +reference, not proof of hardware behavior. + +The Hub-special head windows are compatibility approximations. Reads return +zero and writes are acknowledged without side effects. PROM 6.210 writes the +64-bit word `0x2000` at relative offsets `0x0` and `0x40` before discovering +DRAM; their hardware meaning remains unknown. Bedrock logging records +accesses. This keeps the existing model behavior explicit rather than +claiming a register implementation. diff --git a/docs/o2-ip32.md b/docs/o2-ip32.md index a1ee2daf5..0dba72c24 100644 --- a/docs/o2-ip32.md +++ b/docs/o2-ip32.md @@ -3,8 +3,8 @@ The O2 is SGI's unified-memory workstation: CRIME (the memory controller and rendering engine), MACE (the I/O chip: PCI, Ethernet, serial ports, PS/2, audio, the clocks) and GBE (the display engine, which scans the frame buffer -out of main memory). IRIS emulates it with an R10000 or an R5000, chosen at run -time like the other machines. +out of main memory). IRIS emulates it with an R5000, R10000 or R12000, chosen at +run time like the other machines. It runs IRIX 6.5 (tested with 6.5.22) to the desktop, with networking, audio, SCSI disks and CD-ROMs, X and OpenGL through the CRIME rendering engine, and @@ -28,7 +28,7 @@ serial_log = "serial.log" [machine] profile = "o2_ip32" -cpu = "r10000" # or "r5000" +cpu = "r10000" # or "r5000" or "r12000" [o2] control_port = 2293 @@ -65,14 +65,18 @@ in the file to silence the note. ## CPU, memory and clock -- **R10000 and R5000** both run the shadow cache model that IP28 uses, from +- **R5000, R10000 and R12000** all run the shadow cache model that IP28 uses, from reset: loads and stores go straight to memory. The PROM tests physical page 0 uncached while its stack sits in cache tags for that same page, so while those tags stand the bus answers for the page from a separate 4 KB page. - Linux supports only R5000-class O2s. - `[clock] fixed_mhz` is CP0 Count, 90 MHz unless set; the PROM measures and reports twice it as the CPU clock. R10000 O2s shipped at 175 and 195 MHz - (87.5 and 97.5), R5000s at 180 to 300. + (87.5 and 97.5), R5000s at 180 to 300. The R12000 (revision 3.5, 2 MB secondary + cache) shipped at 400 MHz; it defaults to 200 MHz Count (400 MHz). Its + secondary cache has 64-byte lines like the R10000's, because the PROM fails + with 128-byte ones. The 270 and 300 MHz revision 2.3 part (1 MB) is not + offered. - `banks` is the kit in each of the four DIMM pairs from A: 32, 64, 128 or 256 MB each, largest first, up to 1 GB. The default `[128, 128]` is 256 MB. IRIX boots with any of them, from a single 32 MB kit up. NetBSD's 32-bit diff --git a/docs/octane-ip30.md b/docs/octane-ip30.md index cc652a8e2..0798d29ee 100644 --- a/docs/octane-ip30.md +++ b/docs/octane-ip30.md @@ -59,9 +59,11 @@ overlay = true ``` Paths are relative to the directory iris runs in. Every key is listed in -[FEATURES.md](../FEATURES.md). `machine.cpu` must be `r10000`. `[clock] -fixed_mhz` sets CP0 Count; unset it is 97.5 MHz, half of the R10000's -195 MHz pipeline clock. +[FEATURES.md](../FEATURES.md). `machine.cpu` is `r10000` (revision 2.7), +`r12000` (revision 3.5, 2 MB secondary cache) or `r14000` (revision 2.4, 2 MB). +`[clock] fixed_mhz` sets CP0 Count, half the pipeline clock. Unset it is +97.5 MHz for the 195 MHz R10000, 180 MHz for a 360 MHz R12000 and 300 MHz for a +600 MHz R14000. ## PROM and flash diff --git a/docs/tezro-ip53.md b/docs/tezro-ip53.md new file mode 100644 index 000000000..a1fb8e57f --- /dev/null +++ b/docs/tezro-ip53.md @@ -0,0 +1,144 @@ +# SGI Tezro (IP53 board, IP35 software platform) + +The Tezro is the IP53 board running the IP35 software platform, the same +Bedrock node as the [Fuel](fuel-ip34.md) with a different interface board. +The emulated configuration has 700 MHz R16000 processors (or the 800 MHz +part) with 4 MiB secondary caches, a Bedrock node with 4 GiB of RAM by default, and an IO9 +board behind a PIC PCI host bridge: an IOC4 with four serial ports, a +DS1744-compatible clock and a PS/2 keyboard and mouse, an ISP12160 with two +SCSI channels, and a BCM5701 gigabit Ethernet controller. The machine runs +headless on its L1 or IOC4 serial console. + +## Configuration + +Copy [`tezro.toml.example`](../tezro.toml.example) to `tezro.toml`, provide +the PROM and disk files named there, then run `iris --config tezro.toml` from +the repository root. The sample selects `machine.profile = "tezro_ip53"` and +`machine.cpu = "r16000_700"`, the 700 MHz R16000 (revision 2.1), which is +also what the machine runs with no `cpu` line. `cpu = "r16000"` selects the +800 MHz part (revision 2.2), which a Tezro fits only as four processors, as +QEMU offers it (inferred from the assemblies sheet). Both have a 4 MB +secondary cache. The 800 MHz part with an 8 MB secondary cache (QEMU's +`r16000-800mhz-8mb-rev22`) has no model, since the cache size is fixed when a +CPU model is compiled, and is refused, as is every other processor. `r10000`, +the family, takes the 700 MHz part. The processor clock in the flash record +and the count rate (half the clock) follow the part; `[clock] fixed_mhz` +still sets the count rate. The secondary cache clock and the cache tap +(0xc for the 700 MHz part, 0xa for the 800 MHz one) are not modelled; the tap +appears only as part of the flash record's mode word. + +`[machine] cpus` is the number of processors fitted: 1, 2 or 4, with 4 the +default, as in the Tezros whose inventories have been seen. The count selects +the interface board. The one-processor board has one PIC at XIO widget 15 +carrying the IO9. The two- and four-processor board adds a second PIC at +widget 11, with nothing on its buses. It sits on link 11 of the IO9 PIC's +PXBOW, beside the hub's link 10 and the IO9's link 15, so the PROM and IRIX +find it through that crossbar's link status. Whether it has a crossbar of +its own in front of it is not known. Either board takes a V12 at widget 12 +with `[graphics] board = "v12"`. + +Memory comes from `banks`: one 512 or 1024 MiB bank per locational bank, from +bank 0 with no gaps, up to eight. Left unset, the machine fits the 4 GiB +layout of 1024, 1024 and four 512 MiB banks, a population a real Tezro has +been seen with. One processor needs at least 512 MiB, two or four at least +1 GiB. Banks 4 and 5 of the default sit above the 4 GiB physical address, +outside the processor's direct memory window, so they answer through the +slower decode table. + +The `[tezro]` settings also select the flash backing file, the console and +the L1 port and log. The console options are `l1` (the default) and `ioc4_a` +to `ioc4_d`, the IOC4's ports 0 to 3. IOC4 ports 0 and 1 listen on +`serial_port_a` and `serial_port_b`, ports 2 and 3 on `tezro.serial_port_c` +and `tezro.serial_port_d`. `nvram` names the 32 KiB DS1744 store; its clock +reference is kept beside it with a `.clock` suffix. + +## Firmware inputs + +The PROM is the IP35 PROM 6.210 payload, given raw or in a supported +version-six, single-segment container, as for the Fuel. A fresh flash is +formatted with the payload in the first 8 MiB position and its configuration +block replaced by the Tezro's own (mode word `0x589bad85`, 700 MHz, machine type 0, check +adjustment `0xda`), and the volume label in the second position. A flash +store that already exists keeps the configuration block it holds, so a block +the PROM rewrote survives the next start. The processor resets with the +matching Config `0x6c9bad85`. + +The mode word is a different field from the processor's Config: its low 25 +bits are the Config reset bits, and the bits above carry, among others, the +cache tap. With `cpu = "r16000"` the block carries the 800 MHz part instead: +the QEMU Tezro offering's mode word for it, `0x549baf85`, 800 MHz and check +adjustment `0xff`, and the processor resets with Config `0x6c9baf85`. No +flash profile writes this block; the adjustment keeps the 700 MHz block's +byte sum. A flash store keeps +the block it was formatted with, and the PROM checks its mode word against +the processor's, so give the other part a fresh `[tezro] flash` file. + +## Limits + +Every fitted processor runs, each on its own host thread. The IP35 PROM +assumes its processors left reset together and run at hardware speed +through four early handshakes; at each the model holds the processor that +arrived early until the ones it waits for catch up, or until they have +retired 2,000,000,000 cycles without doing so (`src/ip35/rendezvous.rs`). An +NMI, and a write that clears a running processor's enable bit, are refused +and logged by the Bedrock PI: the CPU has no NMI entry. + +On this Fuel base the PROM stops after printing `Copying PROM code to +memory` and never reaches its menu. Bank 0 covers the reset PROM window at +physical `0x1fc00000` in every Tezro layout, since the smallest bank is +512 MiB, and the PROM runs that stage from cache lines it stages with the +R10000 Index Store Tag and Index Store Data operations. Iris serves such +staged lines only with the R10000 cache index operations, which this base +does not carry. With them on top, the Origin 300 disk boots to the IRIX +login under the interpreter and under jitv2. + +The R16000 model leaves out three R16000-specific choices that QEMU's model +makes: the CP0 diagnostic register's mask and reset value, a trap on a +subnormal operand while FCSR.FS is clear, and the limit on CVT.L results. +Those cases follow the behaviour the rest of the R10000 family has in Iris. +No R16000's FIR has been published for either part; the model reports +`0x0900`, the R12000 and R14000 value, as a model choice. + +Every board identity the machine reports is a software choice of the QEMU +Tezro profiles, not a value measured on a Tezro: Bedrock board word `0x2001`, +L1 brick type `0x39` with revision 1.48.1 and the router `brick` reply, the +PIC's revision 3 profile, the PCI devices (IOC4 at device 1, ISP12160 at 3, +BCM5701 at 4), their revisions and their interrupt inputs, and REQ64# on the +BCM5701's slot. No EEPROM image is fitted for the IO9 board, the node board +or the BCM5701, because no published source gives their contents, and the PROM +warns at every power-on that the module's CPU configuration values mismatch. +PIC DMA to any widget other than Bedrock IO is refused. A configuration read +of an empty PCI slot fails as a master timeout; the error enable gating and +the treatment of device 0 there are inferred. + +The IRIX 6.5.30 disks tested so far were installed on an Origin 300, not a +Tezro, so their device numbering comes from that machine: the IOC4 serial +ports appear as `tty5` and `tty6`. Early in every boot IRIX prints +`NOTICE: ... Detected a link reset due to other partition reset NASID 0` +several times and then continues; what in the model provokes it has not +been found. + +`[graphics] board = "v12"` fits a VPro V12 at widget 12 behind the PXBOW, +on either board, using the VPro model the Octane and the Fuel use. On the +two- and four-processor board this is QEMU's graphics slot. On the +one-processor board the tower guide gives one XIO connector for a VPro board +but names neither its widget nor its route, so widget 12 there is inferred. +CPU 0's stores into its graphics page are counted against PI0's credits and +the V12 returns them along the node's graphics map. Credits the V12 owes are +settled when a store rewrites the map entry naming it. A counted store that +leaves PI0's counter below its bias raises the graphics interrupt and stops +the slot, as QEMU's Bedrock does: the next counted store is answered busy +until the processor takes the interrupt or credits lift the counter above +the bias, for at most a second. The interrupt counter is not modelled. +With the cache index operations on top and IRIX 6.5.30 with the VPro +software, the PROM's Odyssey diagnostics pass, xdm starts Xsgi on the V12, +and a root login at the greeter reaches the 4Dwm desktop. Without the V12 +the machine has no graphics. +Machine snapshots are unsupported. + +## Source basis + +The board composition follows the project's BSD-licensed QEMU SGI models in +`hw/mips/sgi/sn-machine.c` and `hw/mips/sgi/sn1-node.c` and their Tezro +profile layers. That source is an implementation reference, not proof of +hardware behavior. diff --git a/fuel.toml.example b/fuel.toml.example new file mode 100644 index 000000000..1c1325694 --- /dev/null +++ b/fuel.toml.example @@ -0,0 +1,44 @@ +# SGI Fuel (IP34 board, IP35 software platform), serial-only setup. +# PROM 6.210 and IRIX 6.5.30 have been observed reaching the root shell. +# Copy to fuel.toml and run from the repository root: iris --config fuel.toml +# Create ignore/fuel/ and supply your decoded raw PROM payload and IRIX disk. +# PROM also accepts supported version-six single-segment stored/gzip containers. +# No PROM or disk images are included. Convert QCOW2 disks to raw first. + +prom = "ignore/fuel/ip35prom.payload.bin" +nvram = "ignore/fuel/nvram.bin" +banks = [256, 256] +headless = true +no_audio = true +monitor_port = 18888 +serial_port_a = 18880 +serial_port_b = 18881 + +[machine] +profile = "fuel_ip34" +# Required: names the R10000 family. The machine always runs its one +# R14000 at 500 MHz with a 2 MiB secondary cache; other values are refused. +cpu = "r10000" + +[graphics] +# "v12" or "v10" fits a VPro in the graphics slot; IRIX needs a kernel with +# the VPro driver. The console stays on serial either way. +board = "none" + +[fuel] +flash = "ignore/fuel/flash.bin" +console = "l1" +l1_port = 18882 + +[network] +# Disconnected cable. For NAT, use mode = "nat". +# Keep the installed disk's working ef0 address and default route. +# NAT can adopt its static gateway before routing begins. +# Initial defaults: guest 192.168.0.2/24, gateway 192.168.0.1. +mode = "off" + +[scsi.1] +path = "ignore/fuel/irix.raw" +cdrom = false +# Keep the base disk unchanged. Guest writes go to irix.raw.overlay. +overlay = true diff --git a/iris-gui/src/config_ui.rs b/iris-gui/src/config_ui.rs index 891d37d2e..3f351238d 100644 --- a/iris-gui/src/config_ui.rs +++ b/iris-gui/src/config_ui.rs @@ -2035,10 +2035,10 @@ fn show_performance(ui: &mut Ui, cfg: &mut MachineConfig) -> ConfigAction { ui.horizontal(|ui| { ui.add_enabled_ui(octane, |ui| { ComboBox::from_id_salt("machine_cpus") - .selected_text(format!("{}", cfg.machine.cpus)) + .selected_text(format!("{}", cfg.machine.cpus())) .show_ui(ui, |ui| { for n in [1u8, 2] { - ui.selectable_value(&mut cfg.machine.cpus, n, format!("{n}")); + ui.selectable_value(&mut cfg.machine.cpus, Some(n), format!("{n}")); } }); }); diff --git a/src/bin/iris_bench.rs b/src/bin/iris_bench.rs index d26f4a3cd..d8779315a 100644 --- a/src/bin/iris_bench.rs +++ b/src/bin/iris_bench.rs @@ -270,6 +270,13 @@ const CELLS: &[Cell] = &[ Cell { name: "r5000-jitv2", features: "jitv2", cpu: "r5000", expect_cpu: "R5000" }, Cell { name: "r4400-lightning", features: "lightning", cpu: "r4400", expect_cpu: "R4400" }, Cell { name: "r4400-jitv2-lightning", features: "jitv2,lightning", cpu: "r4400", expect_cpu: "R4400" }, + // The R10000 family. It always runs the shadow caches, so the default + // build covers it. + Cell { name: "r10000-interp", features: "", cpu: "r10000", expect_cpu: "R10000" }, + Cell { name: "r12000-interp", features: "", cpu: "r12000", expect_cpu: "R12000" }, + Cell { name: "r14000-interp", features: "", cpu: "r14000", expect_cpu: "R14000" }, + Cell { name: "r16000-interp", features: "", cpu: "r16000", expect_cpu: "R16000" }, + Cell { name: "r16000-700-interp", features: "", cpu: "r16000_700", expect_cpu: "R16000" }, ]; fn build_cell(cell: &Cell, root: &Path, force: bool) -> Result { @@ -1077,12 +1084,20 @@ mod tests { #[test] fn every_cell_name_is_unique_and_maps_to_a_cpu() { for (i, a) in CELLS.iter().enumerate() { - assert!(a.expect_cpu == "R4400" || a.expect_cpu == "R5000"); + assert!(["R4400", "R5000", "R10000", "R12000", "R14000", "R16000"].contains(&a.expect_cpu)); // `expect_cpu` is what the `matrix` guard compares the guest's // `#machine cpu=` line against, so it must describe the `--cpu` // this cell actually passes. CPU is a runtime flag (see `Cell::cpu`), // not a cargo feature, so the two agreeing is the whole contract. - let want = match a.cpu { "r4400" => "R4400", "r5000" => "R5000", c => panic!("{}: unknown --cpu {}", a.name, c) }; + let want = match a.cpu { + "r4400" => "R4400", + "r5000" => "R5000", + "r10000" => "R10000", + "r12000" => "R12000", + "r14000" => "R14000", + "r16000" | "r16000_700" => "R16000", + c => panic!("{}: unknown --cpu {}", a.name, c), + }; assert_eq!(a.expect_cpu, want, "{}", a.name); for b in CELLS.iter().skip(i + 1) { assert_ne!(a.name, b.name); } } diff --git a/src/bin/jitv2_analyze.rs b/src/bin/jitv2_analyze.rs index aba0b247e..5b2ca948d 100644 --- a/src/bin/jitv2_analyze.rs +++ b/src/bin/jitv2_analyze.rs @@ -108,8 +108,8 @@ fn main() { }; let entry_word = entry.offset / 4; - let page_base = entry.pfn.wrapping_mul(iris::cpu::jitv2::PAGE_SIZE); - let (result, non_empty) = analyzer.walk(&page, entry_word, page_base); + let page_base = iris::cpu::jitv2::jitv2::key_phys_base(entry.pfn); + let (result, non_empty) = analyzer.walk(&page, entry_word, page_base as u32); stats.files_walked += 1; if entry_word == 1 { diff --git a/src/bin/jitv2_pcp_dump.rs b/src/bin/jitv2_pcp_dump.rs index 06f7765f0..1868d5c25 100644 --- a/src/bin/jitv2_pcp_dump.rs +++ b/src/bin/jitv2_pcp_dump.rs @@ -76,7 +76,7 @@ fn main() { } }; - let page_base_phys = dump.pfn * PAGE_SIZE; + let page_base_phys = iris::cpu::jitv2::jitv2::key_phys_base(dump.pfn); println!("=== {} ===", path); println!("pfn={:#010x} phys_base={:#010x}", dump.pfn, page_base_phys); println!("current_gen={} entry_gen={} {}", diff --git a/src/config.rs b/src/config.rs index 1ddc8fce1..2ad3e116b 100644 --- a/src/config.rs +++ b/src/config.rs @@ -612,6 +612,12 @@ pub enum MachineProfile { /// R10000. Configured by the shared keys and the `[octane]` section; see /// `src/ip30_machine.rs` and `docs/Octane/ip30-address-map.md`. OctaneIp30, + /// Serial-only Fuel IP34 board using the IP35 software platform. + FuelIp34, + /// Serial-only Tezro: the IP53 board on the IP35 software platform, + /// configured by the shared keys and the `[tezro]` section; see + /// `src/ip53_machine.rs`. + TezroIp53, } impl MachineProfile { @@ -624,6 +630,17 @@ impl MachineProfile { Self::IndyIp24, Self::Indigo2Ip22, Self::Indigo2Ip28, Self::O2Ip32, Self::OctaneIp30, ]; + /// The processor a `[machine]` with no `cpu` key gets: the part the + /// product shipped with where the profile fixes it, else the generic + /// default. + pub fn default_cpu(self) -> CpuModel { + match self { + Self::FuelIp34 => CpuModel::R14000, + Self::TezroIp53 => CpuModel::R16000Mhz700, + _ => CpuModel::default(), + } + } + /// The bank sizes in MB (0 is an empty bank) that `validate()` accepts /// for this profile. The Octane's banks are matched DIMM pairs of 64 MB /// to 2 GB; the IP22/IP24 MC stops at 128 MB. An O2's banks are DIMM-pair @@ -634,6 +651,8 @@ impl MachineProfile { Self::OctaneIp30 => &[0, 64, 128, 256, 512, 1024, 2048], Self::O2Ip32 => &[0, 32, 64, 128, 256], Self::Indigo2Ip28 => MC_BANK_SIZES, + Self::FuelIp34 => &[0, 256], + Self::TezroIp53 => &[0, 512, 1024], _ => &[0, 8, 16, 32, 64, 128], } } @@ -657,12 +676,14 @@ impl MachineProfile { /// Every profile, in the order `workstation::build` lists them. `ALL` is /// only the ones the GUI offers. - pub const EVERY: [Self; 5] = [ + pub const EVERY: [Self; 7] = [ Self::IndyIp24, Self::Indigo2Ip22, Self::Indigo2Ip28, Self::O2Ip32, Self::OctaneIp30, + Self::FuelIp34, + Self::TezroIp53, ]; /// The name `[machine] profile` and `--machine` give this profile. @@ -673,6 +694,8 @@ impl MachineProfile { Self::Indigo2Ip28 => "indigo2_ip28", Self::O2Ip32 => "o2_ip32", Self::OctaneIp30 => "octane_ip30", + Self::FuelIp34 => "fuel_ip34", + Self::TezroIp53 => "tezro_ip53", } } @@ -689,16 +712,21 @@ impl MachineProfile { Self::EVERY.iter().map(|p| format!("{:<14}{}\n", p.name(), p.label())).collect() } - /// The processors fitted at most: two on the Octane, one elsewhere. + /// The processors fitted at most: four on the Tezro, two on the Octane, + /// one elsewhere. pub fn max_cpus(self) -> u8 { - if self == Self::OctaneIp30 { 2 } else { 1 } + match self { + Self::TezroIp53 => 4, + Self::OctaneIp30 => 2, + _ => 1, + } } /// Whether this profile can have `cpus` processors, and why not if it /// cannot. pub fn accepts_cpus(self, cpus: u8) -> Result<(), String> { if cpus != 1 && self.max_cpus() == 1 { - Err(format!("[machine] cpus = {cpus}: only the Octane has more than one processor")) + Err(format!("[machine] cpus = {cpus}: only the Octane and the Tezro have more than one processor")) } else if !(1..=self.max_cpus()).contains(&cpus) { Err(format!("[machine] cpus = {cpus}: IRIX runs an Octane with 1 or 2")) } else { @@ -707,12 +735,26 @@ impl MachineProfile { } /// Whether this profile can have `cpu`, and why not if it cannot. The - /// Octane builds only the part below; the O2 constructor still refuses - /// its own, after `validate()`. + /// Octane takes an R10000, R12000 or R14000, the IP28 an R10000 and the + /// Fuel an R14000; the O2 constructor still refuses its own, after + /// `validate()`. pub fn accepts_cpu(self, cpu: CpuModel) -> Result<(), String> { match self { - Self::OctaneIp30 if cpu != CpuModel::R10000 => { - Err("an Octane has an R10000 ([machine] cpu = \"r10000\")".into()) + Self::OctaneIp30 if !matches!(cpu, CpuModel::R10000 | CpuModel::R12000 | CpuModel::R14000) => Err( + "an Octane has an R10000, R12000 or R14000 ([machine] cpu = \"r10000\", \"r12000\" or \"r14000\")".into(), + ), + Self::Indigo2Ip28 + if matches!(cpu, CpuModel::R12000 | CpuModel::R14000 | CpuModel::R16000 | CpuModel::R16000Mhz700) => + { + Err("an Indigo2 IMPACT (IP28) has an R10000 ([machine] cpu = \"r10000\")".into()) + } + // The machine always models its one R14000; `r10000` names the + // family, as the key did before the R14000 could be named. + Self::FuelIp34 if !matches!(cpu, CpuModel::R10000 | CpuModel::R14000) => { + Err("a Fuel has an R14000: set [machine] cpu = \"r14000\"".into()) + } + Self::TezroIp53 if !matches!(cpu, CpuModel::R10000 | CpuModel::R16000 | CpuModel::R16000Mhz700) => { + Err(TEZRO_CPU_REFUSAL.into()) } _ => Ok(()), } @@ -725,6 +767,8 @@ impl MachineProfile { Self::Indigo2Ip28 => "SGI Indigo2 IMPACT (IP28)", Self::O2Ip32 => "SGI O2 (IP32)", Self::OctaneIp30 => "SGI Octane (IP30)", + Self::FuelIp34 => "SGI Fuel (IP34)", + Self::TezroIp53 => "SGI Tezro (IP53)", } } @@ -734,7 +778,7 @@ impl MachineProfile { /// connector), one on the Indy. pub fn scsi_controllers(self) -> u8 { match self { - Self::Indigo2Ip22 | Self::O2Ip32 | Self::OctaneIp30 => 2, + Self::Indigo2Ip22 | Self::O2Ip32 | Self::OctaneIp30 | Self::FuelIp34 | Self::TezroIp53 => 2, _ => 1, } } @@ -743,35 +787,45 @@ impl MachineProfile { /// buses are wide, with ids 0 to 15 and the controller on 0. The Indy's /// and Indigo2's WD33C93A chips are narrow, with ids 0 to 7. The O2's /// Adaptec buses are wide too, but stay at 7 until they are widened on - /// purpose. + /// purpose. The Fuel's and the Tezro's ISP12160 buses are wide. pub fn max_scsi_id(self) -> u8 { match self { - Self::OctaneIp30 => 15, + Self::OctaneIp30 | Self::FuelIp34 | Self::TezroIp53 => 15, _ => 7, } } - /// How many memory banks the machine has: four on all of them (the MC's - /// four banks; the O2's four DIMM pairs). `banks` has room for more. + /// How many memory banks the machine has: four on the MC, HEART and O2 + /// (the O2's four DIMM pairs), two on Fuel, eight on Tezro. `banks` has + /// room for more; unused entries must be zero. pub fn memory_banks(self) -> usize { - 4 + match self { + Self::FuelIp34 => 2, + Self::TezroIp53 => MAX_BANKS, + _ => 4, + } } pub fn guinness(self) -> bool { matches!(self, Self::IndyIp24) } - /// The CPU models this machine can be built with. The O2 has an R5000 or - /// an R10000 (an R4400 is refused when the machine is built, not by - /// `validate()`); the Octane shipped only with the 195 MHz R10000; the - /// Indy and the Indigo2 take any of the three. + /// The CPU models this machine can be built with. The O2 has an R5000, + /// an R10000 or an R12000 (the others are refused when the machine is + /// built, not by `validate()`); the Octane has an R10000, R12000 or + /// R14000; the Fuel has an R14000 (or the R10000 family key); the Tezro + /// one of its two R16000 parts (or the R10000 family key); the Indy and the + /// Indigo2 take any part, except that the IP28 is refused anything but the + /// R10000 by `validate()`. /// /// Single source of truth for the front-ends' Processor pickers, so they /// cannot offer a CPU the machine will refuse. pub fn cpus(self) -> &'static [CpuModel] { match self { - Self::OctaneIp30 => &[CpuModel::R10000], - Self::O2Ip32 => &[CpuModel::R5000, CpuModel::R10000], + Self::OctaneIp30 => &[CpuModel::R10000, CpuModel::R12000, CpuModel::R14000], + Self::O2Ip32 => &[CpuModel::R5000, CpuModel::R10000, CpuModel::R12000], + Self::FuelIp34 => &[CpuModel::R10000, CpuModel::R14000], + Self::TezroIp53 => &[CpuModel::R16000Mhz700, CpuModel::R16000, CpuModel::R10000], _ => &CpuModel::ALL, } } @@ -811,18 +865,18 @@ pub enum GraphicsBoard { /// VPro V8: BUZZ A, 128 MB. Octane only. #[clap(name = "v8", alias = "vpro:v8", alias = "vpro_v8")] V8, - /// VPro V10: BUZZ B, 32 MB. Octane only. It shipped only in the Octane2 - /// and needs Crossbow 1.4, so the machine takes the Octane2's Crossbow and - /// Bridge revisions. + /// VPro V10: BUZZ B, 32 MB. Octane and Fuel. On the Octane it shipped + /// only in the Octane2 and needs Crossbow 1.4, so the machine takes the + /// Octane2's Crossbow and Bridge revisions. #[clap(name = "v10", alias = "vpro:v10", alias = "vpro_v10")] V10, - /// VPro V12: BUZZ B, 128 MB. Octane only. It shipped only in the Octane2 - /// and needs Crossbow 1.4, so the machine takes the Octane2's Crossbow and - /// Bridge revisions. + /// VPro V12: BUZZ B, 128 MB. Octane, Fuel and Tezro. On the Octane it + /// shipped only in the Octane2 and needs Crossbow 1.4, so the machine + /// takes the Octane2's Crossbow and Bridge revisions. #[clap(name = "v12", alias = "vpro:v12", alias = "vpro_v12")] V12, /// No graphics board: the machine runs on its serial console. Octane - /// only. + /// and Fuel. #[clap(name = "none")] None, } @@ -884,6 +938,8 @@ impl GraphicsBoard { pub fn supports(self, profile: MachineProfile) -> bool { match profile { MachineProfile::OctaneIp30 => self.is_impact() || self.is_vpro() || self == Self::None, + MachineProfile::FuelIp34 => matches!(self, Self::V10 | Self::V12 | Self::None), + MachineProfile::TezroIp53 => matches!(self, Self::None | Self::V12), MachineProfile::Indigo2Ip22 => !self.is_vpro() && self != Self::None, _ => !self.is_vpro() && !matches!(self, Self::Extreme | Self::None), } @@ -893,6 +949,7 @@ impl GraphicsBoard { pub fn default_for(profile: MachineProfile) -> Self { match profile { MachineProfile::OctaneIp30 => Self::SolidImpact, + MachineProfile::FuelIp34 | MachineProfile::TezroIp53 => Self::None, _ => Self::Newport, } } @@ -952,7 +1009,7 @@ impl std::fmt::Display for GraphicsBoard { #[serde(deny_unknown_fields)] pub struct GraphicsSection { /// Graphics board: Newport, GR2 XZ/Extreme, IMPACT Solid/High/Maximum, - /// or none (Octane only). Unset: the machine's default + /// or none (Octane and Fuel). Unset: the machine's default /// ([`GraphicsBoard::default_for`]); read it with /// [`MachineConfig::graphics_board`]. Extreme is restricted to IP22, /// the Octane to IMPACT or none; non-Newport boards require one head @@ -1037,16 +1094,41 @@ pub enum CpuModel { /// the Indigo2 IMPACT (IP28). The machine uses shadow cache arrays; /// see `docs/indigo2-ip28.md` and `src/cpu/mips_cache_shadow.rs`. R10000, + /// MIPS R12000: the R10000's programming model. The Octane and the O2 + /// have it as revision 3.5 with a 2 MB secondary cache. + R12000, + /// MIPS R14000 with a 2 MB secondary cache: revision 2.4 on the Octane, + /// 2.3 on the Fuel. Selectable on those two. + R14000, + /// MIPS R16000, revision 2.2, 800 MHz part, with a 4 MB secondary cache. + /// Selectable on the Tezro. `iris-bench --cpu r16000` also runs it + /// through the Indy and Indigo2 (non-IP28) profiles, which build any part. + R16000, + /// MIPS R16000, revision 2.1, 700 MHz part, with a 4 MB secondary cache: + /// the `r16000` model with the earlier revision's PRId and Config word. + /// The Tezro's default. + #[serde(rename = "r16000_700")] + #[clap(name = "r16000_700")] + R16000Mhz700, } impl CpuModel { - pub const ALL: [Self; 3] = [Self::R4400, Self::R5000, Self::R10000]; + pub const ALL: [Self; 7] = + [Self::R4400, Self::R5000, Self::R10000, Self::R12000, Self::R14000, Self::R16000, Self::R16000Mhz700]; + + /// Whether the part is in the R10000 family. + pub fn is_r10000_family(self) -> bool { + matches!(self, Self::R10000 | Self::R12000 | Self::R14000 | Self::R16000 | Self::R16000Mhz700) + } pub fn label(self) -> &'static str { match self { Self::R4400 => "MIPS R4400", Self::R5000 => "MIPS R5000", Self::R10000 => "MIPS R10000", + Self::R12000 => "MIPS R12000", + Self::R14000 => "MIPS R14000", + Self::R16000 | Self::R16000Mhz700 => "MIPS R16000", } } } @@ -1062,7 +1144,7 @@ impl CpuModel { /// need no tag check against it. /// - `auto`: `shadow` in a `jitv2` build, `full` otherwise. /// -/// The R10000 (IP28) only has the shadow model. +/// The R10000 family only has the shadow model. #[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq, Default, clap::ValueEnum)] #[serde(rename_all = "snake_case")] #[clap(rename_all = "lower")] @@ -1077,7 +1159,7 @@ impl CacheModel { /// Whether `cpu` runs the shadow cache under this setting. pub fn is_shadow(self, cpu: CpuModel) -> bool { match (cpu, self) { - (CpuModel::R10000, _) => true, + (cpu, _) if cpu.is_r10000_family() => true, (_, Self::Full) => false, (_, Self::Shadow) => true, (_, Self::Auto) => cfg!(feature = "jitv2"), @@ -1095,18 +1177,28 @@ pub struct MachineSection { pub cpu: CpuModel, #[serde(default)] pub cache: CacheModel, - /// Processors fitted. More than one only on the Octane. - #[serde(default = "default_cpus")] - pub cpus: u8, + /// Processors fitted, or the profile's default when unset: read it with + /// [`MachineSection::cpus`]. More than one only on the Octane and the + /// Tezro. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub cpus: Option, } -fn default_cpus() -> u8 { - 1 +impl MachineSection { + /// The processors fitted: `cpus` as set, or the profile's default, four + /// on the Tezro as in the Tezros whose inventories have been seen and + /// one elsewhere. + pub fn cpus(&self) -> u8 { + self.cpus.unwrap_or(match self.profile { + MachineProfile::TezroIp53 => 4, + _ => 1, + }) + } } impl Default for MachineSection { fn default() -> Self { - Self { profile: MachineProfile::default(), cpu: CpuModel::default(), cache: CacheModel::default(), cpus: default_cpus() } + Self { profile: MachineProfile::default(), cpu: CpuModel::default(), cache: CacheModel::default(), cpus: None } } } @@ -1624,6 +1716,161 @@ mod scsi_keys { } } +/// Fuel's independent serial connections. The selected console serves CI and serial_log. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum FuelConsole { + #[default] + L1, + Ioc3A, + Ioc3B, +} + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct FuelSection { + /// Persistent 16 MiB boot flash store; absent means volatile flash. + #[serde(default)] + pub flash: Option, + #[serde(default)] + pub console: FuelConsole, + /// L1 host console TCP port; IOC3 keeps serial_port_a and serial_port_b. + #[serde(default)] + pub l1_port: Option, + /// Separate L1 log when an IOC3 port is selected as console. + #[serde(default)] + pub l1_log: Option, + /// Fit the PS/2 keyboard and mouse controller in the onboard IOC3 and + /// feed it the host's keyboard and mouse. + #[serde(default = "default_fuel_input")] + pub ps2_input: bool, + /// Plug a USB keyboard and mouse into the OHCI root ports and feed them + /// the same host input. Off by default: the Fuel's manual and the PROM + /// treat PS/2 as the standard keyboard and mouse. + #[serde(default)] + pub usb_input: bool, + /// L1 firmware revision as "major.minor.patch", reported to request + /// 0x0005. Unset: 1.48.1. + #[serde(default)] + pub l1_revision: Option, + /// Brick serial reported to L1 request 0x0004, at most 37 bytes. Unset: + /// the marked synthetic `EMUN00`. + #[serde(default)] + pub brick_serial: Option, +} + +fn default_fuel_input() -> bool { true } + +impl Default for FuelSection { + fn default() -> Self { + Self { + flash: None, + console: FuelConsole::default(), + l1_port: None, + l1_log: None, + ps2_input: true, + usb_input: false, + l1_revision: None, + brick_serial: None, + } + } +} + +impl FuelSection { + /// Left out of a written config when nothing in it was set. + fn is_default(&self) -> bool { + self.flash.is_none() + && self.console == FuelConsole::default() + && self.l1_port.is_none() + && self.l1_log.is_none() + && self.ps2_input + && !self.usb_input + && self.l1_revision.is_none() + && self.brick_serial.is_none() + } +} + +/// The Tezro's serial connections: the L1 and the IOC4's four ports. The +/// selected console serves CI and serial_log. +#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, Serialize, Deserialize)] +#[serde(rename_all = "snake_case")] +pub enum TezroConsole { + #[default] + L1, + Ioc4A, + Ioc4B, + Ioc4C, + Ioc4D, +} + +impl TezroConsole { + /// The connection's index: 0 for the L1, then the IOC4's ports. + pub fn index(self) -> usize { + self as usize + } +} + +/// The processor counts a Tezro board fits. +pub const TEZRO_CPUS: [u8; 3] = [1, 2, 4]; + +/// Why a Tezro refuses a `[machine] cpu`: the parts it takes, and the one it +/// cannot model. +const TEZRO_CPU_REFUSAL: &str = "a Tezro has R16000s: set [machine] cpu = \"r16000_700\" (700 MHz, the default) \ + or \"r16000\" (800 MHz), both with a 4 MB secondary cache; the 800 MHz part with an 8 MB \ + secondary cache (QEMU's r16000-800mhz-8mb-rev22) is not modelled: the secondary cache size \ + is fixed when a CPU model is compiled"; + +/// The processors an 800 MHz Tezro fits. QEMU offers its 800 MHz Tezro with +/// four processors only, inferred from the assemblies sheet. +const TEZRO_800_MHZ_CPUS: u8 = 4; + +#[derive(Debug, Clone, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct TezroSection { + /// Persistent 16 MiB boot flash store; absent means volatile flash. + #[serde(default)] + pub flash: Option, + #[serde(default)] + pub console: TezroConsole, + /// L1 host console TCP port; IOC4 ports 0 and 1 keep serial_port_a and + /// serial_port_b. + #[serde(default)] + pub l1_port: Option, + /// Separate L1 log when an IOC4 port is selected as console. + #[serde(default)] + pub l1_log: Option, + /// Host TCP ports of IOC4 ports 2 and 3. + #[serde(default)] + pub serial_port_c: Option, + #[serde(default)] + pub serial_port_d: Option, +} + +impl Default for TezroSection { + fn default() -> Self { + Self { + flash: None, + console: TezroConsole::default(), + l1_port: None, + l1_log: None, + serial_port_c: None, + serial_port_d: None, + } + } +} + +impl TezroSection { + /// Left out of a written config when nothing in it was set. + fn is_default(&self) -> bool { + self.flash.is_none() + && self.console == TezroConsole::default() + && self.l1_port.is_none() + && self.l1_log.is_none() + && self.serial_port_c.is_none() + && self.serial_port_d.is_none() + } +} + /// Top-level machine configuration. /// /// Field order matters for TOML export: the `toml` serializer requires every @@ -1669,6 +1916,12 @@ pub struct MachineConfig { #[serde(default = "default_banks", with = "banks_serde")] pub banks: [u32; MAX_BANKS], + /// Whether `banks` was given (in the config file or by `--bank0..3`), as + /// opposed to left at the generic default, so that a value equal to the + /// default still counts as chosen. Only the Tezro reads it. + #[serde(skip)] + pub banks_set: bool, + /// Window scale factor (1 = native, 2 = 2× for HiDPI/4K). CLI --2x overrides this. #[serde(default = "default_scale")] pub scale: u32, @@ -1864,6 +2117,14 @@ pub struct MachineConfig { #[serde(default, skip_serializing_if = "OctaneSection::is_default")] pub octane: OctaneSection, + /// Serial console and flash storage for the Fuel. + #[serde(default, skip_serializing_if = "FuelSection::is_default")] + pub fuel: FuelSection, + + /// Processors, serial console and flash storage for the Tezro. + #[serde(default, skip_serializing_if = "TezroSection::is_default")] + pub tezro: TezroSection, + /// Host performance tuning (`[perf]` section). #[serde(default)] pub perf: PerfConfig, @@ -2050,6 +2311,7 @@ impl Default for MachineConfig { nvram: default_nvram(), nveeprom: default_nveeprom(), banks: default_banks(), + banks_set: false, scsi: default_scsi(), scale: default_scale(), nfs: None, @@ -2091,6 +2353,8 @@ impl Default for MachineConfig { jitv2: Jitv2Config::default(), o2: O2Section::default(), octane: OctaneSection::default(), + fuel: FuelSection::default(), + tezro: TezroSection::default(), perf: PerfConfig::default(), clock: ClockConfig::default(), rtc_offset: RtcOffset::default(), @@ -2140,6 +2404,12 @@ impl MachineConfig { } // The Octane has a PCI bridge with a free slot. MachineProfile::OctaneIp30 => Ok(()), + MachineProfile::FuelIp34 => Err( + "[virtio_serial] is not wired on the Fuel: its XBridge has no virtio-pci attachment".into(), + ), + MachineProfile::TezroIp53 => Err( + "[virtio_serial] is not wired on the Tezro: its PIC has no virtio-pci attachment".into(), + ), MachineProfile::O2Ip32 => Err( "[virtio_serial] is not wired on the O2 yet: the virtio-pci transport exists, but the \ MACE PCI bridge has no interrupt route for a third slot (see docs/virtio-serial.md)" @@ -2154,6 +2424,20 @@ impl MachineConfig { [self.serial_port_a.unwrap_or(a), self.serial_port_b.unwrap_or(b)] } + /// Parse a config file's text, remembering whether it listed `banks`, giving + /// a `[machine]` with no `cpu` its profile's processor, and folding in the + /// spellings older versions wrote. + fn parse_toml(text: &str) -> Result { + let mut cfg = toml::from_str::(text)?; + let table = toml::from_str::(text)?; + cfg.banks_set = table.contains_key("banks"); + if !table.get("machine").and_then(|m| m.get("cpu")).is_some() { + cfg.machine.cpu = cfg.machine.profile.default_cpu(); + } + cfg.fold_deprecated(); + Ok(cfg) + } + /// Load from `iris.toml` if it exists, otherwise return defaults. /// /// A *missing* file is fine (defaults are used). A file that exists but @@ -2177,11 +2461,8 @@ impl MachineConfig { let Ok(text) = std::fs::read_to_string(path) else { return Ok(Self::default()); }; - match toml::from_str::(&text) { - Ok(mut cfg) => { - cfg.fold_deprecated(); - Ok(cfg) - } + match Self::parse_toml(&text) { + Ok(cfg) => Ok(cfg), Err(e) => { let mut msg = format!("failed to parse {path}:\n{e}"); // Common footgun: backslashes in a double-quoted string (Windows @@ -2287,9 +2568,25 @@ impl MachineConfig { self.banks[..n].to_vec() } + /// The Tezro's memory: `banks` from locational bank 0, or the board's + /// 4 GiB default when `banks` was never set (the generic default names a + /// bank size no Tezro takes, so a `banks` equal to it is a mistake to + /// report, not a request for the default). + pub fn tezro_memory(&self) -> Result { + use crate::dev::bedrock_mem::MemoryPopulation; + if !self.banks_set && self.banks == default_banks() { + return Ok(MemoryPopulation::default()); + } + let n = self.banks.iter().rposition(|&b| b != 0).map_or(0, |i| i + 1); + if let Some(gap) = self.banks[..n].iter().position(|&b| b == 0) { + return Err(format!("banks: bank {gap} is empty but a later bank is populated; fill the banks from 0")); + } + MemoryPopulation::from_mib(&self.banks[..n]).map_err(|e| format!("banks: {e}")) + } + /// Validate bank sizes, returns a description of any errors. /// The graphics board: `[graphics] board`, or the machine's default - /// when it is not set (Solid IMPACT on the Octane, Newport elsewhere). + /// (Solid IMPACT on Octane, none on Fuel, Newport elsewhere). pub fn graphics_board(&self) -> GraphicsBoard { self.graphics.board.unwrap_or_else(|| GraphicsBoard::default_for(self.machine.profile)) } @@ -2297,7 +2594,7 @@ impl MachineConfig { pub fn validate(&self) -> Result<(), String> { self.validate_virtio_serial()?; self.machine.profile.accepts_cpu(self.machine.cpu)?; - self.machine.profile.accepts_cpus(self.machine.cpus)?; + self.machine.profile.accepts_cpus(self.machine.cpus())?; if self.graphics.heads != 1 && self.graphics.heads != 2 { return Err(format!( "graphics.heads {} is invalid (valid: 1, 2)", @@ -2315,6 +2612,8 @@ impl MachineConfig { if !board.supports(self.machine.profile) { return Err(if self.machine.profile == MachineProfile::OctaneIp30 { format!("graphics.board \"{name}\" is not modelled on the Octane (valid: solidimpact, highimpact, maximpact, v6, v8, v10, v12, none)") + } else if self.machine.profile == MachineProfile::FuelIp34 { + format!("graphics.board \"{name}\" is not a Fuel board (valid: v12, v10, none)") } else { format!("graphics.board \"{name}\" is only valid on the Octane") }); @@ -2356,9 +2655,9 @@ impl MachineConfig { "scsi.{} and scsi.{} both name {}", other, key, scsi_label(controller, id) )); } - if dev.is_daynaport() && self.machine.profile == MachineProfile::O2Ip32 { + if dev.is_daynaport() && matches!(self.machine.profile, MachineProfile::O2Ip32 | MachineProfile::FuelIp34 | MachineProfile::TezroIp53) { return Err(format!( - "scsi.{}: kind = \"daynaport\" is not available on the O2 (its Ethernet is built in)", key + "scsi.{}: kind = \"daynaport\" is not available on {} (its Ethernet is built in)", key, self.machine.profile.label() )); } if dev.is_daynaport() && self.network.mode == NetMode::Stream { @@ -2393,6 +2692,41 @@ impl MachineConfig { } } crate::dev::crime_mem::check_kits(&self.o2_kits()).map_err(|e| format!("banks: {e}"))?; + } else if self.machine.profile == MachineProfile::FuelIp34 { + if self.banks[..2] != [256, 256] { + return Err("Fuel currently requires banks = [256, 256] (512 MiB)".into()); + } + if !board.supports(MachineProfile::FuelIp34) { + return Err(format!("graphics.board \"{}\" is not a Fuel board (valid: v12, v10, none)", board.as_str())); + } + if let Some(text) = &self.fuel.l1_revision { + crate::dev::l1::parse_revision(text).map_err(|e| format!("fuel.l1_revision: {e}"))?; + } + if self.fuel.brick_serial.as_ref().is_some_and(|s| s.len() > 37 || s.contains('\0')) { + return Err("fuel.brick_serial must be at most 37 bytes with no NUL".into()); + } + } else if self.machine.profile == MachineProfile::TezroIp53 { + // The machine runs one of the two R16000 parts the CPU models + // define; `r10000` names the family, as on the Fuel, and takes the + // default part. The 800 MHz part with an 8 MB secondary cache has + // no model: the cache size is fixed when a model is compiled. + if !matches!(self.machine.cpu, CpuModel::R10000 | CpuModel::R16000 | CpuModel::R16000Mhz700) { + return Err(TEZRO_CPU_REFUSAL.into()); + } + if !TEZRO_CPUS.contains(&self.machine.cpus()) { + return Err(format!("[machine] cpus = {} is not a Tezro board's population (valid: {TEZRO_CPUS:?})", self.machine.cpus())); + } + if self.machine.cpu == CpuModel::R16000 && self.machine.cpus() != TEZRO_800_MHZ_CPUS { + return Err(format!( + "[machine] cpus = {}: the 800 MHz Tezro (cpu = \"r16000\") has four processors only, inferred from \ + the assemblies sheet as QEMU's offering is; one or two fit cpu = \"r16000_700\"", + self.machine.cpus() + )); + } + self.tezro_memory()?.check_for_cpus(usize::from(self.machine.cpus())).map_err(|e| format!("banks: {e}"))?; + if !board.supports(MachineProfile::TezroIp53) { + return Err(format!("graphics.board \"{}\" is not a Tezro board (valid: v12, none)", board.as_str())); + } } else if self.machine.profile == MachineProfile::OctaneIp30 { // A HEART bank is a matched DIMM pair: 64 MB to 2 GB, filled from // bank 0 without a gap, 8 GB at most (SGI's memory upgrade guide). @@ -2438,8 +2772,8 @@ impl MachineConfig { } } match (self.network.mode, self.machine.profile) { - (NetMode::Off, p) if p != MachineProfile::O2Ip32 && p != MachineProfile::OctaneIp30 => { - return Err("network.mode = \"off\" is only available on the O2 and the Octane for now; the Indy and Indigo2 always have their Ethernet".into()); + (NetMode::Off, p) if !matches!(p, MachineProfile::O2Ip32 | MachineProfile::OctaneIp30 | MachineProfile::FuelIp34 | MachineProfile::TezroIp53) => { + return Err("network.mode = \"off\" is only available on the O2, Octane, Fuel and Tezro; the Indy and Indigo2 always have their Ethernet".into()); } (NetMode::Stream, _) => { let ep = self.network.stream.as_deref() @@ -2716,16 +3050,20 @@ pub struct Cli { #[arg(long = "no-scsi-deferred-int", default_value_t = false)] pub no_scsi_deferred_int: bool, - /// Emulated CPU: `r4400` (default), `r5000`, or `r10000`. Overrides `[machine] cpu`. + /// Emulated CPU: `r4400` (default), `r5000`, `r10000`, `r12000`, `r14000`, `r16000` or `r16000_700`. Overrides `[machine] cpu`. /// - /// All three models are compiled into every build — each is its own + /// Every model is compiled into every build — each is its own /// monomorphisation, so the hot path carries no per-model branch — and this /// picks between them at construction. The `r5k` Cargo feature is removed. + /// The Octane takes `r10000`, `r12000` and `r14000`; the O2 `r5000`, + /// `r10000` and `r12000`; the IP28 `r10000`; the Tezro `r16000_700` (the + /// 700 MHz part, its default), `r16000` (the 800 MHz part) or `r10000`, + /// its family. #[arg(long = "cpu", value_name = "MODEL")] pub cpu: Option, /// Cache model: `auto` (default: `shadow` in a jitv2 build, else `full`), - /// `full` or `shadow`. Overrides `[machine] cache`. The R10000 is always + /// `full` or `shadow`. Overrides `[machine] cache`. The R10000 family is always /// `shadow`. #[arg(long = "cache", value_name = "MODEL")] pub cache: Option, @@ -2872,10 +3210,10 @@ impl Cli { if let Some(p) = &self.prom { cfg.prom = p.clone(); } if let Some(p) = &self.nvram { cfg.nvram = p.clone(); } if let Some(p) = &self.nveeprom { cfg.nveeprom = p.clone(); } - if let Some(v) = self.bank0 { cfg.banks[0] = v; } - if let Some(v) = self.bank1 { cfg.banks[1] = v; } - if let Some(v) = self.bank2 { cfg.banks[2] = v; } - if let Some(v) = self.bank3 { cfg.banks[3] = v; } + if let Some(v) = self.bank0 { cfg.banks[0] = v; cfg.banks_set = true; } + if let Some(v) = self.bank1 { cfg.banks[1] = v; cfg.banks_set = true; } + if let Some(v) = self.bank2 { cfg.banks[2] = v; cfg.banks_set = true; } + if let Some(v) = self.bank3 { cfg.banks[3] = v; cfg.banks_set = true; } if let Some(n) = self.jitv2_threads { cfg.jitv2.threads = n; } // Helper: insert or update a SCSI device entry. @@ -3425,7 +3763,7 @@ mod export_tests { assert_eq!(apply(&["--ip22"]).machine.profile, MachineProfile::Indigo2Ip22); assert!(Cli::try_parse_from(["iris", "--ip22", "--machine", "o2_ip32"]).is_err()); let e = MachineProfile::from_name("vax").unwrap_err(); - assert_eq!(e, "unknown machine \"vax\" (valid: indy_ip24, indigo2_ip22, indigo2_ip28, o2_ip32, octane_ip30)"); + assert_eq!(e, "unknown machine \"vax\" (valid: indy_ip24, indigo2_ip22, indigo2_ip28, o2_ip32, octane_ip30, fuel_ip34, tezro_ip53)"); let list = MachineProfile::list(); assert_eq!(list.lines().count(), MachineProfile::EVERY.len()); assert!(list.starts_with("indy_ip24 SGI Indy (IP24)\n"), "{list}"); @@ -3434,7 +3772,8 @@ mod export_tests { for p in MachineProfile::EVERY { match p { MachineProfile::IndyIp24 | MachineProfile::Indigo2Ip22 | MachineProfile::Indigo2Ip28 - | MachineProfile::O2Ip32 | MachineProfile::OctaneIp30 => {} + | MachineProfile::O2Ip32 | MachineProfile::OctaneIp30 | MachineProfile::FuelIp34 + | MachineProfile::TezroIp53 => {} } let cfg: MachineConfig = toml::from_str(&format!("[machine]\nprofile = \"{}\"", p.name())).unwrap(); assert_eq!(cfg.machine.profile, p); @@ -3447,21 +3786,27 @@ mod export_tests { #[test] fn a_profile_says_which_processors_it_accepts() { let accepted = |p: MachineProfile| CpuModel::ALL.into_iter().filter(|&c| p.accepts_cpu(c).is_ok()).collect::>(); - assert_eq!(accepted(MachineProfile::OctaneIp30), [CpuModel::R10000]); - assert_eq!(accepted(MachineProfile::IndyIp24), [CpuModel::R4400, CpuModel::R5000, CpuModel::R10000]); + assert_eq!(accepted(MachineProfile::OctaneIp30), [CpuModel::R10000, CpuModel::R12000, CpuModel::R14000]); + assert_eq!(accepted(MachineProfile::TezroIp53), [CpuModel::R10000, CpuModel::R16000, CpuModel::R16000Mhz700]); + assert_eq!(accepted(MachineProfile::IndyIp24), CpuModel::ALL); + assert_eq!(accepted(MachineProfile::FuelIp34), [CpuModel::R10000, CpuModel::R14000]); // The counts are what the checks in validate() and the Octane // constructor allowed before they moved here: one on every machine // but the Octane, one or two on the Octane. - for p in [MachineProfile::IndyIp24, MachineProfile::Indigo2Ip22, MachineProfile::Indigo2Ip28, MachineProfile::O2Ip32, MachineProfile::OctaneIp30] { + for p in [MachineProfile::IndyIp24, MachineProfile::Indigo2Ip22, MachineProfile::Indigo2Ip28, MachineProfile::O2Ip32, MachineProfile::OctaneIp30, MachineProfile::TezroIp53] { for cpus in 0..=4u8 { - let old_ok = if p == MachineProfile::OctaneIp30 { (1..=2).contains(&cpus) } else { cpus == 1 }; + let old_ok = match p { + MachineProfile::OctaneIp30 => (1..=2).contains(&cpus), + MachineProfile::TezroIp53 => (1..=4).contains(&cpus), + _ => cpus == 1, + }; assert_eq!(p.accepts_cpus(cpus).is_ok(), old_ok, "{p:?} with {cpus}"); } - if p != MachineProfile::OctaneIp30 { + if !matches!(p, MachineProfile::OctaneIp30 | MachineProfile::TezroIp53) { assert_eq!( p.accepts_cpus(2).unwrap_err(), - "[machine] cpus = 2: only the Octane has more than one processor" + "[machine] cpus = 2: only the Octane and the Tezro have more than one processor" ); } } @@ -3470,11 +3815,11 @@ mod export_tests { octane.machine.profile = MachineProfile::OctaneIp30; octane.machine.cpu = CpuModel::R10000; octane.banks = banks_from4([128, 0, 0, 0]); - octane.machine.cpus = 2; + octane.machine.cpus = Some(2); octane.validate().expect("two processors"); - octane.machine.cpus = 3; + octane.machine.cpus = Some(3); assert!(octane.validate().unwrap_err().contains("1 or 2")); - octane.machine.cpus = 1; + octane.machine.cpus = Some(1); octane.machine.cpu = CpuModel::R4400; assert!(octane.validate().unwrap_err().contains("an Octane has an R10000")); @@ -3532,6 +3877,65 @@ mod export_tests { assert_eq!(back.serial_socket_a.as_deref(), Some("/run/x.sock")); } + /// `cpu` names each part, and the R10000 family runs the shadow caches. + #[test] + fn every_cpu_part_is_selectable_by_name() { + for (name, cpu, family) in [ + ("r4400", CpuModel::R4400, false), + ("r5000", CpuModel::R5000, false), + ("r10000", CpuModel::R10000, true), + ("r12000", CpuModel::R12000, true), + ("r14000", CpuModel::R14000, true), + ("r16000", CpuModel::R16000, true), + ("r16000_700", CpuModel::R16000Mhz700, true), + ] { + let cfg: MachineConfig = toml::from_str(&format!("[machine]\ncpu = \"{name}\"\n")).unwrap(); + assert_eq!(cfg.machine.cpu, cpu, "{name}"); + assert_eq!(cpu.is_r10000_family(), family, "{name}"); + assert_eq!(CacheModel::Full.is_shadow(cpu), family, "{name}"); + assert!(CpuModel::ALL.contains(&cpu), "{name}"); + } + } + + /// The IP28 shipped with the R10000 only, so it refuses the later family + /// parts; the Octane and the O2 each name the parts they take. + #[test] + fn machines_name_the_cpus_they_accept() { + let mut cfg = MachineConfig::default(); + cfg.machine.profile = MachineProfile::Indigo2Ip28; + for cpu in [CpuModel::R12000, CpuModel::R14000, CpuModel::R16000, CpuModel::R16000Mhz700] { + cfg.machine.cpu = cpu; + let err = cfg.validate().expect_err("the IP28 has an R10000"); + assert_eq!(err, "an Indigo2 IMPACT (IP28) has an R10000 ([machine] cpu = \"r10000\")"); + } + cfg.machine.cpu = CpuModel::R10000; + cfg.validate().expect("the IP28 takes the R10000"); + + // The Fuel's part check is accepts_cpu's, and validate() reports it. + cfg.machine.profile = MachineProfile::FuelIp34; + cfg.machine.cpu = CpuModel::R12000; + assert_eq!(cfg.validate().unwrap_err(), "a Fuel has an R14000: set [machine] cpu = \"r14000\""); + + // Both machines check the part before they look for a PROM. + let refusal = |profile, cpu| { + let mut cfg = MachineConfig::default(); + cfg.machine.profile = profile; + cfg.machine.cpu = cpu; + match profile { + MachineProfile::OctaneIp30 => crate::ip30_machine::OctaneMachine::new(&cfg).err(), + _ => crate::ip32_machine::O2Machine::new(&cfg).err(), + } + }; + let octane = "an Octane has an R10000, R12000 or R14000 ([machine] cpu = \"r10000\", \"r12000\" or \"r14000\")"; + let o2 = "an O2 has an R5000, R10000 or R12000 ([machine] cpu = \"r5000\", \"r10000\" or \"r12000\")"; + for cpu in [CpuModel::R4400, CpuModel::R5000, CpuModel::R16000, CpuModel::R16000Mhz700] { + assert_eq!(refusal(MachineProfile::OctaneIp30, cpu).as_deref(), Some(octane), "{cpu:?}"); + } + for cpu in [CpuModel::R4400, CpuModel::R14000, CpuModel::R16000, CpuModel::R16000Mhz700] { + assert_eq!(refusal(MachineProfile::O2Ip32, cpu).as_deref(), Some(o2), "{cpu:?}"); + } + } + /// Every old `[o2]` key lands on the shared key it stood for, and the /// section keeps only what is the O2's alone. #[test] @@ -3881,3 +4285,324 @@ mod ci_overlay_tests { assert!(a.contains(&format!("iris-ci-{}-", std::process::id())), "{a}"); } } + +#[cfg(test)] +mod fuel_input_tests { + use super::*; + + /// A config saved for another machine must not grow a `[fuel]` table, + /// and a Fuel's own settings must survive a save and reload. + #[test] + fn only_a_configured_fuel_section_is_saved_and_it_round_trips() { + let other = MachineConfig::default(); + let saved = toml::to_string(&other).unwrap(); + assert!(!saved.contains("[fuel]"), "{saved}"); + let mut fuel = MachineConfig::default(); + fuel.fuel.console = FuelConsole::Ioc3B; + fuel.fuel.flash = Some("flash.bin".into()); + fuel.fuel.l1_port = Some(18882); + fuel.fuel.l1_log = Some("l1.log".into()); + fuel.fuel.usb_input = true; + fuel.fuel.ps2_input = false; + let saved = toml::to_string(&fuel).unwrap(); + assert!(saved.contains("[fuel]"), "{saved}"); + let loaded: MachineConfig = toml::from_str(&saved).unwrap(); + assert_eq!(loaded.fuel.console, FuelConsole::Ioc3B); + assert_eq!(loaded.fuel.flash.as_deref(), Some("flash.bin")); + assert_eq!(loaded.fuel.l1_port, Some(18882)); + assert_eq!(loaded.fuel.l1_log.as_deref(), Some("l1.log")); + assert!(!loaded.fuel.ps2_input && loaded.fuel.usb_input); + assert!(!MachineProfile::ALL.contains(&MachineProfile::FuelIp34)); + } + + /// PS/2 is the Fuel's standard keyboard and mouse; USB is opt-in, and a + /// `[fuel]` table that names neither key keeps both defaults. + #[test] + fn fuel_input_defaults_to_ps2_with_usb_opt_in() { + let none: MachineConfig = toml::from_str("[fuel]\nflash = \"flash.bin\"\n").unwrap(); + assert!(none.fuel.ps2_input && !none.fuel.usb_input); + assert!(MachineConfig::default().fuel.is_default()); + let usb: MachineConfig = toml::from_str("[fuel]\nusb_input = true\n").unwrap(); + assert!(usb.fuel.ps2_input && usb.fuel.usb_input); + assert!(!usb.fuel.is_default(), "an opt-in is written back"); + } + + /// The Fuel's processor is fixed, so `machine.cpu` may only name it (or the + /// family, as the key did before the R14000 had a name); the default + /// `r4400` is refused rather than silently ignored. + /// The L1 identity keys parse, validate and round trip; unset they are + /// left out of a written config. + #[test] + fn fuel_l1_identity_keys_are_validated_and_round_trip() { + let base = "banks = [256, 256]\n[machine]\nprofile = \"fuel_ip34\"\ncpu = \"r14000\"\n"; + let ok: MachineConfig = toml::from_str(&format!( + "{base}[fuel]\nl1_revision = \"1.50.2\"\nbrick_serial = \"EMUN03\"\n" + )).unwrap(); + ok.validate().unwrap(); + let saved = toml::to_string(&ok).unwrap(); + let back: MachineConfig = toml::from_str(&saved).unwrap(); + assert_eq!(back.fuel.l1_revision.as_deref(), Some("1.50.2")); + assert_eq!(back.fuel.brick_serial.as_deref(), Some("EMUN03")); + let plain: MachineConfig = toml::from_str(base).unwrap(); + plain.validate().unwrap(); + assert!(plain.fuel.l1_revision.is_none() && plain.fuel.brick_serial.is_none()); + let bad_rev: MachineConfig = toml::from_str(&format!("{base}[fuel]\nl1_revision = \"1.48\"\n")).unwrap(); + assert!(bad_rev.validate().unwrap_err().contains("fuel.l1_revision")); + let long = "x".repeat(38); + let bad_serial: MachineConfig = toml::from_str(&format!("{base}[fuel]\nbrick_serial = \"{long}\"\n")).unwrap(); + assert!(bad_serial.validate().unwrap_err().contains("fuel.brick_serial")); + } + + #[test] + fn fuel_requires_the_r14000_cpu_key() { + let mut cfg = MachineConfig::default(); + cfg.machine.profile = MachineProfile::FuelIp34; + cfg.banks.fill(0); + cfg.banks[0] = 256; + cfg.banks[1] = 256; + cfg.scsi.clear(); + assert!(cfg.validate().unwrap_err().contains("r14000")); + cfg.machine.cpu = CpuModel::R10000; + cfg.validate().unwrap(); + cfg.machine.cpu = CpuModel::R14000; + cfg.validate().unwrap(); + cfg.machine.cpu = CpuModel::R12000; + assert!(cfg.validate().is_err(), "no source has a Fuel with an R12000"); + } + + #[test] + fn fuel_requires_its_fixed_memory_population_and_a_vpro_or_no_graphics() { + let mut cfg = MachineConfig::default(); + cfg.machine.profile = MachineProfile::FuelIp34; + cfg.machine.cpu = CpuModel::R10000; + cfg.banks.fill(0); + cfg.banks[0] = 256; + cfg.banks[1] = 256; + cfg.scsi.clear(); + assert_eq!(cfg.graphics_board(), GraphicsBoard::None); + cfg.network.mode = NetMode::Off; + cfg.validate().unwrap(); + cfg.network.mode = NetMode::Pcap; + cfg.validate().unwrap(); + cfg.graphics.board = Some(GraphicsBoard::Newport); + assert!(cfg.validate().unwrap_err().contains("none")); + for board in [GraphicsBoard::V12, GraphicsBoard::V10] { + cfg.graphics.board = Some(board); + cfg.validate().unwrap(); + } + cfg.graphics.board = Some(GraphicsBoard::V6); + assert!(cfg.validate().unwrap_err().contains("not a Fuel board")); + cfg.graphics.board = None; + cfg.banks[1] = 0; + assert!(cfg.validate().unwrap_err().contains("256, 256")); + } + + #[test] + fn fuel_accepts_wide_scsi_ids_on_both_channels_without_changing_narrow_profiles() { + let text = "banks = [256, 256]\n[machine]\nprofile = \"fuel_ip34\"\ncpu = \"r10000\"\n[scsi.15]\npath = \"disk.raw\"\n[scsi.31]\npath = \"disk1.raw\"\n"; + let mut cfg: MachineConfig = toml::from_str(text).unwrap(); + cfg.validate().unwrap(); + // The Octane's buses are wide as well; the Indy's WD33C93A is narrow. + cfg.machine.profile = MachineProfile::IndyIp24; + assert!(cfg.validate().unwrap_err().contains("out of range")); + } +} + +#[cfg(test)] +mod tezro_config_tests { + use super::*; + + fn tezro() -> MachineConfig { + let mut cfg = MachineConfig::default(); + cfg.machine.profile = MachineProfile::TezroIp53; + cfg.machine.cpu = CpuModel::R16000Mhz700; + cfg.scsi.clear(); + cfg + } + + /// Left at the generic default, `banks` gives the Tezro its 4 GiB layout; + /// a listed population is taken bank by bank. + #[test] + fn tezro_memory_defaults_to_the_4_gib_layout_and_takes_listed_banks() { + let mut cfg = tezro(); + cfg.validate().unwrap(); + assert_eq!(cfg.tezro_memory().unwrap(), crate::dev::bedrock_mem::MemoryPopulation::default()); + cfg.banks = [1024, 512, 0, 0, 0, 0, 0, 0]; + cfg.validate().unwrap(); + assert_eq!(cfg.tezro_memory().unwrap().banks_mib(), [1024, 512]); + cfg.banks = [1024, 0, 512, 0, 0, 0, 0, 0]; + assert!(cfg.validate().unwrap_err().contains("fill the banks")); + cfg.banks = [256, 256, 0, 0, 0, 0, 0, 0]; + assert!(cfg.validate().unwrap_err().contains("512 or 1024")); + } + + /// A `[machine]` with no `cpu` gets the processor its product shipped + /// with; one that names a processor keeps it. Other profiles keep the + /// generic default. + #[test] + fn an_unset_cpu_defaults_to_the_product_processor() { + let tezro = MachineConfig::parse_toml("[machine]\nprofile = \"tezro_ip53\"\n").unwrap(); + assert_eq!(tezro.machine.cpu, CpuModel::R16000Mhz700, "the 700 MHz part"); + tezro.validate().unwrap(); + let fast = MachineConfig::parse_toml("[machine]\nprofile = \"tezro_ip53\"\ncpu = \"r16000\"\n").unwrap(); + assert_eq!(fast.machine.cpu, CpuModel::R16000, "the 800 MHz part by name"); + fast.validate().unwrap(); + let mut two = fast.clone(); + two.machine.cpus = Some(2); + assert!(two.validate().unwrap_err().contains("four processors only"), "the 800 MHz part with two"); + let fuel = MachineConfig::parse_toml("banks = [256, 256]\n[machine]\nprofile = \"fuel_ip34\"\n").unwrap(); + assert_eq!(fuel.machine.cpu, CpuModel::R14000); + let family = MachineConfig::parse_toml("[machine]\nprofile = \"tezro_ip53\"\ncpu = \"r10000\"\n").unwrap(); + assert_eq!(family.machine.cpu, CpuModel::R10000, "an explicit cpu wins"); + let wrong = MachineConfig::parse_toml("[machine]\nprofile = \"tezro_ip53\"\ncpu = \"r4400\"\n").unwrap(); + assert_eq!(wrong.machine.cpu, CpuModel::R4400); + assert!(wrong.validate().unwrap_err().contains("r16000_700")); + for text in ["", "[machine]\nprofile = \"octane_ip30\"\n", "[machine]\nprofile = \"o2_ip32\"\n"] { + assert_eq!(MachineConfig::parse_toml(text).unwrap().machine.cpu, CpuModel::default(), "{text:?}"); + } + let cli = Cli::parse_from(["iris", "--cpu", "r10000"]); + assert_eq!(cli.apply(tezro).machine.cpu, CpuModel::R10000, "--cpu overrides the default"); + } + + /// The file loader applies the same profile defaults as the text parser, + /// and preserves the errors for explicit and unknown CPU values. + #[test] + fn try_load_applies_tezro_defaults_and_cpu_errors() { + let dir = std::env::temp_dir().join(format!("iris-tezro-try-load-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("tezro.toml"); + let load = |text: &str| { + std::fs::write(&path, text).unwrap(); + MachineConfig::try_load(path.to_str().unwrap()) + }; + + let cfg = load("[machine]\nprofile = \"tezro_ip53\"\n").unwrap(); + assert_eq!(cfg.machine.cpu, CpuModel::R16000Mhz700); + assert!(!cfg.banks_set); + assert_eq!(cfg.tezro_memory().unwrap().total_mib(), 4096); + + let cfg = load("[machine]\nprofile = \"tezro_ip53\"\ncpus = 4\n").unwrap(); + assert_eq!(cfg.machine.cpu, CpuModel::R16000Mhz700); + cfg.validate().unwrap(); + assert!(!cfg.banks_set); + + let cfg = load("[machine]\nprofile = \"tezro_ip53\"\ncpu = \"r16000\"\ncpus = 2\n").unwrap(); + assert!(cfg.validate().unwrap_err().contains("four processors only")); + + let e = load("[machine]\nprofile = \"tezro_ip53\"\ncpu = \"not_a_cpu\"\n").unwrap_err(); + assert!(e.contains("unknown variant") && e.contains("not_a_cpu"), "{e}"); + let _ = std::fs::remove_dir_all(&dir); + } + + /// A `banks` equal to the generic default is a choice, not "unset": the + /// Tezro takes no 128 MB bank, so `--bank0 128 --bank1 128` and a + /// `banks = [128, 128, 0, 0]` line are refused rather than read as the + /// board's 4 GiB default. + #[test] + fn tezro_honours_banks_that_equal_the_generic_default() { + let cli = Cli::parse_from(["iris", "--bank0", "128", "--bank1", "128"]); + let cfg = cli.apply(tezro()); + assert!(cfg.validate().unwrap_err().contains("512 or 1024")); + + let text = "banks = [128, 128, 0, 0]\n[machine]\nprofile = \"tezro_ip53\"\n"; + let cfg = MachineConfig::parse_toml(text).unwrap(); + assert!(cfg.validate().unwrap_err().contains("512 or 1024")); + + let cfg = MachineConfig::parse_toml("[machine]\nprofile = \"tezro_ip53\"\n").unwrap(); + assert_eq!(cfg.tezro_memory().unwrap(), crate::dev::bedrock_mem::MemoryPopulation::default()); + } + + /// The processor is one of the two R16000 parts (or the family, as on + /// the Fuel), with no graphics board or a V12. Every other part is + /// refused, and the refusal names the 8 MB part the model does not have. + #[test] + fn tezro_requires_an_r16000_and_takes_no_graphics_or_a_v12() { + let mut cfg = tezro(); + for cpu in [CpuModel::R16000Mhz700, CpuModel::R10000] { + cfg.machine.cpu = cpu; + cfg.validate().unwrap(); + } + cfg.machine.cpu = CpuModel::R16000; + cfg.validate().unwrap(); + cfg.machine.cpus = Some(1); + assert!(cfg.validate().unwrap_err().contains("four processors only")); + cfg.machine.cpus = Some(4); + for cpu in [CpuModel::R4400, CpuModel::R5000, CpuModel::R12000, CpuModel::R14000] { + cfg.machine.cpu = cpu; + let e = cfg.validate().unwrap_err(); + assert!(e.contains("r16000_700") && e.contains("r16000-800mhz-8mb-rev22"), "{cpu:?}: {e}"); + } + assert_eq!(MachineProfile::TezroIp53.cpus(), [CpuModel::R16000Mhz700, CpuModel::R16000, CpuModel::R10000]); + let mut cfg = tezro(); + assert_eq!(cfg.graphics_board(), GraphicsBoard::None); + cfg.graphics.board = Some(GraphicsBoard::Newport); + assert!(cfg.validate().is_err()); + cfg.graphics.board = Some(GraphicsBoard::V10); + assert!(cfg.validate().is_err(), "the V10 is not a Tezro board"); + for cpus in [1, 4] { + cfg.graphics.board = Some(GraphicsBoard::V12); + cfg.machine.cpus = Some(cpus); + cfg.validate().unwrap(); + } + let mut cfg = tezro(); + cfg.network.mode = NetMode::Off; + cfg.validate().unwrap(); + } + + /// `[machine] cpus` is the processors fitted: four on a Tezro unless + /// set, one elsewhere, and only the Octane and the Tezro take more. + #[test] + fn machine_cpus_defaults_to_four_on_the_tezro_and_one_elsewhere() { + let mut cfg = tezro(); + assert_eq!(cfg.machine.cpus(), 4); + cfg.validate().unwrap(); + for cpus in [1, 2] { + cfg.machine.cpus = Some(cpus); + cfg.validate().unwrap(); + assert_eq!(cfg.machine.cpus(), cpus); + } + assert_eq!(MachineConfig::default().machine.cpus(), 1); + let loaded: MachineConfig = toml::from_str("[machine]\nprofile = \"tezro_ip53\"\ncpu = \"r10000\"\ncpus = 2\n").unwrap(); + assert_eq!(loaded.machine.cpus(), 2); + let mut fuel = MachineConfig::default(); + fuel.machine.profile = MachineProfile::FuelIp34; + fuel.machine.cpu = CpuModel::R14000; + fuel.machine.cpus = Some(2); + assert!(fuel.validate().unwrap_err().contains("only the Octane and the Tezro")); + } + + #[test] + fn tezro_refuses_a_processor_count_its_board_does_not_fit() { + for cpus in [0, 3, 5] { + let mut cfg = tezro(); + cfg.machine.cpus = Some(cpus); + assert!(cfg.validate().unwrap_err().contains("[machine] cpus"), "{cpus}"); + } + } + + /// The sample configuration in the repository root is a valid Tezro. + #[test] + fn the_sample_tezro_configuration_validates() { + let cfg: MachineConfig = toml::from_str(include_str!("../tezro.toml.example")).unwrap(); + assert_eq!(cfg.machine.profile, MachineProfile::TezroIp53); + cfg.validate().unwrap(); + assert_eq!(cfg.tezro_memory().unwrap().total_mib(), 4096); + } + + /// A `[tezro]` table is written only when something in it was set, and + /// what was set survives a save and reload. + #[test] + fn only_a_configured_tezro_section_is_saved_and_it_round_trips() { + let saved = toml::to_string(&MachineConfig::default()).unwrap(); + assert!(!saved.contains("[tezro]"), "{saved}"); + let text = "[machine]\nprofile = \"tezro_ip53\"\ncpu = \"r10000\"\n[tezro]\nconsole = \"ioc4_b\"\nflash = \"flash.bin\"\nl1_port = 18882\nserial_port_c = 18883\n"; + let loaded: MachineConfig = toml::from_str(text).unwrap(); + assert_eq!(loaded.machine.profile, MachineProfile::TezroIp53); + assert_eq!(loaded.tezro.console, TezroConsole::Ioc4B); + assert_eq!(loaded.tezro.console.index(), 2); + let again: MachineConfig = toml::from_str(&toml::to_string(&loaded).unwrap()).unwrap(); + assert_eq!(again.tezro.flash.as_deref(), Some("flash.bin")); + assert_eq!((again.tezro.l1_port, again.tezro.serial_port_c), (Some(18882), Some(18883))); + assert!(!MachineProfile::ALL.contains(&MachineProfile::TezroIp53)); + } +} diff --git a/src/cpu/decode_cache.rs b/src/cpu/decode_cache.rs index 293d41af0..387faca80 100644 --- a/src/cpu/decode_cache.rs +++ b/src/cpu/decode_cache.rs @@ -79,6 +79,10 @@ pub struct DecodeCache { bitmap: u64, /// The bitmap's region size, as `ppmem::window_shift`. window_shift: u32, + /// Pages read through the bus word by word even where the window maps + /// them: the shadow cache serves part of such a page from a line the guest + /// staged, which the window does not hold. Normally empty. + bus_only: Vec, } impl Default for DecodeCache { @@ -92,7 +96,7 @@ impl DecodeCache { } pub fn new() -> Self { - Self { pages: (0..PAGES).map(|_| Page::new()).collect(), bitmap: 0, window_shift: crate::ppmem::BITMAP_SHIFT } + Self { pages: (0..PAGES).map(|_| Page::new()).collect(), bitmap: 0, window_shift: crate::ppmem::BITMAP_SHIFT, bus_only: Vec::new() } } /// Forget everything: memory may have changed without any counter moving. @@ -102,6 +106,27 @@ impl DecodeCache { } } + /// Read page `pfn` through the bus word by word (`on`), or through the + /// window again. Drops the page's decoded slots either way: its pointer + /// into the window is what changes. Costs the fast path nothing: a page + /// without a pointer misses it and takes `fetch_slow`. + pub fn set_bus_only(&mut self, pfn: u64, on: bool) { + self.bus_only.retain(|&p| p != pfn); + if on { + self.bus_only.push(pfn); + } + let page = &mut self.pages[index(pfn)]; + if page.pfn == pfn { + page.pfn = EMPTY; + } + } + + /// Every page back to the window. + pub fn clear_bus_only(&mut self) { + self.bus_only.clear(); + self.flush(); + } + /// The decoded slot for the instruction at `phys`, filled on a miss. /// /// `window` and `bitmap` are ppmem's window base @@ -157,10 +182,11 @@ impl DecodeCache { let k = (phys as usize >> 2) & (SLOTS - 1); let (w, bit) = (k >> 6, 1u64 << (k & 63)); let window_shift = self.window_shift; + let bus_only = !self.bus_only.is_empty() && self.bus_only.contains(&pfn); let page = &mut self.pages[index(pfn)]; let _ = ic_line; let host_for = || -> *const u32 { - if !window.is_null() && crate::ppmem::bitmap_maps_at(bitmap, window_shift, phys) { + if !bus_only && !window.is_null() && crate::ppmem::bitmap_maps_at(bitmap, window_shift, phys) { unsafe { window.add((pfn << 12) as usize) as *const u32 } } else { std::ptr::null() @@ -303,6 +329,26 @@ mod tests { assert_eq!(unsafe { (*p).raw }, 0x2400_0009); } + /// A page marked bus-only reads word by word even where the window maps + /// it, and goes back to the window when the mark is lifted. + #[test] + fn a_bus_only_page_bypasses_the_window_until_released() { + let mut ram = vec![0u32; 2 * 1024]; + ram[1024 + 3] = 0x2400_0007; // phys 0x100c + let window = ram.as_ptr() as *const u8; + let bus_word = |_| -> Result { Ok(0x2400_0042) }; + let mut dc = DecodeCache::new(); + assert_eq!(unsafe { (*dc.fetch(0x100c, window, 1, 16, |_| panic!("window")).unwrap()).raw }, 0x2400_0007); + + dc.set_bus_only(0x1000 >> 12, true); + assert_eq!(unsafe { (*dc.fetch(0x100c, window, 1, 16, bus_word).unwrap()).raw }, 0x2400_0042); + // Served again, and checked against the bus each time. + assert_eq!(unsafe { (*dc.fetch(0x100c, window, 1, 16, |_| Ok(0x2400_0043)).unwrap()).raw }, 0x2400_0043); + + dc.set_bus_only(0x1000 >> 12, false); + assert_eq!(unsafe { (*dc.fetch(0x100c, window, 1, 16, |_| panic!("window")).unwrap()).raw }, 0x2400_0007); + } + /// Raw mode: a fused slot depends on the next word too, so changing only /// the delay slot must refill it. #[test] diff --git a/src/cpu/jitv2/codegen.rs b/src/cpu/jitv2/codegen.rs index ee31f90cf..a9a02a8bf 100644 --- a/src/cpu/jitv2/codegen.rs +++ b/src/cpu/jitv2/codegen.rs @@ -868,6 +868,10 @@ impl Codegen { for v in isa.isa_flags() { let _ = write!(s, " {v}"); } + // Only when set, so machines without the behaviour keep their warm cache. + if self.jit_consts.lwr_keeps_upper { + s.push_str(" lwr_upper=1"); + } s.push_str(" ops="); s.extend(crate::cpu::jitv2::opcode_support::enabled_snapshot().iter().map(|&b| if b { '1' } else { '0' })); Some(blake3::hash(s.as_bytes()).as_bytes()[..16].try_into().unwrap()) @@ -4379,6 +4383,11 @@ pub struct JitConsts { /// to loading from memory, exactly as before, so an un-stamped `Codegen` /// stays correct (just no faster). pub core: Option, + /// A partial LWR keeps bits 63:32 of the destination, as the R10000 + /// does (`CpuModel::PARTIAL_LWR_KEEPS_UPPER`). A model fact rather than a + /// core one, published beside the core address because both reach a worker's + /// `Codegen` the same way. The code-cache fingerprint carries it when set. + pub lwr_keeps_upper: bool, } impl JitConsts { @@ -10042,12 +10051,52 @@ fn emit_lwr_ldr(ctx: &mut EmitCtx, size: MemSize) { let result = if size == MemSize::B8 { result_narrow + } else if size == MemSize::B4 && ctx.jit_consts.lwr_keeps_upper { + // Partial words keep rt[63:32]; a full word (offset 3) still + // sign-extends. Mirrors `MipsExecutor::exec_lwr`. + // Only while 64-bit operations are on for the current mode. + let upper = ctx.builder.ins().band_imm_s(rt_val, 0xFFFF_FFFF_0000_0000u64 as i64); + let lower = ctx.builder.ins().uextend(ir::types::I64, result_narrow); + let merged = ctx.builder.ins().bor(upper, lower); + let not_full = ctx.builder.ins().icmp_imm_s(ir::condcodes::IntCC::NotEqual, byte_offset, align_mask); + let ops64 = emit_ops64_enabled(ctx); + let partial = ctx.builder.ins().band(not_full, ops64); + let full = ctx.builder.ins().sextend(ir::types::I64, result_narrow); + ctx.builder.ins().select(partial, merged, full) } else { ctx.builder.ins().sextend(ir::types::I64, result_narrow) }; emit_write_gpr(ctx, field_rt(ctx.raw), result); } +/// Whether 64-bit operations are enabled for the current privilege mode, read +/// from the live Status at run time: the KX, SX or UX bit selected by +/// EXL/ERL/KSU. Mirrors `MipsCore::is_64bit_mode` (a reserved KSU value counts +/// as kernel). Only emitted on paths that need it (`lwr_keeps_upper`), so +/// models without such a path pay nothing. Run time rather than compile time +/// because a page's code can run in either mode and a region does not end on +/// a mode change by itself. +fn emit_ops64_enabled(ctx: &mut EmitCtx) -> Value { + use crate::cpu::mips_core::{STATUS_ERL, STATUS_EXL, STATUS_KSU_SHIFT, STATUS_KX, STATUS_SX, STATUS_UX}; + let mem = MemFlagsData::trusted(); + let status_off = ir::immediates::Offset32::new(core_offset_of_cp0_status()); + let status = ctx.builder.ins().load(ir::types::I32, mem, ctx.core_ptr, status_off); + let ksu_sh = ctx.builder.ins().ushr_imm_u(status, STATUS_KSU_SHIFT as i64); + let ksu = ctx.builder.ins().band_imm_s(ksu_sh, 3); + let is_super = ctx.builder.ins().icmp_imm_s(IntCC::Equal, ksu, 1); + let is_user = ctx.builder.ins().icmp_imm_s(IntCC::Equal, ksu, 2); + let kx = ctx.builder.ins().iconst(ir::types::I32, STATUS_KX as i64); + let sx = ctx.builder.ins().iconst(ir::types::I32, STATUS_SX as i64); + let ux = ctx.builder.ins().iconst(ir::types::I32, STATUS_UX as i64); + let by_ksu_hi = ctx.builder.ins().select(is_user, ux, kx); + let by_ksu = ctx.builder.ins().select(is_super, sx, by_ksu_hi); + let lvl = ctx.builder.ins().band_imm_s(status, (STATUS_EXL | STATUS_ERL) as i64); + let forced_kernel = ctx.builder.ins().icmp_imm_s(IntCC::NotEqual, lvl, 0); + let bit = ctx.builder.ins().select(forced_kernel, kx, by_ksu); + let on = ctx.builder.ins().band(status, bit); + ctx.builder.ins().icmp_imm_s(IntCC::NotEqual, on, 0) +} + fn emit_lwl(ctx: &mut EmitCtx) { emit_lwl_ldl(ctx, MemSize::B4); } @@ -10811,6 +10860,18 @@ mod tests { (op << 26) | (rs << 21) | (rt << 16) | (rd << 11) | (sa << 6) | funct } + #[test] + fn cache_fingerprint_includes_the_lwr_choice() { + let base = Codegen::new().cache_fingerprint(false).expect("default codegen is cacheable"); + let mut codegen = Codegen::new(); + codegen.jit_consts.lwr_keeps_upper = true; + assert_ne!( + codegen.cache_fingerprint(false).expect("lwr codegen is cacheable"), + base, + "the LWR choice must select a different persistent cache namespace", + ); + } + /// `is_inlinable` must reject on *both* of its conditions, not just the /// bounds one. The bounds half is what panicked /// (`clanker/jitv2/nested-delay-slot-across-page-boundary.md`); the diff --git a/src/cpu/jitv2/comp.rs b/src/cpu/jitv2/comp.rs index 1979b36ee..bb6ce1e23 100644 --- a/src/cpu/jitv2/comp.rs +++ b/src/cpu/jitv2/comp.rs @@ -127,7 +127,8 @@ fn prepare_multi_entry_compile( #[cfg(feature = "developer")] stats: &crate::cpu::jitv2::JitStats, ) -> PrepareOutcome { let page = unsafe { &*req.page }; - let phys_base = page.pfn as u64 * PAGE_SIZE as u64; + let key = page.pfn; + let phys_base = page.phys_base(); // Dirty-cache gate. The seqlock snapshot below reads RAM off the bus; the // guest CPU sees RAM overlaid with its own dirty L1-D/L2 lines, and a @@ -166,7 +167,7 @@ fn prepare_multi_entry_compile( let gen0 = page.current_gen(); let mut read_ok = true; for (i, w) in words.iter_mut().enumerate() { - let r = bus.read32(phys_base + i as u64 * 4); + let r = bus.read32(crate::cpu::jitv2::jitv2::key_bus_addr(key, (i as u32) * 4)); if !r.is_ok() { read_ok = false; break; diff --git a/src/cpu/jitv2/equiv_test.rs b/src/cpu/jitv2/equiv_test.rs index 315eaa286..b2fb11b2a 100644 --- a/src/cpu/jitv2/equiv_test.rs +++ b/src/cpu/jitv2/equiv_test.rs @@ -4355,6 +4355,394 @@ mod tests { } } + /// Run `words` from `pc` once through the interpreter and once as a + /// compiled region, both on CPU model `C` over a fresh `bus()` and seeded + /// with `gpr` and CP0 Status `status`. The JIT learns the model's choices from `install_jit_hooks`, + /// as in production. + fn interpreter_and_jit(bus: impl Fn() -> Arc, pc: u64, gpr: [u64; 32], status: u32, words: &[u32]) + -> (CoreSnapshot, CoreSnapshot) + where + C: crate::cpu::mips_cache_v2::CpuModel + From>, + { + let build = || { + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))); + let mut exec = Box::new(MipsExecutor::::new(bus(), PassthroughTlb::default(), &cfg)); + exec.core.gpr = gpr; + exec.core.pc = pc; + exec.core.cp0_status = status; + exec.core.hot.interrupts.store(0, std::sync::atomic::Ordering::Relaxed); + #[cfg(feature = "fetchverify")] + { exec.core.fetch_verify_on = false; } + exec + }; + let mut interp = build(); + interp.jitv2_dispatch_enabled = false; + for (i, &word) in words.iter().enumerate() { + if interp.core.pc != pc + 4 * i as u64 { + break; // an exception redirected the PC + } + interp.exec(word); + } + + let mut page = [0u32; ENTRIES_PER_PAGE]; + page[..words.len()].copy_from_slice(words); + let page_base = (pc & !(PAGE_SIZE as u64 - 1)) as u32; + let mut analyzer = Analyzer::new(); + let (walked, non_empty) = analyzer.walk_bounded(&page, 0, page_base, words.len()); + assert!(non_empty); + let mut instrs_owned = *walked; + let mut codegen = Codegen::new(); + codegen.jit_consts.lwr_keeps_upper = C::PARTIAL_LWR_KEEPS_UPPER; + let jit_fn: JitFn = codegen.compile_region(&mut instrs_owned, 0, true, false) + .expect("region must be compilable for this test to be meaningful"); + let mut jit = build(); + jit.install_jit_hooks(); + // The compile pool's workers read the model's choice from here, so a + // missing publication would not show if `codegen` were stamped by hand. + assert_eq!(jit.jitv2.lock().jit_consts.lock().lwr_keeps_upper, C::PARTIAL_LWR_KEEPS_UPPER); + unsafe { jit_fn(&mut jit.core as *mut MipsCore) }; + std::mem::forget(codegen); + (CoreSnapshot::capture(&interp.core), CoreSnapshot::capture(&jit.core)) + } + + /// The R10000 family keeps bits 63:32 on a partial LWR while 64-bit + /// operations are enabled for the current mode and the R5000 sign-extends; + /// with them off (a 32-bit user process, UX=0) the family sign-extends too. + /// The JIT and the interpreter must agree at every offset in every mode, and + /// the interpreter's result is the one + /// `partial_lwr_keeps_the_upper_word_on_the_r10000_family` pins. + #[test] + fn jit_lwr_matches_the_interpreter_on_the_r10000_family_and_r5000() { + use crate::cpu::mips_cache_shadow::{R10000ShadowCache, R12000ShadowCache, R16000ShadowCache, R5000ShadowCache}; + fn check>>(model_keeps_upper: bool) { + use crate::cpu::mips_core::{STATUS_EXL, STATUS_KX, STATUS_SX, STATUS_UX}; + const USER: u32 = 2 << 3; + const SUPER: u32 = 1 << 3; + // (Status, whether 64-bit operations are on): each mode's own + // bit decides, whatever the other two hold. + for (status, ops64, mode) in [ + (STATUS_KX, true, "kernel KX=1"), + (0, false, "kernel KX=0"), + (STATUS_UX | STATUS_SX, false, "kernel KX=0, UX/SX=1"), + (USER | STATUS_UX, true, "user UX=1"), + (USER | STATUS_KX | STATUS_SX, false, "user UX=0, KX/SX=1"), + (SUPER | STATUS_SX, true, "supervisor SX=1"), + (SUPER | STATUS_KX | STATUS_UX, false, "supervisor SX=0"), + (USER | STATUS_EXL | STATUS_KX, true, "EXL forces kernel, KX=1"), + (USER | STATUS_EXL | STATUS_UX, false, "EXL forces kernel, UX=1 KX=0"), + ] { + for offset in 0..4u64 { + let mut gpr = [0u64; 32]; + gpr[1] = 0x2000 + offset; + gpr[2] = 0x1122_3344_5566_7788; + let bus = || -> Arc { + let mem = Arc::new(MockMemory::new_not_compilable()); + mem.set_word(0x2000, 0xDEAD_BEEF); + mem + }; + let lwr = make_i(crate::cpu::mips_isa::OP_LWR, 1, 2, 0); + let (interp, jit) = interpreter_and_jit::(bus, 0x1000, gpr, status, &[lwr]); + assert_eq!(jit, interp, "{} JIT/interpreter LWR offset {offset}, {mode}", C::NAME); + let upper_kept = interp.gpr[2] >> 32 == 0x1122_3344; + assert_eq!(upper_kept, model_keeps_upper && ops64 && offset != 3, + "{} LWR offset {offset}, {mode}", C::NAME); + } + } + } + check::(true); + check::(true); + check::(true); + check::(false); + } + + /// A secondary line staged with Index Store Tag and Index Store Data is + /// fetched as code, not the page the JIT compiled from memory: staging it + /// drops the page from the JIT (the micro-TLB flush in `exec_cache`, then + /// `jitv2_track_pcp` declining the page), and invalidating the line puts + /// the page back on memory and the JIT. Production dispatch, inline + /// compile, an R10000-family shadow cache. + #[test] + fn a_staged_line_is_fetched_as_code_instead_of_the_compiled_page() { + use crate::cpu::mips_cache_shadow::R10000ShadowCache; + use crate::cpu::mips_isa::{OP_ADDIU, OP_CACHE}; + let pc = 0xFFFF_FFFF_8000_1000u64; + let ra = 0xFFFF_FFFF_8000_2000u64; + let addiu = |imm: u32| make_i(OP_ADDIU, 0, 2, imm as u16); + let jr = make_r(crate::cpu::mips_isa::OP_SPECIAL, 31, 0, 0, 0, crate::cpu::mips_isa::FUNCT_JR); + let mem = Arc::new(MockMemory::new()); + for (i, w) in [addiu(0x11), jr, 0].into_iter().enumerate() { + mem.set_word(0x1000 + 4 * i as u64, w); + } + let bus: Arc = mem.clone(); + let mut exec = Box::new(MipsExecutor::::new( + bus, PassthroughTlb::default(), &MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))))); + exec.core.cp0_config = (exec.core.cp0_config & !7) | 3; // KSEG0 cacheable + exec.install_jit_hooks(); + exec.jitv2_inline_compile = true; + exec.core.hot.interrupts.store(0, std::sync::atomic::Ordering::Relaxed); + #[cfg(feature = "fetchverify")] + { exec.core.fetch_verify_on = false; } + let run = |exec: &mut MipsExecutor| -> u64 { + exec.core.pc = pc; + exec.core.gpr[31] = ra; + exec.core.gpr[2] = 0; + for _ in 0..16 { + if exec.core.pc == ra { break; } + exec.step_jit(); + } + assert_eq!(exec.core.pc, ra, "the code returns"); + exec.core.gpr[2] + }; + let cache_op = |exec: &mut MipsExecutor, op: u32, lo: u32, hi: u32| { + exec.core.gpr[4] = pc; + exec.core.cp0_taglo = lo as u64; + exec.core.cp0_taghi = hi; + assert_eq!(exec.exec(make_i(OP_CACHE, 4, op, 0)), crate::cpu::mips_exec::EXEC_COMPLETE); + }; + + // From memory, until the page is compiled and entered. + for _ in 0..4 { + assert_eq!(run(&mut exec), 0x11); + } + assert!(!exec.pcp.is_null(), "the page is on the JIT"); + assert!(unsafe { (*exec.pcp).is_published(0) }, "and compiled"); + + // Stage the line at PA 0x1000: valid CleanExclusive, tag PA[39:18] = 0. + cache_op(&mut exec, 0x0b, 2 << 10, 0); + assert!(exec.pcp.is_null(), "staging dropped the page from the JIT"); + // TagLo is the word at the lower address (the inferred refill order). + cache_op(&mut exec, 0x1f, addiu(0x22), jr); + for _ in 0..4 { + assert_eq!(run(&mut exec), 0x22, "the staged word runs, not the compiled page"); + } + assert!(exec.pcp.is_null(), "and the page stays off it"); + + // Invalidate the line (Index Writeback Invalidate): memory and the JIT again. + cache_op(&mut exec, 0x03, 0, 0); + for _ in 0..4 { + assert_eq!(run(&mut exec), 0x11); + } + assert!(!exec.pcp.is_null(), "the page is back on the JIT"); + } + + /// The same for a line staged in the primary instruction cache with + /// Index Store Tag (I) and 16 Index Store Data (I), on way 1: compiled + /// code for the page is dropped, the staged words run, and an Index + /// Invalidate (I) puts the page back on memory and the JIT. + #[test] + fn a_staged_primary_instruction_line_is_fetched_instead_of_the_compiled_page() { + use crate::cpu::mips_cache_shadow::R10000ShadowCache; + use crate::cpu::mips_isa::{OP_ADDIU, OP_CACHE}; + let pc = 0xFFFF_FFFF_8000_1000u64; + let ra = 0xFFFF_FFFF_8000_2000u64; + let addiu = |imm: u32| make_i(OP_ADDIU, 0, 2, imm as u16); + let jr = make_r(crate::cpu::mips_isa::OP_SPECIAL, 31, 0, 0, 0, crate::cpu::mips_isa::FUNCT_JR); + let mem = Arc::new(MockMemory::new()); + for (i, w) in [addiu(0x11), jr, 0].into_iter().enumerate() { + mem.set_word(0x1000 + 4 * i as u64, w); + } + let bus: Arc = mem.clone(); + let mut exec = Box::new(MipsExecutor::::new( + bus, PassthroughTlb::default(), &MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))))); + exec.core.cp0_config = (exec.core.cp0_config & !7) | 3; + exec.install_jit_hooks(); + exec.jitv2_inline_compile = true; + exec.core.hot.interrupts.store(0, std::sync::atomic::Ordering::Relaxed); + #[cfg(feature = "fetchverify")] + { exec.core.fetch_verify_on = false; } + let run = |exec: &mut MipsExecutor| -> u64 { + exec.core.pc = pc; + exec.core.gpr[31] = ra; + exec.core.gpr[2] = 0; + for _ in 0..16 { + if exec.core.pc == ra { break; } + exec.step_jit(); + } + assert_eq!(exec.core.pc, ra, "the code returns"); + exec.core.gpr[2] + }; + let cache_op = |exec: &mut MipsExecutor, op: u32, off: u64, lo: u32, hi: u32| { + exec.core.gpr[4] = pc + off; + exec.core.cp0_taglo = lo as u64; + exec.core.cp0_taghi = hi; + assert_eq!(exec.exec(make_i(OP_CACHE, 4, op, 0)), crate::cpu::mips_exec::EXEC_COMPLETE); + }; + for _ in 0..4 { + assert_eq!(run(&mut exec), 0x11); + } + assert!(unsafe { (*exec.pcp).is_published(0) }, "compiled"); + // Block at PA 0x1000: tag PA[35:12] = 1 in TagLo[31:8], valid in bit 6. + cache_op(&mut exec, 0x08, 1, (1 << 8) | (1 << 6), 0); + assert!(exec.pcp.is_null(), "staging dropped the page from the JIT"); + let words = [addiu(0x22), jr]; + for w in 0..16u64 { + let word = words.get(w as usize).copied().unwrap_or(0); + // The model's raw encoding (p[35:32] = 0xf), with even parity. + exec.core.cp0_ecc = (0xfu64 << 32 | word as u64).count_ones() & 1; + cache_op(&mut exec, 0x1c, 4 * w + 1, word, 0xf); + } + for _ in 0..4 { + assert_eq!(run(&mut exec), 0x22, "the staged words run, not the compiled page"); + } + cache_op(&mut exec, 0x00, 1, 0, 0); // Index Invalidate (I), way 1 + for _ in 0..4 { + assert_eq!(run(&mut exec), 0x11); + } + assert!(!exec.pcp.is_null(), "back on the JIT"); + } + + /// The data side: compiled loads reach RAM through the micro-TLB and + /// ppmem's window, never the cache. Warm the micro-TLB on a page, stage a + /// secondary line in it, and the compiled load returns the staged word; + /// invalidating the line sends it back to the window. + #[test] + fn a_compiled_load_hits_a_staged_line_not_the_window() { + use crate::cpu::mips_cache_shadow::R10000ShadowCache; + use crate::cpu::mips_cache_v2::MipsCache; + use crate::cpu::mips_isa::{OP_CACHE, OP_LW}; + let pc = 0xFFFF_FFFF_8000_1000u64; + let ra = 0xFFFF_FFFF_8000_8000u64; + let data_va = 0xFFFF_FFFF_8000_2000u64; // PA 0x2000 + let lw = make_i(OP_LW, 4, 2, 0); + let jr = make_r(crate::cpu::mips_isa::OP_SPECIAL, 31, 0, 0, 0, crate::cpu::mips_isa::FUNCT_JR); + let mem = Arc::new(MockMemory::new()); + for (i, w) in [lw, jr, 0].into_iter().enumerate() { + mem.set_word(0x1000 + 4 * i as u64, w); + } + let bus: Arc = mem.clone(); + let mut exec = Box::new(MipsExecutor::::new( + bus, PassthroughTlb::default(), &MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))))); + exec.core.cp0_config = (exec.core.cp0_config & !7) | 3; // KSEG0 cacheable + // Out of reset ERL is set and the micro-TLB declines to fill; this test + // is about what it holds. + exec.core.cp0_status &= !crate::cpu::mips_core::STATUS_ERL; + exec.resync_privilege_state(); + // A window over the first 64 KiB of RAM, region 0 mapped, as ppmem + // publishes them: u32 words in native order. + let mut window = vec![0u8; 0x10000]; + unsafe { *(window.as_mut_ptr().add(0x2000) as *mut u32) = 0x0a0a_0a0a }; + let mut gens: Vec = (0..16).map(|_| AtomicU64::new(0)).collect(); + unsafe { + exec.set_tcache_window(window.as_mut_ptr()); + exec.set_tcache_gen_window(gens.as_mut_ptr()); + *exec.tcache_bitmap_ptr() = 1; + } + exec.core.ppmem_bitmap = 1; + exec.install_jit_hooks(); + assert!(exec.cache.jit_dc_geometry().supported, "the inline path is on"); + exec.jitv2_inline_compile = true; + exec.core.hot.interrupts.store(0, std::sync::atomic::Ordering::Relaxed); + #[cfg(feature = "fetchverify")] + { exec.core.fetch_verify_on = false; } + let run = |exec: &mut MipsExecutor| -> u64 { + exec.core.pc = pc; + exec.core.gpr[31] = ra; + exec.core.gpr[4] = data_va; + exec.core.gpr[2] = 0; + for _ in 0..16 { + if exec.core.pc == ra { break; } + exec.step_jit(); + } + assert_eq!(exec.core.pc, ra, "the code returns"); + exec.core.gpr[2] + }; + let cache_op = |exec: &mut MipsExecutor, op: u32, lo: u32, hi: u32| { + exec.core.gpr[5] = data_va; + exec.core.cp0_taglo = lo as u64; + exec.core.cp0_taghi = hi; + assert_eq!(exec.exec(make_i(OP_CACHE, 5, op, 0)), crate::cpu::mips_exec::EXEC_COMPLETE); + }; + for _ in 0..4 { + assert_eq!(run(&mut exec), 0x0a0a_0a0a, "from the window"); + } + // PA 0x2000: valid CleanExclusive, tag PA[39:18] = 0. TagLo is the + // word at the lower address (the inferred refill order). + cache_op(&mut exec, 0x0b, 2 << 10, 0); + cache_op(&mut exec, 0x1f, 0x5a5a_5a5a, 0x1111_1111); + for _ in 0..4 { + assert_eq!(run(&mut exec), 0x5a5a_5a5a, "from the staged line"); + } + cache_op(&mut exec, 0x03, 0, 0); + for _ in 0..4 { + assert_eq!(run(&mut exec), 0x0a0a_0a0a, "from the window again"); + } + } + + /// The store side: a compiled store through a warm write micro-TLB entry + /// goes inline into ppmem's window. Once the page holds a staged line the + /// store lands in the line instead: the window and memory do not change, + /// a cached load sees the value, and invalidating the line (it is + /// DirtyExclusive after the store) writes the value back to memory. + #[test] + fn a_compiled_store_to_a_staged_line_stays_in_the_line() { + use crate::cpu::mips_cache_shadow::R10000ShadowCache; + use crate::cpu::mips_cache_v2::MipsCache; + use crate::cpu::mips_isa::{OP_CACHE, OP_SW}; + let pc = 0xFFFF_FFFF_8000_1000u64; + let ra = 0xFFFF_FFFF_8000_8000u64; + let data_va = 0xFFFF_FFFF_8000_2000u64; // PA 0x2000 + let sw = make_i(OP_SW, 4, 5, 0); + let jr = make_r(crate::cpu::mips_isa::OP_SPECIAL, 31, 0, 0, 0, crate::cpu::mips_isa::FUNCT_JR); + let mem = Arc::new(MockMemory::new()); + for (i, w) in [sw, jr, 0].into_iter().enumerate() { + mem.set_word(0x1000 + 4 * i as u64, w); + } + let bus: Arc = mem.clone(); + let mut exec = Box::new(MipsExecutor::::new( + bus, PassthroughTlb::default(), &MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))))); + exec.core.cp0_config = (exec.core.cp0_config & !7) | 3; // KSEG0 cacheable + exec.core.cp0_status &= !crate::cpu::mips_core::STATUS_ERL; + exec.resync_privilege_state(); + let mut window = vec![0u8; 0x10000]; + let mut gens: Vec = (0..16).map(|_| AtomicU64::new(0)).collect(); + unsafe { + exec.set_tcache_window(window.as_mut_ptr()); + exec.set_tcache_gen_window(gens.as_mut_ptr()); + *exec.tcache_bitmap_ptr() = 1; + } + exec.core.ppmem_bitmap = 1; + exec.install_jit_hooks(); + assert!(exec.cache.jit_dc_geometry().supported, "the inline path is on"); + exec.jitv2_inline_compile = true; + exec.core.hot.interrupts.store(0, std::sync::atomic::Ordering::Relaxed); + #[cfg(feature = "fetchverify")] + { exec.core.fetch_verify_on = false; } + let win = |window: &Vec| unsafe { *(window.as_ptr().add(0x2000) as *const u32) }; + let run = |exec: &mut MipsExecutor, v: u64| { + exec.core.pc = pc; + exec.core.gpr[31] = ra; + exec.core.gpr[4] = data_va; + exec.core.gpr[5] = v; + for _ in 0..16 { + if exec.core.pc == ra { break; } + exec.step_jit(); + } + assert_eq!(exec.core.pc, ra, "the code returns"); + }; + let cache_op = |exec: &mut MipsExecutor, op: u32, lo: u32, hi: u32| { + exec.core.gpr[6] = data_va; + exec.core.cp0_taglo = lo as u64; + exec.core.cp0_taghi = hi; + assert_eq!(exec.exec(make_i(OP_CACHE, 6, op, 0)), crate::cpu::mips_exec::EXEC_COMPLETE); + }; + for v in 1..=4u64 { + run(&mut exec, v); + assert_eq!(win(&window), v as u32, "into the window"); + } + cache_op(&mut exec, 0x0b, 2 << 10, 0); + cache_op(&mut exec, 0x1f, 0, 0); + for v in 0x10..0x14u64 { + run(&mut exec, v); + assert_eq!(win(&window), 4, "the window is left alone"); + assert_eq!(exec.cache.read::<4>(data_va, 0x2000).data, v, "the line has it"); + } + assert_eq!(mem.get_word(0x2000), 0, "and memory"); + cache_op(&mut exec, 0x03, 0, 0); + assert_eq!(mem.get_word(0x2000), 0x13, "written back when the line goes"); + run(&mut exec, 0x20); + assert_eq!(win(&window), 0x20, "back to the window"); + } + #[test] fn lwl_lwr_combine_to_a_full_unaligned_word_load_matches_interpreter() { // The canonical compiler-generated idiom for an unaligned word @@ -7879,4 +8267,74 @@ mod tests { println!("{:<12} {:8.3} ns/instruction", name, ns); } } + + /// R10000: a refused uncached store completes in compiled code exactly as + /// in the interpreter, through both store sites (SW, and SWL by the + /// masked-store helper), and the instructions after it run. The same access + /// as a load, and a store on the R5000, still fault in both engines. + #[test] + fn refused_uncached_store_matches_the_interpreter_on_the_r10000_family_and_r5000() { + use crate::cpu::mips_cache_v2::{R10000Cache, R5000Cache, CpuModel}; + use crate::cpu::mips_core::STATUS_EXL; + use crate::cpu::mips_isa::{OP_ADDIU, OP_LW, OP_SW, OP_SWL}; + + /// Nothing answers: every read and write is refused (the trait's defaults). + struct RefusingBus; + impl BusDevice for RefusingBus {} + + fn run>>(op: u32) -> (CoreSnapshot, CoreSnapshot) { + let pc = 0xFFFF_FFFF_8000_0000u64; + let mut gpr = [0u64; 32]; + gpr[16] = 0xFFFF_FFFF_A100_0000; // kseg1: uncached, and nothing answers + gpr[6] = 0x1234_5678; + let words = [make_i(op, 16, 6, 0), make_i(OP_ADDIU, 0, 7, 2)]; + let build = || { + let bus: Arc = Arc::new(RefusingBus); + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))); + let mut exec = Box::new(MipsExecutor::::new(bus, PassthroughTlb::default(), &cfg)); + exec.core.gpr = gpr; + exec.core.pc = pc; + exec.core.hot.interrupts.store(0, std::sync::atomic::Ordering::Relaxed); + #[cfg(feature = "fetchverify")] + { exec.core.fetch_verify_on = false; } + exec + }; + let mut interp = build(); + interp.jitv2_dispatch_enabled = false; + for (i, &word) in words.iter().enumerate() { + if interp.core.pc != pc + 4 * i as u64 { + break; // an exception redirected the PC + } + interp.exec(word); + } + + let mut page = [0u32; ENTRIES_PER_PAGE]; + page[..words.len()].copy_from_slice(&words); + let page_base = (pc & !(PAGE_SIZE as u64 - 1)) as u32; + let mut analyzer = Analyzer::new(); + let (walked, non_empty) = analyzer.walk_bounded(&page, 0, page_base, words.len()); + assert!(non_empty); + let mut instrs_owned = *walked; + let mut codegen = Codegen::new(); + let jit_fn: JitFn = codegen.compile_region(&mut instrs_owned, 0, true, false) + .expect("region must be compilable for this test to be meaningful"); + let mut jit = build(); + jit.install_jit_hooks(); + unsafe { jit_fn(&mut jit.core as *mut MipsCore) }; + std::mem::forget(codegen); + (CoreSnapshot::capture(&interp.core), CoreSnapshot::capture(&jit.core)) + } + + for op in [OP_SW, OP_SWL] { + let (interp, jit) = run::(op); + assert_eq!(jit, interp, "op {op:#x}: JIT and interpreter diverged"); + assert_eq!(jit.cp0_status & STATUS_EXL, 0, "op {op:#x}: no exception"); + assert_eq!(jit.gpr[7], 2, "op {op:#x}: the instruction after the store ran"); + } + for (name, (interp, jit)) in [("R10000 load", run::(OP_LW)), ("R5000 store", run::(OP_SW))] { + assert_eq!(jit, interp, "{name}: JIT and interpreter diverged"); + assert_ne!(jit.cp0_status & STATUS_EXL, 0, "{name}: must take its Data Bus Error"); + assert_eq!(jit.gpr[7], 0, "{name}: nothing after the fault ran"); + } + } } diff --git a/src/cpu/jitv2/jitv2.rs b/src/cpu/jitv2/jitv2.rs index 9fd970165..766285d9a 100644 --- a/src/cpu/jitv2/jitv2.rs +++ b/src/cpu/jitv2/jitv2.rs @@ -80,6 +80,43 @@ pub fn pfn_of(phys_addr: u64, frames: u64) -> Option { (pfn < frames).then_some(pfn as Pfn) } +/// Set in a page key when the CPU fetched the page uncached on a machine +/// whose uncached bus view decodes differently from its cached one (IP35). +/// The key is `pfn | UNCACHED_VIEW_PFN`; the other machines never set it. +/// It is bit 22, above every frame of every machine: the largest physical +/// window is 16 GiB, [`PFN_MAP_ENTRIES`] = 1 << 22 frames, and [`pfn_of`] +/// admits no frame at or past the machine's `frames`. So stripping it is a +/// no-op on a machine that never sets it, however much memory that machine +/// has (an 8 GB Octane's frames reach bit 21). A constant, so the hot paths +/// that strip it need no load. See [`code_key`]. +pub const UNCACHED_VIEW_PFN: Pfn = PFN_MAP_ENTRIES as Pfn; + +/// Set in the bus address a code page is read or generation-tracked through +/// when the page was fetched uncached. A bus that has no separate uncached +/// code view never sees it (the key bit is never set there). +pub const UNCACHED_VIEW_ADDR: u64 = 1 << 63; + +/// The page key for a fetch at `phys_addr`: its frame below `frames`, plus +/// [`UNCACHED_VIEW_PFN`] when `uncached_view` says this fetch reached memory +/// through the uncached view. +#[inline(always)] +pub fn code_key(phys_addr: u64, frames: u64, uncached_view: bool) -> Option { + pfn_of(phys_addr, frames).map(|pfn| if uncached_view { pfn | UNCACHED_VIEW_PFN } else { pfn }) +} + +/// Physical base address of the frame a page key names, without the view bit. +#[inline(always)] +pub fn key_phys_base(key: Pfn) -> u64 { + (key & !UNCACHED_VIEW_PFN) as u64 * PAGE_SIZE as u64 +} + +/// Bus address for reading or tracking the page `key`, in the view it names. +#[inline(always)] +pub fn key_bus_addr(key: Pfn, offset: u32) -> u64 { + let addr = key_phys_base(key) + offset as u64; + if key & UNCACHED_VIEW_PFN != 0 { addr | UNCACHED_VIEW_ADDR } else { addr } +} + /// Direct-mapped `pfn -> PageSlot` lookup, replacing the `HashMap` both page pools used to carry. /// @@ -1285,6 +1322,18 @@ impl PhysicalCodePage { self.pfn != UNCLAIMED_PFN } + /// Physical base address of the frame this page covers. + #[inline] + pub fn phys_base(&self) -> u64 { + key_phys_base(self.pfn) + } + + /// Whether the CPU fetched this page through the uncached view. + #[inline] + pub fn is_uncached_view(&self) -> bool { + self.pfn != UNCLAIMED_PFN && self.pfn & UNCACHED_VIEW_PFN != 0 + } + pub fn claim(&mut self, pfn: Pfn, gen: *const AtomicU64, fr1: bool) { debug_assert!(std::ptr::eq(self.gen, &NEVER_COMPILABLE_GEN) && self.pfn == UNCLAIMED_PFN, "claim() called on a slot that wasn't clean (pfn={:#x}) — every path that reuses a slot must reset it first (see reset_to_unclaimed)", @@ -2307,6 +2356,10 @@ pub struct Jitv2 { /// lands on a different PFN than the currently-tracked one) — not on /// every fetch. Direct-mapped, not hashed — see [`PfnMap`]. pfn_to_slot: PfnMap, + /// Frames of the machine's memory, the plain keys `pfn_to_slot` holds. + frames: usize, + /// `pfn_to_slot` also holds uncached-view keys (`enable_uncached_views`). + uncached_views: bool, /// Pool capacity, fixed at construction (== `pages.len()`). Claiming past /// this (i.e. with the free list exhausted) triggers `mega_flush` (the /// "ran out of PCPs" resource-exhaustion trigger). @@ -2414,6 +2467,8 @@ impl Jitv2 { pages, free_head: if capacity > 0 { 0 } else { NO_SLOT }, pfn_to_slot: PfnMap::new(frames), + frames, + uncached_views: false, capacity, compile_queue: CompileQueue::new(), codegen: Mutex::new(Some(crate::cpu::jitv2::codegen::Codegen::new())), @@ -2449,13 +2504,38 @@ impl Jitv2 { } let slot = self.free_head; self.free_head = self.pages[slot as usize].next; - let gen = bus.gen_ptr(phys_addr); + // The generation counter belongs to the view the page is read + // through: `phys_addr` names the frame, the key says which view. + let gen_addr = phys_addr | if pfn & UNCACHED_VIEW_PFN != 0 { UNCACHED_VIEW_ADDR } else { 0 }; + let gen = bus.gen_ptr(gen_addr); self.pages[slot as usize].claim(pfn, gen, fr1); self.pages[slot as usize].next = NO_SLOT; self.pfn_to_slot.insert(pfn, slot); Some(slot) } + /// Whether the pool holds uncached-view pages (`enable_uncached_views`). + pub fn uncached_views_enabled(&self) -> bool { + self.uncached_views + } + + /// Let the pool hold uncached-view pages ([`UNCACHED_VIEW_PFN`]): the map + /// grows from `frames` entries to the view bit plus `frames`, so each + /// view's keys index it directly, as plain frames do. Call before any + /// page is claimed and before an executor captures the lookup pointers + /// (`jitv2_bind_fast_lookup`); the map is reallocated, to 16 MiB plus + /// four bytes a frame. + pub fn enable_uncached_views(&mut self) { + if self.uncached_views { + return; + } + assert_eq!(self.pages_used(), 0, "uncached code views must be enabled before any page is claimed"); + assert!(self.frames <= UNCACHED_VIEW_PFN as usize, + "memory reaches past the page table's largest window ({} frames)", self.frames); + self.pfn_to_slot = PfnMap::new(UNCACHED_VIEW_PFN as usize + self.frames); + self.uncached_views = true; + } + /// Pointers for lock-free `pfn -> *mut PhysicalCodePage` fast lookup. pub fn fast_lookup_ptrs(&mut self) -> (*const AtomicU32, *mut PhysicalCodePage) { (self.pfn_to_slot.as_ptr(), self.pages.as_mut_ptr()) @@ -5167,6 +5247,52 @@ mod tests { assert_eq!(pfn_of(0x3000_0000, indy), None); } + /// A machine that never sets the view bit keeps its frames' true + /// addresses, including frames at or above 4 GiB (bit 20 of a frame + /// number), as an Octane with 8 GB has. + #[test] + fn frames_above_4_gib_keep_their_address_without_the_view() { + let octane = frames_for(&crate::ip30::memory_map(&[2048, 2048, 2048, 2048])); + for addr in [0x1_0000_1000u64, 0x1_ffff_f000, 0x2_1fff_f000] { + let key = code_key(addr, octane, false).unwrap(); + assert_eq!(key, (addr / PAGE_SIZE as u64) as Pfn); + assert_eq!(key & UNCACHED_VIEW_PFN, 0, "{addr:#x}"); + assert_eq!(key_phys_base(key), addr & !(PAGE_SIZE as u64 - 1), "{addr:#x}"); + assert_eq!(key_bus_addr(key, 8), (addr & !(PAGE_SIZE as u64 - 1)) + 8, "{addr:#x} has no view address bit"); + } + let dev = FakeDevice(AtomicU64::new(0)); + let mut jit = Jitv2::with_frames(4, octane as usize); + let slot = jit.page_for(0x100001, 0x1_0000_1000, &dev, false).unwrap(); + let page = unsafe { &*jit.page_ptr(slot) }; + assert_eq!(page.phys_base(), 0x1_0000_1000); + assert!(!page.is_uncached_view()); + assert_eq!(key_bus_addr(page.pfn, 0), 0x1_0000_1000); + } + + /// An uncached-view key still names its frame and view when the frame is + /// above 4 GiB, and the view bit stays clear of the largest frame. + #[test] + fn the_uncached_view_survives_frames_above_4_gib() { + let frames = PFN_MAP_ENTRIES as u64; + assert_eq!(UNCACHED_VIEW_PFN as u64, frames); + for addr in [0x1fc0_0000u64, 0x1_0000_1000, 0x3_ffff_f000] { + let key = code_key(addr, frames, true).unwrap(); + assert_ne!(key & UNCACHED_VIEW_PFN, 0); + assert_eq!(key_phys_base(key), addr & !(PAGE_SIZE as u64 - 1), "{addr:#x}"); + assert_eq!(key_bus_addr(key, 0), (addr & !(PAGE_SIZE as u64 - 1)) | UNCACHED_VIEW_ADDR, "{addr:#x}"); + } + let dev = FakeDevice(AtomicU64::new(0)); + let mut jit = Jitv2::with_frames(4, 0xd0000); + jit.enable_uncached_views(); + let key = code_key(0x1fc0_0000, 0xd0000, true).unwrap(); + let slot = jit.page_for(key, 0x1fc0_0000, &dev, false).unwrap(); + let page = unsafe { &*jit.page_ptr(slot) }; + assert!(page.is_uncached_view()); + assert_eq!(page.phys_base(), 0x1fc0_0000); + let plain = jit.page_for(0x1fc00, 0x1fc0_0000, &dev, false).unwrap(); + assert_ne!(plain, slot, "the two views of a frame are two pages"); + } + #[test] fn page_for_allocates_once_and_caches_by_pfn() { let dev = FakeDevice(AtomicU64::new(0)); diff --git a/src/cpu/jitv2/mod.rs b/src/cpu/jitv2/mod.rs index 60e01f749..6aeeb5b6e 100644 --- a/src/cpu/jitv2/mod.rs +++ b/src/cpu/jitv2/mod.rs @@ -92,6 +92,7 @@ mod zz_corpus { // (baked immediate vs load) is identical to production. crate::cpu::jitv2::codegen::JitConsts { core: core::num::NonZeroUsize::new(core as *mut MipsCore as usize), + ..Default::default() } }; // Synthetic corpus-measurement geometry: 16 KiB direct-mapped L1-D diff --git a/src/cpu/jitv2/pcp_dump.rs b/src/cpu/jitv2/pcp_dump.rs index 7c0032b0c..d19bf713e 100644 --- a/src/cpu/jitv2/pcp_dump.rs +++ b/src/cpu/jitv2/pcp_dump.rs @@ -134,11 +134,11 @@ impl PcpDump { /// possibility for a debugging dump, not a correctness-critical path). /// Returns `Err` with the failing physical address if any word read fails. pub fn capture(page: &PhysicalCodePage, bus: &dyn BusDevice) -> Result { - let phys_base = page.pfn as u64 * PAGE_SIZE as u64; + let phys_base = page.phys_base(); let mut words = [0u32; ENTRIES_PER_PAGE]; for (i, w) in words.iter_mut().enumerate() { let addr = phys_base + i as u64 * 4; - let r = bus.read32(addr); + let r = bus.read32(crate::cpu::jitv2::jitv2::key_bus_addr(page.pfn, (i as u32) * 4)); if !r.is_ok() { return Err(addr); } diff --git a/src/cpu/jitv2_html_j2wp.rs b/src/cpu/jitv2_html_j2wp.rs index 3bf2585b5..0bcdf4adf 100644 --- a/src/cpu/jitv2_html_j2wp.rs +++ b/src/cpu/jitv2_html_j2wp.rs @@ -477,13 +477,13 @@ pub fn write_jitv2_html( // (Jitv2::code_bytes_used doc comment) — fine for an on-demand dump, // not a hot path. for page in jit.claimed_pages() { - let phys_addr = page.pfn as u64 * crate::cpu::jitv2::PAGE_SIZE as u64; + let phys_addr = page.phys_base(); let window = memory.find(phys_addr).map_or(OTHER_REGION, |r| r.name); let page_gen = page.current_gen(); let mut words = [0u32; crate::cpu::jitv2::ENTRIES_PER_PAGE]; for (i, w) in words.iter_mut().enumerate() { - *w = bus.read32(phys_addr + i as u64 * 4).data; + *w = bus.read32(crate::cpu::jitv2::jitv2::key_bus_addr(page.pfn, (i as u32) * 4)).data; } // §13: one function/gen/instr_count/code_size/call_count/ diff --git a/src/cpu/mips_cache_shadow.rs b/src/cpu/mips_cache_shadow.rs index 53b6673eb..6c9436fdc 100644 --- a/src/cpu/mips_cache_shadow.rs +++ b/src/cpu/mips_cache_shadow.rs @@ -23,19 +23,34 @@ //! to it and returns them unchanged, so a walking-1s test over tag or data //! SRAM passes without anyone having to know the hardware's field layout. //! Where the emulator *must* know the layout — to decide whether a line is -//! valid, say — that is a separate question from storing the bits. +//! valid, say — that is a separate question from storing the bits. The one +//! exception is the R10000's secondary tag, whose reserved bits read as zero +//! (`R10K_L2_TAG_MASK`). //! -//! The shadow is deliberately not consulted by `read`/`write`/`fetch`. If it -//! ever needs to be, this type is the wrong shape and should say so loudly -//! rather than growing a slow path. +//! The shadow is not consulted by `read`/`write`/`fetch`, with one exception +//! that costs a normal access one predicted branch: on the R10000 family, a +//! secondary line the guest *staged* — a valid tag written by Index Store Tag +//! and data by Index Store Data — is a real hit for later cached loads, stores +//! and fetches, served from the shadow without a bus access. A line staged +//! in the primary instruction cache the same way serves fetches, before the +//! secondary cache. Its words are stored predecoded, and decode back for the +//! sampled classes only (`l1i_decode`); other classes read memory, with a +//! report. See [`Staged`]. +//! Nothing else about the shadow reaches the data path: a line the guest never +//! staged is memory, as before, and ordinary misses never fill or evict. So a +//! staged line lives until an explicit index or hit operation, an Index Store +//! Tag or power-on ends it; a guest that reuses the memory without +//! invalidating reads the stale line, where hardware would have evicted it. use std::cell::UnsafeCell; +use std::collections::HashMap; use std::sync::Arc; use crate::cpu::mips_cache_v2::{ cache_op_name, model, r4k_load_taglo, r4k_store_taglo, CpuModel, FetchInstrResult, MipsCache, - C_ILT, C_IST, C_R10K_CBARRIER, C_R10K_ILD, C_R10K_ISD, CACH_PD, CACH_PI, CACH_SD, CACH_SI, + C_HINV, C_HWBINV, C_IINV, C_ILT, C_IST, C_IWBINV, C_R10K_CBARRIER, C_R10K_ILD, C_R10K_ISD, CACH_PD, CACH_PI, CACH_SD, CACH_SI, }; +use crate::cpu::mips_exec::EXEC_CACHE_ERR; use crate::traits::{BusDevice, BusRead64, BUS_OK}; use crate::devlog::{LogModule, CACHE_LOG_HIT, CACHE_LOG_OP, devlog_is_active, devlog_mask}; @@ -90,15 +105,180 @@ const MRU_BIT: u64 = 1 << 63; /// R4400 and R5000 index differently — see `tag_slot`. const WAYS: usize = 2; -/// Bits a secondary-cache tag retains. +/// Bits an R10000 secondary-cache tag retains: MRU (bit 63), the physical tag +/// (TagHi 3:0 and TagLo 31:14), state, virtual index and ECC. TagLo bits 9, 12 +/// and 13 and TagHi 30:4 are reserved and read as zero (R10000 UM p. 270, +/// figure 11-31). Other parts keep every bit; see `CpuModel::SECONDARY_TAG_MASK`. +const R10K_L2_TAG_MASK: u64 = 0x8000_000f_ffff_cdff; + +/// The secondary tag fields a cached access is compared against. The tag +/// register pair holds PA[35:18] in TagLo[31:14] and PA[39:36] in TagHi[3:0], +/// which the assembled 64-bit tag keeps at bits 31:14 and 35:32. NEC +/// U10278EJ4V0UM pp.267-270. +const L2_TAG_PA_MASK: u64 = 0xf_ffff_c000; + +/// Where the secondary state field sits in the assembled tag: TagLo[11:10], +/// 0 Invalid, 1 Shared, 2 CleanExclusive, 3 DirtyExclusive. +const L2_STATE_SHIFT: u32 = 10; +const L2_STATE_DIRTY_EXCLUSIVE: u64 = 3; + +/// The primary instruction-cache tag, from the register pair (NEC +/// U10278EJ4V0UM pp.267-270): TagLo[31:8] is PA[35:12], TagLo[6] the state (1 valid), and +/// TagHi[3:0] PA[39:36], kept at bits 35:32 of the assembled tag. +const L1I_TAG_PA_MASK: u64 = 0xf_ffff_ff00; +const L1I_VALID: u64 = 1 << 6; + +/// CacheErr.ED: a data-field error (NEC U10278EJ4V0UM p.265). +const L1I_CACHEERR_ED: u32 = 1 << 29; + +/// Even parity: the ones in the 36 data bits plus the parity bit total an even +/// number (NEC U10278EJ4V0UM pp.190-191). +#[inline(always)] +fn l1i_parity_ok(p: u64, parity: u32) -> bool { + (p.count_ones() + parity) & 1 == 0 +} + +/// The parity bit that makes `p` even. +#[cfg(test)] +fn l1i_parity_of(p: u64) -> u32 { + p.count_ones() & 1 +} + +/// A primary instruction-cache word is the instruction predecoded: 36 bits +/// that fetch decodes back. Neither NEC manual gives the layout. This is the +/// one inferred from the words a shipped PROM stages, for sampled opcodes +/// only, with `p` the 36-bit word and `f` the instruction: +/// p[25:20] = f[5:0], p[19:15] = f[10:6], p[14:10] = f[25:21], +/// p[28:26] = f[28:26], p[35:29] = a 7-bit class, +/// p[4:0] and p[9:5] = f[20:16] and f[15:11] (rt and rd positions), in the +/// order the opcode gives. +/// f[31:29] is not in any field, so it comes from the class. The class table +/// is what the samples cover, and it is all this model knows: an entry gives +/// the class, f[31:29], and whether p[4:0] is the rt position. Every other +/// class is refused (`Err(class)`), never guessed. /// -/// All of them, now. A 36-bit mask was inferred here from the PROM's -/// walking-0s phase, which writes an all-ones TagLo and expects back -/// 0x0000000f_ffffcdfe — but that truncation comes from the tag being -/// assembled as `(TagHi << 32) | TagLo[31:0]`, not from the array dropping -/// bits. Once the executor carried TagHi properly the mask did nothing except -/// discard the MRU bit, which the PROM writes as TagHi[31] and reads back. -const L2_TAG_MASK: u64 = u64::MAX; +/// A sampled example: `addiu $4,$0,2` (0x24040002) is 0x3_4420_0004. +const L1I_CLASSES: &[L1iClass] = &[ + // Every row is inferred from sampled words. + L1iClass { class: 0x00, hi3: 0b000, rt_low: false, ops: &[0], funct: &[0x00] }, + L1iClass { class: 0x08, hi3: 0b000, rt_low: false, ops: &[0], funct: &[0x38, 0x3c] }, // dsll, dsll32 + L1iClass { class: 0x18, hi3: 0b000, rt_low: false, ops: &[0], funct: &[0x25] }, // or + L1iClass { class: 0x1a, hi3: 0b001, rt_low: true, ops: &[0, 1, 5, 7], funct: &[] }, // addi addiu ori lui + L1iClass { class: 0x1e, hi3: 0b011, rt_low: true, ops: &[1], funct: &[] }, // daddiu + L1iClass { class: 0x35, hi3: 0b101, rt_low: false, ops: &[7], funct: &[] }, // cache + L1iClass { class: 0x36, hi3: 0b110, rt_low: true, ops: &[7], funct: &[] }, // ld + L1iClass { class: 0x3f, hi3: 0b010, rt_low: false, ops: &[0], funct: &[] }, // mtc0 + L1iClass { class: 0x40, hi3: 0b000, rt_low: false, ops: &[3], funct: &[] }, // jal + L1iClass { class: 0x41, hi3: 0b000, rt_low: false, ops: &[0, 2], funct: &[0x08] }, // jr (op 0, funct 8), j (op 2) + L1iClass { class: 0x71, hi3: 0b000, rt_low: false, ops: &[1], funct: &[] }, // REGIMM +]; + +/// One row of the class table. `ops` are the p[28:26] values (f[28:26]) the +/// class holds, `funct` the function fields allowed for its SPECIAL members +/// (checked when p[28:26] is 0 and the class is a SPECIAL one), `rt_low` +/// whether p[4:0] is the rt position (else the rd position). +struct L1iClass { + class: u8, + hi3: u32, + rt_low: bool, + ops: &'static [u8], + funct: &'static [u8], +} + +/// **Model choice, hardware contents unknown.** A word whose p[35:32] is 0xf +/// is this model's encoding of a raw instruction: p[31:0] is the instruction. +/// It exists so a word that was never in the sampled set can still round-trip +/// through Index Load Data and Index Store Data and execute. It is not a +/// claim about what hardware stores. It uses a p[35:32] no sampled class has. +const L1I_RAW: u64 = 0xf; + +/// The instruction a staged word encodes, or the class that is refused. +fn l1i_decode(p: u64) -> Result { + if (p >> 32) & 0xf == L1I_RAW { + return Ok(p as u32); + } + let class = (p >> 29) as u8 & 0x7f; + let row = L1I_CLASSES.iter().find(|c| c.class == class).ok_or(class)?; + let op = (p >> 26) as u8 & 7; + let funct = (p >> 20) as u8 & 0x3f; + // The class, p[28:26] and for the SPECIAL members the function field + // identify the instruction; anything else in a known class is refused. + let special = !row.funct.is_empty() && op == 0; + if !row.ops.contains(&op) || (special && !row.funct.contains(&funct)) || (class == 0 && p != 0) { + return Err(class); + } + let (hi3, rt_low) = (row.hi3, row.rt_low); + let (low, high) = ((p & 0x1f) as u32, ((p >> 5) & 0x1f) as u32); + let (rt, rd) = if rt_low { (low, high) } else { (high, low) }; + Ok(hi3 << 29 + | ((p >> 26) as u32 & 7) << 26 + | ((p >> 10) as u32 & 0x1f) << 21 + | rt << 16 + | rd << 11 + | ((p >> 15) as u32 & 0x1f) << 6 + | ((p >> 20) as u32 & 0x3f)) +} + +#[inline(always)] +fn l1i_tag_of(pa: u64) -> u64 { + (((pa >> 12) & 0xff_ffff) << 8) | (((pa >> 36) & 0xf) << 32) +} + +/// Does the secondary tag hold a valid line? Every state but Invalid does. +#[inline(always)] +fn l2_tag_valid(tag: u64) -> bool { + (tag >> L2_STATE_SHIFT) & 3 != 0 +} + +/// The tag bits a physical address is compared with. +#[inline(always)] +fn l2_tag_of(pa: u64) -> u64 { + (((pa >> 18) & 0x3_ffff) << 14) | (((pa >> 36) & 0xf) << 32) +} + +/// **Inferred, not established.** Which word of a staged doubleword a word +/// access at the lower address sees. A secondary quadword's doubleword is +/// staged as TagHi (bits 63:32) and TagLo (bits 31:0), but neither NEC manual +/// says how those halves become the words of a primary refill. The only +/// evidence is software that stages a line this way and then fetches it: it +/// is consistent +/// only with the TagLo word landing at the lower address, and that is a +/// simulation, not a measurement. `false` is the manual-literal order. Index +/// Load Data is unaffected; it returns the registers as they were stored. +const STAGED_TAGLO_AT_LOWER_ADDRESS: bool = true; + +/// A staged doubleword as a big-endian memory doubleword: what a `ld` at its +/// address returns. The conversion is its own inverse. +#[inline(always)] +fn staged_to_memory(d: u64) -> u64 { + if STAGED_TAGLO_AT_LOWER_ADDRESS { d.rotate_left(32) } else { d } +} + +/// The secondary lines a guest has made valid, and so what a cached access +/// must consult before memory. +/// +/// A line is staged when its tag is valid; its data are whatever Index Store +/// Data put in the data shadow, so the shadow arrays *are* the staged lines and +/// this holds only the index needed to avoid looking: `valid` for the hot paths +/// and `pages` for the page-granular ones. With `valid == 0`, which is every +/// guest that never stages a line, `read`, `write` and `fetch` pay one load and +/// a not-taken branch. +/// +/// The page set exists because translations are cached per page. The +/// interpreter's and the JIT's inline loads and stores reach RAM through +/// ppmem's window without calling the cache, and the JIT compiles a code page +/// from memory; both are kept off a page in `pages` (`MipsCache::page_staged`), +/// so every access to it comes through the cache. That is the guard against +/// stale compiled code that SMC needs too, done once at the moment a line +/// becomes valid instead of on every store. +struct Staged { + /// Secondary tags with a valid state. + valid: u32, + /// Pages holding at least one such line, and how many. + pages: HashMap, + /// `pages` gained or lost a member since `take_staged_changed`. + changed: bool, +} /// Is the shadow cache's operation trace on, at this level of detail? /// @@ -164,6 +344,17 @@ pub struct ShadowCache< decode: UnsafeCell, /// See [`MipsCache::set_pd_tag_store_hook`]. Consulted by `cache_op` only. pd_tag_hook: UnsafeCell>>, + /// Secondary lines the guest staged. R10000 family only; empty otherwise. + staged: UnsafeCell, + /// The translated address of the CACHE operand about to arrive in + /// `cache_op`: the R10000's secondary index and way come from it. + op_pa: UnsafeCell>, + /// Cache Error raised by the last fetch from a staged primary I word with + /// bad parity (`take_cache_err`), and whether errors are detected now. + l1i_err: UnsafeCell, + l1i_detect: UnsafeCell, + /// Predecode classes already reported as refused, to warn once each. + l1i_refused: UnsafeCell>, } // Safety: the CPU thread is the only accessor, as for every other cache model @@ -191,8 +382,12 @@ impl< const FIR: u32, const TLB_ENTRIES: usize, const MODEL: u8, > ShadowCache { - const R10K: bool = MODEL == model::R10000; + const R10K: bool = model::r10k_family(MODEL); const R5K: bool = MODEL == model::R5000; + /// Index Writeback Invalidate (D) also zeroes the primary data tag on the + /// R12000 and later; the R10000 leaves it as it was. NEC VR12000 user's + /// manual, Appendix B.20. The R14000 and R16000 are taken to match. + const IWBI_CLEARS_PD_TAG: bool = Self::R10K && MODEL != model::R10000; const IC_LINES: usize = if IC_LINE == 0 { 0 } else { IC_SIZE / IC_LINE }; const DC_LINES: usize = if DC_LINE == 0 { 0 } else { DC_SIZE / DC_LINE }; const L2_LINES: usize = if L2_LINE == 0 { 0 } else { L2_SIZE / L2_LINE }; @@ -204,11 +399,11 @@ impl< lladdr: UnsafeCell::new(0), ecc_in: UnsafeCell::new(0), ecc_out: UnsafeCell::new(0), - ic: UnsafeCell::new(Shadow::new(Self::IC_LINES, 0)), - dc: UnsafeCell::new(Shadow::new(Self::DC_LINES, 0)), - // Only the R10000 secondary keeps a data shadow: it is the one - // the IP28 PROM walks with Index_Store_Data, and a 1 MB array is - // cheap once. The R4000 family has no index data ops. + // The R10000 keeps data arrays for all three caches, for the PROM's + // Index_Load_Data / Index_Store_Data walks. The R4000 family has no + // index data ops. + ic: UnsafeCell::new(Shadow::new(Self::IC_LINES, if Self::R10K { IC_SIZE / 4 } else { 0 })), + dc: UnsafeCell::new(Shadow::new(Self::DC_LINES, if Self::R10K { DC_SIZE / 4 } else { 0 })), l2: UnsafeCell::new(Shadow::new(Self::L2_LINES, if Self::R10K { L2_SIZE / 8 } else { 0 })), tc_base: UnsafeCell::new(std::ptr::null_mut()), tc_bitmap: UnsafeCell::new(0), @@ -217,6 +412,11 @@ impl< tc_gen: UnsafeCell::new(std::ptr::null_mut()), decode: UnsafeCell::new(crate::cpu::decode_cache::DecodeCache::new()), pd_tag_hook: UnsafeCell::new(None), + staged: UnsafeCell::new(Staged { valid: 0, pages: HashMap::new(), changed: false }), + op_pa: UnsafeCell::new(None), + l1i_err: UnsafeCell::new(0), + l1i_detect: UnsafeCell::new(true), + l1i_refused: UnsafeCell::new(Vec::new()), } } @@ -296,17 +496,428 @@ impl< /// Data slot for an R10000 `Index_Load_Data` / `Index_Store_Data`. /// - /// Same shape: way in bit 0, the rest addressing the array. The PROM walks - /// it at `…00`, `…01`, `…10`, `…11`, `…20` — one doubleword per operation - /// with the way bit shifted in beneath it. - fn data_slot(&self, len: usize, virt_addr: u64) -> usize { + /// Same shape: way in bit 0, the rest addressing the array, with four-byte + /// primary and eight-byte secondary slots. This is a model choice, enough + /// for a data walk to read back what it wrote. + fn data_slot(&self, sel: u32, len: usize, virt_addr: u64) -> usize { if len == 0 { return 0; } let way = (virt_addr as usize) & (WAYS - 1); - let word = (virt_addr as usize) >> 4; + let shift = if matches!(sel, CACH_PI | CACH_PD) { 2 } else { 3 }; + let word = (virt_addr as usize) >> shift; (word * WAYS + way) % len } + + /// R10000 `Index_Store_Data`, kept out of line: it runs only in a PROM's + /// diagnostic walk, and inlining it grew `cache_op` and moved the code + /// laid out after it. + #[cold] + #[inline(never)] + fn index_store_data(&self, sel: u32, virt_addr: u64, phys_addr: u64) { + let s = self.shadow(sel); + if s.data.is_empty() { + return; + } + let slot = self.data_slot(sel, s.data.len(), virt_addr); + // Diagnostic field widths: R10000 UM pp. 262, 271-272. + let (data_mask, ecc_mask) = if sel == CACH_PI { + (0xf_ffff_ffff, 1) + } else if sel == CACH_PD { + (0xffff_ffff, 0xf) + } else { + (u64::MAX, 0x3ff) + }; + let ecc = unsafe { *self.ecc_in.get() } & ecc_mask; + s.data[slot] = phys_addr & data_mask; + s.ecc[slot] = ecc; + if matches!(sel, CACH_SI | CACH_SD) { + // Index Store Data (S) writes a quadword: the selected + // doubleword plus a zero partner, sharing the check bits + // (R10000 UM p. 91, figure 5-7). + let partner = slot ^ WAYS; + s.data[partner] = 0; + s.ecc[partner] = ecc; + } + if l2_op_log(CACHE_LOG_HIT) { + crate::dlog!(LogModule::L2c, "shadow: -> ISD slot={slot} data={phys_addr:#018x} ecc={:#x}", + s.ecc[slot]); + } + } + + // ---- Staged secondary lines --------------------------------------- + // + // Everything below is R10000-family only and runs only once a guest has + // made a secondary tag valid. A cached access that misses the primary + // caches looks up both secondary ways at the access's index, and a way + // whose tag holds the access's physical address in a valid state supplies + // the data. Index Store Data (S) wrote a whole quadword, so the other + // doubleword of it reads as zero rather than as memory. + // + // Not modelled, and loudly absent rather than guessed: + // - the primary data cache. A line staged there with the primary index + // operations is not consulted by loads. (A staged primary instruction + // line is: see "Staged primary instruction-cache lines".) A valid + // primary instruction line is served before the secondary cache; + // - replacement. An ordinary cached miss never fills, so never evicts: + // a staged line lives until an index or hit operation, or an Index + // Store Tag, ends it (`clanker/cpu/staged-secondary-cache-lines.md`); + // - tag ECC and data check bits. Nothing is checked when a line hits, so + // a line staged with bad check bits hits without a Cache Error; + // - the virtual-index field, TagLo[8:7]. A mismatch with the access's + // VA[13:12] purges primary lines at the recorded index, and there are + // no primary lines here to purge; + // - a CACHE operand on an uncached page. It is undefined on the R10000; + // the operation goes ahead as if the page were cached; + // - the BadVAddr, Context and EntryHi values after a CACHE-raised TLB + // exception, which the manuals do not give; + // - coherence. Another CPU's or a DMA device's write to memory a CPU + // holds in a staged line does not reach the line, and the line's data + // are not visible to them; + // - the part. Only the R10000 is documented. The R12000, R14000 and + // R16000 use the same tag layout and rules here, which is an inference; + // every part keeps only the bits of `R10K_L2_TAG_MASK`; + // - undefined encodings. Operations with op[1:0] = 10 (`CACH_SI`) are + // undefined on the R10000 and do nothing. + + /// Are there any staged lines? The test every cached access makes. + #[inline(always)] + fn staged_any(&self) -> bool { + Self::R10K_FAMILY && unsafe { (*self.staged.get()).valid } != 0 + } + + /// The tag index and data-shadow slot of the doubleword at `pa` in + /// whichever way holds it as a valid line, if one does. Same layout as `data_slot`: a way's + /// doublewords are every `WAYS`-th slot, indexed by `PA[N-2:3]`. + fn staged_line(&self, pa: u64) -> Option<(usize, usize)> { + let l2 = unsafe { &*self.l2.get() }; + if l2.data.is_empty() { + return None; + } + let set = (pa as usize / L2_LINE) % (Self::L2_LINES / WAYS).max(1); + let want = l2_tag_of(pa); + (0..WAYS).find_map(|way| { + let tag = l2.tags[set * WAYS + way]; + (l2_tag_valid(tag) && (tag ^ want) & L2_TAG_PA_MASK == 0) + .then(|| (set * WAYS + way, (((pa as usize) >> 3) % (l2.data.len() / WAYS)) * WAYS + way)) + }) + } + + /// The data-shadow slot of the doubleword at `pa`, if a staged line holds it. + fn staged_slot(&self, pa: u64) -> Option { + self.staged_line(pa).map(|(_, slot)| slot) + } + + /// A cached load of `SIZE` bytes at `pa` from a staged line, as the + /// zero-extended value; `None` if no staged line holds it. + #[inline(never)] + fn staged_read(&self, pa: u64) -> Option { + let slot = self.staged_slot(pa)?; + let m = staged_to_memory(unsafe { (*self.l2.get()).data[slot] }); + let shift = 64 - 8 * SIZE as u32 - 8 * (pa & 7) as u32; + Some(if SIZE == 8 { m } else { (m >> shift) & ((1u64 << (8 * SIZE)) - 1) }) + } + + /// A cached store of `SIZE` bytes at `pa` into a staged line. The line + /// is a write-back cache's: the store stays in it and memory is not + /// written. Returns whether a staged line took the store. + #[inline(never)] + fn staged_write(&self, pa: u64, val: u64) -> bool { + let shift = 64 - 8 * SIZE as u32 - 8 * (pa & 7) as u32; + let mask = if SIZE == 8 { u64::MAX } else { ((1u64 << (8 * SIZE)) - 1) << shift }; + self.staged_write_masked(pa, val << shift, mask) + } + + /// `staged_write` for bytes selected by `mask` of the doubleword at `pa`, + /// `val` already in memory-doubleword position. + fn staged_write_masked(&self, pa: u64, val: u64, mask: u64) -> bool { + let Some((idx, slot)) = self.staged_line(pa) else { return false }; + let l2 = unsafe { &mut *self.l2.get() }; + let m = (staged_to_memory(l2.data[slot]) & !mask) | (val & mask); + l2.data[slot] = staged_to_memory(m); + // **Inferred.** A store to a CleanExclusive or Shared secondary block + // asks the secondary cache to make it DirtyExclusive (NEC + // U10278EJ4V0UM p.246, performance counter events 14 and 15). The + // manual does not state the transition, so this takes it as + // immediate. A later writeback invalidate then sends the stored bytes + // to memory instead of dropping them. + l2.tags[idx] |= L2_STATE_DIRTY_EXCLUSIVE << L2_STATE_SHIFT; + true + } + + /// A staged line moved from page `old` to page `new` (`None`: it was not, + /// or is no longer, valid). The first line in a page and the last one + /// leaving it change which reads of that page the decode cache may serve + /// from the window, and which translations may be cached. + fn page_move(&self, old: Option, new: Option) { + let st = unsafe { &mut *self.staged.get() }; + let decode = unsafe { &mut *self.decode.get() }; + if let Some(pfn) = old { + st.valid -= 1; + if let Some(n) = st.pages.get_mut(&pfn) { + *n -= 1; + if *n == 0 { + st.pages.remove(&pfn); + st.changed = true; + decode.set_bus_only(pfn, false); + } + } + } + if let Some(pfn) = new { + st.valid += 1; + let n = st.pages.entry(pfn).or_insert(0); + *n += 1; + if *n == 1 { + st.changed = true; + decode.set_bus_only(pfn, true); + } + } + } + + /// Store a secondary tag, keeping the staged-line index in step. + fn set_l2_tag(&self, idx: usize, new: u64) { + let l2 = unsafe { &mut *self.l2.get() }; + let old = l2.tags[idx]; + l2.tags[idx] = new; + let page = |tag: u64| l2_tag_valid(tag).then(|| self.l2_line_pa(tag, idx / WAYS) >> 12); + if l2_tag_valid(old) || l2_tag_valid(new) { + self.page_move(page(old), page(new)); + } + } + + // ---- Staged primary instruction-cache lines ------------------------- + // + // Index Store Tag (I) and Index Store Data (I) make a primary I line + // exactly as the secondary pair does, and a cached fetch that finds one + // takes its word from it before the secondary cache is consulted (NEC + // U10278EJ4V0UM pp.87-88). The primary index is the virtual address: + // VA[13:6] picks the set, VA[0] the way, VA[13:2] the word. The tag + // compares the fetch's physical address. Index Store Data (I) writes one + // word per operation, so a block takes 16. + // + // Model choices, each labelled by how well the sources settle it: + // - a fetch compares both ways at VA[13:6] against the fetch's translated + // PA[39:12]. Stated for Hit Invalidate (I), inferred for a fetch; + // - a fetch hit does not update LRU. Unstated, and no replacement is + // modelled; + // - the stored word is the 36-bit predecoded form, decoded on a fetch hit + // (`l1i_decode`) for the sampled classes only, per the inferred + // layout. Any other class is refused: reported once, and the + // fetch is treated as a miss and reads the secondary cache or memory. + // Chosen over a Cache Error because memory holds the instruction the + // word was built from in all but a few stale entries, so a miss + // executes the right instruction for nearly every word a guest stages, + // where an error would stop a guest the hardware runs; the report keeps + // the refusal visible. This can still hide a stale entry whose class is + // unsampled, which is why it is reported; + // - parity is checked on a fetch hit, even over the 36 bits and the parity + // bit. A bad word posts a Cache Error unless Status.DE or ERL is set; + // - Index Load Data (I) returns the stored word and parity as they were + // stored, so a readback stored back verbatim round-trips and executes. + // The only refill is from a staged secondary line (`l1i_refill`); other + // lines the guest never staged read back as the shadow's contents. Model + // choice, hardware contents unknown: a word with p[35:32] = 0xf is a raw + // instruction in p[31:0] (`L1I_RAW`); + // + // The primary data cache is not served: its tag and data layouts are + // known, but a dirty line writes back to the secondary cache and the + // modifier field has no model here. + + /// Physical address of the line a primary I `tag` at `set` names. + fn l1i_line_pa(&self, tag: u64, set: usize) -> u64 { + (((tag >> 8) & 0xff_ffff) << 12) + | (((tag >> 32) & 0xf) << 36) + | ((set as u64 * IC_LINE as u64) & 0xfff) + } + + /// Store a primary I tag, keeping the staged-line index in step. + fn set_l1i_tag(&self, idx: usize, new: u64) { + let ic = unsafe { &mut *self.ic.get() }; + let old = ic.tags[idx]; + ic.tags[idx] = new; + + let page = |tag: u64| (tag & L1I_VALID != 0).then(|| self.l1i_line_pa(tag, idx / WAYS) >> 12); + if old & L1I_VALID != 0 || new & L1I_VALID != 0 { + self.page_move(page(old), page(new)); + } + } + + /// The tag slot of the valid primary I line holding `pa` at the set `va` + /// indexes, if any. + fn staged_l1i_line(&self, va: u64, pa: u64) -> Option { + let ic = unsafe { &*self.ic.get() }; + if ic.data.is_empty() { + return None; + } + let set = (va as usize / IC_LINE) % (Self::IC_LINES / WAYS).max(1); + let want = l1i_tag_of(pa); + (0..WAYS) + .map(|way| set * WAYS + way) + .find(|&idx| ic.tags[idx] & L1I_VALID != 0 && (ic.tags[idx] ^ want) & L1I_TAG_PA_MASK == 0) + } + + /// The instruction a fetch at `va`/`pa` finds in a staged primary I line, + /// then (primary before secondary) in a staged secondary line. `Err` is a + /// Cache Error status. `fetched` is the word being fetched, as opposed to + /// the one after it that opcode fusion peeks at, which never raises. + #[inline(never)] + fn staged_fetch_word(&self, va: u64, pa: u64, fetched: bool) -> Result, u32> { + if let Some(idx) = self.staged_l1i_line(va, pa) { + let ic = unsafe { &*self.ic.get() }; + let slot = (((va as usize) >> 2) % (ic.data.len() / WAYS)) * WAYS + idx % WAYS; + let (p, parity) = (ic.data[slot] & 0xf_ffff_ffff, ic.ecc[slot] & 1); + // Even parity over the 36 bits and the parity bit (NEC + // U10278EJ4V0UM pp.190-191). A bad word posts a Cache Error unless + // errors are not being detected (Status.DE, ERL). + if fetched && !l1i_parity_ok(p, parity) && unsafe { *self.l1i_detect.get() } { + // ED: data field error; the primary index in PIdx. + unsafe { *self.l1i_err.get() = L1I_CACHEERR_ED | (pa as u32 & 0x003f_fff8) | (va >> 12) as u32 & 7 }; + return Err(EXEC_CACHE_ERR); + } + match l1i_decode(p) { + Ok(w) => return Ok(Some(w)), + Err(class) => { + // Refused: see `l1i_decode`. The fetch is treated as a + // miss and reads the secondary cache or memory, and the + // class is reported once. + let seen = unsafe { &mut *self.l1i_refused.get() }; + if !seen.contains(&class) { + seen.push(class); + eprintln!("iris: staged primary I word {p:#011x} at {pa:#x}: predecode class {class:#04x} is not in the sampled set; fetching memory instead"); + } + } + } + } + if fetched && self.staged_any() && self.staged_slot(pa).is_some() { + self.l1i_refill(va, pa); + return self.staged_fetch_word_primary(va, pa); + } + Ok(self.staged_read::<4>(pa).map(|w| w as u32)) + } + + /// The word of the primary line just refilled for `va`/`pa`. + fn staged_fetch_word_primary(&self, va: u64, pa: u64) -> Result, u32> { + match self.staged_l1i_line(va, pa) { + Some(idx) => { + let ic = unsafe { &*self.ic.get() }; + let slot = (((va as usize) >> 2) % (ic.data.len() / WAYS)) * WAYS + idx % WAYS; + Ok(Some(ic.data[slot] as u32)) + } + None => Ok(self.staged_read::<4>(pa).map(|w| w as u32)), + } + } + + /// A cached fetch missed the primary I cache and hit a staged secondary + /// line: refill the 64-byte primary block from it (NEC U10278EJ4V0UM + /// pp.87-88: a primary instruction miss refills 64 bytes from the + /// secondary cache). Only this path fills a primary line; ordinary fetches + /// do not track residency. + /// Model choices: the victim is an Invalid way before a Valid one, and way + /// 0 when both are valid (the manual states no replacement for a refill); + /// the refilled words are stored in the model's raw encoding + /// (`L1I_RAW`), the architectural word carried in place of the predecoded + /// form hardware stores, with its even parity. + fn l1i_refill(&self, va: u64, pa: u64) { + let ic = unsafe { &mut *self.ic.get() }; + if ic.data.is_empty() { + return; + } + let set = (va as usize / IC_LINE) % (Self::IC_LINES / WAYS).max(1); + let way = (0..WAYS).find(|&w| ic.tags[set * WAYS + w] & L1I_VALID == 0).unwrap_or(0); + let idx = set * WAYS + way; + let (vbase, pbase) = (va & !(IC_LINE as u64 - 1), pa & !(IC_LINE as u64 - 1)); + let per_way = ic.data.len() / WAYS; + for i in 0..(IC_LINE / 4) as u64 { + let w = self.staged_read::<4>(pbase + 4 * i).unwrap_or(0); + let p = L1I_RAW << 32 | w; + let slot = (((vbase >> 2) + i) as usize % per_way) * WAYS + way; + ic.data[slot] = p; + ic.ecc[slot] = p.count_ones() & 1; + } + self.set_l1i_tag(idx, l1i_tag_of(pa) | L1I_VALID); + } + + /// Invalidating a secondary block invalidates the primary instruction + /// lines it holds (NEC U12754EJ1V0UMJ1 p.17, 19). + fn l1i_purge_block(&self, pbase: u64, len: u64) { + let ic = unsafe { &*self.ic.get() }; + for idx in 0..ic.tags.len() { + let t = ic.tags[idx]; + if t & L1I_VALID != 0 { + let a = self.l1i_line_pa(t, idx / WAYS); + if a >= pbase && a < pbase + len { + self.l1i_invalidate(idx); + } + } + } + } + + /// Clear the valid bit of the primary I line at `idx`. + fn l1i_invalidate(&self, idx: usize) { + let tag = unsafe { (*self.ic.get()).tags[idx] }; + if tag & L1I_VALID != 0 { + self.set_l1i_tag(idx, tag & !L1I_VALID); + } + } + + /// Physical address of the line a secondary `tag` at `set` names: the + /// tag's PA[39:18] and the set's own PA[17:B] below it. + fn l2_line_pa(&self, tag: u64, set: usize) -> u64 { + (((tag >> 14) & 0x3_ffff) << 18) + | (((tag >> 32) & 0xf) << 36) + | ((set as u64 * L2_LINE as u64) & 0x3_ffff) + } + + /// Invalidate the secondary line at tag slot `idx`: a DirtyExclusive one + /// is first written back to the system interface, then its state goes to + /// Invalid (NEC U12754EJ1V0UMJ1 pp.19, 25, 27). The tag keeps its other + /// fields. The manual also rewrites the tag with the operand's physical + /// address and generates its ECC; neither is modelled, because the + /// shadow stores tags verbatim and has no ECC. + fn l2_invalidate(&self, idx: usize, writeback: bool) { + let l2 = unsafe { &*self.l2.get() }; + let tag = l2.tags[idx]; + if !l2_tag_valid(tag) { + return; + } + if writeback && (tag >> L2_STATE_SHIFT) & 3 == L2_STATE_DIRTY_EXCLUSIVE { + self.l2_write_line_back(idx); + } + let base = self.l2_line_pa(tag, idx / WAYS); + self.set_l2_tag(idx, tag & !(3 << L2_STATE_SHIFT)); + self.l1i_purge_block(base, L2_LINE as u64); + } + + /// Send the block at tag slot `idx` to the system interface, doubleword by + /// doubleword in memory order, at the address its tag names. + fn l2_write_line_back(&self, idx: usize) { + let l2 = unsafe { &*self.l2.get() }; + if l2.data.is_empty() { + return; + } + let (set, way) = (idx / WAYS, idx % WAYS); + let base = self.l2_line_pa(l2.tags[idx], set); + let per_way = l2.data.len() / WAYS; + for i in 0..(L2_LINE / 8) { + let slot = ((set * L2_LINE / 8 + i) % per_way) * WAYS + way; + // A bus error on the way out has nowhere to go. + let _ = self.downstream.write64(base + 8 * i as u64, staged_to_memory(l2.data[slot])); + } + } + + /// Hit Invalidate / Hit Writeback Invalidate (S): act on whichever way + /// holds `pa`. A miss does nothing. Status.CH is not modelled. + fn l2_hit_invalidate(&self, pa: u64, writeback: bool) { + let set = (pa as usize / L2_LINE) % (Self::L2_LINES / WAYS).max(1); + let want = l2_tag_of(pa); + for way in 0..WAYS { + let idx = set * WAYS + way; + let tag = unsafe { (*self.l2.get()).tags[idx] }; + if l2_tag_valid(tag) && (tag ^ want) & L2_TAG_PA_MASK == 0 { + self.l2_invalidate(idx, writeback); + } + } + } } impl< @@ -332,11 +943,10 @@ impl< const PRID: u32 = PRID; const FIR: u32 = FIR; const TLB_ENTRIES: usize = TLB_ENTRIES; - // The R10000 implements 44 virtual address bits where the R4x00 implements - // 40; the shadow cache is only used for it, but key this off the same flag - // that selects its cache encodings rather than asserting it unconditionally. - const VA_BITS: u32 = if MODEL == model::R10000 { 44 } else { 40 }; - const PA_BITS: u32 = if MODEL == model::R10000 { 40 } else { 36 }; + // The R10000 family implements 44 virtual address bits where the R4x00 + // implements 40. + const VA_BITS: u32 = if Self::R10K { 44 } else { 40 }; + const PA_BITS: u32 = if Self::R10K { 40 } else { 36 }; // The CPU's name, which snapshots check: the same as the full model's, so // a mismatch reads as a cache-kind difference rather than a different CPU. // The R10000 keeps its historical "shadow", which existing IP28 @@ -344,10 +954,20 @@ impl< const NAME: &'static str = match MODEL { model::R4400 => "R4400", model::R5000 => "R5000", + model::R12000 => "R12000", + model::R16000 => "R16000", _ => "shadow", }; const CACHE_KIND: &'static str = "shadow"; - const R10K_CACHE_OPS: bool = MODEL == model::R10000; + const R10K_FAMILY: bool = Self::R10K_CACHE_OPS; + // The upper word of a partial LWR is implementation-defined. Keeping it + // is what the IRIX CPU instruction test (olperi) expects and what + // qemu-sgi-origami models; for the R12000 and later it is a model choice. + const PARTIAL_LWR_KEEPS_UPPER: bool = Self::R10K_FAMILY; + const R10K_CACHE_OPS: bool = Self::R10K; + const POSTS_UNCACHED_STORES: bool = Self::R10K_CACHE_OPS; + const SECONDARY_TAG_MASK: u64 = if Self::R10K_CACHE_OPS { R10K_L2_TAG_MASK } else { u64::MAX }; + const CONFIG0_RESET: u32 = if MODEL == model::R16000 { R16000_800_CONFIG0 } else { 0 }; } impl< @@ -391,6 +1011,14 @@ impl< /// nothing a guest can see. fn fetch_is_pure(&self) -> bool { true } + fn page_staged(&self, phys: u64) -> bool { + self.staged_any() && unsafe { (*self.staged.get()).pages.contains_key(&(phys >> 12)) } + } + + fn take_staged_changed(&self) -> bool { + Self::R10K_FAMILY && std::mem::take(unsafe { &mut (*self.staged.get()).changed }) + } + #[cfg(feature = "jitv2")] fn tcache_gen_ptr(&self) -> *mut u8 { unsafe { *self.tc_gen.get() as *mut u8 } } @@ -417,10 +1045,22 @@ impl< } #[inline(always)] - fn fetch(&self, _virt_addr: u64, phys_addr: u64) -> FetchInstrResult { + fn fetch(&self, virt_addr: u64, phys_addr: u64) -> FetchInstrResult { // Decoded instructions by physical page — see `decode_cache`. let dc = unsafe { &mut *self.decode.get() }; + // Only a page the decode cache reads word by word (`set_bus_only`) + // comes here when a line is staged, so this costs a RAM page nothing. let word = |a: u64| { + if self.staged_any() { + // The word being read is at `a`; its virtual address is the + // fetch's, moved by the same distance. + let va = virt_addr.wrapping_add(a.wrapping_sub(phys_addr)); + match self.staged_fetch_word(va, a, a == phys_addr) { + Ok(Some(w)) => return Ok(w), + Ok(None) => {} + Err(status) => return Err(status), + } + } let r = self.downstream.read32(a); if r.is_ok() { Ok(r.data) } else { Err(r.status) } }; @@ -435,6 +1075,11 @@ impl< const { assert!(SIZE == 1 || SIZE == 2 || SIZE == 4 || SIZE == 8, "invalid memory access SIZE") }; + if self.staged_any() { + if let Some(v) = self.staged_read::(phys_addr) { + return BusRead64::ok(v); + } + } // RAM: one load out of ppmem's window. Its layout is u32 words in // native order — byte at `off ^ 3`, halfword at `(off >> 1) ^ 1`, // u64 as rotate_left(32) — the same as CpuCache::tc_read. @@ -471,6 +1116,9 @@ impl< const { assert!(SIZE == 1 || SIZE == 2 || SIZE == 4 || SIZE == 8, "invalid memory access SIZE") }; + if self.staged_any() && self.staged_write::(phys_addr, val) { + return BUS_OK; + } if let Some(base) = self.ram(phys_addr) { let off = phys_addr as usize; unsafe { @@ -499,6 +1147,9 @@ impl< } fn write64_masked(&self, _virt_addr: u64, phys_addr: u64, val: u64, mask: u64) -> u32 { + if self.staged_any() && self.staged_write_masked(phys_addr, val, mask) { + return BUS_OK; + } if let Some(base) = self.ram(phys_addr) { // Compare-exchange, not a read and a store: another CPU's store // to the rest of this doubleword must not be lost. No lock would @@ -530,6 +1181,7 @@ impl< fn cache_op(&self, cache_op: u32, virt_addr: u64, phys_addr: u64) -> u64 { let sel = cache_op & 3; let op = cache_op & 0x1C; + let op_pa = unsafe { (*self.op_pa.get()).take() }; if op == C_IST && sel == CACH_PD { if let Some(hook) = unsafe { &*self.pd_tag_hook.get() } { @@ -545,6 +1197,20 @@ impl< cache_op_name(cache_op)); } + // op[1:0] = 10 is not defined for the R10000 (NEC U12754EJ1V0UMJ1 + // p.16). Do nothing, so it cannot stage or end a line. + if Self::R10K_FAMILY && sel == CACH_SI { + return 0; + } + // The secondary set, way and quadword come from the operand's physical + // address, the primaries' from its virtual address (NEC U12754EJ1V0UMJ1 + // pp.14-15). A caller that gives no physical address (a unit test, or + // an operand that was not translated) is indexed by the address given. + let virt_addr = match op_pa { + Some(pa) if Self::R10K_FAMILY && sel == CACH_SD => pa, + _ => virt_addr, + }; + // R4400/R5000: a tag keeps only its fields, and reads back with // generated check bits — exactly what the full model returns. if !Self::R10K && matches!(op, C_IST | C_ILT) { @@ -562,12 +1228,16 @@ impl< } match op { - // Index_Store_Tag / Index_Load_Tag. Stored and returned verbatim: - // a tag test is a round trip, and round trips do not require - // knowing what the bits mean. + // Index_Store_Tag / Index_Load_Tag. Stored and returned verbatim + // unless the part sets a secondary mask: a tag test is a round trip, + // and round trips do not require knowing what the bits mean. C_IST => { let idx = self.tag_slot(sel, virt_addr); - let mask = if matches!(sel, CACH_SI | CACH_SD) { L2_TAG_MASK } else { u64::MAX }; + let mask = if matches!(sel, CACH_SI | CACH_SD) { + ::SECONDARY_TAG_MASK + } else { + u64::MAX + }; if l2_op_log(CACHE_LOG_OP) { crate::dlog!(LogModule::L2c, "shadow: -> IST va={virt_addr:#012x} idx={idx} stores {:#018x}", phys_addr & mask & !MRU_BIT); @@ -581,7 +1251,14 @@ impl< // here destroyed real tag bits. let set = (idx / WAYS).min(s.mru.len() - 1); s.mru[set] = phys_addr & MRU_BIT != 0; - s.tags[idx] = phys_addr & mask & !MRU_BIT; + let tag = phys_addr & mask & !MRU_BIT; + if Self::R10K_FAMILY && sel == CACH_SD { + self.set_l2_tag(idx, tag); + } else if Self::R10K_FAMILY && sel == CACH_PI { + self.set_l1i_tag(idx, tag); + } else { + s.tags[idx] = tag; + } } 0 } @@ -607,7 +1284,7 @@ impl< C_R10K_CBARRIER if Self::R10K && sel == CACH_PI => 0, C_R10K_ILD if Self::R10K && matches!(sel, CACH_PI | CACH_PD | CACH_SD) => { let s = self.shadow(sel); - let slot = self.data_slot(s.data.len(), virt_addr); + let slot = self.data_slot(sel, s.data.len(), virt_addr); if s.data.is_empty() { 0 } else { @@ -619,19 +1296,49 @@ impl< s.data[slot] } } - C_R10K_ISD if Self::R10K && matches!(sel, CACH_SI | CACH_SD) => { + C_R10K_ISD if Self::R10K => { + self.index_store_data(sel, virt_addr, phys_addr); + 0 + } + + C_IWBINV if Self::IWBI_CLEARS_PD_TAG && sel == CACH_PD => { + let idx = self.tag_slot(sel, virt_addr); let s = self.shadow(sel); - let slot = self.data_slot(s.data.len(), virt_addr); - if !s.data.is_empty() { - s.data[slot] = phys_addr; - s.ecc[slot] = unsafe { *self.ecc_in.get() }; - if l2_op_log(CACHE_LOG_HIT) { - crate::dlog!(LogModule::L2c, "shadow: -> ISD slot={slot} data={phys_addr:#018x} ecc={:#x}", - s.ecc[slot]); - } + if idx < s.tags.len() { + s.tags[idx] = 0; + } + 0 + } + // The secondary operations that end a staged line. The other + // primary ones, and every other cache's, fall through: they move data + // between a cache and memory, and here those never disagree. + C_IWBINV if Self::R10K_FAMILY && sel == CACH_SD => { + let idx = self.tag_slot(sel, virt_addr); + self.l2_invalidate(idx, true); + 0 + } + // Index Invalidate (I) clears the valid bit of the way named, Hit + // Invalidate (I) of whichever way holds the address (NEC + // U12754EJ1V0UMJ1 pp.18, 27). + C_IINV if Self::R10K_FAMILY && sel == CACH_PI => { + let idx = self.tag_slot(sel, virt_addr); + self.l1i_invalidate(idx); + 0 + } + C_HINV if Self::R10K_FAMILY && sel == CACH_PI => { + if let Some(idx) = self.staged_l1i_line(virt_addr, phys_addr) { + self.l1i_invalidate(idx); } 0 } + C_HINV if Self::R10K_FAMILY && sel == CACH_SD => { + self.l2_hit_invalidate(phys_addr, false); + 0 + } + C_HWBINV if Self::R10K_FAMILY && sel == CACH_SD => { + self.l2_hit_invalidate(phys_addr, true); + 0 + } // Invalidate, writeback, fill, and the R4000 hit operations. All // no-ops: there is nothing held that could be stale or dirty. @@ -647,16 +1354,142 @@ impl< s.tags.fill(0); s.mru.fill(false); } + let st = unsafe { &mut *self.staged.get() }; + st.changed |= st.valid != 0; + st.valid = 0; + st.pages.clear(); + unsafe { (*self.decode.get()).clear_bus_only() }; unsafe { (*self.decode.get()).flush() }; } - /// A restore replaces RAM wholesale; drop every decoded instruction. The - /// shadow arrays are not snapshotted (nothing reads them but diagnostics). - fn load_cache_state(&self, _v: &toml::Value) -> Result<(), String> { + /// The staged secondary lines, which cached accesses read: tag, MRU bit, + /// and the line's doublewords with their check bits. The rest of the shadow + /// arrays is for diagnostics, which nothing reads back after a restore, and + /// is not saved. Empty when nothing is staged. + fn save_cache_state(&self) -> toml::Value { + let mut t = toml::value::Table::new(); + if !self.staged_any() { + return toml::Value::Table(t); + } + let l2 = unsafe { &*self.l2.get() }; + let per_way = l2.data.len() / WAYS; + let u = |v: u64| toml::Value::Array(vec![toml::Value::Integer((v & 0xffff_ffff) as i64), toml::Value::Integer((v >> 32) as i64)]); + let mut lines = Vec::new(); + for (idx, &tag) in l2.tags.iter().enumerate() { + if !l2_tag_valid(tag) { + continue; + } + let (set, way) = (idx / WAYS, idx % WAYS); + let slots = (0..L2_LINE / 8).map(|i| ((set * L2_LINE / 8 + i) % per_way) * WAYS + way); + let mut e = toml::value::Table::new(); + e.insert("idx".into(), toml::Value::Integer(idx as i64)); + e.insert("tag".into(), u(tag)); + e.insert("mru".into(), toml::Value::Boolean(l2.mru[set.min(l2.mru.len() - 1)])); + e.insert("data".into(), toml::Value::Array(slots.clone().map(|s| u(l2.data[s])).collect())); + e.insert("ecc".into(), toml::Value::Array(slots.map(|s| toml::Value::Integer(l2.ecc[s] as i64)).collect())); + lines.push(toml::Value::Table(e)); + } + t.insert("staged_l2".into(), toml::Value::Array(lines)); + let ic = unsafe { &*self.ic.get() }; + let per_way = (ic.data.len() / WAYS).max(1); + let mut ilines = Vec::new(); + for (idx, &tag) in ic.tags.iter().enumerate().filter(|(_, &t)| t & L1I_VALID != 0) { + let (set, way) = (idx / WAYS, idx % WAYS); + let words = (0..IC_LINE / 4).map(|i| toml::Value::Integer(ic.data[((set * IC_LINE / 4 + i) % per_way) * WAYS + way] as i64)); + let mut e = toml::value::Table::new(); + e.insert("idx".into(), toml::Value::Integer(idx as i64)); + e.insert("tag".into(), u(tag)); + e.insert("words".into(), toml::Value::Array(words.collect())); + let parity = (0..IC_LINE / 4).map(|i| toml::Value::Integer(ic.ecc[((set * IC_LINE / 4 + i) % per_way) * WAYS + way] as i64)); + e.insert("parity".into(), toml::Value::Array(parity.collect())); + ilines.push(toml::Value::Table(e)); + } + t.insert("staged_l1i".into(), toml::Value::Array(ilines)); + toml::Value::Table(t) + } + + /// A restore replaces RAM wholesale: drop every decoded instruction, end + /// every staged line and bring back the saved ones. + fn load_cache_state(&self, v: &toml::Value) -> Result<(), String> { + let l2 = unsafe { &mut *self.l2.get() }; + if Self::R10K_FAMILY { + for idx in 0..l2.tags.len() { + if l2_tag_valid(l2.tags[idx]) { + self.set_l2_tag(idx, l2.tags[idx] & !(3 << L2_STATE_SHIFT)); + } + } + } + let ic = unsafe { &mut *self.ic.get() }; + if Self::R10K_FAMILY { + for idx in 0..ic.tags.len() { + self.l1i_invalidate(idx); + } + } + let lines = v.get("staged_l2").and_then(|l| l.as_array()); + let u = |v: &toml::Value| -> Result { + let a = v.as_array().filter(|a| a.len() == 2).ok_or("staged line: bad doubleword")?; + let w = |x: &toml::Value| x.as_integer().map(|i| i as u64 & 0xffff_ffff).ok_or("staged line: bad word"); + Ok(w(&a[0])? | w(&a[1])? << 32) + }; + for e in lines.into_iter().flatten().filter(|_| Self::R10K_FAMILY) { + let idx = e.get("idx").and_then(|i| i.as_integer()).ok_or("staged line: no index")? as usize; + if idx >= l2.tags.len() || l2.data.is_empty() { + return Err(format!("staged line index {idx} outside the secondary cache")); + } + let (set, way) = (idx / WAYS, idx % WAYS); + let per_way = l2.data.len() / WAYS; + let data = e.get("data").and_then(|d| d.as_array()).ok_or("staged line: no data")?; + let ecc = e.get("ecc").and_then(|d| d.as_array()).ok_or("staged line: no check bits")?; + if data.len() != L2_LINE / 8 || ecc.len() != data.len() { + return Err("staged line: wrong size for this cache".into()); + } + for (i, (d, c)) in data.iter().zip(ecc).enumerate() { + let slot = ((set * L2_LINE / 8 + i) % per_way) * WAYS + way; + l2.data[slot] = u(d)?; + l2.ecc[slot] = c.as_integer().ok_or("staged line: bad check bits")? as u32; + } + l2.mru[set.min(l2.mru.len() - 1)] = e.get("mru").and_then(|m| m.as_bool()).unwrap_or(false); + self.set_l2_tag(idx, u(e.get("tag").ok_or("staged line: no tag")?)?); + } + for e in v.get("staged_l1i").and_then(|l| l.as_array()).into_iter().flatten().filter(|_| Self::R10K_FAMILY) { + let idx = e.get("idx").and_then(|i| i.as_integer()).ok_or("staged line: no index")? as usize; + let words = e.get("words").and_then(|d| d.as_array()).ok_or("staged line: no words")?; + if idx >= ic.tags.len() || words.len() != IC_LINE / 4 || ic.data.is_empty() { + return Err("staged primary line does not fit this cache".into()); + } + let (set, way) = (idx / WAYS, idx % WAYS); + let per_way = ic.data.len() / WAYS; + let parity = e.get("parity").and_then(|d| d.as_array()).ok_or("staged line: no parity")?; + for (i, w) in words.iter().enumerate() { + let slot = ((set * IC_LINE / 4 + i) % per_way) * WAYS + way; + ic.data[slot] = w.as_integer().ok_or("staged line: bad word")? as u64; + ic.ecc[slot] = parity.get(i).and_then(|x| x.as_integer()).ok_or("staged line: bad parity")? as u32; + } + self.set_l1i_tag(idx, u(e.get("tag").ok_or("staged line: no tag")?)?); + } unsafe { (*self.decode.get()).flush() }; Ok(()) } + /// Dirty staged lines go to memory, and stay staged. + fn writeback_all(&self) { + if !self.staged_any() { + return; + } + let l2 = unsafe { &*self.l2.get() }; + for idx in 0..l2.tags.len() { + if l2_tag_valid(l2.tags[idx]) && (l2.tags[idx] >> L2_STATE_SHIFT) & 3 == L2_STATE_DIRTY_EXCLUSIVE { + self.l2_write_line_back(idx); + } + } + } + + /// `l2 probe` in the monitor: how many secondary lines are staged. + fn debug_probe(&self, _cache_name: &str, _virt_addr: u64, _phys_addr: u64) -> String { + let st = unsafe { &*self.staged.get() }; + format!("staged secondary lines: {} in {} page(s)", st.valid, st.pages.len()) + } + fn get_config(&self, cache_target: u32) -> (usize, usize) { match cache_target { CACH_PI => (IC_SIZE, IC_LINE), @@ -675,12 +1508,28 @@ impl< unsafe { *self.ecc_out.get() } } + fn set_error_control(&self, _ce: bool, detect: bool) { + unsafe { *self.l1i_detect.get() = detect }; + } + + fn take_cache_err(&self) -> u32 { + unsafe { *self.l1i_err.get() } + } + + fn set_cache_op_pa(&self, pa: u64) { + unsafe { *self.op_pa.get() = Some(pa) }; + } + fn downstream(&self) -> Arc { self.downstream.clone() } fn cas_ram32(&self, phys: u64, old: u32, new: u32) -> Option { use std::sync::atomic::{AtomicU32, Ordering}; + // A staged line is not in RAM: the caller stores the ordinary way. + if self.staged_any() && self.staged_slot(phys).is_some() { + return None; + } // Memory beyond the window (the Octane's above 4 GB) is on the bus. let Some(base) = self.ram(phys) else { return self.downstream.cas32(phys, old, new); @@ -695,6 +1544,9 @@ impl< fn cas_ram64(&self, phys: u64, old: u64, new: u64) -> Option { use std::sync::atomic::{AtomicU64, Ordering}; + if self.staged_any() && self.staged_slot(phys).is_some() { + return None; + } let Some(base) = self.ram(phys) else { return self.downstream.cas64(phys, old, new); }; @@ -733,6 +1585,77 @@ impl< pub type R10000ShadowCache = ShadowCache<32768, 64, 32768, 32, 1048576, 128, true, 0x0000_0925, 0x0000_0900, 64, { model::R10000 }>; +/// R12000 3.5 as the Octane has it: the R10000's programming model with a +/// 2 MB secondary cache. PRId 0x0e35, FIR 0x900, 128-byte secondary lines +/// (SB: a model choice). +/// +/// The Octane's R14000 is this type with `R14000_PRID` set at runtime, as the +/// Octane's own R10000 revision is; only the PRId differs. Both report +/// "R12000" as `NAME`. +pub type R12000ShadowCache = + ShadowCache<32768, 64, 32768, 32, 2097152, 128, true, 0x0000_0e35, 0x0000_0900, 64, { model::R12000 }>; + +/// PRId of the R14000 2.4 in the Octane, to set on an `R12000ShadowCache`. +/// Implementation 0x0f is shared with the R16000, which is told apart by a +/// major revision above 2. +pub const R14000_PRID: u32 = 0x0000_0f24; + +/// R12000 3.5 as the O2 has it: the Octane part with 64-byte secondary lines +/// (SB clear). With 128-byte lines the O2 PROM panics before its banner +/// (write TLB miss at 0x0); 64 boots. IRIX on the O2 never reads SB and sizes +/// the secondary cache as 512 KB << SS. +pub type R12000O2ShadowCache = + ShadowCache<32768, 64, 32768, 32, 2097152, 64, true, 0x0000_0e35, 0x0000_0900, 64, { model::R12000 }>; + +/// R14000, as the Fuel has it: the Octane's part at revision 2.3 (PRId +/// 0x0f23), the revision a public software inventory of a Fuel prints. The +/// raw CP0 word, the FIR and the 128-byte lines are taken from the Octane +/// part (`R12000ShadowCache` with `R14000_PRID`), not read from a Fuel. Like +/// the Octane's R14000 it reports "R12000" as `NAME`. +pub type R14000FuelShadowCache = + ShadowCache<32768, 64, 32768, 32, 2097152, 128, true, 0x0000_0f23, 0x0000_0900, 64, { model::R12000 }>; + +/// R16000 2.2, the 800 MHz part, with a 4 MB secondary cache. The Tezro +/// builds it for both its parts, setting the 700 MHz part's PRId and Config +/// word below on it. +/// +/// Values and where they come from: +/// - PRId 0x0f42: the low 16 bits are printed in hinv listings (revision 2.2 +/// under the R16000 name; the major nibble is the printed major plus 2). +/// Bits 31:16 are zero by inference from every full PRId read on this family. +/// - FIR 0x900: a model choice. No R16000 FIR has been printed; this is the +/// R12000 and R14000 value, kept for compatibility. +/// - Config reset word 0x009baf85 (`R16000_800_CONFIG0`): derived +/// from the PROM's per-part clock table (printed clocks and cache size fit +/// it), not read from a running part. SS (4 MB) is inferred from the printed +/// size, the clock fields from the printed 800 MHz CPU and 400 MHz cache +/// clocks. 128-byte lines (SB) are a model choice, and the K0, DN, PM, SK, +/// CT, PE, DDR and bit 24 fields are unknown, carried as the table has them. +/// - Cache tap 0xa is printed in hinv listings and is not modelled. +pub type R16000ShadowCache = + ShadowCache<32768, 64, 32768, 32, 4194304, 128, true, 0x0000_0f42, 0x0000_0900, 64, { model::R16000 }>; + +/// The Config bits an R16000 part's reset word defines: the low 25. IC and DC +/// above them stay the family's. +pub const CONFIG0_RESET_MASK: u32 = 0x01ff_ffff; + +/// Config reset word of `R16000ShadowCache`, the 800 MHz 4 MB part. Same +/// grades as listed there. +pub const R16000_800_CONFIG0: u32 = 0x009b_af85; + +/// PRId of the R16000 2.1, the 700 MHz part, to set on an `R16000ShadowCache`. +/// The low 16 bits are printed in hinv listings as revision 2.1; bits 31:16 are +/// zero by inference. Nothing else in the model differs: both parts have a +/// 4 MB secondary cache. +pub const R16000_MHZ700_PRID: u32 = 0x0000_0f41; + +/// Config reset word of the 700 MHz part. It differs from the 800 MHz word in +/// the EC field only (6 where the 800 MHz part has 7). Derived from the PROM's +/// per-part clock table with the same grades as the 800 MHz word: the 700 MHz +/// CPU clock and 350 MHz cache clock are printed, the other fields unknown. +/// Cache tap 0xc is printed and not modelled. +pub const R16000_MHZ700_CONFIG0: u32 = 0x009b_ad85; + /// SGI O2 R10000 (IP32): the IP28 model with 64-byte secondary-cache lines. /// /// The O2 PROM (4.18 and earlier) handles only that line size. With Config's @@ -767,33 +1690,40 @@ mod tests { } /// A tag test is a round trip, so the shadow must return exactly the bits - /// it was given — including any the real layout would not use. The cache + /// it was given, except the reserved ones the manual reads as zero. The cache /// model that preceded this one decoded TagLo into ptag/state/pidx fields /// and re-encoded on the way out, which silently dropped the low seven /// bits and every state code it did not recognise. A walking-1s test then /// failed on its very first bit. #[test] - fn tags_round_trip_every_bit() { + fn tags_round_trip_every_defined_bit() { let c = cache(); for bit in 0..64 { // 32 and 63 are MRU control/state, not tag storage. if bit == 32 || bit == 63 { continue; } let v = 1u64 << bit; c.cache_op(C_IST | CACH_SD, 0, v); + let kept = ::SECONDARY_TAG_MASK & v; assert_eq!( c.cache_op(C_ILT | CACH_SD, 0, 0), - v, + kept, "bit {bit} did not survive a store/load tag round trip" ); } } - /// Index_Store_Data / Index_Load_Data likewise. + /// Index_Store_Data / Index_Load_Data likewise, in the secondary and the + /// primary caches: a PROM's data walk round-trips. #[test] fn secondary_data_round_trips() { let c = cache(); c.cache_op(C_R10K_ISD | CACH_SD, 0x40, 0xdeadbeef); assert_eq!(c.cache_op(C_R10K_ILD | CACH_SD, 0x40, 0), 0xdeadbeef); + + c.cache_op(C_R10K_ISD | CACH_PI, 4, 0x1234); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PI, 4, 0), 0x1234); + c.cache_op(C_R10K_ISD | CACH_PD, 8, 0x5678); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_PD, 8, 0), 0x5678); } /// The MRU bit is **shared between the ways of a set**: written through @@ -861,23 +1791,39 @@ mod tests { assert_eq!(c.cache_op_ecc(), 0x3fe, "way 1's check bits"); } + /// The NEC layout: ten check bits, and a secondary store writes a + /// quadword whose other doubleword is zero and shares the check bits. + #[test] + fn nec_layout_stores_a_quadword_with_ten_check_bits() { + let c = cache(); + c.set_cache_ecc(0); + c.cache_op(C_R10K_ISD | CACH_SD, 8, 0xdead_beef); + c.set_cache_ecc(0xffff_fc01); + c.cache_op(C_R10K_ISD | CACH_SD, 0, 0x1111_2222_3333_4444); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_SD, 0, 0), 0x1111_2222_3333_4444); + assert_eq!(c.cache_op_ecc(), 0x001, "ten check bits"); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_SD, 8, 0), 0, "zero partner"); + assert_eq!(c.cache_op_ecc(), 0x001, "shared check bits"); + } + /// An untouched set reports no MRU, so an ordinary tag read is not /// contaminated by it. #[test] fn an_untouched_set_has_no_mru_bit() { let c = cache(); - c.cache_op(C_IST | CACH_SD, 0, 0xdead_beef); - assert_eq!(c.cache_op(C_ILT | CACH_SD, 0, 0), 0xdead_beef); + // Only defined tag bits (TagLo bits 9, 12 and 13 read as zero). + c.cache_op(C_IST | CACH_SD, 0, 0xdead_8cef); + assert_eq!(c.cache_op(C_ILT | CACH_SD, 0, 0), 0xdead_8cef); } /// Separate lines must not alias onto one another. #[test] fn distinct_lines_hold_distinct_tags() { let c = cache(); - c.cache_op(C_IST | CACH_SD, 0, 0x1111_1111); - c.cache_op(C_IST | CACH_SD, 128, 0x2222_2222); - assert_eq!(c.cache_op(C_ILT | CACH_SD, 0, 0), 0x1111_1111); - assert_eq!(c.cache_op(C_ILT | CACH_SD, 128, 0), 0x2222_2222); + c.cache_op(C_IST | CACH_SD, 0, 0x1111_0111); + c.cache_op(C_IST | CACH_SD, 128, 0x2222_0022); + assert_eq!(c.cache_op(C_ILT | CACH_SD, 0, 0), 0x1111_0111); + assert_eq!(c.cache_op(C_ILT | CACH_SD, 128, 0), 0x2222_0022); } /// Bit 0 of a CACHE index selects the **way**, and the two ways of a set @@ -905,14 +1851,14 @@ mod tests { for set in 0..4u64 { for way in 0..2u64 { let va = set * 128 + way; - c.cache_op(C_IST | CACH_SD, va, 0x1000 + set * 16 + way); + c.cache_op(C_IST | CACH_SD, va, 0x4000 + set * 16 + way); } } for set in 0..4u64 { for way in 0..2u64 { let va = set * 128 + way; assert_eq!(c.cache_op(C_ILT | CACH_SD, va, 0), - 0x1000 + set * 16 + way, + 0x4000 + set * 16 + way, "set {set} way {way} aliased"); } } @@ -931,6 +1877,25 @@ mod tests { assert_eq!(c.read::<4>(0, 0x2000).data, 0x1234_5678); } + /// On the R12000 and later Index Writeback Invalidate (D) zeroes the + /// line's primary data tag, which Index Load Tag then reads back; the + /// R10000 keeps the tag. + #[test] + fn index_writeback_invalidate_d_clears_the_tag_from_the_r12000_on() { + fn run>>() -> u64 { + let mem: Arc = Arc::new(Memory::new(1)); + let c = C::from(mem); + let tag = 0x0000_0001_2345_6780u64; + c.cache_op(crate::cpu::mips_cache_v2::C_IST | CACH_PD, 0x40, tag); + assert_eq!(c.cache_op(crate::cpu::mips_cache_v2::C_ILT | CACH_PD, 0x40, 0), tag); + c.cache_op(crate::cpu::mips_cache_v2::C_IWBINV | CACH_PD, 0x40, 0); + c.cache_op(crate::cpu::mips_cache_v2::C_ILT | CACH_PD, 0x40, 0) + } + assert_eq!(run::(), 0); + assert_eq!(run::(), 0); + assert_eq!(run::(), 0x0000_0001_2345_6780); + } + /// Geometry is what CP0 Config is built from, so it must be the real /// part's even though the model holds nothing. #[test] @@ -960,6 +1925,21 @@ mod tests { assert_eq!(::NAME, "shadow", "existing IP28 snapshots"); } + /// The R10000 returns zero for the reserved bits of a secondary tag and + /// keeps MRU, tag, state, index and ECC (R10000 UM p. 270); the R4400 + /// stores the tag as written. + #[test] + fn the_r10000_masks_secondary_tags() { + fn stored>>(tag: u64) -> u64 { + let c = C::from(Arc::new(Memory::new(1)) as Arc); + c.cache_op(C_IST | CACH_SD, 0, tag); + c.cache_op(C_ILT | CACH_SD, 0, 0) + } + assert_eq!(stored::(u64::MAX), 0x8000_000f_ffff_cdff); + assert_eq!(stored::(u64::MAX), 0x8000_000f_ffff_cdff); + assert_eq!(stored::(0x3000) & 0x3000, 0x3000, "R4400 keeps TagLo bits 12 and 13"); + } + /// The O2 R10000 differs from the IP28's only in its secondary line size. #[test] fn the_o2_r10000_is_the_ip28_part_with_shorter_secondary_lines() { @@ -972,6 +1952,24 @@ mod tests { (::PRID, ::FIR, ::MIPS4, ::TLB_ENTRIES, ::VA_BITS, ::R10K_CACHE_OPS)); } + /// Each R10000-family part reports its own identity and geometry and shares + /// the family's address widths and cache operations; the R5000 does not. + #[test] + fn the_r10000_family_parts_report_their_own_identity() { + fn id() -> (u32, usize, usize, &'static str, u32, u32, bool) { + (C::PRID, C::L2_SIZE, C::L2_LINE, C::NAME, C::VA_BITS, C::PA_BITS, C::R10K_CACHE_OPS) + } + assert_eq!(id::(), (0x925, 1 << 20, 128, "shadow", 44, 40, true)); + assert_eq!(id::(), (0xe35, 2 << 20, 128, "R12000", 44, 40, true)); + assert_eq!(id::(), (0xe35, 2 << 20, 64, "R12000", 44, 40, true)); + assert_eq!(id::(), (0xf42, 4 << 20, 128, "R16000", 44, 40, true)); + assert_eq!(id::().6, false); + // The 700 MHz part is the same type with a PRId set at build time, as + // the Octane's R14000 is the R12000 type. + assert_eq!(R16000_MHZ700_PRID, 0xf41); + assert_eq!(R16000_MHZ700_PRID >> 4 & 0xf, 4, "major nibble is printed major 2 plus 2"); + } + /// The tag store hook sees primary data tag stores and nothing else. #[test] fn the_tag_store_hook_sees_only_primary_data_tag_stores() { @@ -1015,6 +2013,1014 @@ mod tests { check::("R5000", &[CACH_PI, CACH_PD]); } + // ---- Index-operation staging --------------------------------------- + + use std::sync::atomic::{AtomicU32, Ordering}; + use std::sync::Mutex; + use crate::cpu::mips_cache_v2::{C_HINV, C_HWBINV, C_IWBINV}; + + /// Memory that reads 0xee everywhere, counts reads and records writes, so + /// a staged value can never be mistaken for it and a bus access shows. + #[derive(Default)] + struct Bus { + reads: AtomicU32, + writes: Mutex>, + } + + impl BusDevice for Bus { + fn read8(&self, _: u64) -> crate::traits::BusRead8 { + self.reads.fetch_add(1, Ordering::Relaxed); + crate::traits::BusRead8::ok(0xee) + } + fn read16(&self, _: u64) -> crate::traits::BusRead16 { + self.reads.fetch_add(1, Ordering::Relaxed); + crate::traits::BusRead16::ok(0xeeee) + } + fn read32(&self, _: u64) -> crate::traits::BusRead32 { + self.reads.fetch_add(1, Ordering::Relaxed); + crate::traits::BusRead32::ok(0xeeee_eeee) + } + fn read64(&self, _: u64) -> BusRead64 { + self.reads.fetch_add(1, Ordering::Relaxed); + BusRead64::ok(0xeeee_eeee_eeee_eeee) + } + fn write8(&self, a: u64, v: u8) -> u32 { self.writes.lock().unwrap().push((a, v as u64)); BUS_OK } + fn write16(&self, a: u64, v: u16) -> u32 { self.writes.lock().unwrap().push((a, v as u64)); BUS_OK } + fn write32(&self, a: u64, v: u32) -> u32 { self.writes.lock().unwrap().push((a, v as u64)); BUS_OK } + fn write64(&self, a: u64, v: u64) -> u32 { self.writes.lock().unwrap().push((a, v)); BUS_OK } + } + + /// An R10000 secondary cache (1 MiB, 128-byte lines, 4096 sets) over `Bus`. + fn staged_cache() -> (R10000ShadowCache, Arc) { + let bus = Arc::new(Bus::default()); + (R10000ShadowCache::from(bus.clone() as Arc), bus) + } + + /// A line in set 0x20 (PA[18:7]), tag PA[35:18] = 4. + const PA: u64 = 0x0010_1000; + const MEM: u64 = 0xeeee_eeee_eeee_eeee; + + /// The operand VA of an index operation on `pa`'s set, `way` in bit 0. + fn operand(pa: u64, way: u64) -> u64 { + 0xffff_ffff_8000_0000 | (pa & 0x1fff_fff8) | way + } + + /// Index Store Tag (S): the line of `pa`, in `way`, with `state`. + fn stage_tag(c: &R10000ShadowCache, pa: u64, way: u64, state: u64) { + c.cache_op(C_IST | CACH_SD, operand(pa & !127, way), l2_tag_of(pa) | state << L2_STATE_SHIFT); + } + + /// Index Store Data (S): the doubleword of `pa` becomes TagHi:TagLo = `d`. + fn stage_data(c: &R10000ShadowCache, pa: u64, way: u64, d: u64) { + c.cache_op(C_R10K_ISD | CACH_SD, operand(pa & !7, way), d); + } + + fn load(c: &R10000ShadowCache, pa: u64) -> u64 { + let r = c.read::(operand(pa, 0), pa); + assert!(r.is_ok()); + r.data + } + + /// Encoding: the 5-bit op field is `op[4:2]` over `cache[1:0]`, and + /// the secondary operations are the ones the manual lists. ISUM pp.70-72. + #[test] + fn the_secondary_index_and_hit_operations_have_the_listed_encodings() { + assert_eq!(C_IWBINV | CACH_SD, 0x03); + assert_eq!(C_ILT | CACH_SD, 0x07); + assert_eq!(C_IST | CACH_SD, 0x0b); + assert_eq!(C_HINV | CACH_SD, 0x13); + assert_eq!(C_HWBINV | CACH_SD, 0x17); + assert_eq!(C_R10K_ILD | CACH_SD, 0x1b); + assert_eq!(C_R10K_ISD | CACH_SD, 0x1f); + // Primary data and instruction, for the neighbours a decoder mixes up. + assert_eq!((C_IST | CACH_PD, C_R10K_ISD | CACH_PD), (0x09, 0x1d)); + assert_eq!((C_IST | CACH_PI, C_R10K_ISD | CACH_PI), (0x08, 0x1c)); + assert_eq!(cache_op_name(0x0b), "Index_Store_Tag(SD)"); + } + + /// The whole thing through the CACHE instruction (major opcode 0x2f, op in + /// rt, base register and offset) and the interpreter's cached load: TagLo + /// and TagHi are what the operations store, and `lw` hits the line. + #[test] + fn cache_instructions_stage_a_line_a_cached_load_hits() { + use crate::cpu::mips_exec::{MipsCpuConfig, MipsExecutor, EXEC_COMPLETE}; + use crate::cpu::mips_isa::{OP_CACHE, OP_LW}; + use crate::cpu::mips_tlb::PassthroughTlb; + let bus = Arc::new(Bus::default()); + let mut exec = MipsExecutor::::new( + bus.clone() as Arc, PassthroughTlb::default(), + &MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false)))); + #[cfg(feature = "jitv2")] + { exec.jitv2_dispatch_enabled = false; } + exec.core.cp0_config = (exec.core.cp0_config & !7) | 3; // KSEG0: cacheable + let i = |op: u32, rs: u32, rt: u32, imm: u32| (op << 26) | (rs << 21) | (rt << 16) | imm; + exec.core.gpr[4] = operand(PA, 0); + let tag = l2_tag_of(PA) | 2 << L2_STATE_SHIFT; + exec.core.cp0_taglo = tag as u32 as u64; + exec.core.cp0_taghi = (tag >> 32) as u32; + assert_eq!(exec.exec(i(OP_CACHE, 4, 0x0b, 0)), EXEC_COMPLETE); + exec.core.cp0_taglo = 0x1111_2222u64; // word at the lower address (inferred) + exec.core.cp0_taghi = 0x3333_4444; + assert_eq!(exec.exec(i(OP_CACHE, 4, 0x1f, 0)), EXEC_COMPLETE); + assert_eq!(exec.exec(i(OP_LW, 4, 5, 0)), EXEC_COMPLETE); + assert_eq!(exec.core.gpr[5], 0x1111_2222); + assert_eq!(exec.exec(i(OP_LW, 4, 6, 4)), EXEC_COMPLETE); + assert_eq!(exec.core.gpr[6], 0x3333_4444); + assert_eq!(bus.reads.load(Ordering::Relaxed), 0, "no system-bus read"); + } + + /// Way select (ISUM pp.71-72): address bit 0 picks the way for every + /// index operation, and the two ways of a set hold different lines. + #[test] + fn the_way_comes_from_address_bit_0() { + let (c, bus) = staged_cache(); + let other = PA + (1 << 19); // same set, tag PA[35:18] differs + stage_tag(&c, PA, 0, 2); + stage_data(&c, PA, 0, 0xaaaa_aaaa_0000_0001); + stage_tag(&c, other, 1, 2); + stage_data(&c, other, 1, 0xbbbb_bbbb_0000_0002); + assert_eq!(load::<8>(&c, PA).rotate_left(32), 0xaaaa_aaaa_0000_0001); + assert_eq!(load::<8>(&c, other).rotate_left(32), 0xbbbb_bbbb_0000_0002); + assert_eq!(bus.reads.load(Ordering::Relaxed), 0); + // The tags are read back from the way that was addressed. + let tag = |way| c.cache_op(C_ILT | CACH_SD, operand(PA, way), 0); + assert_eq!(tag(0), l2_tag_of(PA) | 2 << L2_STATE_SHIFT); + assert_eq!(tag(1), l2_tag_of(other) | 2 << L2_STATE_SHIFT); + // Ending way 1 leaves way 0. + stage_tag(&c, other, 1, 0); + assert_eq!(load::<8>(&c, other), MEM); + assert_ne!(load::<8>(&c, PA), MEM); + } + + /// Tag state (UM p.84): Shared, CleanExclusive and DirtyExclusive are + /// valid; Invalid misses. + #[test] + fn every_state_but_invalid_is_valid() { + for (state, hits) in [(0, false), (1, true), (2, true), (3, true)] { + let (c, bus) = staged_cache(); + stage_tag(&c, PA, 0, state); + stage_data(&c, PA, 0, 0x1234_5678_9abc_def0); + let got = load::<8>(&c, PA); + assert_eq!(got != MEM, hits, "state {state}"); + assert_eq!(bus.reads.load(Ordering::Relaxed), if hits { 0 } else { 1 }, "state {state}"); + } + } + + /// The hit after Index Store Tag + Index Store Data: the staged data, at + /// every access width, with no bus read; the other doubleword of the + /// quadword reads zero (UM p.91, ISUM p.30), not memory. + #[test] + fn a_staged_line_hits_with_its_data_and_no_bus_read() { + let (c, bus) = staged_cache(); + stage_tag(&c, PA, 0, 2); + // The word at the lower address is TagLo: the inferred order, below. + stage_data(&c, PA + 8, 0, 0x0102_0304_a1a2_a3a4); + let m = 0xa1a2_a3a4_0102_0304u64; // as `ld` at PA+8 sees it + assert_eq!(load::<8>(&c, PA + 8), m); + assert_eq!(load::<4>(&c, PA + 8), m >> 32); + assert_eq!(load::<4>(&c, PA + 12), m & 0xffff_ffff); + assert_eq!(load::<2>(&c, PA + 10), (m >> 32) & 0xffff); + assert_eq!(load::<1>(&c, PA + 11), (m >> 32) & 0xff); + assert_eq!(load::<1>(&c, PA + 15), m & 0xff); + assert_eq!(load::<8>(&c, PA), 0, "the quadword's other doubleword is zero"); + assert_eq!(bus.reads.load(Ordering::Relaxed), 0); + // Two Index Store Data to one quadword leave only the second. + stage_data(&c, PA, 0, 0x5555_5555_6666_6666); + assert_eq!(load::<8>(&c, PA + 8), 0); + assert_ne!(load::<8>(&c, PA), 0); + } + + /// A miss: the tag does not match, the state is invalid, or the access + /// is in another set. The access reads the bus and sees memory. + #[test] + fn a_tag_that_does_not_match_misses() { + let (c, bus) = staged_cache(); + stage_tag(&c, PA, 0, 2); + stage_data(&c, PA, 0, 0x1234_5678_9abc_def0); + for bit in 14..32 { + // TagLo[31:14] is PA[35:18]: a flipped bit is another address. + let pa = PA ^ (1 << (bit + 4)); + let before = bus.reads.load(Ordering::Relaxed); + assert_eq!(load::<8>(&c, pa), MEM, "PA bit {}", bit + 4); + assert_eq!(bus.reads.load(Ordering::Relaxed), before + 1); + } + // TagHi[3:0] is PA[39:36]. + let before = bus.reads.load(Ordering::Relaxed); + assert_eq!(load::<8>(&c, PA | 1 << 36), MEM); + assert_eq!(bus.reads.load(Ordering::Relaxed), before + 1); + // Another set, same tag. + assert_eq!(load::<8>(&c, PA + 128), MEM); + // The line itself still hits. + assert_ne!(load::<8>(&c, PA), MEM); + } + + /// **Inferred.** Staging TagHi = W1 and TagLo = W0 into one doubleword: + /// a word load at its lower address returns W0 (TagLo lands at the lower + /// address), the higher address W1, and a doubleword load has W0 on top. + /// The manual-literal order would return W1 first. Index Load Data + /// returns the registers as stored under either order. + #[test] + fn the_inferred_refill_order_puts_taglo_at_the_lower_address() { + let (c, _) = staged_cache(); + let (w0, w1) = (0x1111_1111u64, 0x2222_2222u64); + stage_tag(&c, PA, 0, 2); + stage_data(&c, PA, 0, w1 << 32 | w0); + assert!(STAGED_TAGLO_AT_LOWER_ADDRESS, "the inferred order is the one modelled"); + assert_eq!(load::<4>(&c, PA), w0, "lower address: TagLo"); + assert_eq!(load::<4>(&c, PA + 4), w1, "higher address: TagHi"); + assert_eq!(load::<8>(&c, PA), w0 << 32 | w1); + assert_eq!(c.cache_op(C_R10K_ILD | CACH_SD, operand(PA, 0), 0), w1 << 32 | w0, + "Index Load Data is independent of the order"); + // The conversion itself, for either order. + assert_eq!(staged_to_memory(w1 << 32 | w0), w0 << 32 | w1); + assert_eq!(staged_to_memory(staged_to_memory(7)), 7); + } + + /// A fetch hits too, and sees the line end when it is invalidated: the + /// decode cache does not keep serving the staged word. + #[test] + fn a_fetch_runs_the_staged_word_until_the_line_is_invalidated() { + let (c, bus) = staged_cache(); + let word = |c: &R10000ShadowCache| unsafe { + (*c.fetch(operand(PA, 0), PA).instr).raw + }; + assert_eq!(word(&c), 0xeeee_eeee, "memory before staging"); + let reads = bus.reads.load(Ordering::Relaxed); + stage_tag(&c, PA, 0, 2); + stage_data(&c, PA, 0, 0x2402_0022_0000_0000 | 0x0003_e008); // TagHi = addiu, TagLo = jr ra + assert_eq!(word(&c), 0x0003_e008, "TagLo is the word at the lower address"); + assert_eq!(word(&c), 0x0003_e008); + let f = c.fetch(operand(PA, 0) + 4, PA + 4); + assert_eq!(unsafe { (*f.instr).raw }, 0x2402_0022); + let f = c.fetch(operand(PA, 0) + 8, PA + 8); + assert_eq!(unsafe { (*f.instr).raw }, 0, "the quadword's pad is a nop"); + assert_eq!(bus.reads.load(Ordering::Relaxed), reads); + c.cache_op(C_IWBINV | CACH_SD, operand(PA, 0), 0); + assert_eq!(word(&c), 0xeeee_eeee, "back to memory"); + } + + /// Eviction and invalidation end the hit: Index Writeback Invalidate by + /// index, Hit Invalidate and Hit Writeback Invalidate by address, an + /// Index Store Tag that writes another tag or an invalid state, and + /// power-on. An operation on another line does not. + #[test] + fn invalidation_and_eviction_end_the_hit() { + let ends: [(&str, Box); 6] = [ + ("Index Writeback Invalidate", Box::new(|c| { c.cache_op(C_IWBINV | CACH_SD, operand(PA, 0), 0); })), + ("Hit Invalidate", Box::new(|c| { c.cache_op(C_HINV | CACH_SD, operand(PA, 0), PA); })), + ("Hit Writeback Invalidate", Box::new(|c| { c.cache_op(C_HWBINV | CACH_SD, operand(PA, 0), PA); })), + ("Index Store Tag, invalid", Box::new(|c| stage_tag(c, PA, 0, 0))), + ("Index Store Tag, another line", Box::new(|c| stage_tag(c, PA + (1 << 19), 0, 2))), + ("power-on", Box::new(|c| c.power_on())), + ]; + for (name, end) in ends { + let (c, bus) = staged_cache(); + stage_tag(&c, PA, 0, 2); + stage_data(&c, PA, 0, 0x1234_5678_9abc_def0); + assert_ne!(load::<8>(&c, PA), MEM, "{name}: hits before"); + end(&c); + assert_eq!(load::<8>(&c, PA), MEM, "{name}: misses after"); + assert_eq!(bus.reads.load(Ordering::Relaxed), 1, "{name}"); + assert!(c.take_staged_changed(), "{name}"); + assert!(!c.page_staged(PA) || name == "Index Store Tag, another line"); + } + // Another line's invalidation, and a hit operation that misses, leave it. + let (c, _) = staged_cache(); + stage_tag(&c, PA, 0, 2); + stage_data(&c, PA, 0, 0x1234_5678_9abc_def0); + c.cache_op(C_IWBINV | CACH_SD, operand(PA + 128, 0), 0); + c.cache_op(C_HINV | CACH_SD, operand(PA, 0), PA + (1 << 19)); + c.cache_op(C_IWBINV | CACH_SD, operand(PA, 1), 0); // the other way + assert_ne!(load::<8>(&c, PA), MEM); + } + + /// Writeback (ISUM pp.18-19, 27): invalidating a DirtyExclusive line + /// sends its block to the system interface at the tag's address; other + /// states only notify. Staged data and memory order agree. + #[test] + fn a_dirty_exclusive_line_is_written_back_when_it_goes() { + for (state, written) in [(3, true), (2, false), (1, false)] { + let (c, bus) = staged_cache(); + stage_tag(&c, PA, 0, state); + stage_data(&c, PA + 16, 0, 0x1111_1111_2222_2222); + c.cache_op(C_IWBINV | CACH_SD, operand(PA, 0), 0); + let w = bus.writes.lock().unwrap().clone(); + assert_eq!(!w.is_empty(), written, "state {state}"); + if written { + assert_eq!(w.len(), 16, "a 128-byte block, doubleword by doubleword"); + assert_eq!(w[0], (PA & !127, 0), "the line at the tag's address"); + assert_eq!(w[2], (PA + 16, 0x2222_2222_1111_1111), "in memory order"); + } + } + } + + /// A cached store to a staged line stays in the cache: memory is not + /// written, and a later load sees it. A store elsewhere is memory's. + #[test] + fn a_store_to_a_staged_line_stays_in_the_cache() { + let (c, bus) = staged_cache(); + stage_tag(&c, PA, 0, 2); + stage_data(&c, PA, 0, 0); + assert_eq!(c.write::<4>(operand(PA, 0) + 4, PA + 4, 0xdead_beef), BUS_OK); + assert_eq!(c.write::<1>(operand(PA, 0) + 1, PA + 1, 0x7f), BUS_OK); + assert_eq!(load::<4>(&c, PA + 4), 0xdead_beef); + assert_eq!(load::<8>(&c, PA), 0x007f_0000_dead_beef); + assert!(bus.writes.lock().unwrap().is_empty()); + c.write::<4>(operand(PA, 0) + 0x1000, PA + 0x1000, 5); + assert_eq!(bus.writes.lock().unwrap().len(), 1); + } + + /// **Inferred** (NEC U10278EJ4V0UM p.246: a store to a CleanExclusive or + /// Shared block requests DirtyExclusive). A store moves the line to + /// DirtyExclusive, so invalidating with writeback sends the stored bytes + /// to memory instead of dropping them. Hit Invalidate still drops them. + #[test] + fn a_store_makes_the_line_dirty_and_the_writeback_carries_it() { + for state in [1, 2, 3] { + let (c, bus) = staged_cache(); + stage_tag(&c, PA, 0, state); + stage_data(&c, PA, 0, 0); + c.write::<4>(operand(PA, 0), PA, 0x0123_4567); + assert_eq!((c.cache_op(C_ILT | CACH_SD, operand(PA, 0), 0) >> L2_STATE_SHIFT) & 3, 3, "state {state}"); + c.cache_op(C_IWBINV | CACH_SD, operand(PA, 0), 0); + let w = bus.writes.lock().unwrap().clone(); + assert_eq!(w.len(), 16, "state {state}"); + assert_eq!(w[0], (PA & !127, 0x0123_4567_0000_0000), "state {state}: the stored word, at the lower address"); + } + let (c, bus) = staged_cache(); + stage_tag(&c, PA, 0, 2); + stage_data(&c, PA, 0, 0); + c.write::<4>(operand(PA, 0), PA, 1); + c.cache_op(C_HINV | CACH_SD, operand(PA, 0), PA); + assert!(bus.writes.lock().unwrap().is_empty(), "Hit Invalidate does not write back"); + // A load does not dirty a line. + let (c, bus) = staged_cache(); + stage_tag(&c, PA, 0, 2); + stage_data(&c, PA, 0, 5); + load::<4>(&c, PA); + c.cache_op(C_IWBINV | CACH_SD, operand(PA, 0), 0); + assert!(bus.writes.lock().unwrap().is_empty()); + } + + /// op[1:0] = 10 is undefined on the R10000: it neither stages nor ends a line. + #[test] + fn the_undefined_secondary_instruction_encoding_does_nothing() { + let (c, bus) = staged_cache(); + c.cache_op(C_IST | CACH_SI, operand(PA & !127, 0), l2_tag_of(PA) | 2 << L2_STATE_SHIFT); + assert!(!c.page_staged(PA)); + assert_eq!(load::<8>(&c, PA), MEM); + stage_tag(&c, PA, 0, 2); + stage_data(&c, PA, 0, 7); + c.cache_op(C_IWBINV | CACH_SI, operand(PA, 0), 0); + c.cache_op(C_HINV | CACH_SI, operand(PA, 0), PA); + c.cache_op(C_HWBINV | CACH_SI, operand(PA, 0), PA); + assert_ne!(load::<8>(&c, PA), MEM, "still staged"); + assert_eq!(bus.reads.load(Ordering::Relaxed), 1); + } + + /// The load's VA[13:12] may differ from TagLo[8:7]. + /// The lookup is physical, so the line hits whatever virtual address is used. + #[test] + fn a_different_virtual_index_still_hits() { + let (c, bus) = staged_cache(); + c.cache_op(C_IST | CACH_SD, operand(PA & !127, 0), l2_tag_of(PA) | 2 << L2_STATE_SHIFT | 3 << 7); + stage_data(&c, PA, 0, 0x1111_2222_3333_4444); + for va_bits in [0u64, 0x1000, 0x2000, 0x3000] { + let r = c.read::<8>(0xffff_ffff_8000_0000 | (PA & 0xfff) | va_bits, PA); + assert_eq!(r.data, 0x3333_4444_1111_2222, "VA[13:12] = {}", va_bits >> 12); + } + assert_eq!(bus.reads.load(Ordering::Relaxed), 0); + } + + /// The same rules at every geometry the family ships: 2 MiB with 128-byte + /// lines (Octane, Fuel), 1 MiB with 64-byte lines (O2), 4 MiB (R16000). + /// Two lines of one set differ by a way's worth of address. + #[test] + fn staging_works_at_every_cache_geometry() { + use crate::cpu::mips_cache_v2::MipsCache as _; + fn check>>(name: &str) { + let bus = Arc::new(Bus::default()); + let c = C::from(bus.clone() as Arc); + let line = C::L2_LINE as u64; + let way_bytes = (C::L2_SIZE / 2) as u64; + let (a, b) = (0x0020_3000u64 + line, 0x0020_3000u64 + line + way_bytes); + let op = |pa: u64, way: u64| 0xffff_ffff_8000_0000 | (pa & 0x1fff_fff8) | way; + let tag = |pa: u64, way: u64, state: u64| { + c.cache_op(C_IST | CACH_SD, op(pa & !(line - 1), way), l2_tag_of(pa) | state << L2_STATE_SHIFT); + }; + let data = |pa: u64, way: u64, d: u64| { c.cache_op(C_R10K_ISD | CACH_SD, op(pa & !7, way), d); }; + let load = |pa: u64| c.read::<8>(op(pa, 0), pa).data; + tag(a, 0, 2); data(a, 0, 0x0a0a_0a0a_0b0b_0b0b); + tag(b, 1, 2); data(b, 1, 0x0c0c_0c0c_0d0d_0d0d); + assert_eq!(load(a), 0x0b0b_0b0b_0a0a_0a0a, "{name}: way 0"); + assert_eq!(load(b), 0x0d0d_0d0d_0c0c_0c0c, "{name}: way 1"); + assert_eq!(load(a + 8), 0, "{name}: the line's unstaged doubleword"); + assert_eq!(bus.reads.load(Ordering::Relaxed), 0, "{name}"); + // Flip each tag address bit above the index. The first one is + // the other line's, which is staged in way 1. + assert_eq!(load(b), 0x0d0d_0d0d_0c0c_0c0c, "{name}: the other way's line"); + let index_top = (C::L2_SIZE / 2).trailing_zeros(); + for bit in (index_top + 1)..36 { + assert_eq!(load(a ^ (1 << bit)), MEM, "{name}: PA bit {bit}"); + } + // Dirty write back covers the whole line, at this cache's line size. + tag(a, 0, 3); + c.cache_op(C_IWBINV | CACH_SD, op(a, 0), 0); + let w = bus.writes.lock().unwrap().clone(); + assert_eq!(w.len() as u64, line / 8, "{name}"); + assert_eq!(w[0].0, a & !(line - 1), "{name}"); + assert_eq!(load(a), MEM, "{name}: ended"); + assert_ne!(load(b), MEM, "{name}: the other way stays"); + } + check::("R10000 1 MiB/128"); + check::("R10000 O2 1 MiB/64"); + check::("R12000 2 MiB/128"); + check::("R14000 2 MiB/128"); + check::("R16000 4 MiB/128"); + } + + /// A snapshot carries the staged lines, and a restore ends any that were + /// staged before it. Dirty lines also reach memory on `writeback_all`. + #[test] + fn staged_lines_survive_a_snapshot_round_trip() { + let (c, _) = staged_cache(); + assert!(c.save_cache_state().as_table().unwrap().is_empty(), "nothing staged, nothing saved"); + stage_tag(&c, PA, 1, 3); + stage_data(&c, PA + 8, 1, 0x8000_0000_0000_0001); + c.cache_op(C_R10K_ISD | CACH_SD, operand(PA + 8, 1), 0x8000_0000_0000_0001); + let saved = c.save_cache_state(); + + let (d, bus) = staged_cache(); + stage_tag(&d, PA + 0x4000, 0, 2); // staged before the restore: must go + d.load_cache_state(&saved).unwrap(); + assert_eq!(load::<8>(&d, PA + 8), 0x0000_0001_8000_0000); + assert_eq!(load::<8>(&d, PA + 0x4000), MEM); + assert!(d.page_staged(PA) && !d.page_staged(PA + 0x4000)); + assert_eq!(bus.reads.load(Ordering::Relaxed), 1); + assert_eq!(d.cache_op(C_ILT | CACH_SD, operand(PA, 1), 0), l2_tag_of(PA) | 3 << L2_STATE_SHIFT); + + d.writeback_all(); + let w = bus.writes.lock().unwrap().clone(); + assert_eq!(w.len(), 16); + assert_eq!(w[1], (PA & !127 | 8, 0x0000_0001_8000_0000)); + assert_ne!(load::<8>(&d, PA + 8), MEM, "writeback_all keeps the line"); + + let empty = toml::Value::Table(Default::default()); + d.load_cache_state(&empty).unwrap(); + assert_eq!(load::<8>(&d, PA + 8), MEM, "restoring a snapshot with no staged lines ends them"); + } + + /// Index Store Data (S) writes the one + /// quadword PA[6:4] names and leaves + /// the line's other quadwords alone, and a load or fetch finds each + /// quadword by the same bits. A quadword is stored once, the others are + /// rewritten many times, and the first must still read back what was + /// stored: for every quadword of the line, on 64- and 128-byte lines. + #[test] + fn index_store_data_writes_only_the_quadword_the_address_names() { + fn check>>(name: &str) { + let bus = Arc::new(Bus::default()); + let c = C::from(bus.clone() as Arc); + let line = C::L2_LINE as u64; + let pa0 = 0x0020_3000u64 & !(line - 1); + let op = |pa: u64| 0xffff_ffff_8000_0000 | (pa & 0x1fff_fff8); + let quads = line / 16; + for target in 0..quads { + // End the previous round's line (and with it any primary refill). + c.cache_op(C_IWBINV | CACH_SD, op(pa0), 0); + c.cache_op(C_IST | CACH_SD, op(pa0), l2_tag_of(pa0) | 2 << L2_STATE_SHIFT); + // Clear the whole line, then store the target quadword once. + for q in 0..quads { + c.cache_op(C_R10K_ISD | CACH_SD, op(pa0 + 16 * q), 0); + } + // The upper doubleword of the quadword, to catch a mix-up of PA[3]. + let at = pa0 + 16 * target + 8; + let (w0, w1) = (0x1000_0000 + target as u32, 0x2000_0000 + target as u32); + c.cache_op(C_R10K_ISD | CACH_SD, op(at), (w1 as u64) << 32 | w0 as u64); + // Rewrite every other quadword many times, both halves. + for round in 0..40u64 { + for q in (0..quads).filter(|&q| q != target) { + let half = (round + q) & 1; + c.cache_op(C_R10K_ISD | CACH_SD, op(pa0 + 16 * q + 8 * half), 0xdead_0000_0000_0000 | round << 8 | q); + } + } + // Index Load Data (S) of the quadword returns what was stored. The + // check bits on the rewrites are assumed valid: none are modelled. + assert_eq!(c.cache_op(C_R10K_ILD | CACH_SD, op(at), 0), (w1 as u64) << 32 | w0 as u64, + "{name} quadword {target}: Index Load Data"); + let rd = |pa: u64| c.read::<4>(op(pa), pa).data; + assert_eq!(rd(at), w0 as u64, "{name} quadword {target}: word at the lower address"); + assert_eq!(rd(at + 4), w1 as u64, "{name} quadword {target}: word at +4"); + assert_eq!(rd(at - 8), 0, "{name} quadword {target}: its other doubleword"); + assert_eq!(rd(at - 4), 0, "{name} quadword {target}: its other doubleword"); + let f = c.fetch(op(at), at); + assert_eq!(unsafe { (*f.instr).raw } as u64, w0 as u64, "{name} quadword {target}: fetch"); + assert_eq!(bus.reads.load(Ordering::Relaxed), 0, "{name}"); + // And the neighbours still hold what they were last given. + for q in (0..quads).filter(|&q| q != target) { + let half = (39 + q) & 1; + let v = 0xdead_0000_0000_0000u64 | 39 << 8 | q; + assert_eq!(c.read::<8>(op(pa0 + 16 * q + 8 * half), pa0 + 16 * q + 8 * half).data, v.rotate_left(32), + "{name} quadword {q}"); + } + } + } + check::("128-byte lines"); + check::("64-byte lines"); + check::("2 MiB, 128-byte lines"); + } + + // ---- Staged primary instruction-cache lines ------------------------- + + /// Index Store Tag (I) on `pa`'s block at the set `va` indexes, `way`. + fn stage_i_tag(c: &R10000ShadowCache, va: u64, pa: u64, way: u64, valid: bool) { + let tag = l1i_tag_of(pa) | if valid { L1I_VALID } else { 0 }; + c.cache_op(C_IST | CACH_PI, (va & !63) | way, tag); + } + + /// Index Store Data (I): the word at `va`, in `way`, is the 36-bit `p`, + /// with the parity bit that makes it even. + fn stage_p(c: &R10000ShadowCache, va: u64, way: u64, p: u64) { + c.set_cache_ecc(l1i_parity_of(p)); + c.cache_op(C_R10K_ISD | CACH_PI, (va & !3) | way, p); + } + + /// The same for a plain instruction, in the model's raw encoding. + fn stage_i_word(c: &R10000ShadowCache, va: u64, way: u64, w: u32) { + stage_p(c, va, way, L1I_RAW << 32 | w as u64); + } + + fn fetch_raw(c: &R10000ShadowCache, va: u64, pa: u64) -> u32 { + unsafe { (*c.fetch(va, pa).instr).raw } + } + + /// Stage whole primary I blocks on way 1 of several sets (16 words each) + /// and fetch through them: the staged words, with no bus read. + #[test] + fn a_staged_primary_instruction_line_serves_fetches() { + let (c, bus) = staged_cache(); + let base = 0x0030_0000u64; + for set in 0..4u64 { + let (va, pa) = (0xffff_ffff_8000_0000 | (base + set * 64), base + set * 64); + stage_i_tag(&c, va, pa, 1, true); + for w in 0..16u64 { + stage_i_word(&c, va + 4 * w, 1, 0x3c00_0000 | (set << 8) as u32 | w as u32); + } + } + for set in 0..4u64 { + for w in 0..16u64 { + let (va, pa) = (0xffff_ffff_8000_0000 | (base + set * 64 + 4 * w), base + set * 64 + 4 * w); + assert_eq!(fetch_raw(&c, va, pa), 0x3c00_0000 | (set << 8) as u32 | w as u32, "set {set} word {w}"); + } + } + assert_eq!(bus.reads.load(Ordering::Relaxed), 0); + // The index is virtual: the same physical block through a VA with + // other VA[13:6] misses. The tag is physical: another frame misses. + assert_eq!(fetch_raw(&c, 0xffff_ffff_8000_0000 | (base + 0x400), base), 0xeeee_eeee); + assert_eq!(fetch_raw(&c, 0xffff_ffff_8000_0000 | base, base + (1 << 20)), 0xeeee_eeee); + // Loads are not served by an instruction line. + assert_eq!(load::<4>(&c, base), 0xeeee_eeee); + } + + /// A secondary staged line served fetches first; ending it leaves the + /// primary I lines hitting. With both valid the primary wins. + #[test] + fn primary_instruction_lines_outlive_the_secondary_line_and_win_over_it() { + let (c, bus) = staged_cache(); + let (pa, va) = (0x0030_0000u64, 0xffff_ffff_8030_0000u64); + stage_tag(&c, pa, 0, 2); + for q in 0..8 { + stage_data(&c, pa + 16 * q, 0, 0x5555_0000_6666_0000 | q); + } + assert_eq!(fetch_raw(&c, va, pa), 0x6666_0000, "the secondary line"); + // The fetch above refilled way 0; stage over it. + stage_i_tag(&c, va, pa, 0, true); + for w in 0..16 { + stage_i_word(&c, va + 4 * w, 0, 0x7000 + w as u32); + } + assert_eq!(fetch_raw(&c, va, pa), 0x7000, "primary before secondary"); + stage_tag(&c, pa, 0, 0); // the secondary line ends + assert_eq!(fetch_raw(&c, va, pa), 0x7000); + assert_eq!(fetch_raw(&c, va + 8, pa + 8), 0x7002); + assert_eq!(bus.reads.load(Ordering::Relaxed), 0); + } + + /// Ending a primary I line: Index Store Tag with an invalid state, Index + /// Invalidate (the way named), Hit Invalidate (by address), power-on. A + /// fetch served before must not be served again. Another way, another + /// set, and an Index Invalidate of the other way leave the line. + #[test] + fn primary_instruction_lines_end() { + let (pa, va) = (0x0030_0040u64, 0xffff_ffff_8030_0040u64); + let ends: [(&str, Box); 4] = [ + ("Index Store Tag, invalid", Box::new(move |c| stage_i_tag(c, va, pa, 1, false))), + ("Index Invalidate", Box::new(move |c| { c.cache_op(C_IINV | CACH_PI, va | 1, 0); })), + ("Hit Invalidate", Box::new(move |c| { c.cache_op(C_HINV | CACH_PI, va, pa); })), + ("power-on", Box::new(|c| c.power_on())), + ]; + for (name, end) in ends { + let (c, _) = staged_cache(); + stage_i_tag(&c, va, pa, 1, true); + stage_i_word(&c, va, 1, 0x1234); + assert_eq!(fetch_raw(&c, va, pa), 0x1234, "{name}: before"); + assert!(c.page_staged(pa)); + end(&c); + assert_eq!(fetch_raw(&c, va, pa), 0xeeee_eeee, "{name}: after"); + assert!(!c.page_staged(pa), "{name}"); + } + let (c, _) = staged_cache(); + stage_i_tag(&c, va, pa, 1, true); + stage_i_word(&c, va, 1, 0x1234); + c.cache_op(C_IINV | CACH_PI, va, 0); // the other way + c.cache_op(C_IINV | CACH_PI, va + 64, 0); // the next set + c.cache_op(C_HINV | CACH_PI, va, pa + (1 << 20)); // misses + assert_eq!(fetch_raw(&c, va, pa), 0x1234); + } + + /// Decode of a sampled class against a sampled example: + /// `addiu $4,$0,2` (0x24040002) is 0x3_4420_0004. Each field moves where + /// the layout says, so a change to any one fails. + #[test] + fn a_sampled_predecoded_word_decodes_to_its_instruction() { + assert_eq!(l1i_decode(0x3_4420_0004), Ok(0x2404_0002)); + // addiu $17,$9,-1: rs, rt, immediate and shamt/rd bits all move. + let f = 0x2531_ffffu32; // op 001001, rs 9, rt 17, imm 0xffff + let p = (0x1au64 << 29) | ((f as u64 >> 26) & 7) << 26 | (f as u64 & 0x3f) << 20 + | ((f as u64 >> 6) & 0x1f) << 15 | ((f as u64 >> 21) & 0x1f) << 10 + | ((f as u64 >> 11) & 0x1f) << 5 | ((f as u64 >> 16) & 0x1f); + assert_eq!(l1i_decode(p), Ok(f)); + assert_eq!(l1i_decode(0x3_4420_0004 | 0x1f << 5), Ok(0x2404_0002 | 0x1f << 11), "p[9:5] is the rd position for addiu"); + assert_eq!(l1i_decode(0x3_4420_0004 | 0x1f << 10), Ok(0x2404_0002 | 0x1f << 21), "p[14:10] is rs"); + assert_eq!(l1i_decode(0x3_4420_0004 | 0x1f << 15), Ok(0x2404_0002 | 0x1f << 6), "p[19:15] is f[10:6]"); + } + + /// One example per class of the table, built from the instruction by the + /// layout, decoded back: includes dsll/dsll32, jr/j and the nop. + #[test] + fn every_class_in_the_table_decodes() { + fn predecode(class: u64, f: u32, rt_low: bool) -> u64 { + let (rt, rd) = ((f as u64 >> 16) & 0x1f, (f as u64 >> 11) & 0x1f); + let (lo, hi) = if rt_low { (rt, rd) } else { (rd, rt) }; + class << 29 | ((f as u64 >> 26) & 7) << 26 | (f as u64 & 0x3f) << 20 + | ((f as u64 >> 6) & 0x1f) << 15 | ((f as u64 >> 21) & 0x1f) << 10 | hi << 5 | lo + } + let cases: &[(u64, u32, bool, &str)] = &[ + (0x08, 0x0009_20f8, false, "dsll $4,$9,3"), // funct 0x38 + (0x08, 0x0009_20fc, false, "dsll32 $4,$9,3"), // funct 0x3c + (0x18, 0x0109_5025, false, "or $10,$8,$9"), + (0x1a, 0x2084_0010, true, "addi"), + (0x1a, 0x2484_0010, true, "addiu"), + (0x1a, 0x3484_0010, true, "ori"), + (0x1a, 0x3c04_8000, true, "lui"), + (0x1e, 0x6484_0010, true, "daddiu"), + (0x35, 0xbd04_0040, false, "cache"), + (0x36, 0xdd04_0008, true, "ld"), + (0x3f, 0x4084_6000, false, "mtc0"), + (0x40, 0x0c00_1234, false, "jal"), + (0x41, 0x03e0_0008, false, "jr $31"), + (0x41, 0x0800_1234, false, "j"), + (0x71, 0x0481_0004, false, "bgez"), + ]; + for &(class, f, rt_low, name) in cases { + assert_eq!(l1i_decode(predecode(class, f, rt_low)), Ok(f), "{name}"); + } + assert_eq!(l1i_decode(0), Ok(0), "the nop"); + assert_eq!(l1i_decode(0x3_4420_0004), Ok(0x2404_0002)); + // Known class, instruction outside its rows: refused. + assert_eq!(l1i_decode(predecode(0x08, 0x0009_20c0, false)), Err(0x08), "funct 0x00 is not dsll"); + assert_eq!(l1i_decode(predecode(0x1a, 0x2884_0010, true)), Err(0x1a), "op 0x0a is not in 0x1a"); + assert_eq!(l1i_decode(1), Err(0), "class 0 is only the nop"); + // Classes share an opcode top: the class picks f[31:29]. + assert_ne!(l1i_decode(predecode(0x1a, 0x2484_0010, true)), l1i_decode(predecode(0x1e, 0x2484_0010, true))); + } + + /// A class outside the sampled set is refused, not guessed: the fetch is + /// treated as a miss and reads memory, and the refusal is reported. + #[test] + fn an_unsampled_class_is_refused_and_the_fetch_reads_memory() { + assert_eq!(l1i_decode(0x5_0000_0004), Err(0x28)); + assert_eq!(l1i_decode(0x2a << 29), Err(0x2a)); + let (c, bus) = staged_cache(); + let (pa, va) = (0x0030_0000u64, 0xffff_ffff_8030_0000u64); + stage_i_tag(&c, va, pa, 1, true); + stage_p(&c, va, 1, 0x5_0000_0004); + assert_eq!(fetch_raw(&c, va, pa), 0xeeee_eeee); + assert!(bus.reads.load(Ordering::Relaxed) >= 1, "memory was read"); + // The sampled class beside it still executes. + stage_p(&c, va + 4, 1, 0x3_4420_0004); + assert_eq!(fetch_raw(&c, va + 4, pa + 4), 0x2404_0002); + } + + /// A word staged with the wrong parity bit posts a Cache Error on fetch + /// (ED set), unless Status.DE or ERL is set, which executes it. + #[test] + fn bad_parity_posts_a_cache_error_unless_errors_are_not_detected() { + let (c, _) = staged_cache(); + let (pa, va) = (0x0030_0000u64, 0xffff_ffff_8030_0000u64); + stage_i_tag(&c, va, pa, 1, true); + c.set_cache_ecc(l1i_parity_of(0x3_4420_0004) ^ 1); + c.cache_op(C_R10K_ISD | CACH_PI, va | 1, 0x3_4420_0004); + let r = c.fetch(va, pa); + assert_eq!(r.status, EXEC_CACHE_ERR); + assert_ne!(c.take_cache_err() & L1I_CACHEERR_ED, 0); + c.set_error_control(false, false); // Status.DE set + let r = c.fetch(va, pa); + assert_eq!(r.status, crate::cpu::mips_exec::EXEC_COMPLETE); + assert_eq!(unsafe { (*r.instr).raw }, 0x2404_0002); + // Every single-bit flip of the 36 bits is a parity error. + c.set_error_control(false, true); + for bit in 0..36 { + c.set_cache_ecc(l1i_parity_of(0x3_4420_0004)); + c.cache_op(C_R10K_ISD | CACH_PI, va | 1, 0x3_4420_0004 ^ 1 << bit); + assert_eq!(c.fetch(va, pa).status, EXEC_CACHE_ERR, "bit {bit}"); + } + } + + /// A readback stored back verbatim round-trips and executes: for a + /// sampled word and for one the model encodes raw (an unsampled opcode). + #[test] + fn index_load_data_then_store_data_round_trips_and_executes() { + let (c, _) = staged_cache(); + let (pa, va) = (0x0030_0000u64, 0xffff_ffff_8030_0000u64); + stage_i_tag(&c, va, pa, 1, true); + stage_i_tag(&c, va + 0x4000, pa + 0x4000, 0, true); + for (i, (p, want)) in [(0x3_4420_0004u64, 0x2404_0002u32), (L1I_RAW << 32 | 0x7c00_0001, 0x7c00_0001)].into_iter().enumerate() { + let src = va + 4 * i as u64; + stage_p(&c, src, 1, p); + let got = c.cache_op(C_R10K_ILD | CACH_PI, src | 1, 0); + assert_eq!(got, p, "Index Load Data returns the 36 bits"); + let ecc = c.cache_op_ecc(); + assert_eq!(ecc & 1, l1i_parity_of(p)); + // Store the readback into another line, with the readback's parity. + let dst = va + 0x4000 + 4 * i as u64; + c.set_cache_ecc(ecc); + c.cache_op(C_R10K_ISD | CACH_PI, dst, got); + assert_eq!(fetch_raw(&c, dst, pa + 0x4000 + 4 * i as u64), want, "word {i}"); + } + } + + /// A staged word that differs from memory executes the staged word: a + /// stale entry stays until it is invalidated. + #[test] + fn a_staged_word_differing_from_memory_executes() { + let (c, bus) = staged_cache(); + let (pa, va) = (0x0030_0000u64, 0xffff_ffff_8030_0000u64); + assert_eq!(fetch_raw(&c, va, pa), 0xeeee_eeee); + stage_i_tag(&c, va, pa, 1, true); + stage_p(&c, va, 1, 0x3_4420_0004); + // The word after it too: opcode fusion peeks at it. + stage_i_word(&c, va + 4, 1, 0); + let reads = bus.reads.load(Ordering::Relaxed); + for _ in 0..3 { + assert_eq!(fetch_raw(&c, va, pa), 0x2404_0002); + } + assert_eq!(bus.reads.load(Ordering::Relaxed), reads); + } + + /// A fetch that misses the primary I cache and hits a staged secondary + /// line refills a 64-byte primary block into an Invalid way: way 1 here, + /// with way 0 valid under an unrelated tag. Index Load Tag/Data (I) return + /// the refill, and a readback stored verbatim into another line executes. + #[test] + fn a_staged_secondary_hit_refills_the_primary_instruction_cache() { + let (c, _) = staged_cache(); + let (pa, va) = (0x0030_0040u64, 0xffff_ffff_8030_0040u64); + stage_tag(&c, pa & !127, 0, 2); + for q in 0..8 { + stage_data(&c, (pa & !127) + 16 * q, 0, 0x1111_0000_2222_0000 + q); + } + // Way 1 invalid, way 0 valid with an unrelated tag. + stage_i_tag(&c, va, pa, 1, false); + stage_i_tag(&c, va, pa + (1 << 20), 0, true); + // The fetch is served and refills way 1. + assert_eq!(fetch_raw(&c, va, pa), 0x2222_0004, "quadword 4"); + let tag = c.cache_op(C_ILT | CACH_PI, va | 1, 0); + assert_eq!(tag, l1i_tag_of(pa) | L1I_VALID, "way 1 holds the fetch's tag, valid"); + let way0 = c.cache_op(C_ILT | CACH_PI, va, 0); + assert_eq!(way0, l1i_tag_of(pa + (1 << 20)) | L1I_VALID, "way 0 is left alone"); + // The block is the secondary's data: the word at +0 is TagLo of the + // doubleword at +0 (the inferred order), +4 TagHi, and so on. + let dst = va + 0x4000; + stage_i_tag(&c, dst, pa + 0x4000, 0, true); + for i in 0..16u64 { + let got = c.cache_op(C_R10K_ILD | CACH_PI, (va & !63 | 4 * i) | 1, 0); + let ecc = c.cache_op_ecc(); + assert_eq!(got & 0xffff_ffff, fetch_raw(&c, (va & !63) + 4 * i, (pa & !63) + 4 * i) as u64, "word {i}"); + // Verbatim into the other line, then execute from it. + c.set_cache_ecc(ecc); + c.cache_op(C_R10K_ISD | CACH_PI, (dst & !63) + 4 * i, got); + assert_eq!(fetch_raw(&c, (dst & !63) + 4 * i, (pa & !63) + 0x4000 + 4 * i), got as u32, "stored word {i}"); + } + // The line is not refilled again once it is resident, and ending the + // secondary block ends it. + c.cache_op(C_IWBINV | CACH_SD, operand(pa & !127, 0), 0); + assert_eq!(c.cache_op(C_ILT | CACH_PI, va | 1, 0) & L1I_VALID, 0, "purged with its secondary block"); + } + + /// Both ways valid: the model picks way 0. A miss in the secondary cache + /// refills nothing. + #[test] + fn a_refill_needs_a_staged_secondary_hit() { + let (c, _) = staged_cache(); + let (pa, va) = (0x0030_0040u64, 0xffff_ffff_8030_0040u64); + assert_eq!(fetch_raw(&c, va, pa), 0xeeee_eeee); + assert_eq!(c.cache_op(C_ILT | CACH_PI, va, 0) & L1I_VALID, 0); + stage_tag(&c, pa & !127, 0, 2); + stage_i_tag(&c, va, pa + (1 << 20), 0, true); + stage_i_tag(&c, va, pa + (2 << 20), 1, true); + fetch_raw(&c, va, pa); + assert_eq!(c.cache_op(C_ILT | CACH_PI, va, 0), l1i_tag_of(pa) | L1I_VALID); + } + + /// Staged primary I lines survive a snapshot round trip. + #[test] + fn staged_primary_instruction_lines_are_saved() { + let (c, _) = staged_cache(); + let (pa, va) = (0x0030_0000u64, 0xffff_ffff_8030_0000u64); + stage_i_tag(&c, va, pa, 1, true); + for w in 0..16 { + stage_i_word(&c, va + 4 * w, 1, 0x7000 + w as u32); + } + let saved = c.save_cache_state(); + let (d, _) = staged_cache(); + d.load_cache_state(&saved).unwrap(); + assert_eq!(fetch_raw(&d, va + 12, pa + 12), 0x7003); + assert!(d.page_staged(pa)); + d.load_cache_state(&toml::Value::Table(Default::default())).unwrap(); + assert_eq!(fetch_raw(&d, va + 12, pa + 12), 0xeeee_eeee); + } + + // ---- Mapped operands ------------------------------------------------ + + type MappedExec = crate::cpu::mips_exec::MipsExecutor; + + /// An executor, a cached 4 KiB page at 0x200000 mapped to frame `pfn` + /// with cache algorithm `alg` and dirty bit `dirty`, and the bus it sits on. + fn mapped(alg: u64, dirty: u64, pfn: u64) -> (Box, Arc) { + use crate::cpu::mips_exec::{MipsCpuConfig, MipsExecutor, EXEC_COMPLETE}; + let bus = Arc::new(Bus::default()); + let mut exec = Box::new(MipsExecutor::::new( + bus.clone() as Arc, crate::cpu::mips_tlb::MipsTlb::default(), + &MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))))); + #[cfg(feature = "jitv2")] + { exec.jitv2_dispatch_enabled = false; } + exec.core.cp0_config = (exec.core.cp0_config & !7) | 3; // KSEG0 cacheable + exec.core.cp0_status &= !crate::cpu::mips_core::STATUS_ERL; + exec.resync_privilege_state(); + exec.core.cp0_index = 3; + exec.core.cp0_pagemask = 0; + exec.core.cp0_entryhi = 0x100 << 13; + exec.core.cp0_entrylo0 = (pfn << 6) | (alg << 3) | (dirty << 2) | (1 << 1); + exec.core.cp0_entrylo1 = 0; + use crate::cpu::mips_isa::OP_COP0; + assert_eq!(exec.exec((OP_COP0 << 26) | (0x10 << 21) | 0x02), EXEC_COMPLETE); // TLBWI + (exec, bus) + } + + /// CACHE `op` on `va`, with TagLo/TagHi as given; the instruction's status. + fn cache_at(exec: &mut MappedExec, op: u32, va: u64, tag: u64) -> u32 { + exec.core.gpr[4] = va; + exec.core.cp0_taglo = tag & 0xffff_ffff; + exec.core.cp0_taghi = (tag >> 32) as u32; + exec.exec((crate::cpu::mips_isa::OP_CACHE << 26) | (4 << 21) | (op << 16)) + } + + const VA: u64 = 0x20_0000; + + /// The executor is large: run on a big stack, as the other TLB tests do. + fn big_stack(f: impl FnOnce() + Send + 'static) { + std::thread::Builder::new().stack_size(16 * 1024 * 1024).spawn(f).unwrap().join().unwrap(); + } + + /// The secondary set and way come from the physical address of a mapped + /// operand (NEC U12754EJ1V0UMJ1 pp.14-15). Stage through the mapped page, then a cached + /// access to the frame's kseg0 address hits, and the tag is at the frame's + /// set, not the virtual address's. + #[test] + fn a_mapped_index_operation_lands_on_the_physical_set() { + big_stack(|| { + use crate::cpu::mips_exec::EXEC_COMPLETE; + let pfn = 0x3a5; // PA 0x3a5000: set 0x4a0; the VA's set is 0 + let (mut exec, bus) = mapped(3, 1, pfn); + let pa = pfn << 12 | 0x40; + let tag = l2_tag_of(pa) | 2 << L2_STATE_SHIFT; + assert_eq!(cache_at(&mut exec, 0x0b, VA + 0x40, tag), EXEC_COMPLETE); + assert_eq!(cache_at(&mut exec, 0x1f, VA + 0x40, 0x3333_4444_1111_2222), EXEC_COMPLETE); + let kseg0 = 0xffff_ffff_8000_0000 | pa; + assert_eq!(exec.cache.read::<4>(kseg0, pa).data, 0x1111_2222); + assert_eq!(exec.cache.read::<4>(kseg0 + 4, pa + 4).data, 0x3333_4444); + let f = exec.cache.fetch(kseg0, pa); + assert_eq!(unsafe { (*f.instr).raw }, 0x1111_2222, "a cached fetch through the frame"); + assert_eq!(bus.reads.load(Ordering::Relaxed), 0); + // The tag is at the physical set, read back through an unmapped operand. + assert_eq!(cache_at(&mut exec, 0x07, 0xffff_ffff_8000_0000 | (pa & !127), 0), EXEC_COMPLETE); + assert_eq!(exec.core.cp0_taglo | (exec.core.cp0_taghi as u64) << 32, tag); + // And nothing is at the set the virtual address would have chosen. + assert_eq!(cache_at(&mut exec, 0x07, 0xffff_ffff_8000_0000 | VA, 0), EXEC_COMPLETE); + assert_eq!(exec.core.cp0_taglo, 0); + // Index Writeback Invalidate through the mapped operand ends the line. + assert_eq!(cache_at(&mut exec, 0x03, VA + 0x40, 0), EXEC_COMPLETE); + assert_eq!(exec.cache.read::<4>(kseg0, pa).data, 0xeeee_eeee); + }) + + } + + /// Two virtual pages on one frame reach one secondary line; the primary + /// index stays virtual. + #[test] + fn aliases_share_a_secondary_line_and_the_primary_stays_virtual() { + big_stack(|| { + use crate::cpu::mips_exec::EXEC_COMPLETE; + let (mut exec, _) = mapped(3, 1, 0x3a5); + let pa = 0x3a5000u64; + let tag = l2_tag_of(pa) | 2 << L2_STATE_SHIFT; + cache_at(&mut exec, 0x0b, VA, tag); + // Another way to the same line: the unmapped address of the frame. + assert_eq!(cache_at(&mut exec, 0x07, 0xffff_ffff_8000_0000 | pa, 0), EXEC_COMPLETE); + assert_eq!(exec.core.cp0_taglo | (exec.core.cp0_taghi as u64) << 32, tag); + // Primary D: indexed by VA[13:5]. Mapped store, unmapped load at the + // same low 14 bits but another frame. + assert_eq!(cache_at(&mut exec, 0x09, VA + 0x40, 0x0000_0abc), EXEC_COMPLETE); + assert_eq!(cache_at(&mut exec, 0x05, 0xffff_ffff_8000_0000 | 0x40, 0), EXEC_COMPLETE); + assert_eq!(exec.core.cp0_taglo, 0x0abc); + }) + + } + + /// A mapped operand takes TLB Refill when no entry maps it, and the cache + /// does not change. + #[test] + fn an_unmapped_operand_takes_tlb_refill() { + big_stack(|| { + use crate::cpu::mips_exec::{EXEC_IS_EXCEPTION, EXEC_IS_TLB_REFILL}; + let (mut exec, _) = mapped(3, 1, 0x3a5); + let st = cache_at(&mut exec, 0x0b, 0x60_0000, l2_tag_of(0x3a5000) | 2 << L2_STATE_SHIFT); + assert!(st & EXEC_IS_EXCEPTION != 0 && st & EXEC_IS_TLB_REFILL != 0, "status {st:#x}"); + assert!(!exec.cache.page_staged(0x3a5000)); + // The same through an index operation on a data cache. + let st = cache_at(&mut exec, 0x1d, 0x60_0000, 0); + assert!(st & EXEC_IS_EXCEPTION != 0 && st & EXEC_IS_TLB_REFILL != 0, "status {st:#x}"); + }) + + } + + /// A valid entry with the dirty bit clear is accepted: CACHE never takes + /// TLB Modified. + #[test] + fn a_clean_page_takes_no_tlb_modified() { + big_stack(|| { + use crate::cpu::mips_exec::EXEC_COMPLETE; + let (mut exec, _) = mapped(3, 0, 0x3a5); + let tag = l2_tag_of(0x3a5000) | 2 << L2_STATE_SHIFT; + assert_eq!(cache_at(&mut exec, 0x0b, VA, tag), EXEC_COMPLETE); + assert_eq!(cache_at(&mut exec, 0x1f, VA, 7), EXEC_COMPLETE); + assert_eq!(cache_at(&mut exec, 0x03, VA, 0), EXEC_COMPLETE); + }) + + } + + /// A CACHE operand on an uncached page is undefined on the R10000. This + /// model translates it and goes ahead as if the page were cached; pinned + /// here so a change is deliberate. + #[test] + fn an_operand_on_an_uncached_page_goes_ahead() { + big_stack(|| { + use crate::cpu::mips_exec::EXEC_COMPLETE; + let (mut exec, _) = mapped(2, 1, 0x3a5); + let tag = l2_tag_of(0x3a5000) | 2 << L2_STATE_SHIFT; + assert_eq!(cache_at(&mut exec, 0x0b, VA, tag), EXEC_COMPLETE); + assert!(exec.cache.page_staged(0x3a5000)); + }) + + } + + /// Which pages the page-granular fast paths must leave to the cache: + /// exactly those with a valid staged line, reported once as a change. + #[test] + fn the_staged_page_set_follows_the_valid_lines() { + let (c, _) = staged_cache(); + assert!(!c.page_staged(PA) && !c.take_staged_changed()); + stage_tag(&c, PA, 0, 2); + assert!(c.page_staged(PA) && c.page_staged(PA | 0xfff) && !c.page_staged(PA + 0x1000)); + assert!(c.take_staged_changed()); + assert!(!c.take_staged_changed(), "reading clears it"); + stage_tag(&c, PA + 128, 0, 1); // second line, same page: no change + stage_tag(&c, PA, 0, 0); + assert!(c.page_staged(PA) && !c.take_staged_changed()); + stage_tag(&c, PA + 128, 0, 0); + assert!(!c.page_staged(PA) && c.take_staged_changed()); + } + + /// Only the R10000 family stages lines: the R4400's index operations are + /// as before. + #[test] + fn the_r4400_does_not_stage_lines() { + let c = R4400ShadowCache::from(Arc::new(Bus::default()) as Arc); + c.cache_op(C_IST | CACH_SD, 0x8010_1000, 0x0000_0800 | 4 << 14); + assert!(!c.page_staged(PA)); + assert_eq!(c.read::<8>(0x8010_1000, PA).data, MEM); + } + /// Under tcache the JIT serves this model through the window alone, and /// only once both windows are published: the emitted code dereferences /// them without a null check. diff --git a/src/cpu/mips_cache_v2.rs b/src/cpu/mips_cache_v2.rs index 3dffd558c..7912a1218 100644 --- a/src/cpu/mips_cache_v2.rs +++ b/src/cpu/mips_cache_v2.rs @@ -565,6 +565,30 @@ pub trait CpuModel: MipsCache { /// and `Index_Load_Data` returns into it, neither of which is true of the /// R4000 hit operations that share those encodings. const R10K_CACHE_OPS: bool = false; + /// The part is in the R10000 family, which shares one programming model: + /// FrameMask, and 44 virtual and 40 physical address bits. + /// Established for the R10000. The R12000 is inferred to match: its manual's + /// list of changes from the R10000 has no TLB or CP0 change. The R14000 and + /// R16000 are a model choice that gives them the same TLB. + const R10K_FAMILY: bool = false; + /// A partial LWR keeps bits 63:32 of the destination. Observed on the + /// R10000; whether the later parts do the same is not established, so they + /// keep the sign extension for now. + const PARTIAL_LWR_KEEPS_UPPER: bool = false; + /// An uncached store the system refuses completes instead of raising a + /// Data Bus Error. The R10000 sends it as a write request, which gets no + /// response, and Bus Error comes only from an error response to a read or + /// upgrade request or from uncorrectable read data (R10000 UM pp. 104, 316). + /// The R12000 manual documents the same path and the later parts are taken + /// to match. An uncached SC succeeding is a model choice. Loads and fetches + /// still fault. + const POSTS_UNCACHED_STORES: bool = false; + /// Bits `Index_Store_Tag` keeps for the secondary cache: all of them, except + /// on the R10000 family, whose reserved tag bits read as zero (R10000 UM p. 270). + const SECONDARY_TAG_MASK: u64 = u64::MAX; + /// The low 25 bits of Config at reset, where the part's own word is known + /// and differs from the R10000 family's generic one. Zero keeps the generic. + const CONFIG0_RESET: u32 = 0; } pub trait MipsCache: Send + Sync { @@ -595,6 +619,21 @@ pub trait MipsCache: Send + Sync { /// Default `false`: a model that keeps instruction-cache state does. fn fetch_is_pure(&self) -> bool { false } + /// Could a cached access to the page holding `phys` be served from a + /// secondary line the guest staged with Index Store Tag and Index Store + /// Data, instead of from memory? Only the page-granular fast paths that + /// bypass `read`, `write` and `fetch` ask: the micro-TLB fill for the + /// interpreter's and the JIT's inline data accesses, and the JIT's page + /// tracking, which compiles from memory. A page that answers `true` is + /// kept off those paths. Default `false`: a model that serves every + /// access itself has no such line. + fn page_staged(&self, _phys: u64) -> bool { false } + + /// Did the set of pages `page_staged` answers `true` for change since + /// the last call? Reading it clears it. The executor drops the + /// translations that were cached under the old answer. + fn take_staged_changed(&self) -> bool { false } + /// Write every dirty line back to memory, so RAM holds everything. /// Default no-op: a cache that holds no data has nothing to write. fn writeback_all(&self) {} @@ -740,6 +779,12 @@ pub trait MipsCache: Send + Sync { /// takes them from this register and `Index_Load_Data` returns them /// there. Models that do not keep them ignore both of these. fn set_cache_ecc(&self, _v: u32) {} + /// The translated physical address of the CACHE operand about to be + /// passed to `cache_op`. The R10000's secondary index and way come from it + /// rather than from the virtual address (NEC U12754EJ1V0UMJ1 pp.14-15). + /// Taken by the next `cache_op`, which falls back to the virtual address + /// if none was set. Models that index by the virtual address ignore it. + fn set_cache_op_pa(&self, _pa: u64) {} /// The Status bits that govern cache error checking: CE (store the ECC /// register as check bits instead of computing them) and whether errors /// are detected at all (neither DE nor ERL set). Pushed by the executor on @@ -1385,6 +1430,14 @@ pub mod model { pub const R4400: u8 = 0; pub const R5000: u8 = 1; pub const R10000: u8 = 2; + pub const R12000: u8 = 3; + pub const R16000: u8 = 4; + + /// The R10000 family: the R10000 and the parts that follow it, which share + /// one programming model. + pub const fn r10k_family(model: u8) -> bool { + matches!(model, R10000 | R12000 | R16000) + } } /// SGI Indy R4400: direct-mapped 16K L1s, 1 MB unified L2 owning the decode slots. @@ -3595,7 +3648,10 @@ impl self.cp0_entrylo0 & 0x3FFFFFFF, // PFN is 24 bits (29:6), flags in lower bits - 3 => self.cp0_entrylo1 & 0x3FFFFFFF, // PFN is 24 bits (29:6), flags in lower bits + 2 => self.cp0_entrylo0 & self.entrylo_mask, + 3 => self.cp0_entrylo1 & self.entrylo_mask, 4 => self.cp0_context, 5 => self.cp0_pagemask & 0x01FFE000, // PageMask: only bits 24:13 are valid 6 => self.cp0_wired as u64 & 0x3F, // Wired: only bits 5:0 are valid, R4000 manual Table 4-10 @@ -1636,6 +1654,7 @@ impl MipsCore { 18 => self.cp0_watchlo as u64, 19 => self.cp0_watchhi as u64, 20 => self.cp0_xcontext, + 21 => self.cp0_framemask as u64, 25 => { // PerfCnt: R4400 has no performance counters. Log any probe so // a guest expecting them (and routing their overflow onto the @@ -1670,8 +1689,8 @@ impl MipsCore { match reg { 0 => self.cp0_index as u64, 1 => self.cp0_random as u64, - 2 => self.cp0_entrylo0 & 0x3FFFFFFF, - 3 => self.cp0_entrylo1 & 0x3FFFFFFF, + 2 => self.cp0_entrylo0 & self.entrylo_mask, + 3 => self.cp0_entrylo1 & self.entrylo_mask, 4 => self.cp0_context, 5 => self.cp0_pagemask & 0x01FFE000, 6 => self.cp0_wired as u64 & 0x3F, @@ -1688,6 +1707,7 @@ impl MipsCore { 18 => self.cp0_watchlo as u64, 19 => self.cp0_watchhi as u64, 20 => self.cp0_xcontext, + 21 => self.cp0_framemask as u64, 26 => self.cp0_ecc as u64, 27 => self.cp0_cacheerr as u64, 28 => self.cp0_taglo, @@ -1697,6 +1717,14 @@ impl MipsCore { } } + /// An EntryLo value as a TLB write passes it on: FrameMask bits 15:0 clear + /// EntryLo bits 33:18, so the flags IRIX keeps above the PFN's low bits + /// never reach the physical address (R10000 UM v2.0 p. 260). + #[inline] + pub fn framemask_entrylo(&self, lo: u64) -> u64 { + lo & !((self.cp0_framemask as u64) << 18) + } + /// Materialize the virtual CP0 Count: advance `cp0_count` by the whole /// hardware ticks elapsed since the anchor at `count_hz`, move the anchor /// forward by exactly the duration those ticks consumed (the sub-tick @@ -2011,8 +2039,8 @@ impl MipsCore { 0 => self.cp0_index = (value as u32) & (crate::cpu::mips_exec::CP0_INDEX_P | crate::cpu::mips_exec::CP0_INDEX_SLOT_MASK), 1 => { /* Random is read-only */ } - 2 => self.cp0_entrylo0 = value & 0x3FFFFFFF, // PFN is 24 bits (29:6), flags in lower bits - 3 => self.cp0_entrylo1 = value & 0x3FFFFFFF, // PFN is 24 bits (29:6), flags in lower bits + 2 => self.cp0_entrylo0 = value & self.entrylo_mask, + 3 => self.cp0_entrylo1 = value & self.entrylo_mask, 4 => { //eprintln!("MTC0 Context = {:#018x} (PTEBase={:#018x})", value, value & 0xFFFFFFFF_FF800000); self.cp0_context = value; @@ -2035,16 +2063,16 @@ impl MipsCore { } 10 => { // always use 64bit mask because the entries need to be valid in 64 bit mode even when they were set from 32 bit mode // `log mips mask cp0` shows what the guest wrote against - // what survives the mask. The mask is R4400's 40-bit virtual - // address; the R10000 implements 44. - if cp0_log() && (value & !0xC000_00FF_FFFF_E0FF) != 0 { + // what survives the mask: the model's VPN2 width, 40 bits or the + // R10000 family's 44. + if cp0_log() && (value & !self.entryhi_mask) != 0 { crate::dlog!(crate::devlog::LogModule::Mips, "ip28ehi: wrote {value:#018x} -> kept {:#018x} (lost {:#018x}) pc={:#018x}", - value & 0xC000_00FF_FFFF_E0FF, - value & !0xC000_00FF_FFFF_E0FF, + value & self.entryhi_mask, + value & !self.entryhi_mask, self.pc); } - self.cp0_entryhi = value & 0xC000_00FF_FFFF_E0FF; + self.cp0_entryhi = value & self.entryhi_mask; }, 11 => { // Materialize the count *first* so both new_delta below and @@ -2253,6 +2281,10 @@ impl MipsCore { } self.cp0_xcontext = value; } + // FrameMask: bits 15:0 are implemented, the rest read as zero + // (R10000 UM v2.0 p. 260). Other parts have no register 21 and ignore + // the write. + 21 => self.cp0_framemask = value as u32 & self.framemask_mask, 25 => { #[cfg(feature = "developer_ip7")] eprintln!("[ip7] MTC0 PerfCnt (reg 25) write {:#018x} (ignored)", value); diff --git a/src/cpu/mips_exec.rs b/src/cpu/mips_exec.rs index 62aaee8de..3a98bd078 100644 --- a/src/cpu/mips_exec.rs +++ b/src/cpu/mips_exec.rs @@ -577,17 +577,28 @@ pub const TR_GLOBAL: u32 = 1 << 4; /// the privilege epoch that filled it. The only thing rejected here is a VA /// shape that is not a valid address in the first place. #[inline] -fn nutlb_cacheable_segment(va: u64, is_64bit: bool) -> bool { +fn nutlb_cacheable_segment(va: u64, is_64bit: bool) -> bool { if !is_64bit { // Every 32-bit segment (kuseg/kseg0/kseg1/ksseg/kseg3) is a real // address whose finished translation is worth caching. return true; } - // xuseg: bits 63:40 must be zero to be a valid user address. Every other - // 64-bit segment (xsseg, xkphys, xkseg and the 0xFFFFFFFF_* compat - // windows) is fine — translate_64bit_impl raises before returning ok for - // the illegal xkphys sub-ranges, so nothing invalid reaches a fill. - (va >> 62) != 0 || (va >> 40) == 0 + // xuseg: bits 63:`C::VA_BITS` must be zero to be a valid user address. + // Every other 64-bit segment (xsseg, xkphys, xkseg and the 0xFFFFFFFF_* + // compat windows) is fine — translate_64bit_impl raises before returning ok + // for the illegal xkphys sub-ranges, so nothing invalid reaches a fill. + (va >> 62) != 0 || (va >> C::VA_BITS) == 0 +} + +/// The EntryHi bits this model keeps: region, VPN2 up to `C::VA_BITS` and ASID. +#[inline(always)] +const fn model_entryhi_mask() -> u64 { + 0xC000_0000_0000_00FF | crate::cpu::mips_tlb::vpn2_mask(C::VA_BITS) +} + +/// Whether the `n`th refused store is reported: the first four, then each power of two. +pub(crate) fn report_refused_store(n: u64) -> bool { + n <= 4 || n.is_power_of_two() } /// Per-access-class tally of how far a guest load/store would get through the @@ -853,6 +864,9 @@ pub struct MipsExecutor { pub decoded_count: Arc, /// Count of instructions fetched from uncached address space. pub uncached_fetch_count: Arc, + /// Refused uncached stores so far (see `CpuModel::POSTS_UNCACHED_STORES`); + /// sets how often `post_refused_store` reports. + refused_stores: u64, /// Hot-path translation function pointer, updated whenever CP0 Status changes. /// Always the non-debug variant; selects the correct 32/64-bit × privilege specialisation. pub translate_fn: fn(&mut Self, u64, AccessType) -> TranslateResult, @@ -989,6 +1003,16 @@ pub struct MipsExecutor { /// short loop). #[cfg(feature = "jitv2")] pub jitv2_inline_compile: bool, + /// The machine's uncached bus view decodes code differently from its + /// cached one (IP35), so an uncached fetch keys its own code page and + /// is compiled from what the uncached view returns. Set by + /// [`Self::enable_jitv2_uncached_views`]; off on every other machine. + #[cfg(feature = "jitv2")] + pub(crate) jitv2_uncached_views: bool, + /// The bus compiles read code through, when it is not `sysad` itself + /// (see [`Self::enable_jitv2_uncached_views`]). + #[cfg(feature = "jitv2")] + jitv2_code_bus: Option>, /// Scratch analyzer for `jitv2_inline_compile`'s synchronous path — same /// `comp::handle_request` compile sequence the real async compile-thread /// runs, just called directly on the CPU thread. Kept alive across calls @@ -1023,12 +1047,12 @@ pub struct MipsExecutor { pub batch_hook: Option) + Send>>, /// A window of physical addresses that reaches memory at another address, /// `(base, size, target)`, applied to mapped and XKPHYS translations - /// before [`Self::phys_mask`]. The O2 has a no-ECC alias of its memory at + /// after [`Self::phys_mask`]. The O2 has a no-ECC alias of its memory at /// 0x8000_0000 that would otherwise reach the PCI window the bus decodes /// there. None on the other machines. pub phys_alias: Option<(u64, u64, u64)>, /// The physical address bits the board decodes, applied to mapped, - /// XKPHYS and ERL-identity translations after [`Self::phys_alias`]. The + /// XKPHYS and ERL-identity translations before [`Self::phys_alias`]. The /// CPU drives 36 or 40 bits, but the Indy, Indigo2 and O2 decode 32 and /// alias the rest away, and IRIX relies on it: its page-table entries keep /// software bits (the PFN lock among them) just above the PFN, so they @@ -1350,20 +1374,21 @@ unsafe extern "C" fn jit_read64(ctx: *mut core::ffi::c_void #[cfg(feature = "jitv2")] unsafe extern "C" fn jit_write8(ctx: *mut core::ffi::c_void, va: u64, val: u64) -> u32 { let exec = unsafe { &mut *exec_from_core::(ctx) }; - let val = val as u8; // mask to the real width ourselves — see write8_fn's doc comment on why the FFI parameter is u64 + // The whole register, as the interpreter's SB passes it: `write_data` + // narrows it where it stores, and an uncached store hands it to the bus + // (`BusDevice::write_narrow_from`). #[cfg(feature = "jitv2_lockstep")] - { return exec.lockstep_jit_write::<1>(va, val as u64); } + { return exec.lockstep_jit_write::<1>(va, val); } #[cfg(not(feature = "jitv2_lockstep"))] - exec.write_data::<1>(va, val as u64) + exec.write_data::<1>(va, val) } #[cfg(feature = "jitv2")] unsafe extern "C" fn jit_write16(ctx: *mut core::ffi::c_void, va: u64, val: u64) -> u32 { let exec = unsafe { &mut *exec_from_core::(ctx) }; - let val = val as u16; // mask to the real width ourselves — see write16_fn's doc comment #[cfg(feature = "jitv2_lockstep")] - { return exec.lockstep_jit_write::<2>(va, val as u64); } + { return exec.lockstep_jit_write::<2>(va, val); } #[cfg(not(feature = "jitv2_lockstep"))] - exec.write_data::<2>(va, val as u64) + exec.write_data::<2>(va, val) } #[cfg(feature = "jitv2")] unsafe extern "C" fn jit_write32(ctx: *mut core::ffi::c_void, va: u64, val: u64) -> u32 { @@ -2193,6 +2218,22 @@ impl MipsExecutor { self.sysad.describe_fault(phys_addr, size).map_or_else(String::new, |why| format!(" [{why}]")) } + /// An uncached store the system refused, on a model that posts stores + /// (`CpuModel::POSTS_UNCACHED_STORES`): say so and let the store complete. + /// The caller returns `BUS_OK`. Always built, so a missing device or a + /// guest bug stays visible in every build. A probe loop would flood + /// stderr, so it reports the first four and then every power of two, with + /// the running count. + #[cold] + fn post_refused_store(&mut self, virt_addr: u64, phys_addr: u64, size: usize, val: u64, status: u32) { + self.refused_stores += 1; + let n = self.refused_stores; + if report_refused_store(n) { + eprintln!("Unsupported uncached write{} (posted, completes): PC={:016x} VA={:016x} PA={:016x} val={:016x} status={:08x}{} [store {n}]", + size * 8, self.core.pc, virt_addr, phys_addr, val, status, self.fault_note(phys_addr, size)); + } + } + /// Create a new executor from a config and a bus (sysad) and a TLB. /// The cache hierarchy is constructed internally as a unified R4000Cache. pub fn new(sysad: Arc, mut tlb: T, cfg: &MipsCpuConfig) -> Self @@ -2202,7 +2243,16 @@ impl MipsExecutor { let mut core = MipsCore::new(); // Mapped pages decode their C field with this model's table, as KSEG0 // and xkphys do. + tlb.set_va_bits(C::VA_BITS); tlb.set_cache_attrs(Self::cache_attr_table()); + tlb.set_conflict_invalidation(C::R10K_FAMILY); + core.entryhi_mask = model_entryhi_mask::(); + if C::R10K_FAMILY { + core.framemask_mask = 0xFFFF; + // PFN in bits 33:6 and the uncached attribute in 63:62 (R10000 UM v2.0 + // pp. 239-240); 61:34 are reserved. + core.entrylo_mask = 0xC000_0003_FFFF_FFFF; + } // Cache geometry comes from the model type C, so Config describes this CPU. // `mut` is only used by the Triton L2-enable sync below. @@ -2283,16 +2333,15 @@ impl MipsExecutor { c |= log2(32768) << 26; // 32 KB L1D c |= 1 << 15; // big endian if C::L2_LINE == 128 { c |= 1 << 13; } - // Secondary cache size. The encoding is not in anything to hand, - // so it was swept against the PROM rather than guessed. - // - // 1 is the value to keep: IRIX reports "Secondary unified - // instruction/data cache size: 1 Mbyte", which agrees with this - // model's own `L2_SIZE`, and the PROM's power-on diagnostics pass - // and IRIX boots with it. 4 also passes POST but has IRIX report - // 8 MB, contradicting the model. - c |= 1 << 16; + // SS is log2(secondary cache size / 512 KB): 0 is 512 KB, 1 is 1 MB, + // 2 is 2 MB, up to 5 for 16 MB (NEC U10278EJ4V0UM p.230). It follows + // the model's own `L2_SIZE`. EC, SC, DN, PM, DSD and SK stay zero. + c |= (C::L2_SIZE / (512 << 10)).trailing_zeros() << 16; c |= 2; // K0 = uncached at reset + // A part with a known reset word presents it over the low 25 bits. + if C::CONFIG0_RESET != 0 { + c = (c & !crate::cpu::mips_cache_shadow::CONFIG0_RESET_MASK) | C::CONFIG0_RESET; + } config = c; } @@ -2382,6 +2431,7 @@ impl MipsExecutor { ins: DecodedInstr::default(), // scratch slot for uncached fetches decoded_count: Arc::new(AtomicU64::new(0)), uncached_fetch_count: Arc::new(AtomicU64::new(0)), + refused_stores: 0, // Placeholder — overwritten immediately by update_translate_fn below. translate_fn: translate_32_kernel::, // Placeholder — overwritten immediately by update_fpr_mode below. @@ -2450,6 +2500,10 @@ impl MipsExecutor { #[cfg(feature = "jitv2")] jitv2_inline_compile: cfg!(feature = "jitv2_lockstep"), #[cfg(feature = "jitv2")] + jitv2_uncached_views: false, + #[cfg(feature = "jitv2")] + jitv2_code_bus: None, + #[cfg(feature = "jitv2")] jitv2_inline_analyzer: crate::cpu::jitv2::analyzer::Analyzer::with_isa(C::MIPS4), // IRIS_JIT_DISPATCH=off starts with the JIT dispatch gate shut: the // same binary as an interpreter-only reference, for comparing @@ -2618,6 +2672,7 @@ impl MipsExecutor { // set) -- the property a persistent code cache needs. // Several CPUs share compiled code, so no one core's address. core: if self.core.smp || std::env::var_os("IRIS_JIT_PIC").is_some() { None } else { core::num::NonZeroUsize::new(core_addr) }, + lwr_keeps_upper: C::PARTIAL_LWR_KEEPS_UPPER, }; *j.jit_consts.lock() = consts; // The inline-compile path (`jitv2_compile_inline`) takes this @@ -2747,7 +2802,7 @@ impl MipsExecutor { self.core.nanotlb[AT as usize].fill_raw(va_page, result.phys, result.status & 0x7); #[cfg(feature = "jitv2")] if AT == AccessType::Fetch as u8 { - self.jitv2_track_pcp(result.phys); + self.jitv2_track_pcp(result.phys, !result.is_cached()); } } result @@ -2868,12 +2923,19 @@ impl MipsExecutor { return; } + // A page holding a line the guest staged in the secondary cache is + // served by the cache's own read and write: an entry here would let + // the JIT's inline path and the window read memory instead. + if self.cache.page_staged(result.phys) { + return; + } + // Segment shapes we decline to cache at all — currently only a // 64-bit xuseg VA with a nonzero bits 63:40, which is not a valid // user address. Everything else is cacheable regardless of which // mode filled it, because privilege is enforced by flushing on // transition rather than by the tag. - if !nutlb_cacheable_segment(va, self.core.is_64bit_mode()) { + if !nutlb_cacheable_segment::(va, self.core.is_64bit_mode()) { return; } @@ -2961,7 +3023,7 @@ impl MipsExecutor { if !self.smc_seen.insert(key) { return; } - let page_base = pfn as u64 * crate::cpu::jitv2::PAGE_SIZE as u64; + let page_base = crate::cpu::jitv2::jitv2::key_phys_base(pfn); let word = ((phys_addr - page_base) / 4) as usize; eprintln!( "jitv2 SMC: write to the page being executed — pc={:#018x} wrote {} byte(s) \ @@ -2978,12 +3040,16 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", /// [`Self::smc_check_report`]. /// /// `cur_code_pfn == u32::MAX` (no page tracked) can never match a real - /// frame, so the "nothing tracked" case needs no separate test. + /// frame, so the "nothing tracked" case needs no separate test. The + /// comparison masks the view bit off `cur_code_pfn`, which leaves the + /// sentinel as 0xFFBFFFFF: still beyond any frame the page table holds. #[cfg(all(feature = "jitv2", feature = "jitv2_smc_check"))] #[inline(always)] fn smc_check_write(&mut self, virt_addr: u64, phys_addr: u64, size: usize) { let pfn = self.core.cur_code_pfn; - if (phys_addr / crate::cpu::jitv2::PAGE_SIZE as u64) as u32 == pfn { + if (phys_addr / crate::cpu::jitv2::PAGE_SIZE as u64) as u32 + == pfn & !crate::cpu::jitv2::jitv2::UNCACHED_VIEW_PFN + { self.smc_check_report(virt_addr, phys_addr, size, pfn); } } @@ -3022,7 +3088,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", /// may have made the flush unnecessary. #[cfg(feature = "jitv2")] pub(crate) fn jitv2_flush_from_here(&mut self, only_if_exhausted: bool) { - let bus = self.sysad.clone(); + let bus = self.jitv2_compile_bus(); let jit = self.jitv2.clone(); let flush = || { let mut j = jit.lock(); @@ -3051,6 +3117,40 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", paddr, self.memory)) } + /// The page key for a fetch at `phys_addr`: its frame, in the uncached + /// view's key space when `uncached` and this machine keeps one + /// (`jitv2::code_key`). `None` above the page table. + #[cfg(feature = "jitv2")] + pub(crate) fn jitv2_code_key(&self, phys_addr: u64, uncached: bool) -> Option { + crate::cpu::jitv2::jitv2::code_key(phys_addr, self.jitv2_frames, uncached && self.jitv2_uncached_views) + } + + /// Tell the JIT this machine's uncached bus view decodes code differently + /// from its cached view, so uncached fetches get their own code pages and + /// are compiled from `code_bus`, which must serve a page's bytes through + /// the view its key names (see `jitv2::key_bus_addr`). Call while the + /// executor is being built, before it is shared or run. Machines without + /// such views never call this and keep one page per frame. + #[cfg(feature = "jitv2")] + pub fn enable_jitv2_uncached_views(&mut self, code_bus: Arc) { + self.jitv2.lock().enable_uncached_views(); + self.jitv2_uncached_views = true; + self.jitv2_code_bus = Some(code_bus); + self.jitv2_bind_fast_lookup(); + } + + /// The code page the last fetch landed on, or null. + #[cfg(feature = "jitv2")] + pub fn jitv2_current_page(&self) -> *mut crate::cpu::jitv2::PhysicalCodePage { + self.pcp + } + + /// The bus compile workers and the inline compile read code through. + #[cfg(feature = "jitv2")] + pub fn jitv2_compile_bus(&self) -> Arc { + self.jitv2_code_bus.clone().unwrap_or_else(|| self.sysad.clone()) + } + #[cfg(feature = "jitv2")] fn jitv2_share(&mut self, jit: Arc>, world: Arc) { { @@ -3095,13 +3195,23 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", /// nanotlb hit path above and a single PFN comparison here. #[cfg(feature = "jitv2")] #[inline(always)] - fn jitv2_track_pcp(&mut self, phys_addr: u64) { + fn jitv2_track_pcp(&mut self, phys_addr: u64, uncached: bool) { // Code above the JIT's pfn map gets no page and runs in the - // interpreter (`step_jit` checks for the null page). - let Some(pfn) = crate::cpu::jitv2::jitv2::pfn_of(phys_addr, self.jitv2_frames) else { + // interpreter (`step_jit` checks for the null page). That includes + // uncached attributes 1 to 3: their bits 58:57 sit in `phys_addr`, + // so the frame range check rejects them. Where the uncached bus view + // differs from the cached one, an uncached fetch keys a page of its + // own (`code_key`). + let Some(pfn) = crate::cpu::jitv2::jitv2::code_key(phys_addr, self.jitv2_frames, uncached && self.jitv2_uncached_views) else { self.clear_pcp(); return; }; + // Compiled code is built from memory. A page with a staged secondary + // line must run in the interpreter, whose fetch goes through the cache. + if self.cache.page_staged(phys_addr) { + self.clear_pcp(); + return; + } let same_page = !self.pcp.is_null() && unsafe { (*self.pcp).pfn == pfn }; if same_page { return; @@ -3117,7 +3227,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", // on the page-crossing path (not every fetch) to keep the common // same-page case free of this cost even when the toggle is on. if jitv2_pagewb_enabled() { - let page_base = pfn as u64 * crate::cpu::jitv2::PAGE_SIZE as u64; + let page_base = crate::cpu::jitv2::jitv2::key_phys_base(pfn); self.cache.writeback(None, page_base, crate::cpu::jitv2::PAGE_SIZE as u64); } // A page crossing landing exactly on word 0 (the exception/TLB-refill @@ -3153,7 +3263,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", return; } - let page_base = pfn as u64 * crate::cpu::jitv2::PAGE_SIZE as u64; + let page_base = crate::cpu::jitv2::jitv2::key_phys_base(pfn); // §13: live FR mode at first-arrival time, pinned into the page for // its whole lifetime (PhysicalCodePage::fr1's own doc comment) — // ignored by page_for on a lookup hit (an already-claimed page keeps @@ -3322,7 +3432,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", /// cost to the interpreter's per-instruction hot loop. pub fn exec(&mut self, instr: u32) -> ExecStatus { #[cfg(feature = "jitv2")] - self.jitv2_track_pcp(self.core.pc & 0xFFFF_FFFF); + self.jitv2_track_pcp(self.core.pc & 0xFFFF_FFFF, false); self.ins.raw = instr; self.ins.flags = FLAG_NOT_DECODED; decode_into::(&mut self.ins); @@ -4518,11 +4628,11 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", let top_bits = virt_addr >> 62; match top_bits { - // xuseg: 0x0000_0000_0000_0000 - 0x0000_00FF_FFFF_FFFF (user mapped) - // Accessible from all privilege levels + // xuseg is user-mapped below 2^C::VA_BITS (0x0000_00FF_FFFF_FFFF + // for the 40-bit models) and accessible from every privilege level. 0 => { - if (virt_addr >> 40) != 0 { - // Bits 63:40 must be zero for valid user address + if (virt_addr >> C::VA_BITS) != 0 { + // Bits above the model's implemented user VA must be zero if !DEBUG { self.core.cp0_badvaddr = virt_addr; } return TranslateResult::exc(exec_exception(if access_type == AccessType::Write { EXC_ADES } else { EXC_ADEL })); } @@ -4608,15 +4718,17 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", } } - /// A CPU physical address as the board decodes it: through - /// [`Self::phys_alias`] if it falls in it, then cut to [`Self::phys_mask`]. + /// A CPU physical address as the board decodes it: cut to + /// [`Self::phys_mask`], then through [`Self::phys_alias`] if it falls in it. #[inline(always)] fn fold_phys(&self, pa: u64) -> u64 { - let pa = match self.phys_alias { - Some((base, size, target)) if pa.wrapping_sub(base) < size => pa - base + target, + // Cut first: the PTE software bits above the PFN are not part of the + // address the alias window sees. Window and target lie within the mask. + let pa = pa & self.phys_mask; + match self.phys_alias { + Some((base, size, target)) if pa.wrapping_sub(base) < size => (pa - base + target) & self.phys_mask, _ => pa, - }; - pa & self.phys_mask + } } /// TLB translation. When `DEBUG` is true, CP0 side-effect registers are @@ -4649,7 +4761,8 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", TlbResult::Miss { .. } => { if !DEBUG { self.update_tlb_exception_registers::(virt_addr); } // Miss: XTLB vector (0x080) for 64-bit extended addresses, UTLB vector (0x000) otherwise - TranslateResult::exc(if XTLB != 0 { exec_xtlb_miss(tlb_miss_code) } else { exec_tlb_miss(tlb_miss_code) }) + let xrefill = if C::R10K_FAMILY { self.r10k_xtlb_refill::(virt_addr) } else { XTLB != 0 }; + TranslateResult::exc(if xrefill { exec_xtlb_miss(tlb_miss_code) } else { exec_tlb_miss(tlb_miss_code) }) } TlbResult::Invalid { .. } => { if !DEBUG { self.update_tlb_exception_registers::(virt_addr); } @@ -4663,6 +4776,26 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", } } + /// Whether a refill miss on `virt_addr` takes the XTLB vector on the + /// R10000 family. The R10000 chooses by the space the address lies in, + /// not the mode it was referenced from: a user address follows UX, xsseg + /// SX, and kseg3, sseg/ksseg and xkseg follow KX (R10000 UM v2.0 pp. + /// 333-334, 342). The R4400 chooses by the current mode, which is what + /// `XTLB` encodes. So a 64-bit kernel (KX=1, UX=0) touching kuseg takes + /// the 32-bit refill vector here. In 32-bit mode only the low word of the + /// address counts, so it is sign-extended before the region is read. + #[cold] + fn r10k_xtlb_refill(&self, virt_addr: u64) -> bool { + use crate::cpu::mips_core::{STATUS_KX, STATUS_SX, STATUS_UX}; + let va = if XTLB != 0 { virt_addr } else { virt_addr as u32 as i32 as i64 as u64 }; + let x = match va >> 62 { + 0 => STATUS_UX, + 1 => STATUS_SX, + _ => STATUS_KX, + }; + self.core.cp0_status & x != 0 + } + /// Debug helper to translate address without side effects pub fn debug_translate(&mut self, virt_addr: u64) -> TranslateResult { self.translate_impl::(virt_addr, AccessType::Debug) @@ -4673,8 +4806,12 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", #[inline] fn update_tlb_exception_registers(&mut self, virt_addr: u64) { const EH_VPN2_32: u64 = 0x0000_0000_FFFF_E000; - const EH_VPN2_64: u64 = 0x0000_00FF_FFFF_E000; const EH_REGION: u64 = 0xC000_0000_0000_0000; + // VPN2 is bits 43:13 on the R10000 family (R10000 UM pp. 219, 288) and + // 39:13 on the R4x00 and R5000. The R12000 is inferred to match (its + // manual lists no TLB change); the R14000 and R16000 are a model choice + // that follows the same width. + let eh_vpn2_64 = crate::cpu::mips_tlb::vpn2_mask(C::VA_BITS); self.core.cp0_badvaddr = virt_addr; @@ -4697,9 +4834,9 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", // // This is MAME r4000's `EH_WM` (0xC000_00FF_FFFF_E0FF) less the ASID // bits, which are carried separately below. - const EH_VPN_MASK: u64 = EH_REGION | EH_VPN2_64; // 0xC000_00FF_FFFF_E000 + let eh_vpn_mask = EH_REGION | eh_vpn2_64; let asid = self.core.cp0_entryhi & 0xFF; - self.core.cp0_entryhi = (virt_addr & EH_VPN_MASK) | asid; + self.core.cp0_entryhi = (virt_addr & eh_vpn_mask) | asid; // Context: PTEBase[63:23] preserved, BadVPN2 = virt_addr[31:13] in bits [22:4]. // Always 32-bit VPN — Context is used by the 32-bit UTLB handler. @@ -4720,10 +4857,6 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", // the address it computes lands outside the page table it wired, the // nested miss is unrecoverable, and IRIX panics with "Entered tlbmiss // with invalid vaddr". - // - // EntryHi keeps the 40-bit VPN2 field on every model: a 64-bit kernel - // zero-fills those upper bits itself when it builds an EntryHi value, - // and widening the mask here breaks it. self.core.cp0_xcontext = xcontext_on_miss(C::VA_BITS, self.core.cp0_xcontext, virt_addr); } @@ -5068,7 +5201,15 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", if !DEBUG && mips_log(MIPS_LOG_MEM) { dlog_dev!(LogModule::Mips, "Uncached Write{}: PC={:016x} VA={:016x} PA={:016x} Val={:016x}", SIZE*8, self.core.pc, virt_addr, phys_addr, val); } - let ws = if SIZE == 1 { + let ws = if !DEBUG && SIZE <= 2 { + // The bus sees the whole source register: Fuel PROM 6.210's + // Walking Address test reads its check words back from + // Bedrock's ECC view, and the word sits elsewhere in the + // register than the addressed lanes. Ordinary devices take + // the architectural low bits. Debug writes carry just the + // value they name, so they keep the ordinary-width path. + self.sysad.write_narrow_from(phys_addr, SIZE, val) + } else if SIZE == 1 { self.sysad.write8(phys_addr, val as u8) } else if SIZE == 2 { self.sysad.write16(phys_addr, val as u16) @@ -5078,9 +5219,16 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", self.sysad.write64(phys_addr, val) }; if ws != BUS_OK && ws != BUS_BUSY && !DEBUG { - eprintln!("Bus error on uncached write{}: PC={:016x} VA={:016x} PA={:016x} val={:016x} status={:08x}{}", SIZE*8, self.core.pc, virt_addr, phys_addr, val, ws, self.fault_note(phys_addr, SIZE)); + if C::POSTS_UNCACHED_STORES { + self.post_refused_store(virt_addr, phys_addr, SIZE, val, ws); + BUS_OK + } else { + eprintln!("Bus error on uncached write{}: PC={:016x} VA={:016x} PA={:016x} val={:016x} status={:08x}{}", SIZE*8, self.core.pc, virt_addr, phys_addr, val, ws, self.fault_note(phys_addr, SIZE)); + ws + } + } else { + ws } - ws }; // jitv2_lockstep's load/store verification (see MipsCore's // lockstep_mem field doc comment, and read_data_impl's matching @@ -5134,9 +5282,16 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", } let ws = self.sysad.write64_masked(phys_addr, val, mask); if ws != BUS_OK && ws != BUS_BUSY && !DEBUG { - eprintln!("Bus error on uncached write64_masked: PC={:016x} VA={:016x} PA={:016x} val={:016x} mask={:016x} status={:08x}{}", self.core.pc, virt_addr, phys_addr, val, mask, ws, self.fault_note(phys_addr, 8)); + if C::POSTS_UNCACHED_STORES { + self.post_refused_store(virt_addr, phys_addr, mask.count_ones() as usize / 8, val, ws); + BUS_OK + } else { + eprintln!("Bus error on uncached write64_masked: PC={:016x} VA={:016x} PA={:016x} val={:016x} mask={:016x} status={:08x}{}", self.core.pc, virt_addr, phys_addr, val, mask, ws, self.fault_note(phys_addr, 8)); + ws + } + } else { + ws } - ws } } @@ -6354,8 +6509,20 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", let mask = 0xFFFFFFFFu32 >> shift; let result = (mem_word >> shift) | (rt_val & !mask); - // Sign-extend to 64 bits - self.core.write_gpr(rt_reg, result as i32 as i64 as u64); + // The manual leaves the upper word of a partial LWR implementation + // specific (NEC U12754E pp. 125-126). The R10000 keeps + // bits 63:32, as observed reference results show; the later + // parts are not established. A full word + // still sign-extends, and so does any word while 64-bit + // operations are off for the current mode (UX/SX/KX clear): + // there a register must hold a sign-extended 32-bit value. + let value = if C::PARTIAL_LWR_KEEPS_UPPER && byte_offset != 3 && self.core.is_64bit_mode() { + (self.core.read_gpr(rt_reg) & 0xffff_ffff_0000_0000) | result as u64 + } else { + // Sign-extend to 64 bits + result as i32 as i64 as u64 + }; + self.core.write_gpr(rt_reg, value); self.handle_exec_complete() } Err(status) => self.finish_status(status) @@ -6536,11 +6703,15 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", let cache_op = d.rt as u32; // Encoded operation: cache_sel in bits[1:0], op in bits[4:2] let op = cache_op & 0x1C; - // Determine if this is a Hit operation that needs address translation // On an R10000 the encodings for 5/6/7 are index operations, not the - // R4000 hit operations — translating them would fault on an index that - // is not a valid virtual address. See C_R10K_ISD in mips_cache_v2. + // R4000 hit operations. See C_R10K_ISD in mips_cache_v2. let sel = cache_op & 3; + // Bit `cache_op` of this mask: the primary-cache index data operations. + const R10K_PRIMARY_DATA_OPS: u32 = (1 << (C_R10K_ILD | CACH_PI)) + | (1 << (C_R10K_ILD | CACH_PD)) + | (1 << (C_R10K_ISD | CACH_PI)) + | (1 << (C_R10K_ISD | CACH_PD)); + let primary_data_op = C::R10K_CACHE_OPS && (1u32 << cache_op) & R10K_PRIMARY_DATA_OPS != 0; let r10k_index_op = C::R10K_CACHE_OPS && match op { C_R10K_CBARRIER => sel == CACH_PI, @@ -6548,18 +6719,35 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", C_R10K_ISD => matches!(sel, CACH_SI | CACH_SD), _ => false, }; - let needs_translation = - !r10k_index_op && matches!(op, C_CDX | C_HINV | C_HWBINV | C_HWB | C_HSV); + // On the R10000 every CACHE operand is translated like a data address + // before the operation, index operations included (NEC U12754EJ1V0UMJ1 + // pp.70, 72). + // A mapped operand can take TLB Refill, TLB Invalid or a privilege + // Address Error. It is translated as a read, so it never takes TLB + // Modified. The primary caches index by the virtual address and only + // use the translation for its exceptions. The secondary cache takes its + // set and way from the physical address (`set_cache_op_pa`). A Cache + // Barrier names no line and is not translated. An operand on an + // uncached page is undefined on the R10000; it is translated and the + // operation goes ahead as if the page were cached. + // The older models translate hit operations only, as before. + let cache_barrier = op == C_R10K_CBARRIER && sel == CACH_PI; + let needs_translation = if C::R10K_CACHE_OPS { + !cache_barrier + } else { + matches!(op, C_CDX | C_HINV | C_HWBINV | C_HWB | C_HSV) + }; let phys_addr = if needs_translation { - // Hit operations need address translation let tr = self.nanotlb_translate::<{AccessType::Read as u8}>(virt_addr); if tr.is_exception() { return self.handle_exception(tr.status); } tr.phys } else { - // Index operations use virt_addr as index, no translation needed virt_addr }; + if C::R10K_CACHE_OPS && needs_translation { + self.cache.set_cache_op_pa(phys_addr); + } // For Index_Store_Tag, pass TagLo via phys_addr let op = cache_op & 0x1C; @@ -6569,7 +6757,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", // that reads only TagLo sees a different tag from the one written. // Harmless for R4400/R5000, whose tags fit in 32 bits and who leave // TagHi zero: the shift-in contributes nothing there. - let stores_taglo = op == C_IST || (r10k_index_op && op == C_R10K_ISD); + let stores_taglo = op == C_IST || ((r10k_index_op || primary_data_op) && op == C_R10K_ISD); let phys_addr_or_taglo = if stores_taglo { ((self.core.cp0_taghi as u64) << 32) | (self.core.cp0_taglo as u32 as u64) } else { @@ -6620,6 +6808,13 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", } // Call unified cache interface let result = self.cache.cache_op(cache_op, virt_addr, phys_addr_or_taglo); + // A line staged or dropped changes which pages must be served by the + // cache's own read, write and fetch. Translations filled before the + // change (micro-TLB entries the interpreter and the JIT's inline + // loads and stores use, the page being compiled from) do not know. + if self.cache.take_staged_changed() { + self.nanotlb_invalidate(); + } // An I-cache write-back of a line with bad parity raises a Cache // Error from the CACHE instruction itself. #[cfg(feature = "cache-ecc")] @@ -6642,6 +6837,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", // Split back the way it arrived. Zeroing TagHi unconditionally // threw away the top of every tag wider than 32 bits. self.core.cp0_taglo = result & 0xFFFF_FFFF; + // A primary data cache access leaves TagHi zero (a model choice). self.core.cp0_taghi = (result >> 32) as u32; } @@ -7129,13 +7325,23 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", let g1 = (self.core.cp0_entrylo1 & 1) != 0; let g_combined = if g0 && g1 { 1u64 << 12 } else { 0 }; - // EH_WM = 0xC000_00FF_FFFF_E0FF — same mask MAME applies on TLBWI (clears reserved bits). - // Then clear bit 12 (G bit from EntryHi) and set combined G from Lo0&Lo1. - const EH_WM: u64 = 0xC000_00FF_FFFF_E0FF; + // EntryHi's writable mask (0xC000_00FF_FFFF_E0FF on a 40-bit model, MAME's + // `EH_WM`) clears reserved bits, as on TLBWI. Then clear bit 12 (G bit + // from EntryHi) and set combined G from Lo0&Lo1. + // + // The R10000 family stores zeros in the PFN bits the page size leaves + // out of the address, so TLBR does not return them (R10000 UM v2.0 + // pp. 299-300, TLBWI and TLBWR). PageMask bit n covers physical bit + // n-1, EntryLo bit n-7: 0x6000 (16 KB) clears EntryLo 7:6. The R4400 + // keeps the whole PFN. The manual documents this for the R10000 only; + // for the R12000, R14000 and R16000 it is a model choice, as no manual + // describes their TLB write. + let unused_pfn = if C::R10K_FAMILY { (self.core.cp0_pagemask & 0x01FF_E000) >> 7 } else { 0 }; + let lo = |v: u64| self.core.framemask_entrylo(v) & !unused_pfn; TlbEntry { page_mask: self.core.cp0_pagemask, - entry_hi: (self.core.cp0_entryhi & EH_WM & !0x1000) | g_combined, - entry_lo: [self.core.cp0_entrylo0, self.core.cp0_entrylo1], + entry_hi: (self.core.cp0_entryhi & self.core.entryhi_mask & !0x1000) | g_combined, + entry_lo: [lo(self.core.cp0_entrylo0), lo(self.core.cp0_entrylo1)], selector_bit_shift: 0, // all derived fields overwritten by MipsTlb::write() vcmp32: 0, vpn_hi32: 0, vcmp64: 0, vpn_hi64: 0, offset_mask: 0, pfn_base: [0; 2], @@ -7196,6 +7402,9 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", let entry = self.create_tlb_entry_from_cp0(); self.ip28_trace_tlb_write("tlbwi", index, &entry); self.tlb.write(index, entry); + if self.tlb.take_write_conflict() { + self.core.cp0_status |= crate::cpu::mips_core::STATUS_TS; + } // Flushes the nutlb too — the TLB it caches just changed. self.nanotlb_invalidate(); @@ -7220,6 +7429,9 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", let entry = self.create_tlb_entry_from_cp0(); self.ip28_trace_tlb_write("tlbwr", index, &entry); self.tlb.write(index, entry); + if self.tlb.take_write_conflict() { + self.core.cp0_status |= crate::cpu::mips_core::STATUS_TS; + } self.nanotlb_invalidate(); if mips_log(MIPS_LOG_TLB) { dlog_dev!(LogModule::Mips, "TLBWR: Write Random Index {}\n{}", index, self.tlb.format_entry(index)); } @@ -8882,10 +9094,11 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", let over_threshold = self.jitv2.lock().codegen.lock().as_ref() .is_some_and(|c| c.packing_stats().1 > crate::cpu::jitv2::CODEGEN_ARENA_FLUSH_THRESHOLD_BYTES); if over_threshold { - let phys_page_base = page.pfn as u64 * crate::cpu::jitv2::PAGE_SIZE as u64; - unsafe { self.jitv2.lock().flush_from_cpu_thread(self.sysad.clone()); } + let phys_page_base = page.phys_base(); + let uncached = page.is_uncached_view(); + unsafe { self.jitv2.lock().flush_from_cpu_thread(self.jitv2_compile_bus()); } self.clear_pcp(); - self.jitv2_track_pcp(phys_page_base); + self.jitv2_track_pcp(phys_page_base, uncached); return EXEC_RETRY; } let mut codegen = self.jitv2.lock().codegen.lock().take(); @@ -8894,6 +9107,7 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", "jitv2_lockstep: inline compile found no Codegen (async compile queue stole it?) — \ verification would silently be OFF. The queue must not run under lockstep."); let mut ran_out_of_memory = false; + let compile_bus = self.jitv2_compile_bus(); if let Some(codegen) = codegen.as_mut() { // Stamp the L1-D geometry the inline load/store fast path needs. // Done on every take rather than once at startup so a cache @@ -8905,10 +9119,10 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", #[cfg(feature = "developer")] { let stats = self.jitv2.lock().stats.clone(); - ran_out_of_memory = crate::cpu::jitv2::comp::handle_request(&req, &self.sysad, &mut self.jitv2_inline_analyzer, codegen, &stats); + ran_out_of_memory = crate::cpu::jitv2::comp::handle_request(&req, &compile_bus, &mut self.jitv2_inline_analyzer, codegen, &stats); } #[cfg(not(feature = "developer"))] - { ran_out_of_memory = crate::cpu::jitv2::comp::handle_request(&req, &self.sysad, &mut self.jitv2_inline_analyzer, codegen); } + { ran_out_of_memory = crate::cpu::jitv2::comp::handle_request(&req, &compile_bus, &mut self.jitv2_inline_analyzer, codegen); } } *self.jitv2.lock().codegen.lock() = codegen; { @@ -8918,10 +9132,11 @@ va={:#018x} phys={:#010x} (code pfn {:#x}, page {:#010x}, word {}/{})", } if ran_out_of_memory { let page = unsafe { &mut *self.pcp }; - let phys_page_base = page.pfn as u64 * crate::cpu::jitv2::PAGE_SIZE as u64; - unsafe { self.jitv2.lock().flush_from_cpu_thread(self.sysad.clone()); } + let phys_page_base = page.phys_base(); + let uncached = page.is_uncached_view(); + unsafe { self.jitv2.lock().flush_from_cpu_thread(self.jitv2_compile_bus()); } self.clear_pcp(); - self.jitv2_track_pcp(phys_page_base); + self.jitv2_track_pcp(phys_page_base, uncached); return EXEC_RETRY; } // Re-check is_runnable: handle_request can decline this exact @@ -11020,7 +11235,7 @@ impl Saveable for MipsCpu cp0u64!(cp0_taglo); cp0u32!(cp0_taghi); cp0u64!(cp0_badvaddr); cp0u64!(cp0_epc); cp0u64!(cp0_errorepc); cp0u64!(cp0_entrylo0); cp0u64!(cp0_entrylo1); cp0u64!(cp0_context); - cp0u64!(cp0_pagemask); cp0u64!(cp0_entryhi); cp0u64!(cp0_xcontext); + cp0u64!(cp0_pagemask); cp0u64!(cp0_entryhi); cp0u64!(cp0_xcontext); cp0u32!(cp0_framemask); tbl.insert("cp0".into(), toml::Value::Table(cp0)); // FPU @@ -11085,7 +11300,7 @@ impl Saveable for MipsCpu ld32!(cp0_ecc); ld32!(cp0_cacheerr); ld64!(cp0_taglo); ld32!(cp0_taghi); ld64!(cp0_entrylo0); ld64!(cp0_entrylo1); ld64!(cp0_context); ld64!(cp0_pagemask); ld64!(cp0_badvaddr); ld64!(cp0_entryhi); - ld64!(cp0_xcontext); ld64!(cp0_epc); ld64!(cp0_errorepc); + ld64!(cp0_xcontext); ld32!(cp0_framemask); ld64!(cp0_epc); ld64!(cp0_errorepc); // Restart the virtual count from the restored value and re-arm // the compare timer — Instants from the previous run are // meaningless here — with a live ticket for the restored deadline. diff --git a/src/cpu/mips_exec/monitor.rs b/src/cpu/mips_exec/monitor.rs index 1e64255d8..c066e5919 100644 --- a/src/cpu/mips_exec/monitor.rs +++ b/src/cpu/mips_exec/monitor.rs @@ -2027,7 +2027,7 @@ impl MipsCpu { #[cfg(not(feature = "jitv2_lockstep"))] { exec.jitv2_inline_compile = false; - let bus = exec.sysad.clone(); + let bus = exec.jitv2_compile_bus(); let old = exec.jitv2.lock().codegen.lock().take(); match old { Some(old) => { @@ -2555,11 +2555,11 @@ impl MipsCpu { if result.is_exception() { return Err(format!("{:#018x} doesn't translate", pc)); } - let Some(pfn) = exec.jitv2_frame(result.phys) else { + let Some(pfn) = exec.jitv2_code_key(result.phys, !result.is_cached()) else { return Err(format!("phys {:#x} is above the JIT's physical range", result.phys)); }; - let page_base = pfn as u64 * crate::cpu::jitv2::PAGE_SIZE as u64; - let sysad = exec.sysad.clone(); + let page_base = crate::cpu::jitv2::jitv2::key_phys_base(pfn); + let sysad = exec.jitv2_compile_bus(); let mut jit = exec.jitv2.lock(); let fr1 = (exec.core.cp0_status & crate::cpu::mips_core::STATUS_FR) != 0; let slot = jit.page_for(pfn, page_base, sysad.as_ref(), fr1); @@ -2569,7 +2569,7 @@ impl MipsCpu { } }; let page = unsafe { &*page_ptr }; - let sysad = exec.sysad.clone(); + let sysad = exec.jitv2_compile_bus(); let dump = crate::cpu::jitv2::pcp_dump::PcpDump::capture(page, sysad.as_ref()) .map_err(|st| format!("capture failed: bus status {:#x}", st))?; // A path argument is whichever arg is not the address. @@ -2603,7 +2603,7 @@ impl MipsCpu { // the bus, and holding the jitv2 lock across a few // thousand of those plus the filesystem would stall // every dispatching CPU thread for the whole dump. - let sysad = exec.sysad.clone(); + let sysad = exec.jitv2_compile_bus(); let mut unreadable = 0usize; let dumps: Vec = { let jit = exec.jitv2.lock(); @@ -2697,12 +2697,12 @@ impl MipsCpu { writeln!(writer, "No PhysicalCodePage tracked (self.pcp is null) and PC {:#018x} doesn't translate.", pc).unwrap(); return Ok(()); } - let Some(pfn) = exec.jitv2_frame(result.phys) else { + let Some(pfn) = exec.jitv2_code_key(result.phys, !result.is_cached()) else { writeln!(writer, "No PhysicalCodePage tracked (self.pcp is null) and PC {:#018x} is at phys {:#x}, above the JIT's physical range.", pc, result.phys).unwrap(); return Ok(()); }; - let page_base = pfn as u64 * crate::cpu::jitv2::PAGE_SIZE as u64; - let sysad = exec.sysad.clone(); + let page_base = crate::cpu::jitv2::jitv2::key_phys_base(pfn); + let sysad = exec.jitv2_compile_bus(); let fr1 = (exec.core.cp0_status & crate::cpu::mips_core::STATUS_FR) != 0; let mut jit = exec.jitv2.lock(); match jit.page_for(pfn, page_base, sysad.as_ref(), fr1) { @@ -2857,14 +2857,18 @@ impl MipsCpu { if exec.core.smp { return Err("j2 clear: another CPU may be running this page; use j2 flush".to_string()); } - if actual_args.len() < 2 { return Err("Usage: j2 clear ".to_string()); } + if actual_args.len() < 2 { return Err("Usage: j2 clear [uncached]".to_string()); } let symbols_arc = exec.symbols.clone(); let symbols = symbols_arc.lock(); let paddr = monitor_phys(parse_cpu_arg(actual_args[1], &exec.core, Some(&symbols))?); drop(symbols); - let pfn = exec.jitv2_monitor_pfn(paddr)?; - let page_base = pfn as u64 * crate::cpu::jitv2::PAGE_SIZE as u64; - let sysad = exec.sysad.clone(); + // A trailing `uncached` names the page the CPU fetched + // through the uncached view (machines that keep one). + let uncached = actual_args.get(2) == Some(&"uncached") && exec.jitv2_uncached_views; + let frame = exec.jitv2_monitor_pfn(paddr)?; + let pfn = if uncached { frame | crate::cpu::jitv2::jitv2::UNCACHED_VIEW_PFN } else { frame }; + let page_base = crate::cpu::jitv2::jitv2::key_phys_base(pfn); + let sysad = exec.jitv2_compile_bus(); let fr1 = (exec.core.cp0_status & crate::cpu::mips_core::STATUS_FR) != 0; let mut jit = exec.jitv2.lock(); match jit.page_for(pfn, page_base, sysad.as_ref(), fr1) { @@ -2904,15 +2908,19 @@ impl MipsCpu { // still stops the specific `func` this call saw, but // a fresh compile can re-publish the same offset // right after. - if actual_args.len() < 2 { return Err("Usage: j2 deny ".to_string()); } + if actual_args.len() < 2 { return Err("Usage: j2 deny [uncached]".to_string()); } let symbols_arc = exec.symbols.clone(); let symbols = symbols_arc.lock(); let paddr = monitor_phys(parse_cpu_arg(actual_args[1], &exec.core, Some(&symbols))?); drop(symbols); - let pfn = exec.jitv2_monitor_pfn(paddr)?; - let page_base = pfn as u64 * crate::cpu::jitv2::PAGE_SIZE as u64; + // A trailing `uncached` names the page the CPU fetched + // through the uncached view (machines that keep one). + let uncached = actual_args.get(2) == Some(&"uncached") && exec.jitv2_uncached_views; + let frame = exec.jitv2_monitor_pfn(paddr)?; + let pfn = if uncached { frame | crate::cpu::jitv2::jitv2::UNCACHED_VIEW_PFN } else { frame }; + let page_base = crate::cpu::jitv2::jitv2::key_phys_base(pfn); let offset = (paddr & 0xFFF) as usize >> 2; - let sysad = exec.sysad.clone(); + let sysad = exec.jitv2_compile_bus(); let fr1 = (exec.core.cp0_status & crate::cpu::mips_core::STATUS_FR) != 0; let mut jit = exec.jitv2.lock(); match jit.page_for(pfn, page_base, sysad.as_ref(), fr1) { @@ -2976,7 +2984,7 @@ impl MipsCpu { // signal, not a crash risk. let page = unsafe { &*page_ptr }; writeln!(writer, " page: pfn={:#010x} phys_base={:#010x}", - page.pfn, page.pfn as u64 * crate::cpu::jitv2::PAGE_SIZE as u64).unwrap(); + page.pfn, page.phys_base()).unwrap(); } }; match jit.compile_queue.seal_queue_entries() { @@ -3012,7 +3020,7 @@ impl MipsCpu { // implementation. let path = actual_args.get(1).copied().unwrap_or("jitv2.html"); let jitv2_arc = exec.jitv2.clone(); - let bus = exec.sysad.clone(); + let bus = exec.jitv2_compile_bus(); crate::cpu::jitv2_html_j2wp::write_jitv2_html( &jitv2_arc, &bus, &exec.memory.clone(), &mut exec.jitv2_inline_analyzer, path, &mut *writer, )?; diff --git a/src/cpu/mips_exec_test.rs b/src/cpu/mips_exec_test.rs index 4098813ec..c3e9f13da 100644 --- a/src/cpu/mips_exec_test.rs +++ b/src/cpu/mips_exec_test.rs @@ -13,12 +13,18 @@ mod tests { // Mock Memory Interface pub struct MockMemory { pub data: Mutex>, + /// `write8`/`write16` calls as (width, address, value). + pub narrow_writes: Mutex>, + /// `write_narrow_from` calls as (width, address, source register). + pub narrow_from_writes: Mutex>, } impl MockMemory { pub fn new() -> Self { Self { data: Mutex::new(HashMap::new()), + narrow_writes: Mutex::new(Vec::new()), + narrow_from_writes: Mutex::new(Vec::new()), } } @@ -67,6 +73,7 @@ mod tests { } fn write8(&self, addr: u64, val: u8) -> u32 { + self.narrow_writes.lock().unwrap().push((1, addr, val as u64)); self.set_byte(addr as u64, val); BUS_OK } @@ -82,6 +89,7 @@ mod tests { fn write16(&self, addr: u64, val: u16) -> u32 { let aligned_addr = (addr & !1) as u64; + self.narrow_writes.lock().unwrap().push((2, aligned_addr, val as u64)); let bytes = val.to_be_bytes(); for i in 0..2 { self.set_byte(aligned_addr + i as u64, bytes[i]); @@ -111,6 +119,12 @@ mod tests { BUS_OK } + fn write_narrow_from(&self, addr: u64, size: usize, gpr: u64) -> u32 { + self.narrow_from_writes.lock().unwrap().push((size, addr, gpr)); + // Same architectural bytes as the default. + if size == 1 { self.write8(addr, gpr as u8) } else { self.write16(addr, gpr as u16) } + } + // This whole file predates jitv2 and no test here calls // `install_jit_hooks` — every `create_executor*` helper below // explicitly disables both the real dispatch gate @@ -575,6 +589,132 @@ mod tests { assert_eq!(mem.get_byte(0x100B), 0xEF); } + /// Fuel PROM memory diagnostics need the whole source register on the ECC + /// view, so every CPU hands an uncached SB or SH to + /// `BusDevice::write_narrow_from` with it. The default of that call stores + /// the architectural byte or halfword, which is all an ordinary device + /// sees, and the addressed lanes hold it either way. + #[test] + fn uncached_sb_sh_carry_the_whole_source_register() { + fn check>>() { + let mem = Arc::new(MockMemory::new()); + let bus: Arc = mem.clone(); + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))); + let mut exec = MipsExecutor::::new( + bus, + PassthroughTlb::default(), + &cfg, + ); + disable_jitv2(&mut exec); + + let base = 0x2000u64; + let source = 0xFEDC_BA98_7654_3210u64; + let original = 0xAABB_CCDD_EEFF_0011u64; + exec.core.write_gpr(2, base); + exec.core.write_gpr(1, source); + + for (size, op, offsets) in [(1usize, OP_SB, vec![0u64, 1, 2, 3, 4, 5, 6, 7]), + (2usize, OP_SH, vec![0u64, 2, 4, 6])] { + for offset in offsets { + mem.set_double(base, original); + mem.narrow_writes.lock().unwrap().clear(); + mem.narrow_from_writes.lock().unwrap().clear(); + let instr = make_i(op, 2, 1, offset as u16); + assert_eq!(exec.exec(instr), EXEC_COMPLETE); + + let addr = base + offset; + assert_eq!( + *mem.narrow_from_writes.lock().unwrap(), + vec![(size, addr, source)], + "SB/SH source register at byte offset {offset}", + ); + + let first = if size == 1 { source as u8 } else { (source as u16 >> 8) as u8 }; + let last = if size == 1 { first } else { source as u8 }; + assert_eq!(mem.get_byte(addr), first, "stored leading byte at offset {offset}"); + if size == 2 { + assert_eq!(mem.get_byte(addr + 1), last, "stored trailing byte at offset {offset}"); + } + let original_bytes = original.to_be_bytes(); + for lane in 0..8u64 { + if lane < offset || lane >= offset + size as u64 { + assert_eq!(mem.get_byte(base + lane), original_bytes[lane as usize], + "inactive byte lane {lane} at subword offset {offset}"); + } + } + } + } + + // A zero source register still takes the same path and clears + // only the addressed lanes. + exec.core.write_gpr(0, u64::MAX); + mem.set_double(base, original); + mem.narrow_from_writes.lock().unwrap().clear(); + exec.core.write_gpr(2, base + 7); + assert_eq!(exec.exec(make_i(OP_SB, 2, 0, 0)), EXEC_COMPLETE); + assert_eq!(mem.get_byte(base + 7), 0); + assert_eq!(mem.narrow_from_writes.lock().unwrap().len(), 1); + + mem.set_double(base, original); + exec.core.write_gpr(2, base + 6); + assert_eq!(exec.exec(make_i(OP_SH, 2, 0, 0)), EXEC_COMPLETE); + assert_eq!(mem.get_byte(base + 6), 0); + assert_eq!(mem.get_byte(base + 7), 0); + assert_eq!(mem.narrow_from_writes.lock().unwrap().len(), 2); + + mem.narrow_writes.lock().unwrap().clear(); + mem.narrow_from_writes.lock().unwrap().clear(); + assert_eq!(exec.debug_write(base + 1, source, 1, u64::MAX), EXEC_COMPLETE); + assert_eq!(*mem.narrow_writes.lock().unwrap(), vec![(1, base + 1, source as u8 as u64)]); + assert!(mem.narrow_from_writes.lock().unwrap().is_empty(), "debug writes keep the ordinary-width path"); + } + + check::(); + check::(); + check::(); + check::(); + } + + /// JIT store callouts carry the same source register as the interpreter. + #[test] + #[cfg(feature = "jitv2")] + fn jit_subword_hooks_carry_the_source_register() { + fn check>>() { + let mem = Arc::new(MockMemory::new()); + let bus: Arc = mem.clone(); + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))); + let mut exec = MipsExecutor::::new( + bus, + PassthroughTlb::default(), + &cfg, + ); + disable_jitv2(&mut exec); + exec.install_jit_hooks(); + + let base = 0x2000u64; + let source = 0xFEDC_BA98_7654_3210u64; + let write8 = exec.core.write8_fn; + let write16 = exec.core.write16_fn; + let core = &mut exec.core as *mut crate::cpu::mips_core::MipsCore; + // JIT callouts receive the core pointer biased by the shared + // ABI constant; core_from_arg removes this before recovery. + let ctx = core.cast::() + .wrapping_add(crate::cpu::jitv2::codegen::CALLOUT_CORE_BIAS as usize) + .cast(); + unsafe { + write8(ctx, base, source); + write16(ctx, base + 2, source); + } + assert_eq!( + *mem.narrow_from_writes.lock().unwrap(), + vec![(1, base, source), (2, base + 2, source)], + ); + } + + check::(); + check::(); + } + #[test] fn test_load_store_pc_advances() { // Regression test: every load/store dispatch-target handler must @@ -1227,6 +1367,64 @@ mod tests { 0xAAAAAAAA means a stale nanotlb slot survived the switch"); } + /// Two writes that would match together: on the R10000 family the second + /// removes the first and sets Status.TS, for TLBWI and TLBWR alike; on the + /// R4400 and R5000 both entries stay and TS stays clear. The compiled-code + /// path has nothing to add: jitv2 never admits TLBWI or TLBWR into a + /// region (see the `tlb_ops_are_never_admitted_but_eret_is` test), so + /// they always run this interpreter code. + fn overlapping_tlb_writes(r10k: bool) + where C: CpuModel + From> { + use crate::cpu::mips_tlb::{MipsTlb, Tlb}; + use crate::cpu::mips_core::STATUS_TS; + let mem: Arc = Arc::new(MockMemory::new()); + let mut exec = MipsExecutor::::new(mem, MipsTlb::default(), &MipsCpuConfig::indy()); + disable_jitv2(&mut exec); + let tlbwi = (OP_COP0 << 26) | (0x10 << 21) | 0x02; + let tlbwr = (OP_COP0 << 26) | (0x10 << 21) | 0x06; + let write = |exec: &mut MipsExecutor, instr: u32, index: u32, pfn: u64| { + exec.core.cp0_index = index; + exec.core.cp0_pagemask = 0; + exec.core.cp0_entryhi = (0x100 << 13) | 10; + exec.core.cp0_entrylo0 = (pfn << 6) | (3 << 3) | (1 << 2) | (1 << 1); + exec.core.cp0_entrylo1 = ((pfn + 1) << 6) | (3 << 3) | (1 << 2) | (1 << 1); + assert_eq!(exec.exec(instr), EXEC_COMPLETE); + }; + let first_survives = |exec: &MipsExecutor| exec.tlb.read(5).entry_hi == (0x100 << 13) | 10; + + write(&mut exec, tlbwi, 5, 0x50); + assert_eq!(exec.core.cp0_status & STATUS_TS, 0, "a write that overlaps nothing leaves TS clear"); + + write(&mut exec, tlbwi, 6, 0x60); + assert_eq!(exec.core.cp0_status & STATUS_TS != 0, r10k, "TLBWI TS"); + assert_eq!(!first_survives(&exec), r10k, "TLBWI removes the overlapped entry"); + if r10k { + assert_eq!(exec.tlb.read(5).page_mask, 0, "PageMask kept"); + } + + // Software clears TS by writing Status; the next overlap sets it again. + exec.core.cp0_status &= !STATUS_TS; + write(&mut exec, tlbwi, 5, 0x50); + exec.core.cp0_status &= !STATUS_TS; + exec.core.cp0_wired = 7; // Random stays out of 0..7, so it cannot pick entries 5 or 6 + write(&mut exec, tlbwr, 0, 0x70); + assert_eq!(exec.core.cp0_status & STATUS_TS != 0, r10k, "TLBWR TS"); + assert_eq!(!first_survives(&exec), r10k, "TLBWR removes the overlapped entry"); + } + + #[test] + fn overlapping_tlb_writes_follow_the_cpu_family() { + std::thread::Builder::new() + .stack_size(64 * 1024 * 1024) + .spawn(|| { + overlapping_tlb_writes::(true); + overlapping_tlb_writes::(true); + overlapping_tlb_writes::(false); + overlapping_tlb_writes::(false); + }) + .unwrap().join().unwrap(); + } + #[cfg(feature = "tlbcheck")] #[test] fn test_tlbwi_breaks_on_duplicate_entry() { @@ -3744,6 +3942,206 @@ mod tests { check::(0xF_FFFF_FFFF); } + /// FrameMask bits 15:0 clear EntryLo bits 33:18 as a TLB write takes them, + /// on the R10000 only (R10000 UM v2.0 p. 260). Physical bit 24 is EntryLo bit 18, + /// mask bit 0; the R4400 has no such register. + #[test] + fn framemask_clears_entrylo_bits_on_a_tlb_write() { + fn phys>>(framemask: u64) -> (u64, u64) { + const VA: u64 = 0x4000_0000; + let mut exec = map_page_masked::(Arc::new(MockMemory::new()), VA, 0x0300_5000, framemask); + (exec.core.read_cp0(21), exec.debug_translate(VA).phys) + } + use crate::cpu::mips_cache_shadow::R10000ShadowCache; + on_big_stack(|| { + // Mask bit 0 is written with bits above 15, which are ignored. + let all = 0xFFFF_0001; + assert_eq!(phys::(0), (0, 0x0300_5000)); + assert_eq!(phys::(all), (1, 0x0200_5000)); + assert_eq!(phys::(all), (0, 0x0300_5000)); + }); + } + + /// TLBWR masks as TLBWI does, and TLBR returns what the TLB holds: the + /// masked EntryLo, not the value that was written. + #[test] + fn framemask_applies_to_tlbwr_and_tlbr_reads_back_the_masked_entry() { + use crate::cpu::mips_cache_shadow::R10000ShadowCache; + // Write with `funct`, find the entry with TLBP, clear EntryLo0, TLBR, + // and return (physical address of the page, EntryLo0 after TLBR). + fn entry>>(funct: u32) -> (u64, u64) { + const VA: u64 = 0x4000_0000; + let mut exec = map_page_with::(Arc::new(MockMemory::new()), VA, 0x0300_5000, 0xFFFF_0001, funct); + let phys = exec.debug_translate(VA).phys; + for (rd, value) in [(10, VA), (2, 0)] { + exec.core.write_gpr(1, value); + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_DMTC0 << 21) | (1 << 16) | (rd << 11)), EXEC_COMPLETE); + } + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_TLB << 21) | FUNCT_TLBP), EXEC_COMPLETE); + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_TLB << 21) | FUNCT_TLBR), EXEC_COMPLETE); + (phys, exec.core.read_cp0(2)) + } + on_big_stack(|| { + // FrameMask bit 0 clears EntryLo bit 18, physical bit 24. + let unmasked = ((0x0300_5000u64 >> 12) << 6) | (3 << 3) | (1 << 2) | (1 << 1); + let masked = unmasked & !(1 << 18); + for funct in [FUNCT_TLBWI, FUNCT_TLBWR] { + assert_eq!(entry::(funct), (0x0200_5000, masked), "R10000 {funct:#x}"); + assert_eq!(entry::(funct), (0x0300_5000, unmasked), "R4400 {funct:#x}"); + } + }); + } + + /// The R10000 writes zeros into the PFN bits a page size leaves out of the + /// address, so TLBR returns them as zero (R10000 UM v2.0 pp. 299-300, TLBWI + /// and TLBWR); the later parts do the same as a model choice, since no + /// manual describes their TLB write. The R4400 keeps the whole PFN. The values are the PROM + /// diagnostic's patterns on 16 KB pages, where EntryLo bits 7:6 go unused. + /// Translation is the same on both: those bits never reach the address. + #[test] + fn tlb_write_clears_pfn_bits_below_the_page_size_on_the_r10000() { + use crate::cpu::mips_cache_shadow::R10000ShadowCache; + // Write (lo0, lo1) with PageMask 0x6000 through TLBWI or TLBWR, then + // find the entry with TLBP and TLBR it back. Returns (EntryLo0, EntryLo1, physical address of + // VA + 0x3000). + fn entry>>(funct: u32, lo0: u64, lo1: u64) -> (u64, u64, u64) { + const VA: u64 = 0x4000_0000; + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))); + let mut exec = Box::new(MipsExecutor::::new( + Arc::new(MockMemory::new()) as Arc, crate::cpu::mips_tlb::MipsTlb::default(), &cfg)); + disable_jitv2(&mut exec); + exec.install_status_cb(); + let status = (exec.core.cp0_status & !crate::cpu::mips_core::STATUS_ERL) | STATUS_KX; + let dmtc0 = |rd: u32| (OP_COP0 << 26) | (RS_DMTC0 << 21) | (1 << 16) | (rd << 11); + for (rd, value) in [(12, status as u64), (10, VA), (0, 0), (5, 0x6000), (2, lo0), (3, lo1)] { + exec.core.write_gpr(1, value); + assert_eq!(exec.exec(dmtc0(rd)), EXEC_COMPLETE); + } + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_TLB << 21) | funct), EXEC_COMPLETE); + let phys = exec.debug_translate(VA + 0x3000).phys; + // TLBWR picked the slot: find it with TLBP. + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_TLB << 21) | FUNCT_TLBP), EXEC_COMPLETE); + for rd in [2, 3] { + exec.core.write_gpr(1, 0); + assert_eq!(exec.exec(dmtc0(rd)), EXEC_COMPLETE); + } + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_TLB << 21) | FUNCT_TLBR), EXEC_COMPLETE); + (exec.core.read_cp0(2), exec.core.read_cp0(3), phys) + } + on_big_stack(|| { + for funct in [FUNCT_TLBWI, FUNCT_TLBWR] { + let (lo0, lo1) = (0xFFFF_FFFF_A5A5_A5A4, 0x5A5A_5A58); + // EntryLo 7:6 cleared; 63:62 and 33:6 otherwise kept. + assert_eq!(entry::(funct, lo0, lo1).0, 0xC000_0003_A5A5_A524, "R10000 {funct:#x}"); + assert_eq!(entry::(funct, lo0, lo1).1, 0x5A5A_5A18, "R10000 {funct:#x}"); + assert_eq!(entry::(funct, lo1, lo0).0, 0x5A5A_5A18, "R10000 {funct:#x}"); + // The R4400 keeps bits 29:0 as written. + assert_eq!(entry::(funct, lo0, lo1).0, 0x25A5_A5A4, "R4400 {funct:#x}"); + assert_eq!(entry::(funct, lo0, lo1).1, 0x1A5A_5A58, "R4400 {funct:#x}"); + + // A 16 KB page at 0x0300_4000 written with PFN bits for + // 0x0300_7000: both parts translate from the 16 KB frame. + let lo = ((0x0300_7000u64 >> 12) << 6) | (3 << 3) | (1 << 2) | (1 << 1); + let (r10k, _, r10k_pa) = entry::(funct, lo, 0); + let (r4k, _, r4k_pa) = entry::(funct, lo, 0); + assert_eq!((r10k, r10k_pa), (lo & !0xC0, 0x0300_7000), "R10000 {funct:#x}"); + assert_eq!((r4k, r4k_pa), (lo, 0x0300_7000), "R4400 {funct:#x}"); + } + }); + } + + /// On the R10000 family EntryLo holds a 28-bit PFN (33:6) and the uncached + /// attribute in 63:62 (R10000 UM v2.0 pp. 239-240); the R4400 and R5000 keep a + /// 24-bit PFN and nothing above it. The TLB carries the whole PFN. + #[test] + fn entrylo_covers_the_pfn_and_attribute_bits_of_the_r10000_family() { + fn readback>>() -> u64 { + let bus: Arc = Arc::new(MockMemory::new()); + let mut exec = MipsExecutor::::new(bus, PassthroughTlb::default(), &MipsCpuConfig::indy()); + disable_jitv2(&mut exec); + exec.core.write_gpr(1, u64::MAX); + // DMTC0 $1, EntryLo0 and EntryLo1; both read back the same field. + for reg in [2u32, 3] { + assert_eq!(exec.exec((0x10 << 26) | (5 << 21) | (1 << 16) | (reg << 11)), EXEC_COMPLETE); + } + let (lo0, lo1) = (exec.core.read_cp0(2), exec.core.read_cp0(3)); + assert_eq!(lo0, lo1); + lo0 + } + assert_eq!(readback::(), 0x3FFF_FFFF); + // Bits 61:34 are not PFN: they read back as zero, so only 33:32 reach physical 39:38. + assert_eq!(readback::(), 0xC000_0003_FFFF_FFFF); + + // The TLB carries the whole PFN: bits 33:32 are physical 39:38, and FrameMask + // is what keeps software flags there out of the address. + on_big_stack(|| { + const VA: u64 = 0x4000_0000; + let mut exec = map_page::( + Arc::new(MockMemory::new()), VA, 0xC0_2054_C000); + assert_eq!(exec.debug_translate(VA).phys, 0xC0_2054_C000); + // FrameMask 0xC000 is EntryLo 33:32: the page lands where its PFN says. + let mut exec = map_page_masked::( + Arc::new(MockMemory::new()), VA, 0xC0_2054_C000, 0xC000); + assert_eq!(exec.debug_translate(VA).phys, 0x2054_C000); + }); + } + + /// A mapped uncached page's attribute (EntryLo 63:62, carried to physical + /// bits 58:57) reaches the bus on a board that decodes them, and is folded + /// away on the Octane (40 bits), Indigo2 and O2 (32), which is what keeps + /// those machines unchanged. + #[test] + fn mapped_uncached_attribute_is_folded_by_the_boards_physical_mask() { + std::thread::Builder::new() + .stack_size(16 * 1024 * 1024) + .spawn(|| { + use crate::cpu::mips_tlb::MipsTlb; + const VA: u64 = 0x4000_0000; + + fn translate>>( + phys_mask: u64, alias: Option<(u64, u64, u64)>, pfn: u64, + ) -> u64 { + let bus: Arc = Arc::new(MockMemory::new()); + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))); + let mut exec = Box::new(MipsExecutor::::new( + bus, MipsTlb::default(), &cfg)); + disable_jitv2(&mut exec); + exec.install_status_cb(); + let dmtc0 = |exec: &mut Box>, rd: u32, value: u64| { + exec.core.write_gpr(1, value); + let op = (OP_COP0 << 26) | (RS_DMTC0 << 21) | (1 << 16) | (rd << 11); + assert_eq!(exec.exec(op), EXEC_COMPLETE); + }; + let status = (exec.core.cp0_status & !crate::cpu::mips_core::STATUS_ERL) | STATUS_KX; + dmtc0(&mut exec, 12, status as u64); + dmtc0(&mut exec, 10, VA); + dmtc0(&mut exec, 0, 0); + dmtc0(&mut exec, 5, 0); + // C=2 (uncached), valid, dirty, attribute 1 in bits 63:62. + dmtc0(&mut exec, 2, (1 << 62) | (pfn << 6) | (2 << 3) | (1 << 2) | (1 << 1)); + dmtc0(&mut exec, 3, 0); + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_TLB << 21) | FUNCT_TLBWI), EXEC_COMPLETE); + exec.phys_mask = phys_mask; + exec.phys_alias = alias; + exec.debug_translate(VA).phys + } + + type R10K = crate::cpu::mips_cache_shadow::R10000ShadowCache; + type R14K = crate::cpu::mips_cache_shadow::R12000ShadowCache; + const PA: u64 = 0x0d10_0000; + assert_eq!(translate::(u64::MAX, None, PA >> 12), (1 << 57) | PA, + "no mask: the attribute reaches the bus"); + for (name, mask) in [("Octane", (1u64 << 40) - 1), ("Indigo2/O2", u32::MAX as u64)] { + assert_eq!(translate::(mask, None, PA >> 12), PA, "{name} folds it away"); + } + assert_eq!(translate::(u64::MAX, None, PA >> 12), PA, + "a model without the attribute never stores it"); + }) + .unwrap() + .join() + .unwrap(); + } + /// The Indy, Indigo2 and O2 decode 32 physical bits, and IRIX relies on /// it: on the IP28 the kernel stack's wired entry carries PTE software /// bits as PA[34:33]. `phys_mask` must fold them away before the bus. @@ -5688,6 +6086,57 @@ mod tests { assert_ne!(st & EXEC_IS_XTLB_REFILL, 0, "true xkseg is unchanged: XTLB"); } + /// The R10000 picks the refill vector by the address space a missing + /// address lies in and that space's Status bit: user addresses follow + /// UX, xsseg SX, kernel and compat supervisor addresses KX (R10000 UM + /// v2.0 pp. 333-334). The R4400 picks by the current mode. A 64-bit + /// kernel (KX=1, UX=0) missing on kuseg therefore takes the 32-bit refill + /// vector on the R10000 and XTLB on the R4400; the PROM's utlb + /// diagnostic waits at the 32-bit vector for exactly that miss. + #[test] + fn r10000_refill_vector_follows_the_address_space() { + use crate::cpu::mips_cache_shadow::R10000ShadowCache; + use crate::cpu::mips_core::{STATUS_ERL, STATUS_EXL, STATUS_SX, STATUS_UX}; + use crate::cpu::mips_exec::EXEC_IS_XTLB_REFILL; + // Kernel mode with `xbits` set in Status, empty TLB; is a read of + // `va` a refill miss through XTLB? + fn xrefill>>(xbits: u32, va: u64) -> bool { + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))); + let mut exec = Box::new(MipsExecutor::::new( + Arc::new(MockMemory::new()), crate::cpu::mips_tlb::MipsTlb::default(), &cfg)); + disable_jitv2(&mut exec); + exec.install_status_cb(); + let status = (exec.core.cp0_status & !(STATUS_ERL | STATUS_EXL | STATUS_KX | STATUS_SX | STATUS_UX | 0x18)) | xbits; + exec.core.write_gpr(1, status as u64); + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_DMTC0 << 21) | (1 << 16) | (12 << 11)), EXEC_COMPLETE); + let st = exec.read_data::<4>(va).expect_err("empty TLB must miss"); + assert_ne!(st & EXEC_IS_TLB_REFILL, 0, "{va:#x} should be a refill miss"); + st & EXEC_IS_XTLB_REFILL != 0 + } + const KUSEG: u64 = 0; + const XSSEG: u64 = 0x4000_0000_0000_0000; + const CKSSEG: u64 = 0xFFFF_FFFF_C000_0000; + const XKSEG: u64 = 0xC000_0000_0000_0000; + on_big_stack(|| { + type R10k = R10000ShadowCache; + // 64-bit kernel, 32-bit user space: kuseg refills through 0x000. + assert!(!xrefill::(STATUS_KX, KUSEG)); + assert!(xrefill::(STATUS_KX, KUSEG), "R4400 follows the mode"); + assert!(xrefill::(STATUS_KX | STATUS_UX, KUSEG)); + // xsseg follows SX, compat ksseg and xkseg follow KX. + assert!(!xrefill::(STATUS_KX, XSSEG)); + assert!(xrefill::(STATUS_KX | STATUS_SX, XSSEG)); + assert!(xrefill::(STATUS_KX, CKSSEG)); + assert!(xrefill::(STATUS_KX, XKSEG)); + // 32-bit kernel: kseg3-side addresses use 0x000, kuseg follows UX. + assert!(!xrefill::(0, CKSSEG)); + assert!(!xrefill::(STATUS_UX, CKSSEG)); + assert!(!xrefill::(0, KUSEG)); + assert!(xrefill::(STATUS_UX, KUSEG)); + assert!(!xrefill::(STATUS_UX, KUSEG), "R4400 follows the mode"); + }); + } + #[test] fn test_xtlb_compare_width_disambiguates_region_field() { std::thread::Builder::new() @@ -6418,4 +6867,282 @@ mod tests { } assert_eq!(outcome[0], outcome[1], "an explanation must not change what the guest sees"); } + + /// Run `f` on a thread with room for a `MipsTlb`, which is large. + fn on_big_stack(f: impl FnOnce() + Send + 'static) { + std::thread::Builder::new().stack_size(16 * 1024 * 1024).spawn(f).unwrap().join().unwrap(); + } + + /// A 64-bit kernel executor with one TLB entry, written through CP0 and + /// TLBWI, that maps the page at `va` to `phys` (cached, valid, dirty). + fn map_page>>( + bus: Arc, va: u64, phys: u64, + ) -> Box> { + map_page_masked::(bus, va, phys, 0) + } + + /// `map_page` with CP0 FrameMask (register 21) written before the TLB write. + fn map_page_masked>>( + bus: Arc, va: u64, phys: u64, framemask: u64, + ) -> Box> { + map_page_with::(bus, va, phys, framemask, FUNCT_TLBWI) + } + + /// `map_page_masked`, writing the entry with `funct` (TLBWI or TLBWR). + fn map_page_with>>( + bus: Arc, va: u64, phys: u64, framemask: u64, funct: u32, + ) -> Box> { + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))); + let mut exec = Box::new(MipsExecutor::new(bus, crate::cpu::mips_tlb::MipsTlb::default(), &cfg)); + disable_jitv2(&mut exec); + // MTC0 must refresh the boxed executor's translation/accessor state. + exec.install_status_cb(); + let status = (exec.core.cp0_status & !crate::cpu::mips_core::STATUS_ERL) | STATUS_KX; + let entrylo = ((phys >> 12) << 6) | (3 << 3) | (1 << 2) | (1 << 1); + for (rd, value) in [(12, status as u64), (10, va), (0, 0), (5, 0), (21, framemask), (2, entrylo), (3, 0)] { + exec.core.write_gpr(1, value); + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_DMTC0 << 21) | (1 << 16) | (rd << 11)), EXEC_COMPLETE); + } + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_TLB << 21) | funct), EXEC_COMPLETE); + exec + } + + /// EntryHi and the TLB keep every implemented VPN2 bit: 44 virtual-address + /// bits on the R10000, 40 on the R4400. An IRIX n64 stack can sit + /// above the 40-bit limit. + #[test] + fn cp0_entryhi_and_tlb_match_full_xuseg_va() { + on_big_stack(|| { + const VA: u64 = 0x0000_0123_4000_0000; + fn check>>(full_va: bool) { + let mem = Arc::new(MockMemory::new()); + mem.set_word(0x5_0000, 0xdead_beef); + let mut exec = map_page::(mem, VA, 0x5_0000); + let result = exec.read_data::<4>(VA); + if full_va { + assert_eq!(exec.core.cp0_entryhi, VA); + assert_eq!(result, Ok(0xdead_beef)); + } else { + assert_eq!(exec.core.cp0_entryhi, VA & 0xC000_00FF_FFFF_E0FF); + assert_eq!((result.unwrap_err() >> 2) & 0x1f, EXC_ADEL); + } + } + check::(false); + check::(true); + }); + } + + /// A bus where nothing answers: every read and write is refused (the + /// trait's defaults). + struct RefusingBus; + impl BusDevice for RefusingBus {} + + /// R10000: a store the system refuses leaves the processor as a write + /// request, which has no response, so it completes (R10000 UM pp. 104, + /// 316). An uncached SC succeeding is a model choice. A load keeps its + /// Data Bus Error, and so does every access on the R5000. SWL goes through + /// the masked-store path, a separate site. + #[test] + fn refused_uncached_store_completes_only_on_the_r10000() { + fn run>>(op: u32) -> (u32, u64) { + let bus: Arc = Arc::new(RefusingBus); + let mut exec = MipsExecutor::::new(bus, PassthroughTlb::default(), &MipsCpuConfig::indy()); + disable_jitv2(&mut exec); + exec.cache.set_llbit(true); + exec.core.write_gpr(4, 0xFFFF_FFFF_A100_0000); // kseg1: uncached + exec.core.write_gpr(6, 0x1234_5678); + let st = exec.exec(make_i(op, 4, 6, 4)); + if st == EXEC_COMPLETE { + assert_eq!(exec.core.cp0_status & crate::cpu::mips_core::STATUS_EXL, 0, "no exception taken"); + assert_eq!(exec.exec(make_i(OP_ADDIU, 0, 5, 7)), EXEC_COMPLETE); + assert_eq!(exec.core.read_gpr(5), 7, "the next instruction runs"); + } + (st, exec.core.read_gpr(6)) + } + use crate::traits::BUS_ERR; + type R10K = crate::cpu::mips_cache_v2::R10000Cache; + type R10KShadow = crate::cpu::mips_cache_shadow::R10000ShadowCache; + type R5K = crate::cpu::mips_cache_shadow::R5000ShadowCache; + for (op, name) in [(OP_SW, "SW"), (OP_SWL, "SWL")] { + assert_eq!(run::(op).0, EXEC_COMPLETE, "R10000 {name}"); + assert_eq!(run::(op).0, EXEC_COMPLETE, "R10000 shadow {name}"); + assert_eq!(run::(op).0, BUS_ERR, "R5000 {name}"); + } + assert_eq!(run::(OP_SC), (EXEC_COMPLETE, 1), "SC succeeds"); + assert_eq!(run::(OP_SC).0, BUS_ERR, "R5000 SC"); + assert_eq!(run::(OP_LW).0, BUS_ERR, "a refused load still faults"); + } + + #[test] + fn refused_store_reports_are_sparse() { + let shown: Vec = (1..=40).filter(|&n| crate::cpu::mips_exec::report_refused_store(n)).collect(); + assert_eq!(shown, [1, 2, 3, 4, 8, 16, 32]); + } + + /// The O2's no-ECC alias of RAM at 0x8000_0000 is matched on the address the + /// board decodes (32 bits), so a PTE software bit above the PFN cannot make + /// a mapped page miss the alias window. + #[test] + fn phys_alias_is_matched_after_the_physical_mask() { + std::thread::Builder::new() + .stack_size(16 * 1024 * 1024) + .spawn(|| { + use crate::cpu::mips_tlb::MipsTlb; + let (mut exec, _) = create_executor_with_tlb(MipsTlb::default()); + exec.core.cp0_status &= !crate::cpu::mips_core::STATUS_ERL; + exec.phys_mask = u32::MAX as u64; + exec.phys_alias = Some((0x8000_0000, 0x1000_0000, 0)); + // PA 0x1_8000_0000: bit 32 is above what the board decodes. + exec.core.cp0_index = 0; + exec.core.cp0_pagemask = 0; + exec.core.cp0_entryhi = 0x4000_0000; + exec.core.cp0_entrylo0 = (0x18_0000 << 6) | (3 << 3) | (1 << 2) | (1 << 1); + exec.core.cp0_entrylo1 = 0; + assert_eq!(exec.exec((OP_COP0 << 26) | (0x10 << 21) | 0x02), EXEC_COMPLETE); + assert_eq!(exec.debug_translate(0x4000_0000).phys, 0); + }) + .unwrap().join().unwrap(); + } + + /// A partial LWR keeps bits 63:32 of the destination on the R10000 family + /// while 64-bit operations are enabled for the current mode, and sign-extends + /// on the R4400 and R5000 and on the family when they are off (a 32-bit user + /// process, UX=0); a complete word sign-extends on all. + #[test] + fn partial_lwr_keeps_the_upper_word_on_the_r10000_family() { + use crate::cpu::mips_core::{STATUS_KX, STATUS_SX, STATUS_UX}; + const USER: u32 = 2 << 3; + const SUPER: u32 = 1 << 3; + fn check>>(model_keeps_upper: bool) { + let mem = Arc::new(MockMemory::new()); + mem.set_word(0x2000, 0x89ab_cdef); + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))); + let mut exec = Box::new(MipsExecutor::::new(mem, PassthroughTlb::default(), &cfg)); + disable_jitv2(&mut exec); + exec.core.write_gpr(1, 0x2000); + let old = 0x8123_4567_a1b2_c3d4u64; + // (Status, 64-bit operations on): each mode's own bit decides, + // whatever the other two hold. + for (status, ops64, mode) in [ + (STATUS_KX, true, "kernel KX=1"), + (0, false, "kernel KX=0"), + (USER | STATUS_UX, true, "user UX=1"), + (USER | STATUS_KX | STATUS_SX, false, "user UX=0"), + (SUPER | STATUS_SX, true, "supervisor SX=1"), + (SUPER | STATUS_KX | STATUS_UX, false, "supervisor SX=0"), + ] { + exec.core.cp0_status = status; + for (offset, word) in [0xa1b2_c389u32, 0xa1b2_89ab, 0xa189_abcd, 0x89ab_cdef].into_iter().enumerate() { + exec.core.write_gpr(2, old); + assert_eq!(exec.exec(make_i(OP_LWR, 1, 2, offset as u16)), EXEC_COMPLETE); + let expected = if model_keeps_upper && ops64 && offset != 3 { + (old & 0xffff_ffff_0000_0000) | word as u64 + } else { + word as i32 as i64 as u64 + }; + assert_eq!(exec.core.read_gpr(2), expected, "{} LWR offset {offset}, {mode}", C::NAME); + } + } + } + check::(false); + check::(false); + check::(true); + check::(true); + check::(true); + } + + /// FrameMask and the wide EntryLo belong to the whole R10000 family, not to + /// the partial LWR alone. This checks the executor's masks against the + /// family constant the gate reads; the R12000 and R16000 are covered below. + #[test] + fn framemask_and_entrylo_widths_follow_the_family_constant() { + fn check>>() { + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))); + let exec = Box::new(MipsExecutor::::new(Arc::new(MockMemory::new()), PassthroughTlb::default(), &cfg)); + let (framemask, entrylo) = if C::R10K_FAMILY { (0xFFFF, 0xC000_0003_FFFF_FFFF) } else { (0, 0x3FFF_FFFF) }; + assert_eq!(exec.core.framemask_mask, framemask, "{} FrameMask", C::NAME); + assert_eq!(exec.core.entrylo_mask, entrylo, "{} EntryLo", C::NAME); + } + use crate::cpu::mips_cache_shadow::{R10000ShadowCache, R4400ShadowCache, R5000ShadowCache}; + assert!(R10000ShadowCache::R10K_FAMILY); + assert!(!R4400ShadowCache::R10K_FAMILY && !R5000ShadowCache::R10K_FAMILY); + check::(); + check::(); + check::(); + } + + /// Config reports each part's secondary cache: SS (bits 18:16) is + /// log2(secondary size / 512 KB) and SB (bit 13) is set for 128-byte lines + /// (NEC U10278EJ4V0UM p.230). The primary caches read 32 KB. + #[test] + fn config_reports_the_secondary_cache_of_each_family_part() { + use crate::cpu::mips_cache_shadow::*; + fn config>>() -> u64 { + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))); + let mut exec = Box::new(MipsExecutor::::new( + Arc::new(MockMemory::new()), PassthroughTlb::default(), &cfg)); + disable_jitv2(&mut exec); + exec.core.read_cp0(16) + } + // The executor is large; build it on a thread with room. + std::thread::Builder::new().stack_size(16 * 1024 * 1024).spawn(|| { + for (name, c, ss, sb) in [ + ("R10000", config::(), 1, 1), + ("R12000", config::(), 2, 1), + ("R12000 O2", config::(), 2, 0), + ] { + assert_eq!(c >> 26, 0b011_011, "{name}: IC and DC read 32 KB"); + assert_eq!(c & 0x03FF_FFFF, (ss << 16) | (sb << 13) | 0x8000 | 2, "{name}: Config {c:#x}"); + } + // The R16000 presents its part's whole reset word over the low 25 + // bits: SS 3 and SB 1 as for the generic family, plus the clock + // fields. The 700 MHz part is the same type with its word set at + // build time. + let c = config::(); + assert_eq!(c >> 26, 0b011_011, "R16000: IC and DC read 32 KB"); + assert_eq!(c & CONFIG0_RESET_MASK as u64, 0x009b_af85, "R16000: Config {c:#x}"); + assert_eq!((c >> 16) & 7, 3, "R16000: 4 MB secondary cache"); + assert_eq!((c >> 13) & 1, 1, "R16000: 128-byte lines"); + assert_eq!(R16000_MHZ700_CONFIG0 ^ 0x009b_af85, 1 << 9, "the 700 MHz word differs in EC only"); + }).unwrap().join().unwrap(); + } + + /// The R12000 and R16000 keep the 16-bit FrameMask, the 64-bit-family + /// EntryLo mask and the partial-LWR upper word of the R10000. A TLB write on + /// an R12000 with FrameMask set clears EntryLo bit 18, as on the R10000. + #[test] + fn framemask_and_entrylo_widths_hold_on_the_r12000_and_r16000() { + use crate::cpu::mips_cache_shadow::*; + fn masks>>() -> (u32, u64) { + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip22_bus::memory_map(false))); + let exec = Box::new(MipsExecutor::::new(Arc::new(MockMemory::new()), PassthroughTlb::default(), &cfg)); + (exec.core.framemask_mask, exec.core.entrylo_mask) + } + // Write the entry with FrameMask set, find it with TLBP, clear EntryLo0 + // and TLBR it: (FrameMask, EntryLo0 as the TLB holds it, physical page). + fn written>>() -> (u64, u64, u64) { + const VA: u64 = 0x4000_0000; + let mut exec = map_page_masked::(Arc::new(MockMemory::new()), VA, 0x0300_5000, 0xFFFF_0001); + let phys = exec.debug_translate(VA).phys; + for (rd, value) in [(10, VA), (2, 0)] { + exec.core.write_gpr(1, value); + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_DMTC0 << 21) | (1 << 16) | (rd << 11)), EXEC_COMPLETE); + } + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_TLB << 21) | FUNCT_TLBP), EXEC_COMPLETE); + assert_eq!(exec.exec((OP_COP0 << 26) | (RS_TLB << 21) | FUNCT_TLBR), EXEC_COMPLETE); + (exec.core.read_cp0(21), exec.core.read_cp0(2), phys) + } + on_big_stack(|| { + const ENTRYLO: u64 = 0xC000_0003_FFFF_FFFF; + assert!(R12000ShadowCache::R10K_FAMILY && R12000O2ShadowCache::R10K_FAMILY && R16000ShadowCache::R10K_FAMILY); + assert!(R12000ShadowCache::PARTIAL_LWR_KEEPS_UPPER && R12000O2ShadowCache::PARTIAL_LWR_KEEPS_UPPER); + assert!(R16000ShadowCache::PARTIAL_LWR_KEEPS_UPPER); + assert_eq!(masks::(), (0xFFFF, ENTRYLO), "R12000"); + assert_eq!(masks::(), (0xFFFF, ENTRYLO), "R12000 O2"); + assert_eq!(masks::(), (0xFFFF, ENTRYLO), "R16000"); + // Physical bit 24 is EntryLo bit 18, FrameMask bit 0. + let masked = (((0x0300_5000u64 >> 12) << 6) | (3 << 3) | (1 << 2) | (1 << 1)) & !(1 << 18); + assert_eq!(written::(), (1, masked, 0x0200_5000), "R12000"); + }); + } } diff --git a/src/cpu/mips_tlb.rs b/src/cpu/mips_tlb.rs index 05680e0c6..7d8d33a61 100644 --- a/src/cpu/mips_tlb.rs +++ b/src/cpu/mips_tlb.rs @@ -147,6 +147,23 @@ pub enum AccessType { Debug = 3, } +/// The physical base a TLB entry's EntryLo maps. A model that keeps the +/// R10000 family's uncached attribute (EntryLo bits 63:62, see +/// `CpuModel::R10K_FAMILY`) has it carried to physical bits 58:57 +/// for an uncached page, where an unmapped xkphys access carries it; other +/// models never store those bits. For a mapped reference the R10000 takes the +/// two-bit uncached attribute from the TLB's uncached-attribute field, loaded +/// from EntryLo0/1 bits 63:62 (NEC U10278EJ4V0UM, VR10000/VR12000 user's +/// manual, 4th edition, section 11.3; R10000 User's Manual 007-2490-001). +/// +/// Whether the page is uncached is the caller's decode of C through the model's +/// table (`MipsTlb::c_attr`), not another decode here. +#[inline] +fn pfn_base(lo: u64, pfn_mask: u64, uncached: bool) -> u64 { + let attribute = if uncached { ((lo >> 62) & 3) << 57 } else { 0 }; + ((lo << 6) & pfn_mask) | attribute +} + /// TLB Entry structure matching R4000 TLB format #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub struct TlbEntry { @@ -295,6 +312,11 @@ pub enum TlbResult { }, } +/// VPN2 bits of a virtual address `bits` wide: bit 13 up to `bits - 1`. +pub const fn vpn2_mask(bits: u32) -> u64 { + ((1u64 << bits) - 1) & !0x1fff +} + /// TLB interface for MIPS address translation /// /// The TLB translates virtual page numbers to physical page numbers. @@ -351,6 +373,19 @@ pub trait Tlb { fn power_on(&mut self) {} /// Install the CPU model's meaning for each EntryLo C value. fn set_cache_attrs(&mut self, _table: [CacheAttr; 8]) {} + /// Set the implemented VPN width before installing entries. + fn set_va_bits(&mut self, _bits: u32) {} + /// Make a TLB write invalidate every other entry it would match together + /// with, as the R10000 and R12000 do (NEC U10278EJ4V0UM, Status register + /// TS bit, p.223, and "Avoiding TLB Conflict", p.300): they do not shut + /// down on a multiple match. The R14000 and R16000 are given the same + /// rule as a model choice, matching QEMU's r10k_invalidate_conflicts(); + /// no manual for them was read. Off for the R4400 and R5000, whose + /// behaviour on a multiple match is not modelled. + fn set_conflict_invalidation(&mut self, _on: bool) {} + /// Whether a write since the last call invalidated another entry. The + /// executor reports it in Status.TS. + fn take_write_conflict(&mut self) -> bool { false } fn save_state(&self) -> toml::Value { toml::Value::Table(Default::default()) } fn load_state(&mut self, _v: &toml::Value) -> Result<(), String> { Ok(()) } @@ -534,6 +569,7 @@ pub struct MipsTlb { /// Live JTLB entries for the CPU model in use (<= TLB_MAX_ENTRIES). num_entries: usize, + vpn_mask64: u64, /// What each EntryLo C value means on the CPU model in use; the /// executor installs its table with `set_cache_attrs`. c_attr: [CacheAttr; 8], @@ -552,9 +588,56 @@ pub struct MipsTlb { vmap_touched: std::collections::HashSet, #[cfg(feature = "tlbstats")] pub stats: TlbStats, + /// R10000 family: a write invalidates the entries it overlaps. See + /// [`Tlb::set_conflict_invalidation`]. + invalidate_on_write: bool, + /// Entries written since power-on; a never-written slot cannot conflict. + written: u64, + /// A write invalidated another entry since `take_write_conflict`. + write_conflict: bool, } impl MipsTlb { + /// The R10000 family never lets a write create a multiple match: every + /// other entry the incoming one overlaps, wired or not, is taken out of + /// matching before the write lands, and Status.TS records that it + /// happened (see [`Tlb::set_conflict_invalidation`]). Overlap is + /// a lookup that could match both: the region and page number agree + /// under the wider of the two page masks, and the ASIDs agree or either + /// entry is global. Entries that are invalid count: they match and + /// fault. The entry reads back as a never-written slot does, zeros + /// except for its PageMask. Runs on the write path only; a lookup is unchanged. + fn invalidate_overlapped(&mut self, index: usize, new: &TlbEntry) { + let mut live = self.written & !(1u64 << index); + while live != 0 { + let i = live.trailing_zeros() as usize; + live &= live - 1; + if i >= self.num_entries { + continue; + } + let other = &self.entries[i]; + if !(new.is_global() || other.is_global() || new.asid() == other.asid()) { + continue; + } + let both = new.page_mask | other.page_mask | 0x1FFF; + if (new.entry_hi ^ other.entry_hi) & !both & self.vpn_mask64 != 0 { + continue; + } + let old_range = Self::vmap_range(other); + // TLBR of the entry reads zeros for EntryHi and EntryLo but still + // loads PageMask: software that sets PageMask once and relies on a + // read-back restoring it would otherwise write its next entry + // with a 4 KiB mask. + let mut gone = TlbEntry::new(); + gone.page_mask = other.page_mask; + self.entries[i] = gone; + self.shadow[i] = ShadowEntry::invalid(); + self.written &= !(1u64 << i); + self.vmap_vacate(i, old_range); + self.write_conflict = true; + } + } + pub fn new(num_entries: usize) -> Self { assert!(num_entries > 1 && num_entries <= TLB_MAX_ENTRIES, "MipsTlb supports 2..={} entries, got {}", TLB_MAX_ENTRIES, num_entries); @@ -564,12 +647,16 @@ impl MipsTlb { mru_head: [0u8; MRU_LISTS], mru_next: [[MRU_NONE; TLB_MAX_ENTRIES]; MRU_LISTS], num_entries, + vpn_mask64: 0xC000_00FF_FFFF_E000, c_attr: DEFAULT_C_ATTR, vmap: [VMAP_MISS; VMAP_SIZE], #[cfg(feature = "tlbcheck")] vmap_touched: std::collections::HashSet::new(), #[cfg(feature = "tlbstats")] stats: TlbStats::default(), + invalidate_on_write: false, + written: 0, + write_conflict: false, }; for list in 0..MRU_LISTS { tlb.mru_head[list] = 0; @@ -669,6 +756,15 @@ impl MipsTlb { /// step 3 that it now also aliases. #[inline] fn vmap_replace(&mut self, index: usize, old_range: (usize, usize)) { + self.vmap_vacate(index, old_range); + let (new_start, new_count) = Self::vmap_range(&self.entries[index]); + self.vmap_fill_range(index, new_start, new_count); + } + + /// Steps 1 to 3 of [`Self::vmap_replace`]: take `index` out of the map + /// and let the other entries reclaim what it covered. + #[inline] + fn vmap_vacate(&mut self, index: usize, old_range: (usize, usize)) { let (old_start, old_count) = old_range; self.vmap_clear(old_start, old_count); let old_end = old_start + old_count; @@ -685,8 +781,6 @@ impl MipsTlb { self.vmap_fill_range(i, lo, hi - lo); } } - let (new_start, new_count) = Self::vmap_range(&self.entries[index]); - self.vmap_fill_range(index, new_start, new_count); } /// Full internal consistency check (feature = "tlbcheck"). @@ -900,6 +994,16 @@ impl Default for MipsTlb { } } +impl MipsTlb { + /// `pfn_base` for both halves of an entry, deciding "uncached" with the + /// model's C table. + #[inline] + fn pfn_bases(&self, lo: &[u64; 2], pfn_mask: u64) -> [u64; 2] { + let uncached = |lo: u64| self.c_attr[((lo >> 3) & 7) as usize] == CacheAttr::Uncached; + [pfn_base(lo[0], pfn_mask, uncached(lo[0])), pfn_base(lo[1], pfn_mask, uncached(lo[1]))] + } +} + impl Tlb for MipsTlb { #[inline] fn translate(&mut self, virt_addr: u64, asid: u8, access_type: AccessType) -> TlbResult { @@ -1065,14 +1169,15 @@ impl Tlb for MipsTlb { entry.selector_bit_shift = (mask.trailing_ones() - 1) as u8; entry.vcmp32 = !mask & 0x0000_0000_FFFF_E000; entry.vpn_hi32 = entry.entry_hi & entry.vcmp32; - entry.vcmp64 = !mask & 0xC000_00FF_FFFF_E000; + entry.vcmp64 = !mask & self.vpn_mask64; entry.vpn_hi64 = entry.entry_hi & entry.vcmp64; entry.offset_mask = (1u64 << entry.selector_bit_shift) - 1; let pfn_mask = 0xFF_FFFF_FFFF_000 & !entry.offset_mask; - entry.pfn_base = [ - (entry.entry_lo[0] << 6) & pfn_mask, - (entry.entry_lo[1] << 6) & pfn_mask, - ]; + entry.pfn_base = self.pfn_bases(&entry.entry_lo, pfn_mask); + if self.invalidate_on_write { + self.invalidate_overlapped(index, &entry); + } + self.written |= 1 << index; let old_range = Self::vmap_range(&self.entries[index]); self.entries[index] = entry; self.shadow[index] = ShadowEntry::from_entry(&entry, &self.c_attr); @@ -1194,9 +1299,25 @@ impl Tlb for MipsTlb { output } + fn set_conflict_invalidation(&mut self, on: bool) { + self.invalidate_on_write = on; + } + + fn take_write_conflict(&mut self) -> bool { + std::mem::take(&mut self.write_conflict) + } + + fn set_va_bits(&mut self, bits: u32) { + self.vpn_mask64 = 0xC000_0000_0000_0000 | vpn2_mask(bits); + } + fn set_cache_attrs(&mut self, table: [CacheAttr; 8]) { self.c_attr = table; for i in 0..self.num_entries { + // pfn_base carries the attribute for pages the new table decodes + // as uncached, so it follows the table. + let pfn_mask = 0xFF_FFFF_FFFF_000 & !self.entries[i].offset_mask; + self.entries[i].pfn_base = self.pfn_bases(&self.entries[i].entry_lo, pfn_mask); self.shadow[i] = ShadowEntry::from_entry(&self.entries[i], &self.c_attr); } } @@ -1212,6 +1333,8 @@ impl Tlb for MipsTlb { self.mru_next[list][self.num_entries - 1] = MRU_NONE; } self.vmap.fill(VMAP_MISS); + self.written = 0; + self.write_conflict = false; #[cfg(feature = "tlbcheck")] { self.vmap_touched.clear(); } } @@ -1237,7 +1360,7 @@ impl Tlb for MipsTlb { let entry_hi = words[1]; let mask = page_mask | 0x1FFF; let vcmp32 = !mask & 0x0000_0000_FFFF_E000; - let vcmp64 = !mask & 0xC000_00FF_FFFF_E000; + let vcmp64 = !mask & self.vpn_mask64; let shift = (mask.trailing_ones() - 1) as u8; let offset_mask = (1u64 << shift) - 1; let lo = [words[2], words[3]]; @@ -1252,7 +1375,7 @@ impl Tlb for MipsTlb { vcmp64, vpn_hi64: entry_hi & vcmp64, offset_mask, - pfn_base: [(lo[0] << 6) & pfn_mask, (lo[1] << 6) & pfn_mask], + pfn_base: self.pfn_bases(&lo, pfn_mask), }; } } @@ -1268,6 +1391,7 @@ impl Tlb for MipsTlb { self.mru_next[list][self.num_entries - 1] = MRU_NONE; } self.vmap.fill(VMAP_MISS); + self.written = !0u64 >> (64 - self.num_entries); for i in 0..self.num_entries { let (start, count) = Self::vmap_range(&self.entries[i]); self.vmap_fill_range(i, start, count); diff --git a/src/cpu/mips_tlb_test.rs b/src/cpu/mips_tlb_test.rs index 4ebbaeda1..d89b79b83 100644 --- a/src/cpu/mips_tlb_test.rs +++ b/src/cpu/mips_tlb_test.rs @@ -4,6 +4,140 @@ mod tests { use crate::cpu::mips_tlb::*; use crate::cpu::mips_exec::CacheAttr; + /// Region 3 base for the overlap tests below; the frame is arbitrary. + const BASE: u64 = 0xc000_0000_0000_0000; + + fn entry(mask: u64, hi: u64, lo: [u64; 2]) -> TlbEntry { + let mut e = TlbEntry::new(); + e.page_mask = mask; + e.entry_hi = hi; + e.entry_lo = lo; + e + } + + /// CPU 1 of a two-processor Tezro as its TLB looks after the firmware's + /// flush (every entry written invalid: region 3, page number 0, ASID 0, + /// not global, 2 MiB pages) and the kernel's write of the valid global + /// 16 MiB entry at index 1 over the same page number. The firmware + /// entries that overlap it are true duplicates: every compared bit agrees. + fn flushed_then_kernel_mapped(invalidate: bool) -> MipsTlb { + let mut tlb = MipsTlb::new(64); + tlb.set_conflict_invalidation(invalidate); + for index in (0..64).rev() { + tlb.write(index, entry(0x1fe000, BASE, [0; 2])); + } + // PFN 0x40, cacheable, dirty, valid, global, both halves. + tlb.write(1, entry(0x1ffe000, BASE | 0x1000, [(0x40 << 6) | (5 << 3) | 0x7; 2])); + tlb + } + + /// An entry elsewhere, as the kernel writes at index 0 next (wired). + fn write_wired_elsewhere(tlb: &mut MipsTlb) { + tlb.write(0, entry(0x6000, 0xc000_0fff_ffff_9000, [1, 0x2e42f])); + } + + /// Writing the valid entry takes the overlapped firmware entries out of + /// matching, as the R10000 family does, so the fetch the kernel makes + /// next is answered by the valid entry however the lookup map was + /// rebuilt in between. Without the rule the highest-numbered firmware + /// entry answers, as invalid. + #[test] + fn a_write_invalidates_the_entries_it_overlaps() { + std::thread::Builder::new().stack_size(64 << 20).spawn(|| { + let va = BASE + 0x1234; + let mut off = flushed_then_kernel_mapped(false); + write_wired_elsewhere(&mut off); + assert!(matches!(off.translate::<1>(va, 0, AccessType::Fetch), TlbResult::Invalid { .. }), + "without the rule, the duplicate answers"); + assert!(!off.take_write_conflict()); + assert_eq!(off.read(2).entry_hi, BASE, "without the rule, the duplicate stays"); + + let mut tlb = flushed_then_kernel_mapped(true); + assert!(tlb.take_write_conflict(), "the writes overlapped entries"); + write_wired_elsewhere(&mut tlb); + match tlb.translate::<1>(va, 0, AccessType::Fetch) { + TlbResult::Hit { phys_addr, .. } => assert_eq!(phys_addr, 0x1234), + other => panic!("expected the valid entry to answer, got {other:?}"), + } + assert_eq!(tlb.read(2).entry_hi, 0); + assert_eq!(tlb.read(1).page_mask, 0x1ffe000); + // An entry in another region overlaps nothing and removes nothing. + tlb.write(5, entry(0x1fe000, 0x8000_0000_0000_0000, [0; 2])); + assert!(!tlb.take_write_conflict()); + assert!(matches!(tlb.translate::<1>(va, 0, AccessType::Fetch), TlbResult::Hit { .. })); + }).unwrap().join().unwrap(); + } + + /// The rule reaches the wired entries (the low indexes the kernel + /// reserves): a new entry over a wired one removes the wired one. + #[test] + fn a_write_invalidates_wired_entries_too() { + std::thread::Builder::new().stack_size(64 << 20).spawn(|| { + let va = BASE + 0x2345; + let lo = |pfn: u64| [(pfn << 6) | (5 << 3) | 0x7; 2]; + let run = |on: bool| { + let mut tlb = MipsTlb::new(64); + tlb.set_conflict_invalidation(on); + tlb.write(0, entry(0, BASE | 0x2000, lo(0x10))); + tlb.write(40, entry(0, BASE | 0x2000, lo(0x20))); + tlb + }; + let on = run(true); + assert!(on.read(0).entry_hi == 0 && on.read(0).entry_lo == [0; 2], "wired entry 0 is gone"); + assert_eq!(on.read(40).entry_hi, BASE | 0x2000); + let mut on = on; + assert!(on.take_write_conflict()); + match on.translate::<1>(va, 0, AccessType::Read) { + TlbResult::Hit { phys_addr, .. } => assert_eq!(phys_addr, (0x20 << 12) | 0x345), + other => panic!("expected the new entry to answer, got {other:?}"), + } + let off = run(false); + assert_eq!(off.read(0).entry_hi, BASE | 0x2000, "without the rule the wired entry stays"); + }).unwrap().join().unwrap(); + } + + /// TLBR of an invalidated entry still loads its PageMask. The kernel + /// reads a wired sentinel back and reuses PageMask for its next write; a + /// zeroed mask makes that write 4 KiB. + #[test] + fn an_invalidated_entry_keeps_its_page_mask() { + std::thread::Builder::new().stack_size(64 << 20).spawn(|| { + let mut tlb = MipsTlb::new(64); + tlb.set_conflict_invalidation(true); + tlb.write(3, entry(0x1fe000, BASE, [0x7; 2])); + tlb.write(9, entry(0x1ffe000, BASE, [0x7; 2])); + let gone = tlb.read(3); + assert_eq!(gone.page_mask, 0x1fe000); + assert_eq!(gone.entry_hi, 0); + assert_eq!(gone.entry_lo, [0; 2]); + assert_eq!(tlb.read(9).page_mask, 0x1ffe000); + }).unwrap().join().unwrap(); + } + + /// Entries that cannot match together are left alone: a different ASID + /// when neither is global, and a page number that differs above the + /// wider mask. + #[test] + fn entries_that_cannot_match_together_are_kept() { + std::thread::Builder::new().stack_size(64 << 20).spawn(|| { + let mut tlb = MipsTlb::new(64); + tlb.set_conflict_invalidation(true); + tlb.write(3, entry(0, BASE | 0x4000 | 1, [0x6; 2])); // ASID 1, not global + tlb.write(4, entry(0, BASE | 0x4000 | 2, [0x6; 2])); // ASID 2 + tlb.write(5, entry(0, BASE | 0x8000 | 1, [0x6; 2])); // other page + assert!(!tlb.take_write_conflict()); + for i in [3, 4, 5] { + assert_ne!(tlb.read(i).entry_hi, 0, "entry {i} stays"); + } + // A global entry over that page does match both ASIDs. + tlb.write(6, entry(0, BASE | 0x4000 | 0x1000, [0x7; 2])); // global + assert!(tlb.take_write_conflict()); + assert_eq!(tlb.read(3).entry_hi, 0); + assert_eq!(tlb.read(4).entry_hi, 0); + assert_ne!(tlb.read(5).entry_hi, 0); + }).unwrap().join().unwrap(); + } + #[test] fn test_tlb_entry_flags() { let mut entry = TlbEntry::new(); @@ -493,3 +627,52 @@ fn tlb_write_inside_a_larger_entry_count_is_kept() { assert_eq!(r10000.read(63).entry_hi, entry.entry_hi, "...and its last slot"); } } + +/// A mapped uncached page carries EntryLo's uncached attribute (bits 63:62) +/// to the bus as physical bits 58:57; a cached page never does, whatever those bits +/// hold, and no attribute leaves the PFN alone. "Uncached" is the model's C +/// table, so a model that decodes a C value differently, or a table installed +/// after the entry was written, changes the answer. +#[test] +fn only_uncached_pages_carry_the_uncached_attribute() { + use super::{MipsTlb, Tlb, TlbEntry}; + use crate::cpu::mips_exec::CacheAttr; + // MipsTlb is large; give the one instance room, as its neighbours do. + std::thread::Builder::new() + .stack_size(16 * 1024 * 1024) + .spawn(|| { + let pfn = 0x0d100u64 << 6; + let base = |tlb: &MipsTlb| tlb.read(0).pfn_base; + let write = |tlb: &mut MipsTlb, lo0: u64, lo1: u64| { + let mut entry = TlbEntry::new(); + entry.entry_hi = 0xC000_0000_0004_0000; + entry.entry_lo = [lo0, lo1]; + tlb.write(0, entry); + }; + let mut tlb = Box::new(MipsTlb::new(64)); + // The default table: C=2 and the reserved values are uncached. + for (cca, expect) in [(2u64, 1u64 << 57), (6, 1 << 57), (3, 0), (5, 0)] { + let lo = (1 << 62) | pfn | (cca << 3) | 0x6; + write(&mut tlb, lo, lo); + assert_eq!(base(&tlb), [0x0d10_0000 | expect; 2], "cca {cca}"); + } + write(&mut tlb, pfn | (2 << 3), 0); + assert_eq!(base(&tlb)[0], 0x0d10_0000, "no attribute leaves the PFN alone"); + + // The R10000 table reads C=7 (uncached accelerated) as uncached and + // 5 as cacheable; a model that read 7 as cacheable carries nothing. + let mut table = [CacheAttr::Uncached; 8]; + table[3] = CacheAttr::Cacheable; + table[5] = CacheAttr::CacheableCoherent; + tlb.set_cache_attrs(table); + let lo = (1 << 62) | pfn | (7 << 3) | 0x6; + write(&mut tlb, lo, lo); + assert_eq!(base(&tlb), [0x0d10_0000 | 1 << 57; 2], "C=7 uncached under this table"); + table[7] = CacheAttr::Cacheable; + tlb.set_cache_attrs(table); + assert_eq!(base(&tlb), [0x0d10_0000; 2], "the table, installed later, decides"); + }) + .unwrap() + .join() + .unwrap(); +} diff --git a/src/cpu/smp.rs b/src/cpu/smp.rs index dbac65a69..c54368b4e 100644 --- a/src/cpu/smp.rs +++ b/src/cpu/smp.rs @@ -26,6 +26,9 @@ pub const HOST_PARK_BIT: u64 = 1 << 62; pub const HOLD_HEART: u32 = 1 << 0; /// Another thread needs the machine still ([`World::stop_others`]). pub const HOLD_WORLD: u32 = 1 << 1; +/// The board holds this processor at a firmware rendezvous until the ones +/// it must follow catch up (the IP35 node's boot rendezvous). +pub const HOLD_RENDEZVOUS: u32 = 1 << 2; const NOT_A_CPU: u8 = u8::MAX; diff --git a/src/dev/bcm5701.rs b/src/dev/bcm5701.rs new file mode 100644 index 000000000..5e347ca79 --- /dev/null +++ b/src/dev/bcm5701.rs @@ -0,0 +1,574 @@ +//! The Broadcom BCM5701 gigabit Ethernet controller (PCI 14e4:1645), as the +//! single-port SGI board carries it: a descriptor-ring MAC with an integrated +//! copper PHY, which the public Linux tg3 and OpenBSD bge drivers and the +//! IRIX tg driver all drive. +//! +//! BAR0 is 64 KiB: a register file at 0x0000-0x7fff (its first 256 bytes also +//! alias the PCI configuration registers) and a 32 KiB window onto the 32 KiB +//! on-chip SRAM at 0x8000-0xffff, whose base the memory-window base register +//! selects. The driver reaches chip state three ways: the BAR directly, an +//! indirect register window through configuration registers 0x78 and 0x80, +//! and an indirect memory window through 0x7c and 0x84. All three land on the +//! same state. The configuration side is a [`PciConfigHook`] on the +//! function's header, so it works behind any bridge. +//! +//! Modelled: the PCI identity; the chip id in MISC_CTL; the global reset with +//! the on-chip firmware-ready handshake in SRAM and the firmware-progress +//! register; the self-clearing mode-register resets; the MII path to the +//! PHY ([`phy`]); the station address the bootcode leaves in SRAM and in +//! address filter slot 0; the interrupt mailbox, the MISC_CTL global mask and +//! PCISTATE.INTR_NOT_ACTIVE; the buffer manager's delayed enable; and the +//! serial EEPROM controller over a caller-supplied backing ([`eeprom`]); the +//! send ring and its doorbell, the standard, jumbo and return receive rings, +//! the status block DMA, host coalescing, the interrupt it drives, the +//! transmit checksum offload and segmentation, and the receive checksum +//! report ([`dma`]). +//! +//! # What is approximated +//! +//! - Iris has no virtual clock, so timing is wall-clock. Deadlines (the PHY's +//! 100 ms settle, the buffer manager's 8 ms startup, the coalescing budgets +//! and the arrival latency) are evaluated when the device is touched and by +//! the receive thread, each at its own expiry and earliest first, as the +//! model's virtual clock would fire them. +//! - EEPROM transfers complete synchronously; the delay register only reads +//! back, and the 10 ms write spacing is not enforced. Direct serial-pin +//! control is refused and counted. +//! - The management interface ignores the PHY address field; one PHY answers. +//! - A write to a register with no modelled behaviour is stored and reads +//! back, and is recorded in [`Bcm5701::refused`]. +//! - The BCM5704 dual-port variants are not modelled. +//! - Frames reach the host through the shared network link; the chip's +//! virtio-net offload path has no counterpart, so a large TCP buffer is +//! always cut into segments in software. +//! - A transmit is carried out entirely inside the doorbell write, so there +//! is no re-entrancy guard and no deferred flush: the receive thread offers +//! a waiting frame again whenever the guest makes room. +//! +//! Ported from hw/net/bcm570x.c and hw/net/bcm570x_phy.c in +//! jamesbraid/qemu-sgi-origami. + +mod dma; +pub mod eeprom; +pub mod phy; +mod state; + +#[cfg(test)] +mod tests; + +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, Weak}; +use std::time::{Duration, Instant}; + +use parking_lot::Mutex; + +use crate::irq::IrqLine; +use crate::pci::{Bar, PciBusDevice, PciConfig, PciConfigHook, PciConfigPolicy, PciDma, PciRegisterPolicy, PCI_INTA}; +use crate::traits::{Device, Resettable}; +use eeprom::Eeprom; +pub use dma::{DmaWhat, Mailbox, Ring, RxOutcome, MAX_FRAME, RX_ARRIVAL_LATENCY_DEFAULT}; +pub use state::Refused; +use state::{cfg, misc_ctl, pcistate, State}; + +pub const VENDOR_ID: u16 = 0x14e4; +pub const DEVICE_ID: u16 = 0x1645; +/// The SGI board's subsystem identity. +pub const SUBSYSTEM_VENDOR_ID: u16 = 0x10a9; +pub const SUBSYSTEM_ID: u16 = 0x8010; +/// PCI network controller, Ethernet. +const CLASS_ETHERNET: u32 = 0x02_0000; + +pub const BAR0_SIZE: u32 = 0x1_0000; +/// The low 256 bytes of BAR0 are the PCI configuration registers again. +const CFG_ALIAS: u32 = 0x100; + +/// What a board gives the chip: its station address and, optionally, the +/// serial EEPROM the board fits. +pub struct Bcm5701Config { + pub mac: [u8; 6], + pub eeprom: Option, + /// The minimum between a frame arriving and the driver being told, so a + /// completion never beats the arrival it reports. Zero publishes at once. + pub rx_arrival_latency: Duration, +} + +impl Bcm5701Config { + pub fn new(mac: [u8; 6]) -> Self { + Self { mac, eeprom: None, rx_arrival_latency: RX_ARRIVAL_LATENCY_DEFAULT } + } +} + +pub struct Bcm5701 { + config: Arc, + st: Mutex, + irq: Mutex, + /// Serialises delivery of level changes to the interrupt line. + irq_order: Mutex<()>, + link: Mutex>, + rx_wake: Mutex>>, + /// Between `start` and `stop`; the NAT engine watches it too. + running: Arc, + /// The flag the network backend watches, set and cleared with `running`. + net_running: Mutex>>, + /// The network backend, between runs. A NAT engine keeps its flows across + /// a stop. + backend: Arc>>>, + /// The NAT and receive threads, joined in `stop`. + threads: Mutex>>, + me: Weak, + /// Tests freeze the clock to step it exactly. + #[cfg(test)] + frozen: Mutex>, +} + +/// The configuration registers the chip computes: the four indirect-window +/// registers, and the actions behind a MISC_CTL write. +struct Hook(Weak); + +impl PciConfigHook for Hook { + fn read(&self, reg: usize) -> Option { + if !(cfg::REG_BASEADDR..=cfg::MEMWIN_DATA).contains(®) { + return None; + } + let dev = self.0.upgrade()?; + Some(dev.enter(dev.now(), |st| match reg { + cfg::REG_BASEADDR => st.reg_win_base, + cfg::MEMWIN_BASEADDR => st.mem_win_base, + cfg::REG_DATA => st.reg_read(st.reg_win_base), + _ => st.sram_read(st.mem_win_base), + })) + } + + fn write(&self, reg: usize, val: u32, _mask: u32) -> bool { + if !(cfg::REG_BASEADDR..=cfg::MEMWIN_DATA).contains(®) { + return false; + } + let Some(dev) = self.0.upgrade() else { return false }; + dev.enter(dev.now(), |st| match reg { + cfg::REG_BASEADDR => st.reg_win_base = val, + cfg::MEMWIN_BASEADDR => st.mem_win_base = val, + cfg::REG_DATA => st.reg_write(st.reg_win_base, val), + _ => st.sram_write(st.mem_win_base, val), + }); + true + } + + fn written(&self, reg: usize) { + if reg == cfg::MISC_CTL { + if let Some(dev) = self.0.upgrade() { + dev.enter(dev.now(), State::misc_ctl_written); + } + } + } +} + +impl Bcm5701 { + pub fn new(config: Bcm5701Config) -> Result, String> { + if config.mac[0] & 1 != 0 { + return Err("bcm5701: station address must be unicast".into()); + } + let policy = PciConfigPolicy { + registers: &[ + // The chip id in the high half is read-only; the driver's + // byte-order and interrupt-mask bits in the low half stick. + PciRegisterPolicy { + reg_dword: cfg::MISC_CTL, + initial: state::CHIPID_BCM5701 << 16, + writable_mask: misc_ctl::WRITABLE, + }, + // The chip drives PCISTATE; the guest cannot write it. It + // runs as conventional PCI, saw no REQ64#, and starts with + // the interrupt line down. + PciRegisterPolicy { + reg_dword: cfg::PCISTATE, + initial: pcistate::BUSMODE_PCI | pcistate::BUS_32BIT | pcistate::INTR_NOT_ACTIVE, + writable_mask: 0, + }, + // The indirect windows live in the chip, reached by whole + // dwords through the hook. The header bytes behind them are + // what a narrower access sees, and nothing writes them. + PciRegisterPolicy { reg_dword: cfg::REG_BASEADDR, initial: 0, writable_mask: 0 }, + PciRegisterPolicy { reg_dword: cfg::MEMWIN_BASEADDR, initial: 0, writable_mask: 0 }, + PciRegisterPolicy { reg_dword: cfg::REG_DATA, initial: 0, writable_mask: 0 }, + PciRegisterPolicy { reg_dword: cfg::MEMWIN_DATA, initial: 0, writable_mask: 0 }, + ], + ..Default::default() + }; + let pci = Arc::new(PciConfig::new_with_policy( + VENDOR_ID, + DEVICE_ID, + CLASS_ETHERNET, + 0, + [Bar::Mem(BAR0_SIZE), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], + PCI_INTA, + policy, + )); + pci.set_subsystem_identity(u32::from(SUBSYSTEM_ID) << 16 | u32::from(SUBSYSTEM_VENDOR_ID)); + let now = Instant::now(); + let dev = Arc::new_cyclic(|me| Self { + st: Mutex::new(State::new(pci.clone(), config.mac, config.eeprom, config.rx_arrival_latency, now)), + config: pci, + irq: Mutex::new(IrqLine::unconnected()), + irq_order: Mutex::new(()), + link: Mutex::new(None), + rx_wake: Mutex::new(None), + running: Arc::new(AtomicBool::new(false)), + net_running: Mutex::new(None), + backend: Arc::new(Mutex::new(None)), + threads: Mutex::new(Vec::new()), + me: me.clone(), + #[cfg(test)] + frozen: Mutex::new(None), + }); + dev.config.set_hook(Arc::new(Hook(Arc::downgrade(&dev)))); + Ok(dev) + } + + fn now(&self) -> Instant { + #[cfg(test)] + if let Some(t) = *self.frozen.lock() { + return t; + } + Instant::now() + } + + /// Run `f` with the state at `now`, then deliver what it produced: any + /// interrupt level it changed, frames to send, and a wake-up for the + /// receive thread if the guest made room. + fn enter(&self, now: Instant, f: impl FnOnce(&mut State) -> R) -> R { + let (result, wire, wake, purge) = { + let mut st = self.st.lock(); + st.enter(now); + let result = f(&mut st); + ( + result, + std::mem::take(&mut st.wire), + std::mem::take(&mut st.wake_rx), + std::mem::take(&mut st.purge_rx), + ) + }; + if purge { + // A reset drops what the backend was holding for the guest, so a + // frame from before it cannot reach the new rings. + if let Some(link) = self.link.lock().as_mut() { + while link.from_net.pop().is_ok() {} + } + } + self.irq_flush(); + self.send(wire); + if wake { + self.wake_rx(); + } + result + } + + /// Deliver queued interrupt level changes, in order, with no state lock + /// held that the line could need. + fn irq_flush(&self) { + let _order = self.irq_order.lock(); + let events = std::mem::take(&mut self.st.lock().irq_events); + if events.is_empty() { + return; + } + let line = self.irq.lock(); + for level in events { + line.set(level); + } + } + + /// A machine reset: registers, SRAM, PHY and the PCI header as at + /// power-on, the interrupt dropped. The EEPROM keeps its contents. + pub fn reset(&self) { + self.config.reset(); + self.enter(self.now(), State::reset); + } + + /// The backend's cable state; the PHY reports it as link. + pub fn set_backend_link(&self, up: bool) { + self.enter(self.now(), |st| st.phy.set_link_down(!up)); + } + + fn send(&self, frames: Vec>) { + if frames.is_empty() { + return; + } + if let Some(link) = self.link.lock().as_mut() { + for frame in frames { + let _ = link.to_net.push(frame); + } + link.tx_wake.notify(); + } + } + + fn wake_rx(&self) { + if let Some(wake) = self.rx_wake.lock().as_ref() { + wake.notify(); + } + } + + /// Connect to the backend `[network]` selected: frames sent go on + /// `link.to_net`, frames on `link.from_net` are received, and + /// `link.rx_wake` is notified when the guest does something that can + /// unblock a waiting frame. The backend and receive threads run between + /// `start` and `stop`; a NAT engine, which only trades frames with the + /// host and never reaches the guest, keeps its flows across a stop. + pub fn attach(&self, mut link: crate::net::Link) { + let backend = link.backend(); + let running = link.running(); + let rings = link.into_rings(); + *self.rx_wake.lock() = Some(rings.rx_wake.clone()); + *self.link.lock() = Some(rings); + *self.net_running.lock() = Some(running.clone()); + *self.backend.lock() = backend; + if self.running.load(Ordering::SeqCst) { + running.store(true, Ordering::SeqCst); + self.spawn_nat(); + } + } + + fn spawn_nat(&self) { + let slot = self.backend.clone(); + let Some(mut engine) = slot.lock().take() else { return }; + let t = std::thread::Builder::new() + .name("bcm5701-net".into()) + .spawn(move || { + engine.run(); + *slot.lock() = Some(engine); + }) + .expect("bcm5701 NAT thread"); + self.threads.lock().push(t); + } + + /// Take frames into guest memory as they arrive and as the guest makes + /// room, and run the device's timers, until it stops. The wait ends early + /// for a wake from the guest or the NAT engine. + fn rx_loop(me: Weak, running: Arc, wake: Arc) { + while running.load(Ordering::SeqCst) { + let timeout = { + let Some(this) = me.upgrade() else { return }; + let now = this.now(); + this.enter(now, |_| ()); + let next = this.st.lock().deadline(); + next.and_then(|d| d.checked_duration_since(now)).unwrap_or(RX_IDLE) + }; + wake.wait(timeout); + if !running.load(Ordering::SeqCst) { + return; + } + let Some(this) = me.upgrade() else { return }; + this.poll(); + } + } + + /// Take frames from the backend into the buffers the driver has posted. + /// A frame the engine cannot take yet stays at the head of the backend's + /// queue until the guest makes room; one the filter rejects is discarded. + /// Returns whether any frame was consumed. + /// + /// The link is held only to take one frame at a time, so a transmit on the + /// CPU thread never waits behind a burst of deliveries. + pub fn poll(&self) -> bool { + let mut moved = false; + loop { + let now = self.now(); + { + let mut link = self.link.lock(); + let Some(link) = link.as_mut() else { break }; + let outcome = { + let Ok(frame) = link.from_net.peek() else { break }; + let mut st = self.st.lock(); + st.enter(now); + st.receive(frame) + }; + if outcome == RxOutcome::Deferred { + break; + } + let _ = link.from_net.pop(); + moved = true; + } + self.irq_flush(); + } + moved + } + + /// Everything the model was asked for and could not do, with counts. + pub fn refused(&self) -> Vec<(Refused, u64)> { + self.st.lock().refused() + } + + /// The serial EEPROM backing's current contents, if one is fitted. + pub fn eeprom_bytes(&self) -> Option> { + self.st.lock().eeprom_bytes() + } + + /// Name the long-lived locks in the lock registry, which holds only a + /// weak reference to the device. + pub fn register_locks(self: &Arc) { + for (what, probe) in [ + ("st", (|d: &Bcm5701| d.st.is_locked()) as fn(&Bcm5701) -> bool), + ("link", |d| d.link.is_locked()), + ("irq_order", |d| d.irq_order.is_locked()), + ] { + let me = Arc::downgrade(self); + crate::locks::register_lock(format!("bcm5701::{what}"), move || me.upgrade().is_some_and(|d| probe(&d))); + } + } + + /// BAR0 accepts aligned 32-bit accesses everywhere, and byte reads of the + /// first station-address pair. + fn accepts(off: u32, width: usize, write: bool) -> bool { + match width { + 4 => off & 3 == 0, + 1 => !write && (state::reg::MAC_ADDR_0_HIGH..state::reg::MAC_ADDR_0_LOW + 4).contains(&off), + _ => false, + } + } +} + +impl PciBusDevice for Bcm5701 { + fn config(&self) -> &PciConfig { + &self.config + } + + fn bar_read(&self, bar: usize, off: u32, width: usize) -> u32 { + // A rejected read returns zero, as the model it is ported from does + // behind its bridge; the refusal is counted. + if bar != 0 { + return 0; + } + let valid = Self::accepts(off, width, false) && off < BAR0_SIZE; + // The configuration alias goes through the header, whose hook takes + // the state lock itself. + if valid && width == 4 && off < CFG_ALIAS { + return self.config.read((off / 4) as usize); + } + self.enter(self.now(), |st| { + if !valid { + st.refuse(Refused::BarAccess { offset: off, width: width as u8, write: false }); + 0 + } else if width == 1 { + (st.reg_read(off & !3) >> ((off & 3) * 8)) & 0xff + } else if off < state::REG_FILE_SIZE { + st.reg_read(off) + } else { + st.sram_read((st.mem_win_base & !(state::SRAM_SIZE - 1)) + (off - state::REG_FILE_SIZE)) + } + }) + } + + fn bar_write(&self, bar: usize, off: u32, val: u32, width: usize) { + if bar != 0 { + return; + } + let valid = Self::accepts(off, width, true) && off < BAR0_SIZE; + if valid && off < CFG_ALIAS { + self.config.write((off / 4) as usize, val); + return; + } + self.enter(self.now(), |st| { + if !valid { + st.refuse(Refused::BarAccess { offset: off, width: width as u8, write: true }); + } else if off < state::REG_FILE_SIZE { + st.reg_write(off, val); + } else { + st.sram_write((st.mem_win_base & !(state::SRAM_SIZE - 1)) + (off - state::REG_FILE_SIZE), val); + } + }); + } + + /// The chip reflects in PCISTATE whether it saw REQ64# at the end of + /// reset. + fn req64_sampled(&self, asserted: bool) { + self.config.update(cfg::PCISTATE, |v| { + if asserted { + v & !pcistate::BUS_32BIT + } else { + v | pcistate::BUS_32BIT + } + }); + } + + fn connect_dma(&self, dma: Arc) { + self.st.lock().dma = dma; + self.wake_rx(); + } + + fn connect_interrupt(&self, line: IrqLine) { + *self.irq.lock() = line; + // Present the current level to the new line. + self.st.lock().update_irq(true); + self.irq_flush(); + } +} + +/// How long the receive thread sleeps with no deadline to wait for: it is +/// woken by the guest and the NAT engine, so this is only a bound. +const RX_IDLE: Duration = Duration::from_secs(60); + +impl Device for Bcm5701 { + fn step(&self, _cycles: u64) {} + + /// Start the receive thread, and the NAT engine's if one is attached. + fn start(&self) { + if self.running.swap(true, Ordering::SeqCst) { + return; + } + let wake = self.rx_wake.lock().get_or_insert_with(crate::net::Wake::new).clone(); + let (me, running) = (self.me.clone(), self.running.clone()); + let rx = std::thread::Builder::new() + .name("bcm5701-rx".into()) + .spawn(move || Self::rx_loop(me, running, wake)) + .expect("bcm5701 rx thread"); + self.threads.lock().push(rx); + if let Some(r) = self.net_running.lock().as_ref() { + r.store(true, Ordering::SeqCst); + } + self.spawn_nat(); + // Frames queued while stopped. + self.wake_rx(); + } + + /// Join the threads: no frame reaches guest memory and no line moves + /// afterwards. Frames the NAT engine takes meanwhile queue for `start`. + fn stop(&self) { + if !self.running.swap(false, Ordering::SeqCst) { + return; + } + if let Some(r) = self.net_running.lock().as_ref() { + r.store(false, Ordering::SeqCst); + } + self.wake_rx(); + if let Some(l) = self.link.lock().as_ref() { + l.tx_wake.notify(); + } + // The receive thread can hold the last reference for a moment, and + // then this runs on it; it cannot join itself. + let me = std::thread::current().id(); + let threads: Vec<_> = self.threads.lock().drain(..).collect(); + for t in threads { + if t.thread().id() != me { + let _ = t.join(); + } + } + } + + fn is_running(&self) -> bool { + self.running.load(Ordering::SeqCst) + } + + fn get_clock(&self) -> u64 { + 0 + } +} + +impl Resettable for Bcm5701 { + fn power_on(&self) { + self.reset(); + } +} + +impl Drop for Bcm5701 { + fn drop(&mut self) { + Device::stop(&*self); + } +} diff --git a/src/dev/bcm5701/dma.rs b/src/dev/bcm5701/dma.rs new file mode 100644 index 000000000..577c8a99d --- /dev/null +++ b/src/dev/bcm5701/dma.rs @@ -0,0 +1,1014 @@ +//! The descriptor engines: the send ring, the standard and jumbo receive +//! rings and the return ring, the status block, and the host coalescing that +//! decides when the driver is told. +//! +//! Descriptors and the status block are the chip's big-endian internal image, +//! reaching the bus through the MODE_CTL swap controls; frame data uses the +//! frame-data controls. What the host sees of those bytes is the bridge's +//! business ([`PciDma`]). + +use std::time::Duration; + +use super::state::{misc_ctl, reg, Refused, State}; +use crate::pci::{PciDma, PciDmaStatus}; + +/// The largest frame the engines carry, transmit or receive. The part does +/// jumbo frames; the figure matches what the other Origin adapters carry so a +/// guest configured for jumbo behaves the same whichever card it has. It is a +/// model limit, not the hardware's. +pub const MAX_FRAME: usize = 9600; + +const SRAM_SEND_RING_RCB: u32 = 0x0100; +const SRAM_RX_RETURN_RCB: u32 = 0x0200; +/// `nicNoMoreRXBDs`, the one statistics counter the model keeps: the number of +/// times the chip ran out of receive descriptors. +const SRAM_STATS_NO_MORE_RXBDS: u32 = 0x0898; +const RCB_HOSTADDR_HI: u32 = 0x00; +const RCB_HOSTADDR_LO: u32 = 0x04; +const RCB_MAXLEN_FLAGS: u32 = 0x08; +const RCB_FLAG_EXT_RX_ENABLE: u32 = 1; +const RCB_FLAG_DISABLE_RING: u32 = 2; + +const TXBD_SIZE: u64 = 16; +// Offsets in the descriptors' little-endian layout; a 16-bit field's +// image offset is its offset xor 2. +const TXBD_FLAGS: usize = 0x08; +const TXBD_LEN: usize = 0x0a; +const TXBD_MSS: usize = 0x0e; +const TXBDFLAG_TCP_UDP_CKSUM: u16 = 0x0001; +const TXBDFLAG_IP_CKSUM: u16 = 0x0002; +const TXBDFLAG_END: u16 = 0x0004; +const TXBDFLAG_INSERT_SRC_ADDR: u16 = 0x1000; +const TXBDFLAG_CHOOSE_SRC_ADDR: u16 = 0x6000; +const TXBDFLAG_SRC_ADDR_SHIFT: u16 = 13; + +const RXBD_SIZE: u64 = 32; +const RXBD_IDX_LEN: usize = 0x08; +const RXBD_TYPE_FLAGS: usize = 0x0c; +const RXBD_CSUM: usize = 0x10; +const RXBD_OPAQUE: usize = 0x1c; +const RXBDFLAG_END: u32 = 0x0004; +const RXBDFLAG_JUMBO_RING: u32 = 0x0020; +const RXBDFLAG_IP_CKSUM_OK: u32 = 0x1000; +const RXBDFLAG_TCP_UDP_CKSUM_OK: u32 = 0x2000; +const RXBDFLAG_TCP: u32 = 0x4000; +const MODE_RX_NO_PSEUDO: u32 = 1 << 23; +const MODE_NO_FRAME_CRACK: u32 = 1 << 9; + +/// The standard ring holds a fixed number of descriptors. Its control block's +/// length is the size of the buffer each descriptor points at, an MTU rather +/// than a count, while the return ring's really is a ring length. +const RX_STD_RING_LEN: u16 = 512; +const RX_JUMBO_RING_LEN: u16 = 256; +const FCS_LEN: u32 = 4; +const MIN_FRAME: usize = 60; + +const STATUS_BLK_SIZE: usize = 0x50; +const STATFLAG_UPDATED: u32 = 1; +const STATUS_RX_PROD_OFF: usize = 16; +const STATUS_TX_CONS_OFF: usize = 18; +const STATUS_TAG_OFF: usize = 4; +const STATUS_RX_STD_CONS_IMAGE_OFF: usize = 8; +const STATUS_RX_JUMBO_CONS_IMAGE_OFF: usize = 10; +const HCC_MODE_ENABLE: u32 = 2; + +/// Both coalescing tick budgets are programmed in microseconds. +const COAL_TICK: Duration = Duration::from_micros(1); +/// A status DMA the guest's address refused is retried no slower than this. +const COAL_RETRY_MAX: Duration = Duration::from_millis(250); +/// Completions the arrival line can hold; a full line refuses frames. +const RX_LATENCY_DEPTH: usize = 256; +/// The default minimum between a frame arriving and its completion. +pub const RX_ARRIVAL_LATENCY_DEFAULT: Duration = Duration::from_micros(200); + +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub enum Ring { + Standard, + Jumbo, +} + +/// The mailboxes whose index is range-checked. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub enum Mailbox { + TxProducer, + RxStandardProducer, + RxJumboProducer, + RxReturnConsumer, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub enum DmaWhat { + TxDescriptor, + TxPayload, + RxDescriptor, + RxPayload, + RxCompletion, + StatusBlock, +} + +/// The timers the engines keep, ordered by when they fire. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub enum Timer { + PhySettle, + BufferManager, + RxCoalescing, + TxCoalescing, + RxArrival, +} + +/// What the receive path did with a frame the backend offered. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum RxOutcome { + /// Delivered, filtered or dropped: the backend may discard it. + Consumed, + /// Not taken; the frame stays at the head of the backend's queue until + /// the guest makes room. + Deferred, +} + +/// A bus master with nowhere to go, until the bridge connects one. +pub(super) struct NullDma; +impl PciDma for NullDma { + fn dma_read(&self, _addr: u64, buf: &mut [u8]) -> PciDmaStatus { + buf.fill(0); + PciDmaStatus::MasterAbort + } + fn dma_write(&self, _addr: u64, _data: &[u8]) -> PciDmaStatus { + PciDmaStatus::MasterAbort + } +} + +fn be32(b: &[u8], off: usize) -> u32 { + u32::from_be_bytes(b[off..off + 4].try_into().unwrap()) +} + +/// A 16-bit field of the chip's image sits in the half of its 32-bit word +/// the big-endian structure puts it in. +fn ld16(b: &[u8], off: usize) -> u16 { + let o = off ^ 2; + u16::from_be_bytes([b[o], b[o + 1]]) +} + +fn st32(b: &mut [u8], off: usize, v: u32) { + b[off..off + 4].copy_from_slice(&v.to_be_bytes()); +} + +fn st16(b: &mut [u8], off: usize, v: u16) { + let o = off ^ 2; + b[o..o + 2].copy_from_slice(&v.to_be_bytes()); +} + +/// The one's-complement sum of a span, folded. +fn ones_sum(buf: &[u8], seed: u32) -> u16 { + let mut sum = seed; + for pair in buf.chunks(2) { + sum += u32::from(pair[0]) << 8 | u32::from(*pair.get(1).unwrap_or(&0)); + } + while sum >> 16 != 0 { + sum = (sum & 0xffff) + (sum >> 16); + } + sum as u16 +} + +/// Fill in the IP header and/or TCP and UDP checksums of an untagged IPv4 +/// frame. A fragment keeps its transport checksum. +fn checksum_calculate(frame: &mut [u8], ip: bool, transport: bool) { + if frame.len() < 14 + 20 || frame[14] >> 4 != 4 { + return; + } + let ihl = usize::from(frame[14] & 0xf) * 4; + if ihl < 20 || frame.len() < 14 + ihl { + return; + } + if ip { + frame[24] = 0; + frame[25] = 0; + let sum = !ones_sum(&frame[14..14 + ihl], 0); + frame[24..26].copy_from_slice(&sum.to_be_bytes()); + } + if u16::from_be_bytes([frame[20], frame[21]]) & 0x3fff != 0 { + return; + } + let total = usize::from(u16::from_be_bytes([frame[16], frame[17]])); + if total < ihl { + return; + } + let data_len = (total - ihl).min(frame.len() - 14 - ihl); + let proto = frame[23]; + let at = match proto { + 6 if transport && data_len >= 20 => 16, + 17 if transport && data_len >= 8 => 6, + _ => return, + }; + let start = 14 + ihl; + frame[start + at] = 0; + frame[start + at + 1] = 0; + let pseudo = ones_pseudo(&frame[26..34], proto, data_len); + let sum = !ones_sum(&frame[start..start + data_len], pseudo); + frame[start + at..start + at + 2].copy_from_slice(&sum.to_be_bytes()); +} + +fn ones_pseudo(addresses: &[u8], proto: u8, len: usize) -> u32 { + addresses.chunks(2).map(|p| u32::from(p[0]) << 8 | u32::from(p[1])).sum::() + u32::from(proto) + len as u32 +} + +/// Finish the checksums the driver left to the card. The lengths in the frame +/// were composed by the guest, so a header shorter than its own minimum, or +/// lengths reaching past what arrived, decline the offload rather than walk +/// off the buffer. +fn tx_offload_checksum(frame: &mut [u8], flags: u16) { + if frame.len() < 14 + 20 || frame[12..14] != [0x08, 0x00] || frame[14] >> 4 != 4 { + return; + } + let ihl = usize::from(frame[14] & 0xf) * 4; + let total = usize::from(u16::from_be_bytes([frame[16], frame[17]])); + if ihl < 20 || total < ihl || 14 + total > frame.len() { + return; + } + checksum_calculate(frame, flags & TXBDFLAG_IP_CKSUM != 0, flags & TXBDFLAG_TCP_UDP_CKSUM != 0); +} + +struct LsoLayout { + ihl: usize, + thl: usize, + hdr: usize, + payload: usize, +} + +/// Whether `frame` is an unfragmented IPv4 TCP frame that `mss` can cut up. +fn lso_layout(frame: &[u8], mss: usize) -> Option { + if frame.len() < 14 + 20 + 20 || frame[12..14] != [0x08, 0x00] || frame[14] >> 4 != 4 || frame[23] != 6 { + return None; + } + let ihl = usize::from(frame[14] & 0xf) * 4; + if ihl < 20 || 14 + ihl + 20 > frame.len() { + return None; + } + let ip_len = usize::from(u16::from_be_bytes([frame[16], frame[17]])); + if u16::from_be_bytes([frame[20], frame[21]]) & 0x3fff != 0 || ip_len < ihl + 20 || 14 + ip_len != frame.len() { + return None; + } + let thl = usize::from(frame[14 + ihl + 12] >> 4) * 4; + let hdr = 14 + ihl + thl; + if thl < 20 || ihl + thl > ip_len || mss == 0 || hdr + mss > MAX_FRAME { + return None; + } + let payload = ip_len - ihl - thl; + if payload <= mss { + return None; + } + Some(LsoLayout { ihl, thl, hdr, payload }) +} + +/// The checksums of an arriving frame, as the part reports them: the sums it +/// computed rather than a verdict, a field that checks out summing to 0xffff. +/// Anything not plainly IPv4 TCP or UDP is left unclaimed, because reporting +/// a sum for something never summed would have the stack skip its own pass. +fn rx_offload_checksum(frame: &[u8], mode: u32, flags: &mut u32) -> u32 { + if mode & MODE_NO_FRAME_CRACK != 0 { + return 0; + } + if frame.len() < 34 || frame[12..14] != [0x08, 0x00] || frame[14] >> 4 != 4 { + return 0; + } + let ihl = usize::from(frame[14] & 0xf) * 4; + let total = usize::from(u16::from_be_bytes([frame[16], frame[17]])); + if ihl < 20 || total < ihl || 14 + total > frame.len() { + return 0; + } + let ip_sum = ones_sum(&frame[14..14 + ihl], 0); + if ip_sum == 0 || ip_sum == 0xffff { + *flags |= RXBDFLAG_IP_CKSUM_OK; + } + let ip_only = u32::from(ip_sum) << 16; + let proto = frame[23]; + if proto != 6 && proto != 17 { + return ip_only; + } + if proto == 6 { + *flags |= RXBDFLAG_TCP; + } + if u16::from_be_bytes([frame[20], frame[21]]) & 0x3fff != 0 { + return ip_only; + } + let l4len = total - ihl; + if (proto == 6 && l4len < 20) || (proto == 17 && l4len < 8) { + return ip_only; + } + let mut transport_checksum_present = true; + if proto == 6 { + let tcp_hlen = usize::from(frame[14 + ihl + 12] >> 4) * 4; + if tcp_hlen < 20 || tcp_hlen > l4len { + return ip_only; + } + } else { + let udp_len = usize::from(u16::from_be_bytes([frame[14 + ihl + 4], frame[14 + ihl + 5]])); + if udp_len != l4len { + return ip_only; + } + transport_checksum_present = frame[14 + ihl + 6..14 + ihl + 8] != [0, 0]; + } + let pseudo = if mode & MODE_RX_NO_PSEUDO == 0 { ones_pseudo(&frame[26..34], proto, l4len) } else { 0 }; + let l4_sum = ones_sum(&frame[14 + ihl..14 + ihl + l4len], pseudo); + if l4_sum == 0xffff && transport_checksum_present && mode & MODE_RX_NO_PSEUDO == 0 { + *flags |= RXBDFLAG_TCP_UDP_CKSUM_OK; + } + ip_only | u32::from(l4_sum) +} + +impl State { + /// The engine state a reset returns to zero. + pub(super) fn reset_engines(&mut self) { + self.tx_cons = 0; + self.status_tx_cons = 0; + self.status_rx_prod = 0; + self.rx_prod_staged = 0; + self.rx_std_cons = 0; + self.rx_jumbo_cons = 0; + self.rx_shortage_drops = 0; + self.rx_std_shortage_drops = 0; + self.status_tag = 0; + self.rx_coal_pending = 0; + self.tx_coal_pending = 0; + self.rx_coal_expired = false; + self.tx_coal_expired = false; + self.rx_coal_timer = None; + self.tx_coal_timer = None; + self.rx_latency.clear(); + self.wire.clear(); + self.wake_rx = true; + self.purge_rx = true; + } + + /// Whatever made the chip ready again, a posted buffer or a consumer + /// index the driver moved, is the moment to offer the backend's waiting + /// frame again. + pub(super) fn retry_rx_flush(&mut self) { + self.wake_rx = true; + } + + // ---- host memory ---- + + /// The 64-bit host address a pair of control-block words names. The + /// driver word-swaps its host-address writes for the big-endian host via + /// ENDIAN_WORDSWAP; honour it at composition. + fn compose_hostaddr(&self, hi: u32, lo: u32) -> u64 { + if self.cfg.read(super::state::cfg::MISC_CTL) & misc_ctl::ENDIAN_WORDSWAP != 0 { + u64::from(lo) << 32 | u64::from(hi) + } else { + u64::from(hi) << 32 | u64::from(lo) + } + } + + /// The chip's DMA byte-swap controls, MODE_CTL bits 5 and 4 for frame data + /// and bits 2 and 1 for everything else. The chip is big-endian + /// internally with a 64-bit memory: its internal byte at lane l of a + /// doubleword (lane 0 the most significant) goes on PCI byte 7 - l with no + /// swap, and the swaps exchange the halves and reverse bytes within them. + /// The frame-data combinations are established by the programmer's guide; + /// the non-frame single-swap cases are inferred. A transfer's chip-side + /// lane is its PCI address modulo 8, an approximation: the chip's + /// internal buffer alignment is not modelled. + fn lane_xor(&self, frame: bool) -> usize { + let mode = self.rf(reg::MODE_CTL); + let (byte, word) = if frame { (0x10, 0x20) } else { (0x2, 0x4) }; + 7 ^ if mode & byte != 0 { 3 } else { 0 } ^ if mode & word != 0 { 4 } else { 0 } + } + + /// Read `len` bytes of the chip's internal image from `addr`. + fn dma_read_image(&self, addr: u64, len: usize, frame: bool) -> Option> { + if len == 0 { + return Some(Vec::new()); + } + let t = self.lane_xor(frame); + let astart = addr & !7; + let lead = (addr - astart) as usize; + let span = (lead + len + 7) & !7; + let mut raw = vec![0; span]; + if !self.dma.dma_read(astart, &mut raw).is_ok() { + return None; + } + Some( + (0..len) + .map(|j| { + let p = lead + j; + raw[(p & !7) + ((p & 7) ^ t)] + }) + .collect(), + ) + } + + /// Write `data` of the chip's internal image to `addr`, touching only the + /// lanes it covers as the byte enables do: whole doublewords as one + /// transfer, the partial ends a byte at a time. + fn dma_write_image(&self, addr: u64, data: &[u8], frame: bool) -> bool { + let t = self.lane_xor(frame); + let mut i = 0; + while i < data.len() { + let a = addr + i as u64; + if a & 7 == 0 && data.len() - i >= 8 { + let run = (data.len() - i) & !7; + let mut raw = vec![0; run]; + for j in 0..run { + raw[(j & !7) + ((j & 7) ^ t)] = data[i + j]; + } + if !self.dma.dma_write(a, &raw).is_ok() { + return false; + } + i += run; + } else { + let at = (a & !7) + ((a & 7) as usize ^ t) as u64; + if !self.dma.dma_write(at, &data[i..i + 1]).is_ok() { + return false; + } + i += 1; + } + } + true + } + + // ---- status block and coalescing ---- + + /// DMA the status block to the host address the driver programmed, only + /// while host coalescing is enabled. The chip owns the block, so it is + /// built whole each time. `raise` marks an update the driver is meant to + /// be interrupted for. Returns whether it reached the host. + fn post_status_block(&mut self, raise: bool) -> bool { + if self.rf(reg::HCC_MODE) & HCC_MODE_ENABLE == 0 { + return false; + } + let base = self.compose_hostaddr(self.rf(reg::HCC_STATUSBLK_ADDR_HI), self.rf(reg::HCC_STATUSBLK_ADDR_LO)); + let next_tag = self.status_tag.wrapping_add(1); + let mut blk = [0u8; STATUS_BLK_SIZE]; + st32(&mut blk, 0, STATFLAG_UPDATED); + st32(&mut blk, STATUS_TAG_OFF, u32::from(next_tag)); + blk[STATUS_RX_STD_CONS_IMAGE_OFF..][..2].copy_from_slice(&self.rx_std_cons.to_be_bytes()); + blk[STATUS_RX_JUMBO_CONS_IMAGE_OFF..][..2].copy_from_slice(&self.rx_jumbo_cons.to_be_bytes()); + st16(&mut blk, STATUS_TX_CONS_OFF, self.status_tx_cons); + st16(&mut blk, STATUS_RX_PROD_OFF, self.status_rx_prod); + if !self.dma_write_image(base, &blk, false) { + self.refuse(Refused::DmaFault(DmaWhat::StatusBlock)); + return false; + } + self.status_tag = next_tag; + // The block now carries an update the driver has not taken. The line + // asserts while that latch is set and interrupts are unmasked, and + // drops when the driver acknowledges through the interrupt mailbox. + if raise { + self.irq_pending = true; + self.update_irq(false); + } + true + } + + fn coal_arm_one(&mut self, rx: bool) { + let ticks = self.rf(if rx { reg::HCC_RX_COAL_TICKS } else { reg::HCC_TX_COAL_TICKS }); + let deadline = (ticks != 0).then(|| self.now + COAL_TICK * ticks); + if rx { + self.rx_coal_expired = false; + self.rx_coal_timer = deadline; + } else { + self.tx_coal_expired = false; + self.tx_coal_timer = deadline; + } + } + + /// Both budgets restart together: one update reports every ring. + fn coal_arm(&mut self) { + self.coal_arm_one(true); + self.coal_arm_one(false); + } + + /// One host update. All the ring indices are in one status block, so + /// whichever criterion fired reports every ring and restarts every count + /// and timer. + fn status_update(&mut self) -> bool { + if !self.post_status_block(true) { + return false; + } + self.rx_coal_pending = 0; + self.tx_coal_pending = 0; + self.coal_arm(); + true + } + + /// Retry later instead of spinning on an address the guest has not + /// repaired. + fn coal_retry(&mut self, rx: bool) { + let ticks = self.rf(if rx { reg::HCC_RX_COAL_TICKS } else { reg::HCC_TX_COAL_TICKS }); + let wait = (COAL_TICK * ticks.max(1)).min(COAL_RETRY_MAX); + let deadline = Some(self.now + wait); + if rx { + self.rx_coal_timer = deadline; + } else { + self.tx_coal_timer = deadline; + } + } + + fn rx_coal_flush(&mut self) { + if self.rx_coal_pending == 0 || self.status_update() { + return; + } + self.coal_retry(true); + } + + /// The receive budget ran out. It reports whatever arrived since the last + /// update; with nothing to report it stays expired, and the next frame is + /// reported as it lands. + pub(super) fn rx_coal_timer_fired(&mut self) { + self.rx_coal_timer = None; + self.rx_coal_expired = true; + self.rx_coal_flush(); + } + + fn tx_coal_flush(&mut self) { + if self.tx_coal_pending == 0 || self.status_update() { + return; + } + self.coal_retry(false); + } + + pub(super) fn tx_coal_timer_fired(&mut self) { + self.tx_coal_timer = None; + self.tx_coal_expired = true; + self.tx_coal_flush(); + } + + /// Count the descriptors this doorbell completed, and report them once + /// the count criterion is met or the send budget has already run out. + /// + /// A doorbell that completes nothing is reported anyway, which no + /// criterion covers: bring-up leans on that update arriving. This is a + /// known deviation from the coalescing contract, not an established + /// hardware behaviour. + fn tx_coalesce(&mut self, completed: usize) { + let max_bds = self.rf(reg::HCC_TX_MAX_COAL_BDS); + if completed == 0 { + self.status_update(); + return; + } + self.tx_coal_pending = self.tx_coal_pending.wrapping_add(completed as u16); + if (max_bds != 0 && u32::from(self.tx_coal_pending) >= max_bds) || self.tx_coal_expired { + self.tx_coal_flush(); + } + } + + /// A received frame filled a return-ring descriptor. + fn rx_coalesce(&mut self) { + let max_bds = self.rf(reg::HCC_RX_MAX_COAL_BDS); + self.rx_coal_pending = self.rx_coal_pending.wrapping_add(1); + if (max_bds != 0 && u32::from(self.rx_coal_pending) >= max_bds) || self.rx_coal_expired { + self.rx_coal_flush(); + } + } + + pub(super) fn hcc_mode_written(&mut self) { + if self.rf(reg::HCC_MODE) & HCC_MODE_ENABLE != 0 { + // The budgets run from here until the first update reloads them. + self.coal_arm(); + self.retry_rx_flush(); + } else { + self.rx_coal_timer = None; + self.tx_coal_timer = None; + } + } + + /// A budget was reprogrammed. A running timer keeps the value it was + /// loaded with, since the chip reloads one only at a status-block update; + /// one that is not running starts here, because nothing else would. + pub(super) fn coal_ticks_written(&mut self, rx: bool) { + let (expired, running) = if rx { + (self.rx_coal_expired, self.rx_coal_timer.is_some()) + } else { + (self.tx_coal_expired, self.tx_coal_timer.is_some()) + }; + if self.rf(reg::HCC_MODE) & HCC_MODE_ENABLE == 0 || expired || running { + return; + } + self.coal_arm_one(rx); + } + + // ---- transmit ---- + + fn tx_rollback_to_frame(&mut self, frame_start: u16, completed: usize) { + self.tx_cons = frame_start; + if completed != 0 { + self.status_tx_cons = frame_start; + self.tx_coalesce(completed); + } + } + + /// Replace the frame's source address with a filter slot's, as the + /// descriptor asks. + fn tx_replace_source(&self, frame: &mut [u8], flags: u16) { + if flags & TXBDFLAG_INSERT_SRC_ADDR == 0 || frame.len() < 12 { + return; + } + let slot = u32::from((flags & TXBDFLAG_CHOOSE_SRC_ADDR) >> TXBDFLAG_SRC_ADDR_SHIFT); + let hi = self.rf(reg::MAC_ADDR_0_HIGH + slot * reg::MAC_ADDR_STRIDE); + let lo = self.rf(reg::MAC_ADDR_0_LOW + slot * reg::MAC_ADDR_STRIDE); + frame[6] = (hi >> 8) as u8; + frame[7] = hi as u8; + frame[8..12].copy_from_slice(&lo.to_be_bytes()); + } + + /// Cut one oversized TCP buffer into segments the peer will accept. The + /// guest hands down a single frame holding many segments' worth of + /// payload; each piece goes out with its own copy of the headers, its own + /// sequence number, and the finishing flags only on the last. Anything + /// that does not look like the IPv4 TCP frame this is for is sent + /// unchanged rather than guessed at. + fn tx_segment(&mut self, mut frame: Vec, mss: usize, flags: u16) { + let Some(LsoLayout { ihl, thl, hdr, payload }) = lso_layout(&frame, mss) else { + if flags & (TXBDFLAG_IP_CKSUM | TXBDFLAG_TCP_UDP_CKSUM) != 0 { + tx_offload_checksum(&mut frame, flags); + } + self.wire.push(frame); + return; + }; + let seq = be32(&frame, 14 + ihl + 4); + let ip_id = u16::from_be_bytes([frame[18], frame[19]]); + let mut off = 0; + while off < payload { + let piece = mss.min(payload - off); + let mut seg = frame[..hdr].to_vec(); + seg.extend_from_slice(&frame[hdr + off..hdr + off + piece]); + seg[16..18].copy_from_slice(&((ihl + thl + piece) as u16).to_be_bytes()); + seg[18..20].copy_from_slice(&ip_id.wrapping_add((off / mss) as u16).to_be_bytes()); + seg[14 + ihl + 4..14 + ihl + 8].copy_from_slice(&seq.wrapping_add(off as u32).to_be_bytes()); + if off + piece < payload { + // Only the final piece finishes the sender's message. + seg[14 + ihl + 13] &= !0x09; + } + seg[24..26].fill(0); + seg[14 + ihl + 16..14 + ihl + 18].fill(0); + checksum_calculate(&mut seg, true, true); + self.wire.push(seg); + off += piece; + } + } + + /// Ring the transmit doorbell: send every descriptor from the chip's + /// consumer up to the producer index the driver just wrote, then report + /// the new consumer through the status block. + /// + /// The producer is a descriptor index the driver reduces modulo the ring + /// length itself; the failure mode is an index past the ring, which would + /// leave the walk chasing a consumer it never reaches. An out-of-range + /// index is refused, and the walk is bounded by the ring length + /// regardless. A frame whose descriptors are not all there yet, or whose + /// memory cannot be read, is left to retry from its first descriptor. + pub(super) fn tx_kick(&mut self, prod_raw: u32) { + let prod = (prod_raw & 0xffff) as u16; + let ring_len = (self.sram_read(SRAM_SEND_RING_RCB + RCB_MAXLEN_FLAGS) >> 16) as u16; + if ring_len == 0 || prod >= ring_len { + self.refuse(Refused::ProducerOutOfRange(Mailbox::TxProducer)); + return; + } + let ring_base = self.compose_hostaddr( + self.sram_read(SRAM_SEND_RING_RCB + RCB_HOSTADDR_HI), + self.sram_read(SRAM_SEND_RING_RCB + RCB_HOSTADDR_LO), + ); + let mut frame: Vec = Vec::new(); + let mut walked = 0usize; + let mut completed = 0usize; + let mut frame_flags = 0u16; + let mut frame_start = self.tx_cons; + let mut frame_mss = 0usize; + let mut dropped = false; + while self.tx_cons != prod && walked < usize::from(ring_len) { + let Some(bd) = self.dma_read_image(ring_base + u64::from(self.tx_cons) * TXBD_SIZE, TXBD_SIZE as usize, false) + else { + self.refuse(Refused::DmaFault(DmaWhat::TxDescriptor)); + self.tx_rollback_to_frame(frame_start, completed); + return; + }; + let buf_addr = u64::from(be32(&bd, 0)) << 32 | u64::from(be32(&bd, 4)); + let len = usize::from(ld16(&bd, TXBD_LEN)); + let flags = ld16(&bd, TXBD_FLAGS); + if !dropped && frame.len() + len <= MAX_FRAME { + let Some(payload) = self.dma_read_image(buf_addr, len, true) else { + self.refuse(Refused::DmaFault(DmaWhat::TxPayload)); + self.tx_rollback_to_frame(frame_start, completed); + return; + }; + frame.extend_from_slice(&payload); + } else if !dropped { + // The frame overran the buffer. Drop the whole frame and stay + // dropping across the remaining fragments so a later one is + // not mistaken for a fresh packet; resume at the next END. + self.refuse(Refused::TxFrameTooLarge); + dropped = true; + } + frame_flags |= flags; + if frame_mss == 0 { + frame_mss = usize::from(ld16(&bd, TXBD_MSS)); + } + if flags & TXBDFLAG_END != 0 { + if !dropped && !frame.is_empty() { + let mut whole = std::mem::take(&mut frame); + if frame_mss == 0 { + self.tx_replace_source(&mut whole, frame_flags); + } + self.tx_segment(whole, frame_mss, frame_flags); + } + frame.clear(); + frame_flags = 0; + frame_mss = 0; + dropped = false; + } + self.tx_cons = (self.tx_cons + 1) % ring_len; + walked += 1; + if flags & TXBDFLAG_END != 0 { + frame_start = self.tx_cons; + completed = walked; + } + } + if self.tx_cons != frame_start { + self.tx_rollback_to_frame(frame_start, completed); + return; + } + self.status_tx_cons = prod; + self.tx_coalesce(completed); + } + + // ---- receive mailboxes ---- + + pub(super) fn rx_mailbox_written(&mut self, off: u32, val: u32) { + let index = (val & 0xffff) as u16; + match off { + reg::MBX_RX_JUMBO_PROD_LO => { + // Posting jumbo-ring buffers; guard the index like the others. + if index >= RX_JUMBO_RING_LEN { + self.refuse(Refused::ProducerOutOfRange(Mailbox::RxJumboProducer)); + return; + } + self.set_rf(off, val); + // A write that actually posts a buffer ends the shortage. + if index != self.rx_jumbo_cons { + self.rx_shortage_drops = 0; + } + } + reg::MBX_RX_STD_PROD_LO => { + let std_len = self.rf(reg::RX_STD_RCB_MAXLEN_FLAGS) >> 16; + if std_len == 0 || index >= RX_STD_RING_LEN { + self.refuse(Refused::ProducerOutOfRange(Mailbox::RxStandardProducer)); + return; + } + if index != self.rx_std_cons { + self.rx_std_shortage_drops = 0; + } + self.set_rf(off, val); + } + _ => { + // The driver's return-ring acknowledgement, same range guard. + let ret_len = (self.sram_read(SRAM_RX_RETURN_RCB + RCB_MAXLEN_FLAGS) >> 16) as u16; + if ret_len == 0 || index >= ret_len { + self.refuse(Refused::ProducerOutOfRange(Mailbox::RxReturnConsumer)); + return; + } + self.set_rf(off, val); + } + } + // Room where there was none: take anything the peer had waiting. + self.retry_rx_flush(); + } + + // ---- receive ---- + + /// Whether the jumbo ring can take a frame at all, independent of whether + /// one is posted: it needs a host base and must not be disabled. Extended + /// RX Enable asks for the 64-byte descriptor format this model does not + /// parse, so a ring asking for it is treated as disabled. Max_Len plays no + /// part: hardware ignores it for the jumbo ring. + fn jumbo_ring_enabled(&self) -> bool { + let flags = self.rf(reg::RX_JUMBO_RCB_MAXLEN_FLAGS); + let base = self.compose_hostaddr(self.rf(reg::RX_JUMBO_RCB_HADDR_HI), self.rf(reg::RX_JUMBO_RCB_HADDR_LO)); + base != 0 && flags & (RCB_FLAG_DISABLE_RING | RCB_FLAG_EXT_RX_ENABLE) == 0 + } + + /// Whether the engine can take a frame: the rings must be set up and host + /// coalescing enabled, and the host must have posted a buffer the chip has + /// not yet used. + pub(super) fn rx_ready(&self) -> bool { + let std_len = self.rf(reg::RX_STD_RCB_MAXLEN_FLAGS) >> 16; + let ret_len = (self.sram_read(SRAM_RX_RETURN_RCB + RCB_MAXLEN_FLAGS) >> 16) as u16; + let std_prod = (self.rf(reg::MBX_RX_STD_PROD_LO) & 0xffff) as u16; + // A completion the driver has not read yet must not be overwritten; + // when the next would land on the entry the driver has consumed up to, + // the ring is full and the frame is refused. + let ret_cons = (self.rf(reg::MBX_RX_CONS0_LO) & 0xffff) as u16; + // Either ring having a buffer posted is enough; which one a frame + // takes depends on its size. + let jumbo_prod = (self.rf(reg::MBX_RX_JUMBO_PROD_LO) & 0xffff) as u16; + let posted = (std_len != 0 && std_prod != self.rx_std_cons) + || (self.jumbo_ring_enabled() && jumbo_prod != self.rx_jumbo_cons); + if self.rx_latency.len() >= RX_LATENCY_DEPTH { + return false; + } + // Against the staged producer, not the published one: a return + // descriptor whose completion is still held has already been written. + ret_len != 0 + && posted + && self.rf(reg::HCC_MODE) & HCC_MODE_ENABLE != 0 + && (self.rx_prod_staged + 1) % ret_len != ret_cons + } + + /// The address the part filters on is MAC address filter slot 0, which the + /// driver programs, not the address the board gave the device: IRIX + /// installs the system Ethernet address over whatever the card came up + /// with. + fn frame_accepted(&self, frame: &[u8]) -> bool { + if frame.len() < 6 { + return false; + } + if frame[0] & 1 != 0 { + return true; + } + let hi = self.rf(reg::MAC_ADDR_0_HIGH); + let lo = self.rf(reg::MAC_ADDR_0_LOW); + frame[..6] == [(hi >> 8) as u8, hi as u8, (lo >> 24) as u8, (lo >> 16) as u8, (lo >> 8) as u8, lo as u8] + } + + fn count_no_more_rxbds(&mut self) { + let n = self.sram_read(SRAM_STATS_NO_MORE_RXBDS).wrapping_add(1); + self.sram_write(SRAM_STATS_NO_MORE_RXBDS, n); + } + + /// A frame from the backend. Not ready, or no buffer posted, defers it: + /// the only answer that leaves the frame at the peer's queue head. A frame + /// the address filter drops is consumed, never deferred, so the peer is + /// not wedged waiting on a frame that will never be taken. Otherwise the + /// frame is DMA'd into the next posted buffer, a completed descriptor is + /// written into the return ring, and the producer the status block reports + /// advances once the frame's arrival deadline passes. + pub(super) fn receive(&mut self, frame: &[u8]) -> RxOutcome { + // A halted receive MAC is not listening, so the frame is gone rather + // than held: a backend that queued it would hand the driver traffic + // from before the interface existed. + if self.rf(reg::RX_MODE) & 2 == 0 { + return RxOutcome::Consumed; + } + if !self.rx_ready() { + return RxOutcome::Deferred; + } + if !self.frame_accepted(frame) { + return RxOutcome::Consumed; + } + if frame.len() > MAX_FRAME { + // Drop it rather than deliver part of it: a frame cut short looks + // whole, and the failure surfaces wherever the truncated payload + // is finally parsed. + self.refuse(Refused::RxFrameTooLarge); + return RxOutcome::Consumed; + } + // Pad a short frame to the 60-byte Ethernet minimum: a backend can + // hand over a runt, and the driver rejects a descriptor below it. + let mut padded; + let buf: &[u8] = if frame.len() < MIN_FRAME { + padded = frame.to_vec(); + padded.resize(MIN_FRAME, 0); + &padded + } else { + frame + }; + let framelen = buf.len() as u32; + + // Pick the ring by what the frame fits in. The standard ring's Max_Len + // is the driver's statement of how much room its buffers hold; a + // larger frame goes to the jumbo ring when it is enabled, and is + // dropped rather than written past the end of someone's buffer when + // it is not. + let std_size = u32::from((self.rf(reg::RX_STD_RCB_MAXLEN_FLAGS) >> 16) as u16); + let std_prod = (self.rf(reg::MBX_RX_STD_PROD_LO) & 0xffff) as u16; + let jumbo_prod = (self.rf(reg::MBX_RX_JUMBO_PROD_LO) & 0xffff) as u16; + let jumbo_enabled = self.jumbo_ring_enabled(); + let (ring, ring_base, idx) = if framelen <= std_size { + if std_prod == self.rx_std_cons { + // The frame fits a standard buffer and none is posted; it + // cannot take a jumbo one, because a frame's ring is chosen by + // its length against the standard ring's Max_Len. Discard + // rather than refuse: refusing stops the backend reading and + // strands the frames behind it that the jumbo ring can carry. + // Only while a jumbo buffer is posted; without one the caller + // is owed the refusal. + if !jumbo_enabled || jumbo_prod == self.rx_jumbo_cons { + return RxOutcome::Deferred; + } + if self.rx_std_shortage_drops == 0 { + self.refuse(Refused::RxShortage(Ring::Standard)); + } + self.rx_std_shortage_drops += 1; + self.count_no_more_rxbds(); + return RxOutcome::Consumed; + } + let base = self + .compose_hostaddr(self.rf(reg::RX_STD_RCB_HADDR_HI), self.rf(reg::RX_STD_RCB_HADDR_LO)); + (Ring::Standard, base, self.rx_std_cons) + } else if jumbo_enabled { + if jumbo_prod == self.rx_jumbo_cons { + // A frame longer than the standard Max_Len consumes a + // descriptor from the jumbo ring, and nothing is posted. + // Discard rather than refuse, for the reason above, and only + // while a standard buffer is posted. + if std_size == 0 || std_prod == self.rx_std_cons { + return RxOutcome::Deferred; + } + if self.rx_shortage_drops == 0 { + self.refuse(Refused::RxShortage(Ring::Jumbo)); + } + self.rx_shortage_drops += 1; + self.count_no_more_rxbds(); + return RxOutcome::Consumed; + } + let base = self + .compose_hostaddr(self.rf(reg::RX_JUMBO_RCB_HADDR_HI), self.rf(reg::RX_JUMBO_RCB_HADDR_LO)); + (Ring::Jumbo, base, self.rx_jumbo_cons) + } else { + self.refuse(Refused::RxNoRoom(Ring::Standard)); + return RxOutcome::Consumed; + }; + + let ret_len = (self.sram_read(SRAM_RX_RETURN_RCB + RCB_MAXLEN_FLAGS) >> 16) as u16; + let ret_base = self.compose_hostaddr( + self.sram_read(SRAM_RX_RETURN_RCB + RCB_HOSTADDR_HI), + self.sram_read(SRAM_RX_RETURN_RCB + RCB_HOSTADDR_LO), + ); + + // The posted buffer's host address, from the chosen ring's descriptor. + let Some(std_bd) = self.dma_read_image(ring_base + u64::from(idx) * RXBD_SIZE, RXBD_SIZE as usize, false) + else { + self.refuse(Refused::DmaFault(DmaWhat::RxDescriptor)); + return RxOutcome::Deferred; + }; + let buf_addr = u64::from(be32(&std_bd, 0)) << 32 | u64::from(be32(&std_bd, 4)); + let mut rx_flags = RXBDFLAG_END; + if ring == Ring::Jumbo { + // The jumbo ring carries no ring-wide buffer size; the driver + // states each buffer's size in the descriptor it posts. + let buf_len = be32(&std_bd, RXBD_IDX_LEN) & 0xffff; + if framelen > buf_len { + self.refuse(Refused::RxNoRoom(Ring::Jumbo)); + return RxOutcome::Consumed; + } + rx_flags |= RXBDFLAG_JUMBO_RING; + } + if !self.dma_write_image(buf_addr, buf, true) { + self.refuse(Refused::DmaFault(DmaWhat::RxPayload)); + return RxOutcome::Deferred; + } + let csum = rx_offload_checksum(buf, self.rf(reg::MODE_CTL), &mut rx_flags); + + let mut ret_bd = [0u8; RXBD_SIZE as usize]; + st32(&mut ret_bd, 0, (buf_addr >> 32) as u32); + st32(&mut ret_bd, 4, buf_addr as u32); + // The length includes the four-byte FCS the driver strips. + st32(&mut ret_bd, RXBD_IDX_LEN, u32::from(idx) << 16 | ((framelen + FCS_LEN) & 0xffff)); + st32(&mut ret_bd, RXBD_TYPE_FLAGS, rx_flags); + st32(&mut ret_bd, RXBD_CSUM, csum); + st32(&mut ret_bd, RXBD_OPAQUE, be32(&std_bd, RXBD_OPAQUE)); + if !self.dma_write_image(ret_base + u64::from(self.rx_prod_staged) * RXBD_SIZE, &ret_bd, false) { + self.refuse(Refused::DmaFault(DmaWhat::RxCompletion)); + return RxOutcome::Deferred; + } + match ring { + Ring::Jumbo => self.rx_jumbo_cons = (self.rx_jumbo_cons + 1) % RX_JUMBO_RING_LEN, + Ring::Standard => self.rx_std_cons = (self.rx_std_cons + 1) % RX_STD_RING_LEN, + } + self.rx_prod_staged = (self.rx_prod_staged + 1) % ret_len; + // The frame and its return descriptor are in guest memory; the + // producer the status block reports, and the interrupt after it, wait + // for the frame's arrival deadline. + self.rx_hold(self.rx_prod_staged); + RxOutcome::Consumed + } + + // ---- the arrival line ---- + + fn rx_publish(&mut self, status_rx_prod: u16) { + self.status_rx_prod = status_rx_prod; + self.rx_coalesce(); + } + + /// Hold a completion until the frame's minimum arrival latency has passed. + fn rx_hold(&mut self, status_rx_prod: u16) { + if self.rx_floor.is_zero() && self.rx_latency.is_empty() { + self.rx_publish(status_rx_prod); + } else { + self.rx_latency.push_back((self.now + self.rx_floor, status_rx_prod)); + } + } + + /// The head completion's deadline passed. + pub(super) fn rx_arrival_due(&mut self) { + let mut published = false; + while self.rx_latency.front().is_some_and(|(d, _)| *d <= self.now) { + let (_, prod) = self.rx_latency.pop_front().unwrap(); + self.rx_publish(prod); + published = true; + } + if published { + self.retry_rx_flush(); + } + } +} diff --git a/src/dev/bcm5701/eeprom.rs b/src/dev/bcm5701/eeprom.rs new file mode 100644 index 000000000..b56868226 --- /dev/null +++ b/src/dev/bcm5701/eeprom.rs @@ -0,0 +1,113 @@ +//! The BCM5701's serial EEPROM, behind the controller at registers 0x6838 +//! (address and command), 0x683c (data) and 0x6840 (direct serial pins). +//! +//! The controller transfers one aligned 32-bit word at a time. The backing is +//! caller-supplied: nothing here synthesises a signature, a bootstrap, a CRC +//! or a station address, because no complete board image is established. A +//! write goes through to the backing file when there is one, so contents +//! persist across runs. Transfers complete synchronously and the serial bus +//! timing, including the 10 ms spacing between writes, is not modelled. +//! +//! Ported from hw/net/bcm570x.c in jamesbraid/qemu-sgi-origami. + +use std::path::PathBuf; + +pub const MIN_SIZE: usize = 4; +pub const MAX_SIZE: usize = 65536; + +/// Why a transfer did not happen. The register file keeps Complete clear and +/// the data register unchanged; the caller counts and logs the reason. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub enum Fault { + /// No backing is configured, whatever the device ID. + NoBacking, + /// A different device ID than the one configured. + WrongDevice(u8), + /// The word lies beyond the backing. + OutOfRange(u32), + ReadOnly, + /// The backing file could not be written. + Io, +} + +#[derive(Debug)] +pub struct Eeprom { + device_id: u8, + data: Vec, + path: Option, + read_only: bool, +} + +impl Eeprom { + /// An in-memory backing. `device_id` is 0..=7; the image is 4 to 65536 + /// bytes in a multiple of four. + pub fn from_image(device_id: u8, data: Vec) -> Result { + if device_id > 7 { + return Err("bcm5701: eeprom requires a device id in 0..=7".into()); + } + if data.len() < MIN_SIZE || data.len() > MAX_SIZE || data.len() % 4 != 0 { + return Err("bcm5701: EEPROM backing must be 4 to 65536 bytes and a multiple of four".into()); + } + Ok(Self { device_id, data, path: None, read_only: false }) + } + + /// A backing read from `path` and written back to it. A file the + /// filesystem marks read-only makes a read-only backing. + pub fn from_file(device_id: u8, path: PathBuf) -> Result { + let data = std::fs::read(&path).map_err(|e| format!("bcm5701: EEPROM {}: {e}", path.display()))?; + let read_only = std::fs::metadata(&path).map(|m| m.permissions().readonly()).unwrap_or(true); + let mut eeprom = Self::from_image(device_id, data)?; + eeprom.read_only = read_only; + eeprom.path = Some(path); + Ok(eeprom) + } + + /// Refuse writes, as a backing opened read-only does. + pub fn set_read_only(&mut self, read_only: bool) { + self.read_only = read_only; + } + + pub fn device_id(&self) -> u8 { + self.device_id + } + + pub fn is_read_only(&self) -> bool { + self.read_only + } + + pub fn bytes(&self) -> &[u8] { + &self.data + } + + fn word_offset(&self, device: u8, word: u32) -> Result { + if device != self.device_id { + return Err(Fault::WrongDevice(device)); + } + let offset = word as usize * 4; + if offset + 4 > self.data.len() { + return Err(Fault::OutOfRange(word)); + } + Ok(offset) + } + + /// The word at `word`, least significant byte first: the image bytes + /// `66 99 55 aa` read as `0xaa559966`. + pub fn read_word(&self, device: u8, word: u32) -> Result { + let offset = self.word_offset(device, word)?; + Ok(u32::from_le_bytes(self.data[offset..offset + 4].try_into().unwrap())) + } + + pub fn write_word(&mut self, device: u8, word: u32, value: u32) -> Result<(), Fault> { + let offset = self.word_offset(device, word)?; + if self.read_only { + return Err(Fault::ReadOnly); + } + let mut image = self.data.clone(); + image[offset..offset + 4].copy_from_slice(&value.to_le_bytes()); + if let Some(path) = &self.path { + crate::nv_storage::save(&path.to_string_lossy(), &image).map_err(|_| Fault::Io)?; + } + self.data = image; + Ok(()) + } +} diff --git a/src/dev/bcm5701/phy.rs b/src/dev/bcm5701/phy.rs new file mode 100644 index 000000000..f6a37eed8 --- /dev/null +++ b/src/dev/bcm5701/phy.rs @@ -0,0 +1,263 @@ +//! The BCM570x's integrated copper PHY, reached through the MAC's MI_COMM +//! register. +//! +//! The PHY has no bus address of its own: software writes a register number +//! into MI_COMM and reads a word back. The contract is the ordinary MII/GMII +//! one the public brgphy driver drives. Link is modelled without a cable: out +//! of reset the PHY negotiates, and autonegotiation completes a fixed settle +//! later, which restarts when the guest restarts it. Reported link is also +//! gated on the backend's cable state. +//! +//! The identity decodes to Broadcom's organisational identifier and the +//! BCM5701 model, the pair brgphy matches to bind. +//! +//! Ported from hw/net/bcm570x_phy.c in jamesbraid/qemu-sgi-origami. + +use std::time::{Duration, Instant}; + +pub mod reg { + pub const BMCR: u8 = 0x00; + pub const BMSR: u8 = 0x01; + pub const IDR1: u8 = 0x02; + pub const IDR2: u8 = 0x03; + pub const ANAR: u8 = 0x04; + pub const ANLPAR: u8 = 0x05; + pub const GTCR: u8 = 0x09; + pub const GTSR: u8 = 0x0a; + pub const EXTSR: u8 = 0x0f; + /// Broadcom auxiliary status summary. + pub const AUXSTS: u8 = 0x19; +} + +pub const IDR1_VALUE: u16 = 0x0020; +pub const IDR2_BCM5701: u16 = 0x6110; + +pub mod bmcr { + pub const RESET: u16 = 0x8000; + pub const AUTOEN: u16 = 0x1000; + pub const ISOLATE: u16 = 0x0400; + pub const STARTNEG: u16 = 0x0200; + pub const FDX: u16 = 0x0100; + pub const SPEED1000: u16 = 0x0040; +} +mod bmsr { + pub const TX100_FDX: u16 = 0x4000; + pub const TX100_HDX: u16 = 0x2000; + pub const T10_FDX: u16 = 0x1000; + pub const T10_HDX: u16 = 0x0800; + pub const EXTSTAT: u16 = 0x0100; + pub const ANEGDONE: u16 = 0x0020; + pub const CANAUTONEG: u16 = 0x0008; + pub const LINK: u16 = 0x0004; + pub const EXTCAP: u16 = 0x0001; +} +mod anar { + pub const TX100_FDX: u16 = 0x0100; + pub const TX100_HDX: u16 = 0x0080; + pub const T10_FDX: u16 = 0x0040; + pub const T10_HDX: u16 = 0x0020; + pub const CSMA: u16 = 0x0001; + pub const ACK: u16 = 0x4000; +} +const GTCR_ADV_1000TFDX: u16 = 0x0200; +const GTSR_LINKED: u16 = 0x4000 | 0x2000 | 0x1000 | 0x0800; +const EXTSR_1000: u16 = 0x2000 | 0x1000; +const AUXSTS_UP: u16 = 0x8000 | 0x0700; + +const BMCR_RESET_VALUE: u16 = bmcr::AUTOEN | bmcr::FDX | bmcr::SPEED1000; +const ANAR_RESET_VALUE: u16 = + anar::CSMA | anar::T10_HDX | anar::T10_FDX | anar::TX100_HDX | anar::TX100_FDX; + +/// How long autonegotiation takes to settle. It has to finish inside the +/// driver's own patience: the IRIX driver gives up its link wait well before +/// a second. +pub const SETTLE: Duration = Duration::from_millis(100); + +/// What the PHY could not carry out, for the owner to count and log. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub enum Refusal { + /// A read of a register the model does not implement; it reads 0, never + /// the all-ones an absent PHY returns. + ReadUnimplemented(u8), + WriteUnimplemented(u8), + /// A write to a read-only register. + WriteReadOnly(u8), +} + +pub struct Phy { + bmcr: u16, + anar: u16, + gtcr: u16, + autoneg_done: bool, + link_down: bool, + settle_armed: bool, + settle: Option, + pub refused: Vec, +} + +impl Phy { + pub fn new(now: Instant) -> Self { + let mut phy = Self { + bmcr: 0, + anar: 0, + gtcr: 0, + autoneg_done: false, + link_down: false, + settle_armed: false, + settle: None, + refused: Vec::new(), + }; + phy.reset(now); + phy + } + + /// The deadline at which the settle completes. + pub fn deadline(&self) -> Option { + self.settle + } + + /// Complete the settle if its time has come. + pub fn sync(&mut self, now: Instant) { + if self.settle.is_some_and(|d| d <= now) { + self.settle = None; + self.autoneg_done = true; + } + } + + pub fn autoneg_done(&self) -> bool { + self.autoneg_done + } + + pub fn link_up(&self) -> bool { + self.autoneg_done && !self.link_down + } + + /// The far end's cable state; not a register, so a PHY reset leaves it. + pub fn set_link_down(&mut self, down: bool) { + self.link_down = down; + } + + pub fn link_down(&self) -> bool { + self.link_down + } + + fn status(&self) -> u16 { + let mut status = bmsr::EXTCAP + | bmsr::CANAUTONEG + | bmsr::EXTSTAT + | bmsr::TX100_FDX + | bmsr::TX100_HDX + | bmsr::T10_FDX + | bmsr::T10_HDX; + if self.link_up() { + status |= bmsr::LINK | bmsr::ANEGDONE; + } + status + } + + /// A real PHY negotiates as soon as autonegotiation is enabled, so the + /// settle runs from reset and from the write that restarts it, not from + /// the driver's first look at link status. + fn arm_settle(&mut self, now: Instant) { + if !self.settle_armed && !self.autoneg_done && self.bmcr & bmcr::AUTOEN != 0 { + self.settle_armed = true; + self.settle = Some(now + SETTLE); + } + } + + fn refuse(&mut self, what: Refusal) { + if self.refused.len() < 64 { + self.refused.push(what); + } + } + + pub fn read(&mut self, register: u8, now: Instant) -> u16 { + self.sync(now); + let up = self.link_up(); + match register { + reg::BMCR => self.bmcr, + reg::BMSR => { + self.arm_settle(now); + self.status() + } + reg::IDR1 => IDR1_VALUE, + reg::IDR2 => IDR2_BCM5701, + reg::ANAR => self.anar, + // The far end advertises exactly what this end does, which puts + // the driver on the route it trusts. + reg::ANLPAR => { + if up { + self.anar | anar::ACK + } else { + 0 + } + } + reg::GTCR => self.gtcr, + reg::GTSR => { + if up { + GTSR_LINKED + } else { + 0 + } + } + reg::EXTSR => EXTSR_1000, + // The register the driver gates its link-up path on, so a poll + // that reads only this one still brings the link up. + reg::AUXSTS => { + self.arm_settle(now); + if self.link_up() { + AUXSTS_UP + } else { + 0 + } + } + other => { + self.refuse(Refusal::ReadUnimplemented(other)); + 0 + } + } + } + + pub fn write(&mut self, register: u8, value: u16, now: Instant) { + self.sync(now); + match register { + reg::BMCR => { + // Reset and restart are self-clearing. + if value & bmcr::RESET != 0 { + self.reset(now); + return; + } + // Only a restart (or isolating the part) restarts + // negotiation: a driver rewrites this register for other + // reasons, and dropping a working link each time would show + // as the carrier coming back again and again. + if value & (bmcr::ISOLATE | bmcr::STARTNEG) != 0 { + self.autoneg_done = false; + self.settle_armed = false; + self.settle = None; + } + self.bmcr = value & !(bmcr::RESET | bmcr::STARTNEG); + if value & bmcr::ISOLATE == 0 { + self.arm_settle(now); + } + } + reg::ANAR => self.anar = value, + reg::GTCR => self.gtcr = value, + reg::BMSR | reg::IDR1 | reg::IDR2 | reg::ANLPAR | reg::GTSR | reg::EXTSR => { + self.refuse(Refusal::WriteReadOnly(register)); + } + other => self.refuse(Refusal::WriteUnimplemented(other)), + } + } + + pub fn reset(&mut self, now: Instant) { + self.bmcr = BMCR_RESET_VALUE; + self.anar = ANAR_RESET_VALUE; + self.gtcr = GTCR_ADV_1000TFDX; + self.autoneg_done = false; + self.settle_armed = false; + self.settle = None; + // Out of reset autonegotiation is enabled, so the part starts it. + self.arm_settle(now); + } +} diff --git a/src/dev/bcm5701/state.rs b/src/dev/bcm5701/state.rs new file mode 100644 index 000000000..633b7b0cc --- /dev/null +++ b/src/dev/bcm5701/state.rs @@ -0,0 +1,665 @@ +//! The chip's register file, on-chip SRAM, interrupt line and the engines +//! that run inside register writes. The PCI side and the threads are in the +//! parent module. + +use std::collections::{BTreeMap, VecDeque}; +use std::sync::Arc; +use std::time::{Duration, Instant}; + +use super::eeprom::{self, Eeprom}; +use super::phy::{self, Phy}; +use super::dma::{Ring, Timer}; +use crate::pci::{PciConfig, PciDma}; + +pub const REG_FILE_SIZE: u32 = 0x8000; +pub const SRAM_SIZE: u32 = 0x8000; +const REG_WORDS: usize = (REG_FILE_SIZE / 4) as usize; +const SRAM_WORDS: usize = (SRAM_SIZE / 4) as usize; + +/// Dword indices of the PCI configuration registers the chip owns. +pub mod cfg { + pub const MISC_CTL: usize = 0x68 / 4; + pub const PCISTATE: usize = 0x70 / 4; + pub const REG_BASEADDR: usize = 0x78 / 4; + pub const MEMWIN_BASEADDR: usize = 0x7c / 4; + pub const REG_DATA: usize = 0x80 / 4; + pub const MEMWIN_DATA: usize = 0x84 / 4; +} + +pub mod misc_ctl { + pub const CLEAR_INTA: u32 = 0x1; + pub const MASK_PCI_INTR: u32 = 0x2; + pub const ENDIAN_BYTESWAP: u32 = 0x4; + pub const ENDIAN_WORDSWAP: u32 = 0x8; + /// The chip id, bits 31:16, is read-only; the low half is writable. + pub const WRITABLE: u32 = 0xffff; +} + +pub mod pcistate { + pub const INTR_NOT_ACTIVE: u32 = 0x2; + pub const BUSMODE_PCI: u32 = 0x4; + /// The chip did not see REQ64# at the end of reset. + pub const BUS_32BIT: u32 = 0x10; +} + +/// The chip id MISC_CTL[31:16] presents; the driver reads the part from it. +pub const CHIPID_BCM5701: u32 = 0x0105; + +/// Register-file offsets with modelled behaviour. +pub mod reg { + pub const MAC_MODE: u32 = 0x0400; + pub const MAC_STATUS: u32 = 0x0404; + pub const MAC_ADDR_0_HIGH: u32 = 0x0410; + pub const MAC_ADDR_0_LOW: u32 = 0x0414; + pub const MAC_ADDR_STRIDE: u32 = 8; + pub const MI_COMM: u32 = 0x044c; + pub const MI_STS: u32 = 0x0450; + pub const RX_MODE: u32 = 0x0468; + pub const HCC_MODE: u32 = 0x3c00; + pub const BUFMGR_MODE: u32 = 0x4400; + pub const BUFMGR_STATUS: u32 = 0x4404; + pub const FW_PROGRESS: u32 = 0x5034; + pub const MODE_CTL: u32 = 0x6800; + pub const MISC_CFG: u32 = 0x6804; + pub const EEPROM_ADDR: u32 = 0x6838; + pub const EEPROM_DATA: u32 = 0x683c; + pub const EEPROM_PIN: u32 = 0x6840; + pub const EEPROM_DELAY: u32 = 0x6848; + pub const MBX_IRQ0_LO: u32 = 0x0204; + pub const MBX_RX_STD_PROD_LO: u32 = 0x026c; + pub const MBX_RX_JUMBO_PROD_LO: u32 = 0x0274; + pub const MBX_RX_CONS0_LO: u32 = 0x0284; + pub const MBX_TX_PROD_LO: u32 = 0x0304; + pub const HCC_RX_COAL_TICKS: u32 = 0x3c08; + pub const HCC_TX_COAL_TICKS: u32 = 0x3c0c; + pub const HCC_RX_MAX_COAL_BDS: u32 = 0x3c10; + pub const HCC_TX_MAX_COAL_BDS: u32 = 0x3c14; + pub const HCC_STATUSBLK_ADDR_HI: u32 = 0x3c38; + pub const HCC_STATUSBLK_ADDR_LO: u32 = 0x3c3c; + pub const RX_JUMBO_RCB_HADDR_HI: u32 = 0x2440; + pub const RX_JUMBO_RCB_HADDR_LO: u32 = 0x2444; + pub const RX_JUMBO_RCB_MAXLEN_FLAGS: u32 = 0x2448; + pub const RX_STD_RCB_HADDR_HI: u32 = 0x2450; + pub const RX_STD_RCB_HADDR_LO: u32 = 0x2454; + pub const RX_STD_RCB_MAXLEN_FLAGS: u32 = 0x2458; +} + +/// Bit 0 of a mode register is a self-clearing reset; bit 0 of MISC_CFG +/// resets the whole chip. +const MODE_RESET: u32 = 1; +const MISCCFG_RESET_CORE_CLOCKS: u32 = 1; +const RXMODE_ENABLE: u32 = 2; +const RCB_FLAG_EXT_RX_ENABLE: u32 = 1; + +const BUFMGR_MODE_ENABLE: u32 = 2; +/// The buffer manager's enable reads back only after its startup; a +/// functional approximation inside the driver's 10 ms polling budget. +const BUFMGR_START: Duration = Duration::from_millis(8); + +mod eeprom_cmd { + pub const READ: u32 = 1 << 31; + pub const COMPLETE: u32 = 1 << 30; + pub const RESET: u32 = 1 << 29; + pub const DEVICE_ID: u32 = 7 << 26; + pub const START: u32 = 1 << 25; + pub const WORD_ADDR: u32 = 0x3fff << 2; +} + +/// The firmware-ready handshake. The driver writes the magic to SRAM 0x0b50 +/// and polls for its complement, the on-chip bootcode's ready signal. The +/// mailbox returns the complement of the last value written, so the magic +/// reads back as ready whether the driver writes it before the reset or after. +pub const SRAM_FW_MBOX: u32 = 0x0b50; +pub const FW_MAGIC: u32 = 0x4b65_7654; +/// The firmware-progress register holds the all-ones "no progress yet" +/// sentinel, so a driver polling it sees "unchanged" and keeps polling the +/// mailbox. +const FW_PROGRESS_INIT: u32 = 0xffff_ffff; + +/// The station address the on-chip bootcode leaves where a driver looks for +/// it: SRAM 0x0c14 tagged 0x484b in its high half, and MAC address filter +/// slot 0. +const SRAM_MAC_HI: u32 = 0x0c14; +const SRAM_MAC_LO: u32 = 0x0c18; +const MAC_TAG: u32 = 0x484b; + +mod mi { + pub const DATA: u32 = 0x0000_ffff; + pub const REG: u32 = 0x001f_0000; + pub const REG_SHIFT: u32 = 16; + /// Command and address fields, kept across a read. + pub const READ_FIELDS: u32 = 0x0fff_0000; + pub const WRITE: u32 = 0x0400_0000; + pub const READ: u32 = 0x0800_0000; + pub const BUSY: u32 = 0x2000_0000; + pub const LINK: u32 = 1; +} + +/// What the model could not carry out, counted and logged on first sight so +/// an unexpected access is never silent. +#[derive(Clone, Copy, Debug, PartialEq, Eq, PartialOrd, Ord)] +pub enum Refused { + /// A write to a register with no modelled behaviour; the value is stored + /// and reads back. + UnmodelledWrite(u32), + /// A write to the read-only buffer-manager status. + BufmgrStatusWrite, + /// Direct EEPROM serial-pin control. + EepromPins, + EepromFault(eeprom::Fault), + /// An MI_COMM opcode other than read and write. + MiOpcode(u32), + /// A BAR access of a width or alignment the chip does not accept. + BarAccess { offset: u32, width: u8, write: bool }, + Phy(phy::Refusal), + /// A transfer to or from host memory that did not complete. + DmaFault(super::dma::DmaWhat), + /// A ring index written to a mailbox that lies outside the ring. + ProducerOutOfRange(super::dma::Mailbox), + /// A transmit frame larger than the model carries; it is dropped. + TxFrameTooLarge, + /// A received frame larger than the model carries; it is dropped. + RxFrameTooLarge, + /// A frame discarded for want of a posted descriptor, once per drought. + RxShortage(Ring), + /// A frame too big for the ring it would use; it is dropped. + RxNoRoom(Ring), + /// The jumbo ring asks for 64-byte descriptors, which are not modelled; + /// the ring is treated as unavailable. + JumboExtendedDescriptors, +} + +pub(super) struct State { + pub cfg: Arc, + pub mac: [u8; 6], + pub(super) regfile: Box<[u32; REG_WORDS]>, + sram: Box<[u32; SRAM_WORDS]>, + /// The two indirect-window base registers (config 0x78 and 0x7c). + pub reg_win_base: u32, + pub mem_win_base: u32, + eeprom: Option, + bufmgr_start: Option, + /// The mask set through the 0x0204 mailbox, and the latch a status-block + /// update sets and the driver clears. The line is the latch with + /// neither mask (this or MISC_CTL's) set. + pub(super) irq_masked: bool, + pub(super) irq_pending: bool, + irq_level: bool, + pub irq_events: Vec, + pub phy: Phy, + pub now: Instant, + refused: BTreeMap, + pub(super) dma: Arc, + /// The send-ring consumer, and what the status block reports. + pub(super) tx_cons: u16, + pub(super) status_tx_cons: u16, + pub(super) status_rx_prod: u16, + /// Where the next return descriptor goes; it runs ahead of + /// `status_rx_prod` while the arrival line holds a completion. + pub(super) rx_prod_staged: u16, + pub(super) rx_std_cons: u16, + pub(super) rx_jumbo_cons: u16, + /// Frames discarded since a ring's shortage began, so a drought is + /// reported at both ends rather than once per frame. + pub(super) rx_shortage_drops: u64, + pub(super) rx_std_shortage_drops: u64, + pub(super) status_tag: u8, + /// Descriptors consumed on each ring since the last status-block update, + /// and whether each coalescing budget has run out. + pub(super) rx_coal_pending: u16, + pub(super) tx_coal_pending: u16, + pub(super) rx_coal_expired: bool, + pub(super) tx_coal_expired: bool, + pub(super) rx_coal_timer: Option, + pub(super) tx_coal_timer: Option, + /// Completions held until their frame's arrival deadline, with the + /// producer index each publishes. + pub(super) rx_latency: VecDeque<(Instant, u16)>, + pub(super) rx_floor: Duration, + /// Frames the transmit engine produced, for the owner to send. + pub wire: Vec>, + /// Set when something the guest did can unblock a waiting frame. + pub wake_rx: bool, + /// A reset discards the frames the backend still holds for the guest. + pub purge_rx: bool, +} + +impl State { + pub fn new( + cfg: Arc, + mac: [u8; 6], + eeprom: Option, + rx_floor: Duration, + now: Instant, + ) -> Self { + let mut st = Self { + cfg, + mac, + regfile: Box::new([0; REG_WORDS]), + sram: Box::new([0; SRAM_WORDS]), + reg_win_base: 0, + mem_win_base: 0, + eeprom, + bufmgr_start: None, + irq_masked: true, + irq_pending: false, + irq_level: false, + irq_events: Vec::new(), + phy: Phy::new(now), + now, + refused: BTreeMap::new(), + dma: Arc::new(super::dma::NullDma), + tx_cons: 0, + status_tx_cons: 0, + status_rx_prod: 0, + rx_prod_staged: 0, + rx_std_cons: 0, + rx_jumbo_cons: 0, + rx_shortage_drops: 0, + rx_std_shortage_drops: 0, + status_tag: 0, + rx_coal_pending: 0, + tx_coal_pending: 0, + rx_coal_expired: false, + tx_coal_expired: false, + rx_coal_timer: None, + tx_coal_timer: None, + rx_latency: VecDeque::new(), + rx_floor, + wire: Vec::new(), + wake_rx: false, + purge_rx: false, + }; + st.reset(); + st + } + + /// Enter at `now`: run whatever came due since the last entry. + pub fn enter(&mut self, now: Instant) { + if now > self.now { + self.now = now; + } + self.run_timers(); + } + + /// Run whatever came due, earliest first, each at its own deadline: the + /// virtual clock the model is ported from reaches a timer's expiry before + /// it fires, so a timer that arms another counts from there, not from + /// whenever this entry happened. + fn run_timers(&mut self) { + let target = self.now; + while let Some((deadline, timer)) = self.timers().into_iter().filter(|(d, _)| *d <= target).min() { + self.now = deadline; + match timer { + Timer::PhySettle => self.phy.sync(deadline), + Timer::BufferManager => { + self.bufmgr_start = None; + let mode = self.rf(reg::BUFMGR_MODE); + self.set_rf(reg::BUFMGR_MODE, mode | BUFMGR_MODE_ENABLE); + } + Timer::RxCoalescing => self.rx_coal_timer_fired(), + Timer::TxCoalescing => self.tx_coal_timer_fired(), + Timer::RxArrival => self.rx_arrival_due(), + } + } + self.now = target; + } + + fn timers(&self) -> Vec<(Instant, Timer)> { + [ + self.phy.deadline().map(|d| (d, Timer::PhySettle)), + self.bufmgr_start.map(|d| (d, Timer::BufferManager)), + self.rx_coal_timer.map(|d| (d, Timer::RxCoalescing)), + self.tx_coal_timer.map(|d| (d, Timer::TxCoalescing)), + self.rx_latency.front().map(|(d, _)| (*d, Timer::RxArrival)), + ] + .into_iter() + .flatten() + .collect() + } + + /// The earliest moment something here needs attention. + pub fn deadline(&self) -> Option { + self.timers().into_iter().map(|(d, _)| d).min() + } + + pub fn refused(&self) -> Vec<(Refused, u64)> { + self.refused.iter().map(|(what, count)| (*what, *count)).collect() + } + + pub fn eeprom_bytes(&self) -> Option> { + self.eeprom.as_ref().map(|e| e.bytes().to_vec()) + } + + pub fn refuse(&mut self, what: Refused) { + let count = self.refused.entry(what).or_insert(0); + *count += 1; + if *count == 1 { + crate::dlog!(crate::devlog::LogModule::Net, "bcm5701: unsupported or unexpected: {what:?}"); + } + } + + fn drain_phy_refusals(&mut self) { + for what in std::mem::take(&mut self.phy.refused) { + self.refuse(Refused::Phy(what)); + } + } + + // ---- storage ---- + + pub(super) fn rf(&self, off: u32) -> u32 { + self.regfile[((off & (REG_FILE_SIZE - 1)) >> 2) as usize] + } + + pub(super) fn set_rf(&mut self, off: u32, val: u32) { + self.regfile[((off & (REG_FILE_SIZE - 1)) >> 2) as usize] = val; + } + + pub fn sram_read(&self, off: u32) -> u32 { + self.sram[((off & (SRAM_SIZE - 1)) >> 2) as usize] + } + + pub fn sram_write(&mut self, off: u32, val: u32) { + let off = off & (SRAM_SIZE - 1); + let val = if off == SRAM_FW_MBOX { !val } else { val }; + self.sram[(off >> 2) as usize] = val; + } + + // ---- resets ---- + + /// The station address, as the on-chip bootcode owns it: every reset + /// puts it back. + fn load_station_address(&mut self) { + let m = self.mac; + let hi = u32::from(m[0]) << 8 | u32::from(m[1]); + let lo = u32::from_be_bytes([m[2], m[3], m[4], m[5]]); + self.sram_write(SRAM_MAC_HI, MAC_TAG << 16 | hi); + self.sram_write(SRAM_MAC_LO, lo); + self.set_rf(reg::MAC_ADDR_0_HIGH, hi); + self.set_rf(reg::MAC_ADDR_0_LOW, lo); + } + + /// The driver's write of MISC_CFG.RESET_CORE_CLOCKS, by either access + /// path: the register file and SRAM return to their reset values, the + /// handshake mailbox re-arms to the firmware-ready complement and the + /// bootcode's station address comes back. PCI configuration, the chip id + /// in MISC_CTL among it, and the indirect-window bases are untouched. + fn global_reset(&mut self) { + self.bufmgr_start = None; + self.regfile.fill(0); + self.sram.fill(0); + self.reset_engines(); + self.irq_masked = true; + self.irq_pending = false; + self.sram_write(SRAM_FW_MBOX, FW_MAGIC); + self.load_station_address(); + self.update_irq(false); + } + + /// A cold reset: as a global reset, plus the window bases, the PHY, and + /// no firmware-ready complement (only the driver-triggered reset arranges + /// that, because it is the one the driver writes the magic for). + pub fn reset(&mut self) { + self.bufmgr_start = None; + self.regfile.fill(0); + self.sram.fill(0); + self.reg_win_base = 0; + self.mem_win_base = 0; + self.reset_engines(); + self.irq_masked = true; + self.irq_pending = false; + self.phy.reset(self.now); + self.load_station_address(); + self.update_irq(true); + } + + // ---- interrupt ---- + + fn misc_ctl(&self) -> u32 { + self.cfg.read(cfg::MISC_CTL) + } + + /// The interrupt is a level: INTA asserts while a status-block update + /// awaits the driver and neither the mailbox nor MISC_CTL.MASK_PCI_INTR + /// masks it. PCISTATE.INTR_NOT_ACTIVE mirrors the inverse, so the + /// driver's shared-interrupt check reads true exactly when the line is + /// down. `force` delivers the level even if unchanged. + pub fn update_irq(&mut self, force: bool) { + let active = self.irq_pending && !self.irq_masked && self.misc_ctl() & misc_ctl::MASK_PCI_INTR == 0; + self.cfg.update(cfg::PCISTATE, |v| { + if active { + v & !pcistate::INTR_NOT_ACTIVE + } else { + v | pcistate::INTR_NOT_ACTIVE + } + }); + if force || active != self.irq_level { + self.irq_level = active; + self.irq_events.push(active); + } + } + + pub fn irq_level(&self) -> bool { + self.irq_level + } + + /// A write to MISC_CTL acknowledges if CLEAR_INTA was set, then re-derives + /// the line. This is how the driver's "set CLEAR_INTA, clear + /// MASK_PCI_INTR" enables the interrupt. + pub fn misc_ctl_written(&mut self) { + if self.misc_ctl() & misc_ctl::CLEAR_INTA != 0 { + self.irq_pending = false; + self.cfg.update(cfg::MISC_CTL, |v| v & !misc_ctl::CLEAR_INTA); + } + self.update_irq(false); + } + + // ---- EEPROM ---- + + fn eeprom_access(&mut self, command: u32) -> bool { + let device = ((command & eeprom_cmd::DEVICE_ID) >> 26) as u8; + let word = (command & eeprom_cmd::WORD_ADDR) >> 2; + let result = match self.eeprom.as_mut() { + None => { + self.refuse(Refused::EepromFault(eeprom::Fault::NoBacking)); + return false; + } + Some(e) if command & eeprom_cmd::READ != 0 => e.read_word(device, word).map(Some), + Some(e) => { + let value = self.regfile[(reg::EEPROM_DATA >> 2) as usize]; + e.write_word(device, word, value).map(|()| None) + } + }; + match result { + Ok(Some(value)) => { + self.set_rf(reg::EEPROM_DATA, value); + true + } + Ok(None) => true, + Err(fault) => { + self.refuse(Refused::EepromFault(fault)); + false + } + } + } + + fn eeprom_command_write(&mut self, val: u32) { + let old = self.rf(reg::EEPROM_ADDR); + let mut command = val & !(eeprom_cmd::START | eeprom_cmd::COMPLETE | eeprom_cmd::RESET | 3); + let acknowledge = val & eeprom_cmd::COMPLETE != 0; + if val & eeprom_cmd::RESET != 0 { + self.set_rf(reg::EEPROM_ADDR, eeprom_cmd::RESET); + self.set_rf(reg::EEPROM_DATA, 0); + return; + } + if !acknowledge { + command |= old & eeprom_cmd::COMPLETE; + } + self.set_rf(reg::EEPROM_ADDR, command); + // A pending Complete blocks the next transfer until it is + // acknowledged; acknowledging and starting in one write is allowed. + if val & eeprom_cmd::START == 0 || (old & eeprom_cmd::COMPLETE != 0 && !acknowledge) { + return; + } + if self.eeprom_access(command) { + self.set_rf(reg::EEPROM_ADDR, command | eeprom_cmd::COMPLETE); + } + } + + // ---- register file ---- + + /// Register-file writes, reached identically from the BAR and from the + /// indirect register window, so a side effect fires whichever path the + /// driver uses. The 5700/5701 issue the global reset through the window. + pub fn reg_write(&mut self, off: u32, val: u32) { + let off = off & (REG_FILE_SIZE - 1); + let mut val = val; + match off { + reg::MISC_CFG => { + if val & MISCCFG_RESET_CORE_CLOCKS != 0 { + self.global_reset(); + return; + } + } + reg::EEPROM_ADDR => { + self.eeprom_command_write(val); + return; + } + reg::EEPROM_DATA | reg::EEPROM_DELAY => {} + reg::EEPROM_PIN => { + self.refuse(Refused::EepromPins); + self.set_rf(off, val); + return; + } + // The reset bit self-clears; the rest of the write persists. + reg::MAC_MODE => val &= !MODE_RESET, + reg::RX_MODE => { + // A halt takes effect at once. The chip finishes the frame in + // flight first and reads back ENABLE until it has, which a + // model that takes one frame at a time never needs. + val &= !MODE_RESET; + self.set_rf(off, val); + if val & RXMODE_ENABLE != 0 { + self.retry_rx_flush(); + } + return; + } + // Write one to clear: the bits written are the ones acknowledged. + reg::MAC_STATUS => { + let old = self.rf(off); + self.set_rf(off, old & !val); + return; + } + reg::HCC_MODE => { + val &= !MODE_RESET; + self.set_rf(off, val); + self.hcc_mode_written(); + return; + } + reg::HCC_RX_COAL_TICKS | reg::HCC_TX_COAL_TICKS => { + self.set_rf(off, val); + self.coal_ticks_written(off == reg::HCC_RX_COAL_TICKS); + return; + } + reg::MBX_TX_PROD_LO => { + // Store the producer for read-back, then send up to it. + self.set_rf(off, val); + self.tx_kick(val); + return; + } + reg::RX_JUMBO_RCB_MAXLEN_FLAGS => { + // Max_Len is unused for the jumbo ring; only Disable Ring and + // Extended RX Enable do anything, and both are read from the + // stored word. Flag the unmodelled extended format once, here, + // rather than on every frame that finds the ring unavailable. + if val & RCB_FLAG_EXT_RX_ENABLE != 0 { + self.refuse(Refused::JumboExtendedDescriptors); + } + } + reg::MBX_RX_JUMBO_PROD_LO | reg::MBX_RX_STD_PROD_LO | reg::MBX_RX_CONS0_LO => { + self.rx_mailbox_written(off, val); + return; + } + reg::BUFMGR_MODE => { + if val & MODE_RESET != 0 { + self.bufmgr_start = None; + self.set_rf(off, 0); + } + val &= !MODE_RESET; + if val & BUFMGR_MODE_ENABLE == 0 { + self.bufmgr_start = None; + } else if self.rf(off) & BUFMGR_MODE_ENABLE == 0 { + if self.bufmgr_start.is_none() { + self.bufmgr_start = Some(self.now + BUFMGR_START); + } + val &= !BUFMGR_MODE_ENABLE; + } + } + reg::BUFMGR_STATUS => { + self.refuse(Refused::BufmgrStatusWrite); + return; + } + reg::MBX_IRQ0_LO => { + // Every write clears the pending update; a nonzero value + // masks the line and zero re-enables it. + self.irq_pending = false; + self.irq_masked = val != 0; + self.set_rf(off, val); + if val == 0 { + self.retry_rx_flush(); + } + self.update_irq(false); + return; + } + reg::MI_COMM => { + self.mi_comm_write(val); + return; + } + _ => { + self.refuse(Refused::UnmodelledWrite(off)); + self.set_rf(off, val); + return; + } + } + self.set_rf(off, val); + } + + fn mi_comm_write(&mut self, command: u32) { + let register = ((command & mi::REG) >> mi::REG_SHIFT) as u8; + if command & mi::BUSY == 0 { + self.set_rf(reg::MI_COMM, command); + return; + } + let now = self.now; + let opcode = command & (mi::READ | mi::WRITE); + let val = if opcode == mi::READ { + (command & mi::READ_FIELDS) | u32::from(self.phy.read(register, now)) + } else if opcode == mi::WRITE { + self.phy.write(register, (command & mi::DATA) as u16, now); + command & !mi::BUSY + } else { + self.refuse(Refused::MiOpcode(opcode >> 26)); + command & !mi::BUSY + }; + self.set_rf(reg::MI_COMM, val); + self.drain_phy_refusals(); + } + + /// Register-file reads, reached identically from the BAR and the window. + /// Most read back their stored value; MI_STS is the PHY's live link so + /// the driver's auto-poll path sees the carrier. + pub fn reg_read(&mut self, off: u32) -> u32 { + match off & (REG_FILE_SIZE - 1) { + reg::MI_STS => { + self.phy.sync(self.now); + if self.phy.link_up() { + mi::LINK + } else { + 0 + } + } + reg::FW_PROGRESS => FW_PROGRESS_INIT, + _ => self.rf(off), + } + } +} diff --git a/src/dev/bcm5701/tests.rs b/src/dev/bcm5701/tests.rs new file mode 100644 index 000000000..d93d403e6 --- /dev/null +++ b/src/dev/bcm5701/tests.rs @@ -0,0 +1,2410 @@ +//! Register-contract tests for the BCM5701, ported from the QEMU model's +//! qtests. The device is built alone, with a controllable clock and an +//! interrupt probe standing in for the rest of the machine. +//! +//! Ported from tests/qtest/bcm570x-test.c in jamesbraid/qemu-sgi-origami. + +use super::eeprom::{Eeprom, Fault}; +use super::phy::{reg as phy_reg, IDR1_VALUE, IDR2_BCM5701}; +use super::state::{reg, Refused, FW_MAGIC, SRAM_FW_MBOX}; +use super::*; +use crate::irq::LevelProbe; +use crate::pci::PciDmaStatus; +use std::time::Duration; + +const SRAM_WINDOW: u32 = 0x8000; +const MISC_CFG_RESET: u32 = 1; +const MODE_RESET: u32 = 1; +const RXMODE_ENABLE: u32 = 2; +const HCC_ENABLE: u32 = 2; +const BUFMGR_ENABLE: u32 = 2; +const BUFMGR_ATTN: u32 = 4; +const BUFMGR_START: Duration = Duration::from_millis(8); +const PHY_SETTLE: Duration = Duration::from_millis(100); +const MICOMM_BUSY: u32 = 0x2000_0000; +const MICMD_READ: u32 = 0x0800_0000; +const MICMD_WRITE: u32 = 0x0400_0000; +const MICOMM_READ_FIELDS: u32 = 0x0fff_0000; +const PHY_ADDR: u32 = 1; +const MAC: [u8; 6] = [0x52, 0x54, 0x00, 0xab, 0xcd, 0xef]; + +const EEPROM_READ: u32 = 1 << 31; +const EEPROM_COMPLETE: u32 = 1 << 30; +const EEPROM_RESET: u32 = 1 << 29; +const EEPROM_START: u32 = 1 << 25; + +const MEM_SIZE: usize = 0x90_0000; +const TX_RING: u64 = 0x20_0000; +const TX_BUF: u64 = 0x21_0000; +const STATUS: u64 = 0x22_0000; +const STD_RING: u64 = 0x23_0000; +const RET_RING: u64 = 0x24_0000; +const RX_BUF: u64 = 0x25_0000; +const JUMBO_RING: u64 = 0x26_0000; +const JUMBO_BUF: u64 = 0x27_0000; +const STD_STALE_BUF: u64 = 0x28_0000; +/// Past the memory: nothing backs it. +const UNMAPPED: u64 = 0x400_0000; +const TX_RING_LEN: u32 = 4; +const RX_RING_LEN: u32 = 4; +const STD_BUF_SIZE: u32 = 1536; +const JUMBO_BUF_SIZE: u32 = 9216; +const STATUS_BLK_SIZE: usize = 0x50; +const ARRIVAL: Duration = Duration::from_micros(200); +/// Every DMA swap control set: the chip's image goes on the bus as it is. +const MODE_ALL_SWAPS: u32 = 0x36; + +/// Bus-master memory with no bridge in front of it. +struct Mem(std::sync::Mutex>); + +impl PciDma for Mem { + fn dma_read(&self, addr: u64, buf: &mut [u8]) -> PciDmaStatus { + let m = self.0.lock().unwrap(); + let a = addr as usize; + if a.saturating_add(buf.len()) > m.len() { + buf.fill(0); + return PciDmaStatus::MasterAbort; + } + buf.copy_from_slice(&m[a..a + buf.len()]); + PciDmaStatus::Ok + } + fn dma_write(&self, addr: u64, data: &[u8]) -> PciDmaStatus { + let mut m = self.0.lock().unwrap(); + let a = addr as usize; + if a.saturating_add(data.len()) > m.len() { + return PciDmaStatus::MasterAbort; + } + m[a..a + data.len()].copy_from_slice(data); + PciDmaStatus::Ok + } +} + +struct Rig { + dev: Arc, + irq: Arc, + mem: Arc, + /// Frames the guest sent, and a way to hand frames to the guest. + sent: rtrb::Consumer>, + inject: rtrb::Producer>, + ring_slots: usize, +} + +impl Rig { + fn new() -> Self { + Self::with(Bcm5701Config::new(MAC)) + } + + fn with(config: Bcm5701Config) -> Self { + let dev = Bcm5701::new(config).unwrap(); + *dev.frozen.lock() = Some(Instant::now()); + let (irq, line) = LevelProbe::line(); + dev.connect_interrupt(line); + let mem = Arc::new(Mem(std::sync::Mutex::new(vec![0; MEM_SIZE]))); + dev.connect_dma(mem.clone()); + let (link, sent, inject) = crate::net::Link::loopback(); + dev.attach(link); + dev.reset(); + let ring_slots = inject.slots(); + Self { dev, irq, mem, sent, inject, ring_slots } + } + + /// Advance the clock and run whatever came due. + fn step(&self, by: Duration) { + { + let mut t = self.dev.frozen.lock(); + *t = Some(t.unwrap() + by); + } + self.dev.enter(self.dev.now(), |_| ()); + } + + fn mem_write(&self, addr: u64, data: &[u8]) { + let mut m = self.mem.0.lock().unwrap(); + m[addr as usize..addr as usize + data.len()].copy_from_slice(data); + } + fn mem_read(&self, addr: u64, len: usize) -> Vec { + self.mem.0.lock().unwrap()[addr as usize..addr as usize + len].to_vec() + } + fn mem_fill(&self, addr: u64, len: usize, byte: u8) { + self.mem.0.lock().unwrap()[addr as usize..addr as usize + len].fill(byte); + } + fn status_block(&self) -> Vec { + self.mem_read(STATUS, STATUS_BLK_SIZE) + } + fn status_tag(&self) -> u32 { + be32(&self.status_block(), 4) + } + fn status_tx_cons(&self) -> u16 { + get16dma(&self.status_block(), 18) + } + fn status_rx_prod(&self) -> u16 { + get16dma(&self.status_block(), 16) + } + fn line_up(&self) -> bool { + self.cfg_read(0x70 / 4) & 2 == 0 + } + fn nic_no_more_rxbds(&self) -> u32 { + self.readl(SRAM_WINDOW + 0x898) + } + /// The frames the guest has sent so far. + fn sent_frames(&mut self) -> Vec> { + std::iter::from_fn(|| self.sent.pop().ok()).collect() + } + /// Frames handed to the guest that it has not yet taken. + fn waiting(&self) -> usize { + self.ring_slots - self.inject.slots() + } + /// Offer a frame to the receive path, as the backend would. + fn deliver(&mut self, frame: &[u8]) { + self.inject.push(frame.to_vec()).unwrap(); + self.dev.poll(); + } + + fn readl(&self, off: u32) -> u32 { + self.dev.bar_read(0, off, 4) + } + fn readb(&self, off: u32) -> u32 { + self.dev.bar_read(0, off, 1) + } + fn writel(&self, off: u32, val: u32) { + self.dev.bar_write(0, off, val, 4); + } + fn cfg_read(&self, reg: usize) -> u32 { + self.dev.config().read(reg) + } + fn cfg_write(&self, reg: usize, val: u32) { + self.dev.config().write(reg, val); + } + + /// The indirect register window, config 0x78 and 0x80. + fn ind_reg_write(&self, off: u32, val: u32) { + self.cfg_write(0x78 / 4, off); + self.cfg_write(0x80 / 4, val); + } + fn ind_reg_read(&self, off: u32) -> u32 { + self.cfg_write(0x78 / 4, off); + self.cfg_read(0x80 / 4) + } + /// The indirect memory window, config 0x7c and 0x84. + fn ind_mem_write(&self, off: u32, val: u32) { + self.cfg_write(0x7c / 4, off); + self.cfg_write(0x84 / 4, val); + } + fn ind_mem_read(&self, off: u32) -> u32 { + self.cfg_write(0x7c / 4, off); + self.cfg_read(0x84 / 4) + } + + fn mii_read(&self, phy: u32, register: u8) -> u16 { + self.writel(reg::MI_COMM, MICOMM_BUSY | MICMD_READ | phy << 21 | u32::from(register) << 16); + self.readl(reg::MI_COMM) as u16 + } + fn mii_write(&self, phy: u32, register: u8, val: u16) { + self.writel( + reg::MI_COMM, + MICOMM_BUSY | MICMD_WRITE | phy << 21 | u32::from(register) << 16 | u32::from(val), + ); + } + fn sram_word(&self, off: u32) -> u32 { + self.dev.st.lock().sram_read(off) + } + fn split_addr(&self, hi_reg: u32, lo_reg: u32, addr: u64) { + self.writel(hi_reg, (addr >> 32) as u32); + self.writel(lo_reg, addr as u32); + } + fn refusals(&self, what: Refused) -> u64 { + self.dev.refused().iter().find(|(r, _)| *r == what).map_or(0, |(_, n)| *n) + } +} + +fn temp_file(name: &str, contents: &[u8]) -> std::path::PathBuf { + let path = std::env::temp_dir().join(format!("iris-bcm5701-{}-{name}", std::process::id())); + std::fs::write(&path, contents).unwrap(); + path +} + +/// Dword and sub-dword reads of the identity, so a hidden swap in one half +/// cannot hide behind the other. Ported from test_pci_four_tuple. +#[test] +fn the_function_presents_the_sgi_board_identity() { + let rig = Rig::new(); + assert_eq!(rig.cfg_read(0), 0x1645_14e4); + assert_eq!(rig.cfg_read(0x2c / 4), 0x8010_10a9); + assert_eq!(rig.cfg_read(0x08 / 4) >> 8, 0x02_0000); + assert_eq!(rig.cfg_read(0x08 / 4) & 0xff, 0); + assert_eq!(rig.dev.config().interrupt_pin(), PCI_INTA); +} + +/// The chip id in MISC_CTL decodes to the BCM5701, ASIC revision 0, and is +/// read-only while the low half sticks. Ported from test_misc_ctl_asic_revision. +#[test] +fn misc_ctl_presents_the_chip_id() { + let rig = Rig::new(); + let misc = rig.cfg_read(0x68 / 4); + assert_eq!(misc >> 16, 0x0105); + assert_eq!((misc >> 16) >> 12, 0); + rig.cfg_write(0x68 / 4, 0xffff_0004); + assert_eq!(rig.cfg_read(0x68 / 4), 0x0105_0004); +} + +/// Ported from test_indirect_register_window: the window aliases the file. +#[test] +fn the_indirect_register_window_aliases_the_register_file() { + let rig = Rig::new(); + rig.ind_reg_write(0x0410, 0x1234_5678); + assert_eq!(rig.readl(0x0410), 0x1234_5678); + assert_eq!(rig.ind_reg_read(0x0410), 0x1234_5678); + assert_eq!(rig.cfg_read(0x78 / 4), 0x0410); +} + +/// Only whole dwords reach the windows. A narrower configuration read sees +/// the header's own bytes, which stay zero because nothing writes them, as +/// in the model this is ported from, where the window state sits outside +/// the header and the header bytes are read-only. +#[test] +fn narrow_configuration_accesses_bypass_the_indirect_windows() { + let rig = Rig::new(); + rig.ind_reg_write(0x0410, 0x1234_5678); + rig.ind_mem_write(0x0c14, 0xdead_beef); + let config = rig.dev.config(); + for offset in 0x78..0x88 { + assert_eq!(config.read_sized(offset, 1), 0, "byte at {offset:#x}"); + } + for offset in (0x78..0x88).step_by(2) { + assert_eq!(config.read_sized(offset, 2), 0, "halfword at {offset:#x}"); + } + config.write_masked(0x80 / 4, 0xff, 0xff); + config.write_masked(0x7c / 4, 0xffff_0000, 0xffff_0000); + assert_eq!(config.read_sized(0x80, 1), 0); + assert_eq!(config.read_sized(0x7e, 2), 0); + assert_eq!(config.read_sized(0x78, 4), 0x0410); + assert_eq!(config.read_sized(0x80, 4), 0x1234_5678); + assert_eq!(config.read_sized(0x7c, 4), 0x0c14); + assert_eq!(config.read_sized(0x84, 4), 0xdead_beef); +} + +/// Ported from test_indirect_memory_window: SRAM through config and through +/// the BAR window at 0x8000. +#[test] +fn the_indirect_memory_window_aliases_sram() { + let rig = Rig::new(); + rig.ind_mem_write(0x0c14, 0xdead_beef); + assert_eq!(rig.ind_mem_read(0x0c14), 0xdead_beef); + assert_eq!(rig.readl(SRAM_WINDOW + (0x0c14 & 0x7fff)), 0xdead_beef); + // The base register is masked to a 32 KiB boundary for the BAR window, + // and the 32 KiB SRAM has one position. + rig.cfg_write(0x7c / 4, 0x0c14); + assert_eq!(rig.readl(SRAM_WINDOW), rig.sram_word(0)); +} + +/// The magic written before the reset (the bge order) reads back as the +/// complement, and the reset self-clears its trigger and re-arms the mailbox. +/// Ported from test_reset_handshake_via_bar. +#[test] +fn the_reset_handshake_completes_when_the_magic_precedes_the_reset() { + let rig = Rig::new(); + rig.ind_mem_write(SRAM_FW_MBOX, FW_MAGIC); + assert_eq!(rig.ind_mem_read(SRAM_FW_MBOX), !FW_MAGIC); + rig.writel(reg::MISC_CFG, MISC_CFG_RESET); + assert_eq!(rig.readl(reg::MISC_CFG) & MISC_CFG_RESET, 0); + assert_eq!(rig.ind_mem_read(SRAM_FW_MBOX), !FW_MAGIC); +} + +/// The IRIX order: reset first, then the magic. Ported from +/// test_reset_handshake_magic_after_reset. +#[test] +fn the_reset_handshake_completes_when_the_magic_follows_the_reset() { + let rig = Rig::new(); + rig.writel(reg::MISC_CFG, MISC_CFG_RESET); + rig.ind_mem_write(SRAM_FW_MBOX, FW_MAGIC); + assert_eq!(rig.ind_mem_read(SRAM_FW_MBOX), 0xb49a_89ab); +} + +/// Ported from test_reset_handshake_via_indirect: the 5700/5701 reset through +/// the configuration window alone. +#[test] +fn the_reset_handshake_completes_through_the_indirect_windows() { + let rig = Rig::new(); + rig.ind_mem_write(SRAM_FW_MBOX, FW_MAGIC); + rig.ind_reg_write(reg::MISC_CFG, MISC_CFG_RESET); + assert_eq!(rig.ind_mem_read(SRAM_FW_MBOX), 0xb49a_89ab); +} + +/// Ported from test_firmware_progress_register: the all-ones "no progress +/// yet" sentinel survives a reset. +#[test] +fn the_firmware_progress_register_holds_its_sentinel() { + let rig = Rig::new(); + assert_eq!(rig.readl(reg::FW_PROGRESS), 0xffff_ffff); + assert_eq!(rig.ind_reg_read(reg::FW_PROGRESS), 0xffff_ffff); + rig.writel(reg::MISC_CFG, MISC_CFG_RESET); + assert_eq!(rig.readl(reg::FW_PROGRESS), 0xffff_ffff); +} + +/// A global reset keeps the PCI header, the chip id among it, and the +/// window bases, and restores the bootcode's station address. Ported from +/// test_mac_address_registers. +#[test] +fn a_global_reset_restores_the_bootcode_state_only() { + let rig = Rig::new(); + rig.cfg_write(0x68 / 4, 0x0000_0004); + rig.cfg_write(0x7c / 4, 0x1234); + rig.writel(reg::MAC_ADDR_0_HIGH, 0xa1a2); + rig.writel(reg::MISC_CFG, MISC_CFG_RESET); + assert_eq!(rig.cfg_read(0x68 / 4), 0x0105_0004); + assert_eq!(rig.cfg_read(0x7c / 4), 0x1234); + assert_eq!(rig.readl(reg::MAC_ADDR_0_HIGH), 0x5254); +} + +/// Ported from test_self_clearing_reset_bits. +#[test] +fn mode_register_reset_bits_self_clear_and_the_rest_persists() { + let rig = Rig::new(); + rig.writel(reg::MAC_MODE, MODE_RESET | 0x4); + assert_eq!(rig.readl(reg::MAC_MODE) & MODE_RESET, 0); + assert_eq!(rig.readl(reg::MAC_MODE) & 4, 4); + + rig.writel(reg::HCC_MODE, MODE_RESET); + assert_eq!(rig.readl(reg::HCC_MODE) & MODE_RESET, 0); + + rig.writel(reg::RX_MODE, MODE_RESET | RXMODE_ENABLE); + assert_eq!(rig.readl(reg::RX_MODE) & MODE_RESET, 0); + assert_eq!(rig.readl(reg::RX_MODE) & RXMODE_ENABLE, RXMODE_ENABLE); + // The IRIX sequence: reset on its own leaves the receiver halted. + rig.writel(reg::RX_MODE, MODE_RESET); + assert_eq!(rig.readl(reg::RX_MODE), 0); +} + +/// The driver's idle sequence writes ENABLE, then 0, and polls for the whole +/// register to read zero. Ported from test_hcc_mode_idle_poll and +/// test_hcc_mode_clears_to_zero. +#[test] +fn host_coalescing_mode_stores_and_clears() { + let rig = Rig::new(); + rig.writel(reg::HCC_MODE, HCC_ENABLE); + rig.writel(reg::RX_MODE, RXMODE_ENABLE); + assert_eq!(rig.readl(reg::HCC_MODE) & HCC_ENABLE, HCC_ENABLE); + rig.writel(reg::HCC_MODE, 0); + assert_eq!(rig.readl(reg::HCC_MODE) & HCC_ENABLE, 0); + + rig.writel(reg::HCC_MODE, 0x13e); + rig.writel(reg::HCC_MODE, 0); + assert_eq!(rig.readl(reg::HCC_MODE), 0); +} + +/// ENABLE reads back only after the buffer manager's startup, by either +/// access path. Ported from test_bufmgr_registers. +#[test] +fn the_buffer_manager_enable_reads_back_after_its_startup() { + for indirect in [false, true] { + let rig = Rig::new(); + let read = |off| if indirect { rig.ind_reg_read(off) } else { rig.readl(off) }; + let write = |off, val| if indirect { rig.ind_reg_write(off, val) } else { rig.writel(off, val) }; + let on = BUFMGR_ENABLE | BUFMGR_ATTN; + + write(reg::BUFMGR_MODE, BUFMGR_ATTN); + write(reg::BUFMGR_MODE, on); + assert_eq!(read(reg::BUFMGR_MODE), BUFMGR_ATTN); + rig.step(Duration::from_millis(7)); + assert_eq!(read(reg::BUFMGR_MODE), BUFMGR_ATTN); + // A repeated request does not restart the interval. + write(reg::BUFMGR_MODE, on); + rig.step(BUFMGR_START - Duration::from_millis(7) - Duration::from_nanos(1)); + assert_eq!(read(reg::BUFMGR_MODE), BUFMGR_ATTN); + rig.step(Duration::from_nanos(1)); + assert_eq!(read(reg::BUFMGR_MODE), on); + + // Reset cancels startup; the new request starts a fresh interval. + write(reg::BUFMGR_MODE, MODE_RESET | on); + assert_eq!(read(reg::BUFMGR_MODE), BUFMGR_ATTN); + rig.step(BUFMGR_START); + assert_eq!(read(reg::BUFMGR_MODE), on); + + // Halting is immediate and cancels a pending start. + write(reg::BUFMGR_MODE, BUFMGR_ATTN); + assert_eq!(read(reg::BUFMGR_MODE), BUFMGR_ATTN); + write(reg::BUFMGR_MODE, on); + write(reg::BUFMGR_MODE, BUFMGR_ATTN); + rig.step(BUFMGR_START); + assert_eq!(read(reg::BUFMGR_MODE), BUFMGR_ATTN); + + // The status register is read-only, and the refusal is counted. + write(reg::BUFMGR_STATUS, on); + assert_eq!(read(reg::BUFMGR_STATUS), 0); + assert!(rig.refusals(Refused::BufmgrStatusWrite) >= 1); + + // A global reset cancels a pending startup. + rig.writel(reg::BUFMGR_MODE, on); + rig.step(BUFMGR_START / 2); + rig.writel(reg::MISC_CFG, MISC_CFG_RESET); + rig.step(BUFMGR_START); + assert_eq!(rig.readl(reg::BUFMGR_MODE), 0); + } +} + +/// MAC_STATUS bits clear on a write of one, and the write is never readable +/// back as state. Ported from test_mac_status_write_one_clears. +#[test] +fn mac_status_clears_on_write_one() { + let rig = Rig::new(); + assert_eq!(rig.readl(reg::MAC_STATUS), 0); + rig.writel(reg::MAC_STATUS, 0x50); + assert_eq!(rig.readl(reg::MAC_STATUS), 0); + rig.writel(reg::MAC_STATUS, 0xffff_ffff); + assert_eq!(rig.readl(reg::MAC_STATUS), 0); +} + +/// MI_COMM completes at once; a successful read keeps the command and address +/// fields, replaces the data and clears Start and ReadFailed, whatever +/// reserved bits the request carried; a write without Start only stores; an +/// undefined opcode touches no PHY register and is counted. Ported from +/// test_mi_busy_clears. +#[test] +fn mi_comm_completes_reads_and_writes_by_either_path() { + for indirect in [false, true] { + let rig = Rig::new(); + let read = |off| if indirect { rig.ind_reg_read(off) } else { rig.readl(off) }; + let write = |off, val| if indirect { rig.ind_reg_write(off, val) } else { rig.writel(off, val) }; + let anar = |rig: &Rig| { + let command = MICOMM_BUSY | MICMD_READ | PHY_ADDR << 21 | 4 << 16; + if indirect { + rig.ind_reg_write(reg::MI_COMM, command); + rig.ind_reg_read(reg::MI_COMM) + } else { + rig.writel(reg::MI_COMM, command); + rig.readl(reg::MI_COMM) + } + }; + + // The discriminating full-word case: a PHY ID read keeps its fields. + for (command, result) in [(0x2822_ffff, 0x0822_0020), (0xf822_ffff, 0x0822_0020)] { + write(reg::MI_COMM, command); + assert_eq!(read(reg::MI_COMM), result); + } + + let comm = anar(&rig); + assert_eq!(comm & MICOMM_BUSY, 0); + let baseline = comm & 0xffff; + assert_eq!(comm, (MICOMM_BUSY | MICMD_READ | PHY_ADDR << 21 | 4 << 16) & MICOMM_READ_FIELDS | baseline); + + // The 10baseT half-duplex advertisement toggles. + let value = baseline ^ 0x20; + let write_command = MICMD_WRITE | PHY_ADDR << 21 | 4 << 16 | value; + write(reg::MI_COMM, write_command); + assert_eq!(read(reg::MI_COMM), write_command); + write(reg::MI_COMM, write_command); + assert_eq!(read(reg::MI_COMM), write_command); + + // Without Start nothing reaches the PHY. + assert_eq!(anar(&rig) & 0xffff, baseline); + + let id_command = MICMD_READ | PHY_ADDR << 21 | 2 << 16; + write(reg::MI_COMM, id_command); + assert_eq!(read(reg::MI_COMM), id_command); + write(reg::MI_COMM, id_command | MICOMM_BUSY); + assert_eq!(read(reg::MI_COMM), (id_command & MICOMM_READ_FIELDS) | u32::from(IDR1_VALUE)); + + write(reg::MI_COMM, write_command | MICOMM_BUSY); + assert_eq!(read(reg::MI_COMM) & MICOMM_BUSY, 0); + assert_eq!(anar(&rig) & 0xffff, value); + + // Undefined opcodes complete without a PHY access and are counted. + for opcode in [0, MICMD_READ | MICMD_WRITE] { + let command = MICOMM_BUSY | opcode | PHY_ADDR << 21 | 4 << 16 | 0x5a5a; + write(reg::MI_COMM, command); + assert_eq!(read(reg::MI_COMM), command & !MICOMM_BUSY); + assert_eq!(anar(&rig) & 0xffff, value); + } + assert_eq!(rig.refusals(Refused::MiOpcode(0)), 1); + assert_eq!(rig.refusals(Refused::MiOpcode(3)), 1); + } +} + +/// Ported from test_phy_readback: identity, a link that is down while +/// negotiating, a settle that must elapse, and MI_STS following the carrier. +#[test] +fn the_phy_negotiates_and_the_link_follows_the_settle() { + let rig = Rig::new(); + assert_eq!(rig.mii_read(PHY_ADDR, phy_reg::IDR1), IDR1_VALUE); + assert_eq!(rig.mii_read(PHY_ADDR, phy_reg::IDR2), IDR2_BCM5701); + assert_eq!(rig.mii_read(PHY_ADDR, phy_reg::BMSR) & 4, 0); + assert_eq!(rig.readl(reg::MI_STS) & 1, 0); + + // Restarting autonegotiation negotiates again. + rig.mii_write(PHY_ADDR, phy_reg::BMCR, 0x1000 | 0x0200); + assert_eq!(rig.mii_read(PHY_ADDR, phy_reg::BMSR) & 4, 0); + + rig.step(PHY_SETTLE + Duration::from_micros(1)); + assert_eq!(rig.mii_read(PHY_ADDR, phy_reg::BMSR) & 4, 4); + assert_eq!(rig.mii_read(PHY_ADDR, phy_reg::BMSR) & 0x20, 0x20); + assert_eq!(rig.readl(reg::MI_STS) & 1, 1); + + // A cable pulled drops the carrier without touching negotiation. + rig.dev.set_backend_link(false); + assert_eq!(rig.readl(reg::MI_STS) & 1, 0); + rig.dev.set_backend_link(true); + assert_eq!(rig.readl(reg::MI_STS) & 1, 1); +} + +/// Ported from test_phy_aux_status_link_up: never all-ones, ACOMP and the +/// 1000baseT full-duplex resolution once settled. +#[test] +fn the_phy_auxiliary_status_gates_the_link_up_path() { + let rig = Rig::new(); + let aux = rig.mii_read(PHY_ADDR, phy_reg::AUXSTS); + assert_eq!(aux & 0x8000, 0); + assert_ne!(aux, 0xffff); + rig.mii_write(PHY_ADDR, phy_reg::BMCR, 0x1000 | 0x0200); + rig.step(PHY_SETTLE + Duration::from_micros(1)); + let aux = rig.mii_read(PHY_ADDR, phy_reg::AUXSTS); + assert_eq!(aux & 0x8000, 0x8000); + assert_eq!(aux & 0x0700, 0x0700); + assert_ne!(aux, 0xffff); + // The partner advertises what this end does, and gigabit full duplex. + assert_eq!(rig.mii_read(PHY_ADDR, phy_reg::GTSR) & 0x0800, 0x0800); + assert_eq!(rig.mii_read(PHY_ADDR, phy_reg::ANLPAR), rig.mii_read(PHY_ADDR, phy_reg::ANAR) | 0x4000); +} + +/// A PHY register the model lacks reads zero, never the all-ones an absent +/// PHY returns, and the access is counted. +#[test] +fn unimplemented_phy_registers_read_zero_and_are_counted() { + let rig = Rig::new(); + assert_eq!(rig.mii_read(PHY_ADDR, 0x1c), 0); + rig.mii_write(PHY_ADDR, 0x1c, 0x1234); + rig.mii_write(PHY_ADDR, phy_reg::IDR1, 0); + let refused = rig.dev.refused(); + assert!(refused.contains(&(Refused::Phy(phy::Refusal::ReadUnimplemented(0x1c)), 1))); + assert!(refused.contains(&(Refused::Phy(phy::Refusal::WriteUnimplemented(0x1c)), 1))); + assert!(refused.contains(&(Refused::Phy(phy::Refusal::WriteReadOnly(phy_reg::IDR1)), 1))); +} + +/// The station address lands in the SRAM mailbox the driver reads first. +/// Ported from test_mac_address_mailbox. +#[test] +fn the_station_address_is_in_the_sram_mailbox() { + let rig = Rig::new(); + assert_eq!(rig.ind_mem_read(0x0c14), 0x484b_5254); + assert_eq!(rig.ind_mem_read(0x0c18), 0x00ab_cdef); +} + +/// The station address is also in filter slot 0, readable a byte at a time +/// from that pair alone. Ported from test_mac_address_registers. +#[test] +fn the_station_address_is_in_filter_slot_zero_with_narrow_reads_there_only() { + let rig = Rig::new(); + assert_eq!(rig.readl(reg::MAC_ADDR_0_HIGH), 0x0000_5254); + assert_eq!(rig.readl(reg::MAC_ADDR_0_LOW), 0x00ab_cdef); + let bytes: Vec = (0..8).map(|i| rig.readb(reg::MAC_ADDR_0_HIGH + i)).collect(); + assert_eq!(bytes, [0x54, 0x52, 0, 0, 0xef, 0xcd, 0xab, 0]); + + rig.writel(reg::MAC_ADDR_0_HIGH - 4, 0x5566_7788); + rig.writel(reg::MAC_ADDR_0_LOW + 4, 0x1122_3344); + // Adjacent byte reads and halfword accesses are rejected and counted. + assert_eq!(rig.readb(reg::MAC_ADDR_0_HIGH - 1), 0); + assert_eq!(rig.readb(reg::MAC_ADDR_0_LOW + 4), 0); + assert_eq!(rig.dev.bar_read(0, reg::MAC_ADDR_0_HIGH, 2), 0); + // Narrow writes are not part of the byte-read contract. + rig.dev.bar_write(0, reg::MAC_ADDR_0_HIGH, 0, 1); + rig.dev.bar_write(0, reg::MAC_ADDR_0_LOW, 0, 2); + assert_eq!(rig.readl(reg::MAC_ADDR_0_HIGH), 0x5254); + assert_eq!(rig.readl(reg::MAC_ADDR_0_LOW), 0x00ab_cdef); + assert_eq!(rig.readl(reg::MAC_ADDR_0_HIGH - 4), 0x5566_7788); + assert_eq!(rig.readl(reg::MAC_ADDR_0_LOW + 4), 0x1122_3344); + assert!(rig.refusals(Refused::BarAccess { offset: reg::MAC_ADDR_0_HIGH, width: 1, write: true }) >= 1); + assert!(rig.refusals(Refused::BarAccess { offset: reg::MAC_ADDR_0_HIGH, width: 2, write: false }) >= 1); +} + +/// A write to a register with no modelled behaviour reads back and is counted. +#[test] +fn an_unmodelled_register_stores_and_is_counted() { + let rig = Rig::new(); + rig.writel(0x1000, 0xcafe_f00d); + assert_eq!(rig.readl(0x1000), 0xcafe_f00d); + assert_eq!(rig.refusals(Refused::UnmodelledWrite(0x1000)), 1); +} + +/// The first 256 bytes of BAR0 are the configuration registers again; a +/// MISC_CTL acknowledge there lands on the same state. +#[test] +fn bar0_aliases_the_configuration_registers() { + let rig = Rig::new(); + assert_eq!(rig.readl(0x68), 0x0105_0000); + rig.writel(0x68, 0x0000_0008); + assert_eq!(rig.cfg_read(0x68 / 4), 0x0105_0008); + rig.writel(0x7c, 0x0c14); + assert_eq!(rig.readl(0x84), rig.ind_mem_read(0x0c14)); +} + +/// PCISTATE reports conventional PCI and a quiet interrupt line, is chip +/// driven, and CLEAR_INTA self-clears. Ported from the interrupt cases of +/// test_interrupt_mailbox that need no status block. +#[test] +fn pcistate_and_the_interrupt_mailbox_start_quiet() { + let rig = Rig::new(); + assert_eq!(rig.cfg_read(0x70 / 4), 0x16); + rig.cfg_write(0x70 / 4, 0); + assert_eq!(rig.cfg_read(0x70 / 4), 0x16); + rig.writel(reg::MBX_IRQ0_LO, 0); + rig.cfg_write(0x68 / 4, 1); + assert_eq!(rig.cfg_read(0x68 / 4) & 1, 0); + assert!(!rig.irq.level()); + rig.dev.req64_sampled(true); + assert_eq!(rig.cfg_read(0x70 / 4), 0x06); +} + +fn eeprom_rig(image: &[u8], device_id: u8) -> Rig { + Rig::with(Bcm5701Config { eeprom: Some(Eeprom::from_image(device_id, image.to_vec()).unwrap()), ..Bcm5701Config::new(MAC) }) +} + +/// Ported from test_eeprom_bar_read_write: Start cannot replace an +/// unacknowledged completion, acknowledge-and-start works in one write, +/// transfers stay word aligned, writes reach only their word, and Reset wins +/// over Start. +#[test] +fn the_eeprom_controller_reads_and_writes_aligned_words() { + let path = temp_file("bar-rw", &[0x66, 0x99, 0x55, 0xaa, 0x12, 0x34, 0x56, 0x78]); + let rig = Rig::with(Bcm5701Config { eeprom: Some(Eeprom::from_file(3, path.clone()).unwrap()), ..Bcm5701Config::new(MAC) }); + let done = |rig: &Rig| rig.readl(reg::EEPROM_ADDR) & (EEPROM_COMPLETE | EEPROM_START); + + rig.writel(reg::EEPROM_ADDR, EEPROM_READ | EEPROM_START | 3 << 26); + assert_eq!(rig.readl(reg::EEPROM_DATA), 0xaa55_9966); + assert_eq!(done(&rig), EEPROM_COMPLETE); + + // Start cannot replace an unacknowledged completion. + rig.ind_reg_write(reg::EEPROM_ADDR, EEPROM_READ | EEPROM_START | 3 << 26 | 4); + assert_eq!(rig.readl(reg::EEPROM_DATA), 0xaa55_9966); + assert_eq!(rig.readl(reg::EEPROM_ADDR) & EEPROM_COMPLETE, EEPROM_COMPLETE); + + // OpenBSD acknowledges the old completion with the next Start. + rig.ind_reg_write(reg::EEPROM_ADDR, EEPROM_READ | EEPROM_COMPLETE | EEPROM_START | 3 << 26 | 4); + assert_eq!(rig.ind_reg_read(reg::EEPROM_DATA), 0x7856_3412); + assert_eq!(done(&rig), EEPROM_COMPLETE); + + // Address bits 1:0 are ignored: this acknowledges and starts nothing. + rig.writel(reg::EEPROM_DATA, 0x1357_9bdf); + rig.writel(reg::EEPROM_ADDR, EEPROM_COMPLETE); + assert_eq!(rig.readl(reg::EEPROM_DATA), 0x1357_9bdf); + assert_eq!(done(&rig), 0); + assert_eq!(std::fs::read(&path).unwrap()[4..], [0x12, 0x34, 0x56, 0x78]); + + rig.writel(reg::EEPROM_DATA, 0x89ab_cdef); + rig.ind_reg_write(reg::EEPROM_ADDR, EEPROM_START | 3 << 26 | 7); + assert_eq!(rig.readl(reg::EEPROM_ADDR) & EEPROM_COMPLETE, EEPROM_COMPLETE); + assert_eq!(rig.readl(reg::EEPROM_ADDR) & 3, 0); + assert_eq!(rig.readl(reg::EEPROM_DELAY), 0); + assert_eq!(std::fs::read(&path).unwrap(), [0x66, 0x99, 0x55, 0xaa, 0xef, 0xcd, 0xab, 0x89]); + + // Block reset clears the volatile state; the delay register only reads back. + rig.writel(reg::EEPROM_DELAY, 0x000a_2c2a); + rig.writel(reg::EEPROM_ADDR, EEPROM_RESET); + assert_eq!(rig.readl(reg::EEPROM_ADDR), EEPROM_RESET); + assert_eq!(rig.readl(reg::EEPROM_DATA), 0); + assert_eq!(rig.readl(reg::EEPROM_DELAY), 0x000a_2c2a); + + // Reset wins over Start, including while the level remains set. + for data in [0x1122_3344, 0x5566_7788] { + rig.writel(reg::EEPROM_DATA, data); + rig.writel(reg::EEPROM_ADDR, EEPROM_RESET | EEPROM_START | 3 << 26 | 4); + assert_eq!(rig.readl(reg::EEPROM_ADDR), EEPROM_RESET); + assert_eq!(rig.readl(reg::EEPROM_DATA), 0); + } + + // Chip and machine resets do not replace the external bytes. + rig.writel(reg::EEPROM_ADDR, 0); + rig.writel(reg::MISC_CFG, MISC_CFG_RESET); + assert_eq!(rig.readl(reg::EEPROM_DELAY), 0); + rig.writel(reg::EEPROM_ADDR, EEPROM_READ | EEPROM_START | 3 << 26 | 4); + assert_eq!(rig.readl(reg::EEPROM_DATA), 0x89ab_cdef); + rig.dev.reset(); + rig.writel(reg::EEPROM_ADDR, EEPROM_READ | EEPROM_START | 3 << 26 | 4); + assert_eq!(rig.readl(reg::EEPROM_DATA), 0x89ab_cdef); + + // A fresh process sees the guest write in the same file. + let again = Eeprom::from_file(3, path.clone()).unwrap(); + assert_eq!(again.read_word(3, 1), Ok(0x89ab_cdef)); + std::fs::remove_file(path).unwrap(); +} + +/// Ported from test_eeprom_missing_backing: no backing means no transfer and +/// the data register is untouched. +#[test] +fn an_eeprom_command_with_no_backing_does_not_complete() { + let rig = Rig::new(); + rig.writel(reg::EEPROM_DATA, 0x7654_3210); + rig.writel(reg::EEPROM_ADDR, EEPROM_READ | EEPROM_START); + assert_eq!(rig.readl(reg::EEPROM_DATA), 0x7654_3210); + assert_eq!(rig.readl(reg::EEPROM_ADDR) & (EEPROM_COMPLETE | EEPROM_START), 0); + assert_eq!(rig.refusals(Refused::EepromFault(Fault::NoBacking)), 1); +} + +/// Ported from test_eeprom_read_only_backing. +#[test] +fn a_read_only_eeprom_refuses_writes() { + let mut eeprom = Eeprom::from_image(3, vec![0x66, 0x99, 0x55, 0xaa]).unwrap(); + eeprom.set_read_only(true); + assert!(eeprom.is_read_only()); + let rig = Rig::with(Bcm5701Config { eeprom: Some(eeprom), ..Bcm5701Config::new(MAC) }); + rig.writel(reg::EEPROM_DATA, 0x1234_5678); + rig.writel(reg::EEPROM_ADDR, EEPROM_START | 3 << 26); + assert_eq!(rig.readl(reg::EEPROM_DATA), 0x1234_5678); + assert_eq!(rig.readl(reg::EEPROM_ADDR) & (EEPROM_COMPLETE | EEPROM_START), 0); + assert_eq!(rig.dev.eeprom_bytes().unwrap(), [0x66, 0x99, 0x55, 0xaa]); + assert_eq!(rig.refusals(Refused::EepromFault(Fault::ReadOnly)), 1); +} + +/// Ported from test_eeprom_bounds_and_device_id. +#[test] +fn eeprom_words_past_the_backing_and_other_device_ids_fail() { + let rig = eeprom_rig(&[0x66, 0x99, 0x55, 0xaa], 3); + rig.writel(reg::EEPROM_DATA, 0x1234_5678); + for command in [EEPROM_READ | EEPROM_START | 3 << 26 | 4, EEPROM_READ | EEPROM_START | 4 << 26] { + rig.writel(reg::EEPROM_ADDR, command); + assert_eq!(rig.readl(reg::EEPROM_DATA), 0x1234_5678); + assert_eq!(rig.readl(reg::EEPROM_ADDR) & (EEPROM_COMPLETE | EEPROM_START), 0); + } + assert_eq!(rig.refusals(Refused::EepromFault(Fault::OutOfRange(1))), 1); + assert_eq!(rig.refusals(Refused::EepromFault(Fault::WrongDevice(4))), 1); +} + +/// Ported from test_eeprom_last_interface_word: the 64 KiB reach of the +/// address field. +#[test] +fn the_last_eeprom_interface_word_is_reachable() { + let mut image = vec![0u8; 65536]; + image[65532..].copy_from_slice(&[0x66, 0x99, 0x55, 0xaa]); + let rig = eeprom_rig(&image, 3); + rig.writel(reg::EEPROM_ADDR, EEPROM_READ | EEPROM_START | 3 << 26 | 0x3fff << 2); + assert_eq!(rig.readl(reg::EEPROM_DATA), 0xaa55_9966); + rig.writel(reg::EEPROM_DATA, 0x89ab_cdef); + rig.writel(reg::EEPROM_ADDR, EEPROM_COMPLETE | EEPROM_START | 3 << 26 | 0x3fff << 2); + assert_eq!(rig.dev.eeprom_bytes().unwrap()[65532..], [0xef, 0xcd, 0xab, 0x89]); +} + +/// Ported from test_eeprom_rejects_non_word_backing and +/// test_eeprom_id_requires_backing: the configuration is validated up front. +#[test] +fn eeprom_backings_are_validated() { + assert!(Eeprom::from_image(3, vec![1, 2, 3, 4, 5]).unwrap_err().contains("multiple of four")); + assert!(Eeprom::from_image(3, vec![]).is_err()); + assert!(Eeprom::from_image(3, vec![0; 65540]).is_err()); + assert!(Eeprom::from_image(8, vec![0; 4]).unwrap_err().contains("0..=7")); + assert!(Bcm5701::new(Bcm5701Config::new([1, 0, 0, 0, 0, 0])).is_err()); +} + +/// Direct serial-pin control is unsupported and observable, not faked. +#[test] +fn direct_eeprom_pin_control_is_refused() { + let rig = Rig::new(); + rig.writel(reg::EEPROM_PIN, 0x15); + assert_eq!(rig.readl(reg::EEPROM_PIN), 0x15); + assert_eq!(rig.refusals(Refused::EepromPins), 1); +} + +// ---- the descriptor engines ---- + +fn be32(b: &[u8], off: usize) -> u32 { + u32::from_be_bytes(b[off..off + 4].try_into().unwrap()) +} +fn put32(b: &mut [u8], off: usize, v: u32) { + b[off..off + 4].copy_from_slice(&v.to_be_bytes()); +} +/// A 16-bit descriptor field sits in the other half of its word, big-endian. +fn put16dma(b: &mut [u8], off: usize, v: u16) { + b[off ^ 2..(off ^ 2) + 2].copy_from_slice(&v.to_be_bytes()); +} +fn get16dma(b: &[u8], off: usize) -> u16 { + u16::from_be_bytes([b[off ^ 2], b[(off ^ 2) + 1]]) +} + +fn tx_bd(buf: u64, len: u16, flags: u16, mss: u16) -> [u8; 16] { + let mut bd = [0; 16]; + put32(&mut bd, 0, (buf >> 32) as u32); + put32(&mut bd, 4, buf as u32); + put16dma(&mut bd, 8, flags); + put16dma(&mut bd, 10, len); + put16dma(&mut bd, 14, mss); + bd +} + +const TXF_TCP_UDP_CKSUM: u16 = 1; +const TXF_IP_CKSUM: u16 = 2; +const TXF_END: u16 = 4; +const TXF_INSERT_SRC: u16 = 0x1000; +const TXF_CHOOSE_SRC: fn(u16) -> u16 = |n| n << 13; + +fn rx_bd(buf: u64, buf_len: u16) -> [u8; 32] { + let mut bd = [0; 32]; + put32(&mut bd, 0, (buf >> 32) as u32); + put32(&mut bd, 4, buf as u32); + put16dma(&mut bd, 8, buf_len); + bd +} + +impl Rig { + /// Send ring in memory, status block address, coalescing and the MAC on, + /// the way a driver brings the transmit side up. + fn tx_open(&self, ring_len: u32, coal_bds: u32, coal_ticks: u32) { + self.ind_reg_write(reg::MODE_CTL, MODE_ALL_SWAPS); + self.ind_mem_write(0x100, 0); + self.ind_mem_write(0x104, TX_RING as u32); + self.ind_mem_write(0x108, ring_len << 16); + self.split_addr(reg::HCC_STATUSBLK_ADDR_HI, reg::HCC_STATUSBLK_ADDR_LO, STATUS); + self.writel(reg::HCC_TX_MAX_COAL_BDS, coal_bds); + self.writel(reg::HCC_TX_COAL_TICKS, coal_ticks); + self.writel(reg::HCC_MODE, HCC_ENABLE); + self.writel(reg::RX_MODE, RXMODE_ENABLE); + } + + fn tx_post(&self, slot: u64, bd: [u8; 16]) { + self.mem_write(TX_RING + slot * 16, &bd); + } + + /// One whole frame in one descriptor at `slot`, then its doorbell. + fn tx_send_one(&self, slot: u64, frame: &[u8], flags: u16) { + let buf = TX_BUF + slot * 0x800; + self.mem_write(buf, frame); + self.tx_post(slot, tx_bd(buf, frame.len() as u16, flags | TXF_END, 0)); + self.writel(reg::MBX_TX_PROD_LO, slot as u32 + 1); + } + + /// Receive side up: standard ring of `STD_BUF_SIZE` buffers, a return + /// ring of `ret_len`, one completion per update. + fn rx_open(&self, ret_len: u32, coal_bds: u32, coal_ticks: u32) { + self.ind_reg_write(reg::MODE_CTL, MODE_ALL_SWAPS); + self.split_addr(reg::RX_STD_RCB_HADDR_HI, reg::RX_STD_RCB_HADDR_LO, STD_RING); + self.writel(reg::RX_STD_RCB_MAXLEN_FLAGS, STD_BUF_SIZE << 16); + self.ind_mem_write(0x200, 0); + self.ind_mem_write(0x204, RET_RING as u32); + self.ind_mem_write(0x208, ret_len << 16); + self.split_addr(reg::HCC_STATUSBLK_ADDR_HI, reg::HCC_STATUSBLK_ADDR_LO, STATUS); + self.writel(reg::HCC_RX_MAX_COAL_BDS, coal_bds); + self.writel(reg::HCC_RX_COAL_TICKS, coal_ticks); + self.writel(reg::HCC_MODE, HCC_ENABLE); + self.writel(reg::RX_MODE, RXMODE_ENABLE); + self.writel(reg::MBX_IRQ0_LO, 0); + } + + /// Post `count` standard buffers, all at `RX_BUF`. + fn rx_post_std(&self, count: u32) { + for i in 0..u64::from(count) { + self.mem_write(STD_RING + i * 32, &rx_bd(RX_BUF, 0)); + } + self.writel(reg::MBX_RX_STD_PROD_LO, count); + } + + fn rx_open_jumbo(&self) { + self.split_addr(reg::RX_JUMBO_RCB_HADDR_HI, reg::RX_JUMBO_RCB_HADDR_LO, JUMBO_RING); + self.writel(reg::RX_JUMBO_RCB_MAXLEN_FLAGS, JUMBO_BUF_SIZE << 16); + self.mem_write(JUMBO_RING, &rx_bd(JUMBO_BUF, JUMBO_BUF_SIZE as u16)); + self.writel(reg::MBX_RX_JUMBO_PROD_LO, 1); + } + + fn return_descriptor(&self, slot: u64) -> Vec { + self.mem_read(RET_RING + slot * 32, 32) + } +} + +fn broadcast(len: usize, seed: u8) -> Vec { + let mut f: Vec = (0..len).map(|i| (i as u8).wrapping_mul(seed).wrapping_add(1)).collect(); + f[..6].fill(0xff); + f +} + +/// A doorbell posts the status block with UPDATED set, big-endian, once the +/// coalescing block is enabled; disabled it posts nothing and the line stays +/// down. Ported from test_status_block_byte_order and +/// test_status_block_gated_by_hcc. +#[test] +fn the_status_block_is_posted_only_while_host_coalescing_is_enabled() { + let rig = Rig::new(); + rig.ind_mem_write(0x108, TX_RING_LEN << 16); + rig.split_addr(reg::HCC_STATUSBLK_ADDR_HI, reg::HCC_STATUSBLK_ADDR_LO, STATUS); + rig.ind_reg_write(reg::MODE_CTL, MODE_ALL_SWAPS); + rig.writel(reg::MBX_IRQ0_LO, 0); + + rig.writel(reg::MBX_TX_PROD_LO, 0); + assert_eq!(be32(&rig.status_block(), 0), 0); + assert!(!rig.line_up()); + + rig.writel(reg::HCC_MODE, HCC_ENABLE); + rig.writel(reg::RX_MODE, RXMODE_ENABLE); + rig.writel(reg::MBX_TX_PROD_LO, 0); + assert_eq!(rig.status_block()[..4], [0, 0, 0, 1]); + assert!(rig.line_up()); + assert!(rig.irq.level()); +} + +/// The driver word-swaps its 64-bit host addresses for the big-endian host; +/// the chip swaps them back. Ported from test_status_block_hostaddr_wordswap. +#[test] +fn host_addresses_honour_the_word_swap_control() { + let intended = 0x80_0000u64; + let rig = Rig::new(); + rig.ind_mem_write(0x108, TX_RING_LEN << 16); + rig.ind_reg_write(reg::MODE_CTL, MODE_ALL_SWAPS); + rig.cfg_write(0x68 / 4, rig.cfg_read(0x68 / 4) | 8); + // Word-swapped: the low word in ADDR_HI, the high word in ADDR_LO. + rig.writel(reg::HCC_STATUSBLK_ADDR_HI, intended as u32); + rig.writel(reg::HCC_STATUSBLK_ADDR_LO, (intended >> 32) as u32); + rig.writel(reg::HCC_MODE, HCC_ENABLE); + rig.writel(reg::RX_MODE, RXMODE_ENABLE); + rig.writel(reg::MBX_TX_PROD_LO, 0); + assert_eq!(be32(&rig.mem_read(intended, STATUS_BLK_SIZE), 0), 1); +} + +/// Ported from test_transmit_round_trip: the descriptor and its buffer are +/// read, the frame reaches the backend byte for byte, and the status block +/// reports the consumer advanced, in the chip's image order. +#[test] +fn a_transmitted_frame_reaches_the_backend_and_the_status_block_follows() { + let mut rig = Rig::new(); + let frame: Vec = (0..60u8).map(|i| i.wrapping_mul(7).wrapping_add(1)).collect(); + rig.tx_open(TX_RING_LEN, 1, 0); + rig.tx_send_one(0, &frame, 0); + assert_eq!(rig.sent_frames(), [frame]); + let sb = rig.status_block(); + assert_eq!(be32(&sb, 0), 1); + assert_eq!(get16dma(&sb, 18), 1); + // A consumer of 1 at the field's offset xor 2, big-endian. + assert_eq!(sb[16..18], [0, 1]); +} + +/// With no swap controls the chip's image reaches the bus permuted within each +/// doubleword; the frame still arrives intact. Derived from the lane rule the +/// model documents (PG105 Tables 102-109), not a port. +#[test] +fn the_swap_controls_permute_the_bytes_within_each_doubleword() { + for (mode, xor) in [(0u32, 7usize), (0x10, 4), (0x20, 3), (0x30, 0)] { + let mut rig = Rig::new(); + rig.tx_open(TX_RING_LEN, 1, 0); + rig.ind_reg_write(reg::MODE_CTL, mode); + let frame: Vec = (0..64u8).map(|i| i.wrapping_mul(5) ^ 0x21).collect(); + let lanes = |bytes: &[u8], xor: usize| { + let mut out = vec![0; bytes.len()]; + for (i, b) in bytes.iter().enumerate() { + out[(i & !7) + ((i & 7) ^ xor)] = *b; + } + out + }; + // Descriptors take the non-frame controls, which are off here. + rig.mem_write(TX_RING, &lanes(&tx_bd(TX_BUF, 64, TXF_END, 0), 7)); + rig.mem_write(TX_BUF, &lanes(&frame, xor)); + rig.writel(reg::MBX_TX_PROD_LO, 1); + assert_eq!(rig.sent_frames(), [frame], "frame-data control {mode:#x}"); + // The status block is written through the same non-frame lanes. + let sb = lanes(&rig.status_block(), 7); + assert_eq!(be32(&sb, 0), 1); + } +} + +/// A transmit descriptor that cannot be read leaves the ring where it was and +/// the doorbell is retried later. Ported from test_transmit_descriptor_dma_error. +#[test] +fn an_unreadable_send_ring_leaves_the_frame_for_a_retry() { + let mut rig = Rig::new(); + let frame: Vec = (0..60u8).map(|i| i.wrapping_mul(7).wrapping_add(1)).collect(); + rig.tx_open(TX_RING_LEN, 0, 0); + rig.ind_mem_write(0x100, (UNMAPPED >> 32) as u32); + rig.ind_mem_write(0x104, UNMAPPED as u32); + rig.writel(reg::MBX_TX_PROD_LO, 1); + assert!(rig.sent_frames().is_empty()); + assert_eq!(rig.status_tag(), 0); + assert_eq!(rig.refusals(Refused::DmaFault(DmaWhat::TxDescriptor)), 1); + + rig.ind_mem_write(0x104, TX_RING as u32); + rig.mem_write(TX_BUF, &frame); + rig.tx_post(0, tx_bd(TX_BUF, 60, TXF_END, 0)); + rig.writel(reg::MBX_TX_PROD_LO, 1); + assert_eq!(rig.sent_frames(), [frame]); +} + +/// A later fragment's unreadable payload rolls the whole frame back, and an +/// earlier completed frame stays completed. Ported from +/// test_transmit_payload_dma_error and test_transmit_complete_then_fragment_dma_error_retry. +#[test] +fn an_unreadable_fragment_rolls_back_to_the_start_of_its_frame() { + let mut rig = Rig::new(); + let first: Vec = (0..60u8).map(|i| i + 1).collect(); + let second: Vec = (0..60u8).map(|i| i + 100).collect(); + rig.tx_open(8, 1, 0); + rig.mem_write(TX_BUF, &first); + rig.tx_post(0, tx_bd(TX_BUF, 60, TXF_END, 0)); + rig.mem_write(TX_BUF + 0x100, &second[..28]); + rig.tx_post(1, tx_bd(TX_BUF + 0x100, 28, 0, 0)); + rig.tx_post(2, tx_bd(UNMAPPED, 32, TXF_END, 0)); + rig.writel(reg::MBX_TX_PROD_LO, 3); + // The first frame went and is reported; the second waits. + assert_eq!(rig.sent_frames(), [first]); + assert_eq!(rig.status_tx_cons(), 1); + assert_eq!(rig.refusals(Refused::DmaFault(DmaWhat::TxPayload)), 1); + + rig.mem_write(TX_BUF + 0x200, &second[28..]); + rig.tx_post(2, tx_bd(TX_BUF + 0x200, 32, TXF_END, 0)); + rig.writel(reg::MBX_TX_PROD_LO, 3); + assert_eq!(rig.sent_frames(), [second]); + assert_eq!(rig.status_tx_cons(), 3); +} + +/// Ported from test_tx_producer_out_of_range: an index past the ring is +/// refused, observably, and nothing is walked. +#[test] +fn a_transmit_producer_past_the_ring_is_refused() { + let rig = Rig::new(); + rig.ind_mem_write(0x108, TX_RING_LEN << 16); + rig.writel(reg::MBX_TX_PROD_LO, TX_RING_LEN); + rig.writel(reg::MBX_TX_PROD_LO, 0xffff); + assert_eq!(rig.refusals(Refused::ProducerOutOfRange(Mailbox::TxProducer)), 2); + assert_eq!(rig.mii_read(PHY_ADDR, phy_reg::IDR1), IDR1_VALUE); +} + +/// Ported from test_transmit_zero_length_descriptor: nothing goes out, the +/// descriptor still completes. +#[test] +fn a_zero_length_descriptor_completes_without_a_frame() { + let mut rig = Rig::new(); + rig.tx_open(TX_RING_LEN, 1, 0); + rig.tx_post(0, tx_bd(TX_BUF, 0, TXF_END, 0)); + rig.writel(reg::MBX_TX_PROD_LO, 1); + assert_eq!(be32(&rig.status_block(), 0), 1); + assert_eq!(rig.status_tx_cons(), 1); + assert!(rig.sent_frames().is_empty()); +} + +/// Ported from test_transmit_source_address: the descriptor picks one of four +/// filter slots to replace the frame's source, only when it asks to. +#[test] +fn a_descriptor_can_replace_the_source_address_from_a_filter_slot() { + let mut rig = Rig::new(); + rig.tx_open(8, 0, 0); + let slots = [(0x210u32, 0x2030_4050u32), (0x211, 0x2131_4151), (0x212, 0x2232_4252), (0x213, 0x2333_4353)]; + for (i, (hi, lo)) in slots.iter().enumerate() { + rig.writel(reg::MAC_ADDR_0_HIGH + i as u32 * reg::MAC_ADDR_STRIDE, *hi); + rig.writel(reg::MAC_ADDR_0_LOW + i as u32 * reg::MAC_ADDR_STRIDE, *lo); + } + let frame: Vec = (0..60u8).map(|i| 0x80 + i).collect(); + let with_source = |slot: usize| { + let mut f = frame.clone(); + f[6..8].copy_from_slice(&(slots[slot].0 as u16).to_be_bytes()); + f[8..12].copy_from_slice(&slots[slot].1.to_be_bytes()); + f + }; + for (i, selector) in [2u16, 0, 3, 1].into_iter().enumerate() { + rig.tx_send_one(i as u64, &frame, TXF_INSERT_SRC | TXF_CHOOSE_SRC(selector)); + assert_eq!(rig.sent_frames(), [with_source(usize::from(selector))]); + } + // A selector without INSERT leaves the frame untouched. + rig.tx_send_one(4, &frame, TXF_CHOOSE_SRC(3)); + assert_eq!(rig.sent_frames(), std::slice::from_ref(&frame)); + // Split after the first source bytes: both fragments carry the flags. + rig.mem_write(TX_BUF + 5 * 0x800, &frame[..8]); + rig.mem_write(TX_BUF + 6 * 0x800, &frame[8..]); + let flags = TXF_INSERT_SRC | TXF_CHOOSE_SRC(2); + rig.tx_post(5, tx_bd(TX_BUF + 5 * 0x800, 8, flags, 0)); + rig.tx_post(6, tx_bd(TX_BUF + 6 * 0x800, 52, flags | TXF_END, 0)); + rig.writel(reg::MBX_TX_PROD_LO, 7); + assert_eq!(rig.sent_frames(), [with_source(2)]); +} + +fn lso_frame(payload: usize) -> Vec { + let mut f = vec![0u8; 54 + payload]; + f[..14].copy_from_slice(&[0x52, 0x54, 0, 0x11, 0x22, 0x33, 0x52, 0x54, 0, 0xbc, 0x57, 1, 8, 0]); + f[14] = 0x45; + f[16..18].copy_from_slice(&((40 + payload) as u16).to_be_bytes()); + f[18..20].copy_from_slice(&0x1234u16.to_be_bytes()); + f[22] = 64; + f[23] = 6; + f[26..30].copy_from_slice(&[10, 0, 2, 15]); + f[30..34].copy_from_slice(&[10, 0, 2, 2]); + f[34..36].copy_from_slice(&0x3039u16.to_be_bytes()); + f[36..38].copy_from_slice(&0x0017u16.to_be_bytes()); + f[38..42].copy_from_slice(&0x1000u32.to_be_bytes()); + f[46] = 0x50; + f[47] = 0x19; + f[48..50].copy_from_slice(&0x2000u16.to_be_bytes()); + for (i, b) in f[54..].iter_mut().enumerate() { + *b = i as u8; + } + f +} + +fn inet_sum(buf: &[u8], seed: u32) -> u16 { + let mut sum = seed; + for pair in buf.chunks(2) { + sum += u32::from(pair[0]) << 8 | u32::from(*pair.get(1).unwrap_or(&0)); + } + while sum >> 16 != 0 { + sum = (sum & 0xffff) + (sum >> 16); + } + !(sum as u16) +} + +fn tcp_pseudo(frame: &[u8], tcp_len: usize) -> u32 { + frame[26..34].chunks(2).map(|p| u32::from(p[0]) << 8 | u32::from(p[1])).sum::() + 6 + tcp_len as u32 +} + +/// Ported from test_transmit_segmentation: one buffer of three segments goes +/// out as three frames with their own headers, sequence numbers, identifiers +/// and checksums, and the finishing flags only on the last. +#[test] +fn an_oversized_tcp_buffer_is_cut_into_segments() { + const MSS: usize = 512; + let mut rig = Rig::new(); + let frame = lso_frame(MSS * 3); + rig.tx_open(TX_RING_LEN, 0, 0); + rig.mem_write(TX_BUF, &frame); + rig.tx_post(0, tx_bd(TX_BUF, frame.len() as u16, 0x7000 | TXF_END, MSS as u16)); + rig.writel(reg::MBX_TX_PROD_LO, 1); + let sent = rig.sent_frames(); + assert_eq!(sent.len(), 3); + for (i, got) in sent.iter().enumerate() { + let mut expected = frame[..54].to_vec(); + expected.extend_from_slice(&frame[54 + i * MSS..54 + (i + 1) * MSS]); + expected[16..18].copy_from_slice(&((40 + MSS) as u16).to_be_bytes()); + expected[18..20].copy_from_slice(&(0x1234 + i as u16).to_be_bytes()); + expected[38..42].copy_from_slice(&(0x1000 + (i * MSS) as u32).to_be_bytes()); + if i < 2 { + expected[47] &= !0x09; + } + expected[24..26].fill(0); + expected[50..52].fill(0); + let ip = inet_sum(&expected[14..34], 0); + expected[24..26].copy_from_slice(&ip.to_be_bytes()); + let tcp = inet_sum(&expected[34..], tcp_pseudo(&expected, 20 + MSS)); + expected[50..52].copy_from_slice(&tcp.to_be_bytes()); + assert_eq!(got, &expected, "segment {i}"); + } +} + +/// A frame no larger than one segment, or one that is not plain IPv4 TCP, is +/// sent whole. Ported from test_transmit_no_segmentation. +#[test] +fn a_frame_that_needs_no_segmentation_is_sent_whole() { + let mut rig = Rig::new(); + rig.tx_open(TX_RING_LEN, 0, 0); + let small = lso_frame(100); + rig.mem_write(TX_BUF, &small); + rig.tx_post(0, tx_bd(TX_BUF, small.len() as u16, TXF_END, 512)); + rig.writel(reg::MBX_TX_PROD_LO, 1); + assert_eq!(rig.sent_frames(), [small]); +} + +fn cksum_frame(ihl_byte: u8) -> Vec { + let mut f = vec![0u8; 62]; + f[..14].copy_from_slice(&[0x52, 0x54, 0, 0x11, 0x22, 0x33, 0x52, 0x54, 0, 0xbc, 0x57, 1, 8, 0]); + f[14] = ihl_byte; + f[17] = 48; + f[22] = 64; + f[23] = 6; + f[26..30].copy_from_slice(&[10, 0, 2, 15]); + f[30..34].copy_from_slice(&[10, 0, 2, 2]); + f[34..36].copy_from_slice(&[0x30, 0x39]); + f[37] = 0x17; + f[46] = 0x50; + f[47] = 0x02; + f[48] = 0x20; + for (i, b) in f.iter_mut().enumerate().skip(54) { + *b = i as u8; + } + f +} + +/// Ported from test_transmit_checksum_offload: a driver that leaves the +/// checksum fields for the card gets them filled. +#[test] +fn the_card_fills_in_the_checksums_the_driver_asked_for() { + let mut rig = Rig::new(); + let frame = cksum_frame(0x45); + rig.tx_open(TX_RING_LEN, 0, 0); + rig.tx_send_one(0, &frame, TXF_IP_CKSUM | TXF_TCP_UDP_CKSUM); + let got = rig.sent_frames().remove(0); + let ip = inet_sum(&frame[14..34], 0); + let tcp = inet_sum(&frame[34..], tcp_pseudo(&frame, 28)); + assert_eq!(u16::from_be_bytes([got[24], got[25]]), ip); + assert_eq!(u16::from_be_bytes([got[50], got[51]]), tcp); + assert_ne!(ip, 0); + assert_ne!(tcp, 0); + assert_eq!(got[..24], frame[..24]); +} + +/// A header the offload cannot trust is declined, not walked. Ported from +/// test_transmit_checksum_bad_header. +#[test] +fn a_checksum_offload_on_a_malformed_header_is_declined() { + let mut rig = Rig::new(); + let frame = cksum_frame(0x44); + rig.tx_open(TX_RING_LEN, 0, 0); + rig.tx_send_one(0, &frame, TXF_IP_CKSUM | TXF_TCP_UDP_CKSUM); + assert_eq!(rig.sent_frames(), [frame]); +} + +/// One update per group of descriptors, the tick budget reporting a short +/// group, a zero budget never doing so. Ported from test_transmit_coalescing, +/// test_transmit_coalescing_timer and test_transmit_coalescing_zero_ticks. +#[test] +fn transmit_updates_follow_the_count_and_the_tick_budget() { + let mut rig = Rig::new(); + let frame = broadcast(64, 3); + rig.tx_open(64, 4, 0x1000_0000); + for i in 0..8 { + rig.tx_send_one(i, &frame, 0); + } + assert_eq!(rig.sent_frames().len(), 8); + // Eight frames in groups of four are two updates, not eight. + assert_eq!(rig.status_tag(), 2); + + // A short group waits for the budget. + let rig = Rig::new(); + rig.tx_open(64, 8, 50); + for i in 0..3 { + rig.tx_send_one(i, &frame, 0); + } + assert_eq!(rig.status_tag(), 0); + rig.step(Duration::from_micros(49)); + assert_eq!(rig.status_tag(), 0); + rig.step(Duration::from_micros(1)); + assert_eq!(rig.status_tag(), 1); + assert_eq!(rig.status_tx_cons(), 3); + + // Zero disables the timer: only the count reports. + let mut rig = Rig::new(); + rig.tx_open(64, 2, 0); + rig.tx_send_one(0, &frame, 0); + rig.step(Duration::from_secs(1)); + assert_eq!(rig.status_tag(), 0); + rig.tx_send_one(1, &frame, 0); + assert_eq!(rig.status_tag(), 1); + assert_eq!(rig.status_tx_cons(), 2); + assert_eq!(rig.sent_frames().len(), 2); +} + +/// Empty and dropped frames still complete every descriptor they consume. +/// Ported from test_transmit_coalescing_empty_and_dropped. +#[test] +fn empty_and_oversize_frames_still_complete_their_descriptors() { + let mut rig = Rig::new(); + let frame = broadcast(64, 5); + rig.tx_open(64, 4, 0x1000_0000); + rig.mem_write(TX_BUF, &frame); + rig.mem_fill(TX_BUF + 0x1000, 9600, 0xa5); + rig.tx_post(0, tx_bd(TX_BUF, 64, TXF_END, 0)); + rig.tx_post(1, tx_bd(0, 0, TXF_END, 0)); + rig.tx_post(2, tx_bd(TX_BUF + 0x1000, 9600, 0, 0)); + rig.tx_post(3, tx_bd(TX_BUF + 0x4000, 1, TXF_END, 0)); + rig.writel(reg::MBX_IRQ0_LO, 0); + rig.writel(reg::MBX_TX_PROD_LO, 4); + // Only the first is a frame: the oversize pair is dropped whole. + assert_eq!(rig.sent_frames(), [frame]); + assert_eq!(rig.status_tx_cons(), 4); + assert_eq!(rig.status_tag(), 1); + assert!(rig.line_up()); + assert_eq!(rig.refusals(Refused::TxFrameTooLarge), 1); +} + +/// A doorbell that ends inside a frame leaves every fragment to retry; a +/// completed frame earlier is not rolled back. Ported from +/// test_transmit_split_doorbell and test_transmit_complete_then_split_doorbell. +#[test] +fn a_doorbell_that_ends_inside_a_frame_waits_for_the_rest() { + let mut rig = Rig::new(); + let first: Vec = (0..60u8).map(|i| i.wrapping_mul(7).wrapping_add(1)).collect(); + let second: Vec = (0..60u8).map(|i| i.wrapping_mul(3).wrapping_add(9)).collect(); + rig.tx_open(8, 1, 0x1000_0000); + rig.writel(reg::MBX_IRQ0_LO, 0); + rig.mem_write(TX_BUF, &first); + rig.tx_post(0, tx_bd(TX_BUF, 60, TXF_END, 0)); + rig.mem_write(TX_BUF + 0x100, &second[..28]); + rig.mem_write(TX_BUF + 0x200, &second[28..]); + rig.tx_post(1, tx_bd(TX_BUF + 0x100, 28, 0, 0)); + rig.tx_post(2, tx_bd(TX_BUF + 0x200, 32, TXF_END, 0)); + // The doorbell stops after the first fragment of the second frame. + rig.writel(reg::MBX_TX_PROD_LO, 2); + assert_eq!(rig.sent_frames(), [first]); + assert_eq!(rig.status_tx_cons(), 1); + let tag = rig.status_tag(); + rig.writel(reg::MBX_TX_PROD_LO, 3); + assert_eq!(rig.sent_frames(), [second]); + assert_eq!(rig.status_tx_cons(), 3); + assert_eq!(rig.status_tag(), tag + 1); +} + +/// A status DMA the guest's address refused is retried when the address is +/// repaired. Ported from test_transmit_coalescing_status_dma_retry. +#[test] +fn a_refused_status_update_is_retried_by_the_budget_timer() { + let mut rig = Rig::new(); + let frame = broadcast(64, 9); + rig.tx_open(64, 1, 50); + rig.split_addr(reg::HCC_STATUSBLK_ADDR_HI, reg::HCC_STATUSBLK_ADDR_LO, UNMAPPED); + rig.writel(reg::MBX_IRQ0_LO, 0); + rig.tx_send_one(0, &frame, 0); + assert_eq!(rig.sent_frames(), [frame]); + assert_eq!(rig.status_tx_cons(), 0); + assert!(!rig.line_up()); + assert_eq!(rig.refusals(Refused::DmaFault(DmaWhat::StatusBlock)), 1); + + rig.split_addr(reg::HCC_STATUSBLK_ADDR_HI, reg::HCC_STATUSBLK_ADDR_LO, STATUS); + rig.step(Duration::from_micros(51)); + assert_eq!(rig.status_tx_cons(), 1); + assert_eq!(rig.status_tag(), 1); + assert!(rig.line_up()); +} + +/// Ported from test_interrupt_mailbox and test_mailbox_nonzero_masks: the +/// mailbox masks and acknowledges, MISC_CTL's global mask holds the line down +/// without losing the update, and any nonzero value masks. +#[test] +fn the_interrupt_line_follows_the_mailbox_and_the_global_mask() { + let rig = Rig::new(); + rig.ind_mem_write(0x108, TX_RING_LEN << 16); + rig.split_addr(reg::HCC_STATUSBLK_ADDR_HI, reg::HCC_STATUSBLK_ADDR_LO, STATUS); + rig.ind_reg_write(reg::MODE_CTL, MODE_ALL_SWAPS); + rig.writel(reg::HCC_MODE, HCC_ENABLE); + rig.writel(reg::RX_MODE, RXMODE_ENABLE); + + // Masked out of reset: an update posts, but the line stays down. + rig.writel(reg::MBX_TX_PROD_LO, 0); + assert!(!rig.line_up()); + // Enable interrupts; the next update asserts the line. + rig.writel(reg::MBX_IRQ0_LO, 0); + rig.writel(reg::MBX_TX_PROD_LO, 0); + assert!(rig.line_up() && rig.irq.level()); + // Write 1: masks and acknowledges. + rig.writel(reg::MBX_IRQ0_LO, 1); + assert!(!rig.line_up() && !rig.irq.level()); + // An update posted while masked is discarded by the zero write. + rig.writel(reg::MBX_TX_PROD_LO, 0); + rig.writel(reg::MBX_IRQ0_LO, 0); + assert!(!rig.line_up()); + rig.writel(reg::MBX_TX_PROD_LO, 0); + assert!(rig.line_up()); + rig.writel(reg::MBX_IRQ0_LO, 0); + assert!(!rig.line_up()); + rig.writel(reg::MBX_TX_PROD_LO, 0); + assert!(rig.line_up()); + + // The global mask suppresses the line; the update was never taken. + let misc = rig.cfg_read(0x68 / 4); + rig.cfg_write(0x68 / 4, misc | 2); + assert!(!rig.line_up() && !rig.irq.level()); + rig.cfg_write(0x68 / 4, misc); + assert!(rig.line_up() && rig.irq.level()); + // CLEAR_INTA acknowledges and self-clears. + rig.cfg_write(0x68 / 4, misc | 1); + assert!(!rig.line_up()); + assert_eq!(rig.cfg_read(0x68 / 4), misc); + + // Any nonzero In_ISR value masks while the handler runs. + rig.writel(reg::MBX_IRQ0_LO, 2); + rig.writel(reg::MBX_TX_PROD_LO, 0); + assert!(!rig.line_up()); + rig.writel(reg::MBX_IRQ0_LO, 0); + assert!(!rig.line_up()); + rig.writel(reg::MBX_TX_PROD_LO, 0); + assert!(rig.line_up()); +} + +/// A failed status write advances neither its tag nor the interrupt state. +/// Ported from test_status_dma_error. +#[test] +fn a_failed_status_write_advances_nothing() { + let rig = Rig::new(); + rig.ind_mem_write(0x108, TX_RING_LEN << 16); + rig.ind_reg_write(reg::MODE_CTL, MODE_ALL_SWAPS); + rig.split_addr(reg::HCC_STATUSBLK_ADDR_HI, reg::HCC_STATUSBLK_ADDR_LO, UNMAPPED); + rig.writel(reg::HCC_MODE, HCC_ENABLE); + rig.writel(reg::RX_MODE, RXMODE_ENABLE); + rig.writel(reg::MBX_IRQ0_LO, 0); + rig.writel(reg::MBX_TX_PROD_LO, 0); + assert!(!rig.line_up()); + rig.split_addr(reg::HCC_STATUSBLK_ADDR_HI, reg::HCC_STATUSBLK_ADDR_LO, STATUS); + rig.writel(reg::MBX_TX_PROD_LO, 0); + assert_eq!(rig.status_tag(), 1); + assert!(rig.line_up()); +} + +/// Ported from test_receive_round_trip: the frame lands in the posted buffer, +/// the return descriptor and status block describe it in the chip's image +/// order, and a second buffer exposes the next consumer. +#[test] +fn a_received_frame_lands_in_the_posted_buffer_with_its_completion() { + let mut rig = Rig::new(); + let frame = broadcast(64, 5); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + rig.deliver(&frame); + rig.step(ARRIVAL); + assert_eq!(rig.mem_read(RX_BUF, 64), frame); + let ret = rig.return_descriptor(0); + assert_eq!(get16dma(&ret, 0x0a), 0); + assert_eq!(get16dma(&ret, 0x08), 64 + 4); + assert_eq!(get16dma(&ret, 0x0c) & 4, 4); + assert_eq!(ret[0x0a..0x0c], [0, 0x44]); + let sb = rig.status_block(); + assert_eq!(get16dma(&sb, 16), 1); + assert_eq!(sb[8..12], [0, 1, 0, 0]); + assert_eq!(sb[18..20], [0, 1]); + + // The return-ring acknowledgement is accepted; reposting exposes the + // next consumer. + rig.writel(reg::MBX_RX_CONS0_LO, 1); + rig.mem_write(STD_RING + 32, &rx_bd(RX_BUF + 0x1000, 0)); + rig.writel(reg::MBX_RX_STD_PROD_LO, 2); + rig.deliver(&frame); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 2); + assert_eq!(rig.status_block()[8..10], [0, 2]); +} + +/// The return descriptor is written when the frame lands; the producer the +/// status block reports, and the interrupt, wait for the arrival deadline. +/// Ported from test_receive_waits_for_its_arrival_deadline. +#[test] +fn a_completion_waits_for_its_arrival_deadline() { + let mut rig = Rig::new(); + let frame = broadcast(64, 7); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + rig.deliver(&frame); + assert_eq!(rig.mem_read(RX_BUF, 64), frame); + assert_eq!(get16dma(&rig.return_descriptor(0), 0x0c) & 4, 4); + assert_eq!(rig.status_rx_prod(), 0); + assert!(!rig.line_up()); + rig.step(ARRIVAL - Duration::from_nanos(1)); + assert_eq!(rig.status_rx_prod(), 0); + assert!(!rig.line_up()); + rig.step(Duration::from_nanos(1)); + assert_eq!(rig.status_rx_prod(), 1); + assert!(rig.line_up()); +} + +/// Ported from test_receive_jumbo_frame: a frame beyond the standard buffer +/// goes to the jumbo ring and is marked so. +#[test] +fn a_frame_too_big_for_the_standard_buffer_takes_the_jumbo_ring() { + let mut rig = Rig::new(); + let frame = broadcast(9000, 7); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + rig.rx_open_jumbo(); + rig.deliver(&frame); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 1); + let ret = rig.return_descriptor(0); + assert_eq!(get16dma(&ret, 0x08), 9004); + assert_eq!(get16dma(&ret, 0x0c) & 0x20, 0x20); + assert_eq!(rig.mem_read(JUMBO_BUF, 9000), frame); + let sb = rig.status_block(); + assert_eq!(sb[8..12], [0, 0, 0, 1]); +} + +/// With the standard ring empty, a standard-size frame is discarded and +/// counted rather than refused, while a jumbo buffer is posted; the next jumbo +/// frame still gets through. Ported from +/// test_receive_drops_standard_with_empty_standard_ring. +#[test] +fn a_short_frame_with_no_standard_buffer_is_discarded_and_counted() { + let mut rig = Rig::new(); + let first = broadcast(64, 3); + let second = broadcast(64, 11); + let big = broadcast(9000, 13); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + rig.rx_open_jumbo(); + rig.mem_write(STD_RING + 32, &rx_bd(STD_STALE_BUF, 0)); + assert_eq!(rig.nic_no_more_rxbds(), 0); + rig.deliver(&first); + rig.step(ARRIVAL); + rig.writel(reg::MBX_RX_CONS0_LO, 1); + rig.mem_fill(STD_STALE_BUF, 64, 0xa5); + rig.mem_fill(JUMBO_BUF, 64, 0xa5); + rig.deliver(&second); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 1); + assert_eq!(rig.mem_read(STD_STALE_BUF, 64), vec![0xa5; 64]); + assert_eq!(rig.mem_read(JUMBO_BUF, 64), vec![0xa5; 64]); + assert_eq!(rig.nic_no_more_rxbds(), 1); + assert_eq!(rig.refusals(Refused::RxShortage(Ring::Standard)), 1); + // The jumbo ring still takes a jumbo frame. + rig.deliver(&big); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 2); + assert_eq!(rig.mem_read(JUMBO_BUF, 9000), big); + // Posting the standard buffer afterwards does not resurrect the frame. + rig.writel(reg::MBX_RX_STD_PROD_LO, 2); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 2); + assert_eq!(rig.nic_no_more_rxbds(), 1); +} + +/// The mirror: a jumbo frame with the jumbo ring empty. Ported from +/// test_receive_drops_oversize_with_empty_jumbo_ring. +#[test] +fn a_jumbo_frame_with_no_jumbo_buffer_is_discarded_and_counted() { + let mut rig = Rig::new(); + let first = broadcast(9000, 3); + let second = broadcast(9000, 11); + let small = broadcast(64, 15); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(2); + rig.rx_open_jumbo(); + rig.mem_write(JUMBO_RING + 32, &rx_bd(0x28_0000, JUMBO_BUF_SIZE as u16)); + rig.deliver(&first); + rig.step(ARRIVAL); + rig.writel(reg::MBX_RX_CONS0_LO, 1); + rig.mem_fill(0x28_0000, 9000, 0xa5); + rig.deliver(&second); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 1); + assert_eq!(rig.mem_read(0x28_0000, 9000), vec![0xa5; 9000]); + assert_eq!(rig.nic_no_more_rxbds(), 1); + assert_eq!(rig.refusals(Refused::RxShortage(Ring::Jumbo)), 1); + rig.deliver(&small); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 2); + assert_eq!(rig.mem_read(RX_BUF, 64), small); + assert_eq!(get16dma(&rig.return_descriptor(1), 0x0c) & 0x20, 0); +} + +/// A frame too big for a standard buffer with the jumbo ring disabled is +/// dropped, observably. Ported from test_receive_oversize_dropped. +#[test] +fn an_oversize_frame_with_no_jumbo_ring_is_dropped_and_counted() { + let mut rig = Rig::new(); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + rig.mem_fill(RX_BUF, 2048, 0xa5); + rig.deliver(&broadcast(2000, 3)); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 0); + assert_eq!(rig.mem_read(RX_BUF, 2048), vec![0xa5; 2048]); + assert_eq!(rig.refusals(Refused::RxNoRoom(Ring::Standard)), 1); + // The frame was consumed, not left to wedge the queue. + assert_eq!(rig.waiting(), 0); + // And one over the model's limit is dropped too. + rig.deliver(&broadcast(9700, 3)); + assert_eq!(rig.refusals(Refused::RxFrameTooLarge), 1); +} + +/// Ported from test_receive_mac_enable: a halted receiver drops what it is +/// offered; with no buffer the frame waits for the guest to make room. +#[test] +fn a_halted_receiver_drops_and_a_full_one_waits() { + let mut rig = Rig::new(); + let frame = broadcast(64, 3); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + rig.writel(reg::RX_MODE, 0); + rig.deliver(&frame); + assert_eq!(rig.waiting(), 0); + assert_eq!(rig.status_rx_prod(), 0); + rig.writel(reg::RX_MODE, RXMODE_ENABLE); + rig.deliver(&frame); + // The one posted buffer is used; the next frame waits at the head. + rig.deliver(&frame); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 1); + assert_eq!(rig.waiting(), 1); + rig.mem_write(STD_RING + 32, &rx_bd(RX_BUF + 0x1000, 0)); + rig.writel(reg::MBX_RX_STD_PROD_LO, 2); + rig.writel(reg::MBX_RX_CONS0_LO, 1); + rig.dev.poll(); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 2); + assert_eq!(rig.mem_read(RX_BUF + 0x1000, 64), frame); +} + +/// The address filter is slot 0, which the driver programs: other unicast +/// frames are consumed without a completion. Ported from the filter cases of +/// test_receive_checksum_case's setup and test_receive_round_trip. +#[test] +fn the_receive_filter_follows_the_programmed_station_address() { + let mut rig = Rig::new(); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + let mut other = broadcast(64, 3); + other[..6].copy_from_slice(&[2, 0, 0, 0, 0, 9]); + rig.deliver(&other); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 0); + assert_eq!(rig.waiting(), 0); + let mut mine = other.clone(); + mine[..6].copy_from_slice(&MAC); + rig.deliver(&mine); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 1); + // IRIX installs its own address over the card's. + rig.writel(reg::MAC_ADDR_0_HIGH, 0x0200); + rig.writel(reg::MAC_ADDR_0_LOW, 9); + rig.mem_write(STD_RING + 32, &rx_bd(RX_BUF + 0x1000, 0)); + rig.writel(reg::MBX_RX_STD_PROD_LO, 2); + rig.writel(reg::MBX_RX_CONS0_LO, 1); + rig.deliver(&other); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 2); +} + +/// A runt is padded to the Ethernet minimum before it is delivered. +#[test] +fn a_short_frame_is_padded_to_the_ethernet_minimum() { + let mut rig = Rig::new(); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + let mut runt = vec![0xff; 6]; + runt.extend_from_slice(&[1, 2, 3, 4]); + rig.deliver(&runt); + rig.step(ARRIVAL); + let mut padded = runt.clone(); + padded.resize(60, 0); + assert_eq!(rig.mem_read(RX_BUF, 60), padded); + assert_eq!(get16dma(&rig.return_descriptor(0), 0x08), 64); +} + +fn rx_frame(proto: u8, payload: &[u8], fix: impl Fn(&mut Vec)) -> Vec { + let mut f = vec![0xff; 6]; + f.extend_from_slice(&[0x52, 0x54, 0, 0xbc, 0x57, 1, 8, 0]); + let total = 20 + payload.len(); + f.extend_from_slice(&[0x45, 0]); + f.extend_from_slice(&(total as u16).to_be_bytes()); + f.extend_from_slice(&[0, 1, 0, 0, 64, proto, 0, 0, 10, 0, 2, 2, 10, 0, 2, 15]); + f.extend_from_slice(payload); + f.resize(f.len().max(60), 0); + fix(&mut f); + let ip = inet_sum(&f[14..34], 0); + f[24..26].copy_from_slice(&ip.to_be_bytes()); + f +} + +/// The receive checksum report: an intact TCP segment claims both sums +/// (0xffff) and the OK flags; a corrupt one claims the IP sum only; a frame +/// that is not IPv4 TCP or UDP claims nothing. Ported from +/// test_receive_checksum_offload, test_receive_checksum_bad_transport and +/// test_receive_checksum_not_ip. +#[test] +fn the_receive_report_claims_only_the_sums_it_computed() { + let mut tcp = vec![0u8; 20]; + tcp[0..4].copy_from_slice(&[0x30, 0x39, 0, 0x17]); + tcp[12] = 0x50; + let seal = |f: &mut Vec| { + f[14 + 20 + 16..14 + 20 + 18].fill(0); + let sum = inet_sum(&f[34..54], tcp_pseudo(f, 20)); + f[50..52].copy_from_slice(&sum.to_be_bytes()); + }; + for (label, frame, ip_ok, l4_ok, ip_report, l4_report) in [ + ("intact", rx_frame(6, &tcp, seal), true, true, 0xffffu32, 0xffffu32), + ("corrupt", rx_frame(6, &tcp, |_| ()), true, false, 0xffff, 0), + ] { + let mut rig = Rig::new(); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + rig.deliver(&frame); + rig.step(ARRIVAL); + let ret = rig.return_descriptor(0); + let flags = get16dma(&ret, 0x0c); + assert_eq!(flags & 0x1000 != 0, ip_ok, "{label} ip flag"); + assert_eq!(flags & 0x2000 != 0, l4_ok, "{label} transport flag"); + assert_eq!(flags & 0x4000, 0x4000, "{label} tcp flag"); + assert_eq!(u32::from(get16dma(&ret, 0x12)), ip_report, "{label} ip sum"); + let l4 = u32::from(get16dma(&ret, 0x10)); + if l4_ok { + assert_eq!(l4, l4_report); + } else { + assert_ne!(l4, 0xffff); + } + } + // Not IPv4: nothing is claimed. + let mut rig = Rig::new(); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + let mut arp = broadcast(60, 3); + arp[12..14].copy_from_slice(&[0x08, 0x06]); + rig.deliver(&arp); + rig.step(ARRIVAL); + let ret = rig.return_descriptor(0); + assert_eq!(get16dma(&ret, 0x0c) & 0x7000, 0); + assert_eq!(be32(&ret, 0x10), 0); +} + +/// Receive coalescing: with a count of two, two frames make one update; the +/// tick budget reports a short group. Ported from +/// test_receive_interrupt_coalescing and test_receive_coalescing_timer. +#[test] +fn receive_updates_follow_the_count_and_the_tick_budget() { + let mut rig = Rig::new(); + let frame = broadcast(64, 3); + rig.rx_open(16, 2, 0x1000_0000); + rig.rx_post_std(4); + for _ in 0..4 { + rig.deliver(&frame); + } + rig.step(ARRIVAL); + assert_eq!(rig.status_tag(), 2); + assert_eq!(rig.status_rx_prod(), 4); + + let mut rig = Rig::new(); + rig.rx_open(16, 8, 1000); + rig.rx_post_std(4); + rig.deliver(&frame); + rig.deliver(&frame); + rig.step(ARRIVAL); + assert_eq!(rig.status_tag(), 0); + rig.step(Duration::from_micros(800)); + assert_eq!(rig.status_tag(), 1); + assert_eq!(rig.status_rx_prod(), 2); + + // Neither criterion armed: the guest has asked to be told nothing. + let mut rig = Rig::new(); + rig.rx_open(16, 0, 0); + rig.rx_post_std(2); + rig.deliver(&frame); + rig.step(Duration::from_secs(1)); + assert_eq!(rig.status_tag(), 0); +} + +/// Ported from test_rx_producer_out_of_range and the ring-length guards: +/// out-of-range indexes are refused, observably, and not stored. +#[test] +fn receive_mailbox_indexes_past_their_rings_are_refused() { + let rig = Rig::new(); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.writel(reg::MBX_RX_STD_PROD_LO, 512); + rig.writel(reg::MBX_RX_JUMBO_PROD_LO, 256); + rig.writel(reg::MBX_RX_CONS0_LO, RX_RING_LEN); + assert_eq!(rig.readl(reg::MBX_RX_STD_PROD_LO), 0); + assert_eq!(rig.refusals(Refused::ProducerOutOfRange(Mailbox::RxStandardProducer)), 1); + assert_eq!(rig.refusals(Refused::ProducerOutOfRange(Mailbox::RxJumboProducer)), 1); + assert_eq!(rig.refusals(Refused::ProducerOutOfRange(Mailbox::RxReturnConsumer)), 1); +} + +/// A return-ring descriptor that cannot be read or written leaves the frame +/// queued; repairing the address delivers it. Ported from +/// test_receive_descriptor_dma_error and test_receive_completion_dma_error. +#[test] +fn an_unreachable_ring_defers_the_frame_until_it_is_repaired() { + let mut rig = Rig::new(); + let frame = broadcast(64, 3); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.split_addr(reg::RX_STD_RCB_HADDR_HI, reg::RX_STD_RCB_HADDR_LO, UNMAPPED); + rig.mem_write(STD_RING, &rx_bd(RX_BUF, 0)); + rig.writel(reg::MBX_RX_STD_PROD_LO, 1); + rig.deliver(&frame); + assert_eq!(rig.status_rx_prod(), 0); + assert_eq!(rig.waiting(), 1); + assert_eq!(rig.refusals(Refused::DmaFault(DmaWhat::RxDescriptor)), 1); + rig.split_addr(reg::RX_STD_RCB_HADDR_HI, reg::RX_STD_RCB_HADDR_LO, STD_RING); + rig.writel(reg::MBX_RX_STD_PROD_LO, 1); + rig.dev.poll(); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 1); + assert_eq!(rig.mem_read(RX_BUF, 64), frame); + + // The same for the return ring. + let mut rig = Rig::new(); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.ind_mem_write(0x204, UNMAPPED as u32); + rig.ind_mem_write(0x200, (UNMAPPED >> 32) as u32); + rig.rx_post_std(1); + rig.deliver(&frame); + assert_eq!(rig.refusals(Refused::DmaFault(DmaWhat::RxCompletion)), 1); + rig.ind_mem_write(0x200, 0); + rig.ind_mem_write(0x204, RET_RING as u32); + rig.writel(reg::MBX_RX_STD_PROD_LO, 1); + rig.dev.poll(); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 1); +} + +/// A global reset cancels pending coalescing work and the held completions. +/// Ported from test_global_reset_cancels_receive_coalescing and +/// test_system_reset_cancels_transmit_coalescing. +#[test] +fn a_reset_cancels_coalescing_timers_and_held_completions() { + let rig = Rig::new(); + let frame = broadcast(64, 3); + rig.tx_open(64, 8, 50); + rig.tx_send_one(0, &frame, 0); + rig.writel(reg::MISC_CFG, MISC_CFG_RESET); + rig.step(Duration::from_millis(1)); + assert_eq!(rig.status_tag(), 0); + + let mut rig = Rig::new(); + rig.rx_open(16, 1, 0); + rig.rx_post_std(1); + rig.deliver(&frame); + rig.dev.reset(); + rig.step(Duration::from_millis(1)); + assert_eq!(rig.status_rx_prod(), 0); + assert!(!rig.irq.level()); +} + +/// The receive thread takes a frame when the guest posts a buffer and +/// publishes its completion on the clock, with nothing polling by hand; stop +/// joins it. Exercises `Device::start` and `stop`, which the cases above +/// replace with explicit steps. +#[test] +fn the_receive_thread_delivers_and_publishes_on_its_own() { + let mut rig = Rig::new(); + *rig.dev.frozen.lock() = None; + let frame = broadcast(64, 3); + rig.rx_open(RX_RING_LEN, 1, 0); + Device::start(&*rig.dev); + assert!(Device::is_running(&*rig.dev)); + // A frame offered before any buffer is posted waits at the queue head. + rig.inject.push(frame.clone()).unwrap(); + rig.dev.wake_rx(); + rig.rx_post_std(1); + let deadline = Instant::now() + Duration::from_secs(5); + while rig.status_rx_prod() != 1 && Instant::now() < deadline { + std::thread::sleep(Duration::from_millis(2)); + } + Device::stop(&*rig.dev); + assert!(!Device::is_running(&*rig.dev)); + assert_eq!(rig.status_rx_prod(), 1); + assert_eq!(rig.mem_read(RX_BUF, 64), frame); + assert!(rig.line_up()); +} + +// ---- the rest of the plain-memory qtests ---- + +impl Rig { + /// What the receive thread does: take what it can, let the arrival + /// deadline pass. + fn pump(&mut self) { + self.dev.poll(); + self.step(ARRIVAL); + } + fn ret_flags_end(&self, slot: u64) -> u16 { + get16dma(&self.return_descriptor(slot), 0x0c) & 4 + } +} + +/// A completed frame ahead of a descriptor read failure stays completed; the +/// failed frame retries from its first descriptor once the address is +/// repaired. Ported from test_transmit_complete_then_fragment_dma_error_retry +/// (both its empty-first and 60-byte-first variants). +#[test] +fn a_frame_with_an_unreadable_second_fragment_retries_after_the_first_completes() { + for empty_first in [false, true] { + let mut rig = Rig::new(); + let first: Vec = (0..60u8).map(|i| i.wrapping_mul(7).wrapping_add(1)).collect(); + let second: Vec = (0..60u8).map(|i| i.wrapping_mul(5).wrapping_add(3)).collect(); + rig.tx_open(TX_RING_LEN, 1, 0); + rig.writel(reg::MBX_IRQ0_LO, 0); + rig.mem_write(TX_BUF, &first); + rig.mem_write(TX_BUF + 0x100, &second[..28]); + rig.tx_post(0, tx_bd(TX_BUF, if empty_first { 0 } else { 60 }, TXF_END, 0)); + rig.tx_post(1, tx_bd(TX_BUF + 0x100, 28, 0, 0)); + rig.tx_post(2, tx_bd(UNMAPPED, 32, TXF_END, 0)); + let tag = rig.status_tag(); + rig.writel(reg::MBX_TX_PROD_LO, 3); + let expect_first = if empty_first { vec![] } else { vec![first.clone()] }; + assert_eq!(rig.sent_frames(), expect_first); + assert_eq!(rig.status_tx_cons(), 1); + assert_eq!(rig.status_tag(), tag + 1); + assert!(rig.line_up()); + assert_eq!(rig.refusals(Refused::DmaFault(DmaWhat::TxPayload)), 1); + + rig.writel(reg::MBX_IRQ0_LO, 0); + rig.mem_write(TX_BUF + 0x100 + 28, &second[28..]); + rig.tx_post(2, tx_bd(TX_BUF + 0x100 + 28, 32, TXF_END, 0)); + rig.writel(reg::MBX_TX_PROD_LO, 3); + assert_eq!(rig.sent_frames(), [second]); + assert_eq!(rig.status_tx_cons(), 3); + assert_eq!(rig.status_tag(), tag + 2); + assert!(rig.line_up()); + } +} + +/// The same when it is the next descriptor that cannot be read: the ring's +/// last 16 bytes sit at the end of memory. Ported from +/// test_transmit_complete_then_descriptor_dma_error_retry. +#[test] +fn a_frame_with_an_unreadable_next_descriptor_retries_after_the_first_completes() { + let mut rig = Rig::new(); + let first: Vec = (0..60u8).map(|i| i.wrapping_mul(7).wrapping_add(1)).collect(); + let second: Vec = (0..60u8).map(|i| i.wrapping_mul(5).wrapping_add(3)).collect(); + let initial_ring = MEM_SIZE as u64 - 16; + let repaired_ring = TX_RING - 16; + rig.tx_open(TX_RING_LEN, 1, 0); + rig.ind_mem_write(0x104, initial_ring as u32); + rig.writel(reg::MBX_IRQ0_LO, 0); + rig.mem_write(initial_ring, &tx_bd(TX_BUF, 60, TXF_END, 0)); + rig.mem_write(TX_BUF, &first); + let tag = rig.status_tag(); + rig.writel(reg::MBX_TX_PROD_LO, 2); + assert_eq!(rig.sent_frames(), [first]); + assert_eq!(rig.status_tx_cons(), 1); + assert_eq!(rig.status_tag(), tag + 1); + assert!(rig.line_up()); + assert_eq!(rig.refusals(Refused::DmaFault(DmaWhat::TxDescriptor)), 1); + + rig.writel(reg::MBX_IRQ0_LO, 0); + rig.mem_write(repaired_ring, &tx_bd(TX_BUF, 60, TXF_END, 0)); + rig.mem_write(TX_RING, &tx_bd(TX_BUF + 0x100, 60, TXF_END, 0)); + rig.mem_write(TX_BUF + 0x100, &second); + rig.ind_mem_write(0x104, repaired_ring as u32); + rig.writel(reg::MBX_TX_PROD_LO, 2); + assert_eq!(rig.sent_frames(), [second]); + assert_eq!(rig.status_tx_cons(), 2); + assert_eq!(rig.status_tag(), tag + 2); +} + +/// A count of two with a completed frame, a failed one and a repaired retry: +/// the retried frame's two descriptors report at once, and the budget timer +/// has nothing left to say. Ported from +/// test_transmit_partial_error_descriptor_coalescing. +#[test] +fn a_partly_failed_doorbell_coalesces_by_the_descriptors_that_completed() { + let mut rig = Rig::new(); + let first: Vec = (0..60u8).map(|i| i.wrapping_mul(7).wrapping_add(1)).collect(); + let second: Vec = (0..60u8).map(|i| i.wrapping_mul(5).wrapping_add(3)).collect(); + rig.tx_open(TX_RING_LEN, 2, 50); + rig.writel(reg::MBX_IRQ0_LO, 0); + rig.mem_write(TX_BUF, &first); + rig.tx_post(0, tx_bd(TX_BUF, 60, TXF_END, 0)); + rig.tx_post(1, tx_bd(0, 0, TXF_END, 0)); + rig.tx_post(2, tx_bd(UNMAPPED, 60, TXF_END, 0)); + let tag = rig.status_tag(); + rig.writel(reg::MBX_TX_PROD_LO, 3); + assert_eq!(rig.sent_frames(), [first]); + assert_eq!(rig.status_tx_cons(), 2); + assert_eq!(rig.status_tag(), tag + 1); + assert!(rig.line_up()); + // The qtest's wait for the log line steps the clock a millisecond at a + // time, so the budget has run out by the retry. + rig.step(Duration::from_millis(1)); + rig.writel(reg::MBX_IRQ0_LO, 0); + rig.mem_write(TX_BUF + 0x100, &second); + rig.tx_post(2, tx_bd(TX_BUF + 0x100, 60, TXF_END, 0)); + rig.writel(reg::MBX_TX_PROD_LO, 3); + assert_eq!(rig.sent_frames(), [second]); + assert_eq!(rig.status_tx_cons(), 3); + assert_eq!(rig.status_tag(), tag + 2); + assert!(rig.line_up()); + // The budget gives up on a group that already reported. + rig.step(Duration::from_micros(1050)); + assert_eq!(rig.status_tx_cons(), 3); + assert_eq!(rig.status_tag(), tag + 2); +} + +/// A reply handed back straight after the transmit sees the transmit +/// completion already published. The qtest runs an ARP exchange through +/// QEMU's user-mode network; here the backend queue plays the gateway. Adapted +/// from test_user_backend_reply_waits_for_transmit. +#[test] +fn a_reply_to_a_transmit_finds_the_transmit_already_reported() { + let mut rig = Rig::new(); + let mut arp_request = vec![0xff; 6]; + arp_request.extend_from_slice(&[0x52, 0x54, 0, 0x12, 0x34, 0x56, 8, 6, 0, 1, 8, 0, 6, 4, 0, 1]); + arp_request.extend_from_slice(&[0x52, 0x54, 0, 0x12, 0x34, 0x56, 10, 0, 2, 15, 0, 0, 0, 0, 0, 0, 10, 0, 2, 2]); + assert_eq!(arp_request.len(), 42); + rig.writel(reg::MAC_ADDR_0_HIGH, 0x5254); + rig.writel(reg::MAC_ADDR_0_LOW, 0x0012_3456); + rig.tx_open(TX_RING_LEN, 2, 1_000_000); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + rig.tx_send_one(0, &arp_request, 0); + assert_eq!(rig.sent_frames(), [arp_request.clone()]); + // The gateway's reply, to the address just programmed. + let mut reply = arp_request[6..12].to_vec(); + reply.extend_from_slice(&[0x52, 0x55, 10, 0, 2, 2, 8, 6, 0, 1, 8, 0, 6, 4, 0, 2]); + reply.extend_from_slice(&[0x52, 0x55, 10, 0, 2, 2, 10, 0, 2, 2]); + reply.extend_from_slice(&arp_request[22..32]); + reply.resize(60, 0); + rig.deliver(&reply); + rig.step(ARRIVAL); + assert_eq!(rig.status_tx_cons(), 1); + assert_eq!(rig.status_rx_prod(), 1); + assert!(rig.line_up()); + assert_eq!(rig.mem_read(RX_BUF, 60), reply); +} + +/// A frame bigger than the standard buffer, with no jumbo ring, must not +/// write past the buffer. Ported from test_receive_overruns_nothing. +#[test] +fn an_oversize_frame_never_writes_past_the_posted_buffer() { + let mut rig = Rig::new(); + let frame = broadcast(4096, 7); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + rig.mem_fill(RX_BUF, 4096, 0xa5); + rig.deliver(&frame); + rig.step(ARRIVAL); + assert_eq!(rig.mem_read(RX_BUF + u64::from(STD_BUF_SIZE), 4096 - STD_BUF_SIZE as usize), vec![0xa5; 4096 - 1536]); +} + +fn lap_open(rig: &Rig, coal_bds: u32) { + rig.rx_open(8, coal_bds, 0); + rig.writel(reg::RX_STD_RCB_MAXLEN_FLAGS, 1518 << 16); + rig.rx_post_std(64); +} + +/// Five trips round an eight-entry return ring, each completion visible in +/// its slot before the driver acknowledges it. Ported from +/// test_receive_ring_laps. +#[test] +fn the_return_ring_laps_cleanly() { + let mut rig = Rig::new(); + let frame = broadcast(64, 2); + lap_open(&rig, 0); + for i in 0..40u32 { + let idx = u64::from(i % 8); + rig.mem_fill(RET_RING + idx * 32, 32, 0); + rig.deliver(&frame); + rig.step(ARRIVAL); + assert_eq!(rig.ret_flags_end(idx), 4, "frame {i}"); + assert_eq!(get16dma(&rig.return_descriptor(idx), 0x08), 68); + rig.writel(reg::MBX_RX_CONS0_LO, (i + 1) % 8); + rig.writel(reg::MBX_RX_STD_PROD_LO, (64 + i + 1) % 512); + } +} + +/// A full return ring holds frames back rather than overwrite completions the +/// driver owes; freeing space with a consumer write releases them, without +/// another interrupt acknowledgement. Ported from +/// test_receive_ring_full_recovers. +#[test] +fn a_full_return_ring_holds_frames_until_the_driver_frees_it() { + let mut rig = Rig::new(); + let frame = broadcast(64, 3); + lap_open(&rig, 1); + for _ in 0..20 { + rig.deliver(&frame); + } + rig.step(ARRIVAL * 4); + // Seven fit; the last slot stays clear and the rest wait in the backend. + assert_eq!(rig.status_rx_prod(), 7); + assert_eq!(rig.ret_flags_end(7), 0); + assert_eq!(rig.waiting(), 13); + rig.writel(reg::MBX_IRQ0_LO, 0); + assert!(!rig.line_up()); + rig.step(ARRIVAL); + assert_eq!(rig.ret_flags_end(7), 0); + for i in 0..8 { + rig.mem_fill(RET_RING + i * 32, 32, 0); + } + rig.writel(reg::MBX_RX_CONS0_LO, 7); + rig.pump(); + // The retry fills the ring a second time, wrapping to slot 6. + assert_eq!(rig.status_rx_prod(), 6); + assert_eq!(rig.ret_flags_end(6), 0); + for i in 0..8 { + rig.mem_fill(RET_RING + i * 32, 32, 0); + } + rig.writel(reg::MBX_RX_CONS0_LO, 6); + rig.pump(); + assert_eq!(rig.status_rx_prod(), 4); +} + +/// A halted receiver takes nothing and keeps nothing: frames from before it +/// is enabled are gone. Ported from test_receive_mac_enable. +#[test] +fn frames_offered_to_a_halted_receiver_are_not_kept() { + let mut rig = Rig::new(); + let frame = broadcast(64, 11); + rig.rx_open(8, 1, 0); + rig.writel(reg::RX_MODE, 0); + rig.rx_post_std(8); + assert_eq!(rig.readl(reg::RX_MODE) & RXMODE_ENABLE, 0); + for _ in 0..4 { + rig.deliver(&frame); + } + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 0); + assert_eq!(rig.waiting(), 0); + rig.writel(reg::RX_MODE, RXMODE_ENABLE); + rig.pump(); + assert_eq!(rig.status_rx_prod(), 0); + assert_eq!(rig.ret_flags_end(0), 0); + let fresh = broadcast(64, 17); + rig.deliver(&fresh); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 1); + assert!(rig.line_up()); + assert_eq!(get16dma(&rig.return_descriptor(0), 0x08), 68); + // Halting again is not a bring-up special case. + rig.writel(reg::RX_MODE, 0); + rig.deliver(&broadcast(64, 19)); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 1); + rig.writel(reg::RX_MODE, RXMODE_ENABLE); + rig.pump(); + assert_eq!(rig.status_rx_prod(), 1); + rig.deliver(&broadcast(64, 23)); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 2); + assert_eq!(rig.ret_flags_end(1), 4); +} + +/// A frame offered before any buffer is posted waits; posting one releases +/// it. Ported from test_queued_receive_released_by_producer. +#[test] +fn a_waiting_frame_is_released_by_the_producer_mailbox() { + let mut rig = Rig::new(); + let frame = broadcast(64, 29); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.mem_write(STD_RING, &rx_bd(RX_BUF, 0)); + rig.deliver(&frame); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 0); + assert_eq!(rig.ret_flags_end(0), 0); + assert_eq!(rig.waiting(), 1); + rig.writel(reg::MBX_RX_STD_PROD_LO, 1); + rig.pump(); + assert_eq!(rig.status_rx_prod(), 1); + assert_eq!(rig.mem_read(RX_BUF, 64), frame); + assert_eq!(get16dma(&rig.return_descriptor(0), 0x0a), 0); +} + +/// A frame queued before a reset must not reach the rings brought up after +/// it, by chip reset or machine reset. Ported from test_receive_queue_reset. +#[test] +fn a_reset_discards_frames_the_backend_was_holding() { + for machine_reset in [false, true] { + let mut rig = Rig::new(); + let stale = broadcast(64, 0x51); + let fresh = broadcast(64, 0xa7); + // Nothing is posted yet, so the stale frame waits at the head. + rig.rx_open(RX_RING_LEN, 1, 0); + rig.deliver(&stale); + assert_eq!(rig.waiting(), 1); + if machine_reset { + rig.dev.reset(); + } else { + rig.writel(reg::MISC_CFG, MISC_CFG_RESET); + } + assert_eq!(rig.waiting(), 0); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + rig.deliver(&fresh); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 1); + assert_eq!(rig.mem_read(RX_BUF, 64), fresh); + } +} + +/// An unreadable posted buffer defers the frame, leaves the descriptor +/// unconsumed, and the repaired buffer delivers it. Derived from the receive +/// payload read in the model; the qtest of the same subject is +/// test_receive_payload_dma_error, whose log-marker checks are not ported. +#[test] +fn an_unreachable_receive_buffer_defers_the_frame() { + let mut rig = Rig::new(); + let frame = broadcast(64, 3); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.mem_write(STD_RING, &rx_bd(UNMAPPED, 0)); + rig.writel(reg::MBX_RX_STD_PROD_LO, 1); + rig.deliver(&frame); + assert_eq!(rig.waiting(), 1); + assert_eq!(rig.status_rx_prod(), 0); + assert_eq!(rig.refusals(Refused::DmaFault(DmaWhat::RxPayload)), 1); + rig.mem_write(STD_RING, &rx_bd(RX_BUF, 0)); + rig.writel(reg::MBX_RX_STD_PROD_LO, 1); + rig.pump(); + assert_eq!(rig.status_rx_prod(), 1); + assert_eq!(rig.mem_read(RX_BUF, 64), frame); + assert_eq!(get16dma(&rig.return_descriptor(0), 0x0a), 0); +} + +/// With neither criterion armed the receive side stays silent however long +/// the ring is idle, and so does the transmit side. Ported from +/// test_receive_without_a_criterion_is_silent and +/// test_transmit_without_a_criterion_is_silent. +#[test] +fn without_a_criterion_no_update_is_ever_posted() { + let mut rig = Rig::new(); + rig.rx_open(64, 0, 0); + rig.rx_post_std(1); + rig.deliver(&broadcast(64, 3)); + rig.step(Duration::from_secs(1)); + assert_eq!(rig.ret_flags_end(0), 4); + assert_eq!(be32(&rig.status_block(), 0), 0); + assert_eq!(rig.status_rx_prod(), 0); + assert!(!rig.line_up()); + + let mut rig = Rig::new(); + let frame = broadcast(64, 5); + rig.tx_open(64, 0, 0); + rig.writel(reg::MBX_IRQ0_LO, 0); + rig.tx_send_one(0, &frame, 0); + assert_eq!(rig.sent_frames().len(), 1); + rig.step(Duration::from_secs(1)); + assert_eq!(be32(&rig.status_block(), 0), 0); + assert_eq!(rig.status_tx_cons(), 0); + assert!(!rig.line_up()); +} + +/// A budget that has run out reports the next event as it happens, in either +/// direction. Ported from test_receive_expired_timer_reports_at_once and +/// test_transmit_expired_timer_reports_at_once. +#[test] +fn an_expired_budget_reports_the_next_event_at_once() { + let mut rig = Rig::new(); + rig.rx_open(64, 8, 50); + rig.rx_post_std(2); + rig.step(Duration::from_micros(200)); + assert_eq!(rig.status_tag(), 0); + rig.deliver(&broadcast(64, 3)); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 1); + assert_eq!(rig.status_tag(), 1); + + let mut rig = Rig::new(); + rig.tx_open(64, 8, 50); + rig.step(Duration::from_micros(200)); + assert_eq!(rig.status_tag(), 0); + rig.tx_send_one(0, &broadcast(64, 3), 0); + assert_eq!(rig.sent_frames().len(), 1); + assert_eq!(rig.status_tag(), 1); +} + +/// The producer the status block names follows its completion: the +/// descriptor is written, the next slot is still empty. Ported from +/// test_receive_producer_follows_its_completion. +#[test] +fn the_receive_producer_names_only_completed_descriptors() { + let mut rig = Rig::new(); + rig.rx_open(64, 1, 0); + rig.rx_post_std(3); + for i in 0..3u64 { + rig.deliver(&broadcast(64, 0x55)); + rig.step(ARRIVAL); + assert_eq!(u64::from(rig.status_rx_prod()), i + 1); + let bd = rig.return_descriptor(i); + assert_eq!(get16dma(&bd, 0x0c) & 4, 4); + assert_eq!(get16dma(&bd, 0x08), 68); + let next = rig.return_descriptor(i + 1); + assert_eq!(get16dma(&next, 0x08), 0); + assert_eq!(get16dma(&next, 0x0c), 0); + } +} + +/// A receive update the bus refused is retried by the budget timer once the +/// address is repaired. Ported from test_receive_coalescing_status_dma_retry. +#[test] +fn a_refused_receive_update_is_retried_by_the_budget_timer() { + let mut rig = Rig::new(); + rig.rx_open(RX_RING_LEN, 1, 50); + rig.split_addr(reg::HCC_STATUSBLK_ADDR_HI, reg::HCC_STATUSBLK_ADDR_LO, UNMAPPED); + rig.rx_post_std(1); + rig.deliver(&broadcast(64, 0x6d)); + rig.step(ARRIVAL); + assert_eq!(rig.ret_flags_end(0), 4); + assert_eq!(rig.status_rx_prod(), 0); + assert!(!rig.line_up()); + rig.split_addr(reg::HCC_STATUSBLK_ADDR_HI, reg::HCC_STATUSBLK_ADDR_LO, STATUS); + rig.step(Duration::from_micros(1051)); + assert_eq!(rig.status_rx_prod(), 1); + assert_eq!(rig.status_tag(), 1); + assert!(rig.line_up()); +} + +fn cksum_rx_frame() -> Vec { + let mut f = cksum_frame(0x45); + f[..6].fill(0xff); + seal_tcp(&mut f); + f +} +fn seal_tcp(f: &mut [u8]) { + f[24..26].fill(0); + let ip = inet_sum(&f[14..34], 0); + f[24..26].copy_from_slice(&ip.to_be_bytes()); + f[50..52].fill(0); + let tcp = inet_sum(&f[34..62], tcp_pseudo(f, 28)); + f[50..52].copy_from_slice(&tcp.to_be_bytes()); +} +fn seal_udp(f: &mut [u8]) { + f[23] = 17; + f[16] = 0; + f[17] = 48; + f[38] = 0; + f[39] = 28; + f[40..42].fill(0); + f[24..26].fill(0); + let ip = inet_sum(&f[14..34], 0); + f[24..26].copy_from_slice(&ip.to_be_bytes()); + let pseudo: u32 = f[26..34].chunks(2).map(|p| u32::from(p[0]) << 8 | u32::from(p[1])).sum::() + 17 + 28; + let udp = inet_sum(&f[34..62], pseudo); + f[40..42].copy_from_slice(&udp.to_be_bytes()); +} +fn inet_fix_ip(f: &mut [u8]) { + f[24..26].fill(0); + let ip = inet_sum(&f[14..34], 0); + f[24..26].copy_from_slice(&ip.to_be_bytes()); +} + +/// One row of test_receive_checksum_case: the flags, the IP sum and the +/// transport sum a frame earns under a MODE_CTL setting. +fn rx_checksum_case(frame: &[u8], mode: u32, flags: u16, ip: u16, l4: u16, label: &str) { + let mut rig = Rig::new(); + rig.rx_open(RX_RING_LEN, 1, 0); + rig.rx_post_std(1); + rig.ind_reg_write(reg::MODE_CTL, MODE_ALL_SWAPS | mode); + rig.deliver(frame); + rig.step(ARRIVAL); + assert_eq!(rig.status_rx_prod(), 1, "{label}"); + let ret = rig.return_descriptor(0); + let got = get16dma(&ret, 0x0c); + assert_eq!(got, flags | 4, "{label} flags"); + assert_eq!(got & 3, 0, "{label}"); + assert_eq!(get16dma(&ret, 0x12), ip, "{label} ip sum"); + assert_eq!(get16dma(&ret, 0x10), l4, "{label} transport sum"); +} + +/// Ported from test_receive_checksum_offload, _udp, _udp_zero, +/// _malformed_udp_length, _bad_ip, _bad_transport, _no_pseudo, _padding, +/// _fragment, _later_fragment, _malformed_tcp_header, _malformed_length and +/// _raw_mode. +#[test] +fn the_receive_checksum_report_covers_every_malformed_case() { + let tcp = cksum_rx_frame(); + rx_checksum_case(&tcp, 0, 0x7000, 0xffff, 0xffff, "tcp"); + + let mut udp = cksum_frame(0x45); + udp[..6].fill(0xff); + seal_udp(&mut udp); + rx_checksum_case(&udp, 0, 0x3000, 0xffff, 0xffff, "udp"); + + // UDP with the checksum omitted: IP only, and the sum is 0xffff. + let mut udp_zero = udp.clone(); + udp_zero[42..44].copy_from_slice(&[0x62, 0x6a]); + udp_zero[40..42].fill(0); + rx_checksum_case(&udp_zero, 0, 0x1000, 0xffff, 0xffff, "udp, no checksum"); + + let mut bad_udp_len = udp.clone(); + bad_udp_len[39] = 27; + rx_checksum_case(&bad_udp_len, 0, 0x1000, 0xffff, 0, "udp length"); + + let mut bad_ip = tcp.clone(); + bad_ip[22] -= 1; + rx_checksum_case(&bad_ip, 0, 0x6000, 0xfeff, 0xffff, "bad ip"); + + let mut bad_tcp = tcp.clone(); + bad_tcp[51] += 1; + rx_checksum_case(&bad_tcp, 0, 0x5000, 0xffff, 0x0001, "bad transport"); + + rx_checksum_case(&tcp, 1 << 23, 0x5000, 0xffff, !inet_sum(&tcp[34..62], 0), "no pseudo header"); + + let mut padded = tcp.clone(); + padded.extend_from_slice(&[0xa5; 8]); + rx_checksum_case(&padded, 0, 0x7000, 0xffff, 0xffff, "padding"); + + let mut fragment = tcp.clone(); + fragment[20] = 0x20; + inet_fix_ip(&mut fragment); + rx_checksum_case(&fragment, 0, 0x5000, 0xffff, 0, "more fragments"); + + let mut later = tcp.clone(); + later[21] = 1; + inet_fix_ip(&mut later); + rx_checksum_case(&later, 0, 0x5000, 0xffff, 0, "later fragment"); + + let mut short_header = tcp.clone(); + short_header[46] = 0x40; + rx_checksum_case(&short_header, 0, 0x5000, 0xffff, 0, "tcp data offset"); + + let mut bad_length = tcp.clone(); + bad_length[16..18].copy_from_slice(&[1, 0]); + rx_checksum_case(&bad_length, 0, 0, 0, 0, "ip length"); + + rx_checksum_case(&tcp, 1 << 9, 0, 0, 0, "raw mode"); +} diff --git a/src/dev/bedrock_flash.rs b/src/dev/bedrock_flash.rs new file mode 100644 index 000000000..b34170dd7 --- /dev/null +++ b/src/dev/bedrock_flash.rs @@ -0,0 +1,931 @@ +//! Bedrock's boot flash: two independent 8 MiB arrays, selected 32 MiB +//! apart, with the AMD command set and CFI query data. +//! +//! Boot aliases and executable views share the same array bytes. Each array +//! has its own command state; instruction fetches read stored bytes even +//! while register reads return command status. +//! +//! Ported from hw/misc/sgi-bedrock-flash.c and hw/mips/sgi/initial-flash.c +//! in jamesbraid/qemu-sgi-origami. + + + +use super::sflash::amd::{AmdCore, Effect, SECTOR_SIZE}; +use super::sflash::storage::FlashBacking; +use crate::cpu::mips_cache_shadow::R16000_800_CONFIG0; +use crate::traits::{ + BusDevice, BusRead16, BusRead32, BusRead64, BusRead8, Resettable, BUS_ERR, BUS_OK, +}; +use parking_lot::Mutex; +use std::io; +use std::path::Path; +use std::sync::{ + atomic::{AtomicU64, Ordering}, + Arc, +}; + +pub const PART_SIZE: usize = 8 << 20; +pub const FLASH_SIZE: usize = 2 * PART_SIZE; + +/// The environment records that mark the empty processor positions of a +/// two-processor IP35 board, slots B and D, disabled, with their offsets +/// in the store: the records the stock PROM writes itself after its +/// failed-start recovery has given up on each empty position. Seeding a +/// fresh store with them spares the PROM three power cycles per position. +/// That the board ships in this state is a model policy, not a recovered +/// factory image. +/// +/// Ported from tests/functional/mips64/lib_origin300.py in +/// jamesbraid/qemu-sgi-origami. +pub fn two_processor_environment() -> Vec<(usize, Vec)> { + const VALUE: &[u8] = b"32: CPU failed early init."; + let record = |slot: u8| { + let mut bytes = vec![0x10]; + bytes.extend_from_slice(b"Disable"); + bytes.push(slot); + bytes.extend_from_slice(&[0; 7]); + bytes.extend_from_slice(VALUE); + bytes.extend_from_slice(&[0; 22]); + bytes + }; + let records = PART_SIZE + 0x1e0100; + vec![(records, record(b'B')), (records + 0x40, record(b'D'))] +} +pub const SECOND_POSITION: u64 = 32 << 20; + +fn sum(bytes: &[u8]) -> u8 { + bytes.iter().fold(0u8, |s, b| s.wrapping_add(*b)) +} + +/// The fields of the PROM's configuration block that depend on the board. +/// +/// The PROM reads them from flash before it touches the processor, so they +/// must agree with the CPU the machine builds: a mismatch in the boot-mode +/// word sends it to its error routine before it initialises GP. The values +/// are inputs the QEMU models choose for each board; none is a measured +/// flash dump. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct FirmwareRecord { + /// The CPU boot-mode word the PROM compares with the processor's mode. + pub cpu_mode_word: u32, + /// The mode bits the processor latches into CP0 Config at reset: Config + /// bits 24:0 (the R10000 family's reset-mode inputs). + pub reset_mode: u32, + /// The processor clock the PROM reports and times against, in Hz. + pub cpu_clock_hz: u32, + /// The record's machine type field. + pub machine_type: u32, + /// How the record reaches the configuration block. + pub header: Header, +} + +/// How a board's record is put into the configuration block at 0x60. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Header { + /// Rewrite the board's fields over the payload's own block, keeping its + /// byte sum, whenever a store is formatted or opened. QEMU configures a + /// Fuel's flash this way. + Refreshed, + /// Replace the whole block with the one authored for the board's flash + /// seed, once, when a store is formatted. An opened store keeps the block + /// it holds, as QEMU leaves the header of a supplied IP53 flash image + /// alone, so a flash update the PROM wrote survives. + Authored { checksum_adjustment: u8 }, +} + +impl FirmwareRecord { + /// The Fuel's 500 MHz R14000 with a 2 MiB secondary cache. + pub const FUEL: Self = Self { + cpu_mode_word: 0x549a_a985, + reset_mode: 0x009a_a985, + cpu_clock_hz: 500_000_000, + machine_type: 2, + header: Header::Refreshed, + }; + /// The Tezro's 700 MHz R16000 with a 4 MiB secondary cache, as the + /// Tezro flash profile for PROM 6.210 records it: the stock IP35 PROM's + /// word for 700 MHz and 4 MiB with cache tap 0xc, and the check + /// adjustment that profile writes. + pub const TEZRO: Self = Self { + cpu_mode_word: 0x589b_ad85, + reset_mode: 0x009b_ad85, + cpu_clock_hz: 700_000_000, + machine_type: 0, + header: Header::Authored { checksum_adjustment: 0xda }, + }; + /// The Tezro's 800 MHz R16000 with a 4 MiB secondary cache: the QEMU + /// Tezro offering's word for this part (cache tap 0xa), whose low 25 + /// bits are the part's Config reset word in the CPU models. No flash + /// profile writes this block; it is the 700 MHz one with this word and + /// the clock, and a check adjustment chosen to keep its byte sum. + pub const TEZRO_800: Self = Self { + cpu_mode_word: 0x549b_af85, + reset_mode: R16000_800_CONFIG0, + cpu_clock_hz: 800_000_000, + machine_type: 0, + header: Header::Authored { checksum_adjustment: 0xff }, + }; + /// CP0 Config at reset: the fixed 32 KiB primary cache sizes (fields 3 + /// in bits 31:29 and 28:26) with the board's mode bits. + pub const fn reset_config0(&self) -> u32 { + (3 << 29) | (3 << 26) | (self.reset_mode & 0x01ff_ffff) + } + + /// The configuration block written into an IP53 seed: the format word, + /// the board's mode word, the `ip27conf` magic, the processor and + /// 200 MHz controller clocks as 64-bit words, the fixed 1,250,000 word, + /// 16-cycle flash read and write timing, the machine type, the check + /// adjustment, a program count of one and the PROM's 6.210 version. + /// Only that profile defines these fields; their PROM meaning beyond that + /// is not established here. + fn authored_block(&self, checksum_adjustment: u8) -> [u8; CONFIG_LEN] { + let mut block = [0u8; CONFIG_LEN]; + let mut put = |offset: usize, bytes: &[u8]| block[offset..offset + bytes.len()].copy_from_slice(bytes); + put(0x00, &21u32.to_be_bytes()); + put(0x04, &self.cpu_mode_word.to_be_bytes()); + put(0x08, b"ip27conf"); + put(0x10, &u64::from(self.cpu_clock_hz).to_be_bytes()); + put(0x18, &200_000_000u64.to_be_bytes()); + put(0x20, &1_250_000u64.to_be_bytes()); + for (offset, value) in [ + (0x28, 1), + (0x2c, 16), + (0x30, self.machine_type), + (0x34, u32::from(checksum_adjustment)), + (0x38, 1), + (0x3c, 16), + (0x40, 6), + (0x44, 210), + (0x48, 0), + ] { + put(offset, &u32::to_be_bytes(value)); + } + block + } +} + +/// Where the PROM's configuration block sits in the payload, and its size. +const CONFIG_BASE: usize = 0x60; +const CONFIG_LEN: usize = 0x50; + +/// Put the board's record into a store being formatted. +fn format_header(data: &mut [u8], record: &FirmwareRecord) { + match record.header { + Header::Refreshed => configure(data, record), + Header::Authored { checksum_adjustment } => { + data[CONFIG_BASE..CONFIG_BASE + CONFIG_LEN].copy_from_slice(&record.authored_block(checksum_adjustment)); + } + } +} + +/// Put the board's record into a store that was opened rather than formatted. +fn reopen_header(data: &mut [u8], record: &FirmwareRecord) { + match record.header { + Header::Refreshed => configure(data, record), + Header::Authored { .. } => {} + } +} + +/// Apply the board's firmware record while retaining the configuration sum. +fn configure(data: &mut [u8], record: &FirmwareRecord) { + let config = &mut data[CONFIG_BASE..CONFIG_BASE + CONFIG_LEN]; + let old = sum(config); + config[0x34..0x38].fill(0); + config[4..8].copy_from_slice(&record.cpu_mode_word.to_be_bytes()); + config[0x14..0x18].copy_from_slice(&record.cpu_clock_hz.to_be_bytes()); + config[0x30..0x34].copy_from_slice(&record.machine_type.to_be_bytes()); + config[0x38..0x3c].copy_from_slice(&1u32.to_be_bytes()); + let adjustment = old.wrapping_sub(sum(config)); + config[0x34..0x38].copy_from_slice(&(adjustment as u32).to_be_bytes()); +} + +struct State { + data: Vec, + parts: [AmdCore; 2], + backing: Option, +} + +impl State { + fn new(data: Vec) -> Self { + Self { + data, + parts: std::array::from_fn(|_| AmdCore::default()), + backing: None, + } + } + // Keep the in-memory array unchanged until each backing write succeeds. + // If a later range fails, restore the old bytes for all ranges already tried. + fn persist(&mut self, ranges: &[(usize, &[u8])]) -> io::Result<()> { + let Some(backing) = &mut self.backing else { + return Ok(()); + }; + for (index, &(offset, bytes)) in ranges.iter().enumerate() { + if let Err(error) = backing.write_range(offset, bytes) { + for &(restore_offset, restore_bytes) in &ranges[..=index] { + let old = &self.data[restore_offset..restore_offset + restore_bytes.len()]; + if let Err(rollback) = backing.write_range(restore_offset, old) { + log::error!( + "bedrock-flash: backing rollback at {restore_offset:#x} failed: {rollback}" + ); + } + } + return Err(error); + } + } + Ok(()) + } + fn replace(&mut self, offset: usize, bytes: &[u8]) -> io::Result { + if self.data[offset..offset + bytes.len()] == *bytes { + return Ok(false); + } + self.persist(&[(offset, bytes)])?; + self.data[offset..offset + bytes.len()].copy_from_slice(bytes); + Ok(true) + } + fn erase(&mut self, part: usize, sectors: &[usize]) -> io::Result { + let erased = vec![0xff; SECTOR_SIZE]; + let ranges: Vec<_> = sectors + .iter() + .map(|sector| (part * PART_SIZE + sector * SECTOR_SIZE, erased.as_slice())) + .collect(); + self.persist(&ranges)?; + let mut changed = false; + for (offset, bytes) in ranges { + let data = &mut self.data[offset..offset + bytes.len()]; + changed |= data.iter().any(|byte| *byte != 0xff); + data.fill(0xff); + } + Ok(changed) + } +} + +pub struct Flash { + st: Mutex, + gen: AtomicU64, +} + +impl Flash { + /// Format a fresh two-position store from a decoded payload, with the + /// board's firmware record. + pub fn new(payload: &[u8], record: FirmwareRecord) -> io::Result { + Self::seeded(payload, record, &[]) + } + + /// [`Self::new`], with environment `records` written into the volume + /// after its label, each at its own offset into the store. + pub fn seeded(payload: &[u8], record: FirmwareRecord, records: &[(usize, Vec)]) -> io::Result { + if payload.len() > PART_SIZE { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + "PROM payload exceeds 8 MiB", + )); + } + let mut data = vec![0xff; FLASH_SIZE]; + data[..payload.len()].copy_from_slice(payload); + let label = PART_SIZE + 0x1e0000; + data[label + 0x10..label + 0x14].copy_from_slice(&0x504c4f49u32.to_be_bytes()); + data[label + 0x14..label + 0x18].copy_from_slice(&1u32.to_be_bytes()); + data[label + 0x18..label + 0x1c].copy_from_slice(&1u32.to_be_bytes()); + format_header(&mut data, &record); + for (offset, bytes) in records { + data[*offset..offset + bytes.len()].copy_from_slice(bytes); + } + Ok(Self { + st: Mutex::new(State::new(data)), + gen: AtomicU64::new(0), + }) + } + + /// Load an exact 16 MiB store, or seed a missing/empty one. Wrong sizes are + /// refused without writes. Reopening reapplies a refreshed firmware record + /// and leaves an authored one as the store holds it. + /// Completed guest programming and erasure write through to this file. + pub fn with_file(payload: &[u8], path: impl AsRef, record: FirmwareRecord) -> io::Result { + Self::with_file_seeded(payload, path, record, &[]) + } + + /// [`Self::with_file`], seeding a missing or empty store with `records` + /// as [`Self::seeded`] does. A store that exists keeps what it holds. + pub fn with_file_seeded(payload: &[u8], path: impl AsRef, record: FirmwareRecord, records: &[(usize, Vec)]) -> io::Result { + let mut data = Self::seeded(payload, record, records)?.st.into_inner().data; + let mut backing = FlashBacking::open(path, &mut data).map_err(|error| { + if error.kind() == io::ErrorKind::InvalidData { + io::Error::new( + io::ErrorKind::InvalidData, + "flash store must be exactly 16 MiB (or empty)", + ) + } else { + error + } + })?; + reopen_header(&mut data, &record); + // Opening an existing store loads it before a refreshed record's + // fields are applied; an empty store was seeded from this same data. + backing.write_range(0, &data)?; + let flash = Self { + st: Mutex::new(State { + data, + parts: std::array::from_fn(|_| AmdCore::default()), + backing: Some(backing), + }), + gen: AtomicU64::new(0), + }; + Ok(flash) + } + + pub fn contents(&self) -> Vec { + self.st.lock().data.clone() + } + pub fn generation(&self) -> u64 { + self.gen.load(Ordering::Acquire) + } + pub fn reset(&self) { + let mut st = self.st.lock(); + for part in &mut st.parts { + part.reset(); + } + self.gen.fetch_add(1, Ordering::Release); + } + /// Advance the model clock independently of flash reads. + pub fn advance_ns(&self, ns: u64) { + let mut st = self.st.lock(); + for part in 0..2 { + let selecting = st.parts[part].selecting(); + let complete = st.parts[part].advance_ns(ns); + if selecting && !st.parts[part].selecting() { + self.gen.fetch_add(1, Ordering::Release); + } + if !complete { + continue; + } + let sectors: Vec<_> = st.parts[part].selected_sectors().collect(); + let success = match st.erase(part, §ors) { + Ok(_) => true, + Err(e) => { + log::error!("bedrock-flash: position {part} erase backing write failed: {e}"); + false + } + }; + st.parts[part].finish_erase(success); + self.gen.fetch_add(1, Ordering::Release); + } + } + pub fn register_locks(self: &Arc) { + let me = self.clone(); + crate::locks::register_lock_fn("bedrock_flash::data", move || me.st.is_locked()); + } + /// Use the same shared flash for local and remote boot spaces. + pub fn boot_view(self: &Arc) -> FlashView { + FlashView { + flash: self.clone(), + range: None, + } + } + /// A contiguous executable alias. Offsets refer to the 16 MiB backing. + pub fn array_view(self: &Arc, offset: usize, len: usize) -> io::Result { + if offset.checked_add(len).is_none_or(|end| end > FLASH_SIZE) { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "Fuel flash alias exceeds backing", + )); + } + Ok(FlashView { + flash: self.clone(), + range: Some((offset, len)), + }) + } +} +impl Resettable for Flash { + fn power_on(&self) { + self.reset(); + } +} + +pub struct FlashView { + flash: Arc, + range: Option<(usize, usize)>, +} +impl FlashView { + fn offset(&self, addr: u64, width: usize) -> Option { + if let Some((base, len)) = self.range { + let off = usize::try_from(addr).ok()?; + return off + .checked_add(width) + .filter(|end| *end <= len) + .map(|_| base + off); + } + let (base, within) = if addr < PART_SIZE as u64 { + (0, addr) + } else if addr >= SECOND_POSITION { + (PART_SIZE, addr - SECOND_POSITION) + } else { + return None; + }; + let within = usize::try_from(within).ok()?; + within + .checked_add(width) + .filter(|end| *end <= PART_SIZE) + .map(|_| base + within) + } + fn read(&self, addr: u64, width: usize) -> Option { + let off = self.offset(addr, width)?; + let mut st = self.flash.st.lock(); + if self.range.is_none() { + let part = off / PART_SIZE; + if !st.parts[part].array_reads() && (width != 2 || off & 1 != 0) { + return None; + } + if width == 2 && off & 1 == 0 { + if let Some(value) = st.parts[part].read16(off % PART_SIZE) { + return Some(value as u64); + } + } + } + Some( + st.data[off..off + width] + .iter() + .fold(0, |v, b| (v << 8) | *b as u64), + ) + } + fn refuse(&self, addr: u64, width: usize, value: u64) -> u32 { + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "bedrock-flash: unsupported {width}-byte store {value:#x} at {addr:#x}; invalid bus cycle or executable view" + ); + BUS_ERR + } +} +impl BusDevice for FlashView { + fn read8(&self, a: u64) -> BusRead8 { + self.read(a, 1) + .map_or_else(BusRead8::err, |v| BusRead8::ok(v as u8)) + } + fn read16(&self, a: u64) -> BusRead16 { + self.read(a, 2) + .map_or_else(BusRead16::err, |v| BusRead16::ok(v as u16)) + } + fn read32(&self, a: u64) -> BusRead32 { + self.read(a, 4) + .map_or_else(BusRead32::err, |v| BusRead32::ok(v as u32)) + } + fn read64(&self, a: u64) -> BusRead64 { + self.read(a, 8).map_or_else(BusRead64::err, BusRead64::ok) + } + fn write8(&self, a: u64, v: u8) -> u32 { + self.refuse(a, 1, v as u64) + } + fn write16(&self, a: u64, v: u16) -> u32 { + self.refuse(a, 2, v as u64) + } + fn write32(&self, a: u64, v: u32) -> u32 { + self.refuse(a, 4, v as u64) + } + fn write64(&self, a: u64, v: u64) -> u32 { + if self.range.is_some() || a & 7 != 0 || v > u16::MAX as u64 { + return self.refuse(a, 8, v); + } + let (part, within) = if a < SECOND_POSITION { + (0, a) + } else { + (1, a - SECOND_POSITION) + }; + let Some(end) = within.checked_add(8) else { + return self.refuse(a, 8, v); + }; + if end > SECOND_POSITION { + return self.refuse(a, 8, v); + } + let offset = (within / 4) as usize; + let mut st = self.flash.st.lock(); + let effect = match st.parts[part].write16(offset, v as u16) { + Ok(effect) => effect, + Err(()) => { + self.flash.gen.fetch_add(1, Ordering::Release); + return BUS_ERR; + } + }; + if let Effect::Program { offset, value } = effect { + let backing = part * PART_SIZE + offset; + let old = u16::from_be_bytes([st.data[backing], st.data[backing + 1]]); + match st.replace(backing, &(old & value).to_be_bytes()) { + Ok(_) => {} + Err(e) => { + log::error!("bedrock-flash: program backing write at {backing:#x} failed: {e}"); + self.flash.gen.fetch_add(1, Ordering::Release); + return BUS_ERR; + } + } + } + self.flash.gen.fetch_add(1, Ordering::Release); + BUS_OK + } + #[cfg(feature = "jitv2")] + fn gen_ptr(&self, _addr: u64) -> *const AtomicU64 { + &self.flash.gen + } +} + +#[cfg(test)] +mod tests { + // Cases follow tests/qtest/sn1-bedrock-flash-test.c in + // jamesbraid/qemu-sgi-origami: program, sector-erase, erase-polling, + // reset-states and backing-file. + use super::*; + fn unlock(v: &FlashView, part: u64) { + assert_eq!(v.write64(part + 8 * 0x555, 0xaa), crate::traits::BUS_OK); + assert_eq!(v.write64(part + 8 * 0x2aa, 0x55), crate::traits::BUS_OK); + } + fn program(v: &FlashView, part: u64, off: u64, value: u16) { + unlock(v, part); + assert_eq!(v.write64(part + 8 * 0x555, 0xa0), crate::traits::BUS_OK); + assert_eq!( + v.write64(part + 4 * off, value as u64), + crate::traits::BUS_OK + ); + } + fn erase_select(v: &FlashView, part: u64, offset: u64) { + unlock(v, part); + assert_eq!(v.write64(part + 8 * 0x555, 0x80), BUS_OK); + unlock(v, part); + assert_eq!(v.write64(part + 4 * offset, 0x30), BUS_OK); + } + + /// The two-processor seed holds DisableB and DisableD after the volume + /// label, byte for byte as the QEMU Origin 300 boot tests seed them, + /// and an existing store keeps what it holds. + #[test] + fn the_two_processor_seed_disables_slots_b_and_d() { + let records = two_processor_environment(); + let f = Flash::seeded(&[], FirmwareRecord::TEZRO, &records).unwrap(); + let bytes = f.contents(); + let mut expected = vec![0x10]; + expected.extend_from_slice(b"DisableB\0\0\0\0\0\0\032: CPU failed early init."); + expected.extend_from_slice(&[0; 22]); + assert_eq!(&bytes[0x9e0100..0x9e0140], expected.as_slice()); + expected[8] = b'D'; + assert_eq!(&bytes[0x9e0140..0x9e0180], expected.as_slice()); + assert!(bytes[0x9e0180..0x9e0200].iter().all(|&b| b == 0xff)); + assert_eq!(Flash::new(&[], FirmwareRecord::TEZRO).unwrap().contents()[0x9e0100], 0xff); + } + + /// The PROM reads CPU configuration and the volume label from its boot flash. + #[test] + fn the_seed_exposes_fuel_configuration_and_volume_label() { + let seed = vec![0x21; 0xb0]; + let f = Flash::new(&seed, FirmwareRecord::FUEL).unwrap(); + let bytes = f.contents(); + assert_eq!(sum(&bytes[0x60..0xb0]), sum(&seed[0x60..0xb0])); + assert_eq!(&bytes[0x64..0x68], &0x549aa985u32.to_be_bytes()); + assert_eq!(&bytes[0x74..0x78], &500_000_000u32.to_be_bytes()); + assert_eq!(&bytes[0x90..0x94], &2u32.to_be_bytes()); + let label = PART_SIZE + 0x1e0000; + assert_eq!(&bytes[label..label + 16], &[0xff; 16]); + assert_eq!( + &bytes[label + 16..label + 28], + &[0x50, 0x4c, 0x4f, 0x49, 0, 0, 0, 1, 0, 0, 0, 1] + ); + assert!(bytes[label + 28..].iter().all(|b| *b == 0xff)); + assert!(Flash::new(&vec![0; PART_SIZE + 1], FirmwareRecord::FUEL).is_err()); + } + /// The stock image's factory block: the 400 MHz mode word, machine type 0 + /// and a zero check adjustment, before any board configures it. + fn factory_block() -> Vec { + let mut image = vec![0x21; 0xb0]; + image[0x60..0xb0].fill(0); + image[0x64..0x68].copy_from_slice(&0x5219a785u32.to_be_bytes()); + image[0x70..0x78].copy_from_slice(&400_000_000u64.to_be_bytes()); + image + } + + /// A refreshed record leaves the block's byte sum unchanged: the check + /// adjustment absorbs the difference. Follows test_ip35_configure in + /// tests/unit/test-sgi-initial-flash.c, which pins 0x9b for the Fuel. + #[test] + fn a_refreshed_record_configures_the_block_and_keeps_its_byte_sum() { + let record = FirmwareRecord::FUEL; + let image = factory_block(); + let bytes = Flash::new(&image, record).unwrap().contents(); + let block = &bytes[0x60..0xb0]; + assert_eq!(&block[4..8], &record.cpu_mode_word.to_be_bytes()); + assert_eq!(&block[0x14..0x18], &record.cpu_clock_hz.to_be_bytes()); + assert_eq!(&block[0x30..0x34], &record.machine_type.to_be_bytes()); + assert_eq!(&block[0x38..0x3c], &1u32.to_be_bytes(), "flash program count"); + assert_eq!(&block[0x34..0x38], &[0, 0, 0, 0x9b]); + assert_eq!(sum(block), sum(&image[0x60..0xb0])); + } + + /// The Tezro's block is the one the Tezro seed profile writes, byte for + /// byte, whatever block the payload carried. + #[test] + fn the_tezro_seed_writes_its_authored_block() { + const AUTHORED_BLOCK: &str = concat!( + "00000015589bad8569703237636f6e66", + "0000000029b92700000000000bebc200", + "00000000001312d00000000100000010", + "00000000000000da0000000100000010", + "00000006000000d20000000000000000", + ); + let expected: Vec = (0..AUTHORED_BLOCK.len()) + .step_by(2) + .map(|i| u8::from_str_radix(&AUTHORED_BLOCK[i..i + 2], 16).unwrap()) + .collect(); + for payload in [factory_block(), vec![0x5a; 0x200]] { + let bytes = Flash::new(&payload, FirmwareRecord::TEZRO).unwrap().contents(); + assert_eq!(&bytes[0x60..0xb0], expected.as_slice()); + assert_eq!(&bytes[..0x60], &payload[..0x60], "the payload around the block is untouched"); + assert_eq!(&bytes[0xb0..payload.len()], &payload[0xb0..]); + } + } + + /// The Tezro's record carries the QEMU Tezro profile's firmware + /// configuration, and the Config the CPU resets to carries the same mode + /// bits, so the PROM's check of one against the other passes. + #[test] + fn the_tezro_record_carries_its_700_mhz_mode_and_the_config_it_implies() { + let bytes = Flash::new(&factory_block(), FirmwareRecord::TEZRO).unwrap().contents(); + assert_eq!(&bytes[0x64..0x68], &0x589bad85u32.to_be_bytes()); + assert_eq!(&bytes[0x70..0x78], &700_000_000u64.to_be_bytes()); + assert_eq!(&bytes[0x90..0x94], &0u32.to_be_bytes(), "machine type 0"); + assert_eq!(FirmwareRecord::TEZRO.reset_config0(), 0x6c9b_ad85); + assert_eq!(FirmwareRecord::FUEL.reset_config0(), 0x6c9a_a985, "the Fuel's value is unchanged"); + // Config bits 18:16 hold the secondary cache size: 3 is 4 MiB here + // against the Fuel's 2 for 2 MiB. + assert_eq!((FirmwareRecord::TEZRO.reset_config0() >> 16) & 7, 3); + assert_eq!((FirmwareRecord::FUEL.reset_config0() >> 16) & 7, 2); + } + + /// The 800 MHz record differs from the 700 MHz one in the mode bits the + /// CPU models give the part and in its clock, and keeps the authored + /// block's byte sum. Each record's reset Config is its part's. + #[test] + fn the_tezro_800_mhz_record_takes_the_parts_config_and_keeps_the_byte_sum() { + use crate::cpu::mips_cache_shadow::{CONFIG0_RESET_MASK, R16000_MHZ700_CONFIG0}; + assert_eq!(FirmwareRecord::TEZRO.reset_mode, R16000_MHZ700_CONFIG0); + assert_eq!(FirmwareRecord::TEZRO_800.reset_mode, R16000_800_CONFIG0); + assert_eq!(FirmwareRecord::TEZRO_800.cpu_mode_word & CONFIG0_RESET_MASK, R16000_800_CONFIG0); + assert_eq!(FirmwareRecord::TEZRO_800.reset_config0(), 0x6c9b_af85); + let slow = Flash::new(&factory_block(), FirmwareRecord::TEZRO).unwrap().contents(); + let fast = Flash::new(&factory_block(), FirmwareRecord::TEZRO_800).unwrap().contents(); + assert_eq!(&fast[0x64..0x68], &0x549b_af85u32.to_be_bytes()); + assert_eq!(&fast[0x70..0x78], &800_000_000u64.to_be_bytes()); + assert_eq!(sum(&fast[0x60..0xb0]), sum(&slow[0x60..0xb0])); + let differ: Vec = (0x60..0xb0).filter(|&i| fast[i] != slow[i]).collect(); + assert_eq!(differ, [0x64, 0x66, 0x74, 0x75, 0x76, 0x97], "mode word, clock and check adjustment only"); + } + + /// Reopening a store reapplies a refreshed record, so a Fuel follows the + /// machine that opens it, and leaves an authored block as stored, so a + /// block the PROM rewrote on a Tezro survives the next start. + #[test] + fn reopening_refreshes_the_fuels_record_and_keeps_a_tezros_stored_block() { + let dir = std::env::temp_dir().join(format!("bedrock-flash-record-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let fuel_path = dir.join("fuel.bin"); + drop(Flash::with_file(&factory_block(), &fuel_path, FirmwareRecord::TEZRO).unwrap()); + let fuel = Flash::with_file(&[], &fuel_path, FirmwareRecord::FUEL).unwrap().contents(); + assert_eq!(&fuel[0x64..0x68], &0x549aa985u32.to_be_bytes()); + assert_eq!(&fuel[0x90..0x94], &2u32.to_be_bytes()); + + let tezro_path = dir.join("tezro.bin"); + let seeded = Flash::with_file(&factory_block(), &tezro_path, FirmwareRecord::TEZRO).unwrap().contents(); + assert_eq!(std::fs::read(&tezro_path).unwrap(), seeded, "the seed is what the store holds"); + let mut stored = seeded.clone(); + stored[0x64..0x68].copy_from_slice(&0x1234_5678u32.to_be_bytes()); + std::fs::write(&tezro_path, &stored).unwrap(); + let reopened = Flash::with_file(&factory_block(), &tezro_path, FirmwareRecord::TEZRO).unwrap().contents(); + assert_eq!(reopened, stored); + std::fs::remove_dir_all(&dir).unwrap(); + } + /// The PROM programs through one view while executable aliases must + /// keep reading array bytes. + #[test] + fn programming_changes_shared_bytes_without_exposing_status_to_instruction_fetches() { + { + let f = Arc::new(Flash::new(&[], FirmwareRecord::FUEL).unwrap()); + let local = f.boot_view(); + let remote = f.boot_view(); + let execute = f.array_view(0, PART_SIZE).unwrap(); + assert_eq!(local.write64(SECOND_POSITION, 0xf0), crate::traits::BUS_OK); + for part in [0, SECOND_POSITION] { + assert_eq!(local.write64(part + 8 * 0x55, 0x98), crate::traits::BUS_OK); + assert_eq!(remote.read16(part + 0x20).data, b'Q' as u16); + assert_eq!(remote.read16((SECOND_POSITION - part) + 0x20).data, 0xffff); + assert_eq!(execute.read16(0x20).data, 0xffff); + assert_eq!(remote.write64(part, 0xf0), crate::traits::BUS_OK); + assert_eq!(local.write64(part + 8 * 0x555, 0xaa), crate::traits::BUS_OK); + assert_eq!( + remote.write64(part + 8 * 0x2aa, 0x55), + crate::traits::BUS_OK + ); + assert_eq!(local.write64(part + 8 * 0x555, 0xa0), crate::traits::BUS_OK); + assert_eq!( + remote.write64(part + 4 * 0x200, 0x1234), + crate::traits::BUS_OK + ); + assert_eq!(local.read16(part + 0x200).data, 0x1234); + } + assert_eq!(execute.read16(0x200).data, 0x1234); + assert!(!local.read8(PART_SIZE as u64).is_ok()); + } + { + let f = Arc::new(Flash::new(&[], FirmwareRecord::FUEL).unwrap()); + let v = f.boot_view(); + program(&v, 0, 0x200, 0x1234); + program(&v, 0, 0x200, 0xf0ff); + assert_eq!(v.read16(0x200).data, 0x1034); + unlock(&v, 0); + assert_eq!(v.write64(8 * 0x555, 0xa0), crate::traits::BUS_OK); + let gen = f.generation(); + for (addr, value) in [ + (0x800, 0x10000), + (0x804, 0), + (64 << 20, 0), + ] { + assert_eq!(v.write64(addr, value), BUS_ERR); + } + assert_eq!(v.write16(0x800, 0), BUS_ERR); + assert_eq!(f.generation(), gen); + assert_eq!(v.write64(0x800, 0x00ff), crate::traits::BUS_OK); + assert_eq!(v.read16(0x200).data, 0x0034); + let execute = f.array_view(0, PART_SIZE).unwrap(); + unlock(&v, 0); + assert_eq!(execute.write64(8 * 0x555, 0xa0), BUS_ERR); + assert_eq!(v.write64(8 * 0x555, 0xa0), crate::traits::BUS_OK); + assert_eq!(v.write64(0x808, 0x5555), crate::traits::BUS_OK); + assert_eq!(execute.read16(0x202).data, 0x5555); + } + { + let f = Arc::new(Flash::new(&[], FirmwareRecord::FUEL).unwrap()); + let v = f.boot_view(); + assert_eq!(v.write64(0x550, 0x98), BUS_ERR); + assert_eq!(v.read16(0x20).data, 0xffff); + assert_eq!(v.write64(8 * 0x55, 0x98), BUS_OK); + let expected = [ + 0x51, 0x52, 0x59, 2, 0, 0, 0, 0, 0, 0, 0, 0x27, 0x36, 0, 0, 3, 0, 9, 0, 4, 0, 4, 0, 23, + 1, 0, 0, 0, 1, 0x7f, 0, 0, 1, + ]; + for (index, value) in expected.into_iter().enumerate() { + assert_eq!(v.read16(((index + 0x10) * 2) as u64).data, value); + } + assert_eq!(v.read16(0x100).data, 0); + assert!(!v.read8(0x20).is_ok()); + assert!(!v.read16(0x21).is_ok()); + f.reset(); + assert_eq!(v.read16(0x20).data, 0xffff); + } + } + /// The PROM polls erase completion and can reset the chip before + /// selected sectors change. + #[test] + fn erase_completes_only_for_selected_sectors_and_reset_cancels_it() { + let f = Arc::new(Flash::new(&[], FirmwareRecord::FUEL).unwrap()); + let v = f.boot_view(); + program(&v, 0, 0x200, 0); + program(&v, 0, 0x10000, 0); + erase_select(&v, 0, 0x200); + assert!(!v.read8(0x200).is_ok()); + let a = v.read16(0x200).data; + let b = v.read16(0x200).data; + assert_eq!((a ^ b) & 0x44, 0x44); + let a = v.read16(0x10000).data; + let b = v.read16(0x10000).data; + assert_eq!((a ^ b) & 0x44, 0x40); + f.advance_ns(40_000); + assert_eq!(v.write64(4 * 0x10000, 0x30), BUS_OK); + f.advance_ns(40_000); + assert_eq!(v.read16(0x10000).data & 8, 0); + assert_eq!(v.write64(SECOND_POSITION + 8 * 0x55, 0x98), BUS_OK); + assert_eq!(v.read16(SECOND_POSITION + 0x20).data, b'Q' as u16); + f.advance_ns(10_000); + assert_eq!(v.read16(0x200).data & 8, 8); + assert_eq!(v.write64(0, 0xf0), BUS_OK); + assert_eq!(v.read16(0x200).data & 0x80, 0); + f.reset(); + f.advance_ns(u64::MAX); + assert_eq!(v.read16(0x200).data, 0); + assert_eq!(v.read16(0x10000).data, 0); + assert_eq!(v.read16(SECOND_POSITION + 0x20).data, 0xffff); + erase_select(&v, 0, 0x200); + assert_eq!(v.write64(0, 0xf0), BUS_OK); + f.advance_ns(u64::MAX); + assert_eq!(v.read16(0x200).data, 0); + erase_select(&v, 0, 0x200); + assert_eq!(v.write64(4 * 0x10000, 0x30), BUS_OK); + f.advance_ns(50_000 + 512_000_000); + assert_eq!(v.read16(0x200).data, 0xffff); + assert_eq!(v.read16(0x10000).data, 0xffff); + unlock(&v, 0); + assert_eq!(v.write64(8 * 0x555, 0xa0), BUS_OK); + f.reset(); + assert_eq!(v.write64(4 * 0x200, 0x1234), BUS_ERR); + assert_eq!(v.read16(0x200).data, 0xffff); + program(&v, 0, 0x200, 0); + program(&v, 0, 0x10000, 0x3456); + erase_select(&v, 0, 0x200); + f.advance_ns(50_000 + 512_000_000); + assert_eq!(v.read16(0x200).data, 0xffff); + assert_eq!(v.read16(0x10000).data, 0x3456); + } + /// Firmware updates must persist across emulator restarts without + /// overwriting a supplied image. + #[test] + fn completed_programming_and_erases_survive_reopening_the_backing_file() { + { + let path = std::env::temp_dir().join(format!("iris-bedrock-store-{}", std::process::id())); + let mut bytes = vec![0x5a; FLASH_SIZE]; + bytes[PART_SIZE] = 0xc7; + std::fs::write(&path, &bytes).unwrap(); + let f = Flash::with_file(&[0x11], &path, FirmwareRecord::FUEL).unwrap(); + assert_eq!(f.contents()[0], 0x5a); + assert_eq!(f.contents()[PART_SIZE], 0xc7); + assert_eq!(std::fs::read(&path).unwrap(), f.contents()); + drop(f); + std::fs::write(&path, [1, 2, 3]).unwrap(); + assert!(Flash::with_file(&[], &path, FirmwareRecord::FUEL).is_err()); + assert_eq!(std::fs::read(&path).unwrap(), [1, 2, 3]); + std::fs::write(&path, []).unwrap(); + let g = Flash::with_file(&[0x34], &path, FirmwareRecord::FUEL).unwrap(); + assert_eq!(g.contents()[0], 0x34); + assert_eq!(std::fs::metadata(&path).unwrap().len(), FLASH_SIZE as u64); + drop(g); + std::fs::remove_file(path).unwrap(); + } + { + let path = std::env::temp_dir().join(format!("iris-amd-persist-{}", std::process::id())); + std::fs::write(&path, []).unwrap(); + let f = Arc::new(Flash::with_file(&[], &path, FirmwareRecord::FUEL).unwrap()); + let v = f.boot_view(); + program(&v, 0, 0x200, 0x1234); + program(&v, SECOND_POSITION, (PART_SIZE - 2) as u64, 0x5678); + drop(v); + drop(f); + let f = Arc::new(Flash::with_file(&[], &path, FirmwareRecord::FUEL).unwrap()); + let v = f.boot_view(); + assert_eq!(v.read16(0x200).data, 0x1234); + assert_eq!( + v.read16(SECOND_POSITION + PART_SIZE as u64 - 2).data, + 0x5678 + ); + unlock(&v, SECOND_POSITION); + assert_eq!(v.write64(SECOND_POSITION + 8 * 0x555, 0x80), BUS_OK); + unlock(&v, SECOND_POSITION); + assert_eq!( + v.write64(SECOND_POSITION + 4 * (PART_SIZE as u64 - 2), 0x30), + BUS_OK + ); + f.advance_ns(50_000 + 512_000_000); + drop(v); + drop(f); + let f = Arc::new(Flash::with_file(&[], &path, FirmwareRecord::FUEL).unwrap()); + let v = f.boot_view(); + assert_eq!(v.read16(0x200).data, 0x1234); + assert_eq!( + v.read16(SECOND_POSITION + PART_SIZE as u64 - 2).data, + 0xffff + ); + std::fs::remove_file(path).unwrap(); + } + } + + /// A host write failure must preserve flash contents and remain visible + /// to the guest. + #[test] + fn failed_backing_writes_leave_array_bytes_unchanged() { + { + let f = Arc::new(Flash::new(&[], FirmwareRecord::FUEL).unwrap()); + // A read-only descriptor makes writes fail without a special backend. + let path = std::env::temp_dir().join(format!("iris-amd-read-only-{}", std::process::id())); + std::fs::write(&path, []).unwrap(); + f.st.lock().backing = Some(FlashBacking::from_file( + std::fs::File::open(&path).unwrap(), + FLASH_SIZE, + )); + let v = f.boot_view(); + unlock(&v, 0); + assert_eq!(v.write64(8 * 0x555, 0xa0), BUS_OK); + assert_eq!(v.write64(4 * 0x200, 0x1234), BUS_ERR); + assert_eq!(v.read16(0x200).data, 0xffff); + std::fs::remove_file(path).unwrap(); + } + { + let f = Arc::new(Flash::new(&[], FirmwareRecord::FUEL).unwrap()); + let v = f.boot_view(); + program(&v, 0, 0x200, 0); + let path = + std::env::temp_dir().join(format!("iris-amd-erase-failure-{}", std::process::id())); + std::fs::write(&path, []).unwrap(); + f.st.lock().backing = Some(FlashBacking::from_file( + std::fs::File::open(&path).unwrap(), + FLASH_SIZE, + )); + erase_select(&v, 0, 0x200); + f.advance_ns(50_000 + 512_000_000); + assert_eq!(v.read16(0x200).data & 0x20, 0x20); + assert_eq!(f.contents()[0x200], 0); + assert_eq!(v.write64(0, 0xf0), BUS_OK); + assert_eq!(v.read16(0x200).data, 0); + std::fs::remove_file(path).unwrap(); + } + } + +} diff --git a/src/dev/bedrock_io.rs b/src/dev/bedrock_io.rs new file mode 100644 index 000000000..6f999cdfa --- /dev/null +++ b/src/dev/bedrock_io.rs @@ -0,0 +1,889 @@ +//! Bedrock IO routes programmed PIO windows to PCI widgets, applies inbound +//! permission checks to DMA requests and exposes two block-transfer engines. +//! Guests configure the widget masks before touching devices; the BTE moves +//! 128-byte lines and writes completion state through the configured target. +//! +//! # What is approximated here +//! +//! BTE transfers and their completion callbacks run synchronously when a +//! start is written. Timing and unpublished register behavior are not modeled. +//! +//! The plain register words sit in a sparse map keyed by offset, since the +//! 0x30010-byte block holds only a few dozen of them. +//! +//! Ported from hw/misc/sgi-bedrock-io.c in jamesbraid/qemu-sgi-origami. + +/// Length of the block-relative register aperture. +pub const SIZE: u64 = 0x30010; + +/// The public Linux Hub I/O protection-override key literal. +/// +/// Linux `arch/mips/include/asm/sn/sn0/hubio.h` at +/// `db2ddb87143519e20a95aa36c60b36107b736a58` defines +/// `IIO_PROTECT_OVRRD_KEY` as this value (lines 31, 47). This establishes +/// the literal's provenance only; it does not establish a register mapping. +pub const IIO_PROTECT_OVRRD_KEY: u64 = 0x53474972756c6573; + +pub mod reg { + pub const DEVICE_ID: u64 = 0x0; + pub const PROTECTION_KEY: u64 = 0x108; + pub const PIO_PERMISSION: u64 = 0x100; + pub const INBOUND_PERMISSION: u64 = 0x110; + pub const INBOUND_DEVICE_PERMISSION: u64 = 0x120; + pub const XIO_STATUS: u64 = 0x128; + pub const LINK_LOG: u64 = 0x130; + pub const LINK_STATUS: u64 = 0x138; + pub const BTE1_BASE: u64 = 0x10000; + pub const BTE_REGISTER_SIZE: u64 = 0x30; + pub const READ_ONLY: u64 = 0x8; + pub const BTE2_BASE: u64 = 0x20000; + pub const WIDGET_STATUS: u64 = 0x20; + pub const WIDGET_PERMISSION: u64 = 0x118; + pub const REGISTER_0150: u64 = 0x150; + pub const REGISTER_0158: u64 = 0x158; + pub const ITTE_BASE: u64 = 0x160; + pub const ITTE_LAST: u64 = 0x190; + pub const REGISTER_0198_BASE: u64 = 0x198; + pub const REGISTER_01D8_LAST: u64 = 0x1d8; + pub const READ_ONLY_ALIAS: u64 = 0x1f8; + pub const REGISTER_01E0: u64 = 0x1e0; + pub const REGISTER_01E8: u64 = 0x1e8; + pub const REGISTER_01F0: u64 = 0x1f0; + pub const REGISTER_0200: u64 = 0x200; + pub const REGISTER_03A8: u64 = 0x3a8; + pub const REGISTER_03B0: u64 = 0x3b0; + pub const REGISTER_03B8: u64 = 0x3b8; + pub const REGISTER_03C0: u64 = 0x3c0; + pub const RETAINED_WINDOW_BASE: u64 = 0x308; + pub const RETAINED_WINDOW_LAST: u64 = 0x348; + pub const RETAINED_CONFIG_BASE: u64 = 0x400; + pub const RETAINED_CONFIG_LAST: u64 = 0x5d8; + pub const GFX_MAP_BASE: u64 = 0x140; +} +pub mod bte_reg { + pub const STATUS: u64 = 0x0; + pub const SOURCE: u64 = 0x8; + pub const DESTINATION: u64 = 0x10; + pub const CONTROL: u64 = 0x18; + pub const NOTIFICATION: u64 = 0x20; + pub const INTERRUPT: u64 = 0x28; +} + +use crate::traits::{BusDevice, BusRead64, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::Mutex; +use std::collections::{BTreeMap, HashSet}; +use std::sync::{Arc, OnceLock}; +use std::thread::ThreadId; +const ID: u64 = (2 << 28) | (0xc110 << 12) | (0x36 << 1); +const ADDR_MASK: u64 = 0xffffffff80; +const BUSY: u64 = 1 << 20; +const ERROR: u64 = 1 << 16; +#[derive(Clone, Copy, Default)] +struct Bte { + status: u64, + source: u64, + destination: u64, + control: u64, + notification: u64, + interrupt: u64, +} +struct Regs { + words: BTreeMap, + bte: [Bte; 2], + node: u8, + peer: bool, +} +impl Default for Regs { + fn default() -> Self { + let mut words = BTreeMap::new(); + for (a, v) in [ + (0x20, 0xa), + (0x100, !0), + (0x110, !0), + (0x118, !0), + (0x120, !0), + (0x128, 0x3fc0fff6403 & !(1 << 13)), + (0x1e8, 0x111), + ] { + words.insert(a, v); + } + Self { + words, + bte: [Bte::default(); 2], + node: 0, + peer: false, + } + } +} +// Only a synchronous callback on the same thread is recursive. DMA workers +// on other threads may call the thread-safe downstream bus independently. +struct InboundAccess<'a> { + active: &'a Mutex>, + thread: ThreadId, +} +impl Drop for InboundAccess<'_> { + fn drop(&mut self) { + self.active.lock().remove(&self.thread); + } +} + +pub struct BedrockIo { + inbound_threads: Mutex>, + regs: Arc>, + widgets: Mutex<[Option>; 16]>, + memory: Mutex>>, + hub: Mutex>>, + gfx_map_written: OnceLock, + scratch0_written: OnceLock>, +} +/// Told of a store to graphics map entry `entry`: its old value, then the +/// new one. Runs before the new value takes effect, without the register +/// lock, so it may read the map as it stood. +pub type GfxMapHook = Box; +impl Default for BedrockIo { + fn default() -> Self { + Self::new() + } +} +impl BedrockIo { + pub fn new() -> Self { + let regs = Arc::new(Mutex::new(Regs::default())); + crate::locks::register_mutex("bedrock-io", ®s); + Self { + regs, + inbound_threads: Mutex::new(HashSet::new()), + widgets: Mutex::new(std::array::from_fn(|_| None)), + memory: Mutex::new(None), + hub: Mutex::new(None), + gfx_map_written: OnceLock::new(), + scratch0_written: OnceLock::new(), + } + } + /// Called after each store to scratch 0 (0x150) has landed, with no IO + /// lock held. Once. + pub fn on_scratch0_write(&self, hook: Box) { + assert!(self.scratch0_written.set(hook).is_ok(), "scratch 0 hook already set"); + } + /// Watch stores to the graphics map, as a board whose head owes credits + /// must (see [`crate::ip53_machine::OwedCredits`]). Once. + pub fn on_gfx_map_write(&self, hook: GfxMapHook) { + assert!(self.gfx_map_written.set(hook).is_ok(), "graphics map hook already set"); + } + pub fn connect_widget(&self, widget: u8, dev: Arc) { + assert!(widget < 16); + self.widgets.lock()[widget as usize] = Some(dev); + self.set_xio_peer(true); + } + pub fn set_xio_peer(&self, peer: bool) { + let mut r = self.regs.lock(); + r.peer = peer; + if peer { + *r.words.entry(reg::XIO_STATUS).or_default() |= 1 << 13; + } else { + *r.words.entry(reg::XIO_STATUS).or_default() &= !(1 << 13); + } + } + pub fn connect_memory(&self, dev: Arc) { + *self.memory.lock() = Some(dev); + } + pub fn connect_hub(&self, dev: Arc) { + *self.hub.lock() = Some(dev); + } + pub fn set_node_number(&self, node: u8) { + self.regs.lock().node = node; + } + pub fn gfx_map(&self, entry: usize) -> u64 { + assert!(entry < 2); + self.word(reg::GFX_MAP_BASE + entry as u64 * 8) + } + fn word(&self, a: u64) -> u64 { + self.regs.lock().words.get(&a).copied().unwrap_or(0) + } + pub fn disable_device(&self, widget: u8, device: u8) { + if (8..16).contains(&widget) && device < 8 { + *self.regs.lock().words.entry(reg::INBOUND_DEVICE_PERMISSION).or_default() &= + !(1 << ((widget - 8) * 8 + device)); + } + } + fn permitted(r: &Regs, widget: u8, device: u8) -> bool { + widget < 16 + && device < 8 + && r.words.get(®::WIDGET_PERMISSION).copied().unwrap_or(0) & (1 << widget) != 0 + && (widget < 8 + || r.words.get(®::INBOUND_DEVICE_PERMISSION).copied().unwrap_or(0) & (1 << ((widget - 8) * 8 + device)) + != 0) + } + fn retained(a: u64) -> bool { + matches!( + a, + 0x208 + | 0x210 + | 0x218 + | 0x220 + | 0x228 + | 0x300 + | reg::GFX_MAP_BASE + | 0x148 + | 0x388 + | 0x390 + | 0x398 + | 0x3a0 + | 0x600 + | 0x608 + | 0x610 + | 0x30000 + | 0x30008 + ) || (reg::RETAINED_WINDOW_BASE..reg::RETAINED_WINDOW_LAST).contains(&a) + || (reg::RETAINED_CONFIG_BASE..=reg::RETAINED_CONFIG_LAST).contains(&a) + } + fn mask(a: u64) -> Option { + Some(match a { + reg::WIDGET_STATUS => 0x7fe01f, + reg::PIO_PERMISSION | reg::INBOUND_PERMISSION | reg::WIDGET_PERMISSION | reg::INBOUND_DEVICE_PERMISSION | reg::REGISTER_0150 | reg::REGISTER_0158 => !0, + reg::XIO_STATUS => 0x3ff0fff7d3f, + reg::LINK_STATUS => 0x101ff7f, + reg::ITTE_BASE..=reg::ITTE_LAST => 0x1f1f, + reg::REGISTER_0198_BASE..=reg::REGISTER_01D8_LAST => !0, + reg::REGISTER_01E0 => 0x3ffffff, + reg::REGISTER_01E8 => 0x1111, + reg::REGISTER_01F0 => 0x0fffffff03ffffff, + reg::REGISTER_0200 => 0x10f, + reg::REGISTER_03A8 => 0x13ffffffff30f3, + reg::REGISTER_03B0 => 0x1ff0f, + reg::REGISTER_03B8 => 0xff, + reg::REGISTER_03C0 => 0xffffff, + _ if Self::retained(a) => !0, + _ => return None, + }) + } + fn bte_at(a: u64) -> Option<(usize, u64)> { + if (reg::BTE1_BASE..reg::BTE1_BASE + reg::BTE_REGISTER_SIZE).contains(&a) { + Some((0, a - reg::BTE1_BASE)) + } else if (reg::BTE2_BASE..reg::BTE2_BASE + reg::BTE_REGISTER_SIZE).contains(&a) { + Some((1, a - reg::BTE2_BASE)) + } else { + None + } + } + fn target(&self, offset: u64) -> Option<(u8, u64)> { + if offset < 0x10000000 { + Some(((offset >> 24) as u8, offset & 0xffffff)) + } else if offset >= 0x20000000 { + let entry = (offset >> 29) - 1; + if entry >= 7 { + return None; + } + let itte = self.word(reg::ITTE_BASE + entry * 8); + Some(( + ((itte >> 8) & 15) as u8, + ((itte & 31) << 29) + | (offset & 0x1fffffff) + | if itte & (1 << 12) != 0 { 1 << 47 } else { 0 }, + )) + } else { + None + } + } + pub fn pio_read(&self, offset: u64, size: u8) -> BusRead64 { + self.pio(offset, size, None) + } + pub fn pio_write(&self, offset: u64, size: u8, value: u64) -> u32 { + self.pio(offset, size, Some(value)).status + } + fn pio(&self, offset: u64, size: u8, value: Option) -> BusRead64 { + let Some((widget, target)) = self.target(offset) else { + return BusRead64::err(); + }; + { + let r = self.regs.lock(); + if r.words.get(&0x100).copied().unwrap_or(0) & 1 == 0 + || r.words.get(&0x110).copied().unwrap_or(0) & (1 << widget) == 0 + { + return BusRead64::err(); + } + } + if !matches!(size, 1 | 2 | 4 | 8) { + return BusRead64::err(); + } + if widget == 1 && target == 0 { + return BusRead64::ok(ID); + } + let dev = self.widgets.lock()[widget as usize].clone(); + Self::access(dev, target, size, value) + } + /// Device None means every possible DMA master must be permitted. + pub fn inbound_read( + &self, + widget: u8, + device: Option, + address: u64, + size: u8, + ) -> BusRead64 { + self.inbound(widget, device, address, size, None) + } + pub fn inbound_write( + &self, + widget: u8, + device: Option, + address: u64, + size: u8, + value: u64, + ) -> u32 { + self.inbound(widget, device, address, size, Some(value)) + .status + } + fn inbound( + &self, + widget: u8, + device: Option, + address: u64, + size: u8, + value: Option, + ) -> BusRead64 { + if address >= (1 << 48) || size as u64 > (1 << 48) - address { + return BusRead64::err(); + } + { + let r = self.regs.lock(); + let allowed = if let Some(device) = device { + Self::permitted(&r, widget, device) + } else { + (0..8).all(|d| Self::permitted(&r, widget, d)) + }; + if !allowed { + crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock IO inbound access refused from widget {widget} device {device:?}" + ); + return BusRead64::err(); + } + } + let thread = std::thread::current().id(); + if !self.inbound_threads.lock().insert(thread) { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO reentrant inbound packet refused"); + return BusRead64::err(); + } + let _access = InboundAccess { + active: &self.inbound_threads, + thread, + }; + if address & (1 << 47) != 0 { + let node = ((address >> 39) & 255) as u8; + let window = address & ((1 << 39) - 1); + if node != self.regs.lock().node || window & !0x7fffff != 0x1800000 { + return BusRead64::err(); + } + let dev = self.hub.lock().clone(); + Self::access(dev, window & 0x7fffff, size, value) + } else { + let dev = self.memory.lock().clone(); + Self::access(dev, address, size, value) + } + } + fn access(dev: Option>, a: u64, size: u8, value: Option) -> BusRead64 { + let Some(dev) = dev else { + return BusRead64::err(); + }; + if let Some(v) = value { + let status = match size { + 1 => dev.write8(a, v as u8), + 2 => dev.write16(a, v as u16), + 4 => dev.write32(a, v as u32), + 8 => dev.write64(a, v), + _ => BUS_ERR, + }; + BusRead64 { status, data: 0 } + } else { + match size { + 1 => { + let r = dev.read8(a); + BusRead64 { + status: r.status, + data: r.data as u64, + } + } + 2 => { + let r = dev.read16(a); + BusRead64 { + status: r.status, + data: r.data as u64, + } + } + 4 => { + let r = dev.read32(a); + BusRead64 { + status: r.status, + data: r.data as u64, + } + } + 8 => dev.read64(a), + _ => BusRead64::err(), + } + } + } + fn run_bte(&self, n: usize, mut b: Bte) { + let memory = self.memory.lock().clone(); + let count = b.status & 0xffff; + let bytes = count * 128; + let overflow = (b.control & 1 == 0 && b.source > (1 << 40) - bytes) + || b.destination > (1 << 40) - bytes; + let mut remaining = count; + if overflow { + b.status = remaining | ERROR; + } else { + while remaining != 0 { + let mut line = [0u64; 16]; + let mut status = BUS_OK; + if b.control & 1 == 0 { + for (i, v) in line.iter_mut().enumerate() { + let r = Self::access(memory.clone(), b.source + i as u64 * 8, 8, None); + status = r.status; + if status != BUS_OK { + break; + } + *v = r.data; + } + } + if status == BUS_OK { + for (i, v) in line.into_iter().enumerate() { + status = + Self::access(memory.clone(), b.destination + i as u64 * 8, 8, Some(v)) + .status; + if status != BUS_OK { + break; + } + } + } + if status != BUS_OK { + break; + } + remaining -= 1; + b.source = (b.source + 128) & ADDR_MASK; + b.destination = (b.destination + 128) & ADDR_MASK; + } + b.status = if remaining == 0 { 0 } else { remaining | ERROR }; + } + self.regs.lock().bte[n] = b; + if b.control & 0x10 != 0 { + if Self::access(memory, b.notification, 8, Some(b.status)).status != BUS_OK { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO BTE notification write failed"); + } + } else { + let node = ((b.interrupt >> 1) & 255) as u8; + if node != self.regs.lock().node { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO BTE interrupt reaches no node"); + return; + } + let hub = self.hub.lock().clone(); + let offset = if b.interrupt & 1 != 0 { 0x200090 } else { 0x90 }; + if Self::access(hub, offset, 8, Some(0x100 | ((b.interrupt >> 16) & 127))).status + != BUS_OK + { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO BTE completion interrupt failed"); + } + } + } +} +impl Resettable for BedrockIo { + fn power_on(&self) { + let mut r = self.regs.lock(); + let peer = r.peer; + let node = r.node; + *r = Regs::default(); + r.node = node; + r.peer = peer; + if peer { + *r.words.entry(reg::XIO_STATUS).or_default() |= 1 << 13; + } + } +} +impl BusDevice for BedrockIo { + fn read64(&self, a: u64) -> BusRead64 { + if a & 7 != 0 { + return BusRead64::err(); + } + let mut r = self.regs.lock(); + if let Some((n, a)) = Self::bte_at(a) { + let b = &mut r.bte[n]; + return BusRead64::ok(match a { + bte_reg::STATUS => b.status, + bte_reg::SOURCE => b.source, + bte_reg::DESTINATION => b.destination, + bte_reg::CONTROL => { + b.status &= !BUSY; + b.control + } + bte_reg::NOTIFICATION => b.notification, + bte_reg::INTERRUPT => b.interrupt, + _ => 0, + }); + } + let v = match a { + reg::DEVICE_ID => ID, + reg::READ_ONLY | 0x1f8 => 0, + reg::LINK_STATUS => r.words.get(&a).copied().unwrap_or(0) & !(1 << 28), + reg::LINK_LOG => r.words.get(&a).copied().unwrap_or(0), + _ if Self::mask(a).is_some() => r.words.get(&a).copied().unwrap_or(0), + _ => { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO unsupported read {a:#x}"); + 0 + } + }; + BusRead64::ok(v) + } + fn write64(&self, a: u64, v: u64) -> u32 { + if a & 7 != 0 { + return BUS_ERR; + } + if let (Some(hook), true) = (self.gfx_map_written.get(), a == reg::GFX_MAP_BASE || a == reg::GFX_MAP_BASE + 8) { + let entry = ((a - reg::GFX_MAP_BASE) / 8) as usize; + hook(entry, self.gfx_map(entry), v); + } + // Declared before the lock, so it runs once the store has landed and + // the lock is dropped. + let _published = (a == reg::REGISTER_0150).then(|| self.scratch0_written.get()).flatten().map(|hook| ScratchNotice(hook.as_ref())); + let mut r = self.regs.lock(); + if let Some((n, offset)) = Self::bte_at(a) { + let b = &mut r.bte[n]; + let mut run = false; + match offset { + bte_reg::STATUS => b.status = v & (0xffff | BUSY), + bte_reg::SOURCE => b.source = v & ADDR_MASK, + bte_reg::DESTINATION => b.destination = v & ADDR_MASK, + bte_reg::CONTROL => { + b.control = v & 0x111; + if b.control & 0x100 != 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO BTE poison unsupported"); + } else if b.status & BUSY != 0 { + if b.status & 0xffff == 0 { + b.status = 0; + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO zero-count BTE start"); + } else { + run = true; + } + } + } + bte_reg::NOTIFICATION => b.notification = v & ADDR_MASK, + bte_reg::INTERRUPT => b.interrupt = v & 0x7f01ff, + _ => {} + } + let b = *b; + drop(r); + if run { + self.run_bte(n, b); + } + return BUS_OK; + } + match a { + reg::PROTECTION_KEY => { + if v == IIO_PROTECT_OVRRD_KEY { + *r.words.entry(0x100).or_default() |= 1; + } + } + reg::INBOUND_DEVICE_PERMISSION | reg::REGISTER_01E8 => { + *r.words.entry(a).or_default() |= v & Self::mask(a).unwrap(); + } + reg::XIO_STATUS => { + let v = v & Self::mask(a).unwrap() & if r.peer { !0 } else { !(1 << 13) }; + r.words.insert(a, v); + } + reg::LINK_LOG => { + if v == 0 { + r.words.insert(a, 0); + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO nonzero link-log write ignored"); + } + } + reg::READ_ONLY_ALIAS | reg::DEVICE_ID | reg::READ_ONLY => { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO read-only or unsupported clear write {a:#x}"); + } + _ => { + if let Some(mask) = Self::mask(a) { + r.words.insert(a, v & mask); + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock IO unsupported write {a:#x} = {v:#x}"); + } + } + } + BUS_OK + } +} + +/// Where a widget's graphics credits go, as a graphics map entry names it. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct GfxRoute { + pub node: u8, + pub interface: usize, + pub slot: usize, +} + +/// The graphics map's route for `widget`: widget in bits 3:0, interface in +/// bit 4, node in bits 12:5 and slot in bit 16 (Linux sn1/hubio_next.h, +/// IIO_IGFX_INIT). There is no valid bit; the first entry naming the widget +/// wins. +pub fn gfx_route(map: [u64; 2], widget: u8) -> Option { + map.into_iter().find(|entry| entry & 0xf == u64::from(widget)).map(|entry| GfxRoute { + node: ((entry >> 5) & 0xff) as u8, + interface: ((entry >> 4) & 1) as usize, + slot: ((entry >> 16) & 1) as usize, + }) +} + +/// Calls a scratch 0 hook when dropped. +struct ScratchNotice<'a>(&'a (dyn Fn() + Send + Sync)); +impl Drop for ScratchNotice<'_> { + fn drop(&mut self) { + (self.0)() + } +} + +#[cfg(test)] +mod tests { + // Cases follow tests/qtest/sn1-bedrock-io-test.c in + // jamesbraid/qemu-sgi-origami: inbound-protection, bte and + // bte-error-termination. + use super::*; + /// A node watching scratch 0 sees each store to it once it has landed, + /// and no other store. + #[test] + fn a_scratch_0_store_is_reported_after_it_lands() { + let io = Arc::new(BedrockIo::new()); + let seen = Arc::new(Mutex::new(Vec::new())); + let (watch, record) = (Arc::downgrade(&io), seen.clone()); + io.on_scratch0_write(Box::new(move || { + let value = watch.upgrade().unwrap().read64(reg::REGISTER_0150).data; + record.lock().push(value); + })); + io.write64(reg::REGISTER_0150, 0x30); + io.write64(reg::REGISTER_0158, 0x40); + io.write64(reg::REGISTER_0150, 0xf0); + assert_eq!(*seen.lock(), [0x30, 0xf0]); + } + #[derive(Default)] + struct Memory { + words: Mutex>, + fail_read: Mutex>, + fail_write: Mutex>, + } + impl BusDevice for Memory { + fn read64(&self, a: u64) -> BusRead64 { + if *self.fail_read.lock() == Some(a) { + return BusRead64::err(); + } + BusRead64::ok(self.words.lock().get(&a).copied().unwrap_or(0)) + } + fn write64(&self, a: u64, v: u64) -> u32 { + if *self.fail_write.lock() == Some(a) { + return BUS_ERR; + } + self.words.lock().insert(a, v); + BUS_OK + } + } + fn launch(d: &BedrockIo, base: u64, count: u64, src: u64, dst: u64, notify: u64, control: u64) { + d.write64(base, count | BUSY); + d.write64(base + 8, src); + d.write64(base + 0x10, dst); + d.write64(base + 0x20, notify); + d.write64(base + 0x18, control); + } + struct Reentrant(std::sync::Weak); + impl BusDevice for Reentrant { + fn read64(&self, a: u64) -> BusRead64 { + self.0.upgrade().unwrap().inbound_read(15, Some(0), a, 8) + } + } + struct Busy; + impl BusDevice for Busy { + fn read64(&self, _: u64) -> BusRead64 { + BusRead64::busy() + } + fn write64(&self, _: u64, _: u64) -> u32 { + crate::traits::BUS_BUSY + } + } + struct BlockingMemory { + entered: std::sync::mpsc::Sender<()>, + release: Mutex>, + words: Mutex>, + } + impl BusDevice for BlockingMemory { + fn write64(&self, address: u64, value: u64) -> u32 { + if address == 0x1000 { + self.entered.send(()).unwrap(); + self.release + .lock() + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + } + self.words.lock().insert(address, value); + BUS_OK + } + } + + /// The guest must enable widget and requester masks before PIO or inbound DMA succeeds. + #[test] + fn the_io_block_checks_pio_and_inbound_requester_permissions() { + { + let d = BedrockIo::new(); + let m = Arc::new(Memory::default()); + d.connect_widget(8, m.clone()); + d.write64(reg::ITTE_BASE, 0x802); + assert_eq!(d.pio_write(0x20000080, 8, 0xabcd), BUS_OK); + assert_eq!(m.read64(0x40000080).data, 0xabcd); + assert_eq!(d.pio_read(0x20000080, 8).data, 0xabcd); + d.write64(reg::INBOUND_PERMISSION, 0); + assert_eq!(d.pio_read(0x20000080, 8).status, BUS_ERR); + d.write64(reg::INBOUND_PERMISSION, !0); + d.write64(reg::PIO_PERMISSION, 0); + assert_eq!(d.pio_read(0x20000080, 8).status, BUS_ERR); + d.write64(reg::PROTECTION_KEY, IIO_PROTECT_OVRRD_KEY); + assert_eq!(d.pio_read(0x20000080, 8).status, BUS_OK); + assert_eq!(d.pio_read(0x10000000, 8).status, BUS_ERR); + d.power_on(); + assert_eq!(d.read64(reg::PIO_PERMISSION).data, !0); + assert_eq!(d.read64(reg::REGISTER_01E8).data, 0x111); + + } + { + let d = BedrockIo::new(); + let m = Arc::new(Memory::default()); + let h = Arc::new(Memory::default()); + d.connect_memory(m.clone()); + d.connect_hub(h.clone()); + assert_eq!(d.inbound_write(14, Some(2), 0x80, 8, 42), BUS_OK); + assert_eq!(m.read64(0x80).data, 42); + d.disable_device(14, 2); + assert_eq!(d.inbound_read(14, Some(2), 0x80, 8).status, BUS_ERR); + assert_eq!(d.inbound_read(14, None, 0x80, 8).status, BUS_ERR); + assert_eq!(d.inbound_read(14, Some(1), 0x80, 8).status, BUS_OK); + let a = (1 << 47) | 0x1800090; + assert_eq!(d.inbound_write(15, Some(0), a, 8, 0x123), BUS_OK); + assert_eq!(h.read64(0x90).data, 0x123); + assert_eq!( + d.inbound_read(15, Some(0), a | (1 << 39), 8).status, + BUS_ERR + ); + assert_eq!(d.inbound_read(15, Some(0), 1 << 48, 8).status, BUS_ERR); + assert_eq!(d.inbound_read(15, Some(0), a + 0x800000, 8).status, BUS_ERR); + } + } + /// The guest observes BTE line copies and zero-fill through its completion registers. + #[test] + fn the_bte_copies_and_zeroes_lines_before_completion() { + let d = BedrockIo::new(); + let m = Arc::new(Memory::default()); + let h = Arc::new(Memory::default()); + d.connect_memory(m.clone()); + d.connect_hub(h.clone()); + for i in 0..32 { + m.write64(0x1000 + i * 8, 0xfeed0000 | i); + } + launch(&d, reg::BTE1_BASE, 2, 0x1000, 0x2000, 0x3000, 0x10); + for i in 0..32 { + assert_eq!(m.read64(0x2000 + i * 8).data, 0xfeed0000 | i); + } + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 0); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::SOURCE).data, 0x1100); + assert_eq!(m.read64(0x3000).data, 0); + d.write64(reg::BTE2_BASE + bte_reg::INTERRUPT, 53 << 16); + launch(&d, reg::BTE2_BASE, 1, 0, 0x2000, 0, 1); + for i in 0..16 { + assert_eq!(m.read64(0x2000 + i * 8).data, 0); + } + assert_eq!(h.read64(0x90).data, 0x135); + } + /// A failed BTE line stops the transfer and reports the failing source and destination. + #[test] + fn the_bte_reports_transfer_errors_and_stops_after_a_failed_line() { + { + for (read, fail) in [(true, 0x1080), (false, 0x2080)] { + let d = BedrockIo::new(); + let m = Arc::new(Memory::default()); + d.connect_memory(m.clone()); + if read { + *m.fail_read.lock() = Some(fail); + } else { + *m.fail_write.lock() = Some(fail); + } + launch(&d, reg::BTE1_BASE, 3, 0x1000, 0x2000, 0x3000, 0x10); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 2 | ERROR); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::SOURCE).data, 0x1080); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::DESTINATION).data, 0x2080); + assert_eq!(m.read64(0x3000).data, 2 | ERROR); + } + let d = BedrockIo::new(); + let m = Arc::new(Memory::default()); + d.connect_memory(m.clone()); + *m.fail_write.lock() = Some(0x3000); + launch(&d, reg::BTE1_BASE, 1, 0x1000, 0x2000, 0x3000, 0x10); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 0); + launch(&d, reg::BTE1_BASE, 2, 0xffffffff80, 0x2000, 0x3000, 0x10); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 2 | ERROR); + } + { + let d = BedrockIo::new(); + launch(&d, reg::BTE1_BASE, 1, 0, 0, 0, 0x100); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 1 | BUSY); + d.read64(reg::BTE1_BASE + bte_reg::CONTROL); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 1); + launch(&d, reg::BTE1_BASE, 0, 0, 0, 0, 0x10); + assert_eq!(d.read64(reg::BTE1_BASE + bte_reg::STATUS).data, 0); + } + } + /// A downstream callback may reenter the IO block while an inbound transaction is active. + #[test] + fn reentrant_inbound_forwards_are_refused_and_can_be_retried() { + let d = Arc::new(BedrockIo::new()); + d.connect_memory(Arc::new(Reentrant(Arc::downgrade(&d)))); + assert_eq!(d.inbound_read(15, Some(0), 0, 8).status, BUS_ERR); + d.connect_memory(Arc::new(Busy)); + assert_eq!( + d.inbound_read(15, Some(0), 0, 8).status, + crate::traits::BUS_BUSY + ); + d.connect_widget(8, Arc::new(Busy)); + assert_eq!(d.pio_read(0x8000000, 8).status, crate::traits::BUS_BUSY); + assert_eq!(d.pio_write(0x8000000, 8, 42), crate::traits::BUS_BUSY); + + } + /// The two guest BTEs can reach node memory without one worker holding the other off. + #[test] + fn independent_bte_workers_can_reach_memory_concurrently() { + let d = Arc::new(BedrockIo::new()); + let (entered_tx, entered_rx) = std::sync::mpsc::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel(); + let memory = Arc::new(BlockingMemory { + entered: entered_tx, + release: Mutex::new(release_rx), + words: Mutex::new(BTreeMap::new()), + }); + d.connect_memory(memory.clone()); + let first_io = d.clone(); + let first = + std::thread::spawn(move || first_io.inbound_write(14, Some(0), 0x1000, 8, 0x1234)); + entered_rx + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + let second_io = d.clone(); + let second = + std::thread::spawn(move || second_io.inbound_write(15, Some(1), 0x2000, 8, 0x5678)); + let second_status = second.join().unwrap(); + release_tx.send(()).unwrap(); + let first_status = first.join().unwrap(); + assert_eq!(second_status, BUS_OK); + assert_eq!(first_status, BUS_OK); + assert_eq!( + *memory.words.lock(), + BTreeMap::from([(0x1000, 0x1234), (0x2000, 0x5678)]) + ); + } + + #[test] + fn the_graphics_map_names_node_interface_and_slot_and_the_first_match_wins() { + let entry = |node: u64, pi: u64, slot: u64, widget: u64| (slot << 16) | (node << 5) | (pi << 4) | widget; + let map = [entry(3, 1, 1, 0xc), entry(0, 0, 0, 0xc)]; + assert_eq!(gfx_route(map, 0xc), Some(GfxRoute { node: 3, interface: 1, slot: 1 })); + assert_eq!(gfx_route(map, 0xb), None); + assert_eq!(gfx_route([0, 0], 0), Some(GfxRoute { node: 0, interface: 0, slot: 0 }), "no valid bit"); + } +} diff --git a/src/dev/bedrock_lb.rs b/src/dev/bedrock_lb.rs new file mode 100644 index 000000000..a91b83605 --- /dev/null +++ b/src/dev/bedrock_lb.rs @@ -0,0 +1,591 @@ +//! Bedrock LB provides the node and region registers, atomic scratch aliases, +//! error and vector status, and the MicroLAN master used by board identity +//! parts. The guest polls the MicroLAN completion bit after issuing a slot. +//! +//! # What is approximated here +//! +//! The slot is decoded immediately and completes after one microsecond of +//! machine time. Permission registers retain guest writes, but this block's +//! access interface does not carry requester identity, so it cannot enforce +//! those permissions. +//! +//! The plain register words sit in a sparse map keyed by offset, since the +//! 0xc040-byte block holds only a few dozen of them. +//! +//! Ported from hw/misc/sgi-bedrock-lb.c in jamesbraid/qemu-sgi-origami. + +/// Length of the block-relative register aperture. +pub const SIZE: u64 = 0xc040; + +pub mod reg { + pub const NODE_REGION: u64 = 0x0; + pub const PIO_PERMISSION: u64 = 0x4000; + pub const PIO_PERMISSION_KEY: u64 = 0x4008; + pub const INBOUND_PERMISSION: u64 = 0x4010; + pub const SOFT_RESET: u64 = 0x4018; + pub const MICRO_LAN: u64 = 0x4030; + pub const ERROR_STATUS: u64 = 0x4040; + pub const ERROR_CLEAR: u64 = 0x4048; + pub const PIO_REGION_MASK: u64 = 0x4028; + pub const NODE_REGION_CONFIG: u64 = 0x4050; + pub const NODE_REGION_MASK: u64 = 0x4058; + pub const PIO_CONFIG: u64 = 0x4100; + pub const INBOUND_CONFIG: u64 = 0x4200; + pub const INBOUND_MASK: u64 = 0x4208; + pub const VECTOR_REQUEST: u64 = 0xc000; + pub const VECTOR_STATUS: u64 = 0xc020; + pub const VECTOR_STATUS_ALIAS: u64 = 0xc038; + pub const PIO_WINDOW: u64 = 0x8000; + pub const PIO_WINDOW_END: u64 = 0x8020; + pub const ATOMIC_SCRATCH: u64 = 0x8040; + pub const ATOMIC_SCRATCH_END: u64 = 0x8050; + pub const ATOMIC_READ0: u64 = 0x8058; + pub const ATOMIC_READ1: u64 = 0x8060; + pub const REGISTER_4020: u64 = 0x4020; + pub const REGISTER_4060: u64 = 0x4060; + pub const REGISTER_C008: u64 = 0xc008; + pub const REGISTER_C010: u64 = 0xc010; + pub const REGISTER_C028: u64 = 0xc028; + pub const REGISTER_C030: u64 = 0xc030; +} + +use crate::dev::bedrock_io::IIO_PROTECT_OVRRD_KEY; +use crate::dev::w1; +use crate::traits::{BusDevice, BusRead64, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::{Mutex, ReentrantMutex}; +use std::collections::BTreeMap; +use std::sync::Arc; +pub type Clock = Arc u64 + Send + Sync>; +type Level = Arc; +const ID: u64 = (0x36 << 1) | (0x3012 << 12) | (2 << 28); +/// Scratch 1 and 3 in the plain scratch window. +const SCRATCH1: u64 = reg::PIO_WINDOW + 8; +const SCRATCH3: u64 = reg::PIO_WINDOW + 24; +/// Runs its function when dropped. +struct OnDrop(F); +impl Drop for OnDrop { + fn drop(&mut self) { + (self.0)() + } +} +const VECTOR_MASK: u64 = 3 | (0x1fffff << 3) | (0xff << 32) | (0x7ff << 40); +struct Regs { + words: BTreeMap, + scratch: [u64; 5], + node: u8, + region: u8, + deadline: Option, + sample: bool, +} +impl Default for Regs { + fn default() -> Self { + let mut words = BTreeMap::new(); + words.insert(0x4000, !0); + words.insert(0x4010, !0); + Self { + words, + scratch: [0; 5], + node: 0, + region: 0, + deadline: None, + sample: true, + } + } +} +/// What a node does around the processors' scratch registers, called with +/// no LB lock held. With hooks connected, a store to scratch 1 is the +/// hooks' to make, through [`BedrockLb::set_scratch1`]. +pub trait ScratchHooks: Send + Sync { + /// A store of `value` to scratch 1. + fn scratch1_write(&self, value: u64); + /// A read of scratch 1, before it is answered. + fn scratch1_read(&self); + /// A store left scratch 3 clear: the node's hub lock was released. + fn scratch3_released(&self); +} +pub struct BedrockLb { + regs: Arc>, + scratch_hooks: std::sync::OnceLock>, + clock: Clock, + error: Mutex>, + delivery: ReentrantMutex<()>, + driven: Mutex>, + reset: Mutex>>, + microlan: Mutex>, + node_changed: Mutex>>, +} +impl BedrockLb { + pub fn new(clock: Clock) -> Self { + let regs = Arc::new(Mutex::new(Regs::default())); + crate::locks::register_mutex("bedrock-lb", ®s); + Self { + regs, + scratch_hooks: std::sync::OnceLock::new(), + clock, + error: Mutex::new(None), + delivery: ReentrantMutex::new(()), + driven: Mutex::new(None), + reset: Mutex::new(None), + microlan: Mutex::new(None), + node_changed: Mutex::new(None), + } + } + pub fn connect_error(&self, cb: Level) { + let _delivery = self.delivery.lock(); + *self.error.lock() = Some(cb); + *self.driven.lock() = None; + self.update_irq(); + } + /// Connect the node's scratch hooks, once. + pub fn connect_scratch_hooks(&self, hooks: Arc) { + assert!(self.scratch_hooks.set(hooks).is_ok(), "LB scratch hooks connected twice"); + } + /// Store scratch 1, for the hooks that took a store to it. + pub fn set_scratch1(&self, value: u64) { + self.regs.lock().scratch[1] = value; + } + /// Attach the node's 1-wire bus to the local-block MicroLAN master. + pub fn set_microlan(&self, bus: w1::Bus) { + *self.microlan.lock() = Some(bus); + } + pub fn connect_soft_reset(&self, cb: Arc) { + *self.reset.lock() = Some(cb); + } + pub fn connect_node_changed(&self, cb: Arc) { + let _delivery = self.delivery.lock(); + *self.node_changed.lock() = Some(cb); + } + pub fn node_number(&self) -> u8 { + self.regs.lock().node + } + pub fn error_level(&self) -> bool { + self.regs.lock().words.get(&0x4040).copied().unwrap_or(0) != 0 + } + fn update_irq(&self) { + // Serialize the sample and callback, while leaving registers available + // to the callback. Suppress unchanged levels so callbacks can reenter + // register reads that recompute the output. + let _delivery = self.delivery.lock(); + let cb = self.error.lock().clone(); + if let Some(cb) = cb { + let level = self.error_level(); + let changed = { + let mut driven = self.driven.lock(); + let changed = *driven != Some(level); + *driven = Some(level); + changed + }; + if changed { + cb(level); + } + } + } + fn notify_node(&self) { + let _delivery = self.delivery.lock(); + let cb = self.node_changed.lock().clone(); + if let Some(cb) = cb { + cb(self.node_number()); + } + } + fn complete(r: &mut Regs, now: u64) { + if r.deadline.is_some_and(|d| now >= d) { + *r.words.entry(0x4030).or_default() |= 2 | u64::from(r.sample); + r.deadline = None; + } + } + fn mask(a: u64) -> Option { + Some(match a { + reg::PIO_PERMISSION | reg::INBOUND_PERMISSION | reg::REGISTER_4020 | reg::REGISTER_4060 | reg::REGISTER_C008 | reg::REGISTER_C010 | reg::REGISTER_C028 | reg::REGISTER_C030 => !0, + reg::PIO_REGION_MASK => 0x80ff80ff80ff80ff, + reg::ERROR_STATUS => 0x7ff, + reg::NODE_REGION_CONFIG => 0x7f | (0x7ff << 8) | (0x7ff << 20) | (0xf << 32) | (1 << 40) | (1 << 44), + reg::NODE_REGION_MASK => (1 << 38) - 1, + reg::PIO_CONFIG => 0x3f, + reg::INBOUND_CONFIG => 1 | (0x3ff << 4) | (0x3ff << 16) | (1 << 28) | (1 << 32), + reg::INBOUND_MASK => { + 0x1f | (0xfff << 8) | (1 << 20) | (0x3ff << 24) | (0x3ff << 36) | (0x3ff << 48) + } + reg::VECTOR_REQUEST => VECTOR_MASK, + _ => return None, + }) + } +} +impl Resettable for BedrockLb { + fn power_on(&self) { + let mut r = self.regs.lock(); + let survivors = [ + r.scratch[1], + r.scratch[2], + r.words.get(®::ERROR_STATUS).copied().unwrap_or(0), + ]; + *r = Regs::default(); + r.scratch[1] = survivors[0]; + r.scratch[2] = survivors[1]; + r.words.insert(reg::ERROR_STATUS, survivors[2]); + drop(r); + self.update_irq(); + self.notify_node(); + } +} +impl BusDevice for BedrockLb { + fn read64(&self, a: u64) -> BusRead64 { + if a & 7 != 0 { + return BusRead64::err(); + } + if a == SCRATCH1 { + if let Some(hooks) = self.scratch_hooks.get() { + hooks.scratch1_read(); + } + } + let now = (self.clock)(); + let mut r = self.regs.lock(); + Self::complete(&mut r, now); + let v = match a { + reg::NODE_REGION => ID | ((r.node as u64) << 32) | ((r.region as u64) << 46), + reg::PIO_WINDOW..=reg::PIO_WINDOW_END => r.scratch[(a as usize - reg::PIO_WINDOW as usize) / 8], + reg::ATOMIC_READ0 | reg::ATOMIC_READ1 => { + let n = 3 + (a as usize - reg::ATOMIC_READ0 as usize) / 8; + let v = r.scratch[n]; + r.scratch[n] = 1; + v + } + reg::VECTOR_STATUS_ALIAS => r.words.remove(&0xc020).unwrap_or(0), + reg::VECTOR_STATUS | reg::MICRO_LAN => r.words.get(&a).copied().unwrap_or(0), + _ if Self::mask(a).is_some() => r.words.get(&a).copied().unwrap_or(0), + _ => { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB unsupported read {a:#x}"); + 0 + } + }; + BusRead64::ok(v) + } + fn write64(&self, a: u64, v: u64) -> u32 { + if a & 7 != 0 { + return BUS_ERR; + } + let hooks = self.scratch_hooks.get(); + if let (SCRATCH1, Some(hooks)) = (a, hooks) { + hooks.scratch1_write(v); + return BUS_OK; + } + // Declared before the lock, so it runs once the lock is dropped. + let _released = (a == SCRATCH3 && v & 1 == 0).then_some(()).and(hooks).map(|h| OnDrop(|| h.scratch3_released())); + let now = (self.clock)(); + let mut r = self.regs.lock(); + Self::complete(&mut r, now); + match a { + reg::NODE_REGION => { + let region = ((v >> 46) & 3) as u8; + if region == 3 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB invalid region size"); + } else { + r.node = (v >> 32) as u8; + r.region = region; + } + } + reg::PIO_WINDOW..=reg::PIO_WINDOW_END => { + let n = (a as usize - 0x8000) / 8; + r.scratch[n] = if n < 3 { v } else { v & 1 }; + } + reg::ATOMIC_SCRATCH..=reg::ATOMIC_SCRATCH_END => { + let n = (a as usize - reg::ATOMIC_SCRATCH as usize) / 8; + if r.scratch[n] == 0 { + r.scratch[n] = v; + } + } + reg::PIO_PERMISSION_KEY => { + if v == IIO_PROTECT_OVRRD_KEY { + *r.words.entry(reg::PIO_PERMISSION).or_default() |= 1; + } + } + reg::SOFT_RESET => {} + reg::MICRO_LAN => { + if r.deadline.is_some() { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB MicroLAN write while slot pending discarded"); + } else { + r.words.insert(a, v & ((1 << 34) - 4)); + let (pulse, sample) = w1::slot::fields(v); + r.sample = match self.microlan.lock().as_mut() { + Some(bus) => w1::slot::slot(bus, pulse, sample), + None => pulse >= w1::slot::RESET_PULSE + || pulse < w1::slot::ONE_PULSE, + }; + r.deadline = Some(now.saturating_add(1000)); + } + } + reg::ERROR_STATUS => { + *r.words.entry(a).or_default() |= v & 0x7ff; + } + reg::ERROR_CLEAR => { + *r.words.entry(0x4040).or_default() &= !(v & 0x7ff); + } + reg::VECTOR_REQUEST => { + r.words.insert(a, v & VECTOR_MASK); + if r.words.get(®::VECTOR_STATUS).copied().unwrap_or(0) & (1 << 63) != 0 { + *r.words.entry(reg::VECTOR_STATUS).or_default() |= 1 << 62; + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB vector launch reaches no peer; no reply latched"); + } + } + reg::VECTOR_STATUS | reg::VECTOR_STATUS_ALIAS => { + if v == 0 { + r.words.insert(reg::VECTOR_STATUS, 0); + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB nonzero vector status write ignored"); + } + } + _ => { + if let Some(mask) = Self::mask(a) { + r.words.insert(a, v & mask); + if (a == reg::PIO_PERMISSION || a == reg::INBOUND_PERMISSION) && v != !0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB permission enforcement requires requester context"); + } + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock LB unsupported write {a:#x} = {v:#x}"); + } + } + } + drop(r); + self.update_irq(); + if a == reg::NODE_REGION { + self.notify_node(); + } + if a == reg::SOFT_RESET && v & 1 != 0 { + self.power_on(); + let cb = self.reset.lock().clone(); + if let Some(cb) = cb { + cb(); + } + } + BUS_OK + } +} + +#[cfg(test)] +mod tests { + // Cases follow tests/qtest/sn1-bedrock-lb-test.c in + // jamesbraid/qemu-sgi-origami: identity-permissions, scratch-reset and + // microlan-slot. + use super::*; + /// With hooks connected, scratch 1's stores are theirs to make and its + /// reads are reported first; a store that leaves scratch 3 clear, and + /// only that, reports the hub lock released, with the LB unlocked. + #[test] + fn scratch_hooks_see_scratch_1_and_the_hub_lock_release() { + type OnceLockLb = std::sync::OnceLock>; + #[derive(Default)] + struct Seen(Mutex>, OnceLockLb); + impl ScratchHooks for Seen { + fn scratch1_write(&self, value: u64) { + self.0.lock().push(format!("w{value:#x}")); + self.1.get().unwrap().upgrade().unwrap().set_scratch1(value + 1); + } + fn scratch1_read(&self) { + self.0.lock().push("r".into()); + } + fn scratch3_released(&self) { + // The LB is unlocked: reading it does not deadlock. + let lb = self.1.get().unwrap().upgrade().unwrap(); + self.0.lock().push(format!("released {:#x}", lb.read64(SCRATCH3).data)); + } + } + let lb = Arc::new(BedrockLb::new(Arc::new(|| 0))); + let seen = Arc::new(Seen::default()); + seen.1.set(Arc::downgrade(&lb)).unwrap(); + lb.connect_scratch_hooks(seen.clone()); + lb.write64(SCRATCH1, 0x10); + assert_eq!(lb.read64(SCRATCH1).data, 0x11); + lb.write64(SCRATCH3, 1); + lb.write64(SCRATCH3, 0); + assert_eq!(*seen.0.lock(), ["w0x10", "r", "released 0x0"]); + } + use std::sync::atomic::{AtomicBool, AtomicU8, AtomicU64, Ordering}; + /// Hold an IRQ packet's assertion callback while a CPU clears the register. + pub(super) fn check_order( + device: Arc, + install: impl FnOnce(Level), + raise: u64, + value: u64, + clear: impl FnOnce() + Send + 'static, + ) { + let (entered_tx, entered_rx) = std::sync::mpsc::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel(); + let release_rx = Mutex::new(release_rx); + let delivered = Arc::new(AtomicBool::new(false)); + let sink = delivered.clone(); + install(Arc::new(move |level| { + if level { + entered_tx.send(()).unwrap(); + release_rx + .lock() + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + } + sink.store(level, Ordering::SeqCst); + })); + let io = crate::dev::bedrock_io::BedrockIo::new(); + io.connect_hub(device); + let first = std::thread::spawn(move || { + io.inbound_write(14, Some(0), (1 << 47) | 0x1800000 | raise, 8, value) + }); + entered_rx + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + let (done_tx, done_rx) = std::sync::mpsc::channel(); + let second = std::thread::spawn(move || { + clear(); + done_tx.send(()).unwrap(); + }); + // An ordered callback waits for the blocked assertion. A stale publisher + // instead completes the clear early, then overwrites it after release. + let _ = done_rx.recv_timeout(std::time::Duration::from_millis(100)); + release_tx.send(()).unwrap(); + assert_eq!(first.join().unwrap(), BUS_OK); + second.join().unwrap(); + assert!(!delivered.load(Ordering::SeqCst)); + } + + /// The guest configures node permissions and expects selected scratch words to survive reset. + #[test] + fn permission_writes_latch_errors_and_reset_retains_scratch() { + { + let d = BedrockLb::new(Arc::new(|| 0)); + d.write64(reg::NODE_REGION, (0x5a << 32) | (2 << 46)); + assert_eq!(d.node_number(), 0x5a); + d.write64(reg::NODE_REGION, 3 << 46); + assert_eq!(d.node_number(), 0x5a); + d.write64(reg::PIO_PERMISSION, 0); + d.write64(reg::PIO_PERMISSION_KEY, 0); + assert_eq!(d.read64(reg::PIO_PERMISSION).data, 0); + d.write64(reg::PIO_PERMISSION_KEY, IIO_PROTECT_OVRRD_KEY); + assert_eq!(d.read64(reg::PIO_PERMISSION).data, 1); + d.write64(reg::PIO_WINDOW + 8, 42); + d.write64(reg::ERROR_STATUS, !0); + assert_eq!(d.read64(reg::ERROR_STATUS).data, 0x7ff); + d.power_on(); + assert_eq!(d.read64(reg::PIO_WINDOW + 8).data, 42); + assert!(d.error_level()); + d.write64(reg::ERROR_CLEAR, 0x7ff); + assert!(!d.error_level()); + assert_eq!(d.node_number(), 0); + } + { + let d = BedrockLb::new(Arc::new(|| 0)); + d.write64(reg::ATOMIC_SCRATCH, 42); + d.write64(reg::ATOMIC_SCRATCH, 43); + assert_eq!(d.read64(reg::PIO_WINDOW).data, 42); + d.write64(reg::REGISTER_C030, 0xdead); + d.write64(reg::VECTOR_REQUEST, !0); + assert_eq!(d.read64(reg::VECTOR_STATUS).data, 0); + assert_eq!(d.read64(reg::REGISTER_C030).data, 0xdead); + assert_eq!(d.read32(reg::PIO_WINDOW).status, BUS_ERR); + assert_eq!(d.read64(1).status, BUS_ERR); + } + } + /// The guest polls MicroLAN completion and cannot replace a slot that is still pending. + #[test] + fn microlan_slots_finish_after_one_microsecond_and_reject_overlap() { + let time = Arc::new(AtomicU64::new(0)); + let t = time.clone(); + let d = BedrockLb::new(Arc::new(move || t.load(Ordering::SeqCst))); + let word = (0x66 << 27) | (60 << 10) | (0x41 << 2) | 3; + d.write64(reg::MICRO_LAN, word); + assert_eq!(d.read64(reg::MICRO_LAN).data, word & !3); + d.write64(reg::MICRO_LAN, 8 << 10); + assert_eq!(d.read64(reg::MICRO_LAN).data, word & !3); + time.store(1000, Ordering::SeqCst); + assert_eq!(d.read64(reg::MICRO_LAN).data, (word & !3) | 2); + d.write64(reg::MICRO_LAN, 8 << 10); + time.store(2000, Ordering::SeqCst); + assert_eq!(d.read64(reg::MICRO_LAN).data, (8 << 10) | 3); + d.write64(reg::MICRO_LAN, 60 << 10); + d.power_on(); + time.store(3000, Ordering::SeqCst); + assert_eq!(d.read64(reg::MICRO_LAN).data, 0); + } + /// Node-change callbacks may reenter LB while guest writes race. + #[test] + fn notifications_preserve_concurrent_register_order() { + { + let ni = Arc::new(crate::dev::bedrock_ni::BedrockNi::new()); + let a = ni.clone(); + let b = ni.clone(); + check_order( + ni, + move |cb| a.connect_error(cb), + crate::dev::bedrock_ni::reg::ERROR_STATUS, + 1 << 24, + move || { + b.read64(crate::dev::bedrock_ni::reg::ERROR_CLEAR); + }, + ); + } + { + let d = Arc::new(BedrockLb::new(Arc::new(|| 0))); + let a = d.clone(); + let b = d.clone(); + check_order( + d, + move |cb| a.connect_error(cb), + reg::ERROR_STATUS, + 1, + move || { + b.write64(reg::ERROR_CLEAR, 1); + }, + ); + } + { + let d = Arc::new(crate::dev::bedrock_xb::BedrockXb::new()); + let a = d.clone(); + let b = d.clone(); + check_order( + d, + move |cb| a.connect_error(cb), + crate::dev::bedrock_xb::reg::QUEUE_BASE, + 1, + move || { + b.write64(crate::dev::bedrock_xb::reg::CLEAR_QUEUE_BASE, 0); + b.write64(crate::dev::bedrock_xb::reg::CLEAR_FIRST_ERROR, 0); + }, + ); + } + { + let d = Arc::new(BedrockLb::new(Arc::new(|| 0))); + let delivered = Arc::new(AtomicU8::new(0)); + let sink = delivered.clone(); + let (entered_tx, entered_rx) = std::sync::mpsc::channel(); + let (release_tx, release_rx) = std::sync::mpsc::channel(); + let release_rx = Mutex::new(release_rx); + d.connect_node_changed(Arc::new(move |node| { + if node == 1 { + entered_tx.send(()).unwrap(); + release_rx + .lock() + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + } + sink.store(node, Ordering::SeqCst); + })); + let io = crate::dev::bedrock_io::BedrockIo::new(); + io.connect_hub(d.clone()); + let first = std::thread::spawn(move || { + io.inbound_write(14, Some(0), (1 << 47) | 0x1800000, 8, 1 << 32) + }); + entered_rx + .recv_timeout(std::time::Duration::from_secs(5)) + .unwrap(); + let (done_tx, done_rx) = std::sync::mpsc::channel(); + let other = d.clone(); + let second = std::thread::spawn(move || { + other.write64(0, 2 << 32); + done_tx.send(()).unwrap(); + }); + let _ = done_rx.recv_timeout(std::time::Duration::from_millis(100)); + release_tx.send(()).unwrap(); + assert_eq!(first.join().unwrap(), BUS_OK); + second.join().unwrap(); + assert_eq!(d.node_number(), 2); + assert_eq!(delivered.load(Ordering::SeqCst), 2); + } + } +} diff --git a/src/dev/bedrock_md.rs b/src/dev/bedrock_md.rs new file mode 100644 index 000000000..97eb23c07 --- /dev/null +++ b/src/dev/bedrock_md.rs @@ -0,0 +1,2634 @@ +//! Bedrock's memory and directory controller: bank configuration, memory +//! self-test, ECC, scrubbing and the directory and fetch-operation views. +//! +//! Memory is held by `bedrock_mem::BedrockMemory`. CPU coherence requests +//! and hardware protection enforcement are absent; protection registers +//! retain their words. Known unsupported writes are counted and ignored, +//! and unmapped accesses return a bus error. +//! +//! Plain register words, directory entries and reference counters, +//! protection entries and injected ECC check bytes live in sparse maps. The +//! guest touches a few of the millions of possible entries, and an entry it +//! never wrote reads as its default, so a dense table would only cost memory. +//! +//! Ported from hw/misc/sgi-bedrock-md.c in jamesbraid/qemu-sgi-origami. + +use super::bedrock_mem::{BedrockMemory, RamExtent, BANK_SPAN}; +use crate::traits::*; +use parking_lot::Mutex; +use std::collections::{BTreeSet, HashMap, HashSet}; +use std::sync::atomic::{AtomicU64, Ordering}; +use std::sync::{Arc, OnceLock, Weak}; + + +pub mod reg { + pub const CELL: u64 = 0x0; + pub const CONFIG: u64 = 0x8; + pub const ARBITRATION: u64 = 0x10; + pub const MIGRATION_CONFIG: u64 = 0x18; + /// Outgoing reply queue sizes: read-only, so a write changes nothing. + pub const REPLY_QUEUE_SIZE: u64 = 0x60; + pub const PERF_SELECT0: u64 = 0x10000; + pub const PERF_SELECT1: u64 = 0x10008; + pub const PERF_COUNTER0: u64 = 0x10010; + pub const PERF_COUNTER1: u64 = 0x10018; + pub const REFRESH: u64 = 0x20000; + /// Flash, UART and Synergy timing on the local "junk" bus. + pub const JUNK_BUS_TIMING: u64 = 0x20008; + pub const SELFTEST_COMMAND: u64 = 0x20030; + pub const SELFTEST_DATA: u64 = 0x20038; + pub const SELFTEST_ERROR: u64 = 0x20040; + pub const SELFTEST_STATUS: u64 = 0x20048; + pub const ISSUE_DEBUG: u64 = 0x20060; + pub const DIRECTORY_CONFIG: u64 = 0x40000; + pub const DIRECTORY_ERROR: u64 = 0x40010; + /// Reads the directory error record and clears it. + pub const DIRECTORY_ERROR_ALIAS: u64 = 0x40018; + pub const MIGRATION_DIFFERENCE: u64 = 0x40040; + pub const MIGRATION_VALUE: u64 = 0x40048; + /// Outgoing request queue sizes; the model keeps them fixed. + pub const REQUEST_QUEUE_SIZE: u64 = 0x40050; + pub const DIRECTORY_DEBUG: u64 = 0x40060; + pub const ECC_CONFIG: u64 = 0x60000; + pub const MEMORY_ERROR: u64 = 0x60010; + /// Reads the memory error record and clears it. + pub const MEMORY_ERROR_ALIAS: u64 = 0x60018; + pub const MEMORY_DEBUG: u64 = 0x60040; + + /// Reset sizes: 12 entries in each header field and 24 in each data field. + pub const REPLY_QUEUE_SIZES: u64 = 0x0018_1818_0c0c_0c00; + /// Reset sizes: four requests for each of the three queues. + pub const REQUEST_QUEUE_SIZES: u64 = 0x0404_0400; +} + +const CONFIG_MASK: u64 = 0xfbfbfbfb | (3 << 32) | (15 << 44) | (63 << 48) | (7 << 56) | (3 << 60); +const ECC_COLUMNS: [u8; 72] = [ + 0x13, 0x23, 0x43, 0x83, 0x2f, 0xf1, 0x0d, 0x07, 0xd0, 0x70, 0x4f, 0xf8, 0x61, 0x62, 0x64, 0x68, + 0x1c, 0x2c, 0x4c, 0x8c, 0x15, 0x25, 0x45, 0x85, 0x19, 0x29, 0x49, 0x89, 0x1a, 0x2a, 0x4a, 0x8a, + 0x51, 0x52, 0x54, 0x58, 0x91, 0x92, 0x94, 0x98, 0xa1, 0xa2, 0xa4, 0xa8, 0x31, 0x32, 0x34, 0x38, + 0x16, 0x26, 0x46, 0x86, 0x1f, 0xf2, 0x0b, 0x0e, 0xb0, 0xe0, 0x8f, 0xf4, 0xc1, 0xc2, 0xc4, 0xc8, + 1, 2, 4, 8, 16, 32, 64, 128, +]; +pub fn ecc_compute(mut value: u64) -> u8 { + let mut code = 0; + while value != 0 { + code ^= ECC_COLUMNS[value.trailing_zeros() as usize]; + value &= value - 1; + } + code +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum MemoryError { + Correctable, + Uncorrectable, +} +type ErrorCallback = Arc; +/// Runs the closure it is given with every other processor parked, as the +/// board's `World::stop_others` does. See [`BedrockMd::set_remap_guard`]. +pub type RemapGuard = Arc; + +struct State { + cell: u64, + config: u64, + refresh: u64, + directory_config: u64, + ecc_config: u64, + storage: HashMap, + directory: HashMap, + counters: HashMap, + protection: HashMap, + compat: HashMap, + ecc: HashMap, + admitted: [u16; 2], + memory_error: u64, + directory_error: u64, + marked_directory: HashSet, + /// Marks in each 4 KiB page of the dense walk: injected check bytes and + /// marked directory lines. A page is checked while it has any. + marked_pages: HashMap, + /// Pages that gained their first mark since the CPU side last placed + /// them, and how many have, so an access can tell whether it added one. + newly_marked: Vec, + marks_added: u64, + /// A self-test cleared whole banks of marks. + retire_now: bool, + clock_ns: u64, + clock_anchor: Option, + selftest_deadline: Option, + selftest_result: bool, + selftest_status: u64, + selftest_error: u64, +} +impl Default for State { + fn default() -> Self { + Self { + cell: u64::MAX, + config: 0, + refresh: 0, + directory_config: 0, + ecc_config: 0, + storage: HashMap::new(), + directory: HashMap::new(), + counters: HashMap::new(), + protection: HashMap::new(), + compat: HashMap::new(), + ecc: HashMap::new(), + admitted: [0; 2], + memory_error: 0, + directory_error: 0, + marked_directory: HashSet::new(), + marked_pages: HashMap::new(), + newly_marked: Vec::new(), + marks_added: 0, + retire_now: false, + clock_ns: 0, + clock_anchor: None, + selftest_deadline: None, + selftest_result: false, + selftest_status: 0, + selftest_error: 0, + } + } +} + +impl State { + fn page_marked(&mut self, page: u64) { + let marks = self.marked_pages.entry(page).or_insert(0); + *marks += 1; + if *marks == 1 { + self.newly_marked.push(page); + self.marks_added += 1; + } + } + fn page_cleared(&mut self, page: u64) { + if let Some(marks) = self.marked_pages.get_mut(&page) { + *marks -= 1; + if *marks == 0 { + self.marked_pages.remove(&page); + } + } + } + /// Inject check byte `code` over the doubleword `key` of the dense walk. + fn ecc_mark(&mut self, key: u64, code: u8) { + if self.ecc.insert(key, code).is_none() { + self.page_marked(key / (4096 / 8)); + } + } + fn ecc_lift(&mut self, key: u64) { + if self.ecc.remove(&key).is_some() { + self.page_cleared(key / (4096 / 8)); + } + } + /// Mark directory line `line` of the dense walk as holding a bad entry. + fn directory_mark(&mut self, line: u64) { + if self.marked_directory.insert(line) { + self.page_marked(line / (4096 / 128)); + } + } + fn directory_lift(&mut self, line: u64) { + if self.marked_directory.remove(&line) { + self.page_cleared(line / (4096 / 128)); + } + } + /// Drop every mark, as a reset does. + fn lift_all(&mut self) { + self.ecc.clear(); + self.marked_directory.clear(); + self.marked_pages.clear(); + self.newly_marked.clear(); + } +} + +pub struct BedrockMd { + memory: BedrockMemory, + clock: Option, + // Ordinary RAM uses Physical's bank views. Diagnostic metadata, repair + // and fetch operations retain the state lock; direct bank traffic has + // the same concurrency contract as the other machines' RAM. + state: Mutex, + // CPU-side metadata writes and resets serialize snapshot/publication. + // RAM/DMA reads and writes may remove metadata without taking this lock. + remapping: Mutex<()>, + callback: Mutex>, + unsupported: AtomicU64, + guard: OnceLock, +} +#[derive(Clone, Copy)] +enum View { + Registers, + Memory, + Uncached, + Scrub, + Directory, + Fetchop, + Ecc, +} +struct MdBus { + md: Weak, + view: View, + // External endpoints may outlive MD. Keep their generation pointers valid. + // The endpoint owned by Physical itself needs no extra reference. + _memory: Option>, + #[cfg(feature = "jitv2")] + _generations: Arc, +} + +impl BedrockMd { + /// A controller with `banks` fitted, in MiB from locational bank 0; see + /// [`BedrockMemory::new`] for what a board may fit. + pub fn new(banks: &[u32]) -> Result, String> { + Self::build(banks, None) + } + pub fn with_clock(banks: &[u32], clock: crate::dev::heart::Clock) -> Result, String> { + Self::build(banks, Some(clock)) + } + fn build(banks: &[u32], clock: Option) -> Result, String> { + let md = Arc::new(Self { + memory: BedrockMemory::new(banks)?, + clock, + state: Mutex::new(State::default()), + remapping: Mutex::new(()), + callback: Mutex::new(None), + unsupported: AtomicU64::new(0), + guard: OnceLock::new(), + }); + md.memory.set_slow_view(md.view(View::Memory)); + md.rebuild_memory(); + Ok(md) + } + /// Have every rebuild of the whole memory map run inside `guard`. A + /// rebuild scrubs and remaps host pages that another processor may be + /// running from, so on a multiprocessor board it must run with the others + /// parked. Placing and retiring one checked page only takes a region off + /// the direct path and swaps table entries, and needs no guard. The guard + /// is entered before any lock of this controller is taken, so a processor + /// that is waiting for one cannot keep it from stopping the others. + /// The configuration register write, [`Self::rebuild_memory`], + /// [`Self::soft_reset`], a startup-window change and + /// [`Resettable::power_on`] enter it. Set once, before the processors run. + pub fn set_remap_guard(&self, guard: RemapGuard) { + assert!(self.guard.set(guard).is_ok(), "Bedrock MD remap guard already set"); + } + fn stopped(&self, work: impl FnOnce() -> R) -> R { + let Some(guard) = self.guard.get() else { return work() }; + let (mut work, mut done) = (Some(work), None); + guard(&mut || done = work.take().map(|work| work())); + done.expect("the remap guard ran its work") + } + /// Where fitted RAM can answer, for the processor bus's cached view. + pub fn ram_extent(&self) -> RamExtent { + self.memory.extent() + } + pub fn register_locks(self: &Arc) { + let weak = Arc::downgrade(self); + crate::locks::register_lock_fn("bedrock-md::state", move || { + weak.upgrade().is_some_and(|md| md.state.is_locked()) + }); + let weak = Arc::downgrade(self); + crate::locks::register_lock_fn("bedrock-md::remapping", move || { + weak.upgrade().is_some_and(|md| md.remapping.is_locked()) + }); + let weak = Arc::downgrade(self); + crate::locks::register_lock_fn("bedrock-md::callback", move || { + weak.upgrade().is_some_and(|md| md.callback.is_locked()) + }); + } + pub fn physical(&self) -> &Arc { + self.memory.physical() + } + fn view(self: &Arc, view: View) -> Arc { + Arc::new(MdBus { + md: Arc::downgrade(self), + view, + #[cfg(feature = "jitv2")] + _generations: self.memory.generations(), + _memory: if matches!(view, View::Uncached) { + Some(self.physical().clone()) + } else { + None + }, + }) + } + pub fn md_bus(self: &Arc) -> Arc { + self.view(View::Registers) + } + pub fn directory_bus(self: &Arc) -> Arc { + self.view(View::Directory) + } + pub fn fetchop_bus(self: &Arc) -> Arc { + self.view(View::Fetchop) + } + pub fn ecc_bus(self: &Arc) -> Arc { + self.view(View::Ecc) + } + /// DMA uses the same Physical entries as the CPU, without accessing + /// ppmem's CPU-owned bitmap. Hardware protection remains unsupported. + pub fn dma_view(self: &Arc) -> Arc { + Arc::new(DmaView(self.clone())) + } + + fn mapping(s: &State) -> (u64, BTreeSet) { + (s.config, s.marked_pages.keys().copied().collect()) + } + + /// Reinstall RAM after the board sets Physical's default. Call on the + /// CPU thread or with the CPU stopped, before installing board overlays. + /// Ordinary CPU config/diagnostic writes rebuild their changed mappings + /// automatically. This also releases pages whose metadata DMA consumed. + pub fn rebuild_memory(&self) { + self.stopped(|| { + let _remapping = self.remapping.lock(); + let (config, pages) = Self::mapping(&self.state.lock()); + self.memory.remap(config, pages, true); + }) + } + /// The guest stored `tag` in a primary data cache tag: the PROM building + /// its stack in cache lines, or ending that. While they stand the window + /// answers from a page of its own on every path and stays out of the + /// direct window. See [`BedrockMemory::startup_tag`]. Called on the CPU + /// thread, outside the cache's locks. + pub fn startup_tag_store(&self, tag: u32) { + let cpu = crate::cpu::smp::current_cpu(); + let Some(live) = self.memory.startup_tag(cpu, tag) else { return }; + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock MD: CPU {cpu}'s startup window {}", if live { "starts" } else { "ends" }); + self.stopped(|| { + let _remapping = self.remapping.lock(); + if self.memory.set_startup(cpu, live) { + let (config, pages) = Self::mapping(&self.state.lock()); + self.memory.remap(config, pages, true); + } + }) + } + /// Zero every bank, as a power-on does. Physical's own `reset_memory` + /// advances only the banks' counters; code pages follow the logical ones, + /// so a board resets the banks through here. + pub fn reset_memory(&self) { + self.memory.physical().reset_memory(); + self.memory.changed_decode(); + } + /// Load bank `bank` from a file of `Physical::save_bank`'s format, and + /// advance the logical counters the bytes changed under. + pub fn load_bank(&self, bank: usize, path: impl AsRef) -> std::io::Result<()> { + let loaded = self.memory.physical().load_bank(bank, path); + self.memory.changed_decode(); + loaded + } + /// Restore bank `bank` from words `Physical::snapshot_bank_inmem` took, + /// and advance the logical counters the bytes changed under. + pub fn restore_bank(&self, bank: usize, words: &[u32]) { + self.memory.physical().restore_bank_inmem(bank, words); + self.memory.changed_decode(); + } + /// The stable logical-physical counters used by both compiled stores + /// and SysAd page claims. Their storage outlives the CPU via its bus. + #[cfg(feature = "jitv2")] + pub(crate) fn generation_base(&self) -> *mut AtomicU64 { + self.memory.generations().base() + } + pub fn on_error(&self, callback: ErrorCallback) { + *self.callback.lock() = Some(callback); + } + /// Explicit elapsed time for a manually clocked controller. + pub fn advance_ns(&self, elapsed: u64) { + let mut s = self.state.lock(); + Self::advance(&mut s, elapsed); + } + fn advance(s: &mut State, elapsed: u64) { + s.clock_ns = s.clock_ns.saturating_add(elapsed); + if s.selftest_deadline.is_some_and(|end| s.clock_ns >= end) { + s.selftest_deadline = None; + s.selftest_status = if s.selftest_result { 3 } else { 2 }; + s.selftest_error = if s.selftest_result { 0 } else { 1 << 48 }; + } + } + fn catch_up(s: &mut State, now: u64) { + if let Some(anchor) = s.clock_anchor { + s.clock_anchor = Some(anchor.max(now)); + Self::advance(s, now.saturating_sub(anchor)); + } + } + pub fn start_clock(&self) { + if let Some(clock) = &self.clock { + let now = clock(); + self.state.lock().clock_anchor.get_or_insert(now); + } + } + pub fn stop_clock(&self) { + if let Some(clock) = &self.clock { + let now = clock(); + let mut s = self.state.lock(); + Self::catch_up(&mut s, now); + s.clock_anchor = None; + } + } + /// Normal MD memory semantics without re-entering the cached Physical map. + pub fn memory_view(self: &Arc) -> Arc { + self.view(View::Memory) + } + pub fn uncached_view(self: &Arc) -> Arc { + self.view(View::Uncached) + } + pub fn ecc_scrub_bus(self: &Arc) -> Arc { + self.view(View::Scrub) + } + pub fn current_cell(&self) -> u64 { + self.state.lock().cell + } + pub fn unsupported(&self) -> u64 { + self.unsupported.load(Ordering::Relaxed) + } + /// Count and log an access the model does not implement; the caller + /// answers a bus error. + fn refuse(&self, what: &str, at: u64) -> Option { + self.unsupported.fetch_add(1, Ordering::Relaxed); + crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock MD refused {what} access at {at:#x}"); + None + } + /// Called on the CPU thread or with the CPU stopped. + pub fn soft_reset(&self) { + self.stopped(|| self.soft_reset_stopped()) + } + fn soft_reset_stopped(&self) { + let _remapping = self.remapping.lock(); + let mut s = self.state.lock(); + s.cell = u64::MAX; + s.storage.clear(); + s.ecc_config = 0; + s.memory_error = 0; + s.directory_error = 0; + s.lift_all(); + s.selftest_deadline = None; + s.selftest_status = 0; + s.selftest_error = 0; + s.refresh &= (1 << 63) | 0xfff; + let (config, pages) = Self::mapping(&s); + drop(s); + self.memory.remap(config, pages, false); + } + fn access(&self, view: View, addr: u64, bytes: u64, write: Option) -> Option { + if ![1, 2, 4, 8].contains(&bytes) || addr % bytes != 0 { + return self.refuse("access", addr); + } + if write.is_some() && matches!(view, View::Registers) && addr == reg::CONFIG { + return self.stopped(|| self.access_unguarded(view, addr, bytes, write)); + } + self.access_unguarded(view, addr, bytes, write) + } + fn access_unguarded(&self, view: View, addr: u64, bytes: u64, write: Option) -> Option { + // Only CPU register/configuration and diagnostic writes can add + // metadata. Keep their remap off DMA threads: ordinary RAM, scrub + // and fetch operations only remove entries, leaving safe slow pages + // behind until a CPU-side retirement. These write views, like HEART's + // MEMCFG, are CPU-thread-only while the machine runs. + let reconfigure = write.is_some() && matches!(view, View::Registers) && addr == reg::CONFIG; + let marking = write.is_some() && matches!(view, View::Directory | View::Ecc | View::Registers); + let remapping = reconfigure.then(|| self.remapping.lock()); + let mut event = Vec::new(); + // Sample outside state: even an injected clock must not call out under + // the memory lock. Only register observations need the self-test timer. + let now = if matches!(view, View::Registers) { + self.clock.as_ref().map(|clock| clock()) + } else { + None + }; + let (result, snapshot, placing) = { + let mut s = self.state.lock(); + if let Some(now) = now { + Self::catch_up(&mut s, now); + } + let marks_added = s.marks_added; + let result = match view { + View::Registers => self.register(&mut s, addr, bytes, write), + View::Directory => self.directory(&mut s, addr, bytes, write), + View::Ecc => self.ecc(&mut s, addr, bytes, write), + View::Scrub => self.scrub(&mut s, addr, bytes, write, &mut event), + View::Uncached => { + let result = self.ram(&mut s, addr, bytes, write, &mut event); + if write.is_none() && event.contains(&MemoryError::Uncorrectable) { + None + } else { + result + } + } + View::Fetchop => self.fetchop(&mut s, addr, bytes, write, &mut event), + View::Memory => self.ram(&mut s, addr, bytes, write, &mut event), + }; + let placing = marking && (s.marks_added != marks_added || s.retire_now); + (result, reconfigure.then(|| Self::mapping(&s)), placing) + }; + if let Some((config, pages)) = snapshot { + self.memory.reconfigure(config, || pages); + } + drop(remapping); + if placing { + self.place_marks(); + } + if !event.is_empty() { + let callback = self.callback.lock().clone(); + if let Some(callback) = callback { + for kind in event { + callback(kind); + } + } + } + result + } + /// Route the pages that gained a mark through the MD endpoint, and + /// retire the pages whose marks were lifted when enough have piled up or + /// have idled long enough. Only a mark can make a page checked, so this + /// runs after the writes that add one, on the CPU thread. + fn place_marks(&self) { + let _remapping = self.remapping.lock(); + let now = self.clock.as_ref().map(|clock| clock()); + let (added, active, retire_now) = { + let mut s = self.state.lock(); + let added = std::mem::take(&mut s.newly_marked); + (added, s.marked_pages.len(), std::mem::take(&mut s.retire_now)) + }; + self.memory.place(&added); + if retire_now || self.memory.retirement_due(active, now) { + self.retire_lifted(); + } + } + /// Return the pages whose marks are lifted to their banks, in one change. + /// Call on the CPU thread, holding `remapping`. + fn retire_lifted(&self) { + let lifted = { + let s = self.state.lock(); + self.memory.lifted(|page| s.marked_pages.contains_key(&page)) + }; + self.memory.retire(&lifted); + } + /// Return the pages whose ECC or directory marks were lifted to the + /// direct window now, without waiting for the batch to fill or idle. For + /// the board to call from the CPU thread, or with the CPU stopped, when + /// it has a quiet moment. + pub fn retire_lifted_marks(&self) { + let _remapping = self.remapping.lock(); + self.retire_lifted(); + } + fn register(&self, s: &mut State, at: u64, bytes: u64, write: Option) -> Option { + if bytes != 8 { + return self.refuse("register", at); + } + let old = match at { + reg::CELL => s.cell, + reg::CONFIG => s.config, + reg::REPLY_QUEUE_SIZE => reg::REPLY_QUEUE_SIZES, + reg::REFRESH => s.refresh, + reg::SELFTEST_ERROR => s.selftest_error, + reg::SELFTEST_STATUS => s.selftest_status, + reg::DIRECTORY_CONFIG => s.directory_config, + reg::ECC_CONFIG => s.ecc_config, + reg::REQUEST_QUEUE_SIZE => reg::REQUEST_QUEUE_SIZES, + reg::MEMORY_ERROR | reg::MEMORY_ERROR_ALIAS => s.memory_error, + reg::DIRECTORY_ERROR | reg::DIRECTORY_ERROR_ALIAS => s.directory_error, + _ if storage_mask(at).is_some() => *s.storage.get(&at).unwrap_or(&0), + _ if idle(at) => 0, + _ => return self.refuse("register", at), + }; + if let Some(value) = write { + match at { + reg::SELFTEST_COMMAND => return self.selftest(s, value), + reg::CELL => s.cell = value, + reg::CONFIG => { + s.config = value & CONFIG_MASK; + } + reg::REFRESH => s.refresh = value & ((1 << 63) | 0xfff), + reg::DIRECTORY_CONFIG => s.directory_config = value & 3, + reg::ECC_CONFIG => s.ecc_config = value & 1, + _ if storage_mask(at).is_some() => { + s.storage.insert(at, value & storage_mask(at).unwrap()); + } + _ => { + self.unsupported.fetch_add(1, Ordering::Relaxed); + crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock MD ignored register write at {at:#x}: {value:#x}"); + } + } + } else if at == reg::MEMORY_ERROR_ALIAS { + s.memory_error = 0; + } else if at == reg::DIRECTORY_ERROR_ALIAS { + s.directory_error = 0; + } + Some(old) + } + fn directory(&self, s: &mut State, at: u64, bytes: u64, write: Option) -> Option { + if bytes != 8 || at >= 1 << 30 { + return self.refuse("directory", at); + } + let software = at / (BANK_SPAN / 8); + let bank = software ^ ((s.config >> 32) & 3); + if at & 0x7c8 == 0x408 { + let fitted = self.memory.fitted(bank); + if fitted == 0 { + return Some(0); + } + let within = + ((((at % (BANK_SPAN / 8)) >> 11) << 14) | (((at >> 4) & 3) << 7)) % fitted; + let key = ((self.memory.dense_base(bank) + within) >> 14) * 4 + ((within >> 7) & 3); + let old = *s.protection.get(&key).unwrap_or(&0); + if let Some(v) = write { + s.protection.insert(key, v); + } + return Some(old); + } + let within = ((at % (BANK_SPAN / 8)) & !15) * 8; + let field = s.config >> (bank / 2 * 8); + let code = (field >> 4) & 15; + if field & (1 << (bank & 1)) == 0 + || !(1..=5).contains(&code) + || within >= (32 << 20) << code + { + return self.refuse("directory", at); + } + let mask = if at & 8 != 0 || s.directory_config & 1 != 0 { + u64::MAX + } else { + u32::MAX as u64 + }; + let table; + let key; + let fitted = self.memory.fitted(bank); + if within >= fitted { + if !s.compat.contains_key(&at) && s.compat.len() >= 74 { + return self.refuse("directory", at); + } + s.compat.entry(at).or_insert(0); + table = &mut s.compat; + key = at; + } else { + key = (self.memory.dense_base(bank) + within) / 128; + table = if at & 8 != 0 { + &mut s.counters + } else { + &mut s.directory + }; + } + let old = *table.get(&key).unwrap_or(&0) & mask; + if let Some(value) = write { + let value = if at & 8 == 0 && s.directory_config & 2 == 0 { + directory_encode(value, s.directory_config & 1 != 0) + } else { + value + }; + table.insert(key, value & mask); + if within < fitted && at & 8 == 0 { + if directory_syndrome(value, s.directory_config & 1 != 0) != 0 { + s.directory_mark(key); + } else { + s.directory_lift(key); + } + } + } + Some(old) + } + fn checked_word( + &self, + s: &mut State, + bank: usize, + at: u64, + address: u64, + event: &mut Vec, + ) -> u64 { + let mut data = self.memory.raw_read(bank, at); + if s.ecc_config & 1 != 0 || s.ecc.is_empty() { + return data; + } + let key = (self.memory.dense_base(bank as u64) + at) / 8; + if let Some(&check) = s.ecc.get(&key) { + let syndrome = check ^ ecc_compute(data); + if syndrome != 0 { + let bit = ECC_COLUMNS.iter().position(|&v| v == syndrome); + let valid = if bit.is_some() { 1 << 52 } else { 1 << 60 }; + if bit.is_none() + && s.memory_error & (1 << 60) != 0 + && (s.memory_error ^ address) & (((1u64 << 30) - 1) << 3) & !127 != 0 + { + s.memory_error |= 1 << 63; + } + if s.memory_error & (valid | (1 << 60)) != 0 { + s.memory_error |= valid << 1; + } else { + s.memory_error = (s.memory_error + & !((0xff << 40) | (((1u64 << 30) - 1) << 3))) + | ((syndrome as u64) << 40) + | (address & (((1u64 << 30) - 1) << 3)); + } + s.memory_error |= valid; + if let Some(bit) = bit { + if bit < 64 { + data ^= 1 << bit; + self.memory.raw_write(bank, at, data); + } + s.ecc_lift(key); + event.push(MemoryError::Correctable); + } else { + event.push(MemoryError::Uncorrectable); + } + } + } + data + } + fn ram( + &self, + s: &mut State, + address: u64, + bytes: u64, + write: Option, + event: &mut Vec, + ) -> Option { + if let Some(page) = self.memory.startup_device(address, bytes) { + return Some(startup_access(page, address, bytes, write)); + } + let Some((bank, offset)) = self.memory.decode(s.config, address, bytes) else { + return self.refuse("ram", address); + }; + if !s.marked_directory.is_empty() { + let line = (self.memory.dense_base(bank as u64) + offset) / 128; + if s.marked_directory.contains(&line) { + let premium = s.directory_config & 1 != 0; + let word = *s.directory.get(&line).unwrap_or(&0); + if write.is_some() { + s.directory.insert(line, directory_encode(word, premium)); + s.directory_lift(line); + } else if s.directory_config & 2 == 0 { + let syndrome = directory_syndrome(word, premium); + if syndrome != 0 { + let bit = directory_columns(premium) + .iter() + .position(|&v| v == syndrome); + let valid = if bit.is_some() { 1 << 61 } else { 1 << 63 }; + if s.directory_error & ((1 << 63) | valid) != 0 { + s.directory_error |= if bit.is_some() { 1 << 57 } else { 1 << 59 }; + } else { + let carrier = 0x1c0000000 + + (address / BANK_SPAN) * (BANK_SPAN / 8) + + ((offset & !127) / 8); + s.directory_error = (s.directory_error + & !((0x7f << 40) | (((1u64 << 30) - 1) << 3))) + | ((syndrome as u64) << 40) + | (carrier & (((1u64 << 30) - 1) << 3)); + } + s.directory_error |= valid; + if let Some(bit) = bit { + s.directory.insert(line, word ^ (1 << bit)); + s.directory_lift(line); + event.push(MemoryError::Correctable); + } else { + event.push(MemoryError::Uncorrectable); + return None; + } + } + } + } + } + let at = offset & !7; + let shift = (8 - (offset & 7) - bytes) * 8; + let mask = u64::MAX >> ((8 - bytes) * 8); + if let Some(value) = write { + let old = self.memory.raw_read(bank, at); + self.memory.raw_write( + bank, + at, + (old & !(mask << shift)) | ((value & mask) << shift), + ); + if !s.ecc.is_empty() { + s.ecc_lift((self.memory.dense_base(bank as u64) + at) / 8); + } + Some(0) + } else { + Some((self.checked_word(s, bank, at, address, event) >> shift) & mask) + } + } + fn fetchop( + &self, + s: &mut State, + at: u64, + bytes: u64, + write: Option, + event: &mut Vec, + ) -> Option { + if bytes != 4 && bytes != 8 { + return self.refuse("fetchop", at); + } + let op = at & 0x38; + if op > if write.is_some() { 32 } else { 24 } { + return self.refuse("fetchop", at); + } + let address = (at & !63) | (at & 4); + let old = self.ram(s, address, bytes, None, event)?; + let next = match (write, op) { + (None, 0) => return Some(old), + (Some(v), 0) => v, + (_, 8) => old.wrapping_add(1), + (_, 16) => old.wrapping_sub(1), + (None, 24) => 0, + (Some(v), 24) => old & v, + (Some(v), 32) => old | v, + _ => return self.refuse("fetchop", at), + }; + self.ram(s, address, bytes, Some(next), event)?; + Some(old) + } + // Approximation: report whether the selected fitting covers the + // programmed bank, after 1000 guest ns. Only the zero write phase mutates + // memory; other patterns have no published transformation here. + fn selftest(&self, s: &mut State, command: u64) -> Option { + if command == 2 || command == 4 { + s.selftest_deadline = None; + s.selftest_status = 0; + s.selftest_error = 0; + return Some(0); + } + if command & 7 != 1 || command & !0x71 != 0 { + self.unsupported.fetch_add(1, Ordering::Relaxed); + crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock MD ignored self-test command {command:#x}"); + return Some(0); + } + let selected = (command >> 4) & 7; + let configured = selected ^ ((s.config >> 32) & 3); + let field = s.config >> ((configured / 2) * 8); + let code = (field >> 4) & 15; + let size = if (1..=5).contains(&code) { + (32 << 20) << code + } else { + 0 + }; + let fits = field & (1 << (configured & 1)) != 0 + && size != 0 + && size <= self.memory.fitted(selected); + s.selftest_result = fits; + s.selftest_status = 0; + s.selftest_error = 0; + s.selftest_deadline = Some(s.clock_ns.saturating_add(1000)); + if fits && s.storage.get(&0x20038).copied().unwrap_or(0) == 0x80 { + self.memory.zero(selected as usize, size); + let start = self.memory.dense_base(selected); + let words: Vec = s + .ecc + .keys() + .copied() + .filter(|&key| key >= start / 8 && key < (start + size) / 8) + .collect(); + for key in words { + s.ecc_lift(key); + } + // The pages those marks held can go back to the window now. + s.retire_now = true; + let lines: Vec = s + .marked_directory + .iter() + .copied() + .filter(|&line| line >= start / 128 && line < (start + size) / 128) + .collect(); + for line in lines { + let word = *s.directory.get(&line).unwrap_or(&0); + s.directory + .insert(line, directory_encode(word, s.directory_config & 1 != 0)); + s.directory_lift(line); + } + } + Some(0) + } + fn scrub( + &self, + s: &mut State, + at: u64, + bytes: u64, + write: Option, + event: &mut Vec, + ) -> Option { + if write.is_none() { + return self.ecc(s, at, bytes, None); + } + if bytes != 8 || at >= 1 << 30 { + return self.refuse("scrub", at); + } + let software = at / (BANK_SPAN / 8); + let within = at % (BANK_SPAN / 8); + let address = software * BANK_SPAN + within * 8; + let Some((bank, offset)) = self.memory.decode(s.config, address, 8) else { + return self.refuse("scrub", at); + }; + for i in 0..8 { + if self.memory.decode(s.config, address + i * 8, 8).is_some() { + self.checked_word(s, bank, offset + i * 8, address + i * 8, event); + } + } + Some(0) + } + fn ecc(&self, s: &mut State, at: u64, bytes: u64, write: Option) -> Option { + if at >= 1 << 30 || (bytes != 1 && bytes != 2) { + return self.refuse("ecc", at); + } + let software = at / (BANK_SPAN / 8); + let within = (at % (BANK_SPAN / 8)) ^ if bytes == 1 { 7 } else { 6 }; + let address = software * BANK_SPAN + within * 8; + let Some((bank, offset)) = self.memory.decode(s.config, address, bytes * 8) else { + if bytes == 2 && [0x04000006, 0x08000006].contains(&at) { + let i = usize::from(at == 0x08000006); + let old = s.admitted[i]; + if let Some(value) = write { + s.admitted[i] = value as u16; + } + return Some(old as u64); + } + return self.refuse("ecc", at); + }; + let mut old = 0; + for i in 0..bytes { + let raw = offset + i * 8; + let key = (self.memory.dense_base(bank as u64) + raw) / 8; + let computed = ecc_compute(self.memory.raw_read(bank, raw)); + old |= (s.ecc.get(&key).copied().unwrap_or(computed) as u64) << (i * 8); + if let Some(value) = write { + let code = (value >> (i * 8)) as u8; + if code == computed { + s.ecc_lift(key); + } else { + s.ecc_mark(key, code); + } + } + } + Some(old) + } +} +impl Resettable for BedrockMd { + fn power_on(&self) { + // Reset is called on the CPU thread or with it stopped. + self.stopped(|| { + let _remapping = self.remapping.lock(); + *self.state.lock() = State::default(); + let was_live = self.memory.startup_live(); + self.memory.clear_startup(); + self.memory.remap(0, BTreeSet::new(), was_live); + }) + } +} +/// Resolve each DMA access through the atomic Physical table, as HEART +/// does. Do not ask Physical for mem_ptr or gen_ptr: those read the inline +/// CPU bitmap. Default block methods split accesses and respect slow pages +/// even if a caller supplies a block crossing a 4 KiB page boundary. +struct DmaView(Arc); +impl DmaView { + fn at(&self, address: u64, bytes: u64) -> Option<&dyn BusDevice> { + if let Some(page) = self.0.memory.startup_device(address, bytes) { + return Some(page); + } + self.0.memory.decode(self.0.memory.config(), address, bytes)?; + let phys = self.0.physical(); + let entry = phys.entry_at(address); + if crate::ppmem::devmap::entry_eq(entry, phys.default_entry()) { + return None; + } + // SAFETY: the owning MD retains Physical, which keeps every entry + // alive across remaps, including the weak MD slow endpoint. + Some(unsafe { &*entry }) + } +} +macro_rules! dma_rw { + ($read:ident, $write:ident, $result:ident, $ty:ty) => { + fn $read(&self, address: u64) -> $result { + self.at(address, std::mem::size_of::<$ty>() as u64) + .map_or($result::err(), |device| device.$read(address)) + } + fn $write(&self, address: u64, value: $ty) -> u32 { + self.at(address, std::mem::size_of::<$ty>() as u64) + .map_or(BUS_ERR, |device| { + let result = device.$write(address, value); + if result == BUS_OK { self.0.memory.written(address); } + result + }) + } + }; +} +impl BusDevice for DmaView { + dma_rw!(read8, write8, BusRead8, u8); + dma_rw!(read16, write16, BusRead16, u16); + dma_rw!(read32, write32, BusRead32, u32); + dma_rw!(read64, write64, BusRead64, u64); + fn cas32(&self, address: u64, old: u32, new: u32) -> Option { + let swapped = self.at(address, 4)?.cas32(address, old, new)?; + if swapped { self.0.memory.written(address); } + Some(swapped) + } + fn cas64(&self, address: u64, old: u64, new: u64) -> Option { + let swapped = self.at(address, 8)?.cas64(address, old, new)?; + if swapped { self.0.memory.written(address); } + Some(swapped) + } + #[cfg(feature = "jitv2")] + fn gen_ptr(&self, address: u64) -> *const AtomicU64 { + self.0.memory.gen_ptr(address) + } +} + +/// A read or write of `bytes` at `address` in the startup page. The page is +/// plain memory: no ECC, no directory, no fetch-and-op. +fn startup_access(page: &dyn BusDevice, address: u64, bytes: u64, write: Option) -> u64 { + match (bytes, write) { + (1, None) => page.read8(address).data as u64, + (2, None) => page.read16(address).data as u64, + (4, None) => page.read32(address).data as u64, + (_, None) => page.read64(address).data, + (1, Some(v)) => { page.write8(address, v as u8); 0 } + (2, Some(v)) => { page.write16(address, v as u16); 0 } + (4, Some(v)) => { page.write32(address, v as u32); 0 } + (_, Some(v)) => { page.write64(address, v); 0 } + } +} + +impl BusDevice for MdBus { + /// Bedrock's ECC view is the one device that sees the whole source register + /// of an uncached SH (`BusDevice::write_narrow_from`); every other view + /// takes the architectural low bits. + fn write_narrow_from(&self, addr: u64, size: usize, gpr: u64) -> u32 { + if size == 2 && matches!(self.view, View::Ecc) { + // PROM 6.210's memory test stores each check word with `sh` from a + // register whose low 16 bits are zero: the word sits elsewhere in + // the register, by the address of the memory it checks. Observed + // (memory address mod 0x40, carrier offset mod 8, register bits): + // 0x00 6 15:0 + // 0x10 4 63:48 + // 0x20 2 47:32 + // 0x30 0 31:16 + // The shift below gives exactly these four. Which lanes the R14000 + // drives for a narrow store, and how Bedrock wires them to the + // check bits, is unknown; this matches the PROM, not a hardware + // description. + let memory_address = (addr ^ 6).wrapping_mul(8); + return self.write16(addr, (gpr >> (memory_address.wrapping_neg() & 63)) as u16); + } + if size == 1 { self.write8(addr, gpr as u8) } else { self.write16(addr, gpr as u16) } + } + + #[cfg(feature = "jitv2")] + fn gen_ptr(&self, address: u64) -> *const std::sync::atomic::AtomicU64 { + let Some(md) = self.md.upgrade() else { + return std::ptr::null(); + }; + if !matches!(self.view, View::Memory | View::Uncached) { + return std::ptr::null(); + } + md.memory.gen_ptr(address) + } + + fn read8(&self, a: u64) -> BusRead8 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 1, None)) + .map_or(BusRead8::err(), |v| BusRead8::ok(v as u8)) + } + fn read16(&self, a: u64) -> BusRead16 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 2, None)) + .map_or(BusRead16::err(), |v| BusRead16::ok(v as u16)) + } + fn read32(&self, a: u64) -> BusRead32 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 4, None)) + .map_or(BusRead32::err(), |v| BusRead32::ok(v as u32)) + } + fn read64(&self, a: u64) -> BusRead64 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 8, None)) + .map_or(BusRead64::err(), BusRead64::ok) + } + fn write8(&self, a: u64, v: u8) -> u32 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 1, Some(v as u64))) + .map_or(BUS_ERR, |_| BUS_OK) + } + fn write16(&self, a: u64, v: u16) -> u32 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 2, Some(v as u64))) + .map_or(BUS_ERR, |_| BUS_OK) + } + fn write32(&self, a: u64, v: u32) -> u32 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 4, Some(v as u64))) + .map_or(BUS_ERR, |_| BUS_OK) + } + fn write64(&self, a: u64, v: u64) -> u32 { + self.md + .upgrade() + .and_then(|m| m.access(self.view, a, 8, Some(v))) + .map_or(BUS_ERR, |_| BUS_OK) + } +} +fn storage_mask(at: u64) -> Option { + Some(match at { + reg::ARBITRATION => 0xff, + reg::MIGRATION_CONFIG => 0x3ff | (0xff << 16) | (1 << 32), + reg::PERF_SELECT0 | reg::PERF_SELECT1 => { + 3 | (15 << 4) | (127 << 8) | (15 << 16) | (0xffffffffff << 20) | (1 << 63) + } + reg::PERF_COUNTER0 | reg::PERF_COUNTER1 => (1 << 41) - 1, + reg::JUNK_BUS_TIMING => 0xffffffffffff, + reg::SELFTEST_DATA => 0xff, + reg::ISSUE_DEBUG => 3, + reg::MIGRATION_DIFFERENCE => 0x7fff | (7 << 32) | (7 << 35) | (1 << 63), + reg::MIGRATION_VALUE => 0x7fff | (7 << 32) | (7 << 35) | (3 << 62), + reg::DIRECTORY_DEBUG => 3, + reg::MEMORY_DEBUG => 1, + _ => return None, + }) +} +fn idle(at: u64) -> bool { + matches!( + at, + 0x20 | 0x28 + | 0x40 + | 0x48 + | 0x58 + | 0x20010 + | 0x20018 + | 0x20020 + | 0x20028 + | 0x20030 + | 0x20040 + | 0x20048 + | 0x40020 + | 0x40028 + | 0x40030 + | 0x40038 + | 0x40058 + | 0x60020 + | 0x60028 + | 0x60030 + | 0x60038 + ) +} + +#[cfg(test)] +mod tests { + // Cases follow tests/qtest/sn1-bedrock-md-test.c in + // jamesbraid/qemu-sgi-origami: memory-banks, reset, queue-sizes, + // writable-masks, selftest-phases, fetchop-operations, ecc-carrier and + // directory-storage. + use super::*; + use super::super::bedrock_mem::{RETIRED_IDLE_NS, RETIRED_LIMIT}; + const FUEL_BANKS: [u32; 2] = [256, 256]; + const FITTED_BANK: u64 = 256 << 20; + fn fuel() -> Arc { + BedrockMd::new(&FUEL_BANKS).unwrap() + } + fn configured() -> Arc { + let md = fuel(); + assert_eq!(md.md_bus().write64(8, 0x33), BUS_OK); + md + } + + /// What PROM 6.210 stores as primary data cache tags (TagLo), measured: + /// test patterns, then an invalidate, then a valid tag per page of the + /// window, then the invalidate sweep it leaves. + const WINDOW_TAGS: [u32; 8] = [0x100c0, 0x101c1, 0x102c0, 0x103c1, 0x104c0, 0x105c1, 0x106c0, 0x107c1]; + + /// A valid tag for a page of the window starts a private page that every + /// path reaches, whether or not DRAM is configured; an invalid tag ends it + /// and leaves memory as it was. + #[test] + fn a_valid_tag_for_the_window_starts_a_page_every_path_answers() { + let md = fuel(); + let phys = md.physical(); + let (dma, uncached) = (md.dma_view(), md.uncached_view()); + let at = 0x10_0008 + 0x7000; + assert_eq!(phys.read64(at).status, BUS_ERR, "no memory behind the window yet"); + md.startup_tag_store(WINDOW_TAGS[0]); + assert_eq!(phys.write64(at, 0x1122_3344_5566_7788), BUS_OK); + assert_eq!(phys.read64(at).data, 0x1122_3344_5566_7788); + assert_eq!(dma.read64(at).data, 0x1122_3344_5566_7788, "the cached path and DMA"); + assert_eq!(uncached.read32(at + 4).data, 0x5566_7788, "the uncached path"); + assert_eq!(uncached.write8(at, 0xaa), BUS_OK); + assert_eq!(phys.read64(at).data, 0xaa22_3344_5566_7788); + assert_eq!(phys.read64(0x10_8000).status, BUS_ERR, "only the window's 32 KiB"); + // The page wraps in itself, and the bank behind it is untouched. + assert_eq!(md.md_bus().write64(reg::CONFIG, 0x33), BUS_OK); + assert_eq!(phys.read64(at).data, 0xaa22_3344_5566_7788, "configuring DRAM keeps the page"); + assert_eq!(phys.banks()[0].read64(at).data, 0, "while it stands, memory is not the page"); + md.startup_tag_store(0); + assert_eq!(phys.read64(at).data, 0, "the window is memory again"); + assert_eq!(phys.banks()[0].read64(at).data, 0); + } + + /// Two processors in their cache-as-memory phase at once each keep their + /// own stack in the window: a store by one is not seen by the other, as + /// each cache holds its own lines (QEMU's per-processor window, a model + /// policy). When one ends its window it reaches memory there while the + /// other still has its page. A window shared between the processors + /// fails the first check; a window ended for both by one processor's + /// sweep fails the last. + #[test] + fn each_processor_keeps_its_own_startup_window() { + let md = fuel(); + assert_eq!(md.md_bus().write64(reg::CONFIG, 0x33), BUS_OK); + let at = 0x10_0040; + let on = |cpu: u8, work: Box u64 + Send>| { + let md = md.clone(); + std::thread::spawn(move || { + crate::cpu::smp::set_current_cpu(cpu); + work(&md) + }) + .join() + .unwrap() + }; + on(0, Box::new(move |md| { md.startup_tag_store(WINDOW_TAGS[0]); md.physical().write64(at, 0xa0); 0 })); + on(1, Box::new(move |md| { md.startup_tag_store(WINDOW_TAGS[0]); md.physical().write64(at, 0xb1); 0 })); + assert_eq!(on(0, Box::new(move |md| md.physical().read64(at).data)), 0xa0); + assert_eq!(on(1, Box::new(move |md| md.physical().read64(at).data)), 0xb1); + assert_eq!(md.physical().banks()[0].read64(at).data, 0, "memory is neither page"); + // CPU 0's final sweep ends its window alone. + on(0, Box::new(|md| { md.startup_tag_store(0); 0 })); + assert!(md.memory.startup_live(), "CPU 1's still stands"); + assert_eq!(on(0, Box::new(move |md| { md.physical().write64(at, 0xc0); md.physical().read64(at).data })), 0xc0); + assert_eq!(md.physical().banks()[0].read64(at).data, 0xc0, "CPU 0 wrote memory"); + assert_eq!(on(1, Box::new(move |md| md.physical().read64(at).data)), 0xb1); + on(1, Box::new(|md| { md.startup_tag_store(0); 0 })); + assert!(!md.memory.startup_live()); + assert_eq!(on(1, Box::new(move |md| md.physical().read64(at).data)), 0xc0); + } + + /// Test patterns, a tag outside the window, and a kernel's invalid tags + /// leave the decode alone; only the window's valid tags start the page. + #[test] + fn other_tags_leave_the_window_to_memory() { + let md = fuel(); + for tag in [0xffff_ffcf, 0xaaaa_aa8a, 0x5555_5545, 0, 0x108c0, 0xff00c0, 0x1000c0 >> 4] { + md.startup_tag_store(tag); + assert!(!md.memory.startup_live(), "{tag:#x}"); + } + assert_eq!(md.physical().read64(0x10_0000).status, BUS_ERR); + // The tag's node bits above the node's 8 GiB do not move the window. + md.startup_tag_store(0x2001_00c0); + assert!(md.memory.startup_live()); + } + + /// The whole PROM pattern: the page starts at the first install, stays + /// through the rest and across a configuration write, and ends at the + /// first invalid tag of the final sweep. + #[test] + fn the_proms_tag_sequence_starts_and_ends_the_page_once() { + let md = fuel(); + let phys = md.physical(); + for tag in [0xffff_ffcf, 0xaaaa_aa8a, 0x5555_5545, 0] { + md.startup_tag_store(tag); + } + assert!(!md.memory.startup_live()); + for tag in WINDOW_TAGS { + md.startup_tag_store(tag); + assert!(md.memory.startup_live()); + } + assert_eq!(phys.write64(0x10_0000, 7), BUS_OK); + md.md_bus().write64(reg::CONFIG, 0x33); + md.soft_reset(); + assert_eq!(phys.read64(0x10_0000).data, 7, "a controller soft reset is not a power-on"); + md.startup_tag_store(0); + assert!(!md.memory.startup_live()); + md.startup_tag_store(0); + assert!(!md.memory.startup_live()); + // A reboot through the PROM tags again, and the cache still holds + // what the page held. + md.startup_tag_store(WINDOW_TAGS[7]); + assert!(md.memory.startup_live()); + assert_eq!(phys.read64(0x10_0000).data, 7); + } + + /// The final sweep stores 2050 invalid tags; the page ends at the first, + /// and the PROM's later stores to the window reach memory, not the page. + #[test] + fn the_page_ends_at_the_first_invalid_tag_of_the_final_sweep() { + let md = configured(); + let phys = md.physical(); + for tag in WINDOW_TAGS { + md.startup_tag_store(tag); + } + assert_eq!(phys.write64(0x10_0000, 5), BUS_OK); + md.startup_tag_store(0); + assert!(!md.memory.startup_live(), "the first invalid store ends it"); + assert_eq!(phys.read64(0x10_0000).data, 0); + assert_eq!(phys.write64(0x10_0000, 6), BUS_OK); + for _ in 1..2050 { + md.startup_tag_store(0); + } + assert!(!md.memory.startup_live()); + assert_eq!(phys.read64(0x10_0000).data, 6, "memory, untouched by the rest of the sweep"); + } + + /// While the page stands its 64 MiB region is out of the direct window, + /// so cached loads and the JIT reach it through the decode; the rest of + /// the region comes back with the page's end. + #[test] + fn the_page_keeps_its_region_out_of_the_direct_window() { + use crate::ppmem::MappedMemory; + let md = configured(); + let space = md.physical().ppmem_space().unwrap(); + assert_ne!(space.mapped_bitmap() & 1, 0, "configured memory is in the window"); + md.startup_tag_store(WINDOW_TAGS[0]); + assert_eq!(space.mapped_bitmap() & 1, 0, "while the page stands"); + assert_ne!(space.mapped_bitmap() & 2, 0, "the next region is not affected"); + md.startup_tag_store(0); + assert_ne!(space.mapped_bitmap() & 1, 0, "and it returns"); + } + + /// Power-on drops the page, zeroed, and its decode. + #[test] + fn power_on_clears_the_page() { + use crate::traits::Resettable; + let md = configured(); + md.startup_tag_store(WINDOW_TAGS[0]); + assert_eq!(md.physical().write64(0x10_0000, 9), BUS_OK); + md.power_on(); + assert!(!md.memory.startup_live()); + assert_eq!(md.physical().read64(0x10_0000).status, BUS_ERR, "no memory until configured"); + md.startup_tag_store(WINDOW_TAGS[0]); + assert_eq!(md.physical().read64(0x10_0000).data, 0, "the page starts zero"); + } + + /// Ordinary CPU and DMA RAM accesses remain available while MD state is locked. + #[test] + fn ordinary_cpu_and_dma_memory_do_not_take_the_md_lock() { + let md = configured(); + let dma = md.dma_view(); + let _state = md.state.lock(); + assert_eq!(md.physical().write64(0x1000, 0x1122334455667788), BUS_OK); + assert_eq!(md.physical().read64(0x1000).data, 0x1122334455667788); + assert_eq!(dma.read64(0x1000).data, 0x1122334455667788); + assert_eq!(dma.write32(0x1004, 0xabcdef01), BUS_OK); + assert_eq!(md.physical().read64(0x1000).data, 0x11223344abcdef01); + assert_eq!(dma.read64(FITTED_BANK).status, BUS_ERR); + } + + /// An ECC injection removes its direct mapping; DMA repair keeps the page slow. + #[test] + fn injected_checks_disable_the_window_and_dma_repair_keeps_the_slow_page() { + use crate::ppmem::MappedMemory; + let md = configured(); + let phys = md.physical(); + let space = phys.ppmem_space().expect("a window"); + assert_eq!(space.mapped_bitmap() & 3, 3); + assert!(phys.mem_ptr(0).is_some()); + assert_eq!(md.ecc_bus().write8(7, 0x13), BUS_OK); + assert_eq!(space.mapped_bitmap() & 3, 2, "the full 64 MiB region loses its direct bit"); + assert!(phys.mem_ptr(0).is_none()); + let slow = phys.entry_at(0); + assert!(!crate::ppmem::devmap::entry_eq(slow, phys.entry_at(0x1000))); + let dma = md.dma_view(); + let repaired = std::thread::spawn(move || dma.read64(0)).join().unwrap(); + assert_eq!(repaired.status, BUS_OK); + assert_eq!(repaired.data, 1); + assert!(md.state.lock().ecc.is_empty()); + assert_eq!(space.mapped_bitmap() & 3, 2, "a DMA thread never republishes the CPU bitmap"); + assert!(crate::ppmem::devmap::entry_eq(slow, phys.entry_at(0))); + // Adjacent pages still use the ordinary bank entry, even though + // the region no longer has a direct window mapping. + { + let _state = md.state.lock(); + assert_eq!(phys.write64(0x1000, 9), BUS_OK); + assert_eq!(phys.read64(0x1000).data, 9); + } + md.soft_reset(); + assert_eq!(space.mapped_bitmap() & 3, 3); + assert!(phys.mem_ptr(0).is_some()); + assert_eq!(phys.read64(0).data, 1); + } + + /// Directory-marked pages follow their physical bank after a configuration swap. + #[test] + fn marked_directory_pages_follow_their_bank_when_swizzled() { + use crate::ppmem::MappedMemory; + let md = configured(); + let phys = md.physical(); + let space = phys.ppmem_space().expect("a window"); + md.md_bus().write64(reg::DIRECTORY_CONFIG, 2); + md.directory_bus().write64(0, 0x001e0001); + assert_eq!(space.mapped_bitmap() & 1, 0); + assert!(phys.mem_ptr(0).is_none()); + md.md_bus().write64(reg::CONFIG, 0x33 | (1 << 32)); + assert_ne!(space.mapped_bitmap() & 1, 0, "the other bank now answers at zero"); + assert_eq!(space.mapped_bitmap() & (1 << 16), 0); + assert!(phys.mem_ptr(BANK_SPAN).is_none()); + md.md_bus().write64(reg::DIRECTORY_CONFIG, 0); + assert_eq!(phys.read64(BANK_SPAN).status, BUS_OK); + assert_ne!(md.md_bus().read64(reg::DIRECTORY_ERROR).data & (1 << 61), 0); + md.rebuild_memory(); + assert!(phys.mem_ptr(BANK_SPAN).is_some()); + md.power_on(); + assert_eq!(space.mapped_bitmap(), 0); + assert_eq!(phys.read64(BANK_SPAN).status, BUS_ERR); + } + + /// Guest code must not keep a compiled page after its backing bank changes. + /// page_for deliberately retains its generation pointer. A remap must + /// invalidate that page and subsequent writes to its new bank must keep + /// invalidating the same pointer, including after swapping back. + #[cfg(feature = "jitv2")] + #[test] + fn a_claimed_jit_page_tracks_physical_ownership_across_bank_swaps() { + use crate::cpu::jitv2::jitv2::Jitv2; + let md = configured(); + let dma = md.dma_view(); + let mut jit = Jitv2::new(4); + let slot = jit.page_for(0, 0, dma.as_ref(), false).unwrap(); + let page = jit.page_ptr(slot); + let generation = || unsafe { (*page).current_gen() }; + let pointer = dma.gen_ptr(0); + let mut entries = [0u64; crate::cpu::jitv2::jitv2::BITMAP_WORDS]; + entries[0] = 1; + // Exercise the JIT publication/invalidation protocol without + // executing a synthetic function body. + let code = std::ptr::NonNull::<()>::dangling().as_ptr() as *const (); + assert!(unsafe { (*page).publish(&entries, code, generation(), 1, 0, false) }); + assert!(unsafe { (*page).is_runnable(0) }); + let first = generation(); + md.md_bus().write64(reg::CONFIG, 0x33 | (1 << 32)); + assert_eq!(jit.page_for(0, 0, dma.as_ref(), false), Some(slot)); + assert_eq!(dma.gen_ptr(0), pointer); + assert!(generation() > first); + assert!(!unsafe { (*page).is_runnable(0) }); + let swapped = generation(); + assert!(unsafe { (*page).publish(&entries, code, swapped, 1, 0, false) }); + assert!(unsafe { (*page).is_runnable(0) }); + assert_eq!(dma.write64(0, 0x11223344), BUS_OK); + assert!(generation() > swapped); + assert!(!unsafe { (*page).is_runnable(0) }, "code published after a swap sees new-bank writes"); + assert_eq!(md.physical().banks()[1].read64(0).data, 0x11223344); + let written = generation(); + md.md_bus().write64(reg::CONFIG, 0x33); + assert!(generation() > written, "swapping back cannot lower the logical generation"); + let restored = generation(); + md.uncached_view().write64(0, 0x55667788); + assert!(generation() > restored); + let before_fetchop = generation(); + md.fetchop_bus().write64(0, 0); + assert!(generation() > before_fetchop); + md.ecc_bus().write8(7, 0x13); + let before_repair = generation(); + assert_eq!(dma.read64(0).data, 1); + assert!(generation() > before_repair, "ECC repair invalidates logical code pages"); + md.md_bus().write64(reg::CONFIG, 0x13); + md.md_bus().write64(0x20038, 0x80); + let before_zero = generation(); + md.md_bus().write64(reg::SELFTEST_COMMAND, 1); + assert!(generation() > before_zero, "zero self-test invalidates logical code pages"); + let before_disable = generation(); + md.power_on(); + assert!(generation() > before_disable); + assert_eq!(dma.gen_ptr(0), pointer, "a disabled bank keeps its logical counter alive"); + } + + /// The Tezro's banks, in MiB from locational bank 0. + const TEZRO_BANKS: [u32; 6] = [1024, 1024, 512, 512, 512, 512]; + /// Banks 0 and 1 at size code 5 (1 GiB), the other four at code 4 (512 MiB). + const TEZRO_CONFIG: u64 = 0x53 | (0x43 << 8) | (0x43 << 16); + fn tezro() -> Arc { + let md = BedrockMd::new(&TEZRO_BANKS).unwrap(); + assert_eq!(md.md_bus().write64(reg::CONFIG, TEZRO_CONFIG), BUS_OK); + md + } + + /// Each Tezro bank answers in its own 1 GiB slot at its own size: a + /// 1 GiB bank fills its slot, a 512 MiB bank stops halfway, and the + /// slots above bank 5 answer nothing. Banks 4 and 5 sit above 4 GiB, + /// where the CPU's direct window does not reach, and still answer + /// through the bus. Follows the qtest memory-banks and + /// memory-size-boundaries cases in tests/qtest/sn1-bedrock-md-test.c. + #[test] + fn every_tezro_bank_answers_at_its_own_size_including_above_4_gib() { + use crate::ppmem::MappedMemory; + let md = tezro(); + let bus = md.physical(); + let sizes = [1024u64, 1024, 512, 512, 512, 512]; + for (slot, mib) in sizes.iter().enumerate() { + let base = slot as u64 * BANK_SPAN; + let size = mib << 20; + assert_eq!(bus.write64(base, 0x1111 * (slot as u64 + 1)), BUS_OK, "slot {slot}"); + assert_eq!(bus.write64(base + size - 8, 0xe0 + slot as u64), BUS_OK, "slot {slot} top"); + if size < BANK_SPAN { + assert_eq!(bus.read64(base + size).status, BUS_ERR, "slot {slot} past its bank"); + } + } + for slot in 0..6u64 { + assert_eq!(bus.read64(slot * BANK_SPAN).data, 0x1111 * (slot + 1), "slot {slot}"); + let top = (sizes[slot as usize] << 20) - 8; + assert_eq!(bus.read64(slot * BANK_SPAN + top).data, 0xe0 + slot, "slot {slot} top"); + } + for slot in [6, 7] { + assert_eq!(bus.read64(slot * BANK_SPAN).status, BUS_ERR, "slot {slot} holds no bank"); + } + assert_eq!(bus.banks()[4].read64(0).data, 0x1111 * 5, "bank 4 holds what slot 4 wrote"); + assert_eq!(bus.banks()[5].read64(0).data, 0x1111 * 6); + + // The window covers the first 4 GiB: 1 GiB, 1 GiB, 512 MiB, 512 MiB. + let space = bus.ppmem_space().expect("a window"); + assert_eq!(space.mapped_bitmap(), 0xffff | (0xffff << 16) | (0xff << 32) | (0xff << 48)); + assert!(!crate::ppmem::bitmap_maps(space.mapped_bitmap(), 4 * BANK_SPAN)); + assert!(!crate::ppmem::bitmap_maps(space.mapped_bitmap(), 5 * BANK_SPAN)); + assert_eq!(md.dma_view().read64(5 * BANK_SPAN).data, 0x1111 * 6, "DMA reaches it too"); + } + + /// A processor's SC to a bank above 4 GiB, outside its direct window, + /// swaps through the DMA view in one step and leaves a failed compare + /// alone; a successful one advances the page's logical counter. + #[test] + fn a_compare_and_swap_above_4_gib_is_one_step() { + let md = tezro(); + let view = md.dma_view(); + let at = 5 * BANK_SPAN + 0x40; + assert_eq!(view.write64(at, 7), BUS_OK); + #[cfg(feature = "jitv2")] + let before = unsafe { (*view.gen_ptr(at)).load(Ordering::Relaxed) }; + assert_eq!(view.cas64(at, 6, 9), Some(false)); + assert_eq!(view.read64(at).data, 7); + assert_eq!(view.cas64(at, 7, 9), Some(true)); + assert_eq!(view.read64(at).data, 9); + assert_eq!(view.cas32(at + 4, 9, 3), Some(true)); + assert_eq!(view.read64(at).data, 3); + #[cfg(feature = "jitv2")] + assert!(unsafe { (*view.gen_ptr(at)).load(Ordering::Relaxed) } > before); + assert_eq!(view.cas64(7 * BANK_SPAN, 0, 1), None, "no bank there"); + } + + /// Selector 1 swaps the two banks of each pair, so the 512 MiB banks 4 and 5 + /// trade places above 4 GiB. Follows memory-selector-high-bank. + #[test] + fn the_bank_select_swaps_banks_above_4_gib() { + let md = tezro(); + let bus = md.physical(); + assert_eq!(bus.write64(4 * BANK_SPAN + 0x28, 0xfedc_ba98_7654_3210), BUS_OK); + assert_eq!(bus.read64(5 * BANK_SPAN + 0x28).data, 0); + assert_eq!(md.md_bus().write64(reg::CONFIG, TEZRO_CONFIG | (1 << 32)), BUS_OK); + assert_eq!(bus.read64(5 * BANK_SPAN + 0x28).data, 0xfedc_ba98_7654_3210); + assert_eq!(bus.read64(4 * BANK_SPAN + 0x28).data, 0); + // Banks 0 and 1 swap too: the 1 GiB banks have different sizes only + // if the board says so, so give bank 1 a distinct value. + assert_eq!(bus.write64(0, 0xaa), BUS_OK); + assert_eq!(bus.write64(BANK_SPAN, 0xbb), BUS_OK); + assert_eq!(md.md_bus().write64(reg::CONFIG, TEZRO_CONFIG), BUS_OK); + assert_eq!(bus.read64(0).data, 0xbb); + assert_eq!(bus.read64(BANK_SPAN).data, 0xaa); + } + + /// A 1 GiB population is wholly inside ppmem's direct window: all 16 of its + /// 64 MiB regions are mapped, with none left to the decode table. The 4 GiB + /// default maps only what is below 4 GiB, and its banks 4 and 5 answer + /// through the table. + #[test] + fn a_1_gib_population_is_directly_mapped_and_the_default_4_gib_one_is_not_wholly() { + use super::super::bedrock_mem::MemoryPopulation; + use crate::ppmem::MappedMemory; + let md = BedrockMd::new(&MemoryPopulation::one_gib().banks_mib()).unwrap(); + md.md_bus().write64(reg::CONFIG, 0x43 | (0x43 << 8)); + let bus = md.physical(); + let space = bus.ppmem_space().expect("a window"); + assert_eq!(space.mapped_bitmap(), 0xff | (0xff << 16)); + assert_eq!(bus.write64(BANK_SPAN + (512 << 20) - 8, 7), BUS_OK); + assert_eq!(bus.read64(BANK_SPAN + (512 << 20) - 8).data, 7); + assert_eq!(bus.read64(512 << 20).status, BUS_ERR, "bank 0 ends at 512 MiB"); + + let md = BedrockMd::new(&MemoryPopulation::default().banks_mib()).unwrap(); + assert_eq!(MemoryPopulation::default(), MemoryPopulation::four_gib()); + md.md_bus().write64(reg::CONFIG, TEZRO_CONFIG); + let space = md.physical().ppmem_space().expect("a window"); + assert!(!crate::ppmem::bitmap_maps(space.mapped_bitmap(), 4 * BANK_SPAN)); + assert_eq!(md.physical().write64(4 * BANK_SPAN, 9), BUS_OK, "still reachable"); + } + + /// Directory and ECC state is keyed per bank, so banks of different sizes + /// never share an entry. Follows directory-bank-translation. + #[test] + fn directory_entries_of_different_sized_banks_stay_apart() { + let md = tezro(); + md.md_bus().write64(reg::DIRECTORY_CONFIG, 2); + let dir = md.directory_bus(); + // Software bank n's carriers start at n * 128 MiB. + let at = |bank: u64, offset: u64| bank * (BANK_SPAN / 8) + offset; + for bank in 0..6 { + assert_eq!(dir.write64(at(bank, 0x20), 0x1000 + bank), BUS_OK, "bank {bank}"); + } + for bank in 0..6 { + assert_eq!(dir.read64(at(bank, 0x20)).data, 0x1000 + bank, "bank {bank}"); + } + // A 512 MiB bank's directory ends at its size code's reach. + assert_eq!(dir.read64(at(2, 0x20) + (BANK_SPAN / 16)).status, BUS_ERR); + assert_eq!(dir.read64(at(6, 0x20)).status, BUS_ERR, "no bank 6"); + } + + /// A fitting that covers the bank under test zero-fills it, whichever slot + /// the bank is in; a bank that is not fitted fails. Follows the + /// selftest-bank-remap and origin300-selftest-fill cases. + #[test] + fn selftest_fills_a_tezro_bank_above_4_gib_and_fails_a_missing_bank() { + let md = BedrockMd::new(&TEZRO_BANKS).unwrap(); + // Every pair at size code 1 (64 MiB): small enough to fill in a test. + let regs = md.md_bus(); + regs.write64(reg::CONFIG, 0x13 | (0x13 << 8) | (0x13 << 16) | (0x13 << 24)); + let bus = md.physical(); + bus.write64(4 * BANK_SPAN + 0x40, 0x5555); + regs.write64(0x20038, 0x80); + regs.write64(0x20030, (4 << 4) | 1); + md.advance_ns(1000); + assert_eq!(regs.read64(0x20048).data, 3, "bank 4 is fitted"); + assert_eq!(bus.read64(4 * BANK_SPAN + 0x40).data, 0, "and zeroed"); + regs.write64(0x20030, (6 << 4) | 1); + md.advance_ns(1000); + assert_eq!(regs.read64(0x20048).data, 2, "bank 6 is not fitted"); + } + + /// The board names what it fits; nothing else is accepted. A bank is a + /// whole number of direct-window regions and at most the largest size + /// code names. + #[test] + fn a_board_may_only_fit_banks_the_controller_can_decode() { + assert!(BedrockMd::new(&FUEL_BANKS).is_ok()); + assert!(BedrockMd::new(&TEZRO_BANKS).is_ok()); + for banks in [&[][..], &[0, 256], &[2048], &[100], &[64; 9]] { + assert!(BedrockMd::new(banks).is_err(), "{banks:?}"); + } + } + + /// Where RAM can be, for a board's cached view: the swizzle permutes slots + /// within groups of four, so a slot reaches as far as the largest bank of + /// its group. + #[test] + fn the_ram_extent_reaches_as_far_as_the_largest_bank_of_a_slot_group() { + let fuel = RamExtent::new(&FUEL_BANKS); + for slot in 0..4 { + assert!(fuel.contains(slot * BANK_SPAN + FITTED_BANK - 8), "slot {slot}"); + assert!(!fuel.contains(slot * BANK_SPAN + FITTED_BANK), "slot {slot}"); + } + assert!(!fuel.contains(4 * BANK_SPAN)); + let tezro = RamExtent::new(&TEZRO_BANKS); + assert!(tezro.contains(3 * BANK_SPAN + BANK_SPAN - 8), "slot 3 may hold a 1 GiB bank"); + assert!(tezro.contains(5 * BANK_SPAN + (512 << 20) - 8)); + assert!(!tezro.contains(5 * BANK_SPAN + (512 << 20))); + assert!(!tezro.contains(8 * BANK_SPAN)); + assert_eq!(tezro, BedrockMd::new(&TEZRO_BANKS).unwrap().ram_extent()); + } + + /// Code pages above 4 GiB have stable logical generation counters like + /// the rest, and a write bumps them. + #[cfg(feature = "jitv2")] + #[test] + fn code_pages_above_4_gib_keep_a_logical_generation() { + let md = tezro(); + let bus = md.physical(); + let above = 4 * BANK_SPAN + 0x2000; + let pointer = md.dma_view().gen_ptr(above); + assert!(!pointer.is_null()); + assert_ne!(pointer, md.dma_view().gen_ptr(5 * BANK_SPAN + 0x2000)); + assert!(md.dma_view().gen_ptr(8 * BANK_SPAN).is_null(), "past the node's 8 GiB"); + assert!(md.dma_view().gen_ptr(4 * BANK_SPAN + (512 << 20)).is_null(), "past the largest bank there"); + let before = unsafe { (*pointer).load(Ordering::Relaxed) }; + assert_eq!(md.dma_view().write64(above, 1), BUS_OK); + assert!(unsafe { (*pointer).load(Ordering::Relaxed) } > before); + md.md_bus().write64(reg::CONFIG, TEZRO_CONFIG | (1 << 32)); + assert_eq!(md.dma_view().gen_ptr(above), pointer, "a bank swap keeps the counter"); + let _ = bus; + } + + /// The ECC carrier byte for the doubleword at `address`, as the guest + /// addresses it. + fn ecc_carrier(address: u64) -> u64 { + address / BANK_SPAN * (BANK_SPAN / 8) + ((address % BANK_SPAN) / 8 ^ 7) + } + + /// The cost counters now: host mappings, decode-table rewrites and + /// generation counters. + fn counted(md: &Arc) -> [u64; 3] { + md.memory.cost.snapshot() + } + /// The counters one bumped 4 KiB page, and one whole 64 MiB region, cost: + /// the logical counter and the bank's, which only a jitv2 build keeps. + const PAGE_BUMPS: u64 = if cfg!(feature = "jitv2") { 2 } else { 0 }; + const REGION_BUMPS: u64 = PAGE_BUMPS * (64 << 20) / 4096; + fn mark(md: &Arc, address: u64) { + assert_eq!(md.ecc_bus().write8(ecc_carrier(address), 0x13), BUS_OK); + } + /// A doubleword of zero has a check byte of zero, so writing that lifts + /// a mark, and writing 0x13 puts one there. + fn lift(md: &Arc, address: u64) { + assert_eq!(md.ecc_bus().write8(ecc_carrier(address), 0), BUS_OK); + } + fn spent(md: &Arc, work: impl FnOnce()) -> [u64; 3] { + let before = counted(md); + work(); + let after = counted(md); + [after[0] - before[0], after[1] - before[1], after[2] - before[2]] + } + + /// What marking and lifting checked pages costs in host mappings, + /// decode-table rewrites and generation counters. Taking a region off the + /// direct path and putting it back maps nothing: the pages stay mapped + /// and only the bitmap changes. The PROM's memory test + /// marks and lifts the same few hundred doublewords thousands of times + /// (the QEMU model counted 6630 of each for 310 doublewords, and + /// rebuilding the map each time took the test from ten seconds to + /// minutes), so what one costs is what the test costs. Counts, not + /// seconds, so the test can pin them. Before this was incremental each of + /// these was a full rebuild: 15 mappings, 3 or 2 table rewrites and + /// 393,216 counters on the Fuel; 95, 7 or 6 and 2,621,440 on the Tezro. + /// + /// Follows the qtest ecc-lifted-mark-keeps-overlay case in + /// tests/qtest/sn1-bedrock-md-test.c (QEMU commit 4f0f6072dc0b), which + /// pins the same thing in the flat view: a lifted mark's overlay stays, + /// a second mark there is not a map change, and the idle ones go + /// together. + #[test] + fn a_checked_page_costs_a_page_not_a_rebuild() { + let (fuel, tezro) = (configured(), tezro()); + for (md, base, windowed) in [(&fuel, 0, true), (&tezro, 0, true), (&tezro, 4 * BANK_SPAN, false)] { + let (first, second) = (base + 0x5000, base + 0x6000); + // The first page in a region takes the region off the direct + // window and drops code compiled from any of it; the next page + // in that region is one more table entry. Above the window there + // is no region to take. + let cost = spent(md, || mark(md, first)); + assert_eq!(cost, if windowed { [0, 1, REGION_BUMPS] } else { [0, 1, PAGE_BUMPS] }, "first page at {base:#x}"); + assert_eq!(spent(md, || mark(md, second)), [0, 1, PAGE_BUMPS], "second page at {base:#x}"); + // Lifting costs nothing: both pages stay on the MD endpoint. + assert_eq!(spent(md, || { lift(md, first); lift(md, second) }), [0, 0, 0], "lifts at {base:#x}"); + // Marking a lifted page again is not a map change either. + assert_eq!(spent(md, || mark(md, first)), [0, 0, 0], "re-mark at {base:#x}"); + // The retirement returns what is lifted, in one batch, and + // leaves the marked page where it is. + assert_eq!(spent(md, || md.retire_lifted_marks()), [0, 1, PAGE_BUMPS], "retire at {base:#x}"); + lift(md, first); + // The region comes back to the direct path with no host mapping: + // withdrawing it left the pages mapped. + assert_eq!(spent(md, || md.retire_lifted_marks()), [0, 1, PAGE_BUMPS], "last retire at {base:#x}"); + } + } + + fn counter(md: &Arc, address: u64) -> u64 { + #[cfg(feature = "jitv2")] + return unsafe { (*md.memory.gen_ptr(address)).load(Ordering::Relaxed) }; + #[cfg(not(feature = "jitv2"))] + { let _ = (md, address); 0 } + } + + /// With the controller's clock under the test's hand. + fn clocked() -> (Arc, Arc) { + let now = Arc::new(AtomicU64::new(1)); + let hand = now.clone(); + let md = BedrockMd::with_clock(&FUEL_BANKS, Arc::new(move || hand.load(Ordering::SeqCst))).unwrap(); + assert_eq!(md.md_bus().write64(reg::CONFIG, 0x33), BUS_OK); + (md, now) + } + + /// Lifting a mark keeps its page on the MD endpoint, which answers as + /// the bank would: the lifted doubleword reads and writes plainly, a + /// second mark there is checked again, and the page goes back to the bank + /// together with the other idle ones, a second after the first was seen + /// idle, or at once on reset. Follows the qtest + /// ecc-lifted-mark-keeps-overlay case in + /// tests/qtest/sn1-bedrock-md-test.c (QEMU commit 4f0f6072dc0b). + #[test] + fn a_lifted_mark_keeps_its_page_on_the_md_endpoint_until_it_idles() { + use crate::ppmem::MappedMemory; + let (md, now) = clocked(); + let (phys, bus) = (md.physical(), md.ecc_bus()); + let space = phys.ppmem_space().expect("a window"); + let (address, data) = (0x5000, 0x0123_4567_89ab_cdef); + let on_bank = |md: &Arc, address| { + crate::ppmem::devmap::entry_eq(md.physical().entry_at(address), md.physical().entry_at(0x20_0000)) + }; + assert_eq!(phys.write64(address, data), BUS_OK); + assert!(on_bank(&md, address)); + assert_eq!(bus.write8(ecc_carrier(address), ecc_compute(data) ^ ECC_COLUMNS[5]), BUS_OK); + assert!(!on_bank(&md, address) && space.mapped_bitmap() & 1 == 0); + + // A write lifts the mark; the page stays on the endpoint and reads as memory. + assert_eq!(phys.write64(address, data), BUS_OK); + assert!(!on_bank(&md, address) && space.mapped_bitmap() & 1 == 0); + assert_eq!(phys.read64(address).data, data); + assert_eq!(phys.read32(address + 4).data, data as u32); + assert_eq!(phys.write32(address, 0x7654_3210), BUS_OK); + assert_eq!(phys.read64(address).data, 0x7654_3210_89ab_cdef); + assert_eq!(phys.write64(address, data), BUS_OK); + assert_eq!(bus.read8(ecc_carrier(address)).data, ecc_compute(data)); + assert_eq!(md.md_bus().read64(reg::MEMORY_ERROR).data, 0); + + // Marked again, the same doubleword is checked again. + assert_eq!(bus.write8(ecc_carrier(address), ecc_compute(data) ^ ECC_COLUMNS[6]), BUS_OK); + assert_eq!(phys.read64(address).data, data ^ (1 << 6)); + assert_ne!(md.md_bus().read64(reg::MEMORY_ERROR).data & (1 << 52), 0); + assert!(md.state.lock().ecc.is_empty(), "the correcting read lifted it"); + + // Idling a second retires it, on the next publication after that. + let other = |n: u64| 0x1_0000 + n * 0x1000; + assert_eq!(phys.write64(other(0), 0), BUS_OK); + mark(&md, other(0)); + now.store(1 + RETIRED_IDLE_NS - 1, Ordering::SeqCst); + mark(&md, other(1)); + assert!(!on_bank(&md, address), "not yet a second"); + now.store(1 + RETIRED_IDLE_NS, Ordering::SeqCst); + mark(&md, other(2)); + assert!(on_bank(&md, address), "a second"); + assert_eq!(phys.read64(address).data, data ^ (1 << 6)); + assert!(!on_bank(&md, other(0)) && !on_bank(&md, other(2)), "the marked pages stay checked"); + + // Reset takes a kept page back without waiting. + lift(&md, other(0)); + assert_eq!(phys.write64(other(0), 0), BUS_OK); + assert!(!on_bank(&md, other(0))); + md.soft_reset(); + assert!(on_bank(&md, other(0)) && on_bank(&md, other(2))); + assert_eq!(space.mapped_bitmap() & 1, 1); + } + + /// A page kept after its mark is lifted still checks and rewrites a + /// marked directory line in it, as the line's own placement would. + /// Follows the qtest ecc-lifted-mark-over-marked-line case in + /// tests/qtest/sn1-bedrock-md-test.c (QEMU commit 4f0f6072dc0b). + #[test] + fn a_kept_page_checks_a_directory_line_marked_after_the_lift() { + let md = configured(); + let phys = md.physical(); + mark(&md, 0x48); + lift(&md, 0x48); + md.md_bus().write64(reg::DIRECTORY_CONFIG, 2); + let cost = spent(&md, || { md.directory_bus().write64(0, 0x001e0001); }); + assert_eq!(cost, [0, 0, 0], "the page is already on the endpoint"); + md.md_bus().write64(reg::DIRECTORY_CONFIG, 0); + assert_eq!(phys.read64(0x48).status, BUS_OK); + assert_ne!(md.md_bus().read64(reg::DIRECTORY_ERROR).data & (1 << 61), 0, "the read checked the line"); + // Marked again, a write through the kept page rewrites the entry. + md.md_bus().write64(reg::DIRECTORY_CONFIG, 2); + md.directory_bus().write64(0, 0x001e0001); + assert!(!md.state.lock().marked_directory.is_empty()); + assert_eq!(phys.write64(0x48, 0x22), BUS_OK); + assert!(md.state.lock().marked_directory.is_empty()); + assert_eq!(phys.read64(0x48).data, 0x22); + } + + /// The PROM's memory test marks and lifts the same few hundred + /// doublewords over and over; each page is placed once however often. + /// The QEMU model counted 6630 placements for 310 doublewords before it + /// kept lifted overlays. + #[test] + fn marking_the_same_pages_over_and_over_places_each_once() { + let md = configured(); + let pages: Vec = (0..300).map(|i| i * 0x9_0000 + 0x1000).collect(); + let regions: HashSet = pages.iter().map(|page| page >> 26).collect(); + let cost = spent(&md, || { + for _ in 0..20 { + pages.iter().for_each(|&page| mark(&md, page)); + pages.iter().for_each(|&page| lift(&md, page)); + } + }); + let regions = regions.len() as u64; + assert_eq!(cost, [0, 300, regions * REGION_BUMPS + (300 - regions) * PAGE_BUMPS]); + } + + /// Lifted pages pile up only so far: the first mark after + /// `RETIRED_LIMIT` have been lifted returns them all in one batch. + #[test] + fn many_lifted_pages_are_retired_in_one_batch() { + let md = configured(); + let pages: Vec = (0..RETIRED_LIMIT as u64).map(|i| 0x1000 * i).collect(); + pages.iter().for_each(|&page| mark(&md, page)); + pages.iter().for_each(|&page| lift(&md, page)); + // One more page, in another region, makes the 1024 lifted pages due. + let cost = spent(&md, || mark(&md, 0x400_0000 + 0x1000)); + let retired = RETIRED_LIMIT as u64; + assert_eq!(cost, [0, 1 + retired, REGION_BUMPS + PAGE_BUMPS * retired]); + assert!(crate::ppmem::devmap::entry_eq(md.physical().entry_at(0x5000), md.physical().entry_at(0x20_0000))); + } + + /// A zero self-test of a bank clears its marks, and so returns the pages + /// they held. + #[test] + fn a_zero_self_test_returns_the_banks_lifted_pages() { + use crate::ppmem::MappedMemory; + let md = configured(); + let space = md.physical().ppmem_space().expect("a window"); + mark(&md, 0x5000); + assert_eq!(space.mapped_bitmap() & 1, 0); + md.md_bus().write64(0x20038, 0x80); + md.md_bus().write64(reg::SELFTEST_COMMAND, 1); + assert_eq!(space.mapped_bitmap() & 1, 1); + assert!(crate::ppmem::devmap::entry_eq(md.physical().entry_at(0x5000), md.physical().entry_at(0x20_0000))); + } + + /// A guard sees only the rebuilds of the whole map: the configuration + /// write, a rebuild, resets and a startup-window change. Placing and + /// retiring a page, which map and scrub nothing, run without it. + #[test] + fn only_whole_map_rebuilds_enter_the_remap_guard() { + let md = BedrockMd::new(&FUEL_BANKS).unwrap(); + let entered = Arc::new(AtomicU64::new(0)); + let seen = entered.clone(); + md.set_remap_guard(Arc::new(move |work: &mut dyn FnMut()| { + seen.fetch_add(1, Ordering::SeqCst); + work(); + })); + let entries = || entered.load(Ordering::SeqCst); + assert_eq!(md.md_bus().write64(reg::CONFIG, 0x33), BUS_OK); + assert_eq!(entries(), 1); + mark(&md, 0x5000); + lift(&md, 0x5000); + md.retire_lifted_marks(); + assert_eq!(entries(), 1, "marks and retirements are not rebuilds"); + md.rebuild_memory(); + md.soft_reset(); + md.startup_tag_store(WINDOW_TAGS[0]); + md.power_on(); + assert_eq!(entries(), 5); + md.startup_tag_store(0xffff_ffcf); + assert_eq!(entries(), 5, "a tag that changes nothing stops nobody"); + } + + /// A page that is marked, or its region's direct path, drops the code + /// compiled from it; a page elsewhere keeps its counter, and lifting + /// changes nothing until the page is returned. + #[test] + fn marking_a_page_advances_the_counters_of_what_left_the_direct_path() { + let md = configured(); + let (page, neighbour, elsewhere) = (0x5000, 0x9000, 0x400_0000 + 0x5000); + let seen = |md: &Arc| [page, neighbour, elsewhere].map(|a| counter(md, a)); + let start = seen(&md); + #[cfg(feature = "jitv2")] + let bank = |offset: u64| unsafe { (*md.physical().banks()[0].gen_ptr(offset)).load(Ordering::Relaxed) }; + #[cfg(feature = "jitv2")] + let banks = [page, neighbour, elsewhere].map(bank); + mark(&md, page); + let marked = seen(&md); + let step = |a: [u64; 3], b: [u64; 3]| [b[0] - a[0], b[1] - a[1], b[2] - a[2]]; + let jit = u64::from(cfg!(feature = "jitv2")); + assert_eq!(step(start, marked), [jit, jit, 0], "the page and its region"); + #[cfg(feature = "jitv2")] + { + // The bank's own counters follow, as a full remap would bump them. + let now = [page, neighbour, elsewhere].map(bank); + assert!(now[0] > banks[0] && now[1] > banks[1] && now[2] == banks[2]); + } + mark(&md, neighbour); + assert_eq!(step(marked, seen(&md)), [0, jit, 0], "only the second page"); + lift(&md, page); + lift(&md, neighbour); + assert_eq!(step(marked, seen(&md)), [0, jit, 0], "lifting changes nothing"); + md.retire_lifted_marks(); + assert_eq!(step(marked, seen(&md)), [jit, 2 * jit, 0], "returning a page advances it"); + } + + /// A read-only 4 MiB window standing for the reset PROM, whose first + /// word reads 0x0f1e2d3c4b5a6978. + struct Prom; + impl BusDevice for Prom { + fn read64(&self, a: u64) -> BusRead64 { + if a == 0 { BusRead64::ok(0x0f1e_2d3c_4b5a_6978) } else { BusRead64::ok(0) } + } + fn write64(&self, _: u64, _: u64) -> u32 { BUS_ERR } + } + + /// The Tezro's memory wired as the board wires it: the processor bus with + /// uncached RAM views, the cached view, and the reset PROM mapped over the + /// cached plane where the bank leaves room. + fn board(md: &Arc) -> (crate::ip35::Ip35Bus, Arc) { + let mut bus = crate::ip35::Ip35Bus::new(md.physical().clone()); + #[cfg(feature = "jitv2")] + bus.set_ram_generations(md.dma_view()); + md.rebuild_memory(); + bus.map(3, 0, 1 << 33, md.uncached_view()); + bus.map(0, 0, 0x1000_0000, md.uncached_view()); + bus.map(0, 0x1fc0_0000, 0x40_0000, Arc::new(Prom)); + crate::ip35::map_device(md.physical(), 0x1fc0_0000, 0x40_0000, Arc::new(Prom), "test reset PROM"); + let cached = bus.cached_view(md.dma_view(), md.ram_extent()); + (bus, cached) + } + + /// The Tezro's default population, wired and configured as the board does. + fn default_tezro() -> (Arc, crate::ip35::Ip35Bus, Arc) { + let md = BedrockMd::new(&super::super::bedrock_mem::MemoryPopulation::default().banks_mib()).unwrap(); + assert_eq!(md.md_bus().write64(reg::CONFIG, TEZRO_CONFIG), BUS_OK); + let (bus, cached) = board(&md); + (md, bus, cached) + } + /// Asserts a code page's counter moved, in builds that keep counters. + fn assert_advanced(what: &str, before: u64, after: u64) { + if cfg!(feature = "jitv2") { + assert!(after > before, "{what}: {before} -> {after}"); + } + } + + /// A processor store to banks 4 and 5, above the 4 GiB the direct window + /// reaches, goes through the board's cached view or an uncached one into + /// the bank behind the slot and advances that slot's logical counter, + /// and no other slot's. The JIT page lookup and the cache's own bump read + /// the same counter. + #[test] + fn processor_stores_to_banks_4_and_5_advance_their_slot_counters() { + let (md, bus, cached) = default_tezro(); + let uncached = |address: u64| (3u64 << 57) | address; + for (slot, other) in [(4u64, 5u64), (5, 4)] { + let at = slot * BANK_SPAN + 0x2000; + let elsewhere = counter(&md, other * BANK_SPAN + 0x2000); + #[cfg(feature = "jitv2")] + { + assert_eq!(bus.gen_ptr(at), md.memory.gen_ptr(at)); + assert_eq!(cached.gen_ptr(at), md.memory.gen_ptr(at)); + assert_eq!(bus.gen_ptr(uncached(at)), md.memory.gen_ptr(at)); + } + let mut last = counter(&md, at); + let mut stored = |what: &str, status: u32| { + assert_eq!(status, BUS_OK, "{what}"); + let now = counter(&md, at); + assert_advanced(what, last, now); + last = now; + }; + stored("cached store", cached.write64(at, 0x1111)); + stored("cached block store", cached.write_block(at + 8, &[0x2222, 0x3333])); + stored("cached masked store", cached.write64_masked(at + 0x20, 0x4444, 0xff00)); + stored("cached halfword store", cached.write16(at + 0x30, 0x5555)); + stored("uncached store", bus.write64(uncached(at + 0x40), 0x6666)); + stored("uncached byte store", bus.write8(uncached(at + 0x48), 0x77)); + let bank = &md.physical().banks()[slot as usize]; + let word = |offset: u64| bank.read64(0x2000 + offset).data; + assert_eq!([word(0), word(8), word(0x10), word(0x20), word(0x40)], + [0x1111, 0x2222, 0x3333, 0x4400, 0x6666], "slot {slot}"); + assert_eq!(counter(&md, other * BANK_SPAN + 0x2000), elsewhere, "slot {other} untouched"); + } + } + + /// A checked page above 4 GiB keeps its logical counter honest: marking + /// it advances the counter, a store through the page's slow route does + /// too, and so does the store once the page is back on its bank. + #[test] + fn a_checked_page_above_4_gib_advances_its_counter() { + let (md, _bus, cached) = default_tezro(); + let (page, address) = (4 * BANK_SPAN + 0x6000, 4 * BANK_SPAN + 0x6008); + let untouched = counter(&md, 4 * BANK_SPAN + 0x8000); + let start = counter(&md, page); + mark(&md, page); + let marked = counter(&md, page); + assert_advanced("mark", start, marked); + assert!(crate::ppmem::devmap::entry_eq(md.physical().entry_at(address), md.physical().entry_at(page))); + assert!(!crate::ppmem::devmap::entry_eq(md.physical().entry_at(page), md.physical().entry_at(page + 0x2000))); + assert_eq!(cached.write64(address, 0xaa), BUS_OK); + let stored = counter(&md, page); + assert_advanced("store to the checked page", marked, stored); + lift(&md, page); + md.retire_lifted_marks(); + let retired = counter(&md, page); + assert_advanced("retirement", stored, retired); + assert_eq!(cached.write64(address, 0xbb), BUS_OK); + assert_advanced("store after retirement", retired, counter(&md, page)); + assert_eq!(md.physical().banks()[4].read64(0x6008).data, 0xbb); + assert_eq!(counter(&md, 4 * BANK_SPAN + 0x8000), untouched, "a neighbouring page keeps its counter"); + } + + /// After a bank swap the slot's counter, which does not move, is the one + /// a store advances, whichever bank now sits behind the slot, and the + /// slot the bank left is not touched. + #[test] + fn a_store_after_a_bank_swap_advances_the_counter_of_the_slot_it_reached() { + let (md, bus, cached) = default_tezro(); + let (four, five) = (4 * BANK_SPAN + 0x2000, 5 * BANK_SPAN + 0x2000); + #[cfg(feature = "jitv2")] + let pointers = (md.memory.gen_ptr(four), md.memory.gen_ptr(five)); + // Bit 32 swaps the banks of a pair: slot 4 now reaches bank 5. + assert_eq!(md.md_bus().write64(reg::CONFIG, TEZRO_CONFIG | (1 << 32)), BUS_OK); + #[cfg(feature = "jitv2")] + assert_eq!((md.memory.gen_ptr(four), md.memory.gen_ptr(five)), pointers); + let (before_four, before_five) = (counter(&md, four), counter(&md, five)); + assert_eq!(cached.write64(four, 0x44), BUS_OK); + assert_eq!(md.physical().banks()[5].read64(0x2000).data, 0x44, "slot 4 reaches bank 5"); + assert_eq!(md.physical().banks()[4].read64(0x2000).data, 0); + let after_four = counter(&md, four); + assert_advanced("cached store to slot 4", before_four, after_four); + assert_eq!(counter(&md, five), before_five); + assert_eq!(bus.write64((3u64 << 57) | five, 0x55), BUS_OK); + assert_eq!(md.physical().banks()[4].read64(0x2000).data, 0x55, "slot 5 reaches bank 4"); + assert_advanced("uncached store to slot 5", before_five, counter(&md, five)); + assert_eq!(counter(&md, four), after_four); + } + + /// With bank 0 at 1 GiB the RAM extent covers the cached PROM window at + /// 0x1fc0_0000, which a Fuel's 256 MiB banks leave to the PROM. Once + /// bank 0 is enabled the cached plane, the uncached RAM view and DMA all + /// see RAM there, and a cached store lands in it; the PROM answers + /// through Hub-special space only. Before bank 0 is enabled nothing is + /// RAM, the cached plane shows the PROM and a store is refused. + /// + /// The QEMU model agrees for those views and differs for one: its reset + /// window is routed by cache attribute, and a noncoherent cached access + /// (what the stock PROM runs with) keeps seeing the PROM after RAM is + /// enabled (hw/mips/sgi/sn1-node.c, sn1_reset_attrs_to_index and + /// sn1_node_map_reset, "Compatibility policy"). The cache here passes + /// no attribute to the bus, so a noncoherent cached access sees RAM. + #[test] + fn a_1_gib_bank_0_covers_the_cached_prom_window_with_ram() { + let md = BedrockMd::new(&TEZRO_BANKS).unwrap(); + let (bus, cached) = board(&md); + let at = 0x1fc0_0000; + let prom = 0x0f1e_2d3c_4b5a_6978; + let uncached = (3u64 << 57) | at; + assert_eq!(cached.read64(at).data, prom, "before RAM is enabled"); + assert_eq!(cached.write64(at, 0x55), BUS_ERR); + assert_eq!(md.md_bus().write64(reg::CONFIG, TEZRO_CONFIG), BUS_OK); + assert_eq!(bus.read64(at).data, prom, "Hub-special space still shows the PROM"); + for (what, value) in [("cached", cached.read64(at).data), ("uncached RAM", bus.read64(uncached).data), + ("DMA", md.dma_view().read64(at).data)] { + assert_eq!(value, 0, "{what} sees RAM"); + } + assert!(md.physical().mem_ptr(at).is_some(), "the cache fills from RAM too"); + let before = counter(&md, at); + assert_eq!(cached.write64(at, 0x55), BUS_OK); + assert_eq!(md.physical().banks()[0].read64(at).data, 0x55); + assert_advanced("cached store", before, counter(&md, at)); + assert_eq!(bus.read64(at).data, prom, "the PROM is not written"); + // The Fuel's 256 MiB banks never reach it. + let fuel = BedrockMd::new(&FUEL_BANKS).unwrap(); + let (_bus, cached) = board(&fuel); + assert_eq!(fuel.md_bus().write64(reg::CONFIG, 0x33), BUS_OK); + assert_eq!(cached.read64(at).data, prom); + assert_eq!(cached.write64(at, 0x55), BUS_ERR); + } + + /// Physical's bulk writers, on a bank directly, advance only that bank's + /// counters, which nothing that follows code pages reads on this board. + /// The controller's own routes advance the logical ones as well. + #[test] + fn bulk_memory_writes_through_the_controller_advance_the_logical_counters() { + let md = BedrockMd::new(&[64]).unwrap(); + assert_eq!(md.md_bus().write64(reg::CONFIG, 0x11), BUS_OK); + let phys = md.physical(); + let at = 0x5000; + let logical = counter(&md, at); + phys.reset_memory(); + phys.restore_bank_inmem(0, &[1, 2]); + assert_eq!(counter(&md, at), logical, "Physical alone leaves the logical counter"); + md.reset_memory(); + let reset = counter(&md, at); + assert_advanced("reset", logical, reset); + md.restore_bank(0, &[0x0102_0304, 0x0506_0708]); + let restored = counter(&md, at); + assert_advanced("restore", reset, restored); + assert_eq!(phys.read64(0).data, 0x0102_0304_0506_0708); + let path = std::env::temp_dir().join(format!("iris-bedrock-load-{}", std::process::id())); + std::fs::write(&path, [0, 0, 0, 9, 0, 0, 0, 8]).unwrap(); + let loaded = md.load_bank(0, &path); + std::fs::remove_file(&path).unwrap(); + loaded.unwrap(); + assert_advanced("load", restored, counter(&md, at)); + assert_eq!(phys.read64(0).data, 0x0000_0009_0000_0008); + } + + /// Guest density settings remap fitted RAM, and reset restores the default map. + #[test] + fn bank_configuration_remaps_memory_and_reset_restores_defaults() { + let md = configured(); + let bus = md.physical(); + assert_eq!(bus.write64(0, 0x1122334455667788), BUS_OK); + assert_eq!(bus.write64(BANK_SPAN, 0xabcdef), BUS_OK); + assert_eq!(bus.read8(0).data, 0x11); + assert_eq!(bus.read32(4).data, 0x55667788); + assert_eq!(bus.read64(FITTED_BANK).status, BUS_ERR); + assert_eq!(bus.read64(2 * BANK_SPAN).status, BUS_ERR); + assert_eq!(md.md_bus().write64(8, 0x33 | 1 << 32), BUS_OK); + assert_eq!(bus.read64(0).data, 0xabcdef); + assert_eq!(bus.read64(BANK_SPAN).data, 0x1122334455667788); + assert!(bus.mem_ptr(0).is_some()); + md.soft_reset(); + assert_eq!(bus.read64(0).data, 0xabcdef); + md.power_on(); + assert_eq!(bus.read64(0).status, BUS_ERR); + bus.map_unmapped(0x1fc00000, 0x1fc01000); + for (code, size) in [(1, 64 << 20), (2, 128 << 20), (3, FITTED_BANK), + (4, FITTED_BANK), (5, FITTED_BANK)] { + md.md_bus().write64(reg::CONFIG, 1 | (code << 4)); + assert_eq!(bus.read64(size - 8).status, BUS_OK); + assert_eq!(bus.read64(size).status, BUS_ERR); + assert_eq!(bus.read64(0x1fc00000).status, BUS_OK, "a non-RAM mapping survives"); + } + md.md_bus().write64(reg::CONFIG, 0x11); + assert_eq!(bus.read64(64 << 20).status, BUS_ERR, "the old prefix is removed on shrink"); + md.md_bus().write64(reg::CONFIG, 0x61); + assert_eq!(bus.read64(0).status, BUS_ERR, "unsupported density does not map RAM"); + #[cfg(feature = "jitv2")] + { + let md = configured(); + let ram = md.physical(); + let first = ram.gen_ptr(0); + let second = ram.gen_ptr(BANK_SPAN); + assert!(!first.is_null() && !second.is_null()); + assert_ne!(first, second); + let before = unsafe { (*first).load(Ordering::Relaxed) }; + ram.write64(0, 0x55); + assert!(unsafe { (*first).load(Ordering::Relaxed) } > before); + md.md_bus().write64(8, 0x33 | (1 << 32)); + // The window address stays fixed while its backing bank changes. + assert_eq!(ram.gen_ptr(0), first); + assert_eq!(ram.gen_ptr(BANK_SPAN), second); + let bank0 = ram.banks()[0].gen_ptr(0); + let bank1 = ram.banks()[1].gen_ptr(0); + let before0 = unsafe { (*bank0).load(Ordering::Relaxed) }; + let before1 = unsafe { (*bank1).load(Ordering::Relaxed) }; + assert_eq!(ram.write64(0, 0xaa), BUS_OK); + assert_eq!(unsafe { (*bank0).load(Ordering::Relaxed) }, before0); + assert_eq!(unsafe { (*bank1).load(Ordering::Relaxed) }, before1 + 1); + assert_eq!(unsafe { (*first).load(Ordering::Relaxed) }, before1 + 1); + let view = md.uncached_view(); + let logical = view.gen_ptr(0); + assert_eq!(logical, md.memory.gen_ptr(0)); + let logical_before = unsafe { (*logical).load(Ordering::Relaxed) }; + drop(md); + assert_eq!(unsafe { (*logical).load(Ordering::Relaxed) }, logical_before, + "an external endpoint keeps its logical generation pointers alive"); + // The endpoint keeps the backing memory alive after MD disappears. + assert!(unsafe { (*bank1).load(Ordering::Relaxed) } > 0); + assert_eq!(view.read64(0).status, BUS_ERR); + } + + } + /// The guest reads ECC configuration and memory or directory errors through status registers. + #[test] + fn guest_registers_expose_ecc_configuration_and_error_latches() { + { + let md = configured(); + let r = md.md_bus(); + // These carrier offsets are idle; they do not select storage. + for at in [0x20, 0x20010, 0x40020, 0x60020] { + let before = md.unsupported(); + assert_eq!(r.write64(at, u64::MAX), BUS_OK, "{at:#x}"); + assert_eq!(r.read64(at).data, 0, "{at:#x}"); + assert_eq!(md.unsupported(), before + 1); + } + for (at, value) in [(0x60, 0x001818180c0c0c00), (0x40050, 0x04040400)] { + let before = md.unsupported(); + assert_eq!(r.write64(at, 0), BUS_OK); + assert_eq!(r.read64(at).data, value); + assert_eq!(md.unsupported(), before + 1); + } + // Error-clear writes are ignored too: only a read clears the record. + md.ecc_bus().write8(7, 3); + md.uncached_view().read64(0); + r.write64(0x40000, 2); + md.directory_bus().write64(0, 3 << 16); + r.write64(0x40000, 0); + md.physical().read64(0); + let memory_error = r.read64(0x60010).data; + let directory_error = r.read64(0x40010).data; + assert_ne!(memory_error & (1 << 60), 0); + assert_ne!(directory_error & (1 << 63), 0); + for at in [0x40010, 0x40018, 0x60010, 0x60018] { + let before = md.unsupported(); + assert_eq!(r.write64(at, u64::MAX), BUS_OK); + assert_eq!(r.read64(0x60010).data, memory_error); + assert_eq!(r.read64(0x40010).data, directory_error); + assert_eq!(md.unsupported(), before + 1); + } + // An unsupported self-test command leaves an in-flight command alone. + r.write64(0x20030, 1); + let before = md.unsupported(); + assert_eq!(r.write64(0x20030, 3), BUS_OK); + assert_eq!(md.unsupported(), before + 1); + md.advance_ns(1000); + assert_eq!(r.read64(0x20048).data, 3); + assert_eq!(r.write64(0x70000, 1), BUS_ERR); + assert_eq!(r.read64(0x70000).status, BUS_ERR); + assert_eq!(r.write32(0x60, 0), BUS_ERR); + assert_eq!(r.write64(0x61, 0), BUS_ERR); + } + { + let md = configured(); + let regs = md.md_bus(); + assert_eq!(regs.read64(0x60000).data, 0); + assert_eq!(regs.write64(0x60000, u64::MAX), BUS_OK); + assert_eq!(regs.read64(0x60000).data, 1); + assert_eq!(regs.write64(0x60000, !1), BUS_OK); + assert_eq!(regs.read64(0x60000).data, 0); + assert_eq!(regs.write8(0x60000, 1), BUS_ERR); + assert_eq!(regs.write64(0x60001, 1), BUS_ERR); + regs.write64(0x60000, 1); + md.soft_reset(); + assert_eq!(regs.read64(0x60000).data, 0); + assert_eq!(regs.read64(8).data, 0x33); + regs.write64(0x60000, 1); + md.power_on(); + assert_eq!(regs.read64(0x60000).data, 0); + assert_eq!(regs.read64(8).data, 0); + } + { + for (code, kind, status, valid) in [ + (1, MemoryError::Correctable, BUS_OK, 1 << 52), + (3, MemoryError::Uncorrectable, BUS_ERR, 1 << 60), + ] { + let md = configured(); + let regs = md.md_bus(); + let ecc = md.ecc_bus(); + let ram = md.uncached_view(); + let events = Arc::new(Mutex::new(Vec::new())); + let seen = events.clone(); + md.on_error(Arc::new(move |event| seen.lock().push(event))); + assert_eq!(ram.read64(0).status, BUS_OK); + ecc.write8(7, code); + regs.write64(0x60000, u64::MAX); + assert_eq!(ram.read64(0).status, BUS_OK); + assert_eq!(ecc.read8(7).data, code); + assert_eq!(regs.read64(0x60010).data, 0); + assert!(events.lock().is_empty()); + regs.write64(0x60000, !1); + assert_eq!(regs.read64(0x60000).data, 0); + assert_eq!(ram.read64(0).status, status); + assert_ne!(regs.read64(0x60010).data & valid, 0); + assert_eq!(*events.lock(), vec![kind]); + if kind == MemoryError::Correctable { + assert_eq!(ecc.read8(7).data, 0); + assert_eq!(ram.read64(0).status, BUS_OK); + assert_eq!(*events.lock(), vec![kind]); + } + } + } + { + let md = configured(); + let regs = md.md_bus(); + let ecc = md.ecc_bus(); + let ram = md.uncached_view(); + ecc.write8(7, 3); + regs.write64(0x60000, 1); + assert_eq!(ram.write8(7, 0x5a), BUS_OK); + assert_eq!(ecc.read8(7).data, 0x82); + regs.write64(0x60000, 0); + assert_eq!(ram.read64(0).status, BUS_OK); + assert_eq!(ram.read64(0).data, 0x5a); + assert_eq!(regs.read64(0x60010).data, 0); + } + } + /// Self-test completion is visible only after its configured clock interval. + #[test] + fn selftest_waits_for_fitted_clock_cycles_and_zero_fills_ram() { + { + let now = Arc::new(AtomicU64::new(100)); + let time = now.clone(); + let md = BedrockMd::with_clock(&FUEL_BANKS, Arc::new(move || time.load(Ordering::Relaxed))).unwrap(); + let regs = md.md_bus(); + regs.write64(8, 0x33); + md.start_clock(); + regs.write64(0x20030, 1); + now.store(1099, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + now.store(1100, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 3); + now.store(5000, Ordering::Relaxed); + regs.write64(0x20030, 1); + now.store(5500, Ordering::Relaxed); + md.stop_clock(); + now.store(15000, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + md.start_clock(); + now.store(15499, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + now.store(15500, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 3); + regs.write64(0x20030, 1); + now.store(15400, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + now.store(16499, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + now.store(16500, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 3); + regs.write64(0x20030, 1); + regs.write64(0x20030, 2); + now.store(20000, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + md.power_on(); + regs.write64(8, 0x33); + md.start_clock(); + regs.write64(0x20030, 1); + now.store(20999, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 0); + now.store(21000, Ordering::Relaxed); + assert_eq!(regs.read64(0x20048).data, 3); + } + { + let md = configured(); + let r = md.md_bus(); + r.write64(0x20030, 0x21); + md.advance_ns(1000); + assert_eq!(r.read64(0x20048).data, 2); + assert_eq!(r.read64(0x20040).data, 1 << 48); + r.write64(0x20030, 1); + r.write64(0x20030, 2); + md.advance_ns(1000); + assert_eq!(r.read64(0x20048).data, 0); + assert_eq!(r.write64(0x20030, 3), BUS_OK); + } + { + let md = configured(); + let r = md.md_bus(); + r.write64(8, 0x13); + md.physical().write64(0, 0x1234); + md.ecc_bus().write8(7, 1); + r.write64(0x20038, 0x80); + r.write64(0x20030, 1); + md.advance_ns(1000); + assert_eq!(r.read64(0x20048).data, 3); + assert_eq!(md.physical().read64(0).data, 0); + assert_eq!(md.ecc_bus().read8(7).data, 0); + assert_eq!(r.read64(0x60010).data, 0); + } + } + /// Fetch-and-op registers preserve the big-endian lanes used by the processor. + #[test] + fn fetchop_preserves_big_endian_word_lanes() { + let md = configured(); + let ram = md.physical(); + let f = md.fetchop_bus(); + ram.write64(0, 0xffffffff00000009); + assert_eq!(f.read32(8).data, u32::MAX); + assert_eq!(ram.read64(0).data, 9); + assert_eq!(f.read32(12).data, 9); + assert_eq!(ram.read64(0).data, 10); + assert_eq!(f.write32(28, 3), BUS_OK); + assert_eq!(ram.read64(0).data, 2); + assert_eq!(f.write64(32, 8), BUS_OK); + assert_eq!(ram.read64(0).data, 10); + assert_eq!(f.read64(24).data, 10); + assert_eq!(ram.read64(0).data, 0); + assert_eq!(f.read64(32).status, BUS_ERR); + assert_eq!(f.write64(40, 1), BUS_ERR); + assert_eq!(f.read16(0).status, BUS_ERR); + } + /// The guest reads ECC bytes and scrubbing repairs correctable memory. + #[test] + fn ecc_carriers_expose_check_bytes_and_scrub_corrections() { + { + let md = configured(); + let ecc = md.ecc_bus(); + for carrier in 0..8 { + let source = 0x11223344556677a5u64; + assert_eq!(ecc.write8(carrier, source as u8), BUS_OK); + assert_eq!(ecc.read8(carrier).data, source as u8, "carrier {carrier}"); + } + assert_eq!(ecc.write16(4, 0x1234), BUS_OK); + assert_eq!(ecc.read16(4).data, 0x1234); + assert_eq!(ecc.write16(6, 0x2313), BUS_OK); + assert_eq!(ecc.read16(6).data, 0x2313); + assert_eq!(ecc.write16(0x04000006, 0xface), BUS_OK); + assert_eq!(ecc.read16(0x04000006).data, 0xface); + + } + { + let md = configured(); + let events = Arc::new(Mutex::new(Vec::new())); + let seen = events.clone(); + let weak = Arc::downgrade(&md); + md.on_error(Arc::new(move |kind| { + assert_eq!(weak.upgrade().unwrap().current_cell(), u64::MAX); + seen.lock().push(kind); + })); + assert_eq!(md.ecc_bus().write8(7, 3), BUS_OK); + assert_eq!(md.ecc_bus().write8(6, 0x13), BUS_OK); + assert_eq!(md.ecc_bus().write8(5, 0x23), BUS_OK); + assert_eq!(md.ecc_scrub_bus().write64(0, 0), BUS_OK); + assert_eq!( + *events.lock(), + vec![ + MemoryError::Uncorrectable, + MemoryError::Correctable, + MemoryError::Correctable, + ] + ); + let record = md.md_bus().read64(0x60010).data; + assert_ne!(record & (1 << 60), 0); + assert_ne!(record & (1 << 52), 0); + assert_ne!(record & (1 << 53), 0); + assert_eq!((record >> 40) & 0xff, 3); + assert_eq!(record & (((1u64 << 30) - 1) << 3), 0); + assert_eq!(md.ecc_bus().read8(6).data, 0x13); + assert_eq!(md.ecc_bus().read8(5).data, 0x23); + } + } + /// The ECC view takes an uncached SH's check word from wherever the PROM + /// put it in the source register; other views, and bytes, take the low bits. + #[test] + fn ecc_view_reads_the_check_word_from_the_whole_source_register() { + let md = configured(); + let ecc = md.ecc_bus(); + // (carrier offset, register bits holding the check word) + for (carrier, shift) in [(6u64, 0u32), (4, 48), (2, 32), (0, 16)] { + let source = 0x1111_2222_3333_4444u64 ^ (0xa55a_u64 << shift); + let word = (source >> shift) as u16; + assert_eq!(ecc.write_narrow_from(carrier, 2, source), BUS_OK); + assert_eq!(ecc.read16(carrier).data, word, "carrier {carrier}"); + } + assert_eq!(ecc.write_narrow_from(5, 1, 0xfeed_0000_0000_0023), BUS_OK); + assert_eq!(ecc.read8(5).data, 0x23, "a byte keeps the architectural low bits"); + // The memory view is not the ECC view: its halfword is the low bits. + let view = md.uncached_view(); + assert_eq!(view.write_narrow_from(0x40, 2, 0xdead_beef_1234_5678), BUS_OK); + assert_eq!(md.dma_view().read64(0x40).data >> 48, 0x5678); + } + /// Uncached CPU and DMA views report uncorrectable errors with bus status. + #[test] + fn uncached_and_dma_views_report_errors_to_the_guest() { + let md = configured(); + let r = md.md_bus(); + let e = md.ecc_bus(); + e.write8(7, 3); + assert_eq!(md.uncached_view().read64(0).status, BUS_ERR); + assert_eq!(md.dma_view().read64(0).status, BUS_OK); + assert_ne!(r.read64(0x60018).data & (1 << 60), 0); + e.write8(7, 0x13); + assert_eq!(md.ecc_scrub_bus().write64(0, 0), BUS_OK); + assert_eq!(md.physical().read64(0).data, 1); + assert_eq!(e.read8(7).data, 0x13); + assert_eq!(md.ecc_scrub_bus().write32(0, 0), BUS_ERR); + } + /// Directory reads report invalid metadata and protection faults to the guest. + #[test] + fn directory_reads_check_metadata_and_report_protection_errors() { + { + let md = configured(); + let events = Arc::new(Mutex::new(Vec::new())); + let seen = events.clone(); + let weak = Arc::downgrade(&md); + md.on_error(Arc::new(move |kind| { + assert_eq!(weak.upgrade().unwrap().current_cell(), u64::MAX); + seen.lock().push(kind); + })); + let r = md.md_bus(); + md.directory_bus().write64(0, 1); + assert_eq!(md.directory_bus().read64(0).data, 0x001f0001); + r.write64(0x40000, 2); + md.directory_bus().write64(0, 0x001e0001); + r.write64(0x40000, 0); + md.ecc_bus().write8(7, 3); + assert_eq!(md.uncached_view().read64(0).status, BUS_ERR); + assert_eq!(md.directory_bus().read64(0).data, 0x001f0001); + assert_eq!( + *events.lock(), + vec![MemoryError::Correctable, MemoryError::Uncorrectable] + ); + assert_ne!(r.read64(0x40010).data & (1 << 61), 0); + assert_ne!(r.read64(0x60010).data & (1 << 60), 0); + } + { + let md = configured(); + let d = md.directory_bus(); + assert_eq!(d.write64(0x408, 0x1234), BUS_OK); + assert_eq!(d.read64(0x2000408).data, 0x1234); + assert_eq!(d.write64(0x418, 0x5678), BUS_OK); + assert_eq!(d.read64(0x408).data, 0x1234); + assert_eq!(d.read64(0x418).data, 0x5678); + assert_eq!(d.write64(8, u64::MAX), BUS_OK); + assert_eq!(d.read64(8).data, u64::MAX); + assert_eq!(d.read32(0).status, BUS_ERR); + assert_eq!(d.read64(1 << 30).status, BUS_ERR); + } + } +} + +const DIRECTORY_PREMIUM: [u8; 64] = [ + 0x1f, 0x2f, 0x37, 0x3b, 0x3d, 0x3e, 0x07, 0x0b, 0x0d, 0x0e, 0x13, 0x15, 0x16, 0x19, 0x1a, 0x1c, + 0x7f, 0x4f, 0x57, 0x5b, 0x5d, 0x5e, 0x67, 0x6b, 0x6d, 0x6e, 0x73, 0x75, 0x76, 0x79, 0x7a, 0x7c, + 1, 2, 4, 8, 16, 32, 0x23, 0x25, 0x26, 0x29, 0x2a, 0x2c, 0x31, 0x32, 0x34, 0x38, 0x40, 0x43, + 0x45, 0x46, 0x49, 0x4a, 0x4c, 0x51, 0x52, 0x54, 0x58, 0x61, 0x62, 0x64, 0x68, 0x70, +]; +const DIRECTORY_STANDARD: [u8; 32] = [ + 0x1f, 0x2f, 0x37, 0x3b, 0x3d, 0x3e, 0x07, 0x0b, 0x0d, 0x0e, 0x13, 0x15, 0x16, 0x19, 0x1a, 0x1c, + 1, 2, 4, 8, 16, 32, 0x23, 0x25, 0x26, 0x29, 0x2a, 0x2c, 0x31, 0x32, 0x34, 0x38, +]; +fn directory_columns(premium: bool) -> &'static [u8] { + if premium { + &DIRECTORY_PREMIUM + } else { + &DIRECTORY_STANDARD + } +} +fn directory_syndrome(mut word: u64, premium: bool) -> u8 { + if !premium { + word &= u32::MAX as u64; + } + let mut check = 0; + while word != 0 { + check ^= directory_columns(premium)[word.trailing_zeros() as usize]; + word &= word - 1; + } + check +} +pub fn directory_encode(word: u64, premium: bool) -> u64 { + let word = word + & !if premium { + (63 << 32) | (1 << 48) + } else { + 63 << 16 + }; + let check = directory_syndrome(word, premium) as u64; + word | if premium { + (check & 63) << 32 | (check >> 6) << 48 + } else { + check << 16 + } +} diff --git a/src/dev/bedrock_mem.rs b/src/dev/bedrock_mem.rs new file mode 100644 index 000000000..2cd88ceba --- /dev/null +++ b/src/dev/bedrock_mem.rs @@ -0,0 +1,864 @@ +//! A Bedrock node's fitted memory: banks owned by `Physical`, with the bank +//! decode selected by Bedrock's memory configuration register. +//! +//! The board names the fitted banks, locational bank 0 upward, one size in +//! MiB each: two 256 MiB banks on the Fuel, six on the Tezro. Each bank +//! answers in its own 1 GiB software slot, so a Tezro's banks 4 and 5 sit +//! at 4 to 6 GiB, above the 4 GiB window ppmem maps for the CPU's direct +//! access. Those two answer through Physical's decode table only, and +//! their code pages keep the logical generation counters like the rest. +//! +//! These populations follow the QEMU models; they are not measured +//! topologies. Ordinary pages answer through Physical's bank views, without +//! the MD state lock. Pages with injected ECC or marked directory lines use +//! the MD endpoint. Their entire 64 MiB region is absent from the direct +//! window, so cache fills cannot bypass those checks. +//! +//! Remapping runs on the CPU thread, or with the CPU stopped, as HEART's +//! memory descriptors do. A bank configuration change rebuilds the whole +//! map. Marking a page adds only that page: it takes its entry in the decode +//! table, and the first in a 64 MiB region takes the region out of the direct +//! window. Lifting a mark removes nothing. The page stays on the MD endpoint, +//! which answers as the bank would, until a batch retires every lifted page +//! at once: the PROM's memory test marks and lifts the same few hundred pages +//! thousands of times, and a map change for each made it take minutes in the +//! QEMU model. DMA may remove diagnostic metadata but retains the slow +//! mapping until a later retirement. Physical publishes complete +//! decode-table pages atomically; a concurrent bus master sees an old or new +//! entry. This does not promise a transaction across a remap. +//! +//! Ported from hw/misc/sgi-bedrock-md.c in jamesbraid/qemu-sgi-origami. + +use crate::physical::{Physical, RamBank}; +use crate::traits::*; +use crate::ppmem::devmap::SLOT_SHIFT; +use crate::ppmem::{MappedMemory, BITMAP_SHIFT, WINDOW_SIZE}; +use crate::cpu::smp::{current_cpu, MAX_CPUS}; +use parking_lot::Mutex; +use std::collections::{BTreeSet, HashMap, HashSet}; +use std::sync::atomic::{AtomicU64, AtomicU8, Ordering}; +use std::sync::{Arc, OnceLock}; + +pub const BANK_SPAN: u64 = 1 << 30; +/// Software slots of [`BANK_SPAN`] in the node's 8 GiB. +pub const SLOTS: usize = 8; +/// The most memory a bank can hold: what the largest size code names. +const MAX_BANK_MIB: u32 = 1024; +/// Banks are mapped into the direct window in whole regions of this size. +const REGION_MIB: u32 = 1 << (BITMAP_SHIFT - 20); + +/// A DIMM size the Tezro's IP53 board takes. One DIMM makes one bank. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum BankSize { + Mib512, + Mib1024, +} +impl BankSize { + pub const fn mib(self) -> u32 { + match self { + Self::Mib512 => 512, + Self::Mib1024 => 1024, + } + } + fn from_mib(mib: u32) -> Result { + match mib { + 512 => Ok(Self::Mib512), + 1024 => Ok(Self::Mib1024), + _ => Err(format!("a Tezro bank is 512 or 1024 MiB, not {mib} MiB")), + } + } +} + +/// The banks a Tezro fits, locational bank 0 upward: up to eight DIMMs of 512 +/// or 1024 MiB with none left out between them. What the board needs in total +/// depends on its processors; see [`Self::check_for_cpus`]. +/// +/// The bank sizes and the minimums come from the Tezro manuals as the QEMU +/// assembly layer records them. The default is the 4 GiB layout of 1024, 1024 +/// and four 512 MiB banks, a population a real Tezro has been observed to hold. +/// Its banks 4 and 5 lie above 4 GiB, outside ppmem's direct window, so they +/// answer through Physical's decode table, which is slower. +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct MemoryPopulation(Vec); +impl Default for MemoryPopulation { + fn default() -> Self { + Self::four_gib() + } +} +impl MemoryPopulation { + pub fn new(banks: &[BankSize]) -> Result { + if banks.is_empty() || banks.len() > SLOTS { + return Err(format!("memory: {} banks; a Tezro fits 1 to {SLOTS}", banks.len())); + } + Ok(Self(banks.to_vec())) + } + /// From sizes in MiB, rejecting any a Tezro does not take. + pub fn from_mib(banks: &[u32]) -> Result { + let sizes = banks.iter().map(|&mib| BankSize::from_mib(mib)).collect::, _>>()?; + Self::new(&sizes) + } + /// 1 GiB as two banks of 512 MiB. The manual's minimum of 1 GB for two or + /// four processors permits it, but no real Tezro has been observed with + /// less than 4 GiB, and nothing here claims the PROM or IRIX boot at it. All + /// of it lies below 4 GiB, so it is wholly inside ppmem's direct window. + pub fn one_gib() -> Self { + Self(vec![BankSize::Mib512; 2]) + } + /// 4 GiB: 1024, 1024 and four 512 MiB banks, the default. Banks 4 and 5 + /// sit above 4 GiB, outside the direct window. + pub fn four_gib() -> Self { + let mut banks = vec![BankSize::Mib1024; 2]; + banks.extend([BankSize::Mib512; 4]); + Self(banks) + } + pub fn banks(&self) -> &[BankSize] { + &self.0 + } + /// The sizes in MiB, as [`BedrockMd::new`](super::bedrock_md::BedrockMd::new) takes them. + pub fn banks_mib(&self) -> Vec { + self.0.iter().map(|bank| bank.mib()).collect() + } + pub fn total_mib(&self) -> u32 { + self.0.iter().map(|bank| bank.mib()).sum() + } + /// Whether every bank lies below 4 GiB, so the CPU reaches all of it + /// through the direct window. + pub fn fits_direct_window(&self) -> bool { + self.0.len() <= 4 + } + /// One processor needs at least 512 MB; two or four need at least 1 GB. + pub fn check_for_cpus(&self, cpus: usize) -> Result<(), String> { + let needed = match cpus { + 1 => 512, + 2..=4 => 1024, + _ => return Err(format!("{cpus} processors; a Tezro fits 1 to 4")), + }; + if self.total_mib() < needed { + return Err(format!( + "{} MiB of memory; {cpus} processor{} need at least {needed} MiB", + self.total_mib(), if cpus == 1 { "" } else { "s" })); + } + Ok(()) + } +} + +/// Where fitted RAM can answer: for each software slot, the most any bank the +/// bank-select swizzle can place there holds. The swizzle permutes slots +/// within groups of four, so a slot may hold any bank of its group. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct RamExtent([u64; SLOTS]); +impl RamExtent { + /// `banks` are the fitted sizes in MiB from locational bank 0. + pub fn new(banks: &[u32]) -> Self { + let fitted = |bank: usize| banks.get(bank).map_or(0, |&mib| u64::from(mib) << 20); + Self(std::array::from_fn(|slot| { + (0..4).map(|group_slot| fitted(slot / 4 * 4 + group_slot)).max().unwrap_or(0) + })) + } + /// Whether `address` can be fitted RAM under some bank-select setting. + pub fn contains(&self, address: u64) -> bool { + let slot = address / BANK_SPAN; + slot < SLOTS as u64 && address % BANK_SPAN < self.0[slot as usize] + } + /// Slots from 0 through the last that can hold RAM. + #[cfg(feature = "jitv2")] + fn slots(&self) -> usize { + self.0.iter().rposition(|&size| size != 0).map_or(0, |last| last + 1) + } + /// The most RAM software slot `slot` can hold, in bytes. + pub fn prefix(&self, slot: usize) -> u64 { self.0[slot] } +} + +/// The PROM builds its stack and data in primary data cache lines it tags for +/// this node-local window, with no memory behind it: DRAM is not configured +/// yet. See [`BedrockMemory::startup_tag`]. +pub(crate) const STARTUP_BASE: u64 = 0x10_0000; +pub(crate) const STARTUP_SIZE: u64 = 0x8000; +/// A primary data cache tag: the state in bits 7:6, the physical address +/// bits 35:12 above bit 8. Bits above the node's 8 GiB (page bit 21) select +/// the node. +const TAG_STATE: u32 = 0xc0; +const TAG_PAGE_SHIFT: u32 = 8; +const TAG_NODE_PAGES: u32 = 1 << 21; + +/// The startup window as each processor sees it: the processor's own page +/// while its PROM lines stand, its RAM otherwise. Each processor's cache +/// holds its own lines, so each stack is private, QEMU's per-processor +/// cache-as-RAM window. That private storage is a model policy; when a +/// window starts and ends is approximated from the tag stores (see +/// [`BedrockMemory::startup_tag`]). Physical's table holds it over the +/// window while any processor's stands; a processor whose window has ended +/// reaches memory through MD's checked path, as a checked page does. +struct StartupWindow { + /// The window's bytes for each processor. Zero at power-on: the PROM makes + /// the lines valid without a fill. + pages: [RamBank; MAX_CPUS], + /// Bit `cpu` set while that processor's window stands. + live: AtomicU8, + /// The node's memory view, set once when the controller is built. + slow: OnceLock>, +} +impl StartupWindow { + fn new() -> Self { + Self { + pages: std::array::from_fn(|_| { + let page = RamBank::new(1); + page.set_addr_mask(STARTUP_SIZE as u32 - 1); + page + }), + live: AtomicU8::new(0), + slow: OnceLock::new(), + } + } + fn live(&self, cpu: usize) -> bool { + cpu < MAX_CPUS && self.live.load(Ordering::Acquire) & (1 << cpu) != 0 + } + fn any(&self) -> bool { + self.live.load(Ordering::Acquire) != 0 + } + fn set(&self, cpu: usize, live: bool) { + if cpu < MAX_CPUS { + if live { self.live.fetch_or(1 << cpu, Ordering::AcqRel) } else { self.live.fetch_and(!(1 << cpu), Ordering::AcqRel) }; + } + } + fn clear(&self) { + self.live.store(0, Ordering::Release); + for page in &self.pages { + for at in (0..STARTUP_SIZE).step_by(8) { + page.write64(at, 0); + } + } + } + /// The accessing processor's page, while its window stands. + fn own(&self) -> Option<&RamBank> { + let cpu = current_cpu(); + self.live(cpu).then(|| &self.pages[cpu]) + } + /// Where an access that is not to the processor's own page goes. + fn memory(&self) -> Option<&Arc> { + self.slow.get() + } +} +macro_rules! startup_access { + ($r:ident, $w:ident, $t:ty, $out:ident) => { + fn $r(&self, a: u64) -> $out { + match self.own() { + Some(page) => page.$r(a), + None => self.memory().map_or_else($out::err, |m| m.$r(a)), + } + } + fn $w(&self, a: u64, v: $t) -> u32 { + match self.own() { + Some(page) => page.$w(a, v), + None => self.memory().map_or(BUS_ERR, |m| m.$w(a, v)), + } + } + }; +} +impl BusDevice for StartupWindow { + startup_access!(read8, write8, u8, BusRead8); + startup_access!(read16, write16, u16, BusRead16); + startup_access!(read32, write32, u32, BusRead32); + startup_access!(read64, write64, u64, BusRead64); + #[cfg(feature = "jitv2")] + fn gen_ptr(&self, a: u64) -> *const AtomicU64 { + match self.own() { + Some(page) => page.gen_ptr(a), + None => self.memory().map_or(std::ptr::null(), |m| m.gen_ptr(a)), + } + } +} + +#[derive(Clone, Copy, PartialEq, Eq)] +struct Run { + bank: usize, + base: u64, + size: u64, +} + +/// Lifted pages kept on the MD endpoint before a batch retires them. +pub(crate) const RETIRED_LIMIT: usize = 1024; +/// How long a lifted page may idle before the next publication retires it: +/// a second of the controller's clock. +pub(crate) const RETIRED_IDLE_NS: u64 = 1_000_000_000; + +#[derive(Default)] +struct Mapping { + runs: Vec, + /// Whether the decode table's slots over the startup window hold the page. + startup_placed: bool, + /// The pages that answer through the MD endpoint: those with a mark and + /// those whose mark was lifted since the last retirement. Bank-relative + /// 4 KiB page numbers, bank 1 immediately after bank 0. + placed: BTreeSet, + /// How many placed pages each 64 MiB region of the node's address space + /// holds, for the regions inside a run. + held: HashMap, + /// The regions mapped into the direct window now. + windowed: HashSet, + /// When the first lifted page was seen idle, on the controller's clock. + idle_since: Option, + slow: Option>, +} + +struct NoMemory; +impl BusDevice for NoMemory {} + +/// JIT pages belong to physical addresses, not to the bank currently +/// answering there. Keep these counters separate from the banks' counters: +/// a compiled page pins its pointer across later bank reconfiguration. +#[cfg(feature = "jitv2")] +pub(crate) struct Generations(Box<[AtomicU64]>); +#[cfg(feature = "jitv2")] +impl Generations { + /// One counter per 4 KiB page, up to the last slot that can hold RAM. + fn new(extent: &RamExtent) -> Self { + let pages = extent.slots() << (BANK_SPAN.trailing_zeros() - 12); + Self((0..pages).map(|_| AtomicU64::new(0)).collect::>().into_boxed_slice()) + } + pub(crate) fn base(&self) -> *mut AtomicU64 { + self.0.as_ptr() as *mut AtomicU64 + } + fn bump(&self, address: u64) { + if let Some(counter) = self.0.get((address >> 12) as usize) { + counter.fetch_add(1, Ordering::Relaxed); + } + } + fn bump_all(&self, extent: &RamExtent) { + for slot in 0..extent.slots() { + let first = slot * (BANK_SPAN >> 12) as usize; + let pages = (extent.prefix(slot) >> 12) as usize; + for counter in &self.0[first..first + pages] { + counter.fetch_add(1, Ordering::Relaxed); + } + } + } +} + +/// What a remap costs, counted where the work happens so a test can state +/// it without timing anything. +#[cfg(test)] +#[derive(Default)] +pub(crate) struct Cost { + /// Host mappings made or scrubbed in ppmem's direct window. + pub window_ops: AtomicU64, + /// Calls that rewrite decode-table slots. + pub table_ops: AtomicU64, + /// Generation counters incremented, logical and bank. + pub bumps: AtomicU64, +} +#[cfg(test)] +impl Cost { + pub fn add(counter: &AtomicU64, n: usize) { + counter.fetch_add(n as u64, Ordering::Relaxed); + } + pub fn snapshot(&self) -> [u64; 3] { + [&self.window_ops, &self.table_ops, &self.bumps].map(|c| c.load(Ordering::Relaxed)) + } +} + +pub struct BedrockMemory { + physical: Arc, + /// Fitted bytes in each locational bank, bank 0 first. Physical's banks + /// are these, in the same order. + fitted: Vec, + config: AtomicU64, + mapping: Mutex, + /// Each processor's window while its PROM cache lines stand. + startup: Box, + extent: RamExtent, + #[cfg(feature = "jitv2")] + generations: Arc, + #[cfg(test)] + pub(crate) cost: Cost, +} +impl BedrockMemory { + /// Memory with `banks` fitted, locational bank 0 upward, each a size in + /// MiB: at most [`SLOTS`] banks, none empty or over 1 GiB (the largest + /// size code), each a whole number of direct-window regions. + pub fn new(banks: &[u32]) -> Result { + if banks.is_empty() || banks.len() > SLOTS { + return Err(format!("memory: {} banks; fit 1 to {SLOTS}", banks.len())); + } + if let Some((bank, mib)) = banks.iter().enumerate() + .find(|(_, &mib)| mib == 0 || mib > MAX_BANK_MIB || mib % REGION_MIB != 0) + { + return Err(format!( + "memory: bank {bank} is {mib} MiB; a bank is a multiple of {REGION_MIB} MiB up to {MAX_BANK_MIB} MiB")); + } + let physical = Physical::new(banks.iter().map(|&mib| RamBank::new(mib as usize)).collect()); + physical.set_default(Arc::new(NoMemory), "Bedrock unconfigured memory"); + Ok(Self { physical, fitted: banks.iter().map(|&mib| u64::from(mib) << 20).collect(), + config: AtomicU64::new(0), mapping: Mutex::new(Mapping::default()), + startup: Box::new(StartupWindow::new()), + #[cfg(feature = "jitv2")] + generations: Arc::new(Generations::new(&RamExtent::new(banks))), + extent: RamExtent::new(banks), + #[cfg(test)] + cost: Cost::default(), + }) + } + /// Fitted bytes in locational bank `bank`; zero where nothing is fitted. + pub(crate) fn fitted(&self, bank: u64) -> u64 { + self.fitted.get(bank as usize).copied().unwrap_or(0) + } + /// Where the fitted banks start in a dense walk of all of them, bank 0 + /// first. Sparse tables key ECC, directory and protection state by it, so + /// two banks never share a key. + pub(crate) fn dense_base(&self, bank: u64) -> u64 { + self.fitted.iter().take(bank as usize).sum() + } + pub fn extent(&self) -> RamExtent { self.extent } + pub fn physical(&self) -> &Arc { + &self.physical + } + pub(crate) fn config(&self) -> u64 { + self.config.load(Ordering::Acquire) + } + + /// Whether a primary data cache tag store starts or ends the startup + /// window, given its TagLo. PROM 6.210 stores test patterns (state bits + /// set, pages far from the window), an invalidate sweep, then one valid + /// tag for each of the window's eight pages, and a final invalidate sweep + /// once it has memory. So a valid tag for a page of the window starts it + /// and the next invalid one ends it; a kernel stores invalid tags only. + /// + /// The end is the sweep's first store, not its last. Counting accesses + /// that reach the page in an interpreted PROM 6.210 boot, none follow the + /// first of the final sweep's 2050 invalid stores (the count is the same + /// at the first and at the last), and the page is read and written + /// heavily up to that first store. Holding it to the sweep's end would + /// change nothing the PROM sees. + /// `Some(live)` is the state to move to, `None` leaves it alone. + /// + /// Each processor stands its own lines, so its tag stores move its own + /// window, `cpu`'s. + pub(crate) fn startup_tag(&self, cpu: usize, tag: u32) -> Option { + let live = self.startup.live(cpu); + if tag & TAG_STATE == 0 { + return live.then_some(false); + } + let page = (tag >> TAG_PAGE_SHIFT) & (TAG_NODE_PAGES - 1); + let in_window = (STARTUP_BASE >> 12..(STARTUP_BASE + STARTUP_SIZE) >> 12).contains(&(page as u64)); + (in_window && !live).then_some(true) + } + /// Move `cpu`'s window to `live`; whether that changed whether any + /// window stands, and so needs a remap. + pub(crate) fn set_startup(&self, cpu: usize, live: bool) -> bool { + let was = self.startup_live(); + self.startup.set(cpu, live); + was != self.startup_live() + } + /// Whether any processor's window stands. + pub(crate) fn startup_live(&self) -> bool { + self.startup.any() + } + /// Power-on: no cache lines stand, and every page holds zero. + pub(crate) fn clear_startup(&self) { + self.startup.clear(); + } + /// The page, for an access it answers: the window, while the accessing + /// processor's stands. The bus decode, the CPU's cached path and DMA all + /// reach it through here or through Physical's table. A thread that runs + /// no processor reaches CPU 0's, the one a single-processor board has. + pub(crate) fn startup_device(&self, address: u64, bytes: u64) -> Option<&dyn BusDevice> { + let offset = address.checked_sub(STARTUP_BASE)?; + if offset + bytes > STARTUP_SIZE { + return None; + } + self.startup.own().map(|page| page as &dyn BusDevice) + } + #[cfg(feature = "jitv2")] + pub(crate) fn generations(&self) -> Arc { self.generations.clone() } + #[cfg(feature = "jitv2")] + pub(crate) fn gen_ptr(&self, address: u64) -> *const AtomicU64 { + // Disabled RAM must still have a counter: a JIT page may already + // have claimed it before the guest enables or moves a bank. + if self.extent.contains(address) { + unsafe { self.generations.base().add((address >> 12) as usize) } + } else { std::ptr::null() } + } + pub(crate) fn written(&self, _address: u64) { + #[cfg(feature = "jitv2")] + self.generations.bump(_address); + } + + /// The endpoint is weak back to MD: Physical owns it without a cycle. + pub(crate) fn set_slow_view(&self, slow: Arc) { + assert!(self.startup.slow.set(slow.clone()).is_ok(), "the slow view is set once"); + self.mapping.lock().slow = Some(slow); + } + + fn runs(&self, config: u64) -> Vec { + (0..SLOTS as u64).filter_map(|slot| { + let base = slot * BANK_SPAN; + let (bank, _) = self.decode(config, base, 1)?; + let code = (config >> (bank / 2 * 8 + 4)) & 15; + Some(Run { bank, base, size: self.fitted(bank as u64).min((32 << 20) << code) }) + }).collect() + } + + /// Publish a state snapshot after releasing MD's lock. Only the CPU + /// thread, or a caller with the CPU stopped, may change ppmem's bitmap. + /// A forced rebuild follows the machine setting Physical's default. + /// This is the whole-map rebuild: `pages` replaces every placed page, so + /// it retires the lifted ones too. Without `force` it does nothing when + /// neither the runs nor the placed pages would change. + pub(crate) fn remap(&self, config: u64, pages: BTreeSet, force: bool) { + let runs = self.runs(config); + let mut old = self.mapping.lock(); + if !force && old.runs == runs && old.placed == pages { + self.config.store(config, Ordering::Release); + return; + } + self.rebuild(&mut old, config, runs, pages); + } + + /// The bank configuration register changed to `config`. Rebuild, with + /// the marked `pages`, only if it changes which banks answer where. + pub(crate) fn reconfigure(&self, config: u64, pages: impl FnOnce() -> BTreeSet) { + let runs = self.runs(config); + let mut old = self.mapping.lock(); + if old.runs == runs { + self.config.store(config, Ordering::Release); + return; + } + self.rebuild(&mut old, config, runs, pages()); + } + + fn rebuild(&self, old: &mut Mapping, config: u64, runs: Vec, pages: BTreeSet) { + let phys = &self.physical; + let space = phys.ppmem_space(); + // Withdraw the direct window before publishing any checked page. + if let Some(space) = space { + #[cfg(test)] + Cost::add(&self.cost.window_ops, space.mapping_count()); + space.clear_mappings(); + } + // Restore only ranges previously owned by RAM. In particular, the + // unused part of a 1 GiB slot may contain flash or other devices. + if old.startup_placed { + #[cfg(test)] + Cost::add(&self.cost.table_ops, 1); + phys.fill_slots(STARTUP_BASE >> SLOT_SHIFT, STARTUP_SIZE >> SLOT_SHIFT, phys.default_entry()); + } + for run in &old.runs { + let retained = runs.iter().find(|next| next.base == run.base) + .map_or(0, |next| next.size.min(run.size)); + if retained < run.size { + #[cfg(test)] + Cost::add(&self.cost.table_ops, 1); + phys.fill_slots((run.base + retained) >> SLOT_SHIFT, + (run.size - retained) >> SLOT_SHIFT, phys.default_entry()); + } + } + old.held.clear(); + old.windowed.clear(); + old.idle_since = None; + for run in &runs { + #[cfg(test)] + Cost::add(&self.cost.table_ops, 1); + phys.fill_slots(run.base >> SLOT_SHIFT, run.size >> SLOT_SHIFT, phys.bank_view(run.bank, run.base)); + let bank_page = self.dense_base(run.bank as u64) >> SLOT_SHIFT; + let run_pages = run.size >> SLOT_SHIFT; + for &page in pages.range(bank_page..bank_page + run_pages) { + let address = run.base + ((page - bank_page) << SLOT_SHIFT); + *old.held.entry(address >> BITMAP_SHIFT).or_default() += 1; + #[cfg(test)] + Cost::add(&self.cost.table_ops, 1); + phys.map(address, address + (1 << SLOT_SHIFT), + old.slow.as_ref().expect("Bedrock MD slow endpoint").clone(), "Bedrock checked RAM"); + } + if let Some(space) = space { + let region_size = 1 << BITMAP_SHIFT; + for offset in (0..run.size).step_by(region_size as usize) { + // The startup page answers through the decode only, so the + // region holding it stays out of the direct window. + let startup = self.startup_live() && run.base + offset == 0; + let region = (run.base + offset) >> BITMAP_SHIFT; + // Above the window the bank answers through the table alone. + let in_window = run.base + offset + region_size <= WINDOW_SIZE; + if in_window && !startup && !old.held.contains_key(®ion) { + #[cfg(test)] + Cost::add(&self.cost.window_ops, 1); + match space.map_range(run.bank, offset, run.base + offset, region_size) { + Ok(()) => { old.windowed.insert(region); } + Err(error) => crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock RAM window at {:#x}: {error}", run.base + offset), + } + } + } + } + } + let live = self.startup_live(); + if live { + #[cfg(test)] + Cost::add(&self.cost.table_ops, 1); + let window: *const StartupWindow = &*self.startup; + phys.fill_slots(STARTUP_BASE >> SLOT_SHIFT, STARTUP_SIZE >> SLOT_SHIFT, + window as crate::ppmem::devmap::Entry); + } + old.startup_placed = live; + self.config.store(config, Ordering::Release); + // Publish changed ownership before advancing the stable counters, + // so an in-flight compile cannot accept bytes from the old decode. + self.changed_decode(); + old.runs = runs; + old.placed = pages; + } + + /// Where bank-relative `page` lies in the run holding it: the run, and the + /// page's byte offset in the bank. + fn locate(&self, runs: &[Run], page: u64) -> Option<(Run, u64)> { + runs.iter().find_map(|run| { + let first = self.dense_base(run.bank as u64) >> SLOT_SHIFT; + (first..first + (run.size >> SLOT_SHIFT)).contains(&page) + .then(|| (*run, (page - first) << SLOT_SHIFT)) + }) + } + + /// Whether `address` lies in the startup window while its page stands. + fn in_startup(&self, address: u64) -> bool { + self.startup_live() && (STARTUP_BASE..STARTUP_BASE + STARTUP_SIZE).contains(&address) + } + + /// Advance the counters of `pages` 4 KiB pages starting at `address`, the + /// bank's offset `offset` for the same bytes. + fn bump_pages(&self, _address: u64, _bank: usize, _offset: u64, _pages: u64) { + #[cfg(feature = "jitv2")] + { + for page in 0.._pages { + self.generations.bump(_address + (page << SLOT_SHIFT)); + self.physical.banks()[_bank].bump_gen_off((_offset + (page << SLOT_SHIFT)) as usize); + } + #[cfg(test)] + Cost::add(&self.cost.bumps, 2 * _pages as usize); + } + } + + /// Route the pages in `marked` through the MD endpoint, adding to what is + /// placed and changing nothing else. A page already placed, as a lifted + /// one marked again is, costs no map change. The first page in a region + /// takes the region out of the direct window, and a page whose mapping + /// changes advances its counters, so code compiled from it, or from the + /// region's direct path, is dropped. CPU thread only, as `remap`. + pub(crate) fn place(&self, marked: &[u64]) { + let mut m = self.mapping.lock(); + let phys = &self.physical; + let space = phys.ppmem_space(); + let region_size = 1u64 << BITMAP_SHIFT; + for &page in marked { + if !m.placed.insert(page) { + continue; + } + let Some((run, offset)) = self.locate(&m.runs, page) else { continue }; + let address = run.base + offset; + let region = address >> BITMAP_SHIFT; + *m.held.entry(region).or_default() += 1; + // Withdraw the direct window before publishing the checked page. + if m.windowed.remove(®ion) { + if let Some(space) = space { + // Off the direct path, host pages left mapped: another + // processor that read the old bitmap still reaches the bank. + space.withdraw_range(region << BITMAP_SHIFT, region_size); + } + self.bump_pages(region << BITMAP_SHIFT, run.bank, + offset & !(region_size - 1), region_size >> SLOT_SHIFT); + } else { + self.bump_pages(address, run.bank, offset, 1); + } + if !self.in_startup(address) { + #[cfg(test)] + Cost::add(&self.cost.table_ops, 1); + phys.map(address, address + (1 << SLOT_SHIFT), + m.slow.as_ref().expect("Bedrock MD slow endpoint").clone(), "Bedrock checked RAM"); + } + } + } + + /// Whether the lifted pages are due to be retired: `active` pages of the + /// placed ones still hold a mark, and `now` is the controller's clock if + /// it has one. Due when too many lifted pages have piled up, or when the + /// first has idled for a second. Notes the time it first sees one. + pub(crate) fn retirement_due(&self, active: usize, now: Option) -> bool { + let mut m = self.mapping.lock(); + if m.placed.len() <= active { + m.idle_since = None; + return false; + } + if m.placed.len() - active >= RETIRED_LIMIT { + return true; + } + match (now, m.idle_since) { + (Some(now), Some(since)) => now.saturating_sub(since) >= RETIRED_IDLE_NS, + (Some(now), None) => { m.idle_since = Some(now); false } + _ => false, + } + } + + /// The placed pages for which `marked` is false, in order. + pub(crate) fn lifted(&self, marked: impl Fn(u64) -> bool) -> Vec { + self.mapping.lock().placed.iter().copied().filter(|&page| !marked(page)).collect() + } + + /// Return every page of `lifted` to its bank in one change: its table + /// entry, then the direct window for each region left with no placed + /// page. The pages' counters advance as for any change of mapping. CPU + /// thread only, as `remap`. + pub(crate) fn retire(&self, lifted: &[u64]) { + let mut m = self.mapping.lock(); + let phys = &self.physical; + let space = phys.ppmem_space(); + let region_size = 1u64 << BITMAP_SHIFT; + let mut emptied = Vec::new(); + for &page in lifted { + if !m.placed.remove(&page) { + continue; + } + let Some((run, offset)) = self.locate(&m.runs, page) else { continue }; + let address = run.base + offset; + let region = address >> BITMAP_SHIFT; + if !self.in_startup(address) { + #[cfg(test)] + Cost::add(&self.cost.table_ops, 1); + phys.fill_slots(address >> SLOT_SHIFT, 1, phys.bank_view(run.bank, run.base)); + } + self.bump_pages(address, run.bank, offset, 1); + let held = m.held.get_mut(®ion).expect("a placed page's region is held"); + *held -= 1; + if *held == 0 { + m.held.remove(®ion); + emptied.push((region, run, offset & !(region_size - 1))); + } + } + // Return a region to the window only once none of its pages is placed. + for (region, run, offset) in emptied { + let startup = self.startup_live() && region == 0; + let in_window = (region + 1) << BITMAP_SHIFT <= WINDOW_SIZE; + if let (Some(space), true, false) = (space, in_window, startup) { + let base = region << BITMAP_SHIFT; + // A region withdrawn from the path is still mapped; one the + // last rebuild left out because it held a page was never. + if space.restore_range(base, region_size) { + m.windowed.insert(region); + continue; + } + #[cfg(test)] + Cost::add(&self.cost.window_ops, 1); + match space.map_range(run.bank, offset, base, region_size) { + Ok(()) => { m.windowed.insert(region); } + Err(error) => crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock RAM window at {:#x}: {error}", base), + } + } + } + m.idle_since = None; + } + + pub(crate) fn decode(&self, config: u64, addr: u64, bytes: u64) -> Option<(usize, u64)> { + let software = addr / BANK_SPAN; + if software >= SLOTS as u64 { + return None; + } + let bank = software ^ ((config >> 32) & 3); + let field = config >> (bank / 2 * 8); + let code = (field >> 4) & 15; + if self.fitted(bank) == 0 || field & (1 << (bank & 1)) == 0 || !(1..=5).contains(&code) { + return None; + } + let limit = self.fitted(bank).min((32 << 20) << code); + let offset = addr % BANK_SPAN; + (offset < limit && bytes <= limit - offset).then_some((bank as usize, offset)) + } + pub(crate) fn raw_read(&self, bank: usize, at: u64) -> u64 { + self.physical.banks()[bank].read64(at).data + } + pub(crate) fn raw_write(&self, bank: usize, at: u64, value: u64) -> u32 { + let result = self.physical.banks()[bank].write64(at, value); + if result == BUS_OK { + let slot = bank as u64 ^ ((self.config() >> 32) & 3); + self.written(slot * BANK_SPAN + at); + } + result + } + pub(crate) fn zero(&self, bank: usize, size: u64) { + assert!(size <= self.fitted(bank as u64)); + // The bank owns this contiguous mapping; zero has the same representation + // in host word order. Self-test does not synchronize concurrent bank accesses. + unsafe { + std::ptr::write_bytes(self.physical.banks()[bank].host_ptr(), 0, size as usize); + } + #[cfg(feature = "jitv2")] + self.physical.banks()[bank].bump_gen_all(); + #[cfg(feature = "jitv2")] + self.generations.bump_all(&self.extent); + } + pub(crate) fn changed_decode(&self) { + #[cfg(feature = "jitv2")] + self.generations.bump_all(&self.extent); + #[cfg(feature = "jitv2")] + for bank in self.physical.banks() { + bank.bump_gen_all(); + } + #[cfg(all(test, feature = "jitv2"))] + Cost::add(&self.cost.bumps, (0..self.extent.slots()).map(|slot| (self.extent.prefix(slot) >> 12) as usize).sum::() + + self.physical.banks().iter().map(|bank| bank.size() >> 12).sum::()); + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A Tezro takes 512 and 1024 MiB DIMMs, one per bank, none left out + /// between, eight at most. The 4 GiB layout is the default and the 1 GiB + /// one is still constructible. + #[test] + fn only_512_and_1024_mib_banks_are_accepted() { + assert_eq!(MemoryPopulation::default(), MemoryPopulation::four_gib()); + assert_eq!(MemoryPopulation::one_gib().banks_mib(), [512, 512]); + assert_eq!(MemoryPopulation::four_gib().banks_mib(), [1024, 1024, 512, 512, 512, 512]); + assert_eq!(MemoryPopulation::four_gib(), MemoryPopulation::from_mib(&[1024, 1024, 512, 512, 512, 512]).unwrap()); + assert_eq!((MemoryPopulation::one_gib().total_mib(), MemoryPopulation::four_gib().total_mib()), (1024, 4096)); + for bad in [&[][..], &[256], &[2048], &[512, 0], &[768, 512], &[512; 9]] { + assert!(MemoryPopulation::from_mib(bad).is_err(), "{bad:?}"); + } + assert!(MemoryPopulation::from_mib(&[1024; 8]).is_ok()); + assert!(MemoryPopulation::new(&[]).is_err()); + } + + /// The 1 GiB population lies wholly inside ppmem's 4 GiB window; the 4 GiB + /// default does not, since its banks 4 and 5 are above it. The memory model + /// accepts both. + #[test] + fn the_default_fits_the_direct_window_and_the_4_gib_layout_does_not() { + for (population, direct) in [(MemoryPopulation::one_gib(), true), (MemoryPopulation::four_gib(), false)] { + assert_eq!(population.fits_direct_window(), direct); + let end = |bank: usize| bank as u64 * BANK_SPAN + u64::from(population.banks()[bank].mib()) * (1 << 20); + let last = population.banks().len() - 1; + assert_eq!(end(last) <= WINDOW_SIZE, direct, "the last bank ends inside the window"); + assert!(super::super::bedrock_md::BedrockMd::new(&population.banks_mib()).is_ok()); + } + } + + /// One processor needs 512 MB; two or four need 1 GB. Fewer than one or + /// more than four processors is not a Tezro. + #[test] + fn the_memory_must_meet_the_minimum_for_the_processors() { + let half = MemoryPopulation::from_mib(&[512]).unwrap(); + assert!(half.check_for_cpus(1).is_ok()); + for cpus in [2, 3, 4] { + let error = half.check_for_cpus(cpus).unwrap_err(); + assert!(error.contains("1024"), "{error}"); + } + for population in [MemoryPopulation::one_gib(), MemoryPopulation::four_gib()] { + for cpus in [1, 2, 4] { + assert!(population.check_for_cpus(cpus).is_ok()); + } + } + assert!(MemoryPopulation::one_gib().check_for_cpus(0).is_err()); + assert!(MemoryPopulation::one_gib().check_for_cpus(5).is_err()); + // A single 1024 MiB bank meets the two-processor minimum too. + assert!(MemoryPopulation::from_mib(&[1024]).unwrap().check_for_cpus(4).is_ok()); + } +} diff --git a/src/dev/bedrock_ni.rs b/src/dev/bedrock_ni.rs new file mode 100644 index 000000000..bbe67bac6 --- /dev/null +++ b/src/dev/bedrock_ni.rs @@ -0,0 +1,359 @@ +//! Bedrock NI exposes node routing, sideband and error registers. Fuel runs as +//! one node without a NUMAlink peer, but the guest still reads and programs +//! these local control words during machine setup. +//! +//! # What is approximated here +//! +//! Routing and sideband writes are retained; the network fabric and packet +//! delivery are not simulated. Unspecified power-on fields start at zero, and +//! the sideband word survives reset as the current compatibility behavior. +//! +//! Ported from hw/misc/sgi-bedrock-ni.c and hw/misc/sgi-bedrock-ni-regs.c +//! in jamesbraid/qemu-sgi-origami. + +/// Length of the block-relative register aperture. +pub const SIZE: u64 = 0x2008; + +/// Port status bits 1:0 of a node with no cable: bit 1 (link up) clear, bit 0 +/// set. INFERRED, not measured. Bit 1 follows a real Tezro PROM reporting its +/// NUMAlink as down; bit 0 follows a real one-node Fuel taking a PIO bus +/// error with no link message (a 2003 installation kernel). The IRIX 6.5.30 +/// bus-error handler prints a link-reset NOTICE when bits 1:0 read 0 with +/// reset enable 0. A PROM-monitor read of this register on an uncabled node +/// would settle it. +const PORT_STATUS_UNCABLED: u64 = 1; + +pub mod reg { + /// Port status, hub-relative `0x680000` (the first word of the block). + pub const PORT_STATUS: u64 = 0x0; + pub const LOCAL_RESET: u64 = 0x8; + pub const CONTROL_0: u64 = 0x10; + pub const CONTROL_1: u64 = 0x18; + pub const CONTROL_2: u64 = 0x20; + pub const CONTROL_3: u64 = 0x30; + pub const CONTROL_4: u64 = 0x38; + pub const ERROR_STATUS: u64 = 0x100; + pub const ERROR_STATUS_UPPER: u64 = 0x108; + pub const ERROR_STATUS_LOWER: u64 = 0x110; + pub const SIDE_BAND: u64 = 0x118; + pub const ERROR_CLEAR: u64 = 0x120; + pub const ROUTES: u64 = 0x1000; + pub const ROUTES_END: u64 = 0x1400; + pub const GLOBAL_CONTROL: u64 = 0x2000; +} + +use crate::traits::{BusDevice, BusRead64, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::{Mutex, ReentrantMutex}; +use std::sync::Arc; +type Level = Arc; +#[derive(Clone)] +struct Regs { + control: [u64; 8], + routes: [u64; 128], + global: u64, + errors: u64, + sideband: u64, +} +impl Default for Regs { + fn default() -> Self { + Self { + control: [0; 8], + routes: [0; 128], + global: 0, + errors: 0, + sideband: 0, + } + } +} +pub struct BedrockNi { + regs: Arc>, + error: Mutex>, + delivery: ReentrantMutex<()>, + driven: Mutex>, + reset: Mutex>>, +} +impl Default for BedrockNi { + fn default() -> Self { + Self::new() + } +} +impl BedrockNi { + pub fn new() -> Self { + let regs = Arc::new(Mutex::new(Regs::default())); + crate::locks::register_mutex("bedrock-ni", ®s); + Self { + regs, + error: Mutex::new(None), + delivery: ReentrantMutex::new(()), + driven: Mutex::new(None), + reset: Mutex::new(None), + } + } + pub fn connect_error(&self, cb: Level) { + let _delivery = self.delivery.lock(); + *self.error.lock() = Some(cb); + *self.driven.lock() = None; + self.update_irq(); + } + pub fn connect_local_reset(&self, cb: Arc) { + *self.reset.lock() = Some(cb); + } + pub fn error_level(&self) -> bool { + self.regs.lock().errors & 0x1fffffff000000 != 0 + } + fn update_irq(&self) { + // Serialize the sample and callback, while leaving registers available + // to the callback. Suppress unchanged levels so callbacks can reenter + // register reads that recompute the output. + let _delivery = self.delivery.lock(); + let cb = self.error.lock().clone(); + if let Some(cb) = cb { + let level = self.error_level(); + let changed = { + let mut driven = self.driven.lock(); + let changed = *driven != Some(level); + *driven = Some(level); + changed + }; + if changed { + cb(level); + } + } + } + fn mask(a: u64) -> u64 { + match a { + reg::CONTROL_0 => 1, + reg::CONTROL_1 => 7, + reg::CONTROL_2 => 0x1ff, + reg::CONTROL_3 => 0x7efffffff, + reg::CONTROL_4 => 0xffff, + _ => 0, + } + } +} +impl Resettable for BedrockNi { + fn power_on(&self) { + let mut r = self.regs.lock(); + let sideband = r.sideband; + *r = Regs::default(); + r.sideband = sideband; + drop(r); + self.update_irq(); + } +} +impl BusDevice for BedrockNi { + fn read64(&self, a: u64) -> BusRead64 { + if a & 7 != 0 { + return BusRead64::err(); + } + let mut r = self.regs.lock(); + let v = if Self::mask(a) != 0 { + r.control[a as usize / 8] + } else if (reg::ROUTES..reg::ROUTES_END).contains(&a) { + r.routes[(a as usize - 0x1000) / 8] + } else { + match a { + reg::PORT_STATUS => PORT_STATUS_UNCABLED, + reg::ERROR_STATUS => r.errors, + reg::ERROR_STATUS_UPPER | reg::ERROR_STATUS_LOWER => 0, + reg::SIDE_BAND => r.sideband, + reg::ERROR_CLEAR => { + let v = r.errors; + r.errors = 0; + v + } + reg::GLOBAL_CONTROL => r.global, + _ => { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock NI unsupported read {a:#x}"); + 0 + } + } + }; + drop(r); + self.update_irq(); + BusRead64::ok(v) + } + fn write64(&self, a: u64, v: u64) -> u32 { + if a & 7 != 0 { + return BUS_ERR; + } + let mut r = self.regs.lock(); + if Self::mask(a) != 0 { + r.control[a as usize / 8] = v & Self::mask(a); + } else if (reg::ROUTES..reg::ROUTES_END).contains(&a) { + r.routes[(a as usize - 0x1000) / 8] = v & 31; + } else { + match a { + reg::LOCAL_RESET => {} + reg::ERROR_STATUS => r.errors = v & ((1 << 53) - 1), + reg::SIDE_BAND => r.sideband = v & 0xfffff, + reg::GLOBAL_CONTROL => r.global = v & 31, + _ => crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock NI unsupported or read-only write {a:#x} = {v:#x}"), + } + } + drop(r); + self.update_irq(); + if a == reg::LOCAL_RESET && v & 4 != 0 { + let cb = self.reset.lock().clone(); + if let Some(cb) = cb { + cb(); + } + } + BUS_OK + } +} + +#[cfg(test)] +mod tests { + // Cases ported from tests/qtest/sn1-bedrock-test.c in jamesbraid/qemu-sgi-origami. + use super::*; + use std::sync::atomic::{AtomicU64, Ordering}; + + /// The guest programs route fields and reads the error register to acknowledge faults. + #[test] + fn route_registers_store_fields_and_error_reads_clear_pending_errors() { + let d = Arc::new(BedrockNi::new()); + for (a, mask) in [ + (reg::CONTROL_0, 1), + (reg::CONTROL_1, 7), + (reg::CONTROL_2, 0x1ff), + (reg::CONTROL_3, 0x7efffffff), + (reg::CONTROL_4, 0xffff), + ] { + d.write64(a, !0); + assert_eq!(d.read64(a).data, mask); + } + for a in [reg::ROUTES, reg::ROUTES_END - 8] { + d.write64(a, !0); + assert_eq!(d.read64(a).data, 31); + } + d.write64(reg::GLOBAL_CONTROL, !0); + assert_eq!(d.read64(reg::GLOBAL_CONTROL).data, 31); + let weak = Arc::downgrade(&d); + d.connect_error(Arc::new(move |_| { + let d = weak.upgrade().unwrap(); + let _ = d.read64(reg::SIDE_BAND); + })); + let resets = Arc::new(AtomicU64::new(0)); + let count = resets.clone(); + d.connect_local_reset(Arc::new(move || { + count.fetch_add(1, Ordering::SeqCst); + })); + d.write64(reg::CONTROL_0, 0); + d.write64(reg::LOCAL_RESET, 4); + assert_eq!(resets.load(Ordering::SeqCst), 1); + d.write64(reg::ERROR_STATUS, 0xffffff); + assert!(!d.error_level()); + d.write64(reg::ERROR_STATUS, 1 << 24); + assert!(d.error_level()); + d.read64(reg::ERROR_CLEAR); + assert!(!d.error_level()); + d.write64(reg::SIDE_BAND, !0); + d.power_on(); + assert_eq!(d.read64(reg::SIDE_BAND).data, 0xfffff); + assert_eq!(d.read64(reg::CONTROL_3).data, 0); + } + + /// The PROM's one power-on pass writes a sign-extended port-parameter constant and zero + /// to the error and sideband words; each must read back through its defined bits only. + #[test] + fn power_on_programming_pass_reads_back_through_the_defined_bits() { + let d = BedrockNi::new(); + d.write64(reg::CONTROL_0, 1); + d.write64(reg::CONTROL_1, 0); + d.write64(reg::CONTROL_2, 0x166); + d.write64(reg::CONTROL_3, 0xffff_ffff_a3ff_1990); + d.write64(reg::CONTROL_4, 0xc800); + d.write64(reg::ERROR_STATUS, 0); + d.read64(reg::ERROR_CLEAR); + d.write64(reg::SIDE_BAND, 0); + for (a, want) in [ + (reg::CONTROL_0, 1), + (reg::CONTROL_1, 0), + (reg::CONTROL_2, 0x166), + (reg::CONTROL_3, 0x7_a3ff_1990), + (reg::CONTROL_4, 0xc800), + (reg::ERROR_STATUS, 0), + (reg::SIDE_BAND, 0), + (reg::ERROR_CLEAR, 0), + ] { + assert_eq!(d.read64(a).data, want, "register {a:#x}"); + } + } + + /// Only the local-reset bit of the reset word reaches the node; the other bits must + /// not restart it, and the interface's own registers are left to the board's reset. + #[test] + fn only_the_local_reset_bit_requests_a_node_reset() { + let d = BedrockNi::new(); + let resets = Arc::new(AtomicU64::new(0)); + let count = resets.clone(); + d.connect_local_reset(Arc::new(move || { + count.fetch_add(1, Ordering::SeqCst); + })); + let mut expected = 0; + for enable in 0..2 { + d.write64(reg::CONTROL_0, enable); + for command in 0..8u64 { + d.write64(reg::LOCAL_RESET, command); + expected += u64::from(command & 4 != 0); + assert_eq!(resets.load(Ordering::SeqCst), expected, "command {command:#x}"); + } + } + // With nothing wired to the callback the request does not clear this block. + assert_eq!(d.read64(reg::CONTROL_0).data, 1); + } + + /// A latched port error raises the interrupt line through its interrupting bits only; + /// drivers rely on the line dropping when the status is rewritten or cleared. + #[test] + fn stored_error_status_drives_the_error_line_through_the_interrupting_bits() { + let d = BedrockNi::new(); + let level = Arc::new(AtomicU64::new(0)); + let seen = level.clone(); + d.connect_error(Arc::new(move |high| seen.store(u64::from(high), Ordering::SeqCst))); + for (value, high) in [(1u64 << 30, 1), (0xff, 0), (1 << 52, 1), (0, 0)] { + d.write64(reg::ERROR_STATUS, value); + assert_eq!(level.load(Ordering::SeqCst), high, "status {value:#x}"); + } + // Bit 53 is beyond the register and must not be retained. + d.write64(reg::ERROR_STATUS, 1 << 53); + assert_eq!(d.read64(reg::ERROR_STATUS).data, 0); + } + + /// Reset values of the programmable words are unknown on hardware, so the model clears them; + /// a guest that re-runs its setup after a reset must see zeroes, not stale routes. + #[test] + fn reset_clears_every_programmable_register_and_misaligned_accesses_fault() { + let d = BedrockNi::new(); + let offsets = [ + reg::CONTROL_0, reg::CONTROL_1, reg::CONTROL_2, reg::CONTROL_3, reg::CONTROL_4, + reg::ROUTES, reg::ROUTES_END - 8, reg::GLOBAL_CONTROL, + ]; + for a in offsets { + d.write64(a, !0); + } + // A misaligned write must not land in the neighbouring register. + assert_eq!(d.write64(reg::CONTROL_4 + 1, 0), BUS_ERR); + assert_eq!(d.read64(reg::CONTROL_4 + 1).status, BUS_ERR); + assert_eq!(d.read64(reg::CONTROL_4).data, 0xffff); + d.power_on(); + for a in offsets { + assert_eq!(d.read64(a).data, 0, "register {a:#x}"); + } + } + + /// An uncabled one-node port reads status bits 1:0 = 1: link down, bit 0 + /// set. Reading 0 there makes the guest's bus-error handler print a + /// link-reset notice. Bit 18 stays clear (a one-node machine) and a + /// write does not change the value. + #[test] + fn uncabled_port_status_reads_one_and_ignores_writes() { + let d = BedrockNi::new(); + assert_eq!(d.read64(reg::PORT_STATUS).data, 1); + d.write64(reg::PORT_STATUS, !0); + assert_eq!(d.read64(reg::PORT_STATUS).data, 1); + d.power_on(); + assert_eq!(d.read64(reg::PORT_STATUS).data, 1); + } +} diff --git a/src/dev/bedrock_pi.rs b/src/dev/bedrock_pi.rs new file mode 100644 index 000000000..93b736961 --- /dev/null +++ b/src/dev/bedrock_pi.rs @@ -0,0 +1,938 @@ +//! Bedrock PI exposes processor-slot control, pending interrupts, real-time +//! counters, timers and shared graphics registers. A node has two PIs with two +//! processor slots each, A and B; the board names which of the four slots hold +//! a processor ([`Population`]). Fuel fits PI0 slot A only. A Tezro fits one, +//! two or four, filled in the order CPU 0 = PI0/A, CPU 1 = PI1/A, CPU 2 = +//! PI0/B, CPU 3 = PI1/B. Node hooks carry cache-alias changes, CPU enable and +//! NMI events to the board. +//! +//! # What is approximated here +//! +//! The board names which fitted slots have a running processor +//! ([`BedrockPiConfig::running`]); every fitted slot reads present. A +//! running slot starts with its CPU-enable bit set and runs from reset +//! whatever the register later holds: an approximation QEMU's IP35 node +//! makes too, since no cold-reset value for the bit is known and the +//! register is not known to stop a processor. A write that clears a running +//! slot's enable is stored and recorded, not carried out. +//! +//! For a slot without a running processor nothing is delivered, and each +//! kind is recorded the first time it happens ([`Refused`], +//! [`BedrockPi::refused`]) and logged under the `bedrock` module instead of +//! being dropped silently: a write that enables the processor, an NMI, an +//! interrupt output for the slot (an interprocessor interrupt is one) and a +//! timer enable. The enable bit itself is stored, as the register holds it +//! on hardware, and still reaches the node hooks. +//! +//! The CPU-number register answers the reading processor's slice: its slot +//! in bit 0 and the interface in bit 1. A read from a thread that runs no +//! processor (a device, the monitor, a test) answers slot A, as QEMU does +//! for an access with no issuing processor; that is model behavior, since +//! hardware always has a requester. Which processor is in which slot is +//! the board's order ([`Population::slot_of`]). +//! +//! The model accepts Linux's public `SGIrules` key at offset 0x8 for +//! compatibility. Linux documents the literal for Hub I/O, but the available +//! evidence does not establish that this PI offset maps to the same register. +//! Unconnected graphics registers read as zero and are logged as unsupported. +//! +//! Ported from hw/misc/sgi-bedrock-pi.c in jamesbraid/qemu-sgi-origami. + +use crate::dev::bedrock_io::IIO_PROTECT_OVRRD_KEY; +use crate::dev::heart::Clock; +use crate::hptimer::{TimerId, TimerManager, TimerReturn}; +use crate::irq::InterruptController; +use crate::traits::{BusDevice, BusRead32, BusRead64, Device, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::{Mutex, ReentrantMutex}; +use std::sync::{Arc, OnceLock, Weak}; +use std::time::Duration; +const RT_MASK: u64 = (1 << 55) - 1; +const ERROR_MASK: u64 = (1 << 35) - 1; +const MEMORY_ERROR: u64 = 1 << 24; +const REFLECT: u64 = (1 << 54) | (1 << 55) | (1 << 62) | (1 << 63); +const TICK: u64 = 800; +pub const SIZE: u64 = 0x30108; + +pub mod reg { + pub const NODE_REGION: u64 = 0x0; + pub const PROTECTION_KEY: u64 = 0x8; + pub const OFFSET_0010: u64 = 0x10; + pub const OFFSET_0018: u64 = 0x18; + pub const INTERFACE_ID: u64 = 0x20; + pub const CACHED_ALIAS: u64 = 0x28; + pub const OFFSET_0030: u64 = 0x30; + pub const OFFSET_0038: u64 = 0x38; + pub const INTERFACE_SLOT_A: u64 = 0x40; + pub const INTERFACE_SLOT_B: u64 = 0x48; + pub const CPU_A_ENABLE: u64 = 0x50; + pub const CPU_B_ENABLE: u64 = 0x58; + pub const NMI_SLOT_A: u64 = 0x70; + pub const NMI_SLOT_B: u64 = 0x78; + pub const PENDING_MODIFY: u64 = 0x90; + pub const PENDING_STATUS_A: u64 = 0x98; + pub const PENDING_STATUS_B: u64 = 0xa0; + pub const OFFSET_00A8: u64 = 0xa8; + pub const OFFSET_00B0: u64 = 0xb0; + pub const OFFSET_00B8: u64 = 0xb8; + pub const OFFSET_00C0: u64 = 0xc0; + pub const REQUEST_A: u64 = 0xc8; + pub const REQUEST_B: u64 = 0xd0; + pub const PENDING_A_CLEAR: u64 = 0xd8; + pub const PENDING_B_CLEAR: u64 = 0xe0; + pub const OFFSET_00E8: u64 = 0xe8; + pub const OFFSET_00F0: u64 = 0xf0; + pub const RT_COMPARE_A: u64 = 0x108; + pub const RT_COMPARE_B: u64 = 0x110; + pub const PROFILE_COMPARE: u64 = 0x118; + pub const RT_STATUS_A: u64 = 0x120; + pub const RT_STATUS_B: u64 = 0x128; + pub const PROFILE_STATUS_A: u64 = 0x130; + pub const PROFILE_STATUS_B: u64 = 0x138; + pub const RT_ENABLE_A: u64 = 0x140; + pub const RT_ENABLE_B: u64 = 0x148; + pub const PROFILE_ENABLE_A: u64 = 0x150; + pub const PROFILE_ENABLE_B: u64 = 0x158; + pub const PENDING_MODIFY_B: u64 = 0x180; + pub const ERROR_SUMMARY_ALIAS: u64 = 0x3f8; + pub const ERROR_SUMMARY: u64 = 0x400; + pub const ERROR_CAUSE_A: u64 = 0x408; + pub const ERROR_CAUSE_B: u64 = 0x410; + pub const ERROR_ADDRESS_A: u64 = 0x418; + pub const ERROR_ADDRESS_B: u64 = 0x420; + pub const ERROR_CONTROL: u64 = 0x428; + pub const ERROR_CAPTURE_A: u64 = 0x438; + pub const ERROR_CAPTURE_B: u64 = 0x448; + pub const ERROR_CAPTURE_C: u64 = 0x458; + pub const ERROR_CAPTURE_D: u64 = 0x468; + pub const ERROR_SOURCE_A: u64 = 0x490; + pub const ERROR_SOURCE_B: u64 = 0x4c0; + pub const ERROR_DETAIL_A: u64 = 0x4a8; + pub const ERROR_DETAIL_B: u64 = 0x4b0; + pub const ERROR_DETAIL_C: u64 = 0x4b8; + pub const OFFSET_0430: u64 = 0x430; + pub const OFFSET_0440: u64 = 0x440; + pub const OFFSET_0450: u64 = 0x450; + pub const OFFSET_0460: u64 = 0x460; + pub const OFFSET_04C8: u64 = 0x4c8; + pub const OFFSET_04D0: u64 = 0x4d0; + pub const CREDIT_MODE: u64 = 0x20068; + pub const COUNTER: u64 = 0x30100; + pub const GRAPHICS_BASE: u64 = 0x300; + pub const GRAPHICS_END: u64 = 0x350; +} + +/// The processor slots a board fits, as one bit per slot: bit `interface * 2 +/// + slot`, with slot 0 = A and slot 1 = B. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Population(u8); +impl Default for Population { + fn default() -> Self { + Self::SINGLE + } +} +impl Population { + /// One processor, in PI0 slot A. + pub const SINGLE: Self = Self(0b0001); + /// Slots named by `mask`. CPU 0 runs in PI0/A, so it must be fitted, and + /// only the four slots exist. + pub fn new(mask: u8) -> Result { + if mask & !0b1111 != 0 { + return Err(format!("processor mask {mask:#x} names a slot outside the node's four")); + } + if mask & 1 == 0 { + return Err(format!("processor mask {mask:#x} leaves PI0 slot A empty, where CPU 0 runs")); + } + Ok(Self(mask)) + } + /// `count` processors in the board's order: PI0/A, PI1/A, PI0/B, PI1/B. + pub fn fitted_cpus(count: usize) -> Result { + if !(1..=4).contains(&count) { + return Err(format!("{count} processors; a node fits 1 to 4")); + } + Self::new(ORDER[..count].iter().fold(0, |mask, bit| mask | (1 << bit))) + } + /// The slot (interface, 0 = A or 1 = B) CPU `cpu` sits in: the board puts + /// CPU n in the nth slot of this set in its order, one processor per + /// interface before a second on either. + pub fn slot_of(self, cpu: usize) -> Option<(u8, usize)> { + ORDER.iter().filter(|&&bit| self.0 & (1 << bit) != 0).nth(cpu).map(|&bit| (bit / 2, usize::from(bit % 2))) + } + /// The CPU in `slot` of `interface`, if this set holds one. + pub fn cpu_in(self, interface: u8, slot: usize) -> Option { + (0..self.count()).find(|&cpu| self.slot_of(cpu) == Some((interface, slot))) + } + pub fn mask(self) -> u8 { + self.0 + } + pub fn count(self) -> usize { + self.0.count_ones() as usize + } + /// Whether a processor is fitted in `slot` (0 = A, 1 = B) of `interface`. + pub fn fitted(self, interface: u8, slot: usize) -> bool { + self.0 & (1 << (usize::from(interface) * 2 + slot)) != 0 + } +} +/// The board's population order, as physical slot numbers `interface * 2 + +/// slot`: PI0/A, PI1/A, PI0/B, PI1/B. +const ORDER: [u8; 4] = [0, 2, 1, 3]; + +#[derive(Clone, Copy, Debug, Default)] +pub struct BedrockPiConfig { + pub interface: u8, + pub request_region: u8, + pub population: Population, + /// The fitted slots whose processor runs. The Fuel runs PI0/A alone. + pub running: Population, +} +/// Something a processor slot would do or receive that the model does not +/// carry out because no processor runs there. See the module notes. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Refused { + /// A write set the slot's CPU-enable bit. + Start { interface: u8, slot: usize }, + /// A write cleared the CPU-enable bit of a slot whose processor runs; it + /// keeps running. + Stop { interface: u8, slot: usize }, + /// A non-maskable interrupt the node did not take to a processor. + Nmi { interface: u8, slot: usize }, + /// An interrupt output of the slot went high: output line 0..9 of the PI. + Interrupt { interface: u8, line: usize }, + /// A real-time or profile timer interrupt was enabled for the slot. + Timer { interface: u8, slot: usize }, +} +/// Called outside the state lock. Graphics indices 0..9 are the shared PI +/// graphics registers at offsets 0x300..0x348. Disconnected access is observable. +pub trait NodeHooks: Send + Sync { + fn cached_alias_changed(&self, _size: u8) {} + fn rt_compare_written(&self, _interface: u8, _slot: usize) {} + fn rt_counter_read(&self, _interface: u8) {} + fn cpu_enable(&self, _interface: u8, _slot: usize, _enabled: bool) {} + /// Whether the node took the NMI to its processor. No processor model + /// here has an NMI entry, so by default it is not taken. + fn nmi(&self, _interface: u8, _slot: usize) -> bool { + false + } + fn gfx_read(&self, _index: usize) -> Option { + None + } + fn gfx_write(&self, _index: usize, _value: u64) -> bool { + false + } + fn gfx_reset(&self) {} + /// Software cleared graphics interrupt bit 1 + `slot` of bank 0 through + /// the pending-modify register: the processor took it. + fn gfx_interrupt_taken(&self, _slot: usize) {} +} +struct State { + w: [u64; 160], + pending: [u64; 2], + reflected: u64, + uart: bool, + base: u64, + epoch: u64, + paused: Option, + running: bool, + rt_due: [u128; 2], + profile_due: u128, + levels: [u8; 256], + rearm: [u64; 128], + credit_mode: u64, +} +impl State { + fn new(now: u64) -> Self { + let mut s = Self { + w: [0; 160], + pending: [0; 2], + reflected: 0, + uart: false, + base: 0, + epoch: now, + paused: Some(now), + running: false, + rt_due: [0; 2], + profile_due: 0, + levels: [0; 256], + rearm: [0; 128], + credit_mode: 0, + }; + s.w[0] = !0; + s.w[2] = !0; + s.w[3] = !0; + s.arm_all(now); + s + } + fn time(&self, now: u64) -> u64 { + self.paused.unwrap_or(now) + } + fn count(&self, now: u64) -> u64 { + self.base + .wrapping_add(self.time(now).saturating_sub(self.epoch) / TICK) + & RT_MASK + } + fn arm_rt(&mut self, slot: usize, now: u64) { + let now = self.time(now); + let ticks = (self.w[0x108 / 8 + slot] + .wrapping_sub(self.count(now)) + .wrapping_sub(1) + & RT_MASK) + + 1; + self.rt_due[slot] = u128::from(now - now.saturating_sub(self.epoch) % TICK) + + u128::from(ticks) * u128::from(TICK); + } + fn arm_profile(&mut self, now: u64) { + let now = self.time(now); + let ticks = (self.w[0x118 / 8] as u32).wrapping_sub(self.count(now) as u32); + let ticks = if ticks == 0 { + 1u64 << 32 + } else { + u64::from(ticks) + }; + self.profile_due = u128::from(now - now.saturating_sub(self.epoch) % TICK) + + u128::from(ticks) * u128::from(TICK); + } + fn arm_all(&mut self, now: u64) { + self.arm_rt(0, now); + self.arm_rt(1, now); + self.arm_profile(now); + } + fn sync(&mut self, now: u64) { + let now = self.time(now); + for slot in 0..2 { + if u128::from(now) >= self.rt_due[slot] { + self.w[0x120 / 8 + slot] = 1; + self.arm_rt(slot, now); + } + } + if u128::from(now) >= self.profile_due { + self.w[0x130 / 8] = 1; + self.w[0x138 / 8] = 1; + self.arm_profile(now); + } + for level in 7..128 { + if self.rearm[level] != 0 && self.rearm[level] <= now { + self.rearm[level] = 0; + if self.levels.iter().any(|&v| usize::from(v) == level + 1) { + self.pending[level / 64] |= 1 << (level % 64); + } + } + } + } + fn pending(&self, bank: usize) -> u64 { + if bank == 1 { + (self.pending[1] & !REFLECT) | self.reflected + } else { + self.pending[0] + | if self.uart { 1 << 4 } else { 0 } + | if self.w[0xc8 / 8] & self.w[0xe8 / 8] != 0 { + 1 << 5 + } else { + 0 + } + | if self.w[0xd0 / 8] & self.w[0xe8 / 8] != 0 { + 1 << 6 + } else { + 0 + } + } + } + fn outputs(&self) -> [bool; 10] { + let mut out = [false; 10]; + for slot in 0..2 { + out[slot * 5] = self.pending(0) & self.w[0xa8 / 8 + slot * 2] != 0; + out[slot * 5 + 1] = self.pending(1) & self.w[0xb0 / 8 + slot * 2] != 0; + out[slot * 5 + 2] = self.w[0x120 / 8 + slot] & self.w[0x140 / 8 + slot] != 0; + out[slot * 5 + 3] = self.w[0x130 / 8 + slot] & self.w[0x150 / 8 + slot] != 0; + out[slot * 5 + 4] = self.w[0x400 / 8] & self.w[0x408 / 8 + slot] != 0; + } + out + } + fn modify(&mut self, value: u64, reissue: bool, now: u64) { + let bank = ((value >> 6) & 1) as usize; + let bit = value & 63; + let mask = 1 << bit; + if bank == 1 && mask & REFLECT != 0 { + return; + } + if value & 0x100 != 0 { + self.pending[bank] |= mask; + } else { + self.pending[bank] &= !mask; + let level = bank * 64 + bit as usize; + if reissue && self.levels.iter().any(|&v| usize::from(v) == level + 1) { + self.rearm[level] = now.saturating_add(1_000_000); + } + } + } +} +pub struct BedrockPi { + cfg: BedrockPiConfig, + clock: Clock, + state: Mutex, + hooks: Mutex>>, + irq: Mutex>>, + timers: OnceLock>, + timer: Mutex>, + timer_pass: Mutex<()>, + dispatch: ReentrantMutex<()>, + driven: Mutex<[Option; 10]>, + weak: Weak, + refused: Mutex>, +} +impl BedrockPi { + pub fn new(cfg: BedrockPiConfig, clock: Clock) -> Arc { + assert!(cfg.interface < 2 && cfg.request_region < 64); + // The running processor starts with its bus grant; the rest are held + // disabled until a guest enables them. + assert_eq!(cfg.running.mask() & !cfg.population.mask(), 0, "a running slot is fitted"); + let mut state = State::new(clock()); + for slot in 0..2 { + state.w[reg::CPU_A_ENABLE as usize / 8 + slot] = u64::from(cfg.running.fitted(cfg.interface, slot)); + } + Arc::new_cyclic(|weak| Self { + cfg, + state: Mutex::new(state), + clock, + hooks: Mutex::new(None), + irq: Mutex::new(None), + timers: OnceLock::new(), + timer: Mutex::new(None), + timer_pass: Mutex::new(()), + dispatch: ReentrantMutex::new(()), + driven: Mutex::new([None; 10]), + weak: weak.clone(), + refused: Mutex::new(Vec::new()), + }) + } + pub fn set_node_hooks(&self, hooks: Arc) { + let _dispatch = self.dispatch.lock(); + *self.hooks.lock() = Some(hooks); + } + pub fn set_timer_manager(&self, tm: Arc) { + let _ = self.timers.set(tm); + self.schedule(); + } + /// Lines 0..4 are slot A IP2..IP6; lines 5..9 are slot B IP2..IP6. + pub fn set_interrupt_controller(&self, irq: Arc) { + let _dispatch = self.dispatch.lock(); + *self.irq.lock() = Some(irq); + *self.driven.lock() = [None; 10]; + self.drive(); + } + /// Whether a processor runs in `slot` of this interface. + fn started(&self, slot: usize) -> bool { + self.cfg.running.fitted(self.cfg.interface, slot) + } + /// The CPU-number register: the reading processor's slot in bit 0 and + /// this interface in bit 1; slot A for a reader that is no processor + /// here. + fn cpu_number(&self) -> u64 { + let interface = self.cfg.interface; + let slot = crate::cpu::smp::on_cpu_thread() + .and_then(|cpu| self.cfg.running.slot_of(cpu)) + .filter(|&(i, _)| i == interface) + .map_or(0, |(_, slot)| slot as u64); + slot | u64::from(interface) << 1 + } + fn hooks(&self) -> Option> { + self.hooks.lock().clone() + } + /// Report an access to a register the model does not implement. + fn note(&self, off: u64, write: bool) { + crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock PI{} unsupported {} at {off:#x}", self.cfg.interface, + if write { "write" } else { "read" }); + } + /// What the model could not carry out for a processor slot, with how many + /// times each happened. Each kind is logged once, on its first occurrence. + pub fn refused(&self) -> Vec<(Refused, u64)> { + self.refused.lock().clone() + } + fn refuse(&self, what: Refused) { + let mut refused = self.refused.lock(); + if let Some((_, count)) = refused.iter_mut().find(|(seen, _)| *seen == what) { + *count += 1; + } else if refused.len() < 64 { + refused.push((what, 1)); + crate::dlog!(crate::devlog::LogModule::Bedrock, + "Bedrock PI{}: not carried out: {what:?}", self.cfg.interface); + } + } + fn drive(&self) { + // Serialize publication, including reentrant reads by a controller. + // Publish one line at a time: a callback may change another line. + let _dispatch = self.dispatch.lock(); + let irq = self.irq.lock().clone(); + for line in 0..10 { + if self.state.lock().outputs()[line] && !self.started(line / 5) { + self.refuse(Refused::Interrupt { interface: self.cfg.interface, line }); + } + } + if let Some(irq) = irq { + for line in 0..10 { + let level = self.state.lock().outputs()[line]; + let changed = { + let mut driven = self.driven.lock(); + let changed = driven[line] != Some(level); + driven[line] = Some(level); + changed + }; + if changed { + irq.set_line(line as u32, level); + } + } + } + } + fn schedule(&self) { + let _pass = self.timer_pass.lock(); + let Some(tm) = self.timers.get() else { return }; + let mut timer = self.timer.lock(); + if let Some(id) = timer.take() { + tm.remove(id); + } + let s = self.state.lock(); + if !s.running { + return; + } + let next = s + .rt_due + .iter() + .copied() + .chain([s.profile_due]) + .chain(s.rearm.iter().filter(|&&v| v != 0).map(|&v| u128::from(v))) + .min() + .unwrap(); + let delay = next + .saturating_sub(u128::from((self.clock)())) + .min(1_000_000_000) + .max(1) as u64; + drop(s); + *timer = Some( + tm.add_one_shot(Duration::from_nanos(delay), self.weak.clone(), |weak| { + if let Some(pi) = weak.upgrade() { + { + let _pass = pi.timer_pass.lock(); + let mut s = pi.state.lock(); + if !s.running { + return TimerReturn::Delete; + } + s.sync((pi.clock)()); + } + pi.drive(); + pi.schedule(); + } + TimerReturn::Delete + }), + ); + } + pub fn poll(&self) { + self.state.lock().sync((self.clock)()); + self.drive(); + } + pub fn set_pending(&self, bank: usize, bit: u8) { + if bank < 2 && bit < 64 { + self.state.lock().pending[bank] |= 1 << bit; + self.drive(); + } + } + fn reflected(&self, bit: u8, level: bool) { + let mut s = self.state.lock(); + if level { + s.reflected |= 1 << bit; + } else { + s.reflected &= !(1 << bit); + } + drop(s); + self.drive(); + } + pub fn set_lb_error(&self, level: bool) { + self.reflected(55, level); + } + pub fn set_ni_error(&self, level: bool) { + self.reflected(62, level); + } + pub fn set_xb_error(&self, level: bool) { + self.reflected(54, level); + } + pub fn set_uart_irq(&self, level: bool) { + self.state.lock().uart = level; + self.drive(); + } + pub fn set_memory_uncorrectable(&self) { + self.state.lock().w[0x400 / 8] |= MEMORY_ERROR; + self.drive(); + } + pub fn set_level_source(&self, source: usize, level: u8, active: bool) { + if source < 256 && (!active || (7..=127).contains(&level)) { + self.state.lock().levels[source] = if active { level + 1 } else { 0 }; + self.drive(); + } + } + pub fn io_modify(&self, value: u64, region: u8) -> bool { + let mut s = self.state.lock(); + if region >= 64 || s.w[2] & (1 << region) == 0 { + drop(s); + self.note(0x90, true); + return false; + } + let now = s.time((self.clock)()); + s.modify(value, false, now); + drop(s); + self.drive(); + true + } + fn read(&self, off: u64) -> u64 { + if (reg::GRAPHICS_BASE..reg::GRAPHICS_END).contains(&off) { + if let Some(v) = self + .hooks() + .and_then(|h| h.gfx_read(((off - 0x300) / 8) as usize)) + { + return v; + } + self.note(off, false); + return 0; + } + let mut s = self.state.lock(); + s.sync((self.clock)()); + let value = match off { + reg::INTERFACE_ID => self.cpu_number(), + reg::INTERFACE_SLOT_A => u64::from(self.cfg.population.fitted(self.cfg.interface, 0)), + reg::INTERFACE_SLOT_B => u64::from(self.cfg.population.fitted(self.cfg.interface, 1)), + reg::PENDING_STATUS_A => s.pending(0), + reg::PENDING_STATUS_B => s.pending(1), + reg::PENDING_A_CLEAR => s.w[reg::REQUEST_A as usize / 8], + reg::PENDING_B_CLEAR => s.w[reg::REQUEST_B as usize / 8], + reg::ERROR_SUMMARY_ALIAS => s.w[reg::ERROR_SUMMARY as usize / 8], + reg::ERROR_CAPTURE_A | reg::ERROR_CAPTURE_B | reg::ERROR_CAPTURE_C | reg::ERROR_CAPTURE_D => s.w[(off - reg::PROTECTION_KEY) as usize / 8], + reg::CREDIT_MODE => s.credit_mode, + reg::COUNTER => s.count((self.clock)()), + _ if readable(off) => s.w[off as usize / 8], + _ => { + self.note(off, false); + 0 + } + }; + drop(s); + self.drive(); + if off == reg::COUNTER { + if let Some(h) = self.hooks() { + h.rt_counter_read(self.cfg.interface); + } + } + value + } + fn write(&self, off: u64, value: u64) { + // Keep register changes and their node notifications in one order. + // The state lock still drops before callbacks; dispatch is reentrant + // so a node callback can read registers and recompute IRQ outputs. + let _dispatch = self.dispatch.lock(); + if (reg::GRAPHICS_BASE..reg::GRAPHICS_END).contains(&off) { + if !self + .hooks() + .is_some_and(|h| h.gfx_write(((off - 0x300) / 8) as usize, value)) + { + self.note(off, true); + } + return; + } + let now = (self.clock)(); + let mut s = self.state.lock(); + s.sync(now); + let mut taken = None; + let mut alias = None; + let mut compare = None; + let mut enable = None; + let mut nmi = None; + let mut refused = Vec::new(); + match off { + reg::NODE_REGION | reg::OFFSET_0010 | reg::OFFSET_0018 | reg::OFFSET_00A8 | reg::OFFSET_00B0 | reg::OFFSET_00B8 | reg::OFFSET_00C0 | reg::OFFSET_00E8 | reg::OFFSET_04C8 | reg::OFFSET_04D0 => { + s.w[off as usize / 8] = value + } + reg::PROTECTION_KEY => { + if value == IIO_PROTECT_OVRRD_KEY { + s.w[0] |= 1 << self.cfg.request_region; + } + } + reg::CACHED_ALIAS => { + if s.w[5] != value & 15 { + s.w[5] = value & 15; + alias = Some((value & 15) as u8); + } + } + reg::OFFSET_0030 => s.w[6] = value & 255, + reg::OFFSET_0038 => s.w[7] = value & 0xffffff, + reg::CPU_A_ENABLE | reg::CPU_B_ENABLE => { + let slot = ((off - 0x50) / 8) as usize; + s.w[off as usize / 8] = value & 1; + enable = Some((slot, value & 1 != 0)); + if value & 1 != 0 && !self.started(slot) { + refused.push(Refused::Start { interface: self.cfg.interface, slot }); + } else if value & 1 == 0 && self.started(slot) { + refused.push(Refused::Stop { interface: self.cfg.interface, slot }); + } + } + reg::NMI_SLOT_A | reg::NMI_SLOT_B => { + if value & 1 != 0 { + nmi = Some(((off - 0x70) / 8) as usize); + } else { + self.note(off, true); + } + } + reg::PENDING_MODIFY | reg::PENDING_MODIFY_B => { + let now = s.time(now); + s.modify(value, true, now); + // A clear (bit 8) of bank 0 (bit 6) of a graphics bit. + if value & 0x140 == 0 && matches!(value & 0x3f, 1 | 2) { + taken = Some((value & 0x3f) as usize - 1); + } + } + reg::REQUEST_A | reg::REQUEST_B => s.w[off as usize / 8] |= 1 << self.cfg.request_region, + reg::PENDING_A_CLEAR | reg::PENDING_B_CLEAR => s.w[(off - reg::OFFSET_0010) as usize / 8] &= !value, + reg::OFFSET_00F0 => { + s.w[0xf0 / 8] = value & 31; + if value & 31 != 0 { + self.note(off, true); + } + } + reg::RT_COMPARE_A | reg::RT_COMPARE_B => { + let slot = ((off - 0x108) / 8) as usize; + s.w[off as usize / 8] = value & RT_MASK; + s.arm_rt(slot, now); + compare = Some(slot); + } + reg::PROFILE_COMPARE => { + s.w[0x118 / 8] = value & 0xffffffff; + s.arm_profile(now); + } + reg::RT_STATUS_A | reg::RT_STATUS_B | reg::PROFILE_STATUS_A | reg::PROFILE_STATUS_B => s.w[off as usize / 8] = 0, + reg::RT_ENABLE_A | reg::RT_ENABLE_B | reg::PROFILE_ENABLE_A | reg::PROFILE_ENABLE_B => { + s.w[off as usize / 8] = value & 1; + let slot = usize::from(matches!(off, reg::RT_ENABLE_B | reg::PROFILE_ENABLE_B)); + if value & 1 != 0 && !self.started(slot) { + refused.push(Refused::Timer { interface: self.cfg.interface, slot }); + } + } + reg::ERROR_SUMMARY_ALIAS => { + s.w[0x400 / 8] = + (s.w[0x400 / 8] & MEMORY_ERROR) | (value & ERROR_MASK & !MEMORY_ERROR) + } + reg::ERROR_SUMMARY => s.w[reg::ERROR_SUMMARY as usize / 8] &= !(value & ERROR_MASK), + reg::ERROR_CAUSE_A | reg::ERROR_CAUSE_B => s.w[off as usize / 8] = value & ERROR_MASK, + reg::ERROR_ADDRESS_A | reg::ERROR_ADDRESS_B => s.w[off as usize / 8] = value & 0x1fffffff8, + reg::ERROR_CONTROL => { + s.w[0x428 / 8] = value & 15; + if value & 15 != 0 { + self.note(off, true); + } + } + reg::ERROR_CAPTURE_A | reg::ERROR_CAPTURE_B | reg::ERROR_CAPTURE_C | reg::ERROR_CAPTURE_D => { + let slot = usize::from(off >= 0x450); + s.w[0x430 / 8 + slot * 4] = 0; + s.w[0x440 / 8 + slot * 4] = 0; + } + reg::ERROR_SOURCE_A | reg::ERROR_SOURCE_B => s.w[off as usize / 8] = value & 63, + reg::ERROR_DETAIL_A | reg::ERROR_DETAIL_B => s.w[off as usize / 8] = value & 0x1fffff, + reg::ERROR_DETAIL_C => s.w[off as usize / 8] = value & 0xfffff, + reg::CREDIT_MODE => s.credit_mode = value & 1, + reg::COUNTER => { + s.base = value & RT_MASK; + s.epoch = s.time(now); + s.arm_all(now); + } + _ => self.note(off, true), + } + drop(s); + let hooks = self.hooks(); + for what in refused { + self.refuse(what); + } + if let Some(h) = &hooks { + if let Some(size) = alias { + h.cached_alias_changed(size); + } + if let Some(slot) = compare { + h.rt_compare_written(self.cfg.interface, slot); + } + if let Some((slot, value)) = enable { + h.cpu_enable(self.cfg.interface, slot, value); + } + // An NMI no node takes to a running processor is recorded, not lost. + if let Some(slot) = nmi { + if !(h.nmi(self.cfg.interface, slot) && self.started(slot)) { + self.refuse(Refused::Nmi { interface: self.cfg.interface, slot }); + } + } + if let Some(slot) = taken { + h.gfx_interrupt_taken(slot); + } + } + if let (None, Some(slot)) = (&hooks, nmi) { + self.refuse(Refused::Nmi { interface: self.cfg.interface, slot }); + } + self.drive(); + self.schedule(); + } +} +fn readable(off: u64) -> bool { + matches!( + off, + reg::NODE_REGION | reg::OFFSET_0010 + | reg::OFFSET_0018 + | reg::CACHED_ALIAS + | reg::OFFSET_0030 + | reg::OFFSET_0038 + | reg::CPU_A_ENABLE + | reg::CPU_B_ENABLE + | reg::OFFSET_00A8 + | reg::OFFSET_00B0 + | reg::OFFSET_00B8 + | reg::OFFSET_00C0 + | reg::REQUEST_A + | reg::REQUEST_B + | reg::OFFSET_00E8 + | reg::OFFSET_00F0 + | reg::RT_COMPARE_A + | reg::RT_COMPARE_B + | reg::PROFILE_COMPARE + | reg::RT_STATUS_A + | reg::RT_STATUS_B + | reg::PROFILE_STATUS_A + | reg::PROFILE_STATUS_B + | reg::RT_ENABLE_A + | reg::RT_ENABLE_B + | reg::PROFILE_ENABLE_A + | reg::PROFILE_ENABLE_B + | reg::ERROR_SUMMARY + | reg::ERROR_CAUSE_A + | reg::ERROR_CAUSE_B + | reg::ERROR_ADDRESS_A + | reg::ERROR_ADDRESS_B + | reg::ERROR_CONTROL + | reg::OFFSET_0430 + | reg::OFFSET_0440 + | reg::OFFSET_0450 + | reg::OFFSET_0460 + | reg::ERROR_SOURCE_A + | reg::ERROR_DETAIL_A + | reg::ERROR_DETAIL_B + | reg::ERROR_DETAIL_C + | reg::ERROR_SOURCE_B + | reg::OFFSET_04C8 + | reg::OFFSET_04D0 + ) +} +impl BusDevice for BedrockPi { + fn read64(&self, addr: u64) -> BusRead64 { + if addr < SIZE && addr & 7 == 0 { + BusRead64::ok(self.read(addr)) + } else { + self.note(addr, false); + BusRead64::err() + } + } + fn read32(&self, addr: u64) -> BusRead32 { + if addr == reg::COUNTER || addr == reg::COUNTER + 4 { + let value = self.read(reg::COUNTER); + BusRead32::ok(if addr & 4 == 0 { + (value >> 32) as u32 + } else { + value as u32 + }) + } else { + self.note(addr, false); + BusRead32::err() + } + } + fn write64(&self, addr: u64, value: u64) -> u32 { + if addr < SIZE && addr & 7 == 0 { + self.write(addr, value); + BUS_OK + } else { + self.note(addr, true); + BUS_ERR + } + } +} +impl Device for BedrockPi { + fn start(&self) { + let now = (self.clock)(); + let mut s = self.state.lock(); + if s.running { + return; + } + if let Some(paused) = s.paused.take() { + let delta = now.saturating_sub(paused); + s.epoch = s.epoch.saturating_add(delta); + for due in &mut s.rt_due { + *due += u128::from(delta); + } + s.profile_due += u128::from(delta); + for due in &mut s.rearm { + if *due != 0 { + *due = due.saturating_add(delta); + } + } + } + s.running = true; + drop(s); + self.schedule(); + } + fn stop(&self) { + { + let _pass = self.timer_pass.lock(); + let mut s = self.state.lock(); + s.sync((self.clock)()); + s.paused = Some(s.time((self.clock)())); + s.running = false; + if let Some(id) = self.timer.lock().take() { + if let Some(tm) = self.timers.get() { + tm.remove(id); + } + } + } + self.drive(); + } + fn is_running(&self) -> bool { + self.state.lock().running + } + fn step(&self, _cycles: u64) { + self.poll(); + } + fn get_clock(&self) -> u64 { + self.state.lock().count((self.clock)()) + } +} +impl Resettable for BedrockPi { + fn power_on(&self) { + let _dispatch = self.dispatch.lock(); + let now = (self.clock)(); + let mut s = self.state.lock(); + // These controls and captures survive the PI block's local reset. + let old = s.w; + let running = s.running; + let reflected = s.reflected; + let uart = s.uart; + *s = State::new(now); + for off in [ + 0, 0x10, 0x18, 0x28, 0x50, 0x58, 0x400, 0x418, 0x420, 0x428, 0x430, 0x440, 0x450, + 0x460, 0x4c8, 0x4d0, + ] { + s.w[off / 8] = old[off / 8]; + } + s.running = running; + s.paused = if running { None } else { Some(now) }; + s.reflected = reflected; + s.uart = uart; + drop(s); + if let Some(h) = self.hooks() { + h.gfx_reset(); + for slot in 0..2 { + let enabled = self.state.lock().w[0x50 / 8 + slot] != 0; + h.cpu_enable(self.cfg.interface, slot, enabled); + } + } + self.drive(); + self.schedule(); + } +} +#[cfg(test)] +mod tests; diff --git a/src/dev/bedrock_pi/tests.rs b/src/dev/bedrock_pi/tests.rs new file mode 100644 index 000000000..d77d0e230 --- /dev/null +++ b/src/dev/bedrock_pi/tests.rs @@ -0,0 +1,521 @@ +// Cases follow tests/qtest/sn1-bedrock-pi-test.c in jamesbraid/qemu-sgi-origami: +// power-on-values, local-reset, pending-reflection, real-time-comparators, +// profile-counter-wrap, processor-presence, sparse-processor-presence, +// invalid-processor-masks, cpu-number-fallback, sparse-irq-and-nmi-slots and nmi. + +use super::*; +use std::sync::atomic::{AtomicU64, Ordering}; +fn setup() -> (Arc, Arc) { + let now = Arc::new(AtomicU64::new(0)); + let clock = now.clone(); + let pi = BedrockPi::new( + BedrockPiConfig::default(), + Arc::new(move || clock.load(Ordering::SeqCst)), + ); + pi.start(); + (pi, now) +} +fn r(pi: &BedrockPi, off: u64) -> u64 { + pi.read64(off).data +} +/// The PROM reads PI slot presence and relies on the documented reset survivors. +#[test] +fn the_pi_reports_interface_state_and_retains_registers_across_reset() { + { + let (pi, _) = setup(); + assert_eq!(r(&pi, reg::INTERFACE_ID), 0); + assert_eq!(r(&pi, reg::INTERFACE_SLOT_A), 1); + assert_eq!(r(&pi, reg::INTERFACE_SLOT_B), 0); + pi.write64(reg::NODE_REGION, 0); + pi.write64(reg::PROTECTION_KEY, 0x53474972756c6573); + assert_eq!(r(&pi, reg::NODE_REGION), 1); + let absent = BedrockPi::new( + BedrockPiConfig { + interface: 1, + request_region: 0, + population: Population::SINGLE, + running: Population::SINGLE, + }, + Arc::new(|| 0), + ); + assert_eq!(r(&absent, reg::INTERFACE_ID), 2); + assert_eq!(r(&absent, reg::INTERFACE_SLOT_A), 0); + assert_eq!(pi.read32(reg::INTERFACE_ID).status, BUS_ERR); + assert_eq!(pi.write32(reg::COUNTER, 0), BUS_ERR); + } + { + let (pi, _) = setup(); + pi.write64(reg::PENDING_MODIFY, 0x10c); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 1 << 12); + pi.write64(reg::PENDING_MODIFY_B, 12); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 0); + pi.set_lb_error(true); + pi.write64(reg::PENDING_MODIFY, 64 + 55); + assert_eq!(r(&pi, reg::PENDING_STATUS_B), 1 << 55); + pi.write64(reg::PENDING_MODIFY, 0x100 | 64 | 63); + assert_eq!(r(&pi, reg::PENDING_STATUS_B), 1 << 55); + pi.write64(reg::REQUEST_A, 0); + pi.write64(reg::OFFSET_00E8, 1); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 1 << 5); + pi.write64(reg::PENDING_A_CLEAR, 1); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 0); + pi.write64(reg::OFFSET_0010, 0); + assert!(!pi.io_modify(0x108, 0)); + pi.write64(reg::OFFSET_0010, 1); + assert!(pi.io_modify(0x108, 0)); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 1 << 8); + } + { + let (pi, _) = setup(); + pi.write64(reg::CACHED_ALIAS, 9); + pi.write64(reg::CPU_A_ENABLE, 1); + pi.write64(reg::ERROR_ADDRESS_A, !0); + pi.set_memory_uncorrectable(); + pi.write64(reg::ERROR_SUMMARY_ALIAS, 3); + pi.write64(reg::RT_COMPARE_A, 0x24); + pi.power_on(); + assert_eq!(r(&pi, reg::ERROR_SUMMARY), MEMORY_ERROR | 3); + assert_eq!(r(&pi, reg::CACHED_ALIAS), 9); + assert_eq!(r(&pi, reg::CPU_A_ENABLE), 1); + assert_eq!(r(&pi, reg::ERROR_ADDRESS_A), 0x1fffffff8); + assert_eq!(r(&pi, reg::RT_COMPARE_A), 0); + assert_eq!(r(&pi, reg::COUNTER), 0); + pi.write64(reg::ERROR_SUMMARY_ALIAS, 0); + assert_eq!(r(&pi, reg::ERROR_SUMMARY), MEMORY_ERROR); + pi.write64(reg::ERROR_SUMMARY, MEMORY_ERROR); + assert_eq!(r(&pi, reg::ERROR_SUMMARY), 0); + } +} +/// The guest timer drives interrupts even while the processor is stopped. +/// The processor timer can wrap, pause while stopped, and reissue its guest interrupt. +#[test] +fn the_pi_timer_wraps_pauses_and_reissues_interrupts() { + { + let (pi, now) = setup(); + pi.write64(reg::COUNTER, 0x1_0000_0000); + pi.write64(reg::RT_COMPARE_A, 0x1_0000_0002); + now.store(1599, Ordering::SeqCst); + assert_eq!(r(&pi, reg::RT_STATUS_A), 0); + now.store(1600, Ordering::SeqCst); + assert_eq!(r(&pi, reg::RT_STATUS_A), 1); + assert_eq!(pi.read32(reg::COUNTER).data, 1); + assert_eq!(pi.read32(reg::COUNTER + 4).data, 2); + pi.write64(reg::RT_STATUS_A, !0); + assert_eq!(r(&pi, reg::RT_STATUS_A), 0); + pi.write64(reg::COUNTER, RT_MASK); + pi.write64(reg::PROFILE_COMPARE, 0); + now.store(2400, Ordering::SeqCst); + assert_eq!(r(&pi, reg::COUNTER), 0); + assert_eq!(r(&pi, reg::PROFILE_STATUS_A), 1); + assert_eq!(r(&pi, reg::PROFILE_STATUS_B), 1); + pi.write64(reg::PROFILE_STATUS_A, 0); + assert_eq!(r(&pi, reg::PROFILE_STATUS_A), 0); + assert_eq!(r(&pi, reg::PROFILE_STATUS_B), 1); + } + { + let (pi, now) = setup(); + pi.set_level_source(0, 12, true); + pi.write64(reg::PENDING_MODIFY, 0x10c); + pi.write64(reg::PENDING_MODIFY, 12); + now.store(999999, Ordering::SeqCst); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 0); + now.store(1000000, Ordering::SeqCst); + assert_eq!(r(&pi, reg::PENDING_STATUS_A), 1 << 12); + pi.write64(reg::COUNTER, 0); + pi.write64(reg::RT_COMPARE_A, 2); + now.store(1000400, Ordering::SeqCst); + pi.stop(); + now.store(2000000, Ordering::SeqCst); + assert_eq!(r(&pi, reg::COUNTER), 0); + pi.start(); + now.store(2001200, Ordering::SeqCst); + assert_eq!(r(&pi, reg::RT_STATUS_A), 1); + } +} +/// An interrupt callback may inspect PI while an output changes. +/// An interrupt callback may inspect PI while an output changes. +#[test] +fn pi_interrupt_outputs_follow_masks_and_allow_reentrant_reads() { + struct Probe { + pi: Weak, + bits: AtomicU64, + } + impl InterruptController for Probe { + fn set_line(&self, line: u32, level: bool) { + if let Some(pi) = self.pi.upgrade() { + let _ = pi.read64(reg::PENDING_STATUS_A); + } + if level { + self.bits.fetch_or(1 << line, Ordering::SeqCst); + } else { + self.bits.fetch_and(!(1 << line), Ordering::SeqCst); + } + } + } + let (pi, now) = setup(); + let probe = Arc::new(Probe { + pi: Arc::downgrade(&pi), + bits: AtomicU64::new(0), + }); + pi.set_interrupt_controller(probe.clone()); + pi.write64(reg::OFFSET_00A8, 1 << 12); + pi.write64(reg::OFFSET_00B0, 1 << 55); + pi.write64(reg::PENDING_MODIFY, 0x10c); + pi.set_lb_error(true); + pi.write64(reg::RT_COMPARE_A, 2); + pi.write64(reg::PROFILE_COMPARE, 2); + pi.write64(reg::RT_ENABLE_A, 1); + pi.write64(reg::PROFILE_ENABLE_A, 1); + pi.write64(reg::ERROR_CAUSE_A, MEMORY_ERROR); + pi.set_memory_uncorrectable(); + now.store(1600, Ordering::SeqCst); + pi.poll(); + assert_eq!(probe.bits.load(Ordering::SeqCst), 31); + pi.write64(reg::RT_STATUS_A, 0); + assert_eq!(probe.bits.load(Ordering::SeqCst), 27); + pi.write64(reg::PENDING_MODIFY, 12); + pi.set_lb_error(false); + pi.write64(reg::PROFILE_STATUS_A, 0); + pi.write64(reg::ERROR_SUMMARY, MEMORY_ERROR); + assert_eq!(probe.bits.load(Ordering::SeqCst), 0); +} +/// A guest can wait for a timer interrupt without polling its compare register. +/// The guest may wait for a timer interrupt without polling the compare register. +#[test] +fn the_timer_manager_delivers_expirations_without_mmio_polling() { + struct Probe(std::sync::mpsc::Sender<()>); + impl InterruptController for Probe { + fn set_line(&self, line: u32, level: bool) { + if line == 2 && level { + self.0.send(()).unwrap(); + } + } + } + let pi = BedrockPi::new(BedrockPiConfig::default(), crate::dev::heart::host_clock()); + let tm = Arc::new(TimerManager::new()); + pi.set_timer_manager(tm.clone()); + let (tx, rx) = std::sync::mpsc::channel(); + pi.set_interrupt_controller(Arc::new(Probe(tx))); + pi.write64(reg::RT_COMPARE_A, 12500); + pi.write64(reg::RT_ENABLE_A, 1); + pi.start(); + rx.recv_timeout(Duration::from_secs(2)).unwrap(); + pi.stop(); + assert!(!pi.is_running()); +} +/// Node callbacks may reenter PI, so concurrent writes must keep their observed order. +/// Node callbacks may reenter PI, so concurrent writes must preserve observed order. +#[test] +fn concurrent_pi_updates_publish_callbacks_in_register_order() { + use std::sync::mpsc; + use std::thread; + + struct Hooks { + pi: Weak, + offset: u64, + first_value: u64, + entered: mpsc::Sender<()>, + release: Mutex>, + observed: AtomicU64, + } + impl Hooks { + fn publish(&self, value: u64) { + // A node may inspect PI registers while processing a change. + let _ = self.pi.upgrade().unwrap().read64(self.offset); + if value == self.first_value { + self.entered.send(()).unwrap(); + self.release.lock().recv().unwrap(); + } + self.observed.store(value, Ordering::SeqCst); + } + } + impl NodeHooks for Hooks { + fn cached_alias_changed(&self, size: u8) { + self.publish(u64::from(size)); + } + fn cpu_enable(&self, _: u8, _: usize, enabled: bool) { + self.publish(u64::from(enabled)); + } + } + + for (offset, first_value, second_value) in [(reg::CACHED_ALIAS, 1, 2), (reg::CPU_A_ENABLE, 1, 0), (reg::CPU_B_ENABLE, 1, 0)] { + let (pi, _) = setup(); + let (entered_tx, entered_rx) = mpsc::channel(); + let (release_tx, release_rx) = mpsc::channel(); + let hooks = Arc::new(Hooks { + pi: Arc::downgrade(&pi), + offset, + first_value, + entered: entered_tx, + release: Mutex::new(release_rx), + observed: AtomicU64::new(99), + }); + pi.set_node_hooks(hooks.clone()); + let first_pi = pi.clone(); + let first = thread::spawn(move || first_pi.write64(offset, first_value)); + entered_rx.recv_timeout(Duration::from_secs(2)).unwrap(); + let (started_tx, started_rx) = mpsc::channel(); + let (done_tx, done_rx) = mpsc::channel(); + let second_pi = pi.clone(); + let second = thread::spawn(move || { + started_tx.send(()).unwrap(); + second_pi.write64(offset, second_value); + done_tx.send(()).unwrap(); + }); + started_rx.recv_timeout(Duration::from_secs(2)).unwrap(); + let premature = done_rx.recv_timeout(Duration::from_millis(100)); + // Always release and join both writers before checking results. + release_tx.send(()).unwrap(); + assert_eq!(first.join().unwrap(), BUS_OK); + second.join().unwrap(); + assert_eq!( + hooks.observed.load(Ordering::SeqCst), + r(&pi, offset), + "offset {offset:#x}" + ); + assert!(matches!(premature, Err(mpsc::RecvTimeoutError::Timeout))); + assert_eq!(r(&pi, offset), second_value); + } +} + +/// One PI of a node holding the processors `count` names, in the board's order. +fn node(count: usize) -> [Arc; 2] { + let population = Population::fitted_cpus(count).unwrap(); + std::array::from_fn(|interface| { + BedrockPi::new( + BedrockPiConfig { interface: interface as u8, request_region: 0, population, running: Population::SINGLE }, + Arc::new(|| 0), + ) + }) +} + +/// The PROM counts processors from the present bits. The board fills PI0/A, +/// then PI1/A, then PI0/B and PI1/B, so two processors sit one per +/// interface (the qtest's sparse-processor-presence mask 5), where the +/// qtest's processor-count shortcut fills interface 0 first. +#[test] +fn every_fitted_slot_reads_present_in_the_boards_order() { + for (count, present) in [(1, [1, 0, 0, 0]), (2, [1, 0, 1, 0]), (3, [1, 1, 1, 0]), (4, [1, 1, 1, 1])] { + let pis = node(count); + let read = |interface: usize, off| r(&pis[interface], off); + assert_eq!( + [read(0, reg::INTERFACE_SLOT_A), read(0, reg::INTERFACE_SLOT_B), + read(1, reg::INTERFACE_SLOT_A), read(1, reg::INTERFACE_SLOT_B)], + present, "{count} processors"); + } + assert_eq!(Population::fitted_cpus(2).unwrap().mask(), 5); + assert_eq!(Population::fitted_cpus(4).unwrap().count(), 4); +} + +/// A population that cannot hold CPU 0, or names a fifth slot, or a count the +/// node cannot fit, is refused rather than built. +#[test] +fn a_population_without_cpu_0_or_beyond_four_slots_is_refused() { + for mask in [0, 0b0010, 0b1100, 0b1_0001, 0xff] { + assert!(Population::new(mask).is_err(), "{mask:#x}"); + } + for count in [0, 5] { + assert!(Population::fitted_cpus(count).is_err(), "{count}"); + } + assert_eq!(Population::new(0b0101).unwrap(), Population::fitted_cpus(2).unwrap()); +} + +/// Only CPU 0 begins with its bus grant; the other fitted processors are held +/// disabled, and a power-on of the PI leaves that as it was. +#[test] +fn only_cpu_0_starts_enabled() { + let pis = node(4); + let enables = |pis: &[Arc; 2]| { + [r(&pis[0], reg::CPU_A_ENABLE), r(&pis[0], reg::CPU_B_ENABLE), + r(&pis[1], reg::CPU_A_ENABLE), r(&pis[1], reg::CPU_B_ENABLE)] + }; + assert_eq!(enables(&pis), [1, 0, 0, 0]); + pis[1].write64(reg::CPU_B_ENABLE, 1); + for pi in &pis { + pi.power_on(); + } + assert_eq!(enables(&pis), [1, 0, 0, 1], "enables survive the PI's local reset"); + // An empty slot is not granted either. + assert_eq!(r(&node(1)[0], reg::CPU_B_ENABLE), 0); +} + +/// The CPU-number register carries the interface in bit 1; the reader is +/// CPU 0, which sits in slot A, so bit 0 is clear. Follows cpu-number-fallback. +#[test] +fn the_cpu_number_names_the_interface_and_slot_a() { + let pis = node(4); + assert_eq!((r(&pis[0], reg::INTERFACE_ID), r(&pis[1], reg::INTERFACE_ID)), (0, 2)); + for pi in &pis { + pi.power_on(); + } + assert_eq!((r(&pis[0], reg::INTERFACE_ID), r(&pis[1], reg::INTERFACE_ID)), (0, 2)); +} + +/// Enabling a processor that never runs is stored, as the register holds it, +/// and recorded so that the guest's attempt is visible; CPU 0's own enable +/// is not a refusal. +#[test] +fn enabling_another_cpu_is_stored_and_recorded_but_never_started() { + let pis = node(4); + pis[0].write64(reg::CPU_A_ENABLE, 1); + assert!(pis[0].refused().is_empty(), "CPU 0 runs"); + pis[0].write64(reg::CPU_B_ENABLE, 1); + pis[1].write64(reg::CPU_A_ENABLE, 3); + pis[1].write64(reg::CPU_A_ENABLE, 1); + pis[1].write64(reg::CPU_B_ENABLE, 0); + assert_eq!(r(&pis[0], reg::CPU_B_ENABLE), 1); + assert_eq!(pis[0].refused(), [(Refused::Start { interface: 0, slot: 1 }, 1)]); + assert_eq!(pis[1].refused(), [(Refused::Start { interface: 1, slot: 0 }, 2)], + "counted, and a clear bit is not an attempt"); +} + +/// An NMI for a slot with no running processor, or for CPU 0 with no node to +/// take it, is recorded. A node that takes CPU 0's still sees it. Follows nmi. +#[test] +fn an_nmi_no_running_cpu_can_take_is_recorded() { + struct Nmis(std::sync::Mutex>); + impl NodeHooks for Nmis { + fn nmi(&self, interface: u8, slot: usize) -> bool { + self.0.lock().unwrap().push((interface, slot)); + true + } + } + let pis = node(4); + pis[0].write64(reg::NMI_SLOT_A, 1); + pis[0].write64(reg::NMI_SLOT_B, u64::MAX); + pis[1].write64(reg::NMI_SLOT_A, 1); + assert_eq!(pis[0].refused(), [(Refused::Nmi { interface: 0, slot: 0 }, 1), (Refused::Nmi { interface: 0, slot: 1 }, 1)]); + assert_eq!(pis[1].refused(), [(Refused::Nmi { interface: 1, slot: 0 }, 1)]); + + let nmis = Arc::new(Nmis(Default::default())); + let pis = node(4); + pis[0].set_node_hooks(nmis.clone()); + pis[0].write64(reg::NMI_SLOT_A, 1); + assert!(pis[0].refused().is_empty()); + pis[0].write64(reg::NMI_SLOT_B, 1); + assert_eq!(*nmis.0.lock().unwrap(), [(0, 0), (0, 1)], "a node still sees what it is sent"); + assert_eq!(pis[0].refused(), [(Refused::Nmi { interface: 0, slot: 1 }, 1)]); +} + +/// An interrupt output that a processor in another slot would take, such as an +/// interprocessor interrupt aimed at slot A of PI1, goes high on its line and +/// is recorded; CPU 0's own lines are delivered without a record. +/// Follows sparse-irq-and-nmi-slots. +#[test] +fn interrupts_for_slots_without_a_running_cpu_are_recorded() { + let pis = node(2); + pis[1].write64(reg::PENDING_MODIFY, 0x107); + pis[1].write64(reg::OFFSET_00A8, 1 << 7); + assert_eq!(pis[1].refused(), [(Refused::Interrupt { interface: 1, line: 0 }, 1)]); + pis[1].write64(reg::OFFSET_00A8, 0); + assert_eq!(pis[1].refused().len(), 1, "the line going low is not a new event"); + + pis[0].write64(reg::PENDING_MODIFY, 0x107); + pis[0].write64(reg::OFFSET_00A8, 1 << 7); + assert!(pis[0].refused().is_empty(), "PI0 slot A is CPU 0's"); + // Slot B of PI0 shares the pending word through its own mask. + pis[0].write64(reg::OFFSET_00B8, 1 << 7); + assert_eq!(pis[0].refused(), [(Refused::Interrupt { interface: 0, line: 5 }, 1)]); +} + +/// Per-processor timers of a slot that never runs can be set but not +/// taken: enabling one is recorded. +#[test] +fn timers_for_other_slots_are_recorded_when_enabled() { + let pis = node(4); + pis[0].write64(reg::RT_ENABLE_A, 1); + pis[0].write64(reg::PROFILE_ENABLE_A, 1); + assert!(pis[0].refused().is_empty()); + pis[0].write64(reg::RT_ENABLE_B, 1); + pis[0].write64(reg::PROFILE_ENABLE_B, 1); + pis[1].write64(reg::RT_ENABLE_A, 1); + assert_eq!(pis[0].refused(), [(Refused::Timer { interface: 0, slot: 1 }, 2)]); + assert_eq!(pis[1].refused(), [(Refused::Timer { interface: 1, slot: 0 }, 1)]); +} + +/// A node whose `count` fitted processors all run. +fn running_node(count: usize) -> [Arc; 2] { + let population = Population::fitted_cpus(count).unwrap(); + std::array::from_fn(|interface| { + BedrockPi::new( + BedrockPiConfig { interface: interface as u8, request_region: 0, population, running: population }, + Arc::new(|| 0), + ) + }) +} + +/// The board puts CPU n in the nth slot of its order: A, C, B, D. +#[test] +fn each_cpu_has_the_slot_the_boards_order_gives_it() { + let four = Population::fitted_cpus(4).unwrap(); + assert_eq!((0..4).map(|cpu| four.slot_of(cpu).unwrap()).collect::>(), [(0, 0), (1, 0), (0, 1), (1, 1)]); + assert_eq!(four.slot_of(4), None); + assert_eq!(four.cpu_in(0, 1), Some(2)); + let two = Population::fitted_cpus(2).unwrap(); + assert_eq!((two.slot_of(1), two.cpu_in(0, 1)), (Some((1, 0)), None)); +} + +/// Each processor reads its own slice from the CPU-number register of its +/// interface, and a reader that is no processor reads slot A. Follows +/// four-processors-know-their-slots in tests/qtest/sn1-cpu-identity-test.c +/// and cpu-number-fallback. +#[test] +fn four_processors_know_their_slots() { + let pis = Arc::new(running_node(4)); + assert_eq!((r(&pis[0], reg::INTERFACE_ID), r(&pis[1], reg::INTERFACE_ID)), (0, 2), "no issuing processor"); + let slices: Vec = (0..4u8) + .map(|cpu| { + let pis = pis.clone(); + std::thread::spawn(move || { + crate::cpu::smp::set_current_cpu(cpu); + let (interface, _) = Population::fitted_cpus(4).unwrap().slot_of(cpu as usize).unwrap(); + r(&pis[interface as usize], reg::INTERFACE_ID) + }) + .join() + .unwrap() + }) + .collect(); + assert_eq!(slices, [0, 2, 1, 3], "slices A, C, B, D"); +} + +/// Every running slot starts granted. A write is stored either way and +/// starts or stops nothing; clearing a running processor's bit is recorded. +/// Follows cpu-enable-lines and the identity test's enable writes. +#[test] +fn running_slots_start_enabled_and_a_clear_stops_nothing() { + let pis = running_node(4); + let enables = |pis: &[Arc; 2]| { + [r(&pis[0], reg::CPU_A_ENABLE), r(&pis[0], reg::CPU_B_ENABLE), + r(&pis[1], reg::CPU_A_ENABLE), r(&pis[1], reg::CPU_B_ENABLE)] + }; + assert_eq!(enables(&pis), [1, 1, 1, 1]); + pis[1].write64(reg::CPU_A_ENABLE, 0); + assert_eq!(enables(&pis), [1, 1, 0, 1]); + assert_eq!(pis[1].refused(), [(Refused::Stop { interface: 1, slot: 0 }, 1)]); + pis[1].write64(reg::CPU_A_ENABLE, 1); + assert_eq!(pis[1].refused().len(), 1, "re-enabling a running slot is no refusal"); + assert_eq!(enables(&running_node(2)), [1, 0, 1, 0]); +} + +/// The other processors' interrupt outputs and timers are theirs to take: +/// nothing about them is refused once their slots run. +#[test] +fn a_running_slots_interrupts_and_timers_are_not_refused() { + let pis = running_node(4); + pis[1].write64(reg::PENDING_MODIFY, 0x107); + pis[1].write64(reg::OFFSET_00A8, 1 << 7); + pis[0].write64(reg::OFFSET_00B8, 1 << 7); + pis[0].write64(reg::PENDING_MODIFY, 0x107); + pis[0].write64(reg::RT_ENABLE_B, 1); + pis[1].write64(reg::PROFILE_ENABLE_A, 1); + assert!(pis[0].refused().is_empty() && pis[1].refused().is_empty()); +} + +/// An NMI to a running processor that the node cannot take is recorded: +/// the processor has no NMI entry. Follows nmi. +#[test] +fn an_nmi_to_a_running_processor_is_recorded() { + let pis = running_node(2); + pis[1].write64(reg::NMI_SLOT_A, 1); + assert_eq!(pis[1].refused(), [(Refused::Nmi { interface: 1, slot: 0 }, 1)]); +} diff --git a/src/dev/bedrock_xb.rs b/src/dev/bedrock_xb.rs new file mode 100644 index 000000000..6df8e7b1c --- /dev/null +++ b/src/dev/bedrock_xb.rs @@ -0,0 +1,278 @@ +//! Bedrock XB is the node's internal crossbar status block. It records the +//! fabric error queues and lets software service queued errors; the Fuel board +//! connects its outputs to the Bedrock processor interface. +//! +//! # What is approximated here +//! +//! Arbitration timing and diagnostic outputs are not modeled. Unsupported +//! queue reads return zero, while writes are logged and ignored. +//! +//! Ported from hw/misc/sgi-bedrock-xb.c in jamesbraid/qemu-sgi-origami. + +/// Length of the block-relative register aperture. +pub const SIZE: u64 = 0x198; + +pub mod reg { + pub const CONTROL_BASE: u64 = 0x0; + pub const CONTROL_END: u64 = 0x18; + pub const QUEUE_BASE: u64 = 0x20; + pub const QUEUE_END: u64 = 0x88; + pub const FIRST_ERROR: u64 = 0x90; + pub const DEBUG_BASE: u64 = 0x98; + pub const DEBUG_END: u64 = 0xa0; + pub const CLEAR_QUEUE_BASE: u64 = 0x120; + pub const CLEAR_QUEUE_END: u64 = 0x188; + pub const CLEAR_FIRST_ERROR: u64 = 0x190; +} + +use crate::traits::{BusDevice, BusRead64, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::{Mutex, ReentrantMutex}; +use std::sync::Arc; +type Level = Arc; +const MASKS: [u64; 14] = [ + 0x3333, 0x33, 0, 0, 0x3330, 0, 0, 0xf0, 0xfff0, 0, 0, 0, 0x3303, 0x11, +]; +struct Regs { + control: [u64; 4], + queues: [u64; 14], + first: u64, + debug: [u64; 2], + unserviced: u16, +} +impl Default for Regs { + fn default() -> Self { + Self { + control: [0x0008000080000021, 0, 0, 0], + queues: [0; 14], + first: 0, + debug: [0; 2], + unserviced: 0, + } + } +} +pub struct BedrockXb { + regs: Arc>, + error: Mutex>, + delivery: ReentrantMutex<()>, + driven: Mutex>, +} +impl Default for BedrockXb { + fn default() -> Self { + Self::new() + } +} +impl BedrockXb { + pub fn new() -> Self { + let regs = Arc::new(Mutex::new(Regs::default())); + crate::locks::register_mutex("bedrock-xb", ®s); + Self { + regs, + error: Mutex::new(None), + delivery: ReentrantMutex::new(()), + driven: Mutex::new(None), + } + } + pub fn connect_error(&self, cb: Level) { + let _delivery = self.delivery.lock(); + *self.error.lock() = Some(cb); + *self.driven.lock() = None; + self.update_irq(); + } + pub fn error_level(&self) -> bool { + self.regs.lock().first & 0x1000 != 0 + } + fn update_irq(&self) { + // Serialize the sample and callback, while leaving registers available + // to the callback. Suppress unchanged levels so callbacks can reenter + // register reads that recompute the output. + let _delivery = self.delivery.lock(); + let cb = self.error.lock().clone(); + if let Some(cb) = cb { + let level = self.error_level(); + let changed = { + let mut driven = self.driven.lock(); + let changed = *driven != Some(level); + *driven = Some(level); + changed + }; + if changed { + cb(level); + } + } + } +} +impl Resettable for BedrockXb { + fn power_on(&self) { + *self.regs.lock() = Regs::default(); + self.update_irq(); + } +} +impl BusDevice for BedrockXb { + fn read64(&self, a: u64) -> BusRead64 { + if a & 7 != 0 { + return BusRead64::err(); + } + let r = self.regs.lock(); + let v = match a { + reg::CONTROL_BASE..=reg::CONTROL_END => r.control[a as usize / 8], + reg::QUEUE_BASE..=reg::QUEUE_END => { + let n = (a as usize - 0x20) / 8; + if MASKS[n] == 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB unsupported queue {n}"); + } + r.queues[n] + } + reg::FIRST_ERROR => r.first, + reg::DEBUG_BASE | reg::DEBUG_END => r.debug[(a as usize - 0x98) / 8], + _ => { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB unsupported read {a:#x}"); + 0 + } + }; + BusRead64::ok(v) + } + fn write64(&self, a: u64, v: u64) -> u32 { + if a & 7 != 0 { + return BUS_ERR; + } + let mut r = self.regs.lock(); + match a { + reg::CONTROL_BASE..=reg::CONTROL_END => { + let n = a as usize / 8; + r.control[n] = v & [0x000ffffffffffff1, 0xffffffff, 0x7ff, 0xf][n]; + if a == 0x10 && r.control[n] != 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB traffic observation unsupported"); + } + } + reg::QUEUE_BASE..=reg::QUEUE_END => { + let n = (a as usize - 0x20) / 8; + let v = v & MASKS[n]; + if MASKS[n] == 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB unsupported queue {n}"); + } + if v != 0 { + r.queues[n] |= v; + r.unserviced |= 1 << n; + if r.first & 0x1000 == 0 { + r.first = 0x1000 | ((15 - n) as u64) << 8; + } + } + } + reg::CLEAR_QUEUE_BASE..=reg::CLEAR_QUEUE_END => { + let n = (a as usize - 0x120) / 8; + r.queues[n] = 0; + r.unserviced &= !(1 << n); + } + reg::CLEAR_FIRST_ERROR => { + if r.first & 0x1000 != 0 { + let n = 15 - ((r.first >> 8) & 15) as usize; + if r.unserviced & (1 << n) != 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB first error cleared before queue"); + } else { + r.first = 0; + for n in 0..14 { + if r.queues[n] != 0 { + r.first = 0x1000 | ((15 - n) as u64) << 8; + break; + } + } + } + } + } + reg::DEBUG_BASE | reg::DEBUG_END => { + let n = (a as usize - 0x98) / 8; + r.debug[n] = v & [0xffff, 7][n]; + if r.debug[n] != 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB debug output unsupported"); + } + } + _ => crate::dlog!(crate::devlog::LogModule::Bedrock, "Bedrock XB unsupported or read-only write {a:#x} = {v:#x}"), + } + drop(r); + self.update_irq(); + BUS_OK + } +} + +#[cfg(test)] +mod tests { + // Cases follow tests/qtest/sn1-bedrock-xb-test.c in + // jamesbraid/qemu-sgi-origami: control-storage, service-order and + // cold-reset. + use super::*; + + /// The guest sees masked crossbar control words and cleared errors after reset. + #[test] + fn crossbar_control_masks_apply_and_reset_clears_errors() { + let d = BedrockXb::new(); + for (a, mask) in [ + (reg::CONTROL_BASE, 0x000ffffffffffff1), + (reg::CONTROL_BASE + 8, 0xffffffff), + (reg::CONTROL_BASE + 0x10, 0x7ff), + (reg::CONTROL_END, 15), + (reg::DEBUG_BASE, 0xffff), + (reg::DEBUG_END, 7), + ] { + d.write64(a, !0); + assert_eq!(d.read64(a).data, mask); + } + // Keep expected register values independent of the decoder's mask table. + for (address, expected) in [ + (reg::QUEUE_BASE, 0x3333), + (reg::QUEUE_BASE + 8, 0x0033), + (reg::QUEUE_BASE + 0x10, 0), + (reg::QUEUE_BASE + 0x18, 0), + (reg::QUEUE_BASE + 0x20, 0x3330), + (reg::QUEUE_BASE + 0x28, 0), + (reg::QUEUE_BASE + 0x30, 0), + (reg::QUEUE_BASE + 0x38, 0x00f0), + (reg::QUEUE_BASE + 0x40, 0xfff0), + (reg::QUEUE_BASE + 0x48, 0), + (reg::QUEUE_BASE + 0x50, 0), + (reg::QUEUE_BASE + 0x58, 0), + (reg::QUEUE_BASE + 0x60, 0x3303), + (reg::QUEUE_END, 0x0011), + ] { + d.write64(address, !0); + assert_eq!(d.read64(address).data, expected, "register {address:#x}"); + } + d.power_on(); + assert_eq!(d.read64(reg::CONTROL_BASE).data, 0x0008000080000021); + assert!(!d.error_level()); + assert_eq!(d.read32(reg::CONTROL_BASE).status, BUS_ERR); + } + /// The guest services queued crossbar faults in order, one error at a time. + #[test] + fn servicing_one_crossbar_error_exposes_the_next_pending_error() { + { + let d = BedrockXb::new(); + d.write64(reg::QUEUE_BASE, 1); + assert!(d.error_level()); + assert_eq!(d.read64(reg::FIRST_ERROR).data, 0x1f00); + d.write64(reg::CLEAR_FIRST_ERROR, !0); + assert_eq!(d.read64(reg::FIRST_ERROR).data, 0x1f00); + d.write64(reg::CLEAR_QUEUE_END, !0); + d.write64(reg::CLEAR_QUEUE_BASE, 0); + d.write64(reg::CLEAR_FIRST_ERROR, 0); + assert!(!d.error_level()); + d.power_on(); + assert!(!d.error_level()); + } + { + let d = BedrockXb::new(); + d.write64(reg::QUEUE_END, 0x10); + d.write64(reg::QUEUE_BASE, 1); + assert_eq!(d.read64(reg::FIRST_ERROR).data, 0x1200); + d.write64(reg::CLEAR_QUEUE_END, 0); + d.write64(reg::CLEAR_FIRST_ERROR, 0); + assert_eq!(d.read64(reg::FIRST_ERROR).data, 0x1f00); + d.write64(reg::CLEAR_QUEUE_BASE, 0); + d.write64(reg::QUEUE_BASE, 0x10); + d.write64(reg::CLEAR_FIRST_ERROR, 0); + assert_eq!(d.read64(reg::FIRST_ERROR).data, 0x1f00); + d.write64(reg::CLEAR_QUEUE_BASE, 0); + d.write64(reg::CLEAR_FIRST_ERROR, 0); + assert!(!d.error_level()); + } + } +} diff --git a/src/dev/bridge.rs b/src/dev/bridge.rs index 41fcf4fa9..96b267e06 100644 --- a/src/dev/bridge.rs +++ b/src/dev/bridge.rs @@ -1057,7 +1057,7 @@ impl Bridge { let i = (reg >> 2) as usize; let shift = 8 * (reg & 3); let mask = (u64::MAX >> (64 - 8 * size)) << shift; - cfg.write(i, ((cfg.read(i) as u64 & !mask) | ((val << shift) & mask)) as u32); + cfg.write_masked(i, (val << shift) as u32, mask as u32); true } diff --git a/src/dev/deadline_timers.rs b/src/dev/deadline_timers.rs new file mode 100644 index 000000000..6331fd08f --- /dev/null +++ b/src/dev/deadline_timers.rs @@ -0,0 +1,200 @@ +//! One-shot timers for the host-time deadlines a device keeps. +//! +//! A device evaluates its deadlines lazily, on every access, and arms one +//! timer per deadline so the work still happens when the guest is idle. The +//! IOC3 and IOC4 do this for their serial ring timers and transmit delays, +//! and the IOC3 for its interrupt output. This holds the part they share: +//! which timer serves which deadline, replacing one whose deadline moved, +//! and taking them all away on stop. +//! +//! QEMU has no counterpart: its timers belong to the virtual clock and are +//! armed where they are set. + +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, OnceLock, Weak}; +use std::time::{Duration, Instant}; + +use parking_lot::Mutex; + +use crate::hptimer::{TimerId, TimerManager, TimerReturn}; + +/// The timers of a device with `N` kinds of deadline. +pub struct DeadlineTimers { + manager: OnceLock>, + /// The timer pending for each kind, and the deadline it serves. + armed: Mutex<[Option<(TimerId, Instant)>; N]>, +} + +impl DeadlineTimers { + pub fn new() -> Self { + Self { manager: OnceLock::new(), armed: Mutex::new([None; N]) } + } + + /// The timer manager to arm on. Set it before the device starts. + pub fn set_manager(&self, tm: Arc) { + let _ = self.manager.set(tm); + } + + /// Make the pending timers match `want`: one one-shot per deadline. A + /// timer whose deadline moved is replaced. With no manager set, or the + /// device not `running`, deadlines are only evaluated lazily, by the next + /// access. `floor` is the least delay for a kind, for a deadline that can + /// recur faster than a timer should fire; `fire` runs on the timer thread + /// with the device and the kind and deadline it was armed for. + pub fn arm( + &self, + want: &[Option; N], + running: &AtomicBool, + device: &Weak, + floor: fn(usize) -> Duration, + fire: fn(&Weak, usize, Instant) -> TimerReturn, + ) { + let Some(tm) = self.manager.get() else { return }; + let mut armed = self.armed.lock(); + // Checked under the lock `cancel_all` clears the slots with: a pass + // that sees `running` set has its timers removed by that stop. + if !running.load(Ordering::SeqCst) { + return; + } + for (kind, w) in want.iter().enumerate() { + if armed[kind].map(|(_, d)| d) == *w { + continue; + } + if let Some((id, _)) = armed[kind].take() { + tm.remove(id); + } + let Some(due) = *w else { continue }; + let delay = due.saturating_duration_since(Instant::now()).max(floor(kind)); + let id = tm.add_one_shot(delay, device.clone(), move |d| fire(d, kind, due)); + armed[kind] = Some((id, due)); + } + } + + /// A timer for `kind` and `due` has fired: forget it, unless a newer + /// deadline has replaced it. + pub fn fired(&self, kind: usize, due: Instant) { + let mut armed = self.armed.lock(); + if armed[kind].map(|(_, d)| d) == Some(due) { + armed[kind] = None; + } + } + + /// Whether a timer is pending for `kind`. + #[cfg(test)] + pub fn is_armed(&self, kind: usize) -> bool { + self.armed.lock()[kind].is_some() + } + + /// Whether no timer is pending. + #[cfg(test)] + pub fn none_armed(&self) -> bool { + self.armed.lock().iter().all(|a| a.is_none()) + } + + /// Remove every pending timer. + pub fn cancel_all(&self) { + let mut armed = self.armed.lock(); + for slot in armed.iter_mut() { + if let (Some((id, _)), Some(tm)) = (slot.take(), self.manager.get()) { + tm.remove(id); + } + } + } +} + +impl Default for DeadlineTimers { + fn default() -> Self { + Self::new() + } +} + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::atomic::AtomicUsize; + + struct Device { + timers: DeadlineTimers<2>, + running: AtomicBool, + fired: [AtomicUsize; 2], + } + + fn fire(me: &Weak, kind: usize, due: Instant) -> TimerReturn { + if let Some(d) = me.upgrade() { + d.timers.fired(kind, due); + d.fired[kind].fetch_add(1, Ordering::SeqCst); + } + TimerReturn::Delete + } + + fn device() -> Arc { + let d = Arc::new(Device { + timers: DeadlineTimers::new(), + running: AtomicBool::new(true), + fired: Default::default(), + }); + d.timers.set_manager(Arc::new(TimerManager::new())); + d + } + + fn wait_for(f: impl Fn() -> bool) -> bool { + for _ in 0..500 { + if f() { + return true; + } + std::thread::sleep(Duration::from_millis(2)); + } + false + } + + #[test] + fn a_deadline_fires_once_and_its_slot_empties() { + let d = device(); + let due = Instant::now() + Duration::from_millis(5); + d.timers.arm(&[Some(due), None], &d.running, &Arc::downgrade(&d), |_| Duration::ZERO, fire); + assert!(d.timers.is_armed(0)); + assert!(!d.timers.is_armed(1)); + assert!(wait_for(|| d.fired[0].load(Ordering::SeqCst) == 1)); + assert!(d.timers.none_armed()); + assert_eq!(d.fired[1].load(Ordering::SeqCst), 0); + } + + #[test] + fn a_moved_deadline_replaces_its_timer() { + let d = device(); + let me = Arc::downgrade(&d); + let far = Instant::now() + Duration::from_secs(3600); + d.timers.arm(&[Some(far), None], &d.running, &me, |_| Duration::ZERO, fire); + let near = Instant::now() + Duration::from_millis(5); + d.timers.arm(&[Some(near), None], &d.running, &me, |_| Duration::ZERO, fire); + assert!(wait_for(|| d.fired[0].load(Ordering::SeqCst) == 1)); + assert_eq!(d.fired[0].load(Ordering::SeqCst), 1, "the far timer was removed, not left to fire"); + } + + #[test] + fn a_stopped_device_arms_nothing_and_cancel_clears_what_was_armed() { + let d = device(); + let me = Arc::downgrade(&d); + let far = Some(Instant::now() + Duration::from_secs(3600)); + d.running.store(false, Ordering::SeqCst); + d.timers.arm(&[far, far], &d.running, &me, |_| Duration::ZERO, fire); + assert!(d.timers.none_armed()); + d.running.store(true, Ordering::SeqCst); + d.timers.arm(&[far, far], &d.running, &me, |_| Duration::ZERO, fire); + assert!(d.timers.is_armed(0) && d.timers.is_armed(1)); + d.timers.cancel_all(); + assert!(d.timers.none_armed()); + } + + #[test] + fn the_floor_delays_a_kind_that_recurs_quickly() { + let d = device(); + let now = Instant::now(); + d.timers.arm(&[Some(now), Some(now)], &d.running, &Arc::downgrade(&d), |k| { + if k == 0 { Duration::from_millis(40) } else { Duration::ZERO } + }, fire); + assert!(wait_for(|| d.fired[1].load(Ordering::SeqCst) == 1)); + assert_eq!(d.fired[0].load(Ordering::SeqCst), 0, "kind 0 is held back by its floor"); + assert!(wait_for(|| d.fired[0].load(Ordering::SeqCst) == 1)); + } +} diff --git a/src/dev/dp83843.rs b/src/dev/dp83843.rs new file mode 100644 index 000000000..8d3bb40c5 --- /dev/null +++ b/src/dev/dp83843.rs @@ -0,0 +1,573 @@ +//! The DP83843 MII model exposes the control, status and auto-negotiation +//! registers used by the Fuel IOC3 Ethernet port. A guest starts negotiation +//! through MII; the backend link level determines the simulated partner result. +//! +//! # What is approximated here +//! +//! Link settling and partner responses are software models. BMSR reports the +//! current link level rather than the hardware's latched-low status. +//! +//! Ported from hw/net/dp83840.c in jamesbraid/qemu-sgi-origami. + +use std::time::{Duration, Instant}; + +use crate::snapshot::{field, field_bool, field_u32, field_u64, field_u8, toml_u16, StateTable}; +pub mod reg { + pub const BASIC_CONTROL: u8 = 0x0; + pub const BASIC_STATUS: u8 = 0x1; + pub const PHY_ID1: u8 = 0x2; + pub const PHY_ID2: u8 = 0x3; + pub const AUTO_NEG_ADVERTISE: u8 = 0x4; + pub const AUTO_NEG_PARTNER: u8 = 0x5; + pub const AUTO_NEG_EXPANSION: u8 = 0x6; + pub const NEXT_PAGE_TRANSMIT: u8 = 0x7; + pub const PHY_STATUS: u8 = 0x10; + pub const INTERRUPT_CONTROL: u8 = 0x11; + pub const INTERRUPT_STATUS: u8 = 0x12; + pub const DISCONNECT_COUNTER: u8 = 0x13; + pub const FALSE_CARRIER_COUNTER: u8 = 0x14; + pub const RECEIVE_ERROR_COUNTER: u8 = 0x15; + pub const PCS_CONFIG: u8 = 0x16; + pub const LOOPBACK: u8 = 0x17; + pub const TEN_BASE_T_STATUS: u8 = 0x18; + pub const PHY_CONTROL: u8 = 0x19; +} +const SETTLE: Duration = Duration::from_secs(2); +const BREAK: Duration = Duration::from_millis(1500); + +/// Management state for one transceiver. No board GPIO reset is connected. +pub struct Dp83843 { + configured_addr: u8, + regs: [u16; 32], + backend_up: bool, + done: bool, + forced: bool, + resolved: u8, + armed: bool, + deadline: Option, + complete: Option, + pending: bool, + restarting: bool, + pub unimplemented_accesses: u32, +} +impl Dp83843 { + pub fn new(addr: u8) -> Self { + let mut p = Self { + configured_addr: addr & 31, + regs: [0; 32], + backend_up: true, + done: false, + forced: false, + resolved: 0, + armed: false, + deadline: None, + complete: None, + pending: false, + restarting: false, + unimplemented_accesses: 0, + }; + p.reset(); + p + } + pub fn reset(&mut self) { + self.regs = [0; 32]; + self.regs[reg::BASIC_CONTROL as usize] = 0x3100 | if self.configured_addr == 0 { 0x400 } else { 0 }; + self.regs[reg::AUTO_NEG_ADVERTISE as usize] = 0x1e1; + self.regs[reg::NEXT_PAGE_TRANSMIT as usize] = 0x2001; + self.regs[reg::PCS_CONFIG as usize] = 0x20; + self.regs[reg::PHY_CONTROL as usize] = u16::from(self.configured_addr); + self.done = false; + self.forced = false; + self.resolved = 0; + self.armed = false; + self.deadline = None; + self.complete = None; + self.pending = false; + self.restarting = false; + } + pub fn address(&self) -> u8 { + (self.regs[reg::PHY_CONTROL as usize] & 31) as u8 + } + pub fn backend_up(&self) -> bool { + self.backend_up + } + pub fn deadline(&self) -> Option { + self.deadline + } + fn autoneg(&self) -> bool { + self.regs[reg::BASIC_CONTROL as usize] & 0x1000 != 0 + } + fn supported(&self) -> bool { + self.regs[reg::PHY_CONTROL as usize] & 0x220 == 0 + && self.regs[reg::TEN_BASE_T_STATUS as usize] & 0x1a20 == 0 + && self.regs[reg::LOOPBACK as usize] & 0x3f80 == 0 + } + fn link(&self) -> bool { + self.backend_up + && self.supported() + && self.regs[reg::BASIC_CONTROL as usize] & 0x800 == 0 + && if self.autoneg() { + self.done && self.resolved != 0 + } else { + self.forced + } + } + pub fn data_path(&self) -> bool { + self.supported() && self.regs[reg::BASIC_CONTROL as usize] & 0xc00 == 0 + } + pub fn wire_enabled(&self) -> bool { + self.data_path() && self.link() + } + fn loopback_selected(&self) -> bool { + self.regs[reg::BASIC_CONTROL as usize] & 0x4000 != 0 || self.regs[reg::LOOPBACK as usize] & 0x30 != 0 + } + pub fn in_loopback(&self) -> bool { + self.data_path() && self.loopback_selected() + } + fn status(&self) -> u16 { + let mut v = if self.autoneg() { 0x200 } else { 0 }; + if self.loopback_selected() { + v |= 8; + } + if !self.link() { + return v; + } + v |= 1; + let (speed, full) = if self.autoneg() { + v |= 0x20; + (self.resolved <= 2, self.resolved == 1 || self.resolved == 3) + } else { + (self.regs[reg::BASIC_CONTROL as usize] & 0x2000 != 0, self.regs[reg::BASIC_CONTROL as usize] & 0x100 != 0) + }; + if !speed { + v |= 2; + } + if full { + v |= 4; + } + if self.regs[reg::BASIC_CONTROL as usize] & 0x400 == 0 { + v |= 0x800; + } + v + } + fn event(&mut self, before: u16) { + let after = self.status(); + if self.regs[reg::INTERRUPT_CONTROL as usize] & 2 != 0 && ((before ^ after) & 1 != 0 || !before & after & 0x20 != 0) { + self.pending = true; + } + } + fn resolve(&mut self) { + self.resolved = 0; + if self.autoneg() && self.supported() { + for (i, bit) in [0x100, 0x80, 0x40, 0x20].iter().enumerate() { + if self.regs[reg::AUTO_NEG_ADVERTISE as usize] & bit != 0 { + self.resolved = i as u8 + 1; + break; + } + } + } + if self.resolved == 0 { + crate::dlog!(crate::devlog::LogModule::Ioc, "dp83843: no common supported advertised mode"); + } + } + pub fn sync(&mut self, now: Instant) { + if self.deadline.is_some_and(|d| now >= d) { + let before = self.status(); + if let Some(end) = self.complete.take() { + self.regs[reg::BASIC_CONTROL as usize] &= !0x200; + self.deadline = Some(end); + if now < end { + return; + } + } + self.deadline = None; + self.restarting = false; + self.done = true; + self.resolve(); + self.event(before); + } + } + pub fn set_backend_link(&mut self, up: bool, now: Instant) { + self.sync(now); + let before = self.status(); + self.backend_up = up; + self.event(before); + } + fn unimplemented(&mut self, r: u8) { + self.unimplemented_accesses += 1; + crate::dlog!(crate::devlog::LogModule::Ioc, "dp83843: unsupported register or control at {r:#x}"); + } + pub fn read(&mut self, r: u8, now: Instant) -> u16 { + self.sync(now); + let v = match r { + reg::BASIC_CONTROL => self.regs[reg::BASIC_CONTROL as usize], + reg::BASIC_STATUS => { + if !self.armed && !self.done && self.autoneg() { + self.armed = true; + self.deadline = Some(now + SETTLE); + } + let status = self.status(); + 0x7849 | if status & 1 != 0 { 4 } else { 0 } | (status & 0x20) + } + reg::PHY_ID1 => 0x2000, + reg::PHY_ID2 => 0x5c10, + reg::AUTO_NEG_ADVERTISE | reg::NEXT_PAGE_TRANSMIT => self.regs[r as usize], + reg::AUTO_NEG_PARTNER => { + if self.done { + 0x1e1 + } else { + 0 + } + } + reg::AUTO_NEG_EXPANSION => 4 | u16::from(self.done), + reg::PHY_STATUS => { + let v = self.status() | if self.pending { 0x100 } else { 0 }; + self.pending = false; + v + } + reg::INTERRUPT_CONTROL => self.regs[reg::INTERRUPT_CONTROL as usize], + reg::INTERRUPT_STATUS => { + let v = if self.pending { 0x8000 } else { 0 }; + self.pending = false; + v + } + reg::DISCONNECT_COUNTER..=reg::RECEIVE_ERROR_COUNTER => { + let v = self.regs[r as usize]; + self.regs[r as usize] = 0; + v + } + reg::PCS_CONFIG => { + let v = self.regs[reg::PCS_CONFIG as usize]; + if self.autoneg() { + v & !0x4000 + } else if self.regs[reg::PHY_CONTROL as usize] & 0x20 != 0 { + v | 0x4000 + } else { + v + } + } + reg::LOOPBACK => { + self.regs[reg::LOOPBACK as usize] + | if self.regs[reg::PHY_CONTROL as usize] & 0x20 != 0 { + 0x1000 + } else { + 0 + } + } + reg::TEN_BASE_T_STATUS => { + self.regs[reg::TEN_BASE_T_STATUS as usize] + | if self.status() & 4 != 0 || self.regs[reg::PHY_CONTROL as usize] & 0x200 != 0 { + 0x80 + } else { + 0 + } + } + reg::PHY_CONTROL => self.regs[reg::PHY_CONTROL as usize], + _ => { + self.unimplemented(r); + 0 + } + }; + if self.regs[reg::INTERRUPT_CONTROL as usize] & 1 != 0 { + self.pending = true; + } + v + } + pub fn write(&mut self, r: u8, val: u16, now: Instant) { + self.sync(now); + let before = self.status(); + match r { + reg::BASIC_CONTROL => { + let val = val & 0xff80; + if val & 0x8000 != 0 { + self.reset(); + return; + } + if val & 0x80 != 0 { + self.unimplemented(r); + } + self.regs[reg::LOOPBACK as usize] &= !0x380; + if val & 0x4000 != 0 { + self.regs[reg::LOOPBACK as usize] |= 0x30; + } + if val & 0x400 == 0 && val & 0x1200 == 0x1200 { + self.done = false; + self.resolved = 0; + self.forced = false; + self.armed = true; + self.restarting = true; + self.deadline = Some(now + BREAK); + self.complete = Some(now + BREAK + SETTLE); + self.regs[reg::BASIC_CONTROL as usize] = val; + } else if val & 0x400 == 0 && (self.restarting || self.forced && !self.autoneg()) { + if val & 0x1000 == 0 { + self.restarting = false; + self.complete = None; + self.deadline = None; + self.forced = val & 0x800 == 0; + } + self.regs[reg::BASIC_CONTROL as usize] = val & !0x200; + } else { + self.restarting = false; + self.complete = None; + self.deadline = None; + self.forced = false; + self.done = val & 0x400 == 0; + self.armed = self.done; + self.regs[reg::BASIC_CONTROL as usize] = val & !0x8200; + } + if !self.autoneg() { + self.resolved = 0; + self.done = false; + self.forced = self.supported() && self.regs[reg::BASIC_CONTROL as usize] & 0xc00 == 0; + } else if self.done && self.resolved == 0 { + self.resolve(); + } + } + reg::BASIC_STATUS..=reg::PHY_ID2 | reg::AUTO_NEG_PARTNER | reg::AUTO_NEG_EXPANSION | reg::PHY_STATUS | reg::INTERRUPT_STATUS => {} + reg::AUTO_NEG_ADVERTISE => self.regs[reg::AUTO_NEG_ADVERTISE as usize] = val & 0xa5ff, + reg::NEXT_PAGE_TRANSMIT => self.regs[reg::NEXT_PAGE_TRANSMIT as usize] = (val & 0xb7ff) | (self.regs[reg::NEXT_PAGE_TRANSMIT as usize] & 0x800), + reg::INTERRUPT_CONTROL => self.regs[reg::INTERRUPT_CONTROL as usize] = val & 3, + reg::DISCONNECT_COUNTER..=reg::RECEIVE_ERROR_COUNTER => self.regs[r as usize] = val, + reg::PCS_CONFIG => { + if val & !0x20 != 0 { + self.unimplemented(r); + } + let mask = if self.autoneg() || self.regs[reg::PHY_CONTROL as usize] & 0x20 != 0 { + 0xbfef + } else { + 0xffef + }; + self.regs[reg::PCS_CONFIG as usize] = (self.regs[reg::PCS_CONFIG as usize] & !mask) | (val & mask); + } + reg::LOOPBACK => { + if val & 0x4000 != 0 { + self.unimplemented(r); + } + self.regs[reg::LOOPBACK as usize] = val & 0x7fb0; + } + reg::TEN_BASE_T_STATUS => { + if val & 0x55c != 0 { + self.unimplemented(r); + } + self.regs[reg::TEN_BASE_T_STATUS as usize] = val & 0x1ffc; + } + reg::PHY_CONTROL => { + if val & !31 != 0 { + self.unimplemented(r); + } + self.regs[reg::PHY_CONTROL as usize] = val & 0x3bff; + } + _ => self.unimplemented(r), + } + if !self.supported() { + self.resolved = 0; + self.forced = false; + self.unimplemented(r); + } + self.event(before); + if self.regs[reg::INTERRUPT_CONTROL as usize] & 1 != 0 { + self.pending = true; + } + } +} +// The registers and negotiation progress for a snapshot. The two instants +// are saved as the time they had left. +impl Dp83843 { + fn instant_left(at: Option, now: Instant) -> u64 { + // Nanoseconds left plus one; zero for none. + at.map_or(0, |d| d.saturating_duration_since(now).as_nanos() as u64 + 1) + } + + fn instant_from(left: u64, now: Instant) -> Option { + (left != 0).then(|| now + Duration::from_nanos(left - 1)) + } + + pub(crate) fn save_state(&self, now: Instant) -> toml::Value { + StateTable::new() + .u8("configured_addr", self.configured_addr) + .u16s("regs", &self.regs) + .bool("backend_up", self.backend_up) + .bool("done", self.done) + .bool("forced", self.forced) + .u8("resolved", self.resolved) + .bool("armed", self.armed) + .u64("deadline", Self::instant_left(self.deadline, now)) + .u64("complete", Self::instant_left(self.complete, now)) + .bool("pending", self.pending) + .bool("restarting", self.restarting) + .u32("unimplemented_accesses", self.unimplemented_accesses) + .build() + } + + pub(crate) fn load_state(&mut self, v: &toml::Value, now: Instant) -> Result<(), String> { + let items = field(v, "regs")?.as_array().ok_or("regs: not a list")?; + if items.len() != 32 { + return Err(format!("regs: {} entries, expected 32", items.len())); + } + let mut regs = [0u16; 32]; + for (r, item) in regs.iter_mut().zip(items) { + *r = toml_u16(item).ok_or("regs: an entry is out of range")?; + } + *self = Self { + configured_addr: field_u8(v, "configured_addr")?, + regs, + backend_up: field_bool(v, "backend_up")?, + done: field_bool(v, "done")?, + forced: field_bool(v, "forced")?, + resolved: field_u8(v, "resolved")?, + armed: field_bool(v, "armed")?, + deadline: Self::instant_from(field_u64(v, "deadline")?, now), + complete: Self::instant_from(field_u64(v, "complete")?, now), + pending: field_bool(v, "pending")?, + restarting: field_bool(v, "restarting")?, + unimplemented_accesses: field_u32(v, "unimplemented_accesses")?, + }; + Ok(()) + } +} + +impl crate::dev::ioc3_eth::MiiPhy for Dp83843 { + fn address(&self) -> u8 { Dp83843::address(self) } + fn deadline(&self) -> Option { Dp83843::deadline(self) } + fn wire_enabled(&self) -> bool { Dp83843::wire_enabled(self) } + fn in_loopback(&self) -> bool { Dp83843::in_loopback(self) } + fn sync(&mut self, now: Instant) { Dp83843::sync(self, now) } + fn read(&mut self, register: u8, now: Instant) -> u16 { Dp83843::read(self, register, now) } + fn write(&mut self, register: u8, value: u16, now: Instant) { Dp83843::write(self, register, value, now) } + fn set_backend_link(&mut self, up: bool, now: Instant) { Dp83843::set_backend_link(self, up, now) } + fn power_on(&mut self) { Dp83843::reset(self) } + fn save_state(&self, now: Instant) -> toml::Value { Dp83843::save_state(self, now) } + fn load_state(&mut self, v: &toml::Value, now: Instant) -> Result<(), String> { Dp83843::load_state(self, v, now) } +} + +#[cfg(test)] +mod tests { + // Cases ported from tests/qtest/sgi-ioc3-eth-test.c in jamesbraid/qemu-sgi-origami. + use super::*; + + /// The guest observes link and negotiation changes through MII status registers. + #[test] + fn the_phy_restarts_negotiation_and_resets_the_link() { + let now = Instant::now(); + let mut p = Dp83843::new(1); + assert_eq!(p.read(reg::BASIC_STATUS, now) & 0x24, 0); + assert!(!p.wire_enabled()); + assert_eq!(p.read(reg::BASIC_STATUS, now + Duration::from_secs(2)) & 0x24, 0x24); + p.write(reg::BASIC_CONTROL, 0x3300, now + Duration::from_secs(3)); + for (millis, restart, link) in [(4499, true, false), (4500, false, false), (6500, false, true)] { + let at = now + Duration::from_millis(millis); + assert_eq!(p.read(reg::BASIC_CONTROL, at) & 0x200 != 0, restart); + assert_eq!(p.read(reg::BASIC_STATUS, at) & 4 != 0, link); + assert_eq!(p.wire_enabled(), link); + } + let at = now + Duration::from_secs(7); + p.set_backend_link(false, at); + p.write(reg::PHY_CONTROL, 7, at); + p.write(reg::BASIC_CONTROL, 0x8000, at); + assert_eq!(p.address(), 1); + assert_eq!(p.read(reg::BASIC_STATUS, at) & 4, 0); + assert_eq!(p.read(reg::BASIC_STATUS, at + Duration::from_secs(2)) & 4, 0); + p.set_backend_link(true, at + Duration::from_secs(2)); + assert!(p.wire_enabled()); + } + + /// A PROM probing the PHY reads the next-page register's reset value and + /// writes it back; the read-only toggle bit must survive every write. + #[test] + fn the_next_page_register_keeps_its_toggle_bit_and_resets_to_0x2001() { + let now = Instant::now(); + let mut p = Dp83843::new(1); + assert_eq!(p.read(reg::NEXT_PAGE_TRANSMIT, now), 0x2001); + for bit in 0..16 { + let value = 1u16 << bit; + p.write(reg::NEXT_PAGE_TRANSMIT, value, now); + assert_eq!(p.read(reg::NEXT_PAGE_TRANSMIT, now), value & 0xb7ff, "bit {bit}"); + } + p.write(reg::NEXT_PAGE_TRANSMIT, 0xffff, now); + assert_eq!(p.read(reg::NEXT_PAGE_TRANSMIT, now), 0xb7ff); + p.write(reg::BASIC_CONTROL, 0x8000, now); + assert_eq!(p.read(reg::NEXT_PAGE_TRANSMIT, now), 0x2001); + } + + /// The PROM's PHY probe expects zero loopback and masked 10BASE-T status + /// registers, plus the standard BMSR and ANER capability bits, both at + /// power-on and after a BMCR reset. + #[test] + fn the_phy_reads_the_probe_values_the_prom_expects_before_and_after_reset() { + let now = Instant::now(); + let mut p = Dp83843::new(1); + for round in 0..2 { + assert_eq!(p.read(reg::BASIC_STATUS, now) & 0x7849, 0x7849, "round {round}"); + assert_eq!(p.read(reg::AUTO_NEG_EXPANSION, now) & 4, 4); + assert_eq!(p.read(reg::LOOPBACK, now), 0); + assert_eq!(p.read(reg::TEN_BASE_T_STATUS, now) & 0xc77f, 0); + p.write(reg::BASIC_CONTROL, 0x2100, now); + p.write(reg::LOOPBACK, 0x20, now); + p.write(reg::BASIC_CONTROL, 0x8000, now); + } + assert_eq!(p.unimplemented_accesses, 0); + } + + /// Setting BMCR loopback makes the PHY report loopback in the LBR bits the + /// MAC consults, and leaving it must drop the stale bypass bits. + #[test] + fn bmcr_loopback_selects_the_loopback_register_and_clears_the_bypass_bits() { + let now = Instant::now(); + let mut p = Dp83843::new(1); + p.write(reg::LOOPBACK, 0x0180, now); + p.write(reg::BASIC_CONTROL, 0x7300, now); + assert_eq!(p.read(reg::LOOPBACK, now) & 0x03b0, 0x0030); + assert!(p.in_loopback()); + let lbr = p.read(reg::LOOPBACK, now); + p.write(reg::LOOPBACK, lbr | 0x0180, now); + p.write(reg::BASIC_CONTROL, 0x3100, now); + assert_eq!(p.read(reg::LOOPBACK, now) & 0x0380, 0); + p.write(reg::BASIC_CONTROL, 0x8000, now); + assert_eq!(p.read(reg::LOOPBACK, now), 0); + assert!(!p.in_loopback()); + } + + /// Drivers poll the error counters and expect each read to return the count + /// since the previous read, not a running total. + #[test] + fn the_error_counters_clear_when_read() { + let now = Instant::now(); + let mut p = Dp83843::new(1); + for r in [reg::DISCONNECT_COUNTER, reg::FALSE_CARRIER_COUNTER, reg::RECEIVE_ERROR_COUNTER] { + p.write(r, 0xbeef, now); + assert_eq!(p.read(r, now), 0xbeef); + assert_eq!(p.read(r, now), 0); + } + } + + /// A guest that restarts negotiation twice in a row must see the break + /// interval from the second write, not a shortened one from the first. + #[test] + fn a_second_restart_request_rearms_the_break_interval() { + let now = Instant::now(); + let ms = |m: u64| now + Duration::from_millis(m); + let mut p = Dp83843::new(1); + p.write(reg::BASIC_CONTROL, 0x3200, ms(0)); + p.write(reg::BASIC_CONTROL, 0x3200, ms(750)); + assert_ne!(p.read(reg::BASIC_CONTROL, ms(2249)) & 0x200, 0); + assert_eq!(p.read(reg::BASIC_CONTROL, ms(2250)) & 0x200, 0); + assert_eq!(p.read(reg::BASIC_STATUS, ms(4249)) & 0x20, 0); + assert_eq!(p.read(reg::BASIC_STATUS, ms(4250)) & 0x20, 0x20); + } + + /// A snapshot keeps the registers, the negotiation in progress and its + /// time left. + #[test] + fn a_snapshot_round_trips_a_negotiation_in_progress() { + let now = Instant::now(); + let mut src = Dp83843::new(1); + src.write(reg::BASIC_CONTROL, 0x3300, now); + let v = src.save_state(now); + let mut dst = Dp83843::new(1); + dst.load_state(&v, now).unwrap(); + assert_eq!(v, dst.save_state(now)); + assert_eq!(src.deadline(), dst.deadline()); + assert!(dst.load_state(&toml::Value::Table(Default::default()), now).is_err()); + } +} diff --git a/src/dev/ds1744.rs b/src/dev/ds1744.rs new file mode 100644 index 000000000..e5e1c30bd --- /dev/null +++ b/src/dev/ds1744.rs @@ -0,0 +1,450 @@ +//! DS1744-compatible timekeeping SRAM: 32 KiB of battery-backed bytes whose +//! last eight are a BCD calendar. +//! +//! The Tezro's IO9 maps one in the IOC4's BAR (byte-bus +//! select 0 of the IOC4). The part reads like the Fuel's M48T35 [`Timekeeper`] at a +//! glance, and has the same geometry and clock offsets, but differs where +//! software looks: the control register carries the century beside the +//! write and read bits, the day register holds a weekday that advances +//! with the calendar, an oscillator-stop bit rides in the seconds, a commit +//! with an impossible date is refused, and a test-mode bit makes the +//! seconds' low bit run at 1024 Hz. Those are a different calendar rather +//! than a flag on the M48T35's, so this is its own model. +//! +//! The guest's write bit stages a calendar image and the clearing of it +//! commits it; the read bit freezes the visible calendar while the real one +//! runs on. +//! +//! # What is approximated here +//! +//! * The fitted IO9 part and its physical address wiring are unknown. The +//! 32 KiB geometry is the profile's choice. +//! * With a fresh-year base of 1968 the guest's year digits count from that +//! year, as its driver expects: the host date is shifted into 2000 to +//! 2099 and the weekday comes from the unshifted date, so the weekday +//! follows the host's real calendar while the digits follow the shifted +//! one. +//! * Host time and a backing file stand in for the oscillator and battery. +//! A restart restores a completed calendar and lets host time pass; an +//! unfinished write or read transaction is not restored. +//! +//! Ported from hw/rtc/ds1742.c in jamesbraid/qemu-sgi-origami. + +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::time::{Instant, SystemTime, UNIX_EPOCH}; + +use parking_lot::Mutex; + +use crate::traits::{BusDevice, BusRead8, BUS_ERR, BUS_OK}; + +pub const SIZE: usize = 0x8000; +/// The control register, first of the eight calendar bytes. +pub const CONTROL: usize = 0x7ff8; +const WRITE: u8 = 0x80; +const READ: u8 = 0x40; +/// Seconds bit 7: the oscillator is stopped. +const OSCILLATOR_STOP: u8 = 0x80; +/// Day bit 6: test mode; bit 7: the battery flag, read-only and always set. +const TEST_MODE: u8 = 0x40; +const BATTERY_FLAG: u8 = 0x80; +const CLOCK_REF_BYTES: usize = 16; +const CLOCK_REF_MAGIC: &[u8; 8] = b"DS1742R1"; +/// Test mode flips the seconds' low bit every 976562 ns. +const TEST_BIT_NS: i64 = 976_562; + +/// How the guest's year digits relate to the host's calendar. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum YearBase { + /// The host's own year. + Gregorian, + /// The year digits count from this year: the host date is shifted into + /// 2000 to 2099. Only 1968 is accepted. + From(u32), +} + +fn bcd(v: i64) -> u8 { + ((v / 10) * 16 + v % 10) as u8 +} + +fn from_bcd(v: u8) -> i64 { + i64::from((v >> 4) * 10 + (v & 15)) +} + +fn bcd_in(v: u8, lo: i64, hi: i64) -> bool { + v & 15 < 10 && v >> 4 < 10 && (lo..=hi).contains(&from_bcd(v)) +} + +fn is_leap(year: i64) -> bool { + year % 4 == 0 && (year % 100 != 0 || year % 400 == 0) +} + +fn month_days(year: i64, month: i64) -> i64 { + match month { + 2 => 28 + i64::from(is_leap(year)), + 4 | 6 | 9 | 11 => 30, + _ => 31, + } +} + +/// Days since 1970-01-01 of a Gregorian date. +fn days_from_civil(year: i64, month: i64, day: i64) -> i64 { + let y = if month <= 2 { year - 1 } else { year }; + let era = y.div_euclid(400); + let yoe = y - era * 400; + let doy = (153 * ((month + 9) % 12) + 2) / 5 + day - 1; + let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy; + era * 146_097 + doe - 719_468 +} + +/// The Gregorian date of a day count since 1970-01-01. +fn civil_from_days(days: i64) -> (i64, i64, i64) { + let z = days + 719_468; + let era = z.div_euclid(146_097); + let doe = z - era * 146_097; + let yoe = (doe - doe / 1460 + doe / 36_524 - doe / 146_096) / 365; + let doy = doe - (365 * yoe + yoe / 4 - yoe / 100); + let mp = (5 * doy + 2) / 153; + let day = doy - (153 * mp + 2) / 5 + 1; + let month = if mp < 10 { mp + 3 } else { mp - 9 }; + let year = yoe + era * 400 + i64::from(month <= 2); + (year, month, day) +} + +/// Day number of `seconds`, rounding down. +fn day_index(seconds: i64) -> i64 { + seconds.div_euclid(86_400) +} + +/// The 1-based weekday (Sunday is 1) of a count of days since 1970-01-01, +/// a Thursday. +fn weekday_of_day(days: i64) -> u8 { + (days + 4).rem_euclid(7) as u8 + 1 +} + +struct State { + ram: Vec, + control: u8, + /// Test-mode flag and weekday as last written (FT and bits 2:0). + weekday: u8, + staged: [u8; 8], + frozen: [u8; 8], + frozen_valid: bool, + /// Unix seconds of the calendar when it was last set. + epoch: i64, + /// Clock reading (ns) when it was last set. + reference_ns: i64, + stopped: bool, + file: Option, + /// A backing write failed; the next equal write retries it. + storage_failed: bool, +} + +/// The nanosecond clock the calendar runs on. +type Clock = Arc i64 + Send + Sync>; + +pub struct Ds1744 { + st: Mutex, + clock: Clock, +} + +impl Ds1744 { + /// A part whose calendar starts at the host's date and time. + pub fn new(base: YearBase) -> Result { + let host = SystemTime::now().duration_since(UNIX_EPOCH).map_err(|e| e.to_string())?.as_secs() as i64; + let born = Instant::now(); + Ds1744::with_clock(host, base, Arc::new(move || born.elapsed().as_nanos() as i64)) + } + + /// A part seeded with `host_unix_secs` and running on `clock`, a count of + /// nanoseconds. Tests supply a clock they step. + pub fn with_clock(host_unix_secs: i64, base: YearBase, clock: Clock) -> Result { + let mut epoch = host_unix_secs; + let (year, month, day) = civil_from_days(day_index(host_unix_secs)); + let mut control = bcd(year / 100); + let weekday = weekday_of_day(day_index(host_unix_secs)); + match base { + YearBase::Gregorian => {} + YearBase::From(1968) => { + let base = 1968; + if !(base..=base + 99).contains(&year) { + return Err(format!("ds1744: fresh-year-base={base} cannot seed host year {year}")); + } + let shifted = 2000 + (year - base) % 100; + // A leap day in a shifted year without one is the next day. + epoch = days_from_civil(shifted, month, day) * 86_400 + host_unix_secs.rem_euclid(86_400); + control = bcd(shifted / 100); + } + YearBase::From(other) => return Err(format!("ds1744: fresh-year-base must be 0 or 1968, not {other}")), + } + let now = clock(); + Ok(Ds1744 { + st: Mutex::new(State { + ram: vec![0; CONTROL], + control, + weekday, + staged: [0; 8], + frozen: [0; 8], + frozen_valid: false, + epoch, + reference_ns: now, + stopped: false, + file: None, + storage_failed: false, + }), + clock, + }) + } + + fn now_ns(&self) -> i64 { + (self.clock)() + } + + fn seconds(&self, st: &State) -> i64 { + st.epoch + if st.stopped { 0 } else { (self.now_ns() - st.reference_ns) / 1_000_000_000 } + } + + /// The calendar bytes as the part would show them now. + fn image(&self, st: &State) -> [u8; 8] { + let now = self.seconds(st); + let (year, month, day) = civil_from_days(day_index(now)); + let sec = now.rem_euclid(86_400); + let weekday = ((i64::from(st.weekday & 7) - 1 + day_index(now) - day_index(st.epoch)).rem_euclid(7)) as u8; + [ + (st.control & 0xc0) | bcd(year / 100), + bcd(sec % 60) | if st.stopped { OSCILLATOR_STOP } else { 0 }, + bcd(sec / 60 % 60), + bcd(sec / 3600), + BATTERY_FLAG | (st.weekday & TEST_MODE) | (weekday + 1), + bcd(day), + bcd(month), + bcd(year % 100), + ] + } + + /// Make the staged image the calendar, if it is a date. A refusal leaves + /// the calendar as it was. + fn commit(&self, st: &mut State, century: u8) -> bool { + let s = st.staged; + let valid = bcd_in(century, 0, 39) + && bcd_in(s[1] & 0x7f, 0, 59) + && bcd_in(s[2], 0, 59) + && bcd_in(s[3], 0, 23) + && bcd_in(s[4] & 7, 1, 7) + && bcd_in(s[5], 1, 31) + && bcd_in(s[6], 1, 12) + && bcd_in(s[7], 0, 99); + if !valid { + crate::dlog!(crate::devlog::LogModule::Rtc, "DS1744: invalid calendar write"); + return false; + } + let year = from_bcd(century) * 100 + from_bcd(s[7]); + let (month, day) = (from_bcd(s[6]), from_bcd(s[5])); + if day > month_days(year, month) { + crate::dlog!(crate::devlog::LogModule::Rtc, "DS1744: unsupported calendar date {year:04}-{month:02}-{day:02}"); + return false; + } + st.epoch = days_from_civil(year, month, day) * 86_400 + + from_bcd(s[3]) * 3600 + + from_bcd(s[2]) * 60 + + from_bcd(s[1] & 0x7f); + st.reference_ns = self.now_ns(); + st.weekday = s[4] & (TEST_MODE | 7); + st.stopped = s[1] & OSCILLATOR_STOP != 0; + true + } + + // ----- backing file ----- + + /// Back the part with a 32 KiB file and a 16-byte clock reference beside + /// it (`.clock`). A missing file starts blank and is created; a file + /// of another size is refused. A saved calendar is restored and lets the + /// host time since it was saved pass; a blank one keeps the seed. + pub fn set_nvram_file(&self, path: &Path) -> Result<(), String> { + let clock_path = PathBuf::from(format!("{}.clock", path.display())); + let image = match std::fs::read(path) { + Ok(v) if v.len() == SIZE => Some(v), + Ok(_) => return Err("DS1744: backing image must contain exactly 32768 bytes".into()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => None, + Err(e) => return Err(e.to_string()), + }; + let reference = match std::fs::read(&clock_path) { + Ok(v) if v.len() >= CLOCK_REF_BYTES => Some(v), + Ok(_) => return Err("DS1744: clock reference is shorter than 16 bytes".into()), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => None, + Err(e) => return Err(e.to_string()), + }; + let mut st = self.st.lock(); + if let Some(image) = image { + st.ram.copy_from_slice(&image[..CONTROL]); + let cal: [u8; 8] = image[CONTROL..].try_into().expect("eight calendar bytes"); + if [cal[1], cal[2], cal[3], cal[5], cal[6], cal[7]].iter().any(|&b| b != 0) { + self.restore_clock(&mut st, cal, reference.as_deref()); + } + } + st.file = Some(path.to_path_buf()); + self.save_clock(&mut st); + Ok(()) + } + + fn restore_clock(&self, st: &mut State, cal: [u8; 8], reference: Option<&[u8]>) { + st.staged = cal; + if !self.commit(st, cal[0] & 0x3f) { + crate::dlog!(crate::devlog::LogModule::Rtc, "DS1744: invalid saved calendar; using the seed"); + return; + } + st.control = cal[0] & 0x3f; + let mut elapsed = 0; + if !st.stopped { + if let Some(r) = reference.filter(|r| &r[..8] == CLOCK_REF_MAGIC) { + let saved = i64::from_le_bytes(r[8..16].try_into().expect("eight bytes")); + elapsed = (unix_now() - saved).max(0); + } + } + if elapsed != 0 { + let days = day_index(st.epoch + elapsed) - day_index(st.epoch); + st.weekday = (st.weekday & TEST_MODE) | (((i64::from(st.weekday & 7) - 1 + days % 7) % 7) as u8 + 1); + st.epoch += elapsed; + } + } + + /// Write the whole image and the clock reference. Errors are reported + /// and remembered so an equal write retries them. + fn save_clock(&self, st: &mut State) { + let Some(path) = st.file.clone() else { return }; + let mut image = st.ram.clone(); + image.extend_from_slice(&self.image(st)); + let mut reference = CLOCK_REF_MAGIC.to_vec(); + reference.extend_from_slice(&unix_now().to_le_bytes()); + let clock_path = format!("{}.clock", path.display()); + let result = std::fs::write(&path, &image).and_then(|()| std::fs::write(clock_path, &reference)); + st.storage_failed = result.is_err(); + if let Err(e) = result { + log::error!("DS1744: could not save {}: {e}", path.display()); + } + } + + /// Save the calendar now, as a stop or shutdown does. + pub fn flush(&self) { + let mut st = self.st.lock(); + self.save_clock(&mut st); + } + + fn read(&self, a: usize) -> u8 { + let st = self.st.lock(); + if a < CONTROL { + return st.ram[a]; + } + let image = self.image(&st); + if a == CONTROL { + return if st.control & WRITE != 0 { + st.control + } else if st.frozen_valid { + (st.control & 0xc0) | (st.frozen[0] & 0x3f) + } else { + image[0] + }; + } + let i = a - CONTROL; + let mut value = if st.control & WRITE != 0 { + st.staged[i] + } else if st.frozen_valid { + st.frozen[i] + } else { + image[i] + }; + if i == 4 { + value |= BATTERY_FLAG; + } + if i == 1 && st.weekday & TEST_MODE != 0 && !st.stopped { + value = (value & !1) | ((self.now_ns() / TEST_BIT_NS) & 1) as u8; + } + value + } + + fn write(&self, a: usize, value: u8) { + let mut st = self.st.lock(); + if a < CONTROL { + if st.ram[a] == value && !st.storage_failed { + return; + } + st.ram[a] = value; + if let Some(path) = &st.file { + use std::io::{Seek, SeekFrom, Write}; + let result = std::fs::OpenOptions::new().write(true).open(path).and_then(|mut f| { + f.seek(SeekFrom::Start(a as u64))?; + f.write_all(&[value]) + }); + st.storage_failed = result.is_err(); + if let Err(e) = result { + log::error!("DS1744: could not save SRAM byte at {a:#x}: {e}"); + } + } + return; + } + if a != CONTROL { + if st.control & WRITE != 0 { + st.staged[a - CONTROL] = value; + } else { + crate::dlog!(crate::devlog::LogModule::Rtc, "DS1744: calendar write with W clear at {a:#x}"); + } + return; + } + if st.control & (WRITE | READ) == 0 { + st.control = self.image(&st)[0]; + } + let old = st.control; + if value & READ != 0 && old & READ == 0 { + st.frozen = self.image(&st); + st.frozen_valid = true; + } else if value & READ == 0 { + st.frozen_valid = false; + } + if value & WRITE != 0 && old & WRITE == 0 { + st.staged = self.image(&st); + } + if value & WRITE == 0 && old & WRITE != 0 { + let valid = self.commit(&mut st, value & 0x3f); + st.control = (value & 0xc0) | if valid { value & 0x3f } else { old & 0x3f }; + if valid { + self.save_clock(&mut st); + } + return; + } + // The control rules preserve the century while W or R is set. + st.control = (value & 0xc0) | (old & 0x3f); + } +} + +fn unix_now() -> i64 { + SystemTime::now().duration_since(UNIX_EPOCH).map_or(0, |d| d.as_secs() as i64) +} + +impl BusDevice for Ds1744 { + fn read8(&self, addr: u64) -> BusRead8 { + match usize::try_from(addr) { + Ok(a) if a < SIZE => BusRead8 { status: BUS_OK, data: self.read(a) }, + _ => BusRead8::err(), + } + } + + fn write8(&self, addr: u64, val: u8) -> u32 { + match usize::try_from(addr) { + Ok(a) if a < SIZE => { + self.write(a, val); + BUS_OK + } + _ => BUS_ERR, + } + } +} + +impl Drop for Ds1744 { + fn drop(&mut self) { + self.flush(); + } +} + +#[cfg(test)] +#[path = "ds1744_tests.rs"] +mod tests; diff --git a/src/dev/ds1744_tests.rs b/src/dev/ds1744_tests.rs new file mode 100644 index 000000000..4ca13cca8 --- /dev/null +++ b/src/dev/ds1744_tests.rs @@ -0,0 +1,305 @@ +//! DS1744 register-contract tests, through the byte bus with a clock the +//! test steps. +//! +//! Ported from tests/qtest/ds1742-test.c in jamesbraid/qemu-sgi-origami. + +use super::*; +use std::sync::atomic::{AtomicI64, Ordering}; + +const SECOND: i64 = 1_000_000_000; + +fn unix(y: i64, mo: i64, d: i64, h: i64, mi: i64, s: i64) -> i64 { + days_from_civil(y, mo, d) * 86_400 + h * 3600 + mi * 60 + s +} + +struct Rig { + d: Ds1744, + ns: Arc, +} + +impl Rig { + fn at(host: i64, base: YearBase) -> Rig { + let ns = Arc::new(AtomicI64::new(0)); + let n = ns.clone(); + let d = Ds1744::with_clock(host, base, Arc::new(move || n.load(Ordering::SeqCst))).unwrap(); + Rig { d, ns } + } + + /// The qtest's default: January 1st 2024, on the host's calendar. + fn new() -> Rig { + Rig::at(unix(2024, 1, 1, 0, 0, 0), YearBase::Gregorian) + } + + fn step(&self, ns: i64) { + self.ns.fetch_add(ns, Ordering::SeqCst); + } + + fn rd(&self, a: usize) -> u8 { + let r = self.d.read8(a as u64); + assert_eq!(r.status, BUS_OK); + r.data + } + + fn wr(&self, a: usize, v: u8) { + assert_eq!(self.d.write8(a as u64, v), BUS_OK); + } + + /// Set the calendar through the write bit, with a weekday. + #[allow(clippy::too_many_arguments)] + fn set_time_weekday(&self, sec: u8, min: u8, hour: u8, date: u8, month: u8, year: u8, century: u8, weekday: u8) { + self.wr(CONTROL, 0x80); + for (i, v) in [sec, min, hour, weekday, date, month, year].into_iter().enumerate() { + self.wr(CONTROL + 1 + i, v); + } + self.wr(CONTROL, century); + } + + #[allow(clippy::too_many_arguments)] + fn set_time(&self, sec: u8, min: u8, hour: u8, date: u8, month: u8, year: u8, century: u8) { + self.set_time_weekday(sec, min, hour, date, month, year, century, 0x02); + } + + /// The guest-visible calendar: control, seconds, minutes, hours, weekday, + /// date, month, year. + fn calendar(&self) -> [u8; 8] { + std::array::from_fn(|i| self.rd(CONTROL + i)) + } +} + +/// Ports test_fresh_seed: with the 1968 base the guest's year digits count +/// from 1968 and the weekday follows the host's calendar, across a day. +#[test] +fn a_fresh_seed_counts_years_from_1968() { + let r = Rig::at(unix(2024, 2, 28, 23, 59, 59), YearBase::From(1968)); + assert_eq!(r.rd(CONTROL), 0x20); + assert_eq!((r.rd(CONTROL + 7), r.rd(CONTROL + 6), r.rd(CONTROL + 5)), (0x56, 0x02, 0x28)); + assert_eq!(r.rd(CONTROL + 4) & 7, 4); + r.step(2 * SECOND); + assert_eq!(r.rd(CONTROL) & 0x3f, 0x20); + assert_eq!((r.rd(CONTROL + 7), r.rd(CONTROL + 6), r.rd(CONTROL + 5)), (0x56, 0x02, 0x29)); + assert_eq!(r.rd(CONTROL + 1), 0x01); + assert_eq!(r.rd(CONTROL + 4) & 7, 5); + r.step(86_400 * SECOND); + assert_eq!((r.rd(CONTROL + 7), r.rd(CONTROL + 6), r.rd(CONTROL + 5)), (0x56, 0x03, 0x01)); + assert_eq!(r.rd(CONTROL + 4) & 7, 6); +} + +/// Ports test_fresh_seed_default: with no base the guest sees the host's year. +#[test] +fn without_a_base_the_guest_sees_the_host_year() { + let r = Rig::at(unix(2024, 2, 28, 23, 59, 59), YearBase::Gregorian); + assert_eq!(r.rd(CONTROL), 0x20); + assert_eq!((r.rd(CONTROL + 7), r.rd(CONTROL + 6), r.rd(CONTROL + 5)), (0x24, 0x02, 0x28)); + assert_eq!(r.rd(CONTROL + 4) & 7, 4); +} + +/// Ports test_fresh_seed_year_boundary: 2067 maps to 2099, the century +/// register steps to 21 and the shifted 2100 is not a leap year. +#[test] +fn the_shifted_century_rolls_and_2100_is_not_a_leap_year() { + let r = Rig::at(unix(2067, 12, 31, 23, 59, 59), YearBase::From(1968)); + assert_eq!(r.rd(CONTROL), 0x20); + assert_eq!((r.rd(CONTROL + 7), r.rd(CONTROL + 6), r.rd(CONTROL + 5)), (0x99, 0x12, 0x31)); + assert_eq!(r.rd(CONTROL + 4) & 7, 7); + r.step(SECOND); + assert_eq!(r.rd(CONTROL), 0x21); + assert_eq!((r.rd(CONTROL + 7), r.rd(CONTROL + 6), r.rd(CONTROL + 5)), (0x00, 0x01, 0x01)); + assert_eq!(r.rd(CONTROL + 4) & 7, 1); + r.step((58 * 86_400 + 86_399) * SECOND); + assert_eq!((r.rd(CONTROL), r.rd(CONTROL + 7), r.rd(CONTROL + 6), r.rd(CONTROL + 5)), (0x21, 0x00, 0x02, 0x28)); + assert_eq!(r.rd(CONTROL + 4) & 7, 3); + r.step(SECOND); + assert_eq!((r.rd(CONTROL + 6), r.rd(CONTROL + 5)), (0x03, 0x01)); + assert_eq!(r.rd(CONTROL + 4) & 7, 4); +} + +/// Ports test_ds1744_geometry: RAM below the calendar, including its last +/// byte, and an address past the part faults. +#[test] +fn ram_fills_the_part_below_the_calendar() { + let r = Rig::new(); + for (a, v) in [(0, 0x12), (0x800, 0x34), (0x7ff7, 0x56)] { + r.wr(a, v); + } + assert_eq!((r.rd(0), r.rd(0x800), r.rd(0x7ff7)), (0x12, 0x34, 0x56)); + r.set_time(0x12, 0x23, 0x10, 0x25, 0x09, 0x26, 0x20); + assert_eq!(r.rd(CONTROL), 0x20); + assert_eq!((r.rd(CONTROL + 1), r.rd(CONTROL + 2), r.rd(CONTROL + 3)), (0x12, 0x23, 0x10)); + assert_eq!((r.rd(CONTROL + 5), r.rd(CONTROL + 6), r.rd(CONTROL + 7)), (0x25, 0x09, 0x26)); + assert_eq!(r.rd(CONTROL + 4) & 7, 2); + assert_eq!(r.d.read8(SIZE as u64).status, BUS_ERR); + assert_eq!(r.d.write8(SIZE as u64, 1), BUS_ERR); +} + +/// Ports test_calendar: the control register shows the century beside the +/// write and read bits, the battery flag and weekday advance, and a +/// century carries. +#[test] +fn control_shows_the_century_and_the_calendar_advances() { + let r = Rig::new(); + assert_eq!(r.rd(CONTROL), 0x20); + r.wr(CONTROL, 0x40); + assert_eq!(r.rd(CONTROL), 0x60); + r.wr(CONTROL, 0x00); + assert_eq!(r.rd(CONTROL), 0x20); + r.wr(CONTROL, 0x80); + assert_eq!(r.rd(CONTROL), 0xa0); + r.wr(CONTROL, 0x20); + r.set_time(0x59, 0x59, 0x23, 0x31, 0x12, 0x24, 0x20); + assert_eq!(r.rd(CONTROL + 1), 0x59); + assert_eq!(r.rd(CONTROL + 4) & 0x80, 0x80, "battery flag"); + assert_eq!(r.rd(CONTROL + 4) & 7, 2); + r.step(SECOND); + assert_eq!(r.rd(CONTROL + 4) & 7, 3, "the weekday advances with the date"); + assert_eq!((r.rd(CONTROL + 1), r.rd(CONTROL + 5), r.rd(CONTROL + 6), r.rd(CONTROL + 7)), (0, 1, 1, 0x25)); + r.set_time(0x59, 0x59, 0x23, 0x31, 0x12, 0x99, 0x20); + r.step(SECOND); + assert_eq!(r.rd(CONTROL) & 0x3f, 0x21); + assert_eq!(r.rd(CONTROL + 7), 0); + // A commit that is a date is taken, with a different century. + r.wr(CONTROL, 0x80); + r.wr(CONTROL + 6, 0); + r.wr(CONTROL, 0x19); + assert_eq!(r.rd(CONTROL), 0x21, "month 0 is no date, so the century stays"); + assert_eq!(r.rd(CONTROL + 6), 1); + // The read bit freezes what the guest sees while the clock runs. + r.wr(CONTROL, 0x40); + r.step(2 * SECOND); + assert_eq!(r.rd(CONTROL + 1), 0); + r.wr(CONTROL, 0); + assert_eq!(r.rd(CONTROL + 1), 2); +} + +/// Ports test_invalid_dates: a commit that is not a date leaves the calendar +/// alone, a valid leap day is taken, and the stop bit rides in the seconds. +#[test] +fn a_commit_that_is_not_a_date_is_refused() { + let r = Rig::new(); + r.set_time(0x80 | 0x12, 0x34, 0x05, 0x15, 0x01, 0x24, 0x20); + for (date, month, year) in [(0x30, 0x02, 0x24), (0x29, 0x02, 0x23)] { + r.set_time(0x59, 0x59, 0x23, date, month, year, 0x20); + assert_eq!(r.rd(CONTROL + 1), 0x92, "stopped at the earlier time"); + assert_eq!((r.rd(CONTROL + 5), r.rd(CONTROL + 6), r.rd(CONTROL + 7)), (0x15, 0x01, 0x24)); + } + r.set_time(0x59, 0x59, 0x23, 0x29, 0x02, 0x24, 0x20); + r.step(SECOND); + assert_eq!((r.rd(CONTROL + 1), r.rd(CONTROL + 5), r.rd(CONTROL + 6), r.rd(CONTROL + 7)), (0, 1, 3, 0x24)); +} + +/// Ports test_stop_and_test_mode: a stopped oscillator holds the seconds, +/// the battery flag is read-only, and the test-mode bit flips the seconds' +/// low bit every 976562 ns unless the oscillator is stopped. +#[test] +fn test_mode_flips_the_low_second_bit() { + let r = Rig::new(); + r.set_time(0x80 | 0x12, 0x34, 0x05, 0x06, 0x07, 0x26, 0x20); + r.step(3 * SECOND); + assert_eq!(r.rd(CONTROL + 1), 0x92); + r.wr(CONTROL, 0x80); + r.wr(CONTROL + 1, 0x12); + r.wr(CONTROL + 4, 0xc2); + r.wr(CONTROL, 0x20); + assert_eq!(r.rd(CONTROL + 4), 0xc2); + assert_eq!(r.rd(CONTROL + 1) & 0xfe, 0x12); + let bit = r.rd(CONTROL + 1) & 1; + r.step(976_562); + assert_eq!(r.rd(CONTROL + 1) & 1, bit ^ 1); + r.wr(CONTROL, 0x80); + r.wr(CONTROL + 4, 0x02); + r.wr(CONTROL, 0x20); + let bit = r.rd(CONTROL + 1) & 1; + r.step(976_562); + assert_eq!(r.rd(CONTROL + 1) & 1, bit, "test mode off"); + r.wr(CONTROL, 0x80); + r.wr(CONTROL + 4, 0x42); + r.wr(CONTROL + 1, 0x92); + r.wr(CONTROL, 0x20); + let bit = r.rd(CONTROL + 1) & 1; + r.step(3 * 976_562); + assert_eq!(r.rd(CONTROL + 1) & 1, bit, "stopped, so it holds"); +} + +/// Ports test_restart_persistence: RAM and a completed calendar come back +/// from the file, the elapsed host time passes, a stopped clock does not +/// advance, and a file of the wrong size is refused. +#[test] +fn a_backing_file_restores_ram_and_the_calendar() { + let dir = std::env::temp_dir().join(format!("ds1744-{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let path = dir.join("clock"); + { + let r = Rig::new(); + r.d.set_nvram_file(&path).unwrap(); + r.wr(0x100, 0xa5); + r.set_time(0x30, 0x15, 0x08, 0x15, 0x06, 0x30, 0x20); + r.d.flush(); + } + let reopened = Rig::at(unix(2024, 1, 1, 0, 0, 0), YearBase::Gregorian); + reopened.d.set_nvram_file(&path).unwrap(); + assert_eq!(reopened.rd(0x100), 0xa5); + let c = reopened.calendar(); + assert_eq!((c[2], c[3], c[5], c[6], c[7]), (0x15, 0x08, 0x15, 0x06, 0x30)); + assert_eq!(c[0], 0x20); + + let short = dir.join("short"); + std::fs::write(&short, [0u8; 16]).unwrap(); + assert!(Rig::new().d.set_nvram_file(&short).is_err()); + std::fs::remove_dir_all(&dir).unwrap(); +} + +/// One step of a register sequence: write, read and expect, or let time pass. +enum Step { + W(usize, u8), + R(usize, u8), + Advance(i64), +} + +/// The register sequences QEMU's ds1742-test asserts, as one table so the +/// two models can be held to the same bytes. Offsets are from the 32 KiB +/// part's calendar (`CONTROL`); the seed is 2024-01-01T00:00:00. +/// +/// Ports test_calendar, test_invalid_dates and test_stop_and_test_mode +/// from tests/qtest/ds1742-test.c in jamesbraid/qemu-sgi-origami. The +/// model's answers come from the QEMU file's assertions, not from running +/// QEMU here; the end-to-end comparison with a live QEMU is not done. +#[test] +fn the_qemu_register_sequences_read_back_the_same() { + use Step::{Advance, R, W}; + const C: usize = CONTROL; + let set = |s, mi, h, wd, d, mo, y, cent| { + vec![W(C, 0x80), W(C + 1, s), W(C + 2, mi), W(C + 3, h), W(C + 4, wd), W(C + 5, d), W(C + 6, mo), W(C + 7, y), W(C, cent)] + }; + let mut steps = vec![R(C, 0x20), W(C, 0x40), R(C, 0x60), W(C, 0x00), R(C, 0x20), W(C, 0x80), R(C, 0xa0), W(C, 0x20)]; + // A rollover into 2025. + steps.extend(set(0x59, 0x59, 0x23, 0x02, 0x31, 0x12, 0x24, 0x20)); + steps.extend([R(C + 1, 0x59), R(C + 4, 0x82), Advance(1_000_000_000)]); + steps.extend([R(C + 4, 0x83), R(C + 1, 0x00), R(C + 5, 0x01), R(C + 6, 0x01), R(C + 7, 0x25)]); + // A century carry. + steps.extend(set(0x59, 0x59, 0x23, 0x02, 0x31, 0x12, 0x99, 0x20)); + steps.extend([Advance(1_000_000_000), R(C, 0x21), R(C + 7, 0x00)]); + // An impossible month leaves the calendar and its century alone. + steps.extend([W(C, 0x80), W(C + 6, 0x00), W(C, 0x19), R(C, 0x21), R(C + 6, 0x01)]); + // The read bit freezes the display; the clock runs on. + steps.extend([W(C, 0x40), Advance(2_000_000_000), R(C + 1, 0x00), W(C, 0x00), R(C + 1, 0x02)]); + // An impossible date is refused; a leap day is taken. + steps.extend(set(0x92, 0x34, 0x05, 0x02, 0x15, 0x01, 0x24, 0x20)); + steps.extend(set(0x59, 0x59, 0x23, 0x02, 0x30, 0x02, 0x24, 0x20)); + steps.extend([R(C + 1, 0x92), R(C + 5, 0x15), R(C + 6, 0x01), R(C + 7, 0x24)]); + steps.extend(set(0x59, 0x59, 0x23, 0x02, 0x29, 0x02, 0x23, 0x20)); + steps.extend([R(C + 1, 0x92), R(C + 5, 0x15)]); + steps.extend(set(0x59, 0x59, 0x23, 0x02, 0x29, 0x02, 0x24, 0x20)); + steps.extend([Advance(1_000_000_000), R(C + 1, 0x00), R(C + 5, 0x01), R(C + 6, 0x03), R(C + 7, 0x24)]); + // The stop bit holds the seconds; test mode makes the low bit run. + steps.extend(set(0x80 | 0x12, 0x34, 0x05, 0x02, 0x06, 0x07, 0x26, 0x20)); + steps.extend([Advance(3_000_000_000), R(C + 1, 0x92)]); + steps.extend([W(C, 0x80), W(C + 1, 0x12), W(C + 4, 0xc2), W(C, 0x20), R(C + 4, 0xc2)]); + + let r = Rig::new(); + for (i, step) in steps.iter().enumerate() { + match *step { + W(a, v) => r.wr(a, v), + R(a, want) => assert_eq!(r.rd(a), want, "step {i}: read of {a:#x}"), + Advance(ns) => r.step(ns), + } + } +} diff --git a/src/dev/ics1890.rs b/src/dev/ics1890.rs new file mode 100644 index 000000000..04cc6706a --- /dev/null +++ b/src/dev/ics1890.rs @@ -0,0 +1,423 @@ +//! The ICS1890 transceiver fitted to Octane's IOC3. +//! +//! This module is shared by the Octane IOC3 Ethernet model. It was moved here +//! from `ioc3_eth.rs`; its management-register behavior follows the existing +//! Octane behavior and Integrated Circuit Systems, "ICS1890 10Base-T/100Base-TX +//! Integrated PHYceiver", document ICS1890RevG, 21 October 1997. +//! +//! Ported from hw/net/ics1890.c in jamesbraid/qemu-sgi-origami. + +use std::time::{Duration, Instant}; + +use crate::dev::ioc3_eth::MiiPhy; +use crate::snapshot::{field_bool, field_u16, field_u32, field_u64, field_u8, StateTable}; + +/// Ordinary and command-override writable masks, by register. +fn phy_masks(r: usize) -> (u16, u16) { + match r { + 0 => (0xff80, 0), + 1 => (0, 0x7f80), + 2 | 3 => (0, 0xffff), + 4 => (0x21e0, 0x1c1f), + 6 => (0, 0xffe0), + 0x10 => (0x802d, 0), + 0x12 => (0x002f, 0), + 0x13 => (0x0011, 0), + _ => (0, 0), + } +} + +/// The ICS1890 as the Octane carries it: address 1, node mode, software +/// priority, revision 3. +pub struct Ics1890 { + addr: u8, + bmcr: u16, + bmsr_caps: u16, + id1: u16, + id2: u16, + anar: u16, + partner: u16, + exp: u16, + ext_ctl: u16, + tenbt: u16, + ext2: u16, + override_armed: bool, + page_received: bool, + link_low_latch: bool, + last_link: bool, + backend_up: bool, + nego_done: bool, + deadline: Option, + delay: Duration, + /// Accesses to registers the part does not implement. + pub unimplemented_accesses: u32, +} + +impl Ics1890 { + pub fn new(addr: u8, delay: Duration) -> Self { + let mut p = Self { + addr, + bmcr: 0, + bmsr_caps: 0, + id1: 0, + id2: 0, + anar: 0, + partner: 0, + exp: 0, + ext_ctl: 0, + tenbt: 0, + ext2: 0, + override_armed: false, + page_received: false, + link_low_latch: false, + last_link: false, + backend_up: true, + nego_done: false, + deadline: None, + delay, + unimplemented_accesses: 0, + }; + p.reset(Instant::now()); + p + } + + pub fn address(&self) -> u8 { + self.addr + } + + /// Hardware and software reset: defaults, no override, no latches, and a + /// fresh negotiation if enabled. + pub fn reset(&mut self, now: Instant) { + self.bmcr = 0x3000; + self.bmsr_caps = 0x7800; + self.id1 = 0x0015; + self.id2 = 0xf423; + self.anar = 0x01e1; + self.partner = 0; + self.exp = 0; + self.ext_ctl = 0x0008; + self.tenbt = 0; + self.ext2 = 0x0001; + self.override_armed = false; + self.page_received = false; + self.link_low_latch = false; + self.nego_done = false; + self.deadline = None; + self.last_link = false; + self.start_negotiation(now); + } + + fn autoneg(&self) -> bool { + self.bmcr & 0x1000 != 0 + } + fn power_down(&self) -> bool { + self.bmcr & 0x0800 != 0 + } + fn isolated(&self) -> bool { + self.bmcr & 0x0400 != 0 + } + + /// The live link: down in power-down, down without a backend, and with + /// negotiation enabled up only once it has completed. + fn link(&self) -> bool { + !self.power_down() && self.backend_up && (!self.autoneg() || self.nego_done) + } + + fn start_negotiation(&mut self, now: Instant) { + self.nego_done = false; + self.partner = 0; + self.page_received = false; + self.deadline = None; + if self.autoneg() && self.backend_up && !self.power_down() { + // The synthetic peer advertises 0x01e1: IEEE 802.3, no fault. + let common = self.anar & 0x01e0; + if self.anar & 0x1f == 1 && common != 0 && self.anar & 0x2000 == 0 { + self.deadline = Some(now + self.delay); + } + } + } + + /// When a negotiation in progress completes, if one is. + pub fn deadline(&self) -> Option { + self.deadline + } + + /// Advance the negotiation deadline and note link transitions. Reads never + /// start or postpone negotiation; this only observes the clock. + pub fn sync(&mut self, now: Instant) { + if let Some(d) = self.deadline { + if now >= d && !self.power_down() && self.backend_up { + self.deadline = None; + self.nego_done = true; + self.partner = 0x01e1; + self.page_received = true; + } + } + let link = self.link(); + if self.last_link && !link { + self.link_low_latch = true; + } + self.last_link = link; + } + + /// The packet backend's cable state. + pub fn set_backend_link(&mut self, up: bool, now: Instant) { + self.sync(now); + if self.backend_up == up { + return; + } + self.backend_up = up; + if up { + // Recovery starts a fresh negotiation. + self.start_negotiation(now); + } else { + self.deadline = None; + self.nego_done = false; + } + self.sync(now); + } + + /// BMCR loopback turns frames round inside the part. Isolate and + /// power-down take precedence. + pub fn in_loopback(&self) -> bool { + self.bmcr & 0x4000 != 0 && !self.power_down() && !self.isolated() + } + + /// Frames may cross the MII at all. + pub fn data_path(&self) -> bool { + !self.power_down() && !self.isolated() + } + + /// Frames may go to or come from the packet backend. + pub fn wire_enabled(&self) -> bool { + self.data_path() && self.link() + } + + fn resolved(&self) -> (bool, bool) { + // (100 Mb/s, full duplex) + if self.autoneg() { + let common = self.anar & self.partner & 0x01e0; + if common & 0x100 != 0 { + (true, true) + } else if common & 0x80 != 0 { + (true, false) + } else if common & 0x40 != 0 { + (false, true) + } else { + (false, false) + } + } else { + (self.bmcr & 0x2000 != 0, self.bmcr & 0x0100 != 0) + } + } + + pub fn read(&mut self, r: u8, now: Instant) -> u16 { + self.sync(now); + match r { + 0 => self.bmcr, + 1 => { + let mut v = 0x0009 | self.bmsr_caps; + if self.autoneg() && self.nego_done { + v |= 0x0020; + } + if self.link() && !self.link_low_latch { + v |= 0x0004; + } + self.link_low_latch = false; + v + } + 2 => self.id1, + 3 => self.id2, + 4 => self.anar, + 5 => self.partner, + 6 => { + let v = (self.exp & 0xffe0) + | if self.nego_done { 1 } else { 0 } + | if self.page_received { 2 } else { 0 }; + self.page_received = false; + v + } + 0x10 => { + // Bit 4 is a fixed reserved read choice (stock firmware + // expects it set); bit 15 reads the override state. + (self.ext_ctl & 0x002d) + | (u16::from(self.addr & 0x1f) << 6) + | 0x0010 + | if self.override_armed { 0x8000 } else { 0 } + } + 0x11 => { + let link = self.link(); + let (speed, full) = self.resolved(); + let mut v = 0; + if link { + v |= 1; + if speed { + v |= 0x8000; + } + if full { + v |= 0x4000; + } + } + if self.nego_done { + v |= 0x0010 | (7 << 11); + } + v + } + 0x12 => (self.tenbt & 0x002f) | 0x0010, + // Node mode strap in bit 14, software bit 0. + 0x13 => (self.ext2 & 0x0011) | 0x4000, + _ => { + self.unimplemented_accesses += 1; + 0 + } + } + } + + pub fn write(&mut self, r: u8, val: u16, now: Instant) { + self.sync(now); + let r = usize::from(r); + let armed = self.override_armed; + match r { + 0 => { + if val & 0x8000 != 0 { + self.reset(now); + self.sync(now); + return; + } + let old = self.bmcr; + // Reset and restart are self-clearing. + self.bmcr = val & 0x7d80; + let restart = val & 0x0200 != 0; + let pd_now = self.power_down(); + if pd_now && old & 0x0800 == 0 { + // Entering power-down clears the latching status. + self.link_low_latch = false; + self.page_received = false; + self.deadline = None; + } else if !pd_now && old & 0x0800 != 0 { + // Release: enabled negotiation starts afresh. + self.start_negotiation(now); + } else if self.autoneg() && (restart || old & 0x1000 == 0) { + self.start_negotiation(now); + } else if !self.autoneg() { + self.nego_done = false; + self.deadline = None; + } + if armed { + self.override_armed = false; + } + } + 1 | 2 | 3 | 4 | 6 | 0x10 | 0x12 | 0x13 => { + let (ord, ovr) = phy_masks(r); + let mask = ord | if armed { ovr } else { 0 }; + let cell = match r { + 1 => &mut self.bmsr_caps, + 2 => &mut self.id1, + 3 => &mut self.id2, + 4 => &mut self.anar, + 6 => &mut self.exp, + 0x10 => &mut self.ext_ctl, + 0x12 => &mut self.tenbt, + _ => &mut self.ext2, + }; + let before = *cell; + *cell = (before & !mask) | (val & mask); + // The status caps live in bits 14..7 only. + if r == 1 { + self.bmsr_caps &= 0x7f80; + } + if r == 4 && self.anar != before && self.autoneg() { + self.start_negotiation(now); + } + if armed { + self.override_armed = false; + } else if r == 0x10 && val & 0x8000 != 0 { + self.override_armed = true; + } + } + _ => { + // Partner ability, Quick Poll and unimplemented registers + // ignore writes. The part is addressed, so a write still + // consumes an armed override. + if r != 5 && r != 0x11 { + self.unimplemented_accesses += 1; + } + if armed { + self.override_armed = false; + } + } + } + self.sync(now); + } +} + +// The transceiver's registers and auto-negotiation progress for a +// snapshot. Its one deadline is saved as the time it had left. +impl Ics1890 { + pub(crate) fn save_state(&self, now: Instant) -> toml::Value { + StateTable::new() + .u8("addr", self.addr) + .u16("bmcr", self.bmcr) + .u16("bmsr_caps", self.bmsr_caps) + .u16("id1", self.id1) + .u16("id2", self.id2) + .u16("anar", self.anar) + .u16("partner", self.partner) + .u16("exp", self.exp) + .u16("ext_ctl", self.ext_ctl) + .u16("tenbt", self.tenbt) + .u16("ext2", self.ext2) + .bool("override_armed", self.override_armed) + .bool("page_received", self.page_received) + .bool("link_low_latch", self.link_low_latch) + .bool("last_link", self.last_link) + .bool("backend_up", self.backend_up) + .bool("nego_done", self.nego_done) + // Nanoseconds left plus one; zero for none. + .u64("deadline", self.deadline.map_or(0, |d| d.saturating_duration_since(now).as_nanos() as u64 + 1)) + .u64("delay", self.delay.as_nanos() as u64) + .u32("unimplemented_accesses", self.unimplemented_accesses) + .build() + } + + pub(crate) fn load_state(&mut self, v: &toml::Value, now: Instant) -> Result<(), String> { + let deadline = field_u64(v, "deadline")?; + *self = Self { + addr: field_u8(v, "addr")?, + bmcr: field_u16(v, "bmcr")?, + bmsr_caps: field_u16(v, "bmsr_caps")?, + id1: field_u16(v, "id1")?, + id2: field_u16(v, "id2")?, + anar: field_u16(v, "anar")?, + partner: field_u16(v, "partner")?, + exp: field_u16(v, "exp")?, + ext_ctl: field_u16(v, "ext_ctl")?, + tenbt: field_u16(v, "tenbt")?, + ext2: field_u16(v, "ext2")?, + override_armed: field_bool(v, "override_armed")?, + page_received: field_bool(v, "page_received")?, + link_low_latch: field_bool(v, "link_low_latch")?, + last_link: field_bool(v, "last_link")?, + backend_up: field_bool(v, "backend_up")?, + nego_done: field_bool(v, "nego_done")?, + deadline: (deadline != 0).then(|| now + Duration::from_nanos(deadline - 1)), + delay: Duration::from_nanos(field_u64(v, "delay")?), + unimplemented_accesses: field_u32(v, "unimplemented_accesses")?, + }; + Ok(()) + } +} + +impl MiiPhy for Ics1890 { + fn address(&self) -> u8 { Ics1890::address(self) } + fn deadline(&self) -> Option { Ics1890::deadline(self) } + fn wire_enabled(&self) -> bool { Ics1890::wire_enabled(self) } + fn in_loopback(&self) -> bool { Ics1890::in_loopback(self) } + fn sync(&mut self, now: Instant) { Ics1890::sync(self, now) } + fn read(&mut self, register: u8, now: Instant) -> u16 { Ics1890::read(self, register, now) } + fn write(&mut self, register: u8, value: u16, now: Instant) { Ics1890::write(self, register, value, now) } + fn set_backend_link(&mut self, up: bool, now: Instant) { Ics1890::set_backend_link(self, up, now) } + fn power_on(&mut self) { Ics1890::reset(self, Instant::now()) } + fn save_state(&self, now: Instant) -> toml::Value { Ics1890::save_state(self, now) } + fn load_state(&mut self, v: &toml::Value, now: Instant) -> Result<(), String> { Ics1890::load_state(self, v, now) } +} diff --git a/src/dev/ioc3.rs b/src/dev/ioc3.rs index f82bbbde5..955377372 100644 --- a/src/dev/ioc3.rs +++ b/src/dev/ioc3.rs @@ -1,10 +1,12 @@ -//! SGI IOC3 base I/O controller, as the Octane (IP30) fits it. +//! SGI IOC3 base I/O controller, with Octane and Fuel board populations. //! //! One PCI function with one 1 MiB memory BAR. Behind it: two 16550 serial -//! ports with a ring-DMA engine each, a PS/2 keyboard and mouse controller, +//! ports with a ring-DMA engine each (the engine is shared with the IOC4, in +//! `ioc_serial_dma.rs`), a PS/2 keyboard and mouse controller, //! a parallel port, a MicroLAN (one-wire) master, the GPIO pins, a DS1685 -//! clock on the byte bus, the Ethernet buffer RAM and, in the register range -//! at 0xf0, the Ethernet MAC (which lives in `ioc3_eth.rs` behind +//! clock on the Octane byte bus or Fuel's flat system Timekeeper, the +//! Ethernet buffer RAM and, in the register range at 0xf0, +//! the Ethernet MAC (which lives in `ioc3_eth.rs` behind //! [`Ioc3EthPort`]). //! //! Behaviour follows the public Linux and OpenBSD drivers and measured guest @@ -31,8 +33,8 @@ //! # What is approximated here //! //! * Nothing is timed to a baud rate: serial transmit completes inside the -//! write and a loopback character arrives at once (the part takes a -//! character time). +//! write. A character looped back through the ring engine does take its +//! character time to reach the receiver. //! * Serial ring DMA, the receive timer, transmit delays and the interrupt //! output are evaluated lazily on every register access, and an `hptimer` //! one-shot per deadline runs the same catch-up while the guest is idle. @@ -41,6 +43,10 @@ //! has no alarm or periodic interrupts. Its RAM, including the 128-byte //! extended RAM, persists in a 512-byte file. //! +//! Names follow QEMU's: `sgi-ioc3.c` has the same registers under the same +//! names, its `sgi_ioc3_sdma_*` callbacks are the chip's `SerialDmaHost`, and +//! the ring engine is `ioc_serial_dma.rs` (QEMU's `sgi-ioc-serial-dma.c`). +//! //! Ported from hw/misc/sgi-ioc3.c and its serial, parallel and keyboard siblings //! in jamesbraid/qemu-sgi-origami. @@ -54,13 +60,18 @@ use std::time::{Duration, Instant}; use crate::dev::ds1687::MaceRtc; use crate::dev::ioc3_port::Ioc3EthPort; +use crate::dev::ioc3_kbc::{Ioc3Kbc, KbcLine, KbcOwner}; use crate::dev::ps2::Ps2Controller; use crate::dev::w1; use crate::dev::w1::sgi::{self, BoardRecord}; +use crate::dev::deadline_timers::DeadlineTimers; +use crate::dev::ioc_serial_dma::{off as sdma_off, SerialDmaUart, SerialDmaHost, SerialDmaParams, SerialDma, SerialDmaSource}; +#[cfg(test)] +use crate::dev::ioc_serial_dma::{ctl as sdma_ctl, ARM_RX_TIMER, HELD_SHIFT, TIMER_TICK_NS, TX_DELAY, TX_INTERRUPT}; use crate::dev::ns16550::Ns16550; use crate::dev::serial_host::SerialBackend; use crate::irq::{InterruptController, IrqLine}; -use crate::pci::{Bar, PciBusDevice, PciConfig, PciDma, PciDmaStatus, PCI_INTA}; +use crate::pci::{Bar, PciBusDevice, PciConfig, PciConfigPolicy, PciDma, PciDmaStatus, PCI_INTA}; use crate::hptimer::{TimerId, TimerManager, TimerReturn}; use crate::traits::{BusDevice, Device, MachineEvent, Resettable, Saveable}; use crate::snapshot::{ @@ -72,6 +83,8 @@ pub const DEVICE_ID: u16 = 0x0003; pub const REVISION: u8 = 0x01; /// PCI subsystem id of the Octane's system board. pub const SUBSYS_IP30_SYSBOARD: u16 = 0xc304; +/// Fuel system-board population. +pub const SUBSYS_IP34_SYSBOARD: u16 = 0xc30a; pub const BAR_SIZE: u32 = 1 << 20; /// The configuration dwords the chip decodes; the rest of the 16 repeat them. const CONFIG_DECODED_DWORDS: usize = 8; @@ -113,6 +126,7 @@ pub mod reg { pub const PAR_ECP_BASE: u32 = 0x20150; pub const SSRAM: u32 = 0x40000; pub const SSRAM_SIZE: u32 = 0x40000; + pub const BYTEBUS_DEV0: u32 = 0x80000; pub const BYTEBUS_DEV1: u32 = 0xa0000; pub const BYTEBUS_DEV2: u32 = 0xc0000; pub const BYTEBUS_SIZE: u32 = 0x20000; @@ -162,20 +176,8 @@ mod uart_reg { pub const MSR: u32 = 6; pub const SCR: u32 = 7; } -const LSR_DR: u8 = 0x01; -const LSR_OE: u8 = 0x02; -const LSR_PE: u8 = 0x04; -const LSR_FE: u8 = 0x08; -const LSR_BI: u8 = 0x10; -const MCR_DTR: u8 = 0x01; -const MCR_RTS: u8 = 0x02; -const MCR_OUT1: u8 = 0x04; -const MCR_OUT2: u8 = 0x08; -const MCR_LOOP: u8 = 0x10; const MCR_AFE: u8 = 0x20; -const MSR_DCTS: u8 = 0x01; const MSR_CTS: u8 = 0x10; -const MSR_DCD: u8 = 0x80; const LSR_TEST_MASK: u8 = 0x7f; const MSR_TEST_MASK: u8 = 0x0f; /// The reference clock of the UART crystal: the pre-divider divides it, and @@ -222,23 +224,21 @@ impl Uart { self.core.irq() } - /// Move whatever the host end has typed into the receive FIFO and let - /// the receive timeout run out. Returns whether anything changed. + /// Move whatever the host end has typed into the receive FIFO, retry a + /// byte the host end refused and let the receive timeout run out. + /// Returns whether anything changed. fn pump(&mut self) -> bool { + let now = Instant::now(); let moved = self.core.pump(); - self.core.check_timeout(Instant::now()) || moved - } - - /// The 16550 proper: what the ring engine sees. - fn read_core(&mut self, r: u32) -> u8 { - self.core.read(r) + let sent = self.core.poll_tx(now); + self.core.check_timeout(now) || moved || sent } /// A guest's byte-wide read, with the chip's additions: the auto-flow /// bit in modem control, and a factory-test write to line or modem /// status showing in the read that follows it and no later one. fn read(&mut self, r: u32) -> u8 { - let mut v = self.read_core(r); + let mut v = self.core.read(r); match r { uart_reg::MCR => v |= self.mcr_afe, uart_reg::LSR => { @@ -271,10 +271,39 @@ impl Uart { } self.core.write(r, v); } +} + +impl SerialDmaUart for Uart { + fn loopback(&self) -> bool { + self.core.loopback() + } + + fn tx_ready(&self) -> bool { + self.core.tx_ready() + } - /// Modem status as the ring engine reads it with flow control looped back. - fn flow_cts(&self, control: u32) -> bool { - self.loopback() && control & sdma_ctl::FLOW_CONTROL != 0 + fn char_time(&self) -> Duration { + self.core.char_time() + } + + fn read_core(&mut self, r: u32) -> u8 { + self.core.read(r) + } + + fn write(&mut self, r: u32, v: u8) { + Uart::write(self, r, v); + } + + fn lsr(&self) -> u8 { + self.core.lsr() + } + + fn mcr(&self) -> u8 { + self.core.mcr() + } + + fn msr(&self) -> u8 { + self.core.msr() } } @@ -282,48 +311,10 @@ impl Uart { // Serial ring DMA // --------------------------------------------------------------------------- -/// Port DMA register offsets (bytes, from the port's first register). -mod sdma_off { - pub const CONTROL: u32 = 0x00; - pub const TX_PRODUCE: u32 = 0x04; - pub const TX_CONSUME: u32 = 0x08; - pub const RX_PRODUCE: u32 = 0x0c; - pub const RX_CONSUME: u32 = 0x10; - pub const RX_TIMER: u32 = 0x14; - pub const SHADOW: u32 = 0x18; - pub const REGS: u32 = 7; -} -mod sdma_ctl { - pub const HIGH_WATER: u32 = 0x1ff; - pub const TX_TIMER_RUN: u32 = 1 << 16; - pub const FLOW_CONTROL: u32 = 1 << 17; - pub const RX_TO_MEMORY: u32 = 1 << 27; - pub const ENABLE: u32 = 1 << 28; - pub const PAUSE: u32 = 1 << 29; - pub const PAUSED: u32 = 1 << 30; - pub const RESET: u32 = 1 << 31; - pub const UNSTORED: u32 = RESET | RX_TO_MEMORY | PAUSED | TX_TIMER_RUN; -} -const ENTRY: usize = 8; -const ENTRY_CHARS: usize = 4; -const ARM_RX_TIMER: u32 = 1 << 31; -const HELD_SHIFT: u32 = 24; -const TIMER_RELOAD: u32 = 0xfff; -const TIMER_COUNT_SHIFT: u32 = 16; -const TIMER_TICK_NS: u64 = 62_500; // 16 kHz -const TX_TYPE: u8 = 0xc0; -const TX_CHARACTER: u8 = 0x40; -const TX_MODEM: u8 = 0x80; -const TX_DELAY: u8 = 0xc0; -const TX_INTERRUPT: u8 = 0x20; -const RX_OVERRUN: u8 = 0x01; -const RX_PARITY: u8 = 0x02; -const RX_FRAMING: u8 = 0x04; -const RX_BREAK: u8 = 0x08; -const RX_CTS: u8 = 0x10; -const RX_CD: u8 = 0x20; -const RX_MODEM_OK: u8 = 0x40; -const RX_DATA_OK: u8 = 0x80; +/// What IOC3's engine does that IOC4's does not: it reports the packer's held +/// count, and a failed transfer does not stop the port. +const IOC3_SDMA_PARAMS: SerialDmaParams = SerialDmaParams { held_counts: true, halt_on_dma_error: false }; + /// Chip control: where each engine finds its 16550 (byte offset / 8). const CHIP_CTRL_MASK: u32 = 0x0007_ffff; const CHIP_CTRL_PORT_A_SHIFT: u32 = 1; @@ -331,37 +322,6 @@ const CHIP_CTRL_PORT_B_SHIFT: u32 = 8; const CHIP_CTRL_PORT_FIELD: u32 = 0x7f; const CHIP_CTRL_ARB_IDLE: u32 = 1 << 22; -#[derive(Default)] -struct SdmaPort { - reg: [u32; 7], - packer: [u8; ENTRY], - packed: usize, - tx_entry: [u8; ENTRY], - tx_fetched: bool, - tx_slot: usize, - armed: bool, - drain_pending: bool, - busy: bool, - paused: bool, - rx_deadline: Option, - rx_idle_count: u32, - /// Time a frozen timer had left when a pause began. - rx_frozen: Option, - tx_delay_deadline: Option, - tx_delay_frozen: Option, - tx_delay_irq: bool, - /// An entry waits that the receive ring had no room for; drives - /// receive-ring-full while ring DMA is on. - post_refused: bool, -} - -#[derive(Default)] -struct Sdma { - base_hi: u32, - base_lo: u32, - port: [SdmaPort; 2], -} - // --------------------------------------------------------------------------- // DMA as the chip masters the bus // --------------------------------------------------------------------------- @@ -423,38 +383,6 @@ impl PciDma for Ioc3Dma { } } -// --------------------------------------------------------------------------- -// PS/2 controller -// --------------------------------------------------------------------------- - -const KBC_CONTROL: u32 = 0x9c; -const KBC_KBD_RECEIVE: u32 = 0xa0; -const KBC_AUX_RECEIVE: u32 = 0xa4; -const KBC_KBD_TRANSMIT: u32 = 0xa8; -const KBC_AUX_TRANSMIT: u32 = 0xac; -const KBC_RX_SLOTS: usize = 3; -/// Bits 16 and 17 (timeouts) read clear: no stalled line is simulated. -const KBC_CONTROL_MASK: u32 = 0x00fc_ffff; -const KBC_K_CLAMP_1: u32 = 0x0010_0000; -const KBC_M_CLAMP_1: u32 = 0x0020_0000; -const KBC_K_CLAMP_3: u32 = 0x0040_0000; -const KBC_M_CLAMP_3: u32 = 0x0080_0000; -const KBC_RX_SOURCE_AUX: u32 = 1 << 27; -const KBC_RX_OVERFLOW: u32 = 1 << 28; - -#[derive(Default, Clone, Copy)] -struct KbcPort { - data: [u8; KBC_RX_SLOTS], - count: usize, - overflow: bool, -} - -#[derive(Default)] -struct Kbc { - control: u32, - port: [KbcPort; 2], -} - // --------------------------------------------------------------------------- // Parallel port (TL16PIR552 population, 16-byte test FIFO) // --------------------------------------------------------------------------- @@ -734,6 +662,10 @@ fn int_out_valid(v: u32) -> bool { #[derive(Clone, Debug)] pub struct Ioc3Config { pub subsystem_id: u16, + /// Function-specific PCI header write behavior. + pub pci_policy: PciConfigPolicy<'static>, + /// Reset the PCI header along with the chip on power-on. + pub reset_pci_on_power_on: bool, /// The board's station address, which the one-wire part holds. pub mac: [u8; 6], pub serial: [bool; 2], @@ -763,6 +695,30 @@ impl BoardRecordText { } impl Ioc3Config { + /// Fuel's IOC3 fits the keyboard and mouse controller but no indexed Dallas clock. + pub fn fuel(mac: [u8; 6]) -> Self { + Self { + subsystem_id: SUBSYS_IP34_SYSBOARD, + pci_policy: PciConfigPolicy { + status: 0x0280, + status_w1c_mask: 0, + command_mask: 0xffff, + latency_mask: 0xff, + cache_line_mask: 0, + ..Default::default() + }, + reset_pci_on_power_on: true, + mac, + serial: [true, true], + parallel: true, + keyboard_mouse: true, + rtc: false, + gpio_input_mask: 0, + gpio_input_values: 0, + records: Vec::new(), + } + } + /// The Octane's IOC3 (`octane2` selects the Octane2's board records). /// The fictitious barcodes and ROM serials are the /// marked-synthetic ones (barcode `EMU...`). @@ -774,6 +730,8 @@ impl Ioc3Config { }; Self { subsystem_id: SUBSYS_IP30_SYSBOARD, + pci_policy: PciConfigPolicy::default(), + reset_pci_on_power_on: false, mac, serial: [true, true], parallel: true, @@ -814,8 +772,8 @@ struct St { pin_control: u32, pin_data: u32, uart: [Uart; 2], - sdma: Sdma, - kbc: Kbc, + sdma: SerialDma, + kbc: Ioc3Kbc, par: Parallel, rtc: Rtc, ssram: Vec, @@ -854,6 +812,7 @@ struct Ctx<'a> { impl St { fn new(cfg: Ioc3Config) -> St { + let kbc = Ioc3Kbc::new(cfg.keyboard_mouse); St { cfg, int_status: 0, @@ -864,8 +823,8 @@ impl St { pin_control: 0, pin_data: 0, uart: [Uart::new(), Uart::new()], - sdma: Sdma::default(), - kbc: Kbc::default(), + sdma: SerialDma::new(2, IOC3_SDMA_PARAMS), + kbc, par: Parallel::default(), rtc: Rtc::new(), ssram: vec![0; (reg::SSRAM_SIZE / 4) as usize], @@ -879,7 +838,7 @@ impl St { fn reset(&mut self) { self.int_status = 0; self.int_enable = 0; - self.sdma = Sdma::default(); + self.sdma.reset(); for u in &mut self.uart { u.reset(); } @@ -888,7 +847,7 @@ impl St { self.pin_data = 0; self.chip_control = 0; self.int_out = IntOut::default(); - self.kbc = Kbc::default(); + self.kbc.reset(); let backend = self.par.backend.take(); self.par = Parallel::default(); self.par.backend = backend; @@ -907,10 +866,10 @@ impl St { /// bit can reach the line, and the NVRAM file's debounce. fn deadlines(&self) -> Deadlines { let mut d = [None; KINDS]; - for (p, port) in self.sdma.port.iter().enumerate() { - d[K_RX + p] = port.rx_deadline; - // A paused port keeps its transmit delay frozen. - d[K_TX_DELAY + p] = if port.paused { None } else { port.tx_delay_deadline }; + for p in 0..2 { + d[K_RX + p] = self.sdma.rx_deadline(p); + d[K_TX_DELAY + p] = self.sdma.tx_delay_deadline(p); + d[K_LOOP + p] = self.sdma.loop_deadline(p); } if self.int_enable & int::RT_OUTPUT_PULSE != 0 { d[K_INT_OUT] = self.int_out.next; @@ -930,14 +889,12 @@ impl St { self.set_bit(int::serial(p, int::PASSTHROUGH), lvl); // On every level, not on assertion alone: the ring console receives // with the 16550's own interrupts disabled. - if !self.sdma.port[p].busy { - self.sdma_run(p); - } + self.sdma_run(p); } fn uart_read(&mut self, p: usize, r: u32) -> u8 { let mut v = self.uart[p].read(r); - if r == uart_reg::MSR && self.uart[p].flow_cts(self.sdma.port[p].reg[0]) { + if r == uart_reg::MSR && self.sdma.flow_cts(p, self.uart[p].loopback()) { v |= MSR_CTS; } self.uart_changed(p); @@ -959,455 +916,100 @@ impl St { self.dma.as_ref().is_some_and(|d| d.dma_write(addr, data).is_ok()) } - fn sdma_ctl(&self, p: usize) -> u32 { - self.sdma.port[p].reg[0] - } - - fn sdma_ring_size(&self) -> u32 { - if self.sdma.base_lo & 1 != 0 { 4096 } else { 1024 } - } - - fn sdma_mask(&self) -> u32 { - (self.sdma_ring_size() - 1) & !(ENTRY as u32 - 1) - } - - fn sdma_ring(&self, p: usize, dir: u32) -> u64 { - let base = ((self.sdma.base_hi as u64) << 32) | (self.sdma.base_lo & 0xffff_f000) as u64; - base.saturating_add(((p as u64 % 2) * 2 + dir as u64) * self.sdma_ring_size() as u64) - } - - fn sdma_raise(&mut self, p: usize, src: u32) { - self.int_status |= int::serial(p, src); - } - - fn sdma_level(&mut self, p: usize, src: u32, level: bool) { - self.set_bit(int::serial(p, src), level); - } - - /// Chip control places each port's 16550 for its engine; aimed anywhere - /// else, the engine does not reach the port. - fn sdma_reachable(&self, p: usize) -> bool { - if !self.cfg.serial[p] { - return false; - } - let (shift, base) = if p == 1 { - (CHIP_CTRL_PORT_B_SHIFT, reg::SERIAL_B) - } else { - (CHIP_CTRL_PORT_A_SHIFT, reg::SERIAL_A) + /// The engine and the chip around it, borrowed apart so the engine can + /// drive the 16550s, the interrupt status and the bus master. + fn sdma_engine_split(&mut self) -> (&mut SerialDma, Ioc3SerialDmaHost<'_>) { + let host = Ioc3SerialDmaHost { + uart: &mut self.uart, + int_status: &mut self.int_status, + chip_control: self.chip_control, + serial: self.cfg.serial, + dma: self.dma.as_deref(), }; - (self.chip_control >> shift) & CHIP_CTRL_PORT_FIELD == (base & 0xfff) / 8 - } - - fn sdma_running(&self, p: usize) -> bool { - let c = self.sdma_ctl(p); - c & sdma_ctl::ENABLE != 0 && c & sdma_ctl::PAUSE == 0 - } - - /// Transmit-ring-empty is a level: ring DMA on, nothing left in the ring - /// or held by the engine. - fn sdma_update_tx_empty(&mut self, p: usize) { - let mask = self.sdma_mask(); - let port = &self.sdma.port[p]; - let empty = self.sdma_ctl(p) & sdma_ctl::ENABLE != 0 - && (port.reg[1] ^ port.reg[2]) & mask == 0 - && !port.tx_fetched - && port.tx_delay_deadline.is_none(); - self.sdma_level(p, int::TX_RING_EMPTY, empty); - } - - /// The receive high-water mark counts ring entries; a level. So is - /// receive-ring-full: set while ring DMA is on and an entry waits that - /// the ring had no room for, so an acknowledgement does not bring it back - /// with every refused retry, and it falls once the consumer makes room, - /// posted yet or not. A ring filled exactly, with nothing waiting, does - /// not raise it. A model choice; what the part does is not known. - fn sdma_update_high_water(&mut self, p: usize) { - let mask = self.sdma_mask(); - let port = &self.sdma.port[p]; - let hw = self.sdma_ctl(p) & sdma_ctl::HIGH_WATER; - let waiting = port.reg[3].wrapping_sub(port.reg[4]) & mask; - let level = hw != 0 && waiting / ENTRY as u32 >= hw; - if (port.reg[3] + ENTRY as u32) & mask != port.reg[4] & mask { - self.sdma.port[p].post_refused = false; - } - let full = self.sdma_ctl(p) & sdma_ctl::ENABLE != 0 && self.sdma.port[p].post_refused; - self.sdma_level(p, int::RX_HIGH_WATER, level); - self.sdma_level(p, int::RX_RING_FULL, full); - } - - /// Post the packer as one entry. Fails, keeping the characters, when the - /// ring is full or the write is refused. - fn sdma_post(&mut self, p: usize) -> bool { - let mask = self.sdma_mask(); - let ring = self.sdma_ring(p, 1); - let (produce, consume, packer) = { - let port = &self.sdma.port[p]; - (port.reg[3], port.reg[4], port.packer) - }; - let next = (produce + ENTRY as u32) & mask; - if next == consume & mask { - self.sdma.port[p].post_refused = true; - return false; - } - if !self.dma_write(ring.saturating_add((produce & mask) as u64), &packer) { - self.sdma_raise(p, int::DMA_ERROR); - return false; - } - let port = &mut self.sdma.port[p]; - port.reg[3] = next; - port.post_refused = false; - port.packed = 0; - port.packer = [0; ENTRY]; - true - } - - fn sdma_rx_timer_start(&mut self, p: usize, since: Instant) { - let port = &mut self.sdma.port[p]; - let reload = port.reg[5] & TIMER_RELOAD; - port.rx_idle_count = 0; - let dur = Duration::from_nanos(reload as u64 * TIMER_TICK_NS); - if port.paused { - // Started while paused, it waits frozen at its full count. - port.rx_frozen = Some(dur); - } else { - port.rx_deadline = Some(since + dur); - } + (&mut self.sdma, host) } - fn sdma_rx_timer_expired(&mut self, p: usize) { - self.sdma.port[p].rx_idle_count = 0; - self.sdma.port[p].rx_deadline = None; - if self.sdma.port[p].packed != 0 { - self.sdma_post(p); - } - // Raised at every expiry, even over an empty ring; the arm is spent. - if self.sdma.port[p].armed { - self.sdma_raise(p, int::RX_TIMER); - } - self.sdma.port[p].armed = false; - self.sdma_update_high_water(p); + fn sdma_run(&mut self, p: usize) { + let (sdma, mut host) = self.sdma_engine_split(); + sdma.run(p, &mut host); } /// Expiries already due happen before an access. fn sdma_catch_up(&mut self, now: Instant) { - for p in 0..2 { - let delay = self.sdma.port[p].tx_delay_deadline; - if let Some(d) = delay { - if !self.sdma.port[p].paused && d <= now { - self.sdma_tx_delay_ended(p, d); - } - } - if let Some(d) = self.sdma.port[p].rx_deadline { - if d <= now { - self.sdma_rx_timer_expired(p); - } - } - } + let (sdma, mut host) = self.sdma_engine_split(); + sdma.catch_up(now, &mut host); } - fn sdma_tx_delay_ended(&mut self, p: usize, when: Instant) { - let irq = { - let port = &mut self.sdma.port[p]; - port.tx_delay_deadline = None; - std::mem::take(&mut port.tx_delay_irq) - }; - if irq { - self.sdma_raise(p, int::TX_EXPLICIT); - } - self.sdma_run_from(p, when); + fn sdma_port_read(&mut self, p: usize, off: u32) -> u32 { + let (sdma, mut host) = self.sdma_engine_split(); + sdma.port_read(p, off, &mut host) } - fn sdma_receive(&mut self, p: usize, since: Instant) { - while self.sdma_running(p) { - if self.sdma.port[p].packed == ENTRY_CHARS && !self.sdma_post(p) { - break; - } - // Line status before the character: reading it clears the latches. - let lsr = self.uart[p].read_core(uart_reg::LSR); - if lsr & LSR_DR == 0 { - break; - } - let data = self.uart[p].read_core(uart_reg::RBR_THR); - let mut msr = self.uart[p].read_core(uart_reg::MSR); - if self.uart[p].flow_cts(self.sdma_ctl(p)) { - msr |= MSR_CTS; - } - let mut status = RX_DATA_OK | RX_MODEM_OK; - for (bit, flag) in [(LSR_OE, RX_OVERRUN), (LSR_PE, RX_PARITY), (LSR_FE, RX_FRAMING), (LSR_BI, RX_BREAK)] { - if lsr & bit != 0 { - status |= flag; - } - } - if msr & MSR_CTS != 0 { - status |= RX_CTS; - } - if msr & MSR_DCD != 0 { - status |= RX_CD; - } - let start_timer = { - let port = &mut self.sdma.port[p]; - port.packer[port.packed] = data; - port.packer[ENTRY_CHARS + port.packed] = status; - port.packed += 1; - port.armed && port.rx_deadline.is_none() && port.rx_frozen.is_none() - }; - // Started by the first character after the arm; later ones leave - // a running timer alone. - if start_timer { - self.sdma_rx_timer_start(p, since); - } - // A fourth character completes the entry; fewer wait for the - // timer or a drain. - if self.sdma.port[p].packed == ENTRY_CHARS { - self.sdma_post(p); - } - } - self.uart_pass_bit(p); + fn sdma_port_write(&mut self, p: usize, off: u32, val: u32) { + let (sdma, mut host) = self.sdma_engine_split(); + sdma.port_write(p, off, val, &mut host); } +} - /// Keep the pass-through bit on the 16550's level without kicking the - /// engine (it is already running). - fn uart_pass_bit(&mut self, p: usize) { - let lvl = self.uart[p].irq(); - self.set_bit(int::serial(p, int::PASSTHROUGH), lvl); - } +/// IOC3 as the ring engine sees it: two 16550s, the SuperIO status word, and +/// the bus master. +struct Ioc3SerialDmaHost<'a> { + uart: &'a mut [Uart; 2], + int_status: &'a mut u32, + chip_control: u32, + serial: [bool; 2], + dma: Option<&'a Ioc3Dma>, +} - /// Each control byte types its data byte; the empty type ends the entry, - /// wherever it sits. - fn sdma_transmit(&mut self, p: usize, since: Instant) { - let mask = self.sdma_mask(); - let ring = self.sdma_ring(p, 0); - if self.sdma.port[p].tx_delay_deadline.is_some() { - return; - } - loop { - let (produce, consume) = { - let port = &self.sdma.port[p]; - (port.reg[1], port.reg[2]) - }; - if !(self.sdma_running(p) && consume & mask != produce & mask) { - break; - } - if !self.sdma.port[p].tx_fetched { - let mut entry = [0u8; ENTRY]; - if !self.dma_read(ring.saturating_add((consume & mask) as u64), &mut entry) { - self.sdma_raise(p, int::DMA_ERROR); - break; - } - let port = &mut self.sdma.port[p]; - port.tx_entry = entry; - port.tx_fetched = true; - } - let entry = self.sdma.port[p].tx_entry; - let mut slot = self.sdma.port[p].tx_slot; - while slot < ENTRY_CHARS { - let control = entry[ENTRY_CHARS + slot]; - let ty = control & TX_TYPE; - if ty == 0 { - break; - } - match ty { - TX_CHARACTER => { - let looped = self.uart[p].loopback(); - // What has already arrived goes ahead of this one. - if looped { - self.sdma_receive(p, since); - } - self.uart[p].write(uart_reg::RBR_THR, entry[slot]); - // Approximation: a looped-back character arrives at - // once, where the part takes a character time. - self.sdma_receive(p, since); - } - TX_MODEM => self.uart[p].write(uart_reg::MCR, entry[slot]), - TX_DELAY => { - // The data byte is milliseconds of quiet. - if entry[slot] != 0 { - let port = &mut self.sdma.port[p]; - port.tx_slot = slot + 1; - port.tx_delay_deadline = Some(since + Duration::from_millis(entry[slot] as u64)); - port.tx_delay_irq = control & TX_INTERRUPT != 0; - return; - } - } - _ => unreachable!(), - } - slot += 1; - self.sdma.port[p].tx_slot = slot; - if control & TX_INTERRUPT != 0 { - self.sdma_raise(p, int::TX_EXPLICIT); - } - } - let port = &mut self.sdma.port[p]; - port.tx_slot = 0; - port.tx_fetched = false; - port.reg[2] = (port.reg[2] + ENTRY as u32) & mask; +impl Ioc3SerialDmaHost<'_> { + fn set_bit(&mut self, mask: u32, level: bool) { + if level { + *self.int_status |= mask; + } else { + *self.int_status &= !mask; } - self.uart_pass_bit(p); } +} - /// The drain posts what the packer holds and reads back set until it has; - /// it waits while ring DMA is off or paused and while the ring is full. - fn sdma_drain(&mut self, p: usize) { - if !self.sdma.port[p].drain_pending || !self.sdma_running(p) { - return; - } - if self.sdma.port[p].packed != 0 && !self.sdma_post(p) { - return; - } - self.sdma.port[p].drain_pending = false; +impl SerialDmaHost for Ioc3SerialDmaHost<'_> { + fn uart(&mut self, port: usize) -> &mut dyn SerialDmaUart { + &mut self.uart[port] } - fn sdma_run_from(&mut self, p: usize, since: Instant) { - if !self.sdma_reachable(p) || self.sdma.port[p].busy { - return; - } - self.sdma.port[p].busy = true; - self.sdma_transmit(p, since); - self.sdma_receive(p, since); - self.sdma_drain(p); - self.sdma_update_high_water(p); - self.sdma_update_tx_empty(p); - self.sdma.port[p].busy = false; + fn raise(&mut self, port: usize, src: SerialDmaSource) { + *self.int_status |= int::serial(port, src as u32); } - fn sdma_run(&mut self, p: usize) { - self.sdma_run_from(p, Instant::now()); + fn set_level(&mut self, port: usize, src: SerialDmaSource, level: bool) { + self.set_bit(int::serial(port, src as u32), level); } - /// A passive mirror of the 16550's lines; no read side effects. - fn sdma_shadow(&self, p: usize) -> u32 { - let u = &self.uart[p]; - let lsr = u.core.lsr(); - let mcr = u.core.mcr(); - let mut msr = u.core.msr(); - if u.flow_cts(self.sdma_ctl(p)) { - msr |= MSR_CTS; - } - let mut s = lsr as u32; - for (m, bit) in [(MSR_DCTS, 16), (0x08, 19), (MSR_CTS, 20), (MSR_DCD, 23)] { - if msr & m != 0 { - s |= 1 << bit; - } - } - for (m, bit) in [(MCR_DTR, 24), (MCR_RTS, 25), (MCR_OUT1, 26), (MCR_OUT2, 27), (MCR_LOOP, 28)] { - if mcr & m != 0 { - s |= 1 << bit; - } + /// Chip control places each port's 16550 for its engine; aimed anywhere + /// else, the engine does not reach the port. + fn reachable(&self, port: usize) -> bool { + if !self.serial[port] { + return false; } - s + let (shift, base) = if port == 1 { + (CHIP_CTRL_PORT_B_SHIFT, reg::SERIAL_B) + } else { + (CHIP_CTRL_PORT_A_SHIFT, reg::SERIAL_A) + }; + (self.chip_control >> shift) & CHIP_CTRL_PORT_FIELD == (base & 0xfff) / 8 } - fn sdma_port_read(&mut self, p: usize, off: u32) -> u32 { - let now = Instant::now(); - self.sdma_catch_up(now); - if off == sdma_off::SHADOW { - return self.sdma_shadow(p); - } - let port = &self.sdma.port[p]; - let mut v = port.reg[(off / 4) as usize]; - match off { - sdma_off::CONTROL => { - if v & sdma_ctl::PAUSE != 0 { - v |= sdma_ctl::PAUSED; - } - if port.tx_delay_deadline.is_some() { - v |= sdma_ctl::TX_TIMER_RUN; - } - if port.drain_pending { - v |= sdma_ctl::RX_TO_MEMORY; - } - } - sdma_off::RX_CONSUME => { - if port.armed { - v |= ARM_RX_TIMER; - } - } - sdma_off::RX_PRODUCE => v |= (port.packed as u32) << HELD_SHIFT, - sdma_off::RX_TIMER => { - // Ticks left while running or frozen; idle, whatever a reload - // write last loaded. - let mut ticks = port.rx_idle_count; - if let (true, Some(left)) = (port.paused, port.rx_frozen) { - ticks = (left.as_nanos() as u64 / TIMER_TICK_NS).min(TIMER_RELOAD as u64) as u32; - } else if let Some(d) = port.rx_deadline { - let left = d.saturating_duration_since(now); - ticks = (left.as_nanos() as u64 / TIMER_TICK_NS).min(TIMER_RELOAD as u64) as u32; - } - v |= ticks << TIMER_COUNT_SHIFT; - } - _ => {} - } - v + /// Keep the pass-through bit on the 16550's level. + fn uart_line_changed(&mut self, port: usize) { + let level = self.uart[port].irq(); + self.set_bit(int::serial(port, int::PASSTHROUGH), level); } - fn sdma_port_write(&mut self, p: usize, off: u32, val: u32) { - let now = Instant::now(); - self.sdma_catch_up(now); - let mask = self.sdma_mask(); - match off { - sdma_off::CONTROL => { - self.sdma.port[p].reg[0] = val & !sdma_ctl::UNSTORED; - if val & sdma_ctl::RESET != 0 { - // Resets the four indices and nothing else. - let port = &mut self.sdma.port[p]; - port.reg[1..=4].fill(0); - port.armed = false; - port.drain_pending = false; - port.packed = 0; - port.post_refused = false; - port.packer = [0; ENTRY]; - port.tx_slot = 0; - port.tx_fetched = false; - port.tx_delay_deadline = None; - port.tx_delay_irq = false; - port.rx_deadline = None; - port.rx_frozen = None; - port.rx_idle_count = 0; - } - if val & sdma_ctl::RX_TO_MEMORY != 0 { - self.sdma.port[p].drain_pending = true; - } - let port = &mut self.sdma.port[p]; - // A pause freezes the receive timer and a transmit delay; - // resuming lets each finish the time it had left. A model - // choice; no source says what the part does. - if val & sdma_ctl::PAUSE != 0 && !port.paused { - port.paused = true; - port.rx_frozen = port.rx_deadline.take().map(|d| d.saturating_duration_since(now)); - port.tx_delay_frozen = port.tx_delay_deadline.take().map(|d| d.saturating_duration_since(now)); - } else if val & sdma_ctl::PAUSE == 0 && port.paused { - port.paused = false; - port.rx_deadline = port.rx_frozen.take().map(|l| now + l); - port.tx_delay_deadline = port.tx_delay_frozen.take().map(|l| now + l); - } - if val & sdma_ctl::ENABLE == 0 { - port.rx_deadline = None; - port.rx_frozen = None; - } - self.sdma_run(p); - } - sdma_off::TX_PRODUCE | sdma_off::TX_CONSUME | sdma_off::RX_PRODUCE => { - self.sdma.port[p].reg[(off / 4) as usize] = val & mask; - self.sdma_run(p); - } - sdma_off::RX_CONSUME => { - // Bit 31 sets the arm; a write with it clear leaves it. - self.sdma.port[p].reg[4] = val & mask; - if val & ARM_RX_TIMER != 0 { - self.sdma.port[p].armed = true; - } - self.sdma_run(p); - } - sdma_off::RX_TIMER => { - let port = &mut self.sdma.port[p]; - port.reg[5] = val & TIMER_RELOAD; - port.rx_idle_count = val & TIMER_RELOAD; - } - // The shadow discards writes silently. - _ => {} - } - self.sdma_update_tx_empty(p); - self.sdma_update_high_water(p); + fn dma_read(&self, addr: u64, buf: &mut [u8]) -> bool { + self.dma.is_some_and(|d| d.dma_read(addr, buf).is_ok()) + } + + fn dma_write(&self, addr: u64, data: &[u8]) -> bool { + self.dma.is_some_and(|d| d.dma_write(addr, data).is_ok()) } } @@ -1540,7 +1142,10 @@ impl St { } // The set alias reads back the live mask; the clear one does not. reg::INT_ENABLE_SET => self.int_enable, - reg::CHIP_CONTROL => self.chip_control | CHIP_CTRL_ARB_IDLE, + // Bit 22, arbiter idle, polled by the kernel's console setup. It + // reads busy only while the ring engine is inside a transfer, the + // rule IOC4's serial idle bit follows too. + reg::CHIP_CONTROL => self.chip_control | if self.sdma.idle() { CHIP_CTRL_ARB_IDLE } else { 0 }, reg::INT_OUTPUT => { self.io_catch_up(Instant::now()); self.int_out.reg | if self.int_out.state { IO_STATE } else { 0 } @@ -1550,9 +1155,9 @@ impl St { reg::PIN_CONTROL_SET | reg::PIN_CONTROL_CLR => self.pin_control, reg::PIN_DATA => self.pin_data_now(), 0x80..=0x98 => self.par_dma_read(a), - 0x9c | 0xa0 | 0xa4 => self.kbc_read(a, c.ps2), - 0xb0 => self.sdma.base_hi, - 0xb4 => self.sdma.base_lo, + reg::KBC_BASE | 0xa0 | 0xa4 => self.kbc_read(a, c.ps2), + 0xb0 => self.sdma.base_read(0, true), + 0xb4 => self.sdma.base_read(0, false), 0xb8..=0xef => { let (p, off) = Self::sdma_port_of(a); self.sdma_port_read(p, off) @@ -1622,9 +1227,9 @@ impl St { reg::PIN_CONTROL_CLR => self.pin_control &= !val, reg::PIN_DATA => self.pin_data = val, 0x80..=0x98 => self.par_dma_write(a, val), - 0x9c | 0xa8 | 0xac => self.kbc_write(a, val, c.ps2), - 0xb0 => self.sdma.base_hi = val, - 0xb4 => self.sdma.base_lo = val, + reg::KBC_BASE | 0xa8 | 0xac => self.kbc_write(a, val, c.ps2), + 0xb0 => self.sdma.base_write(0, true, val), + 0xb4 => self.sdma.base_write(0, false, val), 0xb8..=0xef => { let (p, off) = Self::sdma_port_of(a); if self.cfg.serial[0] || self.cfg.serial[1] { @@ -1637,97 +1242,25 @@ impl St { // ----- PS/2 ----- - fn kbc_clamp(&self, port: usize) -> usize { - let (one, three) = if port == 1 { (KBC_M_CLAMP_1, KBC_M_CLAMP_3) } else { (KBC_K_CLAMP_1, KBC_K_CLAMP_3) }; - if self.kbc.control & one != 0 { - 1 - } else if self.kbc.control & three != 0 { - KBC_RX_SLOTS - } else { - 0 + /// Move what the host's PS/2 model has sent into the holdings, and latch + /// the keyboard source if a character arrived. + fn kbc_fill(&mut self, ps2: &Ps2Controller) { + if self.kbc.fill_all(ps2) { + self.int_status |= int::KEYBOARD; } } - /// Move what the device has sent into this port's holding, up to what - /// the clamp allows. A clamped port stops with its clock held low and - /// the rest still in the device; an unclamped one loses the surplus and - /// says so through the overflow flag. Every character that arrives - /// latches the keyboard interrupt (a model choice: per character rather - /// than a level over the holding). - fn kbc_fill(&mut self, port: usize, ps2: &Ps2Controller) { - if !self.cfg.keyboard_mouse { - return; - } - let clamp = self.kbc_clamp(port); - let mut arrived = false; - loop { - if clamp != 0 && self.kbc.port[port].count >= clamp { - break; - } - let Some(byte) = ps2.pop_from(port == 1) else { break }; - let p = &mut self.kbc.port[port]; - if p.count >= KBC_RX_SLOTS { - p.overflow = true; - continue; - } - p.data[p.count] = byte; - p.count += 1; - arrived = true; - } + fn kbc_read(&mut self, a: u32, ps2: &Ps2Controller) -> u32 { + let (val, arrived) = self.kbc.read(a - reg::KBC_BASE, ps2); if arrived { self.int_status |= int::KEYBOARD; } - } - - /// Report what the port holds, oldest character in the highest slot, and - /// empty the holding; the clamp releases with it. - fn kbc_receive(&mut self, port: usize, ps2: &Ps2Controller) -> u32 { - let p = self.kbc.port[port]; - let mut val = 0u32; - for (i, &b) in p.data.iter().enumerate().take(p.count) { - let slot = (KBC_RX_SLOTS - 1) - i; - val |= (b as u32) << (8 * slot); - val |= 1 << (29 + slot); - } - if p.count != 0 && port == 1 { - val |= KBC_RX_SOURCE_AUX; - } - if p.overflow { - val |= KBC_RX_OVERFLOW; - } - self.kbc.port[port] = KbcPort::default(); - // The stopped device resumes inside this read; the part takes a - // character time (an approximation). - self.kbc_fill(port, ps2); val } - fn kbc_read(&mut self, a: u32, ps2: &Ps2Controller) -> u32 { - match a { - KBC_CONTROL => self.kbc.control, - KBC_KBD_RECEIVE => self.kbc_receive(0, ps2), - KBC_AUX_RECEIVE => self.kbc_receive(1, ps2), - _ => 0, - } - } - fn kbc_write(&mut self, a: u32, val: u32, ps2: &Ps2Controller) { - match a { - KBC_CONTROL => { - self.kbc.control = val & KBC_CONTROL_MASK; - // Releasing a clamp lets a stopped device send again. - self.kbc_fill(0, ps2); - self.kbc_fill(1, ps2); - } - KBC_KBD_TRANSMIT | KBC_AUX_TRANSMIT => { - if !self.cfg.keyboard_mouse { - return; - } - ps2.write_to(val as u8, a == KBC_AUX_TRANSMIT); - self.kbc_fill(0, ps2); - self.kbc_fill(1, ps2); - } - _ => {} + if self.kbc.write(a - reg::KBC_BASE, val, ps2) { + self.int_status |= int::KEYBOARD; } } @@ -2040,18 +1573,6 @@ impl St { // The device // --------------------------------------------------------------------------- -/// A PS/2 device model's "data waiting" line, which tells the chip a -/// character has arrived. -struct KbcSink(Weak); - -impl InterruptController for KbcSink { - fn set_line(&self, _line: u32, _level: bool) { - if let Some(sh) = self.0.upgrade() { - sh.kbc_poke(); - } - } -} - /// How long a serial receive thread sleeps when its backend has nothing: /// `SerialBackend` has no blocking read to wait on. const RX_IDLE: Duration = Duration::from_millis(2); @@ -2065,9 +1586,15 @@ const K_RX: usize = 0; const K_TX_DELAY: usize = 2; const K_INT_OUT: usize = 4; const K_NVRAM: usize = 5; -const KINDS: usize = 6; +const K_LOOP: usize = 6; +const KINDS: usize = 8; type Deadlines = [Option; KINDS]; +/// The least delay for a kind's timer; see [`IO_REARM_MIN`]. +fn timer_floor(kind: usize) -> Duration { + if kind == K_INT_OUT { IO_REARM_MIN } else { Duration::ZERO } +} + /// A rewrite of the NVRAM file waiting for the writer thread. type NvramJob = (std::path::PathBuf, Vec); @@ -2099,19 +1626,23 @@ struct Shared { /// A PS/2 character arrived while an access held the chip. kbc_dirty: AtomicBool, me: Weak, - timers: OnceLock>, + timers: DeadlineTimers, events: OnceLock>, /// Between `start` and `stop`. running: AtomicBool, /// Held by a timer callback while it runs, so `stop` can wait one out. gate: Mutex<()>, - /// The timer pending for each deadline kind, and the deadline it serves. - armed: Mutex<[Option<(TimerId, Instant)>; KINDS]>, /// The NVRAM image the writer thread has yet to put in the file. nvram_job: Mutex>, nvram_ready: Condvar, } +impl KbcOwner for Shared { + fn kbc_poke(&self) { + self.kbc_poke_impl(); + } +} + impl Shared { /// Put the SuperIO line at the level the chip's state gives, then pick up /// any PS/2 character that arrived while the chip was held (a poke sets @@ -2123,9 +1654,7 @@ impl Shared { if !self.kbc_dirty.swap(false, Ordering::SeqCst) { break; } - let mut st = self.st.lock(); - st.kbc_fill(0, &self.ps2); - st.kbc_fill(1, &self.ps2); + self.st.lock().kbc_fill(&self.ps2); } } @@ -2201,32 +1730,9 @@ impl Shared { } } - /// Make the pending timers match `want`: one one-shot per deadline. A - /// timer whose deadline moved is replaced; with none set or the chip not - /// running, deadlines are only evaluated lazily, by the next access. + /// Arm a timer for each deadline the chip holds. fn arm(&self, want: Deadlines) { - let Some(tm) = self.timers.get() else { return }; - let mut armed = self.armed.lock(); - // Checked under the lock `stop` clears the slots with: a pass that - // sees `running` set has its timers removed by that stop. - if !self.running.load(Ordering::SeqCst) { - return; - } - for (k, w) in want.iter().enumerate() { - if armed[k].map(|(_, d)| d) == *w { - continue; - } - if let Some((id, _)) = armed[k].take() { - tm.remove(id); - } - let Some(due) = *w else { continue }; - let mut delay = due.saturating_duration_since(Instant::now()); - if k == K_INT_OUT { - delay = delay.max(IO_REARM_MIN); - } - let id = tm.add_one_shot(delay, self.me.clone(), move |me| Shared::fire(me, k, due)); - armed[k] = Some((id, due)); - } + self.timers.arm(&want, &self.running, &self.me, timer_floor, Shared::fire); } /// A deadline passed with no guest access to notice it. Runs on the timer @@ -2238,12 +1744,7 @@ impl Shared { if !sh.running.load(Ordering::SeqCst) { return TimerReturn::Delete; } - { - let mut armed = sh.armed.lock(); - if armed[kind].map(|(_, d)| d) == Some(due) { - armed[kind] = None; - } - } + sh.timers.fired(kind, due); if kind == K_NVRAM { // The file is written by its own thread, not here. if let Some(job) = sh.st.lock().rtc.take_dirty(NVRAM_QUIET) { @@ -2308,12 +1809,11 @@ impl Shared { /// very thread, from inside a transmit), that access picks the character /// up when it finishes: the flag is set before the lock is tried, so /// either this thread gets the lock or the holder sees the flag. - fn kbc_poke(&self) { + fn kbc_poke_impl(&self) { self.kbc_dirty.store(true, Ordering::SeqCst); if let Some(mut st) = self.st.try_lock() { if self.kbc_dirty.swap(false, Ordering::SeqCst) { - st.kbc_fill(0, &self.ps2); - st.kbc_fill(1, &self.ps2); + st.kbc_fill(&self.ps2); } drop(st); self.sync_superio(); @@ -2382,6 +1882,7 @@ impl Shared { /// The IOC3 PCI function. pub struct Ioc3 { sh: Arc, + timekeeper: OnceLock>, /// The serial receive threads and the NVRAM writer, joined in `stop`. threads: Mutex>>, } @@ -2389,22 +1890,26 @@ pub struct Ioc3 { impl Ioc3 { pub fn new(cfg: Ioc3Config) -> Ioc3 { let subsystem = cfg.subsystem_id; + let pci_policy = cfg.pci_policy; let sh = Arc::new_cyclic(|weak: &Weak| { - let line = IrqLine::new(Arc::new(KbcSink(weak.clone())), 0); + let line = IrqLine::new(Arc::new(KbcLine(weak.clone())), 0); let ps2 = Arc::new(Ps2Controller::new(Some(line))); // Host key and mouse events are accepted from the start (this // only sets a flag); `stop` and `start` follow the chip's. Device::start(&*ps2); - let pci = PciConfig::new( - VENDOR_ID, - DEVICE_ID, - 0xff_0000, - REVISION, - [Bar::Mem(BAR_SIZE), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], - PCI_INTA, + let bars = [ + Bar::Mem(BAR_SIZE), + Bar::Unused, + Bar::Unused, + Bar::Unused, + Bar::Unused, + Bar::Unused, + ]; + let pci = PciConfig::new_with_policy( + VENDOR_ID, DEVICE_ID, 0xff_0000, REVISION, bars, PCI_INTA, pci_policy, ); // The subsystem identity selects what a guest instantiates. - pci.write(11, ((subsystem as u32) << 16) | VENDOR_ID as u32); + pci.set_subsystem_identity(((subsystem as u32) << 16) | VENDOR_ID as u32); Shared { pci, st: Mutex::new(St::new(cfg)), @@ -2420,16 +1925,15 @@ impl Ioc3 { line_order: Mutex::new(()), kbc_dirty: AtomicBool::new(false), me: weak.clone(), - timers: OnceLock::new(), + timers: DeadlineTimers::new(), events: OnceLock::new(), running: AtomicBool::new(false), gate: Mutex::new(()), - armed: Mutex::new([None; KINDS]), nvram_job: Mutex::new(None), nvram_ready: Condvar::new(), } }); - Ioc3 { sh, threads: Mutex::new(Vec::new()) } + Ioc3 { sh, timekeeper: OnceLock::new(), threads: Mutex::new(Vec::new()) } } /// The Octane's (or Octane2's) onboard IOC3: both serial ports, PS/2, @@ -2438,9 +1942,15 @@ impl Ioc3 { Ioc3::new(Ioc3Config::octane(mac, octane2)) } - /// The timer manager the deadlines are armed on. Call before `start`. + /// Attach Fuel's independently owned flat select-0 system clock. + pub fn attach_timekeeper(&self, clock: Arc) { + assert!(!self.sh.st.lock().cfg.rtc, "indexed Dallas and flat Timekeeper populations conflict"); + assert!(self.timekeeper.set(clock).is_ok(), "IOC3 timekeeper attached more than once"); + } + + /// The timer manager for device deadlines. Set it before start. pub fn set_timer_manager(&self, tm: Arc) { - let _ = self.sh.timers.set(tm); + self.sh.timers.set_manager(tm); } /// Where the DS1685's power-off goes: one `PowerOff` event when the guest @@ -2545,6 +2055,9 @@ impl Ioc3 { /// Ethernet block is a separate device here and keeps its state: the /// machine resets it on its own. pub fn reset(&self) { + if self.sh.st.lock().cfg.reset_pci_on_power_on { + self.sh.pci.reset(); + } self.sh.with_st(false, |st, _| st.reset()); } @@ -2615,14 +2128,7 @@ impl Device for Ioc3 { if !self.sh.running.swap(false, Ordering::SeqCst) { return; } - { - let mut armed = self.sh.armed.lock(); - for slot in armed.iter_mut() { - if let (Some((id, _)), Some(tm)) = (slot.take(), self.sh.timers.get()) { - tm.remove(id); - } - } - } + self.sh.timers.cancel_all(); drop(self.sh.gate.lock()); // The writer waits on the job slot's lock: take it so the wake-up // cannot fall between its check of `running` and its wait. @@ -2703,6 +2209,16 @@ impl PciBusDevice for Ioc3 { if width == 4 && a <= reg::HEADER_LAST && a & 3 == 0 { return self.header_dword((a / 4) as usize); } + if (reg::BYTEBUS_DEV0..reg::BYTEBUS_DEV0 + crate::dev::timekeeper::SIZE as u32).contains(&a) { + if let Some(clock) = self.timekeeper.get() { + if width != 1 { + crate::dlog!(crate::devlog::LogModule::Ioc, "IOC3 Timekeeper {width}-byte read refused"); + return 0; + } + return u32::from(clock.read8(u64::from(a - reg::BYTEBUS_DEV0)).data); + } + } + if (reg::ENET_FIRST..=reg::ENET_LAST).contains(&a) { // Only 32-bit accesses reach the Ethernet block. if width != 4 { @@ -2719,6 +2235,17 @@ impl PciBusDevice for Ioc3 { return; } let a = off ^ (4 - width.min(4) as u32); + if (reg::BYTEBUS_DEV0..reg::BYTEBUS_DEV0 + crate::dev::timekeeper::SIZE as u32).contains(&a) { + if let Some(clock) = self.timekeeper.get() { + if width != 1 { + crate::dlog!(crate::devlog::LogModule::Ioc, "IOC3 Timekeeper {width}-byte write refused"); + return; + } + clock.write8(u64::from(a - reg::BYTEBUS_DEV0), val as u8); + return; + } + } + if (reg::ENET_FIRST..=reg::ENET_LAST).contains(&a) { if width == 4 && a & 3 == 0 { if let Some(e) = self.sh.eth() { @@ -2759,23 +2286,23 @@ impl PciBusDevice for Ioc3 { // was waiting out. Deadlines are encoded as nanoseconds left plus one, // zero for none. -fn until(due: Option, now: Instant) -> u64 { +pub(crate) fn until(due: Option, now: Instant) -> u64 { due.map_or(0, |d| d.saturating_duration_since(now).as_nanos() as u64 + 1) } -fn due(left: u64, now: Instant) -> Option { +pub(crate) fn due(left: u64, now: Instant) -> Option { (left != 0).then(|| now + Duration::from_nanos(left - 1)) } -fn span(d: Option) -> u64 { +pub(crate) fn span(d: Option) -> u64 { d.map_or(0, |d| d.as_nanos() as u64 + 1) } -fn unspan(n: u64) -> Option { +pub(crate) fn unspan(n: u64) -> Option { (n != 0).then(|| Duration::from_nanos(n - 1)) } -fn usize_field(v: &toml::Value, key: &str, limit: usize) -> Result { +pub(crate) fn usize_field(v: &toml::Value, key: &str, limit: usize) -> Result { let n = field_u64(v, key)? as usize; if n > limit { return Err(format!("{key}: {n} is more than {limit}")); @@ -2803,79 +2330,8 @@ impl Uart { } } -impl SdmaPort { - fn save_state(&self, now: Instant) -> toml::Value { - StateTable::new() - .u32s("reg", &self.reg) - .u8s("packer", &self.packer) - .u64("packed", self.packed as u64) - .u8s("tx_entry", &self.tx_entry) - .bool("tx_fetched", self.tx_fetched) - .u64("tx_slot", self.tx_slot as u64) - .bool("armed", self.armed) - .bool("drain_pending", self.drain_pending) - .bool("busy", self.busy) - .bool("paused", self.paused) - .u64("rx_due", until(self.rx_deadline, now)) - .u32("rx_idle_count", self.rx_idle_count) - .u64("rx_frozen", span(self.rx_frozen)) - .u64("tx_delay_due", until(self.tx_delay_deadline, now)) - .u64("tx_delay_frozen", span(self.tx_delay_frozen)) - .bool("tx_delay_irq", self.tx_delay_irq) - .bool("post_refused", self.post_refused) - .build() - } - fn load_state(v: &toml::Value, now: Instant) -> Result { - Ok(Self { - reg: field_u32s(v, "reg")?, - packer: field_u8s(v, "packer")?, - packed: usize_field(v, "packed", ENTRY)?, - tx_entry: field_u8s(v, "tx_entry")?, - tx_fetched: field_bool(v, "tx_fetched")?, - tx_slot: usize_field(v, "tx_slot", ENTRY)?, - armed: field_bool(v, "armed")?, - drain_pending: field_bool(v, "drain_pending")?, - busy: field_bool(v, "busy")?, - paused: field_bool(v, "paused")?, - rx_deadline: due(field_u64(v, "rx_due")?, now), - rx_idle_count: field_u32(v, "rx_idle_count")?, - rx_frozen: unspan(field_u64(v, "rx_frozen")?), - tx_delay_deadline: due(field_u64(v, "tx_delay_due")?, now), - tx_delay_frozen: unspan(field_u64(v, "tx_delay_frozen")?), - tx_delay_irq: field_bool(v, "tx_delay_irq")?, - post_refused: field_bool(v, "post_refused")?, - }) - } -} - -impl KbcPort { - fn save_state(&self) -> toml::Value { - StateTable::new() - .u8s("data", &self.data) - .u64("count", self.count as u64) - .bool("overflow", self.overflow) - .build() - } - fn load_state(v: &toml::Value) -> Result { - Ok(Self { - data: field_u8s(v, "data")?, - count: usize_field(v, "count", KBC_RX_SLOTS)?, - overflow: field_bool(v, "overflow")?, - }) - } -} - impl St { fn save_state(&self, now: Instant) -> toml::Value { - let mut sdma = StateTable::new(); - sdma.u32("base_hi", self.sdma.base_hi) - .u32("base_lo", self.sdma.base_lo) - .value("port0", self.sdma.port[0].save_state(now)) - .value("port1", self.sdma.port[1].save_state(now)); - let mut kbc = StateTable::new(); - kbc.u32("control", self.kbc.control) - .value("port0", self.kbc.port[0].save_state()) - .value("port1", self.kbc.port[1].save_state()); let p = &self.par; let fifo: Vec = p.fifo.iter().copied().collect(); let mut par = StateTable::new(); @@ -2914,8 +2370,8 @@ impl St { .u32("pin_data", self.pin_data) .value("uart0", self.uart[0].save_state()) .value("uart1", self.uart[1].save_state()) - .value("sdma", sdma.build()) - .value("kbc", kbc.build()) + .value("sdma", self.sdma.save_state(now)) + .value("kbc", self.kbc.save_state()) .value("parallel", par.build()) .value("rtc", rtc.build()) .u32s("ssram", &self.ssram) @@ -2931,17 +2387,8 @@ impl St { } let ssram: Vec = ssram.iter().map(crate::snapshot::toml_u32).collect::>() .ok_or("ssram: an entry is not a word")?; - let sdma = field(v, "sdma")?; - let sdma = Sdma { - base_hi: field_u32(sdma, "base_hi")?, - base_lo: field_u32(sdma, "base_lo")?, - port: [SdmaPort::load_state(field(sdma, "port0")?, now)?, SdmaPort::load_state(field(sdma, "port1")?, now)?], - }; - let kbc = field(v, "kbc")?; - let kbc = Kbc { - control: field_u32(kbc, "control")?, - port: [KbcPort::load_state(field(kbc, "port0")?)?, KbcPort::load_state(field(kbc, "port1")?)?], - }; + let sdma = self.sdma.loaded(field(v, "sdma")?, now)?; + let kbc = self.kbc.loaded(field(v, "kbc")?)?; let o = field(v, "int_out")?; let int_out = IntOut { reg: field_u32(o, "reg")?, diff --git a/src/dev/ioc3_eth.rs b/src/dev/ioc3_eth.rs index cbe3c0c88..4b9e9fa63 100644 --- a/src/dev/ioc3_eth.rs +++ b/src/dev/ioc3_eth.rs @@ -1,5 +1,5 @@ -//! The Ethernet MAC inside IOC3, with the ICS1890 transceiver the Octane -//! carries on its management bus. +//! The Ethernet MAC inside IOC3, with the Octane ICS1890 or Fuel DP83843 +//! transceiver on its management bus. //! //! IOC3 forwards the register offsets 0xf0..=0x150 of its register BAR here //! through [`Ioc3EthPort`]. The MAC is SGI's own design with no vendor @@ -17,7 +17,7 @@ //! Ring entries and descriptor words are big-endian bytes at bus addresses, //! frame bytes are in wire order; the Bridge applies any programmed swap. //! -//! Frames go through iris's network engine like the O2's MEC does: see +//! Frames use iris's shared network link. The machine attaches it through //! [`Ioc3Eth::attach`]. //! //! Approximations (the QEMU model this was ported from has the same ones, @@ -38,8 +38,8 @@ //! sends whenever the PHY's data path is open (not powered down or //! isolated), link or not. //! -//! Ported from hw/net/sgi-ioc3-eth.c and hw/net/ics1890.c in -//! jamesbraid/qemu-sgi-origami. +//! Ported from hw/net/sgi-ioc3-eth.c in jamesbraid/qemu-sgi-origami. The +//! Octane PHY model lives in `ics1890.rs`. use std::sync::atomic::{AtomicBool, AtomicU64, Ordering}; use std::sync::{Arc, OnceLock, Weak}; @@ -48,8 +48,9 @@ use parking_lot::Mutex; use std::time::{Duration, Instant}; use crate::dev::ioc3_port::Ioc3EthPort; +use crate::dev::ics1890::Ics1890; use crate::pci::{PciDma, PciDmaStatus}; -use crate::snapshot::{field, field_bool, field_u16, field_u32, field_u32s, field_u64, field_u8, StateTable}; +use crate::snapshot::{field, field_bool, field_u16, field_u32, field_u32s, field_u64, StateTable}; use crate::traits::{Device, Resettable, Saveable}; #[cfg(test)] @@ -163,6 +164,23 @@ pub const PHY_ADDR: u8 = 1; /// How long the synthetic link partner takes to complete negotiation. pub const NEGOTIATION_DELAY: Duration = Duration::from_millis(100); +/// The management and packet-path behavior that varies between IOC3 PHYs. +pub(crate) trait MiiPhy: Send { + fn address(&self) -> u8; + fn deadline(&self) -> Option; + fn wire_enabled(&self) -> bool; + fn in_loopback(&self) -> bool; + fn sync(&mut self, now: Instant); + fn read(&mut self, register: u8, now: Instant) -> u16; + fn write(&mut self, register: u8, value: u16, now: Instant); + fn set_backend_link(&mut self, up: bool, now: Instant); + fn power_on(&mut self); + /// The transceiver's registers and negotiation progress for a snapshot; + /// its deadline is saved as the time it had left. + fn save_state(&self, now: Instant) -> toml::Value; + fn load_state(&mut self, v: &toml::Value, now: Instant) -> Result<(), String>; +} + fn ridx(off: u32) -> usize { ((off - MAC_CONTROL) / 4) as usize } @@ -171,345 +189,6 @@ fn ridx(off: u32) -> usize { // ICS1890 transceiver // --------------------------------------------------------------------------- -/// Ordinary and command-override writable masks, by register. -fn phy_masks(r: usize) -> (u16, u16) { - match r { - 0 => (0xff80, 0), - 1 => (0, 0x7f80), - 2 | 3 => (0, 0xffff), - 4 => (0x21e0, 0x1c1f), - 6 => (0, 0xffe0), - 0x10 => (0x802d, 0), - 0x12 => (0x002f, 0), - 0x13 => (0x0011, 0), - _ => (0, 0), - } -} - -/// The ICS1890 as the Octane carries it: address 1, node mode, software -/// priority, revision 3. -pub struct Ics1890 { - addr: u8, - bmcr: u16, - bmsr_caps: u16, - id1: u16, - id2: u16, - anar: u16, - partner: u16, - exp: u16, - ext_ctl: u16, - tenbt: u16, - ext2: u16, - override_armed: bool, - page_received: bool, - link_low_latch: bool, - last_link: bool, - backend_up: bool, - nego_done: bool, - deadline: Option, - delay: Duration, - /// Accesses to registers the part does not implement. - pub unimplemented_accesses: u32, -} - -impl Ics1890 { - pub fn new(addr: u8, delay: Duration) -> Self { - let mut p = Self { - addr, - bmcr: 0, - bmsr_caps: 0, - id1: 0, - id2: 0, - anar: 0, - partner: 0, - exp: 0, - ext_ctl: 0, - tenbt: 0, - ext2: 0, - override_armed: false, - page_received: false, - link_low_latch: false, - last_link: false, - backend_up: true, - nego_done: false, - deadline: None, - delay, - unimplemented_accesses: 0, - }; - p.reset(Instant::now()); - p - } - - pub fn address(&self) -> u8 { - self.addr - } - - /// Hardware and software reset: defaults, no override, no latches, and a - /// fresh negotiation if enabled. - pub fn reset(&mut self, now: Instant) { - self.bmcr = 0x3000; - self.bmsr_caps = 0x7800; - self.id1 = 0x0015; - self.id2 = 0xf423; - self.anar = 0x01e1; - self.partner = 0; - self.exp = 0; - self.ext_ctl = 0x0008; - self.tenbt = 0; - self.ext2 = 0x0001; - self.override_armed = false; - self.page_received = false; - self.link_low_latch = false; - self.nego_done = false; - self.deadline = None; - self.last_link = false; - self.start_negotiation(now); - } - - fn autoneg(&self) -> bool { - self.bmcr & 0x1000 != 0 - } - fn power_down(&self) -> bool { - self.bmcr & 0x0800 != 0 - } - fn isolated(&self) -> bool { - self.bmcr & 0x0400 != 0 - } - - /// The live link: down in power-down, down without a backend, and with - /// negotiation enabled up only once it has completed. - fn link(&self) -> bool { - !self.power_down() && self.backend_up && (!self.autoneg() || self.nego_done) - } - - fn start_negotiation(&mut self, now: Instant) { - self.nego_done = false; - self.partner = 0; - self.page_received = false; - self.deadline = None; - if self.autoneg() && self.backend_up && !self.power_down() { - // The synthetic peer advertises 0x01e1: IEEE 802.3, no fault. - let common = self.anar & 0x01e0; - if self.anar & 0x1f == 1 && common != 0 && self.anar & 0x2000 == 0 { - self.deadline = Some(now + self.delay); - } - } - } - - /// When a negotiation in progress completes, if one is. - pub fn deadline(&self) -> Option { - self.deadline - } - - /// Advance the negotiation deadline and note link transitions. Reads never - /// start or postpone negotiation; this only observes the clock. - pub fn sync(&mut self, now: Instant) { - if let Some(d) = self.deadline { - if now >= d && !self.power_down() && self.backend_up { - self.deadline = None; - self.nego_done = true; - self.partner = 0x01e1; - self.page_received = true; - } - } - let link = self.link(); - if self.last_link && !link { - self.link_low_latch = true; - } - self.last_link = link; - } - - /// The packet backend's cable state. - pub fn set_backend_link(&mut self, up: bool, now: Instant) { - self.sync(now); - if self.backend_up == up { - return; - } - self.backend_up = up; - if up { - // Recovery starts a fresh negotiation. - self.start_negotiation(now); - } else { - self.deadline = None; - self.nego_done = false; - } - self.sync(now); - } - - /// BMCR loopback turns frames round inside the part. Isolate and - /// power-down take precedence. - pub fn in_loopback(&self) -> bool { - self.bmcr & 0x4000 != 0 && !self.power_down() && !self.isolated() - } - - /// Frames may cross the MII at all. - pub fn data_path(&self) -> bool { - !self.power_down() && !self.isolated() - } - - /// Frames may go to or come from the packet backend. - pub fn wire_enabled(&self) -> bool { - self.data_path() && self.link() - } - - fn resolved(&self) -> (bool, bool) { - // (100 Mb/s, full duplex) - if self.autoneg() { - let common = self.anar & self.partner & 0x01e0; - if common & 0x100 != 0 { - (true, true) - } else if common & 0x80 != 0 { - (true, false) - } else if common & 0x40 != 0 { - (false, true) - } else { - (false, false) - } - } else { - (self.bmcr & 0x2000 != 0, self.bmcr & 0x0100 != 0) - } - } - - pub fn read(&mut self, r: u8, now: Instant) -> u16 { - self.sync(now); - match r { - 0 => self.bmcr, - 1 => { - let mut v = 0x0009 | self.bmsr_caps; - if self.autoneg() && self.nego_done { - v |= 0x0020; - } - if self.link() && !self.link_low_latch { - v |= 0x0004; - } - self.link_low_latch = false; - v - } - 2 => self.id1, - 3 => self.id2, - 4 => self.anar, - 5 => self.partner, - 6 => { - let v = (self.exp & 0xffe0) - | if self.nego_done { 1 } else { 0 } - | if self.page_received { 2 } else { 0 }; - self.page_received = false; - v - } - 0x10 => { - // Bit 4 is a fixed reserved read choice (stock firmware - // expects it set); bit 15 reads the override state. - (self.ext_ctl & 0x002d) - | (u16::from(self.addr & 0x1f) << 6) - | 0x0010 - | if self.override_armed { 0x8000 } else { 0 } - } - 0x11 => { - let link = self.link(); - let (speed, full) = self.resolved(); - let mut v = 0; - if link { - v |= 1; - if speed { - v |= 0x8000; - } - if full { - v |= 0x4000; - } - } - if self.nego_done { - v |= 0x0010 | (7 << 11); - } - v - } - 0x12 => (self.tenbt & 0x002f) | 0x0010, - // Node mode strap in bit 14, software bit 0. - 0x13 => (self.ext2 & 0x0011) | 0x4000, - _ => { - self.unimplemented_accesses += 1; - 0 - } - } - } - - pub fn write(&mut self, r: u8, val: u16, now: Instant) { - self.sync(now); - let r = usize::from(r); - let armed = self.override_armed; - match r { - 0 => { - if val & 0x8000 != 0 { - self.reset(now); - self.sync(now); - return; - } - let old = self.bmcr; - // Reset and restart are self-clearing. - self.bmcr = val & 0x7d80; - let restart = val & 0x0200 != 0; - let pd_now = self.power_down(); - if pd_now && old & 0x0800 == 0 { - // Entering power-down clears the latching status. - self.link_low_latch = false; - self.page_received = false; - self.deadline = None; - } else if !pd_now && old & 0x0800 != 0 { - // Release: enabled negotiation starts afresh. - self.start_negotiation(now); - } else if self.autoneg() && (restart || old & 0x1000 == 0) { - self.start_negotiation(now); - } else if !self.autoneg() { - self.nego_done = false; - self.deadline = None; - } - if armed { - self.override_armed = false; - } - } - 1 | 2 | 3 | 4 | 6 | 0x10 | 0x12 | 0x13 => { - let (ord, ovr) = phy_masks(r); - let mask = ord | if armed { ovr } else { 0 }; - let cell = match r { - 1 => &mut self.bmsr_caps, - 2 => &mut self.id1, - 3 => &mut self.id2, - 4 => &mut self.anar, - 6 => &mut self.exp, - 0x10 => &mut self.ext_ctl, - 0x12 => &mut self.tenbt, - _ => &mut self.ext2, - }; - let before = *cell; - *cell = (before & !mask) | (val & mask); - // The status caps live in bits 14..7 only. - if r == 1 { - self.bmsr_caps &= 0x7f80; - } - if r == 4 && self.anar != before && self.autoneg() { - self.start_negotiation(now); - } - if armed { - self.override_armed = false; - } else if r == 0x10 && val & 0x8000 != 0 { - self.override_armed = true; - } - } - _ => { - // Partner ability, Quick Poll and unimplemented registers - // ignore writes. The part is addressed, so a write still - // consumes an armed override. - if r != 5 && r != 0x11 { - self.unimplemented_accesses += 1; - } - if armed { - self.override_armed = false; - } - } - } - self.sync(now); - } -} - // --------------------------------------------------------------------------- // Checksums and the multicast hash // --------------------------------------------------------------------------- @@ -616,7 +295,7 @@ struct State { mii_data_w: u16, /// Apart from its index: an index-only write must not disarm. rx_timer_armed: bool, - phy: Ics1890, + phy: Box, irq_asserted: bool, /// Level changes not yet delivered to the callback, in order. irq_events: Vec, @@ -626,14 +305,14 @@ struct State { } impl State { - fn new(delay: Duration) -> Self { + fn new(phy: Box) -> Self { Self { regs: [0; NREGS], int_deferred: 0, mii_data: MII_ABSENT, mii_data_w: 0, rx_timer_armed: false, - phy: Ics1890::new(PHY_ADDR, delay), + phy, irq_asserted: false, irq_events: Vec::new(), frame: vec![0; FRAME_MAX], @@ -642,6 +321,20 @@ impl State { } } + fn power_on(&mut self) { + self.phy.power_on(); + self.regs = [0; NREGS]; + self.int_deferred = 0; + self.mii_data = MII_ABSENT; + self.mii_data_w = 0; + self.rx_timer_armed = false; + self.irq_asserted = false; + self.irq_events.clear(); + self.frame.fill(0); + self.tx_frames = 0; + self.rx_frames = 0; + } + fn get(&self, off: u32) -> u32 { self.regs[ridx(off)] } @@ -1025,6 +718,7 @@ type IrqFn = Box; /// through the Bridge and HEART, which never call back into this device. pub struct Ioc3Eth { mac: [u8; 6], + thread_prefix: &'static str, st: Mutex, dma: Mutex>>, irq: Mutex>, @@ -1062,9 +756,19 @@ impl Ioc3Eth { /// As [`Ioc3Eth::new`], with the PHY's synthetic negotiation delay set. pub fn with_negotiation_delay(mac: [u8; 6], delay: Duration) -> Arc { + Self::with_phy(mac, Box::new(Ics1890::new(PHY_ADDR, delay)), "ip30-eth") + } + + /// Fuel uses the DP83843 at management address 1. + pub fn new_fuel(mac: [u8; 6]) -> Arc { + Self::with_phy(mac, Box::new(crate::dev::dp83843::Dp83843::new(PHY_ADDR)), "ioc3-eth") + } + + fn with_phy(mac: [u8; 6], phy: Box, thread_prefix: &'static str) -> Arc { Arc::new_cyclic(|me| Self { mac, - st: Mutex::new(State::new(delay)), + thread_prefix, + st: Mutex::new(State::new(phy)), dma: Mutex::new(None), irq: Mutex::new(None), irq_order: Mutex::new(()), @@ -1101,13 +805,7 @@ impl Ioc3Eth { /// with the state saying low. pub fn reset(&self) { let _order = self.irq_order.lock(); - { - let mut st = self.st.lock(); - let delay = st.phy.delay; - let backend_up = st.phy.backend_up; - *st = State::new(delay); - st.phy.set_backend_link(backend_up, Instant::now()); - } + self.st.lock().power_on(); if let Some(cb) = self.irq.lock().as_ref() { cb(false); } @@ -1175,8 +873,9 @@ impl Ioc3Eth { fn spawn_nat(&self) { let slot = self.backend.clone(); let Some(mut engine) = slot.lock().take() else { return }; + let thread_name = format!("{}-net", self.thread_prefix); let t = std::thread::Builder::new() - .name("ip30-eth-net".into()) + .name(thread_name) .spawn(move || { engine.run(); *slot.lock() = Some(engine); @@ -1281,8 +980,9 @@ impl Device for Ioc3Eth { } let wake = self.rx_wake.lock().get_or_insert_with(crate::net::Wake::new).clone(); let (me, running) = (self.me.clone(), self.running.clone()); + let thread_name = format!("{}-rx", self.thread_prefix); let rx = std::thread::Builder::new() - .name("ip30-eth-rx".into()) + .name(thread_name) .spawn(move || Self::rx_loop(me, running, wake)) .expect("ioc3 eth rx thread"); self.threads.lock().push(rx); @@ -1377,63 +1077,6 @@ impl Ioc3EthPort for Ioc3Eth { } } -// The transceiver's registers and auto-negotiation progress for a -// snapshot. Its one deadline is saved as the time it had left. -impl Ics1890 { - pub(crate) fn save_state(&self, now: Instant) -> toml::Value { - StateTable::new() - .u8("addr", self.addr) - .u16("bmcr", self.bmcr) - .u16("bmsr_caps", self.bmsr_caps) - .u16("id1", self.id1) - .u16("id2", self.id2) - .u16("anar", self.anar) - .u16("partner", self.partner) - .u16("exp", self.exp) - .u16("ext_ctl", self.ext_ctl) - .u16("tenbt", self.tenbt) - .u16("ext2", self.ext2) - .bool("override_armed", self.override_armed) - .bool("page_received", self.page_received) - .bool("link_low_latch", self.link_low_latch) - .bool("last_link", self.last_link) - .bool("backend_up", self.backend_up) - .bool("nego_done", self.nego_done) - // Nanoseconds left plus one; zero for none. - .u64("deadline", self.deadline.map_or(0, |d| d.saturating_duration_since(now).as_nanos() as u64 + 1)) - .u64("delay", self.delay.as_nanos() as u64) - .u32("unimplemented_accesses", self.unimplemented_accesses) - .build() - } - - pub(crate) fn load_state(&mut self, v: &toml::Value, now: Instant) -> Result<(), String> { - let deadline = field_u64(v, "deadline")?; - *self = Self { - addr: field_u8(v, "addr")?, - bmcr: field_u16(v, "bmcr")?, - bmsr_caps: field_u16(v, "bmsr_caps")?, - id1: field_u16(v, "id1")?, - id2: field_u16(v, "id2")?, - anar: field_u16(v, "anar")?, - partner: field_u16(v, "partner")?, - exp: field_u16(v, "exp")?, - ext_ctl: field_u16(v, "ext_ctl")?, - tenbt: field_u16(v, "tenbt")?, - ext2: field_u16(v, "ext2")?, - override_armed: field_bool(v, "override_armed")?, - page_received: field_bool(v, "page_received")?, - link_low_latch: field_bool(v, "link_low_latch")?, - last_link: field_bool(v, "last_link")?, - backend_up: field_bool(v, "backend_up")?, - nego_done: field_bool(v, "nego_done")?, - deadline: (deadline != 0).then(|| now + Duration::from_nanos(deadline - 1)), - delay: Duration::from_nanos(field_u64(v, "delay")?), - unimplemented_accesses: field_u32(v, "unimplemented_accesses")?, - }; - Ok(()) - } -} - // The Ethernet block's state for a snapshot: the MAC's registers, the // MII port and the transceiver. Frames in flight between the receive // thread and the host network are not state: with the threads stopped diff --git a/src/dev/ioc3_eth/tests.rs b/src/dev/ioc3_eth/tests.rs index 64a54bb4f..d0ea63891 100644 --- a/src/dev/ioc3_eth/tests.rs +++ b/src/dev/ioc3_eth/tests.rs @@ -814,6 +814,42 @@ fn the_last_reference_released_on_another_thread_stops_the_mac() { assert!(!running.load(Ordering::SeqCst), "the last release stops it"); } +#[test] +fn fuel_phy_mdio_decoder_and_machine_reset() { + let eth = Ioc3Eth::new_fuel(STATION); + assert_eq!(eth.mac(), STATION); + let read = |phy: u32, r: u32| { + eth.write32(MII_CONTROL, MII_READ | (phy << 5) | r); + eth.read32(MII_DATA_R) + }; + assert_eq!(read(1, 2), 0x2000); + assert_eq!(read(1, 3), 0x5c10); + assert_eq!(read(0, 2), 0xffff); + eth.write32(MII_DATA_W, 7); + eth.write32(MII_CONTROL, (1 << 5) | 0x19); + assert_eq!(read(1, 2), 0xffff); + assert_eq!(read(7, 2), 0x2000); + // A wrong-address reset has no effect. + eth.write32(MII_DATA_W, 0x8000); + eth.write32(MII_CONTROL, 1 << 5); + assert_eq!(read(7, 2), 0x2000); + eth.reset(); + assert_eq!(read(1, 3), 0x5c10); + assert_eq!(read(7, 2), 0xffff); + // The MAC soft reset leaves PHY programming intact. + eth.write32(MII_DATA_W, 0x21); + eth.write32(MII_CONTROL, (1 << 5) | 4); + eth.write32(MAC_CONTROL, MCR_RESET); + assert_eq!(read(1, 4), 0x21); + // Forced mode exposes the configured backend cable without waiting. + eth.write32(MII_DATA_W, 0x2100); + eth.write32(MII_CONTROL, 1 << 5); + for up in [false, true] { + eth.set_backend_link(up); + assert_eq!(read(1, 1) & 4 != 0, up); + } +} + /// A guest that does nothing but ACK: the whole host side of a lock-step /// TFTP read (transmit descriptor, NAT thread, built-in server, receive /// thread, receive ring), with the guest's own software left out. Every diff --git a/src/dev/ioc3_kbc.rs b/src/dev/ioc3_kbc.rs new file mode 100644 index 000000000..6cc45788f --- /dev/null +++ b/src/dev/ioc3_kbc.rs @@ -0,0 +1,295 @@ +//! SGI IOC3 PS/2 keyboard and mouse controller. +//! +//! A block of the IOC3 that the IOC4 carries too, at another place in its +//! BAR. It is a bit-level PS/2 line engine, not an 8042: five 32-bit +//! registers, a control and status register, a receive register per port +//! and a transmit register per port. Each port holds up to three received +//! characters; the clamp bits in the control register stop a port's device +//! once it has sent one or three, and with no clamp armed a fourth +//! character is lost and the receive register reports the overflow. +//! +//! The host's keyboard and mouse model is a [`Ps2Controller`], a byte queue +//! shared by both ports. This block drains it into the per-port holdings. +//! The chip around it owns the interrupt: every call that can move a +//! character in says so, and the chip latches its own source bit. +//! +//! # What is approximated here +//! +//! * The data-available line is asserted for every character that reaches a +//! holding, not once per drain. The silicon may pulse per character or hold +//! a level over the holding, and no public driver tells them apart. +//! * A device stopped by a clamp resumes inside the read that empties the +//! holding; the part takes a character time. A guest that reads and only +//! then acknowledges can clear the assertion those resumed characters +//! made. Linux acknowledges before it reads and OpenBSD reads until the +//! register comes back empty, so both avoid it. +//! * The control register's timeout bits (16, 17) and the hardware status +//! bits read clear: no stalled line is simulated, and a transmit completes +//! inside the write. +//! * Reset values and the register's endianness are the emulator's choice. +//! +//! Ported from hw/input/sgi-ioc3-kbc.c in jamesbraid/qemu-sgi-origami. QEMU's +//! two `ps2` devices are the shared [`Ps2Controller`] here, and its +//! re-entrancy guard around a fill is the owner's poke flag. + +use crate::dev::ioc3::usize_field; +use crate::snapshot::{field, field_bool, field_u32, field_u8s, StateTable}; +use std::sync::Weak; + +use crate::dev::ps2::Ps2Controller; +use crate::irq::InterruptController; + +/// Register offsets within the controller's window. +pub mod reg { + /// Control and status. + pub const CONTROL: u32 = 0x00; + /// Keyboard receive (read-only). + pub const KBD_RECEIVE: u32 = 0x04; + /// Mouse receive (read-only). + pub const AUX_RECEIVE: u32 = 0x08; + /// Keyboard transmit (write-only). + pub const KBD_TRANSMIT: u32 = 0x0c; + /// Mouse transmit (write-only). + pub const AUX_TRANSMIT: u32 = 0x10; + pub const MMIO_SIZE: u32 = AUX_TRANSMIT + 4; +} + +/// Characters a port holds. +pub const RX_SLOTS: usize = 3; +pub const PORT_KBD: usize = 0; +pub const PORT_AUX: usize = 1; + +/// Control bits, from the public Linux header. +pub mod ctrl { + pub const EMULATION: u32 = 1 << 12; + pub const FAST_CLOCK: u32 = 1 << 13; + pub const K_TO_EN: u32 = 1 << 18; + pub const M_TO_EN: u32 = 1 << 19; + pub const K_CLAMP_1: u32 = 1 << 20; + pub const M_CLAMP_1: u32 = 1 << 21; + pub const K_CLAMP_3: u32 = 1 << 22; + pub const M_CLAMP_3: u32 = 1 << 23; + pub const CLAMP_MASK: u32 = 0x00f0_0000; + /// Writable bits: the low half, the timeout enables and the clamps. + pub const MASK: u32 = 0x0000_ffff | K_TO_EN | M_TO_EN | CLAMP_MASK; +} + +/// Receive register: bit 27 says the characters came from the mouse, bit 28 +/// that one arrived with the slots full and was lost. +pub const RX_SOURCE_AUX: u32 = 1 << 27; +pub const RX_OVERFLOW: u32 = 1 << 28; + +/// The valid flag of receive slot `slot` (2 is the oldest character). +pub const fn rx_valid(slot: usize) -> u32 { + 1 << (29 + slot) +} + +/// What one port holds: the characters it has taken, oldest first, and +/// whether one was lost for want of room. +#[derive(Default, Clone, Copy)] +struct Holding { + data: [u8; RX_SLOTS], + count: usize, + overflow: bool, +} + +/// A chip that owns a [`Ioc3Kbc`] and wants to know when the host's PS/2 +/// model has a character. +pub trait KbcOwner: Send + Sync { + /// A device raised its line. If an access holds the chip (possibly this + /// very thread, from inside a transmit), that access picks the character + /// up when it finishes. + fn kbc_poke(&self); +} + +/// The line a PS/2 model drives when it has something to send. +pub struct KbcLine(pub Weak); + +impl InterruptController for KbcLine { + fn set_line(&self, _line: u32, _level: bool) { + if let Some(owner) = self.0.upgrade() { + owner.kbc_poke(); + } + } +} + +pub struct Ioc3Kbc { + /// Board wiring: whether the connectors are fitted. + ports_present: bool, + control: u32, + port: [Holding; 2], +} + +impl Ioc3Kbc { + pub fn new(ports_present: bool) -> Ioc3Kbc { + Ioc3Kbc { ports_present, control: 0, port: [Holding::default(); 2] } + } + + /// Reset: the control register and both holdings clear. + pub fn reset(&mut self) { + self.control = 0; + self.port = [Holding::default(); 2]; + } + + /// How many characters this port may take before its clock line is held + /// low. Zero means no clamp is armed, and then the controller has no way + /// to stop the device. Arming both clamps is not a combination any driver + /// writes, so the tighter one is taken. + fn clamp(&self, port: usize) -> usize { + let (one, three) = if port == PORT_AUX { + (ctrl::M_CLAMP_1, ctrl::M_CLAMP_3) + } else { + (ctrl::K_CLAMP_1, ctrl::K_CLAMP_3) + }; + if self.control & one != 0 { + 1 + } else if self.control & three != 0 { + RX_SLOTS + } else { + 0 + } + } + + /// Move what the device has sent into this port's holding, up to what + /// the clamp allows. A clamped port stops with its clock held low and the + /// rest still in the device; an unclamped one loses the surplus and says + /// so through the overflow flag. Returns whether a character arrived, + /// which is when the chip latches its source. + pub fn fill(&mut self, port: usize, ps2: &Ps2Controller) -> bool { + if !self.ports_present { + return false; + } + let clamp = self.clamp(port); + let mut arrived = false; + loop { + if clamp != 0 && self.port[port].count >= clamp { + break; + } + let Some(byte) = ps2.pop_from(port == PORT_AUX) else { break }; + let p = &mut self.port[port]; + if p.count >= RX_SLOTS { + p.overflow = true; + continue; + } + p.data[p.count] = byte; + p.count += 1; + arrived = true; + } + arrived + } + + /// Fill both ports. + pub fn fill_all(&mut self, ps2: &Ps2Controller) -> bool { + let kbd = self.fill(PORT_KBD, ps2); + let aux = self.fill(PORT_AUX, ps2); + kbd || aux + } + + /// Report what the port holds, oldest character in the highest slot, and + /// empty the holding: the register packs the characters since the last + /// read, so the read clears them, their flags and the overflow flag + /// together. The clamp releases with it. Returns the value and whether + /// the device's resumed characters arrived. + fn receive(&mut self, port: usize, ps2: &Ps2Controller) -> (u32, bool) { + let p = self.port[port]; + let mut val = 0u32; + for (i, &b) in p.data.iter().enumerate().take(p.count) { + let slot = (RX_SLOTS - 1) - i; + val |= u32::from(b) << (8 * slot); + val |= rx_valid(slot); + // The per-character error flag (bit 24 + slot) stays clear: the + // PS/2 layer delivers clean bytes. + } + if p.count != 0 && port == PORT_AUX { + val |= RX_SOURCE_AUX; + } + if p.overflow { + val |= RX_OVERFLOW; + } + self.port[port] = Holding::default(); + let arrived = self.fill(port, ps2); + (val, arrived) + } + + /// A 32-bit read at `off`. Returns the value, and whether a character + /// arrived as a result. + pub fn read(&mut self, off: u32, ps2: &Ps2Controller) -> (u32, bool) { + match off { + // Write-pending and the other status bits read clear: a + // synchronous transmit completes within the register write. + reg::CONTROL => (self.control, false), + reg::KBD_RECEIVE => self.receive(PORT_KBD, ps2), + reg::AUX_RECEIVE => self.receive(PORT_AUX, ps2), + _ => { + crate::dlog!(crate::devlog::LogModule::Ioc, "ioc3-kbc: read of write-only register {off:#x}"); + (0, false) + } + } + } + + /// A 32-bit write at `off`. Returns whether a character arrived. + pub fn write(&mut self, off: u32, val: u32, ps2: &Ps2Controller) -> bool { + match off { + reg::CONTROL => { + self.control = val & ctrl::MASK; + if val & (ctrl::EMULATION | ctrl::FAST_CLOCK) != 0 { + crate::dlog!(crate::devlog::LogModule::Ioc, "ioc3-kbc: emulation/fast-clock effects are not modeled"); + } + // Releasing a clamp lets a stopped device send again. + self.fill_all(ps2) + } + reg::KBD_TRANSMIT | reg::AUX_TRANSMIT => { + if !self.ports_present { + crate::dlog!(crate::devlog::LogModule::Ioc, "ioc3-kbc: transmit on an unwired port"); + return false; + } + ps2.write_to(val as u8, off == reg::AUX_TRANSMIT); + self.fill_all(ps2) + } + _ => { + crate::dlog!(crate::devlog::LogModule::Ioc, "ioc3-kbc: write of {val:#x} to read-only register {off:#x}"); + false + } + } + } +} + +// The controller's state for a snapshot of the IOC3 that carries it: the +// control register and each port's held characters. Whether the connectors +// are fitted is the board's. + +impl Holding { + fn save_state(&self) -> toml::Value { + StateTable::new() + .u8s("data", &self.data) + .u64("count", self.count as u64) + .bool("overflow", self.overflow) + .build() + } + fn load_state(v: &toml::Value) -> Result { + Ok(Self { + data: field_u8s(v, "data")?, + count: usize_field(v, "count", RX_SLOTS)?, + overflow: field_bool(v, "overflow")?, + }) + } +} + +impl Ioc3Kbc { + pub fn save_state(&self) -> toml::Value { + StateTable::new() + .u32("control", self.control) + .value("port0", self.port[0].save_state()) + .value("port1", self.port[1].save_state()) + .build() + } + + /// The controller `save_state` described, on this board's connectors. + pub fn loaded(&self, v: &toml::Value) -> Result { + Ok(Ioc3Kbc { + ports_present: self.ports_present, + control: field_u32(v, "control")?, + port: [Holding::load_state(field(v, "port0")?)?, Holding::load_state(field(v, "port1")?)?], + }) + } +} diff --git a/src/dev/ioc3_tests.rs b/src/dev/ioc3_tests.rs index ec7f1bfe5..7ae343cf5 100644 --- a/src/dev/ioc3_tests.rs +++ b/src/dev/ioc3_tests.rs @@ -6,6 +6,7 @@ //! Ported from tests/qtest/octane-test.c in jamesbraid/qemu-sgi-origami. use super::*; +use crate::dev::ioc3_kbc::{ctrl, RX_OVERFLOW}; use std::sync::Mutex; use crate::irq::LevelProbe; use crate::traits::MachineEvent; @@ -106,6 +107,15 @@ impl Rig { self.w32(reg::CHIP_CONTROL, 0x0002_2e5e); } + /// Let `chars` looped-back characters cross the line at the reset rate + /// (about a millisecond each), then play the timers. A looped-back + /// character takes a character time to reach the receiver, as it does + /// on the part. + fn line(&self, chars: u64) { + std::thread::sleep(Duration::from_micros(1100 * chars + 3000)); + self.c.service(); + } + /// DMA the way the chip sees memory: internal bytes. fn mem_view(&self) -> Ioc3Dma { Ioc3Dma::new(self.mem.clone()) @@ -528,7 +538,7 @@ fn type_keys(r: &Rig) { #[test] fn the_ps2_receive_holds_three_behind_the_clamp() { let r = Rig::new(); - r.w32(CTRL, KBC_K_CLAMP_3 | KBC_M_CLAMP_3); + r.w32(CTRL, ctrl::K_CLAMP_3 | ctrl::M_CLAMP_3); r.w32(K_TX, 0xf4); // enable scanning let _ = r.r32(K_RX); r.w32(reg::INT_STATUS, int::KEYBOARD); @@ -536,13 +546,87 @@ fn the_ps2_receive_holds_three_behind_the_clamp() { assert_ne!(r.r32(reg::INT_STATUS) & int::KEYBOARD, 0); let first = r.r32(K_RX); assert_eq!(first & (VALID[0] | VALID[1] | VALID[2]), VALID[0] | VALID[1] | VALID[2]); - assert_eq!(first & KBC_RX_OVERFLOW, 0, "nothing lost"); + assert_eq!(first & RX_OVERFLOW, 0, "nothing lost"); // The read released the clamp; the stopped keyboard resumed. let second = r.r32(K_RX); assert_ne!(second & VALID[2], 0, "the rest were held back, not dropped"); assert_ne!((second >> 16) & 0xff, (first >> 16) & 0xff, "and are not a replay"); } +/// Five key presses behind clamp-after-three, with the keyboard scanning +/// and its acknowledge read away. +fn five_keys_behind_the_clamp(r: &Rig) { + use winit::keyboard::KeyCode::{KeyA, KeyB, KeyC, KeyD, KeyE}; + r.w32(CTRL, ctrl::K_CLAMP_3 | ctrl::M_CLAMP_3); + r.w32(K_TX, 0xf4); + let _ = r.r32(K_RX); + r.w32(reg::INT_STATUS, int::KEYBOARD); + for k in [KeyA, KeyB, KeyC, KeyD, KeyE] { + r.c.ps2().push_kb(k, true); + } + assert_ne!(r.r32(reg::INT_STATUS) & int::KEYBOARD, 0); +} + +/// Ports sn0/ioc3/ps2-backlog-survives-an-early-acknowledge: Linux +/// acknowledges before its handler reads. A guest that acknowledges and +/// reads nothing is not handed the same interrupt again, the read that +/// follows still finds all three characters, and the backlog behind the +/// clamp raises the source a second time. +#[test] +fn the_ps2_backlog_survives_an_early_acknowledge() { + let r = Rig::new(); + five_keys_behind_the_clamp(&r); + for _ in 0..4 { + r.w32(reg::INT_STATUS, int::KEYBOARD); + assert_eq!(r.r32(reg::INT_STATUS) & int::KEYBOARD, 0); + } + let all = VALID[0] | VALID[1] | VALID[2]; + assert_eq!(r.r32(K_RX) & all, all); + assert_ne!(r.r32(reg::INT_STATUS) & int::KEYBOARD, 0, "the resumed keyboard raises it again"); + r.w32(reg::INT_STATUS, int::KEYBOARD); + assert_eq!(r.r32(K_RX) & all, VALID[2] | VALID[1]); + assert_eq!(r.r32(reg::INT_STATUS) & int::KEYBOARD, 0, "nothing left, nothing asserts it"); +} + +/// Ports sn0/ioc3/ps2-drain-then-acknowledge-takes-one: OpenBSD reads until +/// the register comes back empty and acknowledges afterwards, so a whole +/// backlog drains in one pass and one acknowledge ends it. +#[test] +fn the_ps2_drain_then_one_acknowledge_ends_the_backlog() { + let r = Rig::new(); + five_keys_behind_the_clamp(&r); + let (mut reads, mut valid) = (0, 0); + loop { + let v = r.r32(K_RX); + valid += VALID.iter().filter(|&&f| v & f != 0).count(); + reads += 1; + assert!(reads <= 4); + assert_ne!(r.r32(reg::INT_STATUS) & int::KEYBOARD, 0, "pending until the guest answers it"); + if v == 0 { + break; + } + } + assert_eq!((valid, reads), (5, 3), "3, 2, then empty"); + r.w32(reg::INT_STATUS, int::KEYBOARD); + assert_eq!(r.r32(reg::INT_STATUS) & int::KEYBOARD, 0); + assert_eq!(r.r32(K_RX), 0); + assert_eq!(r.r32(reg::INT_STATUS) & int::KEYBOARD, 0); +} + +/// Ports sn0/ioc3/ps2-clamp-after-one-splits-a-reply: clamping after one +/// character splits a reset reply, the acknowledge first and the self-test +/// byte next, each alone in the oldest slot. +#[test] +fn clamp_after_one_splits_a_reset_reply() { + let r = Rig::new(); + r.w32(CTRL, ctrl::K_CLAMP_1); + r.w32(K_TX, 0xff); + assert_eq!(r.r32(K_RX), 0x80fa_0000); + assert_ne!(r.r32(reg::INT_STATUS) & int::KEYBOARD, 0); + assert_eq!(r.r32(K_RX), 0x80aa_0000); + assert_eq!(r.r32(K_RX), 0); +} + #[test] fn the_ps2_receive_overflows_without_a_clamp() { let r = Rig::new(); @@ -550,7 +634,7 @@ fn the_ps2_receive_overflows_without_a_clamp() { let _ = r.r32(K_RX); type_keys(&r); let v = r.r32(K_RX); - assert_ne!(v & KBC_RX_OVERFLOW, 0, "a fourth character was lost"); + assert_ne!(v & RX_OVERFLOW, 0, "a fourth character was lost"); } #[test] @@ -567,7 +651,7 @@ fn the_ps2_control_masks_writes() { r.w32(CTRL, 0xffff_ffff); assert_eq!( r.r32(CTRL), - 0x0000_ffff | K_TO_EN | M_TO_EN | KBC_K_CLAMP_1 | KBC_M_CLAMP_1 | KBC_K_CLAMP_3 | KBC_M_CLAMP_3 + 0x0000_ffff | K_TO_EN | M_TO_EN | ctrl::K_CLAMP_1 | ctrl::M_CLAMP_1 | ctrl::K_CLAMP_3 | ctrl::M_CLAMP_3 ); r.c.reset(); assert_eq!(r.r32(CTRL), 0); @@ -682,6 +766,7 @@ fn ring_dma_loops_back_through_port_a() { put(&r, 0x2000, &[b'a', b'b', b'c', b'd', 0x40, 0x40, 0x40, 0x40]); r.w32(A, sdma_ctl::ENABLE); r.w32(A + sdma_off::TX_PRODUCE, 8); + r.line(4); assert_eq!(r.r32(A + sdma_off::TX_CONSUME), 8, "the engine took the entry"); assert_eq!(r.r32(A + sdma_off::RX_PRODUCE) & 0xfff, 8, "and posted one back"); let e = get(&r, 0x3000, 8); @@ -702,6 +787,7 @@ fn ring_dma_runs_on_port_b_too() { put(&r, 0x4000, &[b'w', b'x', b'y', b'z', 0x40, 0x40, 0x40, 0x40]); r.w32(B, sdma_ctl::ENABLE); r.w32(B + sdma_off::TX_PRODUCE, 8); + r.line(4); assert_eq!(r.r32(B + sdma_off::TX_CONSUME), 8); assert_eq!(&get(&r, 0x5000, 4), b"wxyz"); } @@ -716,6 +802,7 @@ fn flow_control_in_loopback_reports_cts() { put(&r, 0x2000, &[b'a', b'b', b'c', b'd', 0x40, 0x40, 0x40, 0x40]); r.w32(A, sdma_ctl::ENABLE | sdma_ctl::FLOW_CONTROL | 32); r.w32(A + sdma_off::TX_PRODUCE, 8); + r.line(4); let e = get(&r, 0x3000, 8); assert_eq!(e[4] & 0xdf, 0xd0, "data valid, modem valid, clear to send"); } @@ -732,6 +819,7 @@ fn an_empty_slot_ends_the_entry() { put(&r, 0x2008, &[b'e', b'f', b'g', b'h', 0x40, 0x40, 0x40, 0x00]); r.w32(A, sdma_ctl::ENABLE); r.w32(A + sdma_off::TX_PRODUCE, 16); + r.line(4); assert_eq!(r.r32(A + sdma_off::TX_CONSUME), 16); // a, then e f g: "aefg" in one receive entry. assert_eq!(&get(&r, 0x3000, 4), b"aefg"); @@ -1375,7 +1463,7 @@ fn running_rig() -> (Rig, Arc) { /// deadline briefly shows none; the callback holds `gate` throughout. fn timer_pending(r: &Rig, kind: usize) -> bool { let _idle = r.c.sh.gate.lock(); - r.c.sh.armed.lock()[kind].is_some() + r.c.sh.timers.is_armed(kind) } /// Poll `cond` for up to two seconds. @@ -1507,7 +1595,7 @@ fn stop_joins_the_threads_and_removes_the_timers() { r.c.stop(); assert!(!r.c.is_running()); assert!(r.c.threads.lock().is_empty()); - assert!(r.c.sh.armed.lock().iter().all(|a| a.is_none())); + assert!(r.c.sh.timers.none_armed()); // Typed input waits in the backend while stopped. r.wire[0].inp.lock().unwrap().extend(b"x"); std::thread::sleep(Duration::from_millis(20)); @@ -1636,3 +1724,82 @@ fn a_short_buffer_ram_is_refused() { chip.insert("ssram".into(), toml::Value::Array(Vec::new())); assert!(c.load_state(&v).unwrap_err().contains("ssram")); } + +/// The Octane keeps the IOC3's PCI header across a chip reset; only the +/// Fuel population resets it along with the chip. +#[test] +fn an_octane_chip_reset_keeps_the_pci_header() { + let r = Rig::new(); + let cfg = r.c.config(); + cfg.write(1, 0xffff_0146); + cfg.write(3, 0xffff_ffff); + r.c.reset(); + assert_eq!(cfg.read(1), 0x0200_0146); + assert_eq!(cfg.read(3), 0xffff); +} + +/// Fuel's IOC3 has its own header write policy and resets the header with +/// the chip, reaches the flat Timekeeper on byte-bus device 0, leaves the +/// Dallas clock ports unfitted, and has no PHY reset on GPIO 5. +#[test] +fn the_fuel_population_keeps_its_clock_and_header_policy_across_reset() { + let r = Rig::with(Ioc3::new(Ioc3Config::fuel(MAC))); + let clock = Arc::new(crate::dev::timekeeper::Timekeeper::fuel_at(2026, 1, 1, 0, 0, 0).unwrap()); + r.c.attach_timekeeper(clock.clone()); + let cfg = r.c.config(); + cfg.write(1, 0xffff_0146); + cfg.write(3, 0xffff_ffff); + assert_eq!(cfg.read(1), 0x0280_0146); + assert_eq!(cfg.read(3), 0x0000_ff00); + let base = reg::BYTEBUS_DEV0; + r.w8(base + 0x7f8, 0xa5); + r.c.reset(); + clock.power_on(); + assert_eq!(r.r8(base + 0x7fff), 0x58); + assert_eq!(r.r8(base + 0x7f8), 0xa5); + assert_eq!(cfg.read(1), 0x0280_0000); + assert_eq!(cfg.read(3), 0); + // Fuel selects the flat clock, leaving indexed Dallas ports unfitted. + r.w8(reg::BYTEBUS_DEV1, 0x40); + r.w8(reg::BYTEBUS_DEV2, 0x55); + assert_eq!(r.r8(reg::BYTEBUS_DEV2), 0); + r.c.attach_eth(crate::dev::ioc3_eth::Ioc3Eth::new_fuel(MAC)); + use crate::dev::ioc3_eth::{bits::MII_READ, reg::{MII_CONTROL, MII_DATA_R, MII_DATA_W}}; + r.w32(MII_DATA_W, 7); + r.w32(MII_CONTROL, (1 << 5) | 0x19); + r.w32(reg::PIN_CONTROL_SET, 1 << 5); + r.w32(reg::PIN_REG_FIRST + 5 * 4, 0); + r.w32(reg::PIN_DATA, 1 << 5); + r.w32(MII_CONTROL, MII_READ | (7 << 5) | 3); + assert_eq!(r.r32(MII_DATA_R), 0x5c10, "Fuel GPIO5 has no PHY reset connection"); +} + +/// Fuel's IOC3 fits the keyboard and mouse controller, as the Octane's does +/// (OpenBSD configures it through the onboard IOC3's keyboard controller). A +/// key pushed from the host arrives in K_RX through the IOC3 registers and +/// latches the keyboard interrupt, and a mouse report arrives in M_RX; both +/// survive a chip reset. +#[test] +fn the_fuel_ioc3_delivers_keyboard_and_mouse_traffic_across_reset() { + let r = Rig::with(Ioc3::new(Ioc3Config::fuel(MAC))); + for _ in 0..2 { + r.c.reset(); + r.c.ps2().power_on(); + r.w32(K_TX, 0xff); + let v = r.r32(K_RX); + assert_ne!(v & VALID[2], 0, "the keyboard answers a reset"); + assert_eq!((v >> 16) & 0xff, 0xfa, "with an ACK"); + r.w32(M_TX, 0xf2); + let v = r.r32(M_RX); + assert_ne!(v & (1 << 27), 0, "the mouse answers from its own port"); + // Enable scanning, then inject a key from the host. + r.w32(K_TX, 0xf4); + while r.r32(K_RX) & VALID[2] != 0 {} + r.w32(reg::INT_STATUS, int::KEYBOARD); + r.c.ps2().push_kb(winit::keyboard::KeyCode::KeyA, true); + assert_ne!(r.r32(reg::INT_STATUS) & int::KEYBOARD, 0, "a key latches the interrupt"); + let v = r.r32(K_RX); + assert_ne!(v & VALID[2], 0, "and arrives in K_RX"); + assert_eq!(v & (1 << 27), 0, "from the keyboard"); + } +} diff --git a/src/dev/ioc4.rs b/src/dev/ioc4.rs new file mode 100644 index 000000000..0674436ab --- /dev/null +++ b/src/dev/ioc4.rs @@ -0,0 +1,1011 @@ +//! SGI IOC4 base I/O controller, revision 0x53, as the IO9 board fits it. +//! +//! One PCI function with a 1 MiB memory BAR. Behind it: four 16550-compatible +//! serial ports with a ring-DMA engine each, a local interrupt controller for +//! them, the registers that capture a failed ring transfer, the UART clock +//! select, the ATA channel's DMA readback word the PROM's diagnostic probes, +//! and the timekeeping SRAM on byte-bus select 0 when the board fits one +//! ([`Ds1744`]), and the PS/2 keyboard and mouse controller the IO9 always +//! carries ([`Ioc3Kbc`], the block the IOC3 has too). The ATA channel, the +//! GPIO pins and the other byte-bus selects are not modelled; an access to +//! them reads zero or is dropped, and is logged. +//! +//! Behaviour follows the public Linux and OpenBSD drivers; where the model +//! approximates, the comment says so. +//! +//! # One interrupt output +//! +//! Everything the chip reports ORs into the PCI interrupt pin: the serial +//! status word (eight bits a port) and the other word (ATA, DMA errors, +//! keyboard), each behind its own enable. UART pass-through, transmit-ring- +//! empty, receive-ring-full and receive-high-water follow their source's level and ignore a +//! write-one-to-clear; the rest latch until acknowledged. +//! +//! # Byte order +//! +//! IOC4 orders its own bytes. Its slot is self-ordered: the bridge carries the +//! processor's bytes in address order and leaves the address alone, so a +//! byte access to UART register `r` of port `n` arrives at `0x380 + 8n + r`. +//! A PCI function's BAR value is its bytes read little-endian, and IOC4 reads +//! them big-endian, so the chip takes a 32-bit value byte-reversed: the +//! processor's value is the register's value, as QEMU's big-endian IOC4 +//! region reads it. Control registers and the +//! keyboard block take 32-bit accesses only; a UART window or the clock takes +//! bytes only. Anything else is refused (reads zero, writes dropped). Which widths the silicon accepts is not +//! known. +//! +//! # What is approximated here +//! +//! * The reset values, the BAR size and the PCI class are the emulator's +//! choices. +//! * The serial ring engine is the one the IOC3 uses (`ioc_serial_dma.rs`) and +//! carries its approximations: timers run on host time and receive-ring-full +//! is a level that follows a held entry. After a failed ring transfer the port +//! halts until a channel reset. +//! * Register 0x20 reads the DMA-idle bit unless a transfer is under way. +//! Transfers complete inside the access that starts them, so software +//! always finds the engine idle. +//! * The keyboard block's status bits (write-pending, line control, line +//! levels and idle flags, mask `0xc33f`) read clear and drop writes, as the +//! QEMU adapter does. Its port order, write-pending timing, and whether +//! the interrupt is a pulse per character or a level are not established, +//! and the model follows QEMU: a latch per arriving character, which an +//! acknowledge racing an arrival can lose. The IOC3-only receive flags are +//! masked out of the receive registers. The ports are always fitted, as the +//! IO9's are in QEMU. +//! * The ATA readback word at 0x148 is a plain 32-bit latch with no ATA +//! behind it. Only boards that ask for it have it. +//! * The PCI status register reads 0x0220 (66 MHz capable, medium DEVSEL) on +//! boards that ask for it: a value taken from another revision of the chip, +//! not measured on this one. +//! +//! Names follow QEMU's: `sgi-ioc4.c` has the same registers under the same +//! names, its `sgi_ioc4_sdma_*` callbacks are the chip's `SerialDmaHost`, and +//! the ring engine is `ioc_serial_dma.rs` (QEMU's `sgi-ioc-serial-dma.c`). +//! +//! Ported from hw/misc/sgi-ioc4.c in jamesbraid/qemu-sgi-origami. + +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, OnceLock, Weak}; +use std::time::{Duration, Instant}; + +use parking_lot::Mutex; + +use crate::dev::ds1744::Ds1744; +use crate::dev::ioc3_kbc::{self, Ioc3Kbc, KbcLine, KbcOwner}; +use crate::dev::ps2::Ps2Controller; +use crate::dev::ioc_serial_dma::{self, off as sdma_off, SerialDmaUart, SerialDmaHost, SerialDmaParams, SerialDma, SerialDmaSource}; +use crate::dev::ns16550::Ns16550; +use crate::dev::serial_host::SerialBackend; +use crate::dev::deadline_timers::DeadlineTimers; +use crate::hptimer::{TimerManager, TimerReturn}; +use crate::irq::IrqLine; +use crate::pci::{Bar, PciBusDevice, PciConfig, PciConfigPolicy, PciDma, PCI_INTA}; +use crate::traits::{BusDevice, Device, Resettable}; + +pub const VENDOR_ID: u16 = 0x10a9; +pub const DEVICE_ID: u16 = 0x100a; +pub const REVISION: u8 = 0x53; +/// PCI class, subclass and interface: "other", an emulator default. +const PCI_CLASS: u32 = 0x00_ff_00; +pub const BAR_SIZE: u32 = 1 << 20; +/// Status of a 66 MHz capable function with medium DEVSEL timing. +const PCI_STATUS_COMPAT: u16 = 0x0220; +pub const PORTS: usize = 4; + +/// Register offsets within the BAR. +pub mod reg { + /// Captured address of the first failed ring transfer, low and high. + pub const ERROR_LOW: u32 = 0x00; + pub const ERROR_HIGH: u32 = 0x04; + pub const SERIAL_PENDING: u32 = 0x08; + pub const OTHER_PENDING: u32 = 0x0c; + pub const SERIAL_ENABLE_SET: u32 = 0x10; + pub const OTHER_ENABLE_SET: u32 = 0x14; + pub const SERIAL_ENABLE_CLR: u32 = 0x18; + pub const OTHER_ENABLE_CLR: u32 = 0x1c; + pub const SERIAL_CONTROL: u32 = 0x20; + pub const MISC_CONTROL: u32 = 0x24; + /// A word of the ATA channel's DMA block the PROM writes and reads back. + pub const ATA_READBACK: u32 = 0x148; + /// Ring bases, a low and a high word for each pair of ports. + pub const SDMA_BASE: u32 = 0x300; + pub const SDMA_PORT: u32 = 0x310; + pub const SDMA_PORT_STRIDE: u32 = super::sdma_off::REGS * 4; + /// The first 16550 window; eight bytes a port. + pub const UART: u32 = 0x380; + pub const UART_STRIDE: u32 = 8; + /// Byte-bus select 0: the timekeeping SRAM, whose last eight bytes are + /// the clock (at 0x87ff8). + pub const CLOCK: u32 = 0x80000; + pub const CLOCK_SIZE: u32 = crate::dev::ds1744::SIZE as u32; + /// The PS/2 controller's five registers. + pub const KBC: u32 = 0x200; +} + +/// Bits of the serial status word, eight to a port. +pub mod int { + pub const PORT_STRIDE: u32 = 8; + pub const PASSTHROUGH: u32 = 6; + pub const fn serial(port: usize, src: u32) -> u32 { + 1 << (port as u32 * PORT_STRIDE + src) + } + pub const fn pass(port: usize) -> u32 { + serial(port, PASSTHROUGH) + } + /// The other word: a serial DMA error, one bit a port from bit 2. + pub const fn dma_error(port: usize) -> u32 { + 1 << (port as u32 + 2) + } +} + +/// Bits of the captured error address. +mod err { + pub const VALID: u32 = 1; + pub const MULTIPLE: u32 = 1 << 5; + pub const REQUESTER_SHIFT: u32 = 1; + /// The low six address bits are not captured. + pub const ADDRESS_MASK: u64 = 0xffff_ffc0; +} + +/// Other word bit 6: a keyboard or mouse character arrived. +const OTHER_KEYBOARD: u32 = 0x40; +/// Keyboard control bits that report state on the controller: write-pending +/// (0, 1), the last write's line-control bit (2, 3), the data and clock line +/// levels (4, 5, 8, 9) and the state machines' idle flags (14, 15). The model +/// does not simulate the lines, so they all read clear and a write to them is +/// dropped. +const KBC_STATUS: u32 = 0x0000_c33f; +/// The receive registers carry no IOC3-only flags: bits 24 to 28 (per-slot +/// errors, source and overflow) read clear. +const KBC_RX_MASK: u32 = 0xe0ff_ffff; + +/// Serial control bits the model acts on: the threshold, drain, enable, +/// pause and reset. The rest are stored and read back. +const SSCR_MODELLED: u32 = 0x1ff | (1 << 27) | (1 << 28) | (1 << 29) | (1 << 31); +/// The receive consume index also holds the timer arm in bit 31. +const SRCIR_TIMER_ARM: u32 = 1 << 31; +/// The ring size select in a pair base's low word. +const RING_4K: u32 = 1; + +/// Register 0x20: the DMA engines are idle. +const SERIAL_CONTROL_IDLE: u32 = 0x80; + +/// IOC4's engine halts a port on a failed transfer and does not define the +/// packer's held count. +const IOC4_SDMA_PARAMS: SerialDmaParams = SerialDmaParams { held_counts: false, halt_on_dma_error: true }; + +/// The UART input clock the board feeds the chip. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Ioc4Clock { + Mhz33, + Mhz66, +} + +impl Ioc4Clock { + pub fn hz(self) -> u32 { + match self { + Ioc4Clock::Mhz33 => 33_333_333, + Ioc4Clock::Mhz66 => 66_666_667, + } + } +} + +/// What the board makes of the chip. +#[derive(Clone, Copy, Debug)] +pub struct Ioc4Config { + pub clock: Ioc4Clock, + /// Report PCI status 0x0220. + pub pci_status_compat: bool, + /// Latch the ATA DMA readback word. + pub ata_readback: bool, +} + +impl Ioc4Config { + /// The chip with nothing a board asks for beyond its clock. + pub fn bare(clock: Ioc4Clock) -> Self { + Self { clock, pci_status_compat: false, ata_readback: false } + } + + /// The Tezro's IO9: a 66.67 MHz clock, and the status and ATA readback + /// words its PROM expects. + pub fn tezro() -> Self { + Self { clock: Ioc4Clock::Mhz66, pci_status_compat: true, ata_readback: true } + } +} + +/// A 32-bit register in the ring-engine block. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum SdmaReg { + Base { pair: usize, high: bool }, + Port { port: usize, off: u32 }, +} + +/// Ring bases sit at 0x300 (low then high word, a pair each), then seven +/// registers a port from 0x310. +fn sdma_decode(a: u32) -> Option { + if a & 3 != 0 { + return None; + } + if (reg::SDMA_BASE..reg::SDMA_PORT).contains(&a) { + let n = a - reg::SDMA_BASE; + return Some(SdmaReg::Base { pair: (n / 8) as usize, high: n & 4 != 0 }); + } + let end = reg::SDMA_PORT + PORTS as u32 * reg::SDMA_PORT_STRIDE; + if (reg::SDMA_PORT..end).contains(&a) { + let n = a - reg::SDMA_PORT; + return Some(SdmaReg::Port { port: (n / reg::SDMA_PORT_STRIDE) as usize, off: n % reg::SDMA_PORT_STRIDE }); + } + None +} + +/// Where a byte access to the UART windows lands. +fn uart_decode(a: u32) -> Option<(usize, u32)> { + let end = reg::UART + PORTS as u32 * reg::UART_STRIDE; + (reg::UART..end).contains(&a).then(|| (((a - reg::UART) / reg::UART_STRIDE) as usize, (a - reg::UART) % reg::UART_STRIDE)) +} + +fn in_kbc(a: u32) -> bool { + (reg::KBC..reg::KBC + ioc3_kbc::reg::MMIO_SIZE).contains(&a) +} + +fn in_clock(a: u32) -> bool { + (reg::CLOCK..reg::CLOCK + reg::CLOCK_SIZE).contains(&a) +} + +struct St { + cfg: Ioc4Config, + serial_pending: u32, + serial_enable: u32, + other_pending: u32, + other_enable: u32, + /// Sources that follow a level, in the serial word's layout. + level: u32, + error_low: u32, + error_high: u32, + ata_readback: u32, + uart: [Ns16550; PORTS], + sdma: SerialDma, + kbc: Ioc3Kbc, + ps2: Arc, + dma: Option>, +} + +impl St { + fn new(cfg: Ioc4Config, ps2: Arc) -> St { + St { + cfg, + serial_pending: 0, + serial_enable: 0, + other_pending: 0, + other_enable: 0, + level: 0, + error_low: 0, + error_high: 0, + ata_readback: 0, + uart: std::array::from_fn(|_| Ns16550::new(cfg.clock.hz() / 16)), + sdma: SerialDma::new(PORTS, IOC4_SDMA_PARAMS), + // Every IO9 brings the keyboard and mouse out. + kbc: Ioc3Kbc::new(true), + ps2, + dma: None, + } + } + + /// A PCI reset: the interrupt state, the ports and the engines. + fn reset(&mut self) { + self.serial_pending = 0; + self.serial_enable = 0; + self.other_pending = 0; + self.other_enable = 0; + self.level = 0; + self.error_low = 0; + self.error_high = 0; + self.ata_readback = 0; + for u in &mut self.uart { + u.reset(); + } + self.sdma.reset(); + self.kbc.reset(); + self.ps2.power_on(); + } + + /// Move what the host's PS/2 model has sent into the holdings, and latch + /// the keyboard source if a character arrived. + fn kbc_fill(&mut self) { + let ps2 = self.ps2.clone(); + if self.kbc.fill_all(&ps2) { + self.other_pending |= OTHER_KEYBOARD; + } + } + + /// A 32-bit read of the keyboard block at `off` in its window, as the + /// adapter presents it: no write-pending or other status, and none of + /// the IOC3-only receive flags. + fn kbc_read(&mut self, off: u32) -> u32 { + let ps2 = self.ps2.clone(); + let (raw, arrived) = self.kbc.read(off, &ps2); + if arrived { + self.other_pending |= OTHER_KEYBOARD; + } + if off == ioc3_kbc::reg::CONTROL { raw & !KBC_STATUS } else { raw & KBC_RX_MASK } + } + + fn kbc_write(&mut self, off: u32, val: u32) { + let mut val = val; + if off == ioc3_kbc::reg::CONTROL { + // The status bits read clear; a write that differs is a guest + // error and is dropped. + if val & KBC_STATUS != 0 { + crate::dlog!(crate::devlog::LogModule::Ioc, "IOC4 write to read-only KBC status bits {:#x} ignored", val & KBC_STATUS); + } + val &= !KBC_STATUS; + } + let ps2 = self.ps2.clone(); + if self.kbc.write(off, val, &ps2) { + self.other_pending |= OTHER_KEYBOARD; + } + } + + /// The PCI interrupt pin: any enabled source. + fn line_level(&self) -> bool { + (self.serial_pending | self.level) & self.serial_enable != 0 || self.other_pending & self.other_enable != 0 + } + + /// The host-time deadlines the engines hold, for the timers. + fn deadlines(&self) -> Deadlines { + let mut d = [None; KINDS]; + for p in 0..PORTS { + d[K_RX + p] = self.sdma.rx_deadline(p); + d[K_TX_DELAY + p] = self.sdma.tx_delay_deadline(p); + d[K_LOOP + p] = self.sdma.loop_deadline(p); + } + d + } + + fn sdma_engine_split(&mut self) -> (&mut SerialDma, Ioc4SerialDmaHost<'_>) { + let host = Ioc4SerialDmaHost { + uart: &mut self.uart, + serial_pending: &mut self.serial_pending, + level: &mut self.level, + other_pending: &mut self.other_pending, + error_low: &mut self.error_low, + error_high: &mut self.error_high, + dma: self.dma.as_deref(), + }; + (&mut self.sdma, host) + } + + fn sdma_run(&mut self, p: usize) { + let (sdma, mut host) = self.sdma_engine_split(); + sdma.run(p, &mut host); + } + + fn sdma_catch_up(&mut self, now: Instant) { + let (sdma, mut host) = self.sdma_engine_split(); + sdma.catch_up(now, &mut host); + } + + // ----- the 16550s ----- + + /// Keep the pass-through source on the port's interrupt level, and let + /// the engine move: a ring driver receives with the 16550's own + /// interrupts disabled. + fn uart_changed(&mut self, p: usize) { + let level = self.uart[p].irq(); + set_level_bit(&mut self.level, int::pass(p), level); + self.sdma_run(p); + } + + /// Move host input into the port, retry a byte the host end refused and + /// let the receive timeout run out. Returns whether anything changed. + fn uart_pump(&mut self, p: usize) -> bool { + let now = Instant::now(); + let u = &mut self.uart[p]; + let moved = u.pump(); + let sent = u.poll_tx(now); + u.check_timeout(now) || moved || sent + } + + fn uart_read(&mut self, p: usize, r: u32) -> u8 { + let v = self.uart[p].read(r); + self.uart_changed(p); + v + } + + fn uart_write(&mut self, p: usize, r: u32, v: u8) { + if r == 0 && !self.uart[p].dlab() && self.sdma.control(p) & ioc_serial_dma::ctl::ENABLE != 0 { + crate::dlog!(crate::devlog::LogModule::Ioc, "IOC4 PIO transmit on port {p} while DMA is enabled"); + } + self.uart[p].write(r, v); + self.uart_changed(p); + } + + // ----- mode-switch diagnostics ----- + + /// Report the DMA mode switches and the writes the model takes but does + /// not act on. They are reported, not refused: the UART orders PIO and + /// ring DMA itself, and the write goes ahead. + fn check_port_write(&self, p: usize, off: u32, v: u32) { + let log = |what: std::fmt::Arguments| crate::dlog!(crate::devlog::LogModule::Ioc, "IOC4 {what}"); + match off { + sdma_off::CONTROL => { + if v & !SSCR_MODELLED != 0 { + log(format_args!("serial control bits {:#010x} on port {p} are stored but not modelled", v & !SSCR_MODELLED)); + } + self.control_check(p, v); + } + sdma_off::TX_PRODUCE | sdma_off::RX_CONSUME => { + let size = if self.sdma.base_read(p / 2, false) & RING_4K != 0 { 0x1000 } else { 0x400 }; + let keep = (size - 8) | if off == sdma_off::RX_CONSUME { SRCIR_TIMER_ARM } else { 0 }; + if v & !keep != 0 { + let what = if off == sdma_off::TX_PRODUCE { "TX producer" } else { "RX consumer" }; + log(format_args!("{what} {v:#010x} on port {p} has bits outside the pointer field")); + } + } + sdma_off::RX_TIMER if v & !0xfff != 0 => { + log(format_args!("unsupported RX timer bits {:#010x} on port {p}", v & !0xfff)); + } + _ => {} + } + } + + /// Enabling DMA over a base misaligned for its rings, or over PIO data in + /// the UART's FIFOs, and disabling it with ring work pending. + fn control_check(&self, p: usize, v: u32) { + use ioc_serial_dma::ctl; + let old_on = self.sdma.control(p) & ctl::ENABLE != 0; + let want_on = v & ctl::ENABLE != 0; + if v & ctl::RESET != 0 || old_on == want_on { + return; + } + let log = |what: std::fmt::Arguments| crate::dlog!(crate::devlog::LogModule::Ioc, "IOC4 {what}"); + let pair = p / 2; + if want_on { + let base = u64::from(self.sdma.base_read(pair, true)) << 32 | u64::from(self.sdma.base_read(pair, false) & !RING_4K); + let size: u64 = if self.sdma.base_read(pair, false) & RING_4K != 0 { 0x1000 } else { 0x400 }; + if base & (size * 4 - 1) != 0 { + log(format_args!("DMA enabled on port {p} with pair base {base:#x} misaligned for its rings")); + } + if self.uart[p].rx_pending() != 0 || self.uart[p].tx_pending() { + log(format_args!("DMA enabled on port {p} with PIO data in the UART's FIFOs")); + } + } else if !self.sdma.port_quiescent(p) { + log(format_args!("DMA disabled on port {p} with ring work pending")); + } + } + + // ----- control registers ----- + + fn read32(&mut self, a: u32) -> u32 { + self.sdma_catch_up(Instant::now()); + if let Some(r) = sdma_decode(a) { + let (sdma, mut host) = self.sdma_engine_split(); + return match r { + SdmaReg::Base { pair, high } => sdma.base_read(pair, high), + SdmaReg::Port { port, off } => sdma.port_read(port, off, &mut host), + }; + } + match a { + reg::ERROR_LOW => self.error_low, + reg::ERROR_HIGH => self.error_high, + reg::SERIAL_PENDING => self.serial_pending | self.level, + reg::OTHER_PENDING => self.other_pending, + reg::SERIAL_ENABLE_SET => self.serial_enable, + reg::OTHER_ENABLE_SET => self.other_enable, + reg::SERIAL_CONTROL => if self.sdma.idle() { SERIAL_CONTROL_IDLE } else { 0 }, + reg::MISC_CONTROL => u32::from(self.cfg.clock == Ioc4Clock::Mhz66), + reg::ATA_READBACK if self.cfg.ata_readback => self.ata_readback, + _ => { + crate::dlog!(crate::devlog::LogModule::Ioc, "IOC4 read of unsupported register {a:#x}"); + 0 + } + } + } + + fn write32(&mut self, a: u32, v: u32) { + self.sdma_catch_up(Instant::now()); + if let Some(r) = sdma_decode(a) { + match r { + SdmaReg::Base { pair, high } => { + if !self.sdma.pair_quiescent(pair) { + crate::dlog!(crate::devlog::LogModule::Ioc, "IOC4 DMA base of pair {pair} changed while it is active"); + } + self.sdma.base_write(pair, high, v); + } + SdmaReg::Port { port, off } => { + self.check_port_write(port, off, v); + let (sdma, mut host) = self.sdma_engine_split(); + sdma.port_write(port, off, v, &mut host); + } + } + return; + } + match a { + // Acknowledge: write one to clear. Level sources ignore it. + reg::SERIAL_PENDING => self.serial_pending &= !v, + reg::OTHER_PENDING => self.other_pending &= !v, + reg::SERIAL_ENABLE_SET => self.serial_enable |= v, + reg::OTHER_ENABLE_SET => self.other_enable |= v, + reg::SERIAL_ENABLE_CLR => self.serial_enable &= !v, + reg::OTHER_ENABLE_CLR => self.other_enable &= !v, + reg::ATA_READBACK if self.cfg.ata_readback => self.ata_readback = v, + reg::ERROR_LOW | reg::ERROR_HIGH | reg::SERIAL_CONTROL | reg::MISC_CONTROL => { + crate::dlog!(crate::devlog::LogModule::Ioc, "IOC4 write {v:#x} to read-only register {a:#x}"); + } + _ => crate::dlog!(crate::devlog::LogModule::Ioc, "IOC4 write {v:#x} to unsupported register {a:#x}"), + } + } +} + +fn set_level_bit(word: &mut u32, mask: u32, level: bool) { + if level { + *word |= mask; + } else { + *word &= !mask; + } +} + +/// IOC4 as the ring engine sees it: four 16550s, the serial status word, +/// the error capture registers and the bus master. +struct Ioc4SerialDmaHost<'a> { + uart: &'a mut [Ns16550; PORTS], + serial_pending: &'a mut u32, + level: &'a mut u32, + other_pending: &'a mut u32, + error_low: &'a mut u32, + error_high: &'a mut u32, + dma: Option<&'a dyn PciDma>, +} + +impl SerialDmaHost for Ioc4SerialDmaHost<'_> { + fn uart(&mut self, port: usize) -> &mut dyn SerialDmaUart { + &mut self.uart[port] + } + + fn raise(&mut self, port: usize, src: SerialDmaSource) { + *self.serial_pending |= int::serial(port, src as u32); + } + + fn set_level(&mut self, port: usize, src: SerialDmaSource, level: bool) { + set_level_bit(self.level, int::serial(port, src as u32), level); + } + + /// The engines always reach their ports. + fn reachable(&self, _port: usize) -> bool { + true + } + + fn uart_line_changed(&mut self, port: usize) { + let level = self.uart[port].irq(); + set_level_bit(self.level, int::pass(port), level); + } + + fn dma_read(&self, addr: u64, buf: &mut [u8]) -> bool { + self.dma.is_some_and(|d| d.dma_read(addr, buf).is_ok()) + } + + fn dma_write(&self, addr: u64, data: &[u8]) -> bool { + self.dma.is_some_and(|d| d.dma_write(addr, data).is_ok()) + } + + /// The first failure captures its address and its requester: twice the + /// port for a transmit fetch, one more for a receive post. Later ones + /// only set the multiple-error bit. The port's source is in the other + /// word. + fn dma_error(&mut self, port: usize, rx: bool, addr: u64) { + if *self.error_low & err::VALID == 0 { + let requester = (2 * port + usize::from(rx)) as u32; + *self.error_low = (addr & err::ADDRESS_MASK) as u32 | requester << err::REQUESTER_SHIFT | err::VALID; + *self.error_high = (addr >> 32) as u32; + } else { + *self.error_low |= err::MULTIPLE; + } + *self.other_pending |= int::dma_error(port); + crate::dlog!( + crate::devlog::LogModule::Ioc, + "IOC4 serial DMA {} failed for port {port} at {addr:#x}", + if rx { "write" } else { "read" } + ); + } +} + +// --------------------------------------------------------------------------- +// The device +// --------------------------------------------------------------------------- + +/// How long a serial receive thread sleeps when its backend has nothing: +/// `SerialBackend` has no blocking read to wait on. +const RX_IDLE: Duration = Duration::from_millis(2); + +/// The deadlines the chip keeps on host time, one timer slot each. +const K_RX: usize = 0; +const K_TX_DELAY: usize = PORTS; +const K_LOOP: usize = 2 * PORTS; +const KINDS: usize = 3 * PORTS; +type Deadlines = [Option; KINDS]; + +/// Lock order: `gate`, then `line_order`, then `st`; `armed` and `line` are +/// leaves. The engines move their rings with `st` held: the DMA goes down +/// through the bridge, which never calls back into the chip. +struct Shared { + pci: PciConfig, + st: Mutex, + /// PCI interrupt pin A. + line: Mutex>, + /// What the line was last driven to. + line_level: AtomicBool, + /// Serialises driving the line; see [`Shared::sync_line`]. + line_order: Mutex<()>, + me: Weak, + timers: DeadlineTimers, + /// A PS/2 character arrived while an access held the chip. + kbc_dirty: AtomicBool, + /// Between `start` and `stop`. + running: AtomicBool, + /// Held by a timer callback while it runs, so `stop` can wait one out. + gate: Mutex<()>, +} + +impl KbcOwner for Shared { + /// A device raised its line. If an access holds the chip (possibly this + /// very thread, from inside a transmit), that access picks the character + /// up when it finishes: the flag is set before the lock is tried, so + /// either this thread gets the lock or the holder sees the flag. + fn kbc_poke(&self) { + self.kbc_dirty.store(true, Ordering::SeqCst); + if let Some(mut st) = self.st.try_lock() { + if self.kbc_dirty.swap(false, Ordering::SeqCst) { + st.kbc_fill(); + } + drop(st); + self.sync_line(); + } + } +} + +impl Shared { + /// Put the interrupt line at the level the chip's state gives, then pick + /// up any PS/2 character that arrived while the chip was held (a poke + /// sets the flag before it tries the lock, so whoever holds the lock sees + /// it here, after releasing). + fn sync_line(&self) { + loop { + self.drive_line(); + if !self.kbc_dirty.swap(false, Ordering::SeqCst) { + break; + } + self.st.lock().kbc_fill(); + } + } + + /// The level is read after taking `line_order`, so whichever thread + /// drives last has seen the latest state. `st` is never held when this + /// is called. + fn drive_line(&self) { + let _order = self.line_order.lock(); + let level = self.st.lock().line_level(); + if self.line_level.swap(level, Ordering::SeqCst) != level { + if let Some(l) = self.line.lock().as_ref() { + l.set(level); + } + } + } + + /// Run `f` with the chip locked, then arm the timers for the deadlines it + /// left and drive the line from its state. + fn with_st(&self, f: impl FnOnce(&mut St) -> R) -> R { + let (r, want) = { + let mut st = self.st.lock(); + let r = f(&mut st); + (r, st.deadlines()) + }; + self.arm(want); + self.sync_line(); + r + } + + /// Arm a timer for each deadline the chip holds. + fn arm(&self, want: Deadlines) { + self.timers.arm(&want, &self.running, &self.me, |_| Duration::ZERO, Shared::fire); + } + + /// A deadline passed with no guest access to notice it. Runs on the timer + /// thread: short, no waiting beyond the chip's own locks. + fn fire(me: &Weak, kind: usize, due: Instant) -> TimerReturn { + let Some(sh) = me.upgrade() else { return TimerReturn::Delete }; + let _gate = sh.gate.lock(); + if !sh.running.load(Ordering::SeqCst) { + return TimerReturn::Delete; + } + sh.timers.fired(kind, due); + sh.with_st(|st| st.sdma_catch_up(Instant::now())); + TimerReturn::Delete + } + + /// Feed host input from serial backend `port` into its FIFO until the + /// chip stops. + fn rx_loop(me: Weak, port: usize) { + loop { + { + let Some(sh) = me.upgrade() else { return }; + if !sh.running.load(Ordering::SeqCst) { + return; + } + sh.with_st(|st| { + if st.uart_pump(port) { + st.uart_changed(port); + } + }); + } + std::thread::park_timeout(RX_IDLE); + } + } + + /// One synchronous pass of what the receive threads and timers do. + #[cfg(test)] + fn service(&self) { + self.with_st(|st| { + for p in 0..PORTS { + st.uart_pump(p); + st.uart_changed(p); + } + st.sdma_catch_up(Instant::now()); + for p in 0..PORTS { + st.sdma_run(p); + } + }); + } +} + +/// The IOC4 PCI function. +pub struct Ioc4 { + sh: Arc, + /// The DS1744 on byte-bus select 0, when the board fits one. + clock: OnceLock>, + /// The serial receive threads, joined in `stop`. + threads: Mutex>>, +} + +impl Ioc4 { + pub fn new(cfg: Ioc4Config) -> Ioc4 { + let sh = Arc::new_cyclic(|weak: &Weak| { + let line = IrqLine::new(Arc::new(KbcLine(weak.clone())), 0); + let ps2 = Arc::new(Ps2Controller::new(Some(line))); + // Host key and mouse events are accepted from the start (this + // only sets a flag); `stop` and `start` follow the chip's. + Device::start(&*ps2); + let bars = [Bar::Mem(BAR_SIZE), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused]; + let policy = PciConfigPolicy { + status: if cfg.pci_status_compat { PCI_STATUS_COMPAT } else { 0 }, + status_w1c_mask: 0, + command_mask: 0xffff, + ..Default::default() + }; + Shared { + pci: PciConfig::new_with_policy(VENDOR_ID, DEVICE_ID, PCI_CLASS, REVISION, bars, PCI_INTA, policy), + st: Mutex::new(St::new(cfg, ps2)), + line: Mutex::new(None), + line_level: AtomicBool::new(false), + line_order: Mutex::new(()), + kbc_dirty: AtomicBool::new(false), + me: weak.clone(), + timers: DeadlineTimers::new(), + running: AtomicBool::new(false), + gate: Mutex::new(()), + } + }); + Ioc4 { sh, clock: OnceLock::new(), threads: Mutex::new(Vec::new()) } + } + + /// Fit the timekeeping SRAM on byte-bus select 0. Its 32 KiB fill the + /// BAR from 0x80000 and its calendar sits in the last eight bytes. The + /// chip's reset does not touch it: it is memory with a battery. + pub fn attach_clock(&self, clock: Arc) { + assert!(self.clock.set(clock).is_ok(), "IOC4 clock attached more than once"); + } + + /// The timer manager for device deadlines. Set it before start. + pub fn set_timer_manager(&self, tm: Arc) { + self.sh.timers.set_manager(tm); + } + + /// Wire the host end of serial port `port` (0 is the console): a TCP + /// listener, a log tee, or nothing. + pub fn set_serial_backend(&self, port: usize, backend: Arc) { + self.sh.st.lock().uart[port].set_backend(backend); + } + + /// The PS/2 keyboard and mouse model, for the host side to push keys and + /// pointer motion into. + pub fn ps2(&self) -> Arc { + self.sh.st.lock().ps2.clone() + } + + /// A PCI reset of the chip. + pub fn reset(&self) { + self.sh.pci.reset(); + self.sh.with_st(St::reset); + } + + /// One synchronous pass of the work the receive threads and the timers + /// do between guest accesses. Tests use it to settle the chip without + /// waiting on a thread. + #[cfg(test)] + pub fn service(&self) { + self.sh.service(); + } + + /// Name the chip's long-lived locks in the lock registry, which holds + /// only a weak reference to them. + pub fn register_locks(&self) { + for (what, probe) in [ + ("st", (|s: &Shared| s.st.is_locked()) as fn(&Shared) -> bool), + ("line_order", |s| s.line_order.is_locked()), + ] { + let sh = Arc::downgrade(&self.sh); + crate::locks::register_lock(format!("ioc4::{what}"), move || sh.upgrade().is_some_and(|s| probe(&s))); + } + } + + /// Say that an access the chip does not take was refused. + fn refuse(what: &str, width: usize, off: u32) { + crate::dlog!(crate::devlog::LogModule::Ioc, "IOC4 {what} of width {width} at {off:#x} refused"); + } +} + +impl Device for Ioc4 { + fn step(&self, _cycles: u64) {} + + /// Start the serial receive threads and arm the timers for any deadline + /// the chip already holds. + fn start(&self) { + if self.sh.running.swap(true, Ordering::SeqCst) { + return; + } + Device::start(&*self.sh.st.lock().ps2); + let mut threads = self.threads.lock(); + for p in 0..PORTS { + let me = self.sh.me.clone(); + threads.push( + std::thread::Builder::new() + .name(format!("IOC4-RX-{p}")) + .spawn(move || Shared::rx_loop(me, p)) + .expect("spawn ioc4 receive thread"), + ); + } + drop(threads); + let want = self.sh.st.lock().deadlines(); + self.sh.arm(want); + } + + /// Remove the timers, wait out a callback in flight and join the threads. + /// Nothing touches memory or the line afterwards, until `start`. + fn stop(&self) { + if !self.sh.running.swap(false, Ordering::SeqCst) { + return; + } + self.sh.timers.cancel_all(); + drop(self.sh.gate.lock()); + let mut threads = self.threads.lock(); + for t in threads.iter() { + t.thread().unpark(); + } + for t in threads.drain(..) { + let _ = t.join(); + } + drop(threads); + Device::stop(&*self.sh.st.lock().ps2); + } + + fn is_running(&self) -> bool { + self.sh.running.load(Ordering::SeqCst) + } + + fn get_clock(&self) -> u64 { + 0 + } +} + +impl Resettable for Ioc4 { + fn power_on(&self) { + self.reset(); + } +} + +impl Drop for Ioc4 { + fn drop(&mut self) { + Device::stop(&*self); + } +} + +impl PciBusDevice for Ioc4 { + fn config(&self) -> &PciConfig { + &self.sh.pci + } + + fn bar_read(&self, bar: usize, off: u32, width: usize) -> u32 { + if bar != 0 || off >= BAR_SIZE { + return 0; + } + if let Some((p, r)) = uart_decode(off) { + if width != 1 { + Ioc4::refuse("UART read", width, off); + return 0; + } + return u32::from(self.sh.with_st(|st| st.uart_read(p, r))); + } + if let Some(clock) = self.clock.get().filter(|_| in_clock(off)) { + if width != 1 { + Ioc4::refuse("clock read", width, off); + return 0; + } + return u32::from(clock.read8(u64::from(off - reg::CLOCK)).data); + } + if in_kbc(off) { + if width != 4 || off & 3 != 0 { + Ioc4::refuse("KBC read", width, off); + return 0; + } + return self.sh.with_st(|st| st.kbc_read(off - reg::KBC)).swap_bytes(); + } + if width != 4 || off & 3 != 0 { + Ioc4::refuse("read", width, off); + return 0; + } + self.sh.with_st(|st| st.read32(off)).swap_bytes() + } + + fn bar_write(&self, bar: usize, off: u32, val: u32, width: usize) { + if bar != 0 || off >= BAR_SIZE { + return; + } + if let Some((p, r)) = uart_decode(off) { + if width != 1 { + Ioc4::refuse("UART write", width, off); + return; + } + self.sh.with_st(|st| st.uart_write(p, r, val as u8)); + return; + } + if let Some(clock) = self.clock.get().filter(|_| in_clock(off)) { + if width != 1 { + Ioc4::refuse("clock write", width, off); + return; + } + clock.write8(u64::from(off - reg::CLOCK), val as u8); + return; + } + if in_kbc(off) { + if width != 4 || off & 3 != 0 { + Ioc4::refuse("KBC write", width, off); + return; + } + self.sh.with_st(|st| st.kbc_write(off - reg::KBC, val.swap_bytes())); + return; + } + if width != 4 || off & 3 != 0 { + Ioc4::refuse("write", width, off); + return; + } + self.sh.with_st(|st| st.write32(off, val.swap_bytes())); + } + + /// The PCI interrupt pin: any enabled source. + fn connect_interrupt(&self, line: IrqLine) { + { + let _order = self.sh.line_order.lock(); + line.set(self.sh.line_level.load(Ordering::SeqCst)); + *self.sh.line.lock() = Some(line); + } + self.sh.sync_line(); + } + + fn connect_dma(&self, dma: Arc) { + self.sh.st.lock().dma = Some(dma); + } +} + +#[cfg(test)] +#[path = "ioc4_tests.rs"] +mod tests; diff --git a/src/dev/ioc4_tests.rs b/src/dev/ioc4_tests.rs new file mode 100644 index 000000000..0d5a39ded --- /dev/null +++ b/src/dev/ioc4_tests.rs @@ -0,0 +1,1757 @@ +//! IOC4 register-contract tests. +//! +//! Accesses go through `PciBusDevice` the way the bridge makes them: IOC4 +//! orders its own bytes, so a register is at its address and a UART register +//! is a byte access at its window. Each test builds only the chip and a +//! block of bus memory for the rings. +//! +//! Ported from tests/qtest/sgi-ioc4-test.c in jamesbraid/qemu-sgi-origami. +//! That file drives the chip through an XBridge with chardev sockets and the +//! virtual clock; here the sockets are a `Wire` per port and time is the host +//! clock, so timers are given a few milliseconds and `service` plays the part +//! of the main loop. + +use super::*; +use crate::dev::ds1744::{self, Ds1744, YearBase}; +use crate::irq::LevelProbe; +use std::sync::atomic::{AtomicI64, Ordering as AtomicOrdering}; +use crate::pci::PciDmaStatus; +use std::collections::VecDeque; + +const MEM_SIZE: usize = 0x10000; +/// Where the rings sit in bus memory. +const RINGS: u64 = 0x4000; +/// An address no memory answers. +const BAD_BASE: u64 = 0x20_0000; +const TX_VALID: u8 = 0x40; +const TX_EXPLICIT: u8 = 0x20; +const TX_MCR: u8 = 0x80; +const TX_DELAY: u8 = 0xc0; +/// Received status from the line: data and modem valid, CTS and DCD up. +const RX_FROM_LINE: u8 = 0xf0; +/// Received status looped back: RTS returns as CTS, and OUT2 clear is no DCD. +const RX_LOOPBACK: u8 = 0xd0; +const ENABLE: u32 = 1 << 28; +const RESET: u32 = 1 << 31; +const THRESHOLD: u32 = 1; +const ARM: u32 = 1 << 31; +const DRAIN: u32 = 1 << 27; +const SHADOW_OUTPUTS: u32 = 0x1f00_0000; +const ERROR_VALID: u32 = 1; +const ERROR_MULTIPLE: u32 = 1 << 5; + +/// A sink for a serial line: what the guest sent, and what the host types. +#[derive(Default)] +struct Wire { + out: std::sync::Mutex>, + inp: std::sync::Mutex>, +} + +impl SerialBackend for Wire { + fn send_byte(&self, b: u8) { + self.out.lock().unwrap().push(b); + } + fn recv_byte(&self) -> std::io::Result { + self.inp + .lock() + .unwrap() + .pop_front() + .ok_or_else(|| std::io::Error::new(std::io::ErrorKind::WouldBlock, "empty")) + } +} + +/// Bus memory, as bytes at bus addresses. Addresses past its end abort. +struct TestMem(std::sync::Mutex>); + +impl PciDma for TestMem { + fn dma_read(&self, addr: u64, buf: &mut [u8]) -> PciDmaStatus { + let m = self.0.lock().unwrap(); + let a = addr as usize; + if addr > u64::from(u32::MAX) || a + buf.len() > m.len() { + buf.fill(0); + return PciDmaStatus::MasterAbort; + } + buf.copy_from_slice(&m[a..a + buf.len()]); + PciDmaStatus::Ok + } + fn dma_write(&self, addr: u64, data: &[u8]) -> PciDmaStatus { + let mut m = self.0.lock().unwrap(); + let a = addr as usize; + if addr > u64::from(u32::MAX) || a + data.len() > m.len() { + return PciDmaStatus::MasterAbort; + } + m[a..a + data.len()].copy_from_slice(data); + PciDmaStatus::Ok + } +} + +struct Rig { + c: Ioc4, + mem: Arc, + wire: [Arc; PORTS], + pin: Arc, +} + +impl Rig { + fn new() -> Rig { + Rig::with(Ioc4Config::bare(Ioc4Clock::Mhz33)) + } + + fn with(cfg: Ioc4Config) -> Rig { + let c = Ioc4::new(cfg); + let mem = Arc::new(TestMem(std::sync::Mutex::new(vec![0; MEM_SIZE]))); + c.connect_dma(mem.clone()); + let wire: [Arc; PORTS] = std::array::from_fn(|_| Arc::new(Wire::default())); + for (p, w) in wire.iter().enumerate() { + c.set_serial_backend(p, w.clone()); + } + let (pin, line) = LevelProbe::line(); + c.connect_interrupt(line); + Rig { c, mem, wire, pin } + } + + /// A 32-bit register as the processor sees it through the self-ordered + /// slot: the BAR value is the address-order bytes read little-endian. + fn r32(&self, a: u32) -> u32 { + self.c.bar_read(0, a, 4).swap_bytes() + } + fn w32(&self, a: u32, v: u32) { + self.c.bar_write(0, a, v.swap_bytes(), 4) + } + fn r8(&self, a: u32) -> u8 { + self.c.bar_read(0, a, 1) as u8 + } + fn w8(&self, a: u32, v: u8) { + self.c.bar_write(0, a, u32::from(v), 1) + } + + /// UART register `r` of port `p`. + fn ur(&self, p: usize, r: u32) -> u8 { + self.r8(reg::UART + reg::UART_STRIDE * p as u32 + r) + } + fn uw(&self, p: usize, r: u32, v: u8) { + self.w8(reg::UART + reg::UART_STRIDE * p as u32 + r, v) + } + + /// A port's DMA register. + fn sr(&self, p: usize, off: u32) -> u32 { + self.r32(reg::SDMA_PORT + reg::SDMA_PORT_STRIDE * p as u32 + off) + } + fn sw(&self, p: usize, off: u32, v: u32) { + self.w32(reg::SDMA_PORT + reg::SDMA_PORT_STRIDE * p as u32 + off, v) + } + + /// Point a pair's rings at `base` with `size` byte rings. + fn pair_base(&self, pair: u32, base: u64, size: u32) { + let lo = reg::SDMA_BASE + 8 * pair; + self.w32(lo + 4, (base >> 32) as u32); + self.w32(lo, base as u32 | u32::from(size == 0x1000)); + } + + /// Where port `p`'s ring for `dir` (0 transmit, 1 receive) is, with its + /// pair based at `RINGS`. + fn ring(&self, p: usize, dir: u64, size: u64) -> usize { + (RINGS + ((p as u64 & 1) * 2 + dir) * size) as usize + } + + fn put(&self, at: usize, bytes: &[u8]) { + self.mem.0.lock().unwrap()[at..at + bytes.len()].copy_from_slice(bytes); + } + + fn get(&self, at: usize, n: usize) -> Vec { + self.mem.0.lock().unwrap()[at..at + n].to_vec() + } + + /// One transmit entry in port `p`'s ring, `at` bytes in. + fn tx_entry(&self, p: usize, size: u64, at: usize, data: [u8; 4], control: [u8; 4]) { + let ring = self.ring(p, 0, size) + at; + self.put(ring, &data); + self.put(ring + 4, &control); + } + + /// Wait out `ms` of host time, then play the timers. + fn settle(&self, ms: u64) { + std::thread::sleep(Duration::from_millis(ms)); + self.c.service(); + } + + /// Play the timers until `done`, a millisecond of host time apart. + fn until(&self, what: &str, done: impl Fn(&Rig) -> bool) { + for _ in 0..3000 { + self.c.service(); + if done(self) { + return; + } + std::thread::sleep(Duration::from_millis(1)); + } + panic!("timed out waiting for {what}"); + } + + /// Divisor one: a character crosses the line in about five microseconds + /// instead of a millisecond, so a long loopback run stays short. + fn fast_line(&self, p: usize) { + self.uw(p, 3, 0x80); + self.uw(p, 0, 1); + self.uw(p, 1, 0); + self.uw(p, 3, 0x03); + } + + fn sent(&self, p: usize) -> Vec { + self.wire[p].out.lock().unwrap().clone() + } + + fn type_in(&self, p: usize, bytes: &[u8]) { + self.wire[p].inp.lock().unwrap().extend(bytes.iter().copied()); + self.c.service(); + } +} + +// --------------------------------------------------------------------------- +// PCI function +// --------------------------------------------------------------------------- + +/// Ports test_identity_and_reset: vendor, device, revision 0x53, the +/// emulator's class and interrupt pin, and a 1 MiB BAR. +#[test] +fn identity_and_bar() { + let r = Rig::new(); + let cfg = r.c.config(); + assert_eq!(cfg.read(0), 0x100a_10a9); + assert_eq!(cfg.read(2), 0x00ff_0053); + assert_eq!(cfg.interrupt_pin(), PCI_INTA); + cfg.write(4, 0xffff_ffff); + assert_eq!(cfg.read(4), 0xfff0_0000, "one megabyte of memory space"); +} + +/// The IP53 status policy: 0x0220 survives every write and a reset, and an +/// instance without the policy keeps its own value. +#[test] +fn status_follows_the_board() { + let r = Rig::with(Ioc4Config::tezro()); + let cfg = r.c.config(); + assert_eq!(cfg.read(1) >> 16, 0x0220); + cfg.write(1, 0x0000_0006); + assert_eq!(cfg.read(1), 0x0220_0006); + cfg.write(1, 0xffff_0000); + assert_eq!(cfg.read(1) >> 16, 0x0220, "a status write of ones leaves the read-only fields"); + r.c.reset(); + assert_eq!(cfg.read(1), 0x0220_0000); + + let bare = Rig::new(); + assert_eq!(bare.c.config().read(1) >> 16, 0); +} + +/// Ports the reset half of test_identity_and_reset: the driver's +/// disable-and-acknowledge is not a reset, but a reset clears what it left. +#[test] +fn reset_clears_the_interrupt_state_and_the_ports() { + let r = Rig::new(); + r.w32(reg::SERIAL_ENABLE_SET, int::pass(0)); + r.w32(reg::OTHER_ENABLE_SET, 0x40); + r.uw(0, 3, 0x80); + r.uw(0, 0, 0x34); + r.uw(0, 3, 0x03); + r.uw(0, 1, 0x02); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::pass(0), int::pass(0)); + assert!(r.pin.level()); + + r.c.reset(); + assert_eq!(r.r32(reg::SERIAL_ENABLE_SET), 0); + assert_eq!(r.r32(reg::OTHER_ENABLE_SET), 0); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::pass(0), 0); + assert!(!r.pin.level()); + assert_eq!(r.ur(0, 3), 0, "line control"); + assert_eq!(r.ur(0, 1), 0, "interrupt enable"); + assert_eq!(r.ur(0, 2) & 0xc0, 0, "FIFOs off"); + // The driver's own initialisation: raw pending, separate from enabled. + r.uw(0, 1, 2); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::pass(0), int::pass(0)); + assert!(!r.pin.level(), "pending but not enabled"); +} + +/// The pin follows the enabled sources of either word. Ports the IRQ half +/// of test_identity_and_reset. +#[test] +fn the_pin_follows_enabled_sources() { + let r = Rig::new(); + r.uw(0, 1, 0x02); + assert!(!r.pin.level()); + r.w32(reg::SERIAL_ENABLE_SET, int::pass(0)); + assert!(r.pin.level()); + r.w32(reg::SERIAL_ENABLE_CLR, int::pass(0)); + assert!(!r.pin.level()); + r.w32(reg::OTHER_ENABLE_SET, int::dma_error(1)); + assert!(!r.pin.level(), "nothing pending in the other word"); +} + +// --------------------------------------------------------------------------- +// The four 16550s and the local interrupt controller +// --------------------------------------------------------------------------- + +/// Ports test_uart_and_clock: the misc-control bit names the board's clock, +/// writes to it are ignored, and each port's input is the clock over 16. +#[test] +fn clock_select_reads_back_and_sets_the_baud_base() { + for (clock, bit, hz) in [(Ioc4Clock::Mhz33, 0, 33_333_333u32), (Ioc4Clock::Mhz66, 1, 66_666_667)] { + let r = Rig::with(Ioc4Config::bare(clock)); + assert_eq!(r.r32(reg::MISC_CONTROL) & 1, bit); + r.w32(reg::MISC_CONTROL, u32::from(bit == 0)); + assert_eq!(r.r32(reg::MISC_CONTROL) & 1, bit, "the clock is the board's"); + for p in 0..PORTS { + assert_eq!(r.c.sh.st.lock().uart[p].baudbase(), hz / 16); + } + } +} + +/// Ports test_uart_and_clock's per-port register loop. +#[test] +fn every_port_has_its_own_registers() { + let r = Rig::new(); + for n in 0..PORTS { + let n8 = n as u8; + assert_eq!(r.ur(n, 5) & 0x60, 0x60, "transmitter empty after reset"); + r.uw(n, 3, 0x80); + r.uw(n, 0, 0x31 + n8); + r.uw(n, 1, 0x12 + n8); + assert_eq!(r.ur(n, 0), 0x31 + n8, "divisor low"); + assert_eq!(r.ur(n, 1), 0x12 + n8, "divisor high"); + r.uw(n, 3, 3); + r.uw(n, 2, 1); + assert_eq!(r.ur(n, 2) & 0xc0, 0xc0, "FIFOs enabled"); + r.uw(n, 2, 7); + } + for n in 0..PORTS { + assert_eq!(r.ur(n, 3), 3, "no port inherited another's line control"); + } +} + +/// Ports test_interrupts: the enable-set and enable-clear aliases, raw +/// pending against enabled, and one port's source isolated from the others'. +#[test] +fn serial_interrupts_mask_pending_and_acknowledge() { + let r = Rig::new(); + r.w32(reg::SERIAL_ENABLE_CLR, u32::MAX); + assert_eq!(r.r32(reg::SERIAL_ENABLE_SET), 0); + r.w32(reg::OTHER_ENABLE_CLR, u32::MAX); + assert_eq!(r.r32(reg::OTHER_ENABLE_SET), 0); + r.w32(reg::SERIAL_PENDING, u32::MAX); + r.w32(reg::OTHER_PENDING, u32::MAX); + for n in 0..PORTS { + r.uw(n, 1, 2); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::pass(n), int::pass(n)); + assert!(!r.pin.level(), "pending but masked"); + r.w32(reg::SERIAL_ENABLE_SET, int::pass(n)); + assert_eq!(r.r32(reg::SERIAL_ENABLE_SET), int::pass(n)); + assert!(r.pin.level()); + r.w32(reg::SERIAL_PENDING, int::pass(n)); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::pass(n), int::pass(n), "a level ignores the acknowledge"); + assert!(r.pin.level()); + assert_eq!(r.ur(n, 2) & 0xf, 2, "transmit-empty cause"); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::pass(n), 0, "the identification read quieted it"); + assert!(!r.pin.level()); + r.w32(reg::SERIAL_ENABLE_CLR, int::pass(n)); + assert_eq!(r.r32(reg::SERIAL_ENABLE_SET), 0); + r.uw(n, 1, 0); + } + for n in 0..PORTS { + r.uw(n, 1, 2); + r.w32(reg::SERIAL_ENABLE_SET, int::pass(n)); + } + let all = (0..PORTS).fold(0, |m, n| m | int::pass(n)); + assert_eq!(r.r32(reg::SERIAL_PENDING) & all, all); + r.ur(0, 2); + assert_eq!(r.r32(reg::SERIAL_PENDING) & all, all & !int::pass(0), "only port 0 quieted"); + assert!(r.pin.level()); + assert_eq!(r.r32(reg::OTHER_PENDING), 0, "no ATA or error source"); + r.w32(reg::OTHER_ENABLE_SET, 0x40); + assert_eq!(r.r32(reg::OTHER_ENABLE_SET), 0x40); + r.w32(reg::OTHER_ENABLE_CLR, 0x40); + assert_eq!(r.r32(reg::OTHER_ENABLE_SET), 0); +} + +/// Ports test_uart_pass_through_is_a_level: a write cannot clear a UART's +/// source, only the UART can. +#[test] +fn uart_pass_through_follows_the_uart() { + let r = Rig::new(); + r.w32(reg::SERIAL_PENDING, u32::MAX); + r.uw(0, 1, 2); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::pass(0), int::pass(0)); + r.w32(reg::SERIAL_PENDING, int::pass(0)); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::pass(0), int::pass(0)); + r.uw(1, 1, 2); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::pass(1), int::pass(1)); + assert_eq!(r.ur(1, 2) & 0xf, 2); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::pass(1), 0, "quieted by the UART alone, it falls"); +} + +/// Ports test_pio_fifo: a loopback burst fills the sixteen-byte FIFO and +/// comes back in order, and a receive FIFO reset drops what is held. +#[test] +fn programmed_io_goes_through_the_fifo() { + let r = Rig::new(); + for n in 0..PORTS { + r.uw(n, 3, 3); + r.uw(n, 2, 1); + r.uw(n, 4, 0x10); + for i in 0..16 { + r.uw(n, 0, 0x40 + i); + } + assert_eq!(r.ur(n, 5) & 1, 1); + for i in 0..16 { + assert_eq!(r.ur(n, 0), 0x40 + i); + } + assert_eq!(r.ur(n, 5) & 1, 0); + r.uw(n, 0, 0xaa); + assert_eq!(r.ur(n, 5) & 1, 1); + r.uw(n, 2, 3); + assert_eq!(r.ur(n, 5) & 1, 0, "a receive FIFO reset drops it"); + } +} + +/// Ports test_uart_rejects_aligned_wide_write: a wider access to a UART +/// window is refused whole and reaches no register. +#[test] +fn uart_windows_take_bytes_only() { + let r = Rig::new(); + r.uw(0, 4, 0x10); + assert_eq!(r.ur(0, 4), 0x10); + r.c.bar_write(0, reg::UART, 0x5a5a_5a5a, 4); + assert_eq!(r.ur(0, 5) & 1, 0, "no character arrived"); + r.c.bar_write(0, reg::UART, 0x5a5a, 2); + assert_eq!(r.ur(0, 5) & 1, 0); + assert_eq!(r.c.bar_read(0, reg::UART, 4), 0); + r.uw(0, 0, 0x5a); + assert_eq!(r.ur(0, 5) & 1, 1); + assert_eq!(r.ur(0, 0), 0x5a); +} + +/// Ports test_ata_readback_rejects_width: the control registers take 32-bit +/// accesses and nothing narrower, and the words around the latch are not it. +#[test] +fn control_registers_take_words_only() { + let r = Rig::with(Ioc4Config::tezro()); + r.w32(reg::ATA_READBACK, 0x1234_5678); + assert_eq!(r.c.bar_read(0, reg::ATA_READBACK, 1), 0); + assert_eq!(r.c.bar_read(0, reg::ATA_READBACK, 2), 0); + r.c.bar_write(0, reg::ATA_READBACK, 0xaa, 1); + r.c.bar_write(0, reg::ATA_READBACK, 0xbbcc, 2); + assert_eq!(r.r32(reg::ATA_READBACK), 0x1234_5678); + assert_eq!(r.r32(reg::ATA_READBACK - 4), 0); + assert_eq!(r.r32(reg::ATA_READBACK + 4), 0); + r.w32(reg::ATA_READBACK - 4, 0xaabb_ccdd); + r.w32(reg::ATA_READBACK + 4, 0x5566_7788); + assert_eq!(r.r32(reg::ATA_READBACK), 0x1234_5678); +} + +/// Ports test_ata_readback_latch and test_ata_readback_default_unsupported: +/// every one-hot value and all ones read back, a reset clears the word, and +/// a chip whose board did not ask for it reads zero. +#[test] +fn ata_readback_is_a_latch_where_the_board_asks_for_it() { + let r = Rig::with(Ioc4Config::tezro()); + let before = r.get(RINGS as usize, 8); + for bit in 0..32 { + r.w32(reg::ATA_READBACK, 1 << bit); + assert_eq!(r.r32(reg::ATA_READBACK), 1 << bit); + } + r.w32(reg::ATA_READBACK, u32::MAX); + assert_eq!(r.r32(reg::ATA_READBACK), u32::MAX); + assert_eq!(r.get(RINGS as usize, 8), before, "no DMA behind it"); + assert!(!r.pin.level()); + r.c.reset(); + assert_eq!(r.r32(reg::ATA_READBACK), 0); + + let bare = Rig::new(); + bare.w32(reg::ATA_READBACK, u32::MAX); + assert_eq!(bare.r32(reg::ATA_READBACK), 0); +} + +/// Ports test_serial_idle_status_default_pio: with every transfer finishing +/// inside the access that starts it, the DMA status reads idle, read-only. +#[test] +fn serial_control_reads_idle() { + let r = Rig::new(); + assert_eq!(r.r32(reg::SERIAL_CONTROL), 0x80); + r.w32(reg::SERIAL_CONTROL, 0); + assert_eq!(r.r32(reg::SERIAL_CONTROL), 0x80); +} + +// --------------------------------------------------------------------------- +// Serial ring DMA +// --------------------------------------------------------------------------- + +/// Ports test_dma_tx_control_entries: a modem-control slot writes the 16550's +/// register, a delay slot holds the walk for its milliseconds, and an +/// explicit-interrupt flag raises the port's completion bit. +#[test] +fn tx_control_entries_drive_the_uart() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.tx_entry(0, 0x1000, 0, [0x1f, 3, 0, 0], [TX_MCR | TX_EXPLICIT, TX_DELAY | TX_EXPLICIT, 0, 0]); + r.sw(0, sdma_off::TX_PRODUCE, 8); + assert_eq!(r.sr(0, sdma_off::SHADOW) & SHADOW_OUTPUTS, SHADOW_OUTPUTS, "the modem control reached the UART"); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::serial(0, 7), int::serial(0, 7), "the first slot's completion"); + r.w32(reg::SERIAL_PENDING, int::serial(0, 7)); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 0, "the delay holds the entry"); + assert_eq!(r.sr(0, sdma_off::CONTROL) & (1 << 16), 1 << 16, "and reads busy"); + r.settle(10); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 8); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::serial(0, 7), int::serial(0, 7), "the delay's completion"); +} + +/// Ports test_dma_tx_wrap_1k: the consumer wraps at the 1 KiB ring's last +/// entry, and the empty source comes back with it. +#[test] +fn tx_ring_wraps_at_one_kilobyte() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x400); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.sw(0, sdma_off::TX_PRODUCE, 0x3f8); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 0x3f8, "empty entries are skipped"); + r.tx_entry(0, 0x400, 0x3f8, [0x5e, 0, 0, 0], [TX_VALID, 0, 0, 0]); + r.sw(0, sdma_off::TX_PRODUCE, 0); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 0, "wrapped"); + assert_eq!(r.sent(0), [0x5e]); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::serial(0, 0), int::serial(0, 0), "transmit ring empty"); +} + +/// Ports test_dma_tx_empty_follows_the_ring: transmit-ring-empty is a level +/// that is clear while an entry is in the engine and deaf to a write. +#[test] +fn tx_empty_follows_the_ring() { + let r = Rig::new(); + let empty = int::serial(0, 0); + r.pair_base(0, RINGS, 0x1000); + assert_eq!(r.r32(reg::SERIAL_PENDING) & empty, 0, "off with ring DMA off"); + r.sw(0, sdma_off::CONTROL, ENABLE); + assert_eq!(r.r32(reg::SERIAL_PENDING) & empty, empty); + r.w32(reg::SERIAL_PENDING, empty); + assert_eq!(r.r32(reg::SERIAL_PENDING) & empty, empty); + r.tx_entry(0, 0x1000, 0, [3, 0, 0, 0], [TX_DELAY, 0, 0, 0]); + r.sw(0, sdma_off::TX_PRODUCE, 8); + assert_eq!(r.r32(reg::SERIAL_PENDING) & empty, 0); + r.settle(10); + assert_eq!(r.r32(reg::SERIAL_PENDING) & empty, empty); +} + +/// Ports test_dma_rx_drain_and_threshold: characters wait in the packer +/// until a drain posts them, with the line's modem state, and the high-water +/// source follows the entries waiting. +#[test] +fn rx_drain_posts_the_packer_and_raises_the_threshold() { + let r = Rig::new(); + let high = int::serial(0, 2); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE | THRESHOLD); + // A long reload, so the drain and not the timer posts the entry. + r.sw(0, sdma_off::RX_TIMER, 0xfff); + r.sw(0, sdma_off::RX_CONSUME, ARM); + r.type_in(0, b"abc"); + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE) & 0xff8, 0); + r.sw(0, sdma_off::CONTROL, ENABLE | THRESHOLD | DRAIN); + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE) & 0xff8, 8); + let entry = r.get(r.ring(0, 1, 0x1000), 8); + assert_eq!(entry, [b'a', b'b', b'c', 0, RX_FROM_LINE, RX_FROM_LINE, RX_FROM_LINE, 0]); + assert_eq!(r.r32(reg::SERIAL_PENDING) & high, high); + r.w32(reg::SERIAL_PENDING, high); + assert_eq!(r.r32(reg::SERIAL_PENDING) & high, high, "a level"); + r.sw(0, sdma_off::RX_CONSUME, 8); + assert_eq!(r.r32(reg::SERIAL_PENDING) & high, 0, "the consumer made room"); +} + +/// Ports test_dma_rx_timer_expiry: the timer posts a lone character when +/// its reload runs out and raises the expiry and threshold sources. +#[test] +fn rx_timer_posts_a_lone_character() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE | THRESHOLD); + r.sw(0, sdma_off::RX_TIMER, 2); + r.sw(0, sdma_off::RX_CONSUME, ARM); + r.type_in(0, &[0x6d]); + r.settle(10); + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE) & 0xff8, 8); + let entry = r.get(r.ring(0, 1, 0x1000), 8); + assert_eq!((entry[0], entry[4]), (0x6d, RX_FROM_LINE)); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::serial(0, 3), int::serial(0, 3), "timer expired"); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::serial(0, 2), int::serial(0, 2), "threshold"); +} + +/// Ports test_dma_threshold_is_a_level. +#[test] +fn threshold_is_a_level() { + let r = Rig::new(); + let high = int::serial(0, 2); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, THRESHOLD); + r.sw(0, sdma_off::RX_PRODUCE, 8); + assert_eq!(r.r32(reg::SERIAL_PENDING) & high, high); + r.w32(reg::SERIAL_PENDING, high); + assert_eq!(r.r32(reg::SERIAL_PENDING) & high, high); + r.sw(0, sdma_off::RX_CONSUME, 8); + assert_eq!(r.r32(reg::SERIAL_PENDING) & high, 0); +} + +/// Ports test_dma_channel_reset_keeps_settings: a channel reset zeroes the +/// four indices and keeps the timer reload, the UART's registers and what +/// the write that reset it stored. +#[test] +fn channel_reset_keeps_settings() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::RX_TIMER, 0x140); + r.uw(0, 4, 0x03); + r.sw(0, sdma_off::TX_PRODUCE, 0x40); + r.sw(0, sdma_off::CONTROL, RESET | 5); + assert_eq!(r.sr(0, sdma_off::TX_PRODUCE), 0); + assert_eq!(r.sr(0, sdma_off::RX_TIMER) & 0xfff, 0x140); + assert_eq!(r.sr(0, sdma_off::SHADOW) & SHADOW_OUTPUTS, 0x0300_0000); + assert_eq!(r.sr(0, sdma_off::CONTROL), 5); +} + +/// Ports test_dma_full_reset_clears_shadow: the shadow mirrors the UART and +/// ignores a write, and a chip reset puts it back to the UART's reset +/// modem control. +#[test] +fn shadow_mirrors_the_uart() { + let r = Rig::new(); + r.uw(0, 4, 0x13); + assert_eq!(r.sr(0, sdma_off::SHADOW) & SHADOW_OUTPUTS, 0x1300_0000); + r.sw(0, sdma_off::SHADOW, 0x1f00_0000); + assert_eq!(r.sr(0, sdma_off::SHADOW) & SHADOW_OUTPUTS, 0x1300_0000); + r.c.reset(); + let mcr = u32::from(r.ur(0, 4)); + assert_eq!(r.sr(0, sdma_off::SHADOW) & SHADOW_OUTPUTS, mcr << 24); +} + +/// Ports test_dma_loopback_tx_mcr in miniature: with the UART looped back +/// the engine's transmitted characters return through the receive ring with +/// the looped modem state (DTR and RTS set, so RTS comes back as CTS), and +/// none reach the line. +#[test] +fn loopback_returns_through_the_rings() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x400); + r.fast_line(0); + r.uw(0, 4, 0x13); + r.sw(0, sdma_off::CONTROL, ENABLE | THRESHOLD); + r.tx_entry(0, 0x400, 0, *b"wxyz", [TX_VALID; 4]); + r.sw(0, sdma_off::TX_PRODUCE, 8); + r.until("the receive entry", |r| r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8 == 8); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 8); + assert_eq!(r.get(r.ring(0, 1, 0x400), 8), [b'w', b'x', b'y', b'z', RX_LOOPBACK, RX_LOOPBACK, RX_LOOPBACK, RX_LOOPBACK]); + assert!(r.sent(0).is_empty(), "nothing reached the line"); +} + +/// A looped-back character takes a character time to reach the receiver, and +/// the walk waits for it: at 9600 baud the second character of an entry is +/// not sent for a millisecond. Ports the "still on the line" case of +/// test_dma_loopback_tx_mcr. +#[test] +fn a_looped_back_character_takes_a_character_time() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x400); + r.uw(0, 4, 0x13); + r.sw(0, sdma_off::CONTROL, ENABLE | THRESHOLD); + r.tx_entry(0, 0x400, 0, *b"abcd", [TX_VALID; 4]); + r.sw(0, sdma_off::TX_PRODUCE, 8); + assert!(r.c.sh.st.lock().sdma.loop_deadline(0).is_some(), "the first character is on the line"); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 0, "the entry is not done"); + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8, 0, "and nothing has been posted"); + r.until("the entry", |r| r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8 == 8); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 8); +} + +/// Ports test_dma_loopback_pio_mcr's loopback half: a full 127-entry transmit +/// ring crosses the line and fills the receive ring, entry for entry. +#[test] +fn loopback_fills_the_receive_ring() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x400); + r.fast_line(0); + r.uw(0, 4, 0x13); + assert_eq!(r.ur(0, 4), 0x13); + assert_eq!(r.sr(0, sdma_off::SHADOW) & SHADOW_OUTPUTS, 0x1300_0000); + r.sw(0, sdma_off::CONTROL, ENABLE | THRESHOLD); + let expected = fill_loopback_ring(&r); + r.until("the transmit ring", |r| r.sr(0, sdma_off::TX_CONSUME) & 0x3f8 == 0x3f8); + r.until("the receive ring", |r| r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8 == 0x3f8); + assert_eq!((r.sr(0, sdma_off::TX_PRODUCE), r.sr(0, sdma_off::TX_CONSUME)), (0x3f8, 0x3f8)); + assert_eq!(r.sr(0, sdma_off::RX_CONSUME), 0); + assert_eq!(r.get(r.ring(0, 1, 0x400), 0x3f8), expected); + assert!(r.sent(0).is_empty()); +} + +/// Ports test_dma_loopback_tx_mcr: a modem-control slot ahead of the +/// characters writes the UART's own register, and its characters then loop +/// back, entry by entry, the first holding three. +#[test] +fn a_modem_control_slot_leads_a_looped_back_run() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x400); + r.fast_line(0); + r.uw(0, 4, 0); + r.sw(0, sdma_off::CONTROL, ENABLE | THRESHOLD); + let mut expected = vec![0u8; 0x3f8]; + let mut count = 0usize; + for i in 0..127usize { + let (data, control): ([u8; 4], [u8; 4]) = if i == 0 { + ([0x13, 0, 1, 2], [TX_MCR, TX_VALID, TX_VALID, TX_VALID]) + } else { + (std::array::from_fn(|j| (i * 4 + j - 1) as u8), [TX_VALID; 4]) + }; + for &byte in &data[usize::from(i == 0)..] { + expected[count / 4 * 8 + count % 4] = byte; + expected[count / 4 * 8 + 4 + count % 4] = RX_LOOPBACK; + count += 1; + } + r.tx_entry(0, 0x400, i * 8, data, control); + } + assert_eq!(count, 507); + r.sw(0, sdma_off::TX_PRODUCE, 0x3f8); + r.until("the transmit ring", |r| r.sr(0, sdma_off::TX_CONSUME) & 0x3f8 == 0x3f8); + r.until("the last full entry", |r| r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8 == 0x3f0); + assert_eq!(r.ur(0, 4), 0x13, "the modem-control slot wrote the UART's register"); + assert_eq!(r.sr(0, sdma_off::SHADOW) & SHADOW_OUTPUTS, 0x1300_0000); + // Three characters wait in the packer for a drain. + r.sw(0, sdma_off::CONTROL, ENABLE | THRESHOLD | DRAIN); + r.until("the drain", |r| r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8 == 0x3f8); + assert_eq!(r.get(r.ring(0, 1, 0x400), 0x3f8), expected); + assert!(r.sent(0).is_empty()); +} + +/// Ports test_dma_loopback_shadow_reset: a write to the shadow changes +/// nothing, and a channel reset leaves the outputs as they were. +#[test] +fn the_shadow_ignores_writes_and_a_channel_reset() { + let r = Rig::new(); + r.uw(0, 4, 0); + r.sw(0, sdma_off::SHADOW, SHADOW_OUTPUTS); + assert_eq!(r.ur(0, 4), 0); + r.uw(0, 4, 0x13); + r.sw(0, sdma_off::SHADOW, 0x1300_0000); + assert_eq!(r.ur(0, 4), 0x13); + r.sw(0, sdma_off::CONTROL, RESET); + assert_eq!(r.sr(0, sdma_off::SHADOW) & SHADOW_OUTPUTS, 0x1300_0000); + assert_eq!(r.ur(0, 4), 0x13); +} + +/// A full 127-entry transmit ring of valid characters; the receive ring it +/// should fill. +fn fill_loopback_ring(r: &Rig) -> Vec { + let mut expected = vec![0u8; 0x3f8]; + for i in 0..127usize { + let data: [u8; 4] = std::array::from_fn(|j| (i * 4 + j) as u8); + expected[i * 8..i * 8 + 4].copy_from_slice(&data); + expected[i * 8 + 4..i * 8 + 8].fill(RX_LOOPBACK); + r.tx_entry(0, 0x400, i * 8, data, [TX_VALID; 4]); + } + r.sw(0, sdma_off::TX_PRODUCE, 0x3f8); + expected +} + +/// Ports test_dma_requester_ids' use of both pairs: ports 2 and 3 take their +/// rings from the second pair's base, apart from the first pair's. +#[test] +fn second_pair_has_its_own_base() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x400); + r.pair_base(1, RINGS + 0x2000, 0x400); + assert_eq!(r.r32(reg::SDMA_BASE), RINGS as u32); + assert_eq!(r.r32(reg::SDMA_BASE + 8), (RINGS + 0x2000) as u32); + for p in [0usize, 2, 3] { + r.sw(p, sdma_off::CONTROL, ENABLE); + } + let at = |pair: u64, p: u64| (RINGS + pair * 0x2000 + (p & 1) * 2 * 0x400) as usize; + r.put(at(1, 2), &[b'c', 0, 0, 0, TX_VALID, 0, 0, 0]); + r.put(at(1, 3), &[b'd', 0, 0, 0, TX_VALID, 0, 0, 0]); + r.put(at(0, 0), &[b'a', 0, 0, 0, TX_VALID, 0, 0, 0]); + r.sw(3, sdma_off::TX_PRODUCE, 8); + r.sw(2, sdma_off::TX_PRODUCE, 8); + r.sw(0, sdma_off::TX_PRODUCE, 8); + assert_eq!((r.sent(0), r.sent(1), r.sent(2), r.sent(3)), (b"a".to_vec(), vec![], b"c".to_vec(), b"d".to_vec())); +} + +/// Ports test_dma_error_capture_and_reset: the first failed transfer +/// captures its address, a second sets the multiple bit, the port's source +/// is in the other word, and a channel reset recovers the port but leaves +/// the capture and the source for software to acknowledge. +#[test] +fn a_failed_ring_transfer_is_captured_and_halts_the_port() { + let r = Rig::new(); + let bad = BAD_BASE + (1 << 32); + let low = BAD_BASE as u32 | ERROR_VALID; + r.pair_base(0, bad, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.sw(0, sdma_off::TX_PRODUCE, 8); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 0, "the fetch failed"); + assert_eq!(r.r32(reg::ERROR_LOW), low); + assert_eq!(r.r32(reg::ERROR_HIGH), 1); + assert_eq!(r.r32(reg::OTHER_PENDING) & int::dma_error(0), int::dma_error(0)); + + r.sw(0, sdma_off::CONTROL, RESET); + assert_eq!(r.r32(reg::ERROR_LOW), low, "a channel reset leaves the capture"); + assert_eq!(r.r32(reg::OTHER_PENDING) & int::dma_error(0), int::dma_error(0)); + r.w32(reg::OTHER_PENDING, int::dma_error(0)); + assert_eq!(r.r32(reg::OTHER_PENDING) & int::dma_error(0), 0); + r.w32(reg::ERROR_LOW, 0); + assert_eq!(r.r32(reg::ERROR_LOW), low, "the capture is read-only"); + + r.sw(0, sdma_off::CONTROL, ENABLE); + r.type_in(0, &[0x73]); + r.sw(0, sdma_off::CONTROL, ENABLE | DRAIN); + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE) & 0xff8, 0, "the post failed"); + assert_eq!(r.r32(reg::ERROR_LOW), low | ERROR_MULTIPLE); + assert_eq!(r.r32(reg::OTHER_PENDING) & int::dma_error(0), int::dma_error(0)); + r.c.reset(); + assert_eq!((r.r32(reg::ERROR_LOW), r.r32(reg::ERROR_HIGH), r.r32(reg::OTHER_PENDING)), (0, 0, 0)); +} + +/// Ports test_dma_requester_ids: a transmit fetch names twice the port and a +/// receive post one more, on every port. +#[test] +fn the_error_capture_names_the_requester() { + for p in 0..PORTS { + let r = Rig::new(); + r.pair_base(p as u32 / 2, BAD_BASE, 0x400); + r.sw(p, sdma_off::CONTROL, ENABLE); + r.sw(p, sdma_off::TX_PRODUCE, 8); + assert_eq!(r.r32(reg::OTHER_PENDING), int::dma_error(p), "only this port's bit"); + assert_eq!(r.r32(reg::ERROR_LOW) >> 1 & 0xf, 2 * p as u32); + assert_eq!(r.r32(reg::ERROR_LOW) & ERROR_VALID, ERROR_VALID); + + r.c.reset(); + r.pair_base(p as u32 / 2, BAD_BASE, 0x400); + r.sw(p, sdma_off::CONTROL, ENABLE); + r.type_in(p, &[1, 2, 3, 4]); + assert_eq!(r.r32(reg::ERROR_LOW) >> 1 & 0xf, 2 * p as u32 + 1); + assert_eq!(r.sr(p, sdma_off::RX_PRODUCE), 0); + } +} + +/// Ports test_dma_port_fault_isolation_and_recovery: a fault on one port +/// leaves its pair's other port running, and the faulted port runs again +/// once reset. +#[test] +fn a_fault_halts_only_its_own_port() { + let r = Rig::new(); + r.pair_base(0, BAD_BASE, 0x400); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.sw(1, sdma_off::CONTROL, ENABLE); + r.sw(1, sdma_off::TX_PRODUCE, 8); + assert_eq!(r.r32(reg::ERROR_LOW) >> 1 & 0xf, 2); + assert_eq!(r.r32(reg::OTHER_PENDING), int::dma_error(1)); + + // The base is shared, so quiet both ports before changing it. + r.sw(0, sdma_off::CONTROL, 0); + r.sw(1, sdma_off::CONTROL, RESET); + assert_eq!((r.sr(1, sdma_off::TX_PRODUCE), r.sr(1, sdma_off::TX_CONSUME)), (0, 0)); + r.pair_base(0, RINGS, 0x400); + assert_eq!(r.r32(reg::SDMA_BASE), RINGS as u32); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.tx_entry(0, 0x400, 0, [b'p', 0, 0, 0], [TX_VALID, 0, 0, 0]); + r.sw(0, sdma_off::TX_PRODUCE, 8); + assert_eq!(r.sent(0), b"p"); + assert_eq!(r.sr(1, sdma_off::TX_CONSUME), 0); + + r.sw(1, sdma_off::CONTROL, ENABLE); + r.tx_entry(1, 0x400, 0, [0x15, 0, 0, 0], [TX_MCR, 0, 0, 0]); + r.sw(1, sdma_off::TX_PRODUCE, 8); + assert_eq!(r.sr(1, sdma_off::TX_CONSUME), 8, "the reset port runs"); + assert_eq!(r.sr(1, sdma_off::SHADOW) & SHADOW_OUTPUTS, 0x1500_0000); +} + +// --------------------------------------------------------------------------- +// The clock on byte-bus select 0 +// --------------------------------------------------------------------------- + +const CLOCK_CONTROL: u32 = reg::CLOCK + ds1744::CONTROL as u32; + +/// A chip with a clock on a stepped nanosecond counter, seeded on +/// 2024-01-01. +fn with_clock() -> (Rig, Arc) { + let r = Rig::new(); + let ns = Arc::new(AtomicI64::new(0)); + let n = ns.clone(); + let seed = 19_723 * 86_400; // 2024-01-01 + let clock = Ds1744::with_clock(seed, YearBase::Gregorian, Arc::new(move || n.load(AtomicOrdering::SeqCst))); + r.c.attach_clock(Arc::new(clock.unwrap())); + (r, ns) +} + +/// Set the calendar through the write bit: seconds, minutes, hours, date, +/// month and year, then the century that commits it. +fn set_time(r: &Rig, [sec, min, hour, date, month, year]: [u8; 6], century: u8) { + r.w8(CLOCK_CONTROL, 0x80); + for (i, v) in [sec, min, hour, 0x02, date, month, year].into_iter().enumerate() { + r.w8(CLOCK_CONTROL + 1 + i as u32, v); + } + r.w8(CLOCK_CONTROL, century); +} + +/// Ports test_rtc_absent: with no clock fitted the window reads zero and +/// drops a write. +#[test] +fn an_unfitted_clock_reads_zero() { + let r = Rig::new(); + assert_eq!(r.r8(CLOCK_CONTROL), 0); + r.w8(CLOCK_CONTROL, 0x55); + assert_eq!(r.r8(CLOCK_CONTROL), 0); + for a in [reg::CLOCK + 0x7ff7, reg::CLOCK + 0x8000, reg::CLOCK] { + assert_eq!(r.r8(a), 0); + } +} + +/// Ports test_rtc_alias and test_nvram_window: the SRAM fills the BAR from +/// 0x80000 to the calendar at 0x87ff8, the frozen calendar reads through it, +/// and neither side of the window answers. +#[test] +fn the_clock_window_holds_ram_and_the_calendar() { + let (r, ns) = with_clock(); + let frozen = [0x60u8, 0x59, 0x59, 0x23, 0x82, 0x31, 0x12, 0x24]; + r.w8(reg::CLOCK, 0xa5); + r.w8(reg::CLOCK + 0x7ff7, 0x5a); + for (i, a) in [0x0000e, 0x004bc, 0x07fcb, 0x07fd0].into_iter().enumerate() { + r.w8(reg::CLOCK + a, 0x31 + i as u8); + } + assert_eq!((r.r8(reg::CLOCK), r.r8(reg::CLOCK + 0x7ff7)), (0xa5, 0x5a)); + for (i, a) in [0x0000e, 0x004bc, 0x07fcb, 0x07fd0].into_iter().enumerate() { + assert_eq!(r.r8(reg::CLOCK + a), 0x31 + i as u8); + } + + set_time(&r, [0x59, 0x59, 0x23, 0x31, 0x12, 0x24], 0x20); + r.w8(CLOCK_CONTROL, 0x40); + ns.fetch_add(2_000_000_000, AtomicOrdering::SeqCst); + for (i, want) in frozen.into_iter().enumerate() { + assert_eq!(r.r8(CLOCK_CONTROL + i as u32), want, "calendar byte {i}"); + } + assert!(!r.pin.level()); + r.w8(CLOCK_CONTROL, 0); + assert_eq!((r.r8(CLOCK_CONTROL + 1), r.r8(CLOCK_CONTROL + 5), r.r8(CLOCK_CONTROL + 6), r.r8(CLOCK_CONTROL + 7)), (1, 1, 1, 0x25)); + + // Past the part on either side nothing answers. + assert_eq!(r.r8(reg::CLOCK + 0x8000), 0); + assert_eq!(r.r8(reg::CLOCK + 0x8008), 0); + assert_eq!(r.r8(reg::CLOCK - 1), 0); + + // A chip reset leaves the SRAM and the calendar alone. + r.w8(CLOCK_CONTROL, 0x40); + r.c.reset(); + assert_eq!((r.r8(reg::CLOCK), r.r8(reg::CLOCK + 0x7ff7)), (0xa5, 0x5a)); + assert_eq!(r.r8(CLOCK_CONTROL), 0x60, "even a read in progress"); +} + +/// Ports test_rtc_rejects_wide_access: a halfword or word to the clock is +/// refused whole and reaches no register. +#[test] +fn the_clock_takes_bytes_only() { + let (r, _) = with_clock(); + set_time(&r, [0x12, 0x34, 0x05, 0x15, 0x01, 0x24], 0x20); + r.w8(CLOCK_CONTROL, 0x80); + r.w8(CLOCK_CONTROL + 1, 0x11); + r.w8(CLOCK_CONTROL + 2, 0x22); + r.c.bar_write(0, CLOCK_CONTROL + 2, 0x4433, 2); + r.c.bar_write(0, CLOCK_CONTROL, 0x2000_0000, 4); + assert_eq!(r.c.bar_read(0, CLOCK_CONTROL + 2, 2), 0); + assert_eq!(r.c.bar_read(0, CLOCK_CONTROL, 4), 0); + assert_eq!((r.r8(CLOCK_CONTROL + 2), r.r8(CLOCK_CONTROL + 3)), (0x22, 0x05)); + r.w8(CLOCK_CONTROL, 0x20); + assert_eq!((r.r8(CLOCK_CONTROL + 2), r.r8(CLOCK_CONTROL + 3)), (0x22, 0x05)); +} + +// --------------------------------------------------------------------------- +// A backend that cannot take a character +// --------------------------------------------------------------------------- + +/// A line that takes nothing while shut. +#[derive(Default)] +struct Gate { + open: std::sync::atomic::AtomicBool, + out: std::sync::Mutex>, +} + +impl SerialBackend for Gate { + fn send_byte(&self, b: u8) { + self.out.lock().unwrap().push(b); + } + fn try_send_byte(&self, b: u8) -> bool { + let open = self.open.load(AtomicOrdering::SeqCst); + if open { + self.send_byte(b); + } + open + } + fn recv_byte(&self) -> std::io::Result { + Err(std::io::Error::new(std::io::ErrorKind::WouldBlock, "empty")) + } +} + +/// Ports test_dma_tx_refusing_backend_holds_then_completes: a backend that +/// never takes a character holds the walk behind it, but only for the +/// 16550 core's bound. With the FIFO off the port takes two characters, one +/// in its shift register and one in its holding register, and the third +/// waits; two seconds on the core gives the held one up and the walk goes +/// on. +#[test] +fn a_refusing_backend_holds_the_walk_then_gives_up() { + let r = Rig::new(); + let gate = Arc::new(Gate::default()); + r.c.set_serial_backend(0, gate.clone()); + r.pair_base(0, RINGS, 0x1000); + for i in 0..3 { + r.tx_entry(0, 0x1000, 8 * i, [0x41, 0, 0, 0], [TX_VALID, 0, 0, 0]); + } + r.sw(0, sdma_off::CONTROL, ENABLE); + r.sw(0, sdma_off::TX_PRODUCE, 24); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 16, "two characters taken, the third waits"); + r.c.service(); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 16, "still refused"); + + let late = Instant::now() + Duration::from_secs(3); + r.c.sh.with_st(|st| { + assert!(st.uart[0].poll_tx(late)); + st.uart_changed(0); + }); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 24, "given up on, so the walk goes on"); + assert!(gate.out.lock().unwrap().is_empty()); +} + +/// The walk resumes without a give-up when the backend opens. +#[test] +fn a_backend_that_opens_releases_the_walk() { + let r = Rig::new(); + let gate = Arc::new(Gate::default()); + r.c.set_serial_backend(0, gate.clone()); + r.pair_base(0, RINGS, 0x1000); + for i in 0..3 { + r.tx_entry(0, 0x1000, 8 * i, [0x41 + i as u8, 0, 0, 0], [TX_VALID, 0, 0, 0]); + } + r.sw(0, sdma_off::CONTROL, ENABLE); + r.sw(0, sdma_off::TX_PRODUCE, 24); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 16); + gate.open.store(true, AtomicOrdering::SeqCst); + r.c.service(); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 24); + assert_eq!(*gate.out.lock().unwrap(), b"ABC"); +} + +// --------------------------------------------------------------------------- +// A receive ring with no room +// --------------------------------------------------------------------------- + +const RX_FULL: u32 = int::serial(0, 1); + +/// Ports test_dma_rx_full_and_staged_recovery: a character the full ring +/// cannot take waits, receive-ring-full is a level that stays while an +/// entry waits for room and ignores an acknowledge, and the entries post as +/// the consumer makes room. Filled exactly with nothing waiting, it stays +/// down. +#[test] +fn receive_ring_full_is_a_level_while_an_entry_waits() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x400); + r.sw(0, sdma_off::CONTROL, ENABLE); + for i in 0..127u32 { + r.type_in(0, &[0x31, 0x32, 0x33, 0x34]); + r.until("the entry", |r| r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8 == ((i + 1) * 8) & 0x3f8); + } + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE), 0x3f8); + assert_eq!(r.r32(reg::SERIAL_PENDING) & RX_FULL, 0, "full, but nothing waits for room"); + + r.sw(0, sdma_off::RX_TIMER, 1); + r.sw(0, sdma_off::RX_CONSUME, ARM); + r.type_in(0, &[0xa1, 0xa2, 0xa3, 0xa4, 0xb5]); + r.until("the source", |r| r.r32(reg::SERIAL_PENDING) & RX_FULL != 0); + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE), 0x3f8); + r.w32(reg::SERIAL_PENDING, RX_FULL); + assert_eq!(r.r32(reg::SERIAL_PENDING) & RX_FULL, RX_FULL, "a level ignores the acknowledge"); + + r.sw(0, sdma_off::RX_CONSUME, 16 | ARM); + r.until("room", |r| r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8 == 0); + let entry = r.get(r.ring(0, 1, 0x400) + 0x3f8, 8); + assert_eq!((&entry[..4], entry[4], entry[7]), (&[0xa1, 0xa2, 0xa3, 0xa4][..], RX_FROM_LINE, RX_FROM_LINE)); + assert_eq!(r.r32(reg::SERIAL_PENDING) & RX_FULL, 0, "it fell once the consumer made room"); + + r.until("the last character", |r| r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8 == 8); + let entry = r.get(r.ring(0, 1, 0x400), 8); + assert_eq!((entry[0], entry[1], entry[4], entry[5]), (0xb5, 0, RX_FROM_LINE, 0)); + assert_eq!(r.r32(reg::SERIAL_PENDING) & RX_FULL, 0, "full again, but nothing waits for room"); +} + +/// Ports test_dma_loopback_full_ring_recovery: a full receive ring does not +/// stall the transmit walk in loopback. Both entries go out and the eight +/// characters wait in the packer and the 16550's FIFO, then land as the +/// consumer frees the ring. +#[test] +fn a_full_ring_does_not_stall_a_loopback_walk() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x400); + r.fast_line(0); + r.uw(0, 2, 0x01); + r.uw(0, 4, 0x13); + r.sw(0, sdma_off::CONTROL, ENABLE | THRESHOLD); + let expected = fill_loopback_ring(&r); + r.until("the transmit ring", |r| r.sr(0, sdma_off::TX_CONSUME) & 0x3f8 == 0x3f8); + r.until("the receive ring", |r| r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8 == 0x3f8); + assert_eq!(r.get(r.ring(0, 1, 0x400), 0x3f8), expected); + + let looped = |c: [u8; 4]| [c[0], c[1], c[2], c[3], RX_LOOPBACK, RX_LOOPBACK, RX_LOOPBACK, RX_LOOPBACK]; + r.tx_entry(0, 0x400, 0x3f8, [0xa1, 0xa2, 0xa3, 0xa4], [TX_VALID; 4]); + r.tx_entry(0, 0x400, 0, [0xb1, 0xb2, 0xb3, 0xb4], [TX_VALID; 4]); + r.sw(0, sdma_off::TX_PRODUCE, 8); + r.until("both entries sent", |r| r.sr(0, sdma_off::TX_CONSUME) & 0x3f8 == 8); + r.until("the source", |r| r.r32(reg::SERIAL_PENDING) & RX_FULL != 0); + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE), 0x3f8); + + r.sw(0, sdma_off::RX_CONSUME, 8); + r.until("the first", |r| r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8 == 0); + assert_eq!(r.get(r.ring(0, 1, 0x400) + 0x3f8, 8), looped([0xa1, 0xa2, 0xa3, 0xa4])); + r.sw(0, sdma_off::RX_CONSUME, 16); + r.until("the second", |r| r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8 == 8); + assert_eq!(r.r32(reg::SERIAL_PENDING) & RX_FULL, 0, "filled exactly with nothing waiting"); + assert_eq!(r.get(r.ring(0, 1, 0x400), 8), looped([0xb1, 0xb2, 0xb3, 0xb4])); + assert!(r.sent(0).is_empty()); +} + +/// Ports test_dma_loopback_overrun_reaches_status: with the receive ring +/// out of room the characters that do not fit are lost to the 16550's +/// overrun, which shows in the status byte of the next character read. +#[test] +fn a_loopback_overrun_reaches_the_status_byte() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x400); + r.fast_line(0); + r.uw(0, 2, 0x01); + r.uw(0, 4, 0x13); + r.sw(0, sdma_off::CONTROL, ENABLE); + // Room for one entry before the ring is full. + r.sw(0, sdma_off::RX_CONSUME, 16); + for i in 0..8usize { + r.tx_entry(0, 0x400, 8 * i, std::array::from_fn(|j| (4 * i + j) as u8), [TX_VALID; 4]); + } + r.sw(0, sdma_off::TX_PRODUCE, 64); + r.until("the walk", |r| r.sr(0, sdma_off::TX_CONSUME) & 0x3f8 == 64); + r.settle(5); + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8, 8); + + r.sw(0, sdma_off::RX_CONSUME, 8); + r.sw(0, sdma_off::CONTROL, ENABLE | DRAIN); + r.until("the drain", |r| r.sr(0, sdma_off::CONTROL) & DRAIN == 0); + let posted = (r.sr(0, sdma_off::RX_PRODUCE) & 0x3f8) / 8; + assert!(posted > 1); + let overrun = (1..posted as usize).any(|i| r.get(r.ring(0, 1, 0x400) + 8 * i + 4, 4).iter().any(|s| s & 1 != 0)); + assert!(overrun, "the loss is marked on a later character"); + assert!(posted * 4 < 32, "and not everything came back"); +} + +/// Ports test_serial_idle_status: the serial control word reads the idle bit +/// set after a write, a pause, a reset, PIO and a ring run, and the writes to +/// it and to the unsupported ATA window disturb no interrupt word. +#[test] +fn serial_idle_status_survives_writes_and_ring_runs() { + let r = Rig::new(); + assert_eq!(r.r32(reg::SERIAL_CONTROL), 0x80); + r.w32(reg::SERIAL_CONTROL, 0x80); + r.c.bar_read(0, reg::SERIAL_CONTROL, 1); + r.c.bar_write(0, reg::SERIAL_CONTROL, 0x80, 1); + assert_eq!(r.r32(reg::SERIAL_CONTROL), 0x80); + assert_eq!(r.r32(0x100), 0); + r.w32(0x100, u32::MAX); + assert_eq!(r.r32(reg::SERIAL_CONTROL), 0x80); + + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE | (1 << 29)); + r.tx_entry(0, 0x1000, 0, [0x5a, 0, 0, 0], [TX_VALID, 0, 0, 0]); + r.sw(0, sdma_off::TX_PRODUCE, 8); + assert_ne!(r.sr(0, sdma_off::CONTROL) & (1 << 30), 0, "paused"); + assert_eq!(r.r32(reg::SERIAL_CONTROL), 0x80); + + r.c.reset(); + r.uw(0, 0, 0x5a); + assert_eq!(r.sent(0), [0x5a]); + assert_eq!(r.r32(reg::SERIAL_CONTROL), 0x80); + + r.pair_base(0, RINGS, 0x1000); + r.tx_entry(1, 0x1000, 0, [0x15, 0, 0, 0], [TX_MCR | TX_EXPLICIT, 0, 0, 0]); + r.sw(1, sdma_off::CONTROL, ENABLE); + r.sw(1, sdma_off::TX_PRODUCE, 8); + assert_eq!(r.sr(1, sdma_off::TX_CONSUME), 8); + let (serial, other) = (r.r32(reg::SERIAL_PENDING), r.r32(reg::OTHER_PENDING)); + assert_ne!(serial, 0); + r.w32(reg::SERIAL_CONTROL, 0x80); + r.w32(0x100, u32::MAX); + assert_eq!((r.r32(reg::SERIAL_PENDING), r.r32(reg::OTHER_PENDING)), (serial, other)); + assert_eq!(r.r32(reg::SERIAL_CONTROL), 0x80); +} + +// --------------------------------------------------------------------------- +// The host end of a port +// --------------------------------------------------------------------------- + +/// Ports test_dma_loopback_discards_external_input: in loopback the UART's +/// serial input is disconnected, as on a PC16550D. Input from outside is +/// discarded on arrival, not kept for later, and new input arrives as usual +/// once the loopback is off. +#[test] +fn loopback_discards_input_from_outside() { + let r = Rig::new(); + let input = [0xa1, 0xa2, 0xa3, 0xa4]; + r.pair_base(0, RINGS, 0x400); + r.uw(0, 4, 0x10); + r.sw(0, sdma_off::CONTROL, ENABLE | THRESHOLD); + r.type_in(0, &input); + r.sw(0, sdma_off::CONTROL, ENABLE | THRESHOLD | DRAIN); + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE), 0); + + // Out of loopback nothing comes back: it was discarded, not held. + r.uw(0, 4, 0); + r.c.service(); + r.sw(0, sdma_off::CONTROL, ENABLE | THRESHOLD | DRAIN); + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE), 0); + + r.type_in(0, &input); + r.until("the entry", |r| r.sr(0, sdma_off::RX_PRODUCE) == 8); + assert_eq!(&r.get(r.ring(0, 1, 0x400), 4), &input); +} + +/// Ports test_dma_chardev_reconnect: a port's backend can be replaced while +/// the machine runs, and both PIO and the ring engine then use the new one. +#[test] +fn replacing_the_backend_keeps_the_engine_working() { + let r = Rig::new(); + let new = Arc::new(Wire::default()); + r.c.set_serial_backend(0, new.clone()); + r.c.set_serial_backend(0, new.clone()); + r.uw(0, 0, b'p'); + assert_eq!(*new.out.lock().unwrap(), b"p"); + assert!(r.sent(0).is_empty(), "the old backend saw nothing"); + + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.tx_entry(0, 0x1000, 0, [0x61, 0x62, 0, 0], [TX_VALID, TX_VALID, 0, 0]); + r.sw(0, sdma_off::TX_PRODUCE, 8); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 8); + assert_eq!(*new.out.lock().unwrap(), b"pab"); + new.inp.lock().unwrap().push_back(0x7a); + r.c.service(); + r.sw(0, sdma_off::CONTROL, ENABLE | DRAIN); + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE) & 0xff8, 8, "and its input arrives"); + assert_eq!(r.get(r.ring(0, 1, 0x1000), 1), [0x7a]); +} + +// --------------------------------------------------------------------------- +// The rest of the ring engine's register cases +// --------------------------------------------------------------------------- + +const TIMER_EXPIRED: u32 = int::serial(0, 3); +const TX_COMPLETION: u32 = int::serial(0, 7); + +/// Ports test_dma_tx_wrap_4k. +#[test] +fn tx_ring_wraps_at_four_kilobytes() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.sw(0, sdma_off::TX_PRODUCE, 0xff8); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 0xff8); + r.tx_entry(0, 0x1000, 0xff8, [0x6e, 0, 0, 0], [TX_VALID, 0, 0, 0]); + r.sw(0, sdma_off::TX_PRODUCE, 0); + assert_eq!(r.sent(0), [0x6e]); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 0); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::serial(0, 0), int::serial(0, 0)); +} + +/// Ports test_dma_control_stores_its_bits: the control register keeps every +/// writable bit as written, including ones the model does not act on. +#[test] +fn control_stores_the_bits_it_does_not_act_on() { + let r = Rig::new(); + let written = (1 << 17) | (1 << 20) | 0x1ff; + r.sw(0, sdma_off::CONTROL, written); + assert_eq!(r.sr(0, sdma_off::CONTROL), written); +} + +/// Ports test_dma_engine_indices_take_writes and +/// test_dma_host_index_masks_its_pointer: the two indices the engine moves +/// still take a write, and every index keeps only its pointer field. +#[test] +fn indices_keep_their_pointer_field() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::TX_CONSUME, u32::MAX); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 0xff8); + r.sw(0, sdma_off::RX_PRODUCE, 0x18); + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE), 0x18); + r.sw(0, sdma_off::TX_PRODUCE, 0x10014); + assert_eq!(r.sr(0, sdma_off::TX_PRODUCE), 0x10); + r.sw(0, sdma_off::RX_CONSUME, 0x2c); + assert_eq!(r.sr(0, sdma_off::RX_CONSUME) & 0xff8, 0x28); +} + +/// Ports test_dma_empty_slot_ends_the_entry: an empty slot ends its entry +/// wherever it sits, so the slots behind it are not sent. +#[test] +fn an_empty_slot_ends_the_entry() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.tx_entry(0, 0x1000, 0, *b"abcd", [TX_VALID, 0, TX_VALID, TX_VALID]); + r.tx_entry(0, 0x1000, 8, [b'e', 0, 0, 0], [TX_VALID, 0, 0, 0]); + r.sw(0, sdma_off::TX_PRODUCE, 16); + assert_eq!(r.sent(0), b"ae"); +} + +/// Ports test_dma_tx_control_decodes_type_and_bit5: a control byte is +/// decoded by its type field and bit 5, undefined low bits do not stop the +/// port, and a slot of type none raises nothing even with bit 5 set. +#[test] +fn a_control_byte_is_decoded_by_type_and_bit_five() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.tx_entry(0, 0x1000, 0, [b'a', b'b', b'c', 0], [TX_VALID | 0x10, TX_EXPLICIT, TX_VALID, 0]); + r.tx_entry(0, 0x1000, 8, [b'd', 0, 0, 0], [TX_VALID, 0, 0, 0]); + r.sw(0, sdma_off::TX_PRODUCE, 16); + assert_eq!(r.sent(0), b"ad"); + assert_eq!(r.sr(0, sdma_off::TX_CONSUME), 16); + assert_eq!(r.r32(reg::SERIAL_PENDING) & TX_COMPLETION, 0); +} + +/// Ports test_dma_tx_delay_sets_the_busy_bit: control bit 16 reads set while +/// a delay entry runs and clear once it has ended. +#[test] +fn a_delay_entry_sets_the_busy_bit() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.tx_entry(0, 0x1000, 0, [3, 0, 0, 0], [TX_DELAY, 0, 0, 0]); + r.sw(0, sdma_off::TX_PRODUCE, 8); + assert_eq!(r.sr(0, sdma_off::CONTROL) & (1 << 16), 1 << 16); + r.settle(10); + assert_eq!(r.sr(0, sdma_off::CONTROL) & (1 << 16), 0); +} + +/// Ports test_dma_rx_timer_due_before_access: an expiry already due +/// happens before a register access, with no timer thread to run it. +#[test] +fn a_due_receive_timer_expires_before_an_access() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.sw(0, sdma_off::RX_TIMER, 2); + r.sw(0, sdma_off::RX_CONSUME, ARM); + r.type_in(0, &[0x64]); + std::thread::sleep(Duration::from_millis(5)); + assert_eq!(r.r32(reg::SERIAL_PENDING) & TIMER_EXPIRED, TIMER_EXPIRED); +} + +/// Ports test_dma_rx_timer_count_reads_back: a reload write loads the live +/// count too and an idle timer holds it; a running one counts down. +#[test] +fn the_receive_timer_count_reads_back() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::RX_TIMER, 0xfff); + assert_eq!(r.sr(0, sdma_off::RX_TIMER) & 0x0fff_0fff, 0x0fff_0fff); + std::thread::sleep(Duration::from_millis(2)); + assert_eq!(r.sr(0, sdma_off::RX_TIMER) & 0x0fff_0fff, 0x0fff_0fff, "idle, it holds"); + r.sw(0, sdma_off::RX_TIMER, 0x140); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.sw(0, sdma_off::RX_CONSUME, ARM); + r.type_in(0, &[0x63]); + let count = r.sr(0, sdma_off::RX_TIMER) >> 16 & 0xfff; + assert!((1..=0x140).contains(&count), "running: {count:#x} of 0x140 left"); +} + +/// Ports test_dma_consume_without_arm_keeps_it: a consume write with bit 31 +/// clear leaves the arm, which reads back and still starts the timer. +#[test] +fn a_consume_write_without_the_arm_keeps_it() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.sw(0, sdma_off::RX_TIMER, 2); + r.sw(0, sdma_off::RX_CONSUME, ARM); + r.sw(0, sdma_off::RX_CONSUME, 0); + assert_eq!(r.sr(0, sdma_off::RX_CONSUME), ARM); + r.type_in(0, &[0x6b]); + r.until("the expiry", |r| r.r32(reg::SERIAL_PENDING) & TIMER_EXPIRED != 0); + assert_eq!(r.sr(0, sdma_off::RX_CONSUME), 0, "and the expiry spent it"); +} + +/// Ports test_dma_zero_reload_expires_at_once. +#[test] +fn a_zero_reload_expires_at_once() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.sw(0, sdma_off::RX_TIMER, 0); + r.sw(0, sdma_off::RX_CONSUME, ARM); + r.type_in(0, &[0x7a]); + r.until("the expiry", |r| r.r32(reg::SERIAL_PENDING) & TIMER_EXPIRED != 0); + assert_eq!(r.sr(0, sdma_off::RX_PRODUCE), 8); +} + +/// Ports test_dma_rx_status_carries_modem_state: a character from the line +/// carries the live CTS and DCD levels, as the UART's modem status shows. +#[test] +fn a_received_character_carries_the_modem_state() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.type_in(0, &[0x6e]); + r.sw(0, sdma_off::CONTROL, ENABLE | DRAIN); + let entry = r.get(r.ring(0, 1, 0x1000), 8); + assert_eq!((entry[0], entry[4]), (0x6e, RX_FROM_LINE)); + assert_eq!(r.ur(0, 6) & 0xb0, 0xb0); +} + +/// Ports test_dma_rx_timer_expiry_after_full_entry: a full entry posts at +/// once, the timer still raises its source when it runs out, and once +/// acknowledged it does not come back without another arm. +#[test] +fn the_timer_still_expires_after_a_full_entry() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.sw(0, sdma_off::RX_TIMER, 2); + r.sw(0, sdma_off::RX_CONSUME, ARM); + r.type_in(0, &[0x31, 0x32, 0x33, 0x34]); + r.until("the entry", |r| r.sr(0, sdma_off::RX_PRODUCE) & 0xff8 == 8); + let entry = r.get(r.ring(0, 1, 0x1000), 8); + assert_eq!((&entry[..4], entry[4], entry[7]), (&[0x31, 0x32, 0x33, 0x34][..], RX_FROM_LINE, RX_FROM_LINE)); + r.until("the expiry", |r| r.r32(reg::SERIAL_PENDING) & TIMER_EXPIRED != 0); + r.w32(reg::SERIAL_PENDING, TIMER_EXPIRED); + r.settle(5); + assert_eq!(r.r32(reg::SERIAL_PENDING) & TIMER_EXPIRED, 0); +} + +/// Ports test_dma_rx_disable_stops_timer_keeps_arm: clearing the enable stops +/// a running receive timer and discards its time, and the arm stays so the +/// first character after re-enabling starts it afresh. +#[test] +fn disabling_the_ring_stops_the_timer_and_keeps_the_arm() { + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.sw(0, sdma_off::RX_TIMER, 0x100); + r.sw(0, sdma_off::RX_CONSUME, ARM); + r.type_in(0, &[0x31]); + assert!(r.c.sh.st.lock().sdma.rx_deadline(0).is_some(), "running"); + r.sw(0, sdma_off::CONTROL, 0); + assert!(r.c.sh.st.lock().sdma.rx_deadline(0).is_none(), "stopped"); + r.settle(20); + assert_eq!(r.r32(reg::SERIAL_PENDING) & TIMER_EXPIRED, 0); + r.sw(0, sdma_off::CONTROL, ENABLE); + assert_eq!(r.sr(0, sdma_off::RX_CONSUME) & ARM, ARM, "the arm stays"); + r.type_in(0, &[0x32]); + assert!(r.c.sh.st.lock().sdma.rx_deadline(0).is_some(), "the next character starts it afresh"); +} + +// --------------------------------------------------------------------------- +// The PS/2 keyboard and mouse controller +// --------------------------------------------------------------------------- + +const KBC_CONTROL: u32 = reg::KBC; +const KBC_KBD_RECEIVE: u32 = reg::KBC + 0x04; +const KBC_AUX_RECEIVE: u32 = reg::KBC + 0x08; +const KBC_KBD_TRANSMIT: u32 = reg::KBC + 0x0c; +const KBC_AUX_TRANSMIT: u32 = reg::KBC + 0x10; +const KBC_IRQ: u32 = 0x40; +const KBC_WRITE_PENDING: u32 = 0x03; + +fn key(r: &Rig, code: winit::keyboard::KeyCode, down: bool) { + r.c.ps2().push_kb(code, down); +} + +/// Ports test_ps2_adapter (QEMU with the read-only status +/// bits): the five registers through the adapter, reply bytes in BE32 +/// presentation, write-pending never set, the keyboard source in the other +/// word with its own enable and acknowledge, isolation from the serial +/// words, a mouse report, and reset. +#[test] +fn the_ps2_adapter_presents_the_controller() { + use winit::keyboard::KeyCode::KeyA; + let r = Rig::new(); + assert_eq!((r.r32(KBC_CONTROL), r.r32(KBC_KBD_RECEIVE), r.r32(KBC_AUX_RECEIVE)), (0, 0, 0)); + r.w32(KBC_CONTROL, u32::MAX); + assert_eq!(r.r32(KBC_CONTROL), 0x00fc_3cc0, "the status bits read clear and drop the write"); + r.w32(KBC_CONTROL, 0); + r.c.bar_read(0, KBC_CONTROL, 1); + r.c.bar_write(0, KBC_KBD_TRANSMIT, 0xff, 2); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0, "a narrow write reached nothing"); + + // 0xff000000 is a zero byte to the keyboard: it only acknowledges. + r.w32(KBC_KBD_TRANSMIT, 0xff00_0000); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0x80fa_0000); + r.w32(KBC_KBD_TRANSMIT, 0xff); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0xc0fa_aa00); + assert_eq!(r.r32(KBC_CONTROL) & KBC_WRITE_PENDING, 0); + // QEMU's keyboard scans again after the reset command; this core waits + // for the driver's enable, as the O2's and Octane's PROMs give it. + r.w32(KBC_KBD_TRANSMIT, 0xf4); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0x80fa_0000); + r.w32(reg::OTHER_PENDING, KBC_IRQ); + + key(&r, KeyA, true); + assert_eq!(r.r32(reg::OTHER_PENDING) & KBC_IRQ, KBC_IRQ); + assert!(!r.pin.level(), "pending but not enabled"); + r.w32(reg::OTHER_ENABLE_SET, KBC_IRQ); + assert!(r.pin.level()); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0x801c_0000); + assert_eq!(r.r32(KBC_KBD_RECEIVE) & 0x1f00_0000, 0); + assert_eq!(r.r32(reg::OTHER_PENDING) & KBC_IRQ, KBC_IRQ, "a drain does not acknowledge"); + + // A serial source holds the pin up when the keyboard's goes. + r.uw(0, 3, 0x80); + r.uw(0, 0, 0x34); + r.uw(0, 1, 0x12); + r.uw(0, 3, 0x03); + r.uw(0, 2, 0x01); + r.uw(0, 1, 0x02); + r.w32(reg::SERIAL_ENABLE_SET, int::pass(0)); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::pass(0), int::pass(0)); + assert!(r.pin.level()); + r.w32(reg::OTHER_ENABLE_CLR, KBC_IRQ); + assert!(r.pin.level()); + r.w32(reg::OTHER_ENABLE_SET, KBC_IRQ); + r.w32(reg::OTHER_PENDING, KBC_IRQ); + assert_eq!(r.r32(reg::OTHER_PENDING) & KBC_IRQ, 0); + assert_eq!(r.r32(reg::SERIAL_PENDING) & int::pass(0), int::pass(0), "the serial source is another word"); + assert!(r.pin.level()); + r.uw(0, 1, 0); + assert!(!r.pin.level()); + + r.w32(KBC_AUX_TRANSMIT, 0xff); + assert_eq!(r.r32(KBC_AUX_RECEIVE), 0xe0fa_aa00); + assert_eq!(r.r32(KBC_CONTROL) & KBC_WRITE_PENDING, 0); + r.w32(reg::OTHER_PENDING, KBC_IRQ); + r.w32(KBC_AUX_TRANSMIT, 0xf4); + assert_eq!(r.r32(KBC_AUX_RECEIVE), 0x80fa_0000); + r.w32(reg::OTHER_PENDING, KBC_IRQ); + r.c.ps2().push_mouse_input(1, 4, 0, 0); + assert_eq!(r.r32(KBC_AUX_RECEIVE), 0xe009_0400); + + r.c.reset(); + assert_eq!((r.r32(KBC_CONTROL), r.r32(KBC_KBD_RECEIVE), r.r32(KBC_AUX_RECEIVE)), (0, 0, 0)); + assert_eq!((r.r32(reg::OTHER_PENDING) & KBC_IRQ, r.r32(reg::OTHER_ENABLE_SET) & KBC_IRQ), (0, 0)); + r.w32(KBC_KBD_TRANSMIT, 0xff); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0xc0fa_aa00); +} + +/// Ports test_ps2_ports_present: every IOC4 has both ports; there is no +/// switch to leave them unplugged. +#[test] +fn both_ps2_ports_are_always_fitted() { + let r = Rig::with(Ioc4Config::bare(Ioc4Clock::Mhz33)); + r.w32(KBC_KBD_TRANSMIT, 0xff); + r.w32(KBC_AUX_TRANSMIT, 0xff); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0xc0fa_aa00); + assert_eq!(r.r32(KBC_AUX_RECEIVE), 0xe0fa_aa00); +} + +/// Ports test_ps2_driver_sequence, the attach and interrupt sequence of +/// OpenBSD's public IOC4 keyboard driver: read-modify-write the control +/// register to clamp after one character, transmit after the write-pending +/// poll, then on each interrupt mask the source, read each receive register +/// until no valid flag is left, acknowledge and unmask. Also the +/// clamp-after-three delivery, and an unclamped port losing a fourth +/// character without showing the IOC3-only overflow flag. +#[test] +fn the_ps2_driver_sequence() { + use winit::keyboard::KeyCode::{KeyA, KeyB, KeyC, KeyD}; + let r = Rig::new(); + let mut csr = r.r32(KBC_CONTROL); + csr &= !0x00c0_0ccc; + csr |= 0x0030_0000; + r.w32(KBC_CONTROL, csr); + assert_eq!(r.r32(KBC_CONTROL), 0x0030_0000); + // Status bits are not storage: a write to them is dropped and logged. + r.w32(KBC_CONTROL, 0x0030_0000 | 0xc33f); + assert_eq!(r.r32(KBC_CONTROL), 0x0030_0000); + r.w32(KBC_CONTROL, 0x0030_0000 | 0x3cc0); + assert_eq!(r.r32(KBC_CONTROL), 0x0030_3cc0, "the writable bits among them stick"); + r.w32(KBC_CONTROL, 0x0030_0000); + r.w32(reg::OTHER_ENABLE_SET, KBC_IRQ); + + // A reset reply is two characters, so the clamp holds the second back. + assert_eq!(r.r32(KBC_CONTROL) & KBC_WRITE_PENDING, 0); + r.w32(KBC_KBD_TRANSMIT, 0xff); + assert!(r.pin.level()); + assert_eq!(r.r32(reg::OTHER_PENDING) & KBC_IRQ, KBC_IRQ); + r.w32(reg::OTHER_ENABLE_CLR, KBC_IRQ); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0x80fa_0000); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0x80aa_0000); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0); + assert_eq!(r.r32(KBC_AUX_RECEIVE), 0); + // As in the adapter test: this keyboard model needs the driver's enable. + r.w32(KBC_KBD_TRANSMIT, 0xf4); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0x80fa_0000); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0); + r.w32(reg::OTHER_PENDING, KBC_IRQ); + r.w32(reg::OTHER_ENABLE_SET, KBC_IRQ); + assert!(!r.pin.level()); + + // Press and release: three characters, one per read under the clamp. + key(&r, KeyA, true); + key(&r, KeyA, false); + assert!(r.pin.level()); + r.w32(reg::OTHER_ENABLE_CLR, KBC_IRQ); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0x801c_0000); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0x80f0_0000); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0x801c_0000); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0); + r.w32(reg::OTHER_PENDING, KBC_IRQ); + r.w32(reg::OTHER_ENABLE_SET, KBC_IRQ); + assert!(!r.pin.level()); + + // Clamp after three: the same press and release arrive in one read. + r.w32(KBC_CONTROL, 0x0040_0000); + key(&r, KeyA, true); + key(&r, KeyA, false); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0xe01c_f01c); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0); + + // No clamp: a fourth character is lost and the flag is not reported. + r.w32(KBC_CONTROL, 0); + for k in [KeyA, KeyB, KeyC, KeyD] { + key(&r, k, true); + } + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0xe01c_3221); + assert_eq!(r.r32(KBC_KBD_RECEIVE), 0); +} + +/// Ports the width rule of test_ps2_adapter: the keyboard block takes +/// aligned 32-bit accesses only, and a refused one reaches no register. +#[test] +fn the_ps2_block_takes_aligned_words_only() { + let r = Rig::new(); + r.w32(KBC_CONTROL, 0x0030_0000); + assert_eq!(r.c.bar_read(0, KBC_CONTROL, 1), 0); + assert_eq!(r.c.bar_read(0, KBC_CONTROL, 2), 0); + r.c.bar_write(0, KBC_CONTROL, 0, 2); + r.c.bar_write(0, KBC_CONTROL + 1, 0, 1); + assert_eq!(r.r32(KBC_CONTROL), 0x0030_0000); + assert_eq!(r.c.bar_read(0, KBC_CONTROL + 2, 4), 0, "unaligned"); + // Beyond the five registers is not the controller. + assert_eq!(r.r32(reg::KBC + 0x14), 0); +} + +/// A key pressed on the host reaches the pin with no guest access, from the +/// host's thread, as the IOC3's does. +#[test] +fn a_host_key_raises_the_pin_without_a_guest_access() { + let r = Rig::new(); + r.w32(reg::OTHER_ENABLE_SET, KBC_IRQ); + r.w32(KBC_KBD_TRANSMIT, 0xf4); + let _ = r.r32(KBC_KBD_RECEIVE); + r.w32(reg::OTHER_PENDING, KBC_IRQ); + assert!(!r.pin.level()); + key(&r, winit::keyboard::KeyCode::KeyA, true); + assert!(r.pin.level()); +} + +// --------------------------------------------------------------------------- +// Reported, not refused +// --------------------------------------------------------------------------- + +/// Everything the `ioc` log module has said in this process. Tests run side +/// by side and share it, so a check looks for its own message text. +fn device_log() -> String { + use std::io::Write; + struct Sink(Arc>>); + impl Write for Sink { + fn write(&mut self, b: &[u8]) -> std::io::Result { + self.0.lock().unwrap().extend_from_slice(b); + Ok(b.len()) + } + fn flush(&mut self) -> std::io::Result<()> { + Ok(()) + } + } + static LOG: std::sync::OnceLock>>> = std::sync::OnceLock::new(); + let log = LOG.get_or_init(|| { + let buf = Arc::new(std::sync::Mutex::new(Vec::new())); + let dl = crate::devlog::DEVLOG.get().cloned().unwrap_or_else(crate::devlog::init_devlog); + dl.add_sink(Arc::new(parking_lot::Mutex::new(Sink(buf.clone())))); + dl.enable(crate::devlog::LogModule::Ioc); + buf + }); + String::from_utf8_lossy(&log.lock().unwrap()).into_owned() +} + +/// Ports test_dma_reports_bad_base_and_pointer: a misaligned pair base and a +/// base change on a busy pair are reported, not refused; the enable and the +/// new base both take effect, and an index keeps its pointer field while the +/// stray bits are reported. +#[test] +fn a_bad_base_and_a_stray_pointer_bit_are_reported_not_refused() { + let _ = device_log(); + let r = Rig::new(); + r.pair_base(0, RINGS + 8, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + assert_eq!(r.sr(0, sdma_off::CONTROL) & ENABLE, ENABLE); + r.sw(0, sdma_off::CONTROL, 0); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.pair_base(0, RINGS + 0x4000, 0x1000); + assert_eq!(r.r32(reg::SDMA_BASE), (RINGS + 0x4000) as u32 | 1); + r.sw(0, sdma_off::CONTROL, RESET); + r.sw(0, sdma_off::TX_PRODUCE, 4); + assert_eq!(r.sr(0, sdma_off::TX_PRODUCE), 0); + r.sw(0, sdma_off::RX_TIMER, 0x1_0002); + r.sw(0, sdma_off::CONTROL, 1 << 20); + let log = device_log(); + for what in [ + "misaligned for its rings", + "changed while it is active", + "TX producer 0x00000004 on port 0 has bits outside the pointer field", + "unsupported RX timer bits 0x00010000 on port 0", + "serial control bits 0x00100000 on port 0 are stored but not modelled", + ] { + assert!(log.contains(what), "no report of: {what}"); + } +} + +/// Ports test_dma_mode_switch_over_pio_is_reported and +/// test_dma_tx_disable_reports_pending_delay: the UART orders PIO and ring +/// DMA itself, so enabling DMA over queued PIO data, a PIO transmit while DMA +/// is on, and clearing the enable over a running delay are reported and go +/// ahead. +#[test] +fn a_mode_switch_over_pio_is_reported_and_goes_ahead() { + let _ = device_log(); + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.uw(0, 2, 0x07); + r.type_in(0, &[0x5a]); + assert_eq!(r.ur(0, 5) & 1, 1); + r.sw(0, sdma_off::CONTROL, ENABLE); + assert_eq!(r.sr(0, sdma_off::CONTROL) & ENABLE, ENABLE, "the enable took effect"); + r.uw(0, 0, 0xa5); + assert_eq!(r.sent(0), [0xa5], "and so did the PIO byte"); + + r.tx_entry(0, 0x1000, 0, [30, 0, 0, 0], [TX_DELAY, 0, 0, 0]); + r.sw(0, sdma_off::TX_PRODUCE, 8); + r.sw(0, sdma_off::CONTROL, 0); + assert_eq!(r.sr(0, sdma_off::CONTROL) & ENABLE, 0); + let log = device_log(); + assert!(log.contains("with PIO data in the UART's FIFOs")); + assert!(log.contains("PIO transmit on port 0 while DMA is enabled")); + assert!(log.contains("DMA disabled on port 0 with ring work pending")); +} + +/// Ports the undefined-bits check of test_dma_tx_control_decodes_type_and_bit5. +#[test] +fn undefined_transmit_control_bits_are_reported() { + let _ = device_log(); + let r = Rig::new(); + r.pair_base(0, RINGS, 0x1000); + r.sw(0, sdma_off::CONTROL, ENABLE); + r.tx_entry(0, 0x1000, 0, [b'q', 0, 0, 0], [TX_VALID | 0x10, 0, 0, 0]); + r.sw(0, sdma_off::TX_PRODUCE, 8); + assert_eq!(r.sent(0), b"q", "the slot is obeyed by its type"); + assert!(device_log().contains("undefined TX control bits 0x10 in an entry on port 0")); +} diff --git a/src/dev/ioc_serial_dma.rs b/src/dev/ioc_serial_dma.rs new file mode 100644 index 000000000..31c276a33 --- /dev/null +++ b/src/dev/ioc_serial_dma.rs @@ -0,0 +1,1124 @@ +//! The serial ring-DMA engine of the SGI IOC3 and IOC4. +//! +//! Both chips put the same engine in front of each of their 16550-compatible +//! ports: seven 32-bit registers per port, a ring of eight-byte entries in +//! host memory for each direction, and a 64-bit ring base shared by a pair of +//! ports. The transmit walk writes the port's 16550; the receive side drains +//! its FIFO into a packer that becomes one ring entry. A ring's address is +//! derived from its port and direction, and no register reads it back. +//! +//! The chip around the engine owns its register window, its interrupt status +//! words and the 16550s. The engine reaches them through [`SerialDmaHost`], so the same +//! walk serves the IOC3 (two ports, nine interrupt bits each) and the IOC4 +//! (four ports, eight bits each, DMA errors reported elsewhere). +//! +//! # What differs between the chips +//! +//! [`SerialDmaParams`] holds what the two do differently and nothing else: +//! +//! * IOC3 reports the packer's held-character count in the receive producer +//! index. IOC4's register map does not define the field, so whether the +//! chip has it is not known. +//! * IOC4 stops a port after a failed transfer until a channel reset (its +//! error sources are named "serial halted by mem err"). Whether an IOC3 +//! port halts is not known. +//! +//! # What is approximated here +//! +//! * The receive timer and a transmit delay run on host time. A pause freezes +//! both where they stand and a resume lets each finish the time it had +//! left. No source says what the part does. +//! * Receive-ring-full is a level that follows a held entry; what the part +//! does is not known. +//! +//! # Mapping to QEMU +//! +//! The split is QEMU's: this file is the shared engine, `ioc3.rs` and +//! `ioc4.rs` are the chips that use it. +//! +//! | QEMU (`hw/char/sgi-ioc-serial-dma.c`) | here | +//! |---|---| +//! | `SGIIOCSerialDMA`, `SGIIOCSerialDMAPort` | [`SerialDma`], [`SerialDmaPort`] | +//! | `SGIIOCSerialDMAParams` | [`SerialDmaParams`] | +//! | `SGIIOCSerialDMAOps` and its `opaque` | [`SerialDmaHost`], implemented by the chip | +//! | `SGIIOCSerialDMASource` | [`SerialDmaSource`] | +//! | the port's `SerialState *uart` | [`SerialDmaUart`] | +//! | `sgi_ioc_sdma_port_read`, `_port_write`, `_kick`, `_catch_up`, `_reset` | `port_read`, `port_write`, `run`, `catch_up`, `reset` | +//! | `sgi_ioc_sdma_base_read`, `_base_write` | `base_read`, `base_write` | +//! +//! Ported from hw/char/sgi-ioc-serial-dma.c in jamesbraid/qemu-sgi-origami, +//! in the form the IOC3 had before the engine was shared. + +use std::time::{Duration, Instant}; + +use crate::dev::ioc3::{due, span, unspan, until, usize_field}; +use crate::snapshot::{field, field_bool, field_u32, field_u32s, field_u64, field_u8s, StateTable}; + +use crate::dev::ns16550::Ns16550; + +use crate::dev::ns16550::com_reg::{ + LSR_BI, LSR_DR, LSR_FE, LSR_OE, LSR_PE, MCR_DTR, MCR_LOOP, MCR_OUT1, MCR_OUT2, MCR_RTS, MSR_CTS, MSR_DCD, + MSR_DCTS, MSR_DDCD, +}; +use crate::dev::ns16550::com_reg::{LSR as UART_LSR, MCR as UART_MCR, MSR as UART_MSR, THR as UART_THR}; + +/// Port DMA register offsets (bytes, from the port's first register). +pub mod off { + pub const CONTROL: u32 = 0x00; + pub const TX_PRODUCE: u32 = 0x04; + pub const TX_CONSUME: u32 = 0x08; + pub const RX_PRODUCE: u32 = 0x0c; + pub const RX_CONSUME: u32 = 0x10; + pub const RX_TIMER: u32 = 0x14; + pub const SHADOW: u32 = 0x18; + /// Registers per port. + pub const REGS: u32 = 7; +} + +/// Bits of the port's control register. +pub mod ctl { + pub const HIGH_WATER: u32 = 0x1ff; + pub const TX_TIMER_RUN: u32 = 1 << 16; + pub const FLOW_CONTROL: u32 = 1 << 17; + pub const RX_TO_MEMORY: u32 = 1 << 27; + pub const ENABLE: u32 = 1 << 28; + pub const PAUSE: u32 = 1 << 29; + pub const PAUSED: u32 = 1 << 30; + pub const RESET: u32 = 1 << 31; + /// Status and commands: these never read back as written. + pub const UNSTORED: u32 = RESET | RX_TO_MEMORY | PAUSED | TX_TIMER_RUN; +} + +/// The most ports one engine drives. +pub const MAX_PORTS: usize = 4; +/// Ports sharing one ring base. +const PAIR: usize = 2; + +pub const ENTRY: usize = 8; +pub const ENTRY_CHARS: usize = 4; +/// Bit 31 of the receive consumer index arms the receive timer. +pub const ARM_RX_TIMER: u32 = 1 << 31; +/// The packer's held count, in the receive producer index (IOC3 only). +pub const HELD_SHIFT: u32 = 24; +const TIMER_RELOAD: u32 = 0xfff; +const TIMER_COUNT_SHIFT: u32 = 16; +pub const TIMER_TICK_NS: u64 = 62_500; // 16 kHz +const RING_4K: u32 = 1; +const RING_ADDRESS: u32 = 0xffff_f000; +const RING_1K_SIZE: u32 = 1024; +const RING_4K_SIZE: u32 = 4096; +const TX_TYPE: u8 = 0xc0; +const TX_CHARACTER: u8 = 0x40; +const TX_MODEM: u8 = 0x80; +pub const TX_DELAY: u8 = 0xc0; +pub const TX_INTERRUPT: u8 = 0x20; +const TX_UNDEFINED: u8 = 0x1f; +const RX_OVERRUN: u8 = 0x01; +const RX_PARITY: u8 = 0x02; +const RX_FRAMING: u8 = 0x04; +const RX_BREAK: u8 = 0x08; +const RX_CTS: u8 = 0x10; +const RX_CD: u8 = 0x20; +const RX_MODEM_OK: u8 = 0x40; +const RX_DATA_OK: u8 = 0x80; + +/// A port's interrupt sources, numbered as the IOC3 lays them out in its +/// status word. The IOC4 places all but the DMA error the same way. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum SerialDmaSource { + TxEmpty = 0, + RxFull = 1, + RxHighWater = 2, + RxTimer = 3, + TxExplicit = 7, + DmaError = 8, +} + +/// What differs between the chips, fixed per chip. +#[derive(Clone, Copy, Debug)] +pub struct SerialDmaParams { + /// The receive producer index reports the packer's held count. + pub held_counts: bool, + /// A failed transfer stops the port until a channel reset. + pub halt_on_dma_error: bool, +} + +/// A 16550 as the engine drives it. +pub trait SerialDmaUart { + fn loopback(&self) -> bool; + /// The port can take a character from the engine. The walk waits while + /// it cannot, and goes on when the chip next runs the engine. + fn tx_ready(&self) -> bool; + /// How long a character takes on the line at the programmed rate. + fn char_time(&self) -> Duration; + /// A read with the chip's side effects left out: the 16550 proper. + fn read_core(&mut self, reg: u32) -> u8; + fn write(&mut self, reg: u32, val: u8); + fn lsr(&self) -> u8; + fn mcr(&self) -> u8; + fn msr(&self) -> u8; +} + +/// The chip around the engine. +pub trait SerialDmaHost { + fn uart(&mut self, port: usize) -> &mut dyn SerialDmaUart; + /// Latch a source; the chip's write-one-to-clear acknowledges it. + fn raise(&mut self, port: usize, src: SerialDmaSource); + /// Drive a source that follows a level and ignores acknowledgement. + fn set_level(&mut self, port: usize, src: SerialDmaSource, level: bool); + /// The port's 16550 is fitted and the engine is aimed at it. + fn reachable(&self, port: usize) -> bool; + /// The 16550's interrupt line may have moved: reflect it in the chip's + /// pass-through source. + fn uart_line_changed(&mut self, port: usize); + fn dma_read(&self, addr: u64, buf: &mut [u8]) -> bool; + fn dma_write(&self, addr: u64, data: &[u8]) -> bool; + /// A transfer failed at `addr`: a receive post or a transmit fetch. + fn dma_error(&mut self, port: usize, _rx: bool, _addr: u64) { + self.raise(port, SerialDmaSource::DmaError); + } +} + +#[derive(Default)] +pub struct SerialDmaPort { + reg: [u32; off::REGS as usize], + packer: [u8; ENTRY], + packed: usize, + tx_entry: [u8; ENTRY], + tx_fetched: bool, + tx_slot: usize, + armed: bool, + drain_pending: bool, + busy: bool, + halted: bool, + /// The last post was turned away for lack of ring room. + post_refused: bool, + paused: bool, + pub rx_deadline: Option, + rx_idle_count: u32, + /// Time a frozen timer had left when a pause began. + rx_frozen: Option, + tx_delay_deadline: Option, + tx_delay_frozen: Option, + tx_delay_irq: bool, + /// A looped-back character still on the line: when it reaches the + /// receiver. The 16550 core loops it back the moment it is written, and + /// the engine does not look at the receiver until this time. + loop_due: Option, +} + +pub struct SerialDma { + params: SerialDmaParams, + base_hi: [u32; MAX_PORTS / PAIR], + base_lo: [u32; MAX_PORTS / PAIR], + /// Transfers under way, for the chip's idle status. Nonzero only inside + /// a call: a chip cannot read its own status mid-walk, so software sees + /// idle unless a bus master it reaches re-enters the chip. + in_flight: u32, + pub port: Vec, +} + +impl SerialDma { + pub fn new(ports: usize, params: SerialDmaParams) -> SerialDma { + assert!((1..=MAX_PORTS).contains(&ports)); + SerialDma { + params, + base_hi: [0; MAX_PORTS / PAIR], + base_lo: [0; MAX_PORTS / PAIR], + in_flight: 0, + port: (0..ports).map(|_| SerialDmaPort::default()).collect(), + } + } + + /// Back to the power-on state: the ring bases and every port. + pub fn reset(&mut self) { + *self = SerialDma::new(self.port.len(), self.params); + } + + pub fn base_read(&self, pair: usize, high: bool) -> u32 { + if high { self.base_hi[pair] } else { self.base_lo[pair] } + } + + pub fn base_write(&mut self, pair: usize, high: bool, val: u32) { + if high { + self.base_hi[pair] = val; + } else { + self.base_lo[pair] = val; + } + } + + /// A port's control register as stored. + pub fn control(&self, p: usize) -> u32 { + self.port[p].reg[0] + } + + /// A register as stored, without the status the read adds. + pub fn stored(&self, p: usize, offset: u32) -> u32 { + self.port[p].reg[(offset / 4) as usize] + } + + /// When the port's receive timer runs out, if it is running. + pub fn rx_deadline(&self, p: usize) -> Option { + self.port[p].rx_deadline + } + + /// When the port's transmit delay ends. A paused port keeps it frozen. + pub fn tx_delay_deadline(&self, p: usize) -> Option { + let port = &self.port[p]; + if port.paused { None } else { port.tx_delay_deadline } + } + + /// No transfer is under way: what a chip's DMA idle status reports. + pub fn idle(&self) -> bool { + self.in_flight == 0 + } + + /// A ring read, counted as in flight while the bus master has it. + fn dma_read(&mut self, host: &mut dyn SerialDmaHost, addr: u64, buf: &mut [u8]) -> bool { + self.in_flight += 1; + let ok = host.dma_read(addr, buf); + self.in_flight -= 1; + ok + } + + /// A ring write, counted as in flight while the bus master has it. + fn dma_write(&mut self, host: &mut dyn SerialDmaHost, addr: u64, data: &[u8]) -> bool { + self.in_flight += 1; + let ok = host.dma_write(addr, data); + self.in_flight -= 1; + ok + } + + /// When a looped-back character reaches the receiver, if one is on the + /// line. + pub fn loop_deadline(&self, p: usize) -> Option { + self.port[p].loop_due + } + + /// Nothing queued, held or pending on the port: what a chip that wants + /// the rings left alone while they move checks before it lets a guest + /// change a mode. + pub fn port_quiescent(&self, p: usize) -> bool { + let port = &self.port[p]; + port.reg[1] == port.reg[2] + && port.reg[3] == port.reg[4] + && !port.tx_fetched + && port.tx_delay_deadline.is_none() + && port.packed == 0 + && !port.armed + && !port.drain_pending + } + + /// Both ports of a pair are off, with their indices at zero and nothing + /// pending. + pub fn pair_quiescent(&self, pair: usize) -> bool { + (pair * PAIR..pair * PAIR + PAIR).all(|p| { + self.control(p) & (ctl::ENABLE | ctl::PAUSE) == 0 + && self.port[p].reg[1..=4].iter().all(|&r| r == 0) + && self.port_quiescent(p) + }) + } + + /// Whether the port is stopped by a failed transfer. + pub fn halted(&self, p: usize) -> bool { + self.port[p].halted + } + + /// Looped back with the engine's hardware flow control on, the port + /// reports clear-to-send asserted even with request-to-send clear. + pub fn flow_cts(&self, p: usize, loopback: bool) -> bool { + loopback && self.control(p) & ctl::FLOW_CONTROL != 0 + } + + fn ring_size(&self, p: usize) -> u32 { + if self.base_lo[p / PAIR] & RING_4K != 0 { RING_4K_SIZE } else { RING_1K_SIZE } + } + + /// The selected ring size is also every index register's pointer width. + fn mask(&self, p: usize) -> u32 { + (self.ring_size(p) - 1) & !(ENTRY as u32 - 1) + } + + /// The size-select bit is not address. + fn ring(&self, p: usize, dir: u32) -> u64 { + let pair = p / PAIR; + let base = (u64::from(self.base_hi[pair]) << 32) | u64::from(self.base_lo[pair] & RING_ADDRESS); + base.saturating_add(((p % PAIR) as u64 * 2 + u64::from(dir)) * u64::from(self.ring_size(p))) + } + + fn running(&self, p: usize) -> bool { + let c = self.control(p); + !self.port[p].halted && c & ctl::ENABLE != 0 && c & ctl::PAUSE == 0 + } + + /// A transfer the fabric refused: the chip reports it, and may halt. + fn dma_failed(&mut self, p: usize, rx: bool, addr: u64, host: &mut dyn SerialDmaHost) { + if self.params.halt_on_dma_error { + self.port[p].halted = true; + } + host.dma_error(p, rx, addr); + } + + /// Transmit-ring-empty is a level: ring DMA on, nothing left in the ring + /// or held by the engine. + fn update_tx_empty(&self, p: usize, host: &mut dyn SerialDmaHost) { + let mask = self.mask(p); + let port = &self.port[p]; + let empty = self.control(p) & ctl::ENABLE != 0 + && (port.reg[1] ^ port.reg[2]) & mask == 0 + && !port.tx_fetched + && port.tx_delay_deadline.is_none(); + host.set_level(p, SerialDmaSource::TxEmpty, empty); + } + + /// The receive high-water mark counts ring entries; a level. So is + /// receive-ring-full: set while ring DMA is on and an entry waits, turned + /// away for lack of ring room. It does not come back with every refused + /// retry after software has acknowledged it, and it falls once the + /// consumer makes room, whether or not the held entry has posted yet. A + /// ring filled exactly, with nothing waiting, does not raise it. A model + /// choice; what the part does is not known. + fn update_high_water(&mut self, p: usize, host: &mut dyn SerialDmaHost) { + let mask = self.mask(p); + let hw = self.control(p) & ctl::HIGH_WATER; + let enabled = self.control(p) & ctl::ENABLE != 0; + let port = &mut self.port[p]; + let waiting = port.reg[3].wrapping_sub(port.reg[4]) & mask; + host.set_level(p, SerialDmaSource::RxHighWater, hw != 0 && waiting / ENTRY as u32 >= hw); + if (port.reg[3] + ENTRY as u32) & mask != port.reg[4] & mask { + port.post_refused = false; + } + host.set_level(p, SerialDmaSource::RxFull, enabled && port.post_refused); + } + + /// Post the packer as one entry. Fails, keeping the characters, when the + /// ring is full or the write is refused. + fn post(&mut self, p: usize, host: &mut dyn SerialDmaHost) -> bool { + let mask = self.mask(p); + let ring = self.ring(p, 1); + let (produce, consume, packer) = { + let port = &self.port[p]; + (port.reg[3], port.reg[4], port.packer) + }; + let next = (produce + ENTRY as u32) & mask; + if next == consume & mask { + // Full: refuse. The characters keep in the packer and the FIFO. + self.port[p].post_refused = true; + return false; + } + let at = ring.saturating_add(u64::from(produce & mask)); + if !self.dma_write(host, at, &packer) { + self.dma_failed(p, true, at, host); + return false; + } + let port = &mut self.port[p]; + port.reg[3] = next; + port.post_refused = false; + port.packed = 0; + port.packer = [0; ENTRY]; + true + } + + fn rx_timer_start(&mut self, p: usize, since: Instant) { + let port = &mut self.port[p]; + let reload = port.reg[5] & TIMER_RELOAD; + port.rx_idle_count = 0; + let dur = Duration::from_nanos(u64::from(reload) * TIMER_TICK_NS); + if port.paused { + // Started while paused, it waits frozen at its full count. + port.rx_frozen = Some(dur); + } else { + port.rx_deadline = Some(since + dur); + } + } + + fn rx_timer_expired(&mut self, p: usize, host: &mut dyn SerialDmaHost) { + self.port[p].rx_idle_count = 0; + self.port[p].rx_deadline = None; + if self.port[p].packed != 0 { + self.post(p, host); + } + // Raised at every expiry, even over an empty ring; the arm is spent. + if self.port[p].armed { + host.raise(p, SerialDmaSource::RxTimer); + } + self.port[p].armed = false; + self.update_high_water(p, host); + } + + /// Expiries already due happen before an access. + pub fn catch_up(&mut self, now: Instant, host: &mut dyn SerialDmaHost) { + for p in 0..self.port.len() { + if let Some(d) = self.port[p].tx_delay_deadline { + if !self.port[p].paused && d <= now { + self.tx_delay_ended(p, d, host); + } + } + // Each arrival can send the next character; the ring bounds it. + while let Some(d) = self.port[p].loop_due.filter(|&d| d <= now) { + self.loop_arrived(p, d, host); + } + if let Some(d) = self.port[p].rx_deadline { + if d <= now { + self.rx_timer_expired(p, host); + } + } + } + } + + /// A looped-back character reaches the receiver; the walk goes on from + /// then. + fn loop_arrived(&mut self, p: usize, when: Instant, host: &mut dyn SerialDmaHost) { + self.port[p].loop_due = None; + self.run_from(p, when, host); + } + + fn tx_delay_ended(&mut self, p: usize, when: Instant, host: &mut dyn SerialDmaHost) { + let irq = { + let port = &mut self.port[p]; + port.tx_delay_deadline = None; + std::mem::take(&mut port.tx_delay_irq) + }; + if irq { + host.raise(p, SerialDmaSource::TxExplicit); + } + self.run_from(p, when, host); + } + + fn receive(&mut self, p: usize, since: Instant, host: &mut dyn SerialDmaHost) { + // The 16550 core loops a character back the moment it is written; on + // the part it takes a character time on the line to arrive. + if self.port[p].loop_due.is_some() { + return; + } + while self.running(p) { + if self.port[p].packed == ENTRY_CHARS && !self.post(p, host) { + break; + } + // Line status before the character: reading it clears the latches. + let uart = host.uart(p); + let lsr = uart.read_core(UART_LSR); + if lsr & LSR_DR == 0 { + break; + } + let data = uart.read_core(UART_THR); + let mut msr = uart.read_core(UART_MSR); + let looped = uart.loopback(); + if self.flow_cts(p, looped) { + msr |= MSR_CTS; + } + let mut status = RX_DATA_OK | RX_MODEM_OK; + for (bit, flag) in [(LSR_OE, RX_OVERRUN), (LSR_PE, RX_PARITY), (LSR_FE, RX_FRAMING), (LSR_BI, RX_BREAK)] { + if lsr & bit != 0 { + status |= flag; + } + } + if msr & MSR_CTS != 0 { + status |= RX_CTS; + } + if msr & MSR_DCD != 0 { + status |= RX_CD; + } + let start_timer = { + let port = &mut self.port[p]; + port.packer[port.packed] = data; + port.packer[ENTRY_CHARS + port.packed] = status; + port.packed += 1; + port.armed && port.rx_deadline.is_none() && port.rx_frozen.is_none() + }; + // Started by the first character after the arm; later ones leave + // a running timer alone. + if start_timer { + self.rx_timer_start(p, since); + } + // A fourth character completes the entry; fewer wait for the + // timer or a drain. + if self.port[p].packed == ENTRY_CHARS { + self.post(p, host); + } + } + host.uart_line_changed(p); + } + + /// The type field and bit 5 decide what a slot does; the low five bits + /// have no defined meaning and are reported, not obeyed. Report them once, + /// when the entry is fetched, for the slots up to the one that ends it. + fn check_entry(p: usize, entry: &[u8; ENTRY]) { + let mut undefined = 0; + for &control in &entry[ENTRY_CHARS..] { + undefined |= control & TX_UNDEFINED; + if control & TX_TYPE == 0 { + break; + } + } + if undefined != 0 { + crate::dlog!( + crate::devlog::LogModule::Ioc, + "serial DMA: undefined TX control bits {undefined:#04x} in an entry on port {p}" + ); + } + } + + /// Each control byte types its data byte; the empty type ends the entry, + /// wherever it sits. + fn transmit(&mut self, p: usize, since: Instant, host: &mut dyn SerialDmaHost) { + let mask = self.mask(p); + let ring = self.ring(p, 0); + if self.port[p].tx_delay_deadline.is_some() { + return; + } + loop { + let (produce, consume) = { + let port = &self.port[p]; + (port.reg[1], port.reg[2]) + }; + if !(self.running(p) && consume & mask != produce & mask) { + break; + } + let was_fetched = self.port[p].tx_fetched; + if !was_fetched { + let mut entry = [0u8; ENTRY]; + let at = ring.saturating_add(u64::from(consume & mask)); + if !self.dma_read(host, at, &mut entry) { + self.dma_failed(p, false, at, host); + break; + } + let port = &mut self.port[p]; + port.tx_entry = entry; + port.tx_fetched = true; + } + let entry = self.port[p].tx_entry; + if !was_fetched { + Self::check_entry(p, &entry); + } + let mut slot = self.port[p].tx_slot; + while slot < ENTRY_CHARS { + let control = entry[ENTRY_CHARS + slot]; + let ty = control & TX_TYPE; + if ty == 0 { + break; + } + match ty { + TX_CHARACTER => { + // In loopback the line carries one character at a time. + if !host.uart(p).tx_ready() || self.port[p].loop_due.is_some() { + return; + } + let looped = host.uart(p).loopback(); + // What has already arrived goes ahead of this one. + if looped { + self.receive(p, since, host); + } + host.uart(p).write(UART_THR, entry[slot]); + if looped { + self.port[p].loop_due = Some(since + host.uart(p).char_time()); + } + // Drain per character or loopback overruns the FIFO. + self.receive(p, since, host); + } + TX_MODEM => host.uart(p).write(UART_MCR, entry[slot]), + TX_DELAY => { + // The data byte is milliseconds of quiet. + if entry[slot] != 0 { + let port = &mut self.port[p]; + port.tx_slot = slot + 1; + port.tx_delay_deadline = Some(since + Duration::from_millis(u64::from(entry[slot]))); + port.tx_delay_irq = control & TX_INTERRUPT != 0; + return; + } + } + _ => unreachable!("the type field has two bits"), + } + slot += 1; + self.port[p].tx_slot = slot; + if control & TX_INTERRUPT != 0 { + host.raise(p, SerialDmaSource::TxExplicit); + } + } + let port = &mut self.port[p]; + port.tx_slot = 0; + port.tx_fetched = false; + port.reg[2] = (port.reg[2] + ENTRY as u32) & mask; + } + host.uart_line_changed(p); + } + + /// The drain posts what the packer holds and reads back set until it has; + /// it waits while ring DMA is off or paused and while the ring is full. + fn drain(&mut self, p: usize, host: &mut dyn SerialDmaHost) { + if !self.port[p].drain_pending || !self.running(p) { + return; + } + if self.port[p].packed != 0 && !self.post(p, host) { + return; + } + self.port[p].drain_pending = false; + } + + /// Walk the port's rings from `since`: where a transmit delay ended, or + /// now. + pub fn run_from(&mut self, p: usize, since: Instant, host: &mut dyn SerialDmaHost) { + if !host.reachable(p) || self.port[p].busy { + return; + } + self.port[p].busy = true; + self.transmit(p, since, host); + self.receive(p, since, host); + self.drain(p, host); + self.update_high_water(p, host); + self.update_tx_empty(p, host); + self.port[p].busy = false; + } + + pub fn run(&mut self, p: usize, host: &mut dyn SerialDmaHost) { + self.run_from(p, Instant::now(), host); + } + + /// A passive mirror of the 16550's lines; no read side effects. + fn shadow(&self, p: usize, host: &mut dyn SerialDmaHost) -> u32 { + let u = host.uart(p); + let (lsr, mcr, mut msr) = (u.lsr(), u.mcr(), u.msr()); + if self.flow_cts(p, u.loopback()) { + msr |= MSR_CTS; + } + let mut s = u32::from(lsr); + for (m, bit) in [(MSR_DCTS, 16), (MSR_DDCD, 19), (MSR_CTS, 20), (MSR_DCD, 23)] { + if msr & m != 0 { + s |= 1 << bit; + } + } + for (m, bit) in [(MCR_DTR, 24), (MCR_RTS, 25), (MCR_OUT1, 26), (MCR_OUT2, 27), (MCR_LOOP, 28)] { + if mcr & m != 0 { + s |= 1 << bit; + } + } + s + } + + pub fn port_read(&mut self, p: usize, offset: u32, host: &mut dyn SerialDmaHost) -> u32 { + let now = Instant::now(); + self.catch_up(now, host); + if offset == off::SHADOW { + return self.shadow(p, host); + } + let port = &self.port[p]; + let mut v = port.reg[(offset / 4) as usize]; + match offset { + off::CONTROL => { + if v & ctl::PAUSE != 0 { + v |= ctl::PAUSED; + } + if port.tx_delay_deadline.is_some() { + v |= ctl::TX_TIMER_RUN; + } + if port.drain_pending { + v |= ctl::RX_TO_MEMORY; + } + } + off::RX_CONSUME => { + if port.armed { + v |= ARM_RX_TIMER; + } + } + off::RX_PRODUCE if self.params.held_counts => v |= (port.packed as u32) << HELD_SHIFT, + off::RX_TIMER => { + // Ticks left while running or frozen; idle, whatever a reload + // write last loaded. + let ticks_in = |d: Duration| (d.as_nanos() as u64 / TIMER_TICK_NS).min(u64::from(TIMER_RELOAD)) as u32; + let mut ticks = port.rx_idle_count; + if let (true, Some(left)) = (port.paused, port.rx_frozen) { + ticks = ticks_in(left); + } else if let Some(d) = port.rx_deadline { + ticks = ticks_in(d.saturating_duration_since(now)); + } + v |= ticks << TIMER_COUNT_SHIFT; + } + _ => {} + } + v + } + + /// Resets the four indices and the work in progress; the settings + /// survive, and a port a failed transfer stopped runs again. + fn reset_port(&mut self, p: usize) { + let port = &mut self.port[p]; + port.halted = false; + port.reg[1..=4].fill(0); + port.armed = false; + port.drain_pending = false; + port.post_refused = false; + port.packed = 0; + port.packer = [0; ENTRY]; + port.tx_slot = 0; + port.tx_fetched = false; + port.tx_delay_deadline = None; + port.tx_delay_irq = false; + port.rx_deadline = None; + port.rx_frozen = None; + port.rx_idle_count = 0; + } + + pub fn port_write(&mut self, p: usize, offset: u32, val: u32, host: &mut dyn SerialDmaHost) { + let now = Instant::now(); + self.catch_up(now, host); + let mask = self.mask(p); + match offset { + off::CONTROL => { + self.port[p].reg[0] = val & !ctl::UNSTORED; + if val & ctl::RESET != 0 { + self.reset_port(p); + } + if val & ctl::RX_TO_MEMORY != 0 { + self.port[p].drain_pending = true; + } + let port = &mut self.port[p]; + // A pause freezes the receive timer and a transmit delay; + // resuming lets each finish the time it had left. A model + // choice; no source says what the part does. + if val & ctl::PAUSE != 0 && !port.paused { + port.paused = true; + port.rx_frozen = port.rx_deadline.take().map(|d| d.saturating_duration_since(now)); + port.tx_delay_frozen = port.tx_delay_deadline.take().map(|d| d.saturating_duration_since(now)); + } else if val & ctl::PAUSE == 0 && port.paused { + port.paused = false; + port.rx_deadline = port.rx_frozen.take().map(|l| now + l); + port.tx_delay_deadline = port.tx_delay_frozen.take().map(|l| now + l); + } + if val & ctl::ENABLE == 0 { + port.rx_deadline = None; + port.rx_frozen = None; + } + self.run(p, host); + } + off::TX_PRODUCE | off::TX_CONSUME | off::RX_PRODUCE => { + self.port[p].reg[(offset / 4) as usize] = val & mask; + self.run(p, host); + } + off::RX_CONSUME => { + // Bit 31 sets the arm; a write with it clear leaves it. + self.port[p].reg[4] = val & mask; + if val & ARM_RX_TIMER != 0 { + self.port[p].armed = true; + } + self.run(p, host); + } + off::RX_TIMER => { + let port = &mut self.port[p]; + port.reg[5] = val & TIMER_RELOAD; + port.rx_idle_count = val & TIMER_RELOAD; + } + // The shadow discards writes silently. + _ => {} + } + self.update_tx_empty(p, host); + self.update_high_water(p, host); + } +} + +impl SerialDmaUart for Ns16550 { + fn loopback(&self) -> bool { + Ns16550::loopback(self) + } + + fn tx_ready(&self) -> bool { + Ns16550::tx_ready(self) + } + + fn char_time(&self) -> Duration { + Ns16550::char_time(self) + } + + fn read_core(&mut self, r: u32) -> u8 { + self.read(r) + } + + fn write(&mut self, r: u32, v: u8) { + Ns16550::write(self, r, v); + } + + fn lsr(&self) -> u8 { + Ns16550::lsr(self) + } + + fn mcr(&self) -> u8 { + Ns16550::mcr(self) + } + + fn msr(&self) -> u8 { + Ns16550::msr(self) + } +} + +#[cfg(test)] +mod tests { + //! The engine against a stand-in chip. The IOC3 and IOC4 register tests + //! cover the engine through each chip; these cover what only the second + //! chip's parameters change. + + use super::*; + + #[derive(Default)] + struct FakeUart { + sent: Vec, + rx: std::collections::VecDeque, + } + + impl SerialDmaUart for FakeUart { + fn loopback(&self) -> bool { + false + } + fn tx_ready(&self) -> bool { + true + } + fn char_time(&self) -> Duration { + Duration::ZERO + } + fn read_core(&mut self, reg: u32) -> u8 { + match reg { + UART_LSR => u8::from(!self.rx.is_empty()), + UART_THR => self.rx.pop_front().unwrap_or(0), + _ => 0, + } + } + fn write(&mut self, reg: u32, val: u8) { + if reg == UART_THR { + self.sent.push(val); + } + } + fn lsr(&self) -> u8 { + 0 + } + fn mcr(&self) -> u8 { + 0 + } + fn msr(&self) -> u8 { + 0 + } + } + + struct Chip { + uart: [FakeUart; MAX_PORTS], + mem: Vec, + /// Transfers below this address are refused. + refuse_below: u64, + raised: Vec<(usize, SerialDmaSource)>, + errors: Vec<(usize, bool, u64)>, + /// Whether the chip reports errors itself, as IOC4 does. + reports_errors: bool, + } + + impl Chip { + fn new() -> Chip { + Chip { + uart: Default::default(), + mem: vec![0; 0x4000], + refuse_below: 0, + raised: Vec::new(), + errors: Vec::new(), + reports_errors: true, + } + } + } + + impl SerialDmaHost for Chip { + fn uart(&mut self, port: usize) -> &mut dyn SerialDmaUart { + &mut self.uart[port] + } + fn raise(&mut self, port: usize, src: SerialDmaSource) { + self.raised.push((port, src)); + } + fn set_level(&mut self, _port: usize, _src: SerialDmaSource, _level: bool) {} + fn reachable(&self, _port: usize) -> bool { + true + } + fn uart_line_changed(&mut self, _port: usize) {} + fn dma_read(&self, addr: u64, buf: &mut [u8]) -> bool { + if addr < self.refuse_below { + return false; + } + buf.copy_from_slice(&self.mem[addr as usize..addr as usize + buf.len()]); + true + } + fn dma_write(&self, _addr: u64, _data: &[u8]) -> bool { + true + } + fn dma_error(&mut self, port: usize, rx: bool, addr: u64) { + if self.reports_errors { + self.errors.push((port, rx, addr)); + } else { + self.raise(port, SerialDmaSource::DmaError); + } + } + } + + const FOUR: SerialDmaParams = SerialDmaParams { held_counts: false, halt_on_dma_error: true }; + const TWO: SerialDmaParams = SerialDmaParams { held_counts: true, halt_on_dma_error: false }; + + /// One transmit entry holding `byte`, in the port's transmit ring. + fn queue_char(sdma: &mut SerialDma, chip: &mut Chip, p: usize, byte: u8) { + let at = sdma.ring(p, 0) as usize; + chip.mem[at..at + ENTRY].copy_from_slice(&[byte, 0, 0, 0, TX_CHARACTER, 0, 0, 0]); + let produce = sdma.stored(p, off::TX_PRODUCE); + sdma.port_write(p, off::TX_PRODUCE, produce + ENTRY as u32, chip); + } + + fn enable(sdma: &mut SerialDma, chip: &mut Chip, p: usize) { + sdma.port_write(p, off::CONTROL, ctl::ENABLE, chip); + } + + #[test] + fn a_port_in_the_second_pair_uses_the_second_ring_base() { + let mut sdma = SerialDma::new(4, FOUR); + let mut chip = Chip::new(); + sdma.base_write(0, false, 0x1000); + sdma.base_write(1, false, 0x2000); + assert_eq!(sdma.ring(0, 0), 0x1000); + assert_eq!(sdma.ring(1, 0), 0x1000 + 2 * 1024); + assert_eq!(sdma.ring(2, 0), 0x2000); + assert_eq!(sdma.ring(3, 1), 0x2000 + 3 * 1024); + enable(&mut sdma, &mut chip, 3); + queue_char(&mut sdma, &mut chip, 3, b'x'); + assert_eq!(chip.uart[3].sent, b"x"); + assert!(chip.uart[0].sent.is_empty(), "the other ports saw nothing"); + } + + #[test] + fn a_failed_fetch_halts_the_port_until_a_channel_reset() { + let mut sdma = SerialDma::new(4, FOUR); + let mut chip = Chip::new(); + sdma.base_write(0, false, 0x1000); + enable(&mut sdma, &mut chip, 0); + chip.refuse_below = 0x2000; + queue_char(&mut sdma, &mut chip, 0, b'a'); + assert_eq!(chip.errors, [(0, false, 0x1000)], "the chip is told where the fetch failed"); + assert!(sdma.halted(0)); + chip.refuse_below = 0; + sdma.run(0, &mut chip); + assert!(chip.uart[0].sent.is_empty(), "a halted port does not resume by itself"); + sdma.port_write(0, off::CONTROL, ctl::ENABLE | ctl::RESET, &mut chip); + assert!(!sdma.halted(0)); + assert_eq!(sdma.stored(0, off::TX_PRODUCE), 0, "the reset cleared the indices"); + queue_char(&mut sdma, &mut chip, 0, b'b'); + assert_eq!(chip.uart[0].sent, b"b"); + } + + #[test] + fn a_failed_fetch_leaves_a_chip_that_does_not_halt_running() { + let mut sdma = SerialDma::new(2, TWO); + let mut chip = Chip::new(); + chip.reports_errors = false; + sdma.base_write(0, false, 0x1000); + enable(&mut sdma, &mut chip, 0); + chip.refuse_below = 0x2000; + queue_char(&mut sdma, &mut chip, 0, b'a'); + assert_eq!(chip.raised.last(), Some(&(0, SerialDmaSource::DmaError)), "the default report is the port's own source"); + assert!(!sdma.halted(0)); + chip.refuse_below = 0; + sdma.run(0, &mut chip); + assert_eq!(chip.uart[0].sent, b"a", "the walk retries once the transfer works"); + } + + #[test] + fn only_a_chip_that_reports_held_counts_shows_them() { + for (params, shown) in [(TWO, true), (FOUR, false)] { + let mut sdma = SerialDma::new(2, params); + let mut chip = Chip::new(); + sdma.base_write(0, false, 0x1000); + enable(&mut sdma, &mut chip, 0); + chip.uart[0].rx.extend([1, 2]); + sdma.run(0, &mut chip); + let produce = sdma.port_read(0, off::RX_PRODUCE, &mut chip); + assert_eq!(produce >> HELD_SHIFT & 7, if shown { 2 } else { 0 }); + } + } + + #[test] + fn the_engine_is_idle_between_calls_whether_a_transfer_worked_or_not() { + let mut sdma = SerialDma::new(4, FOUR); + let mut chip = Chip::new(); + assert!(sdma.idle()); + sdma.base_write(0, false, 0x1000); + enable(&mut sdma, &mut chip, 0); + queue_char(&mut sdma, &mut chip, 0, b'a'); + assert!(sdma.idle(), "after a good transfer"); + chip.refuse_below = 0x2000; + queue_char(&mut sdma, &mut chip, 0, b'b'); + assert!(sdma.halted(0)); + assert!(sdma.idle(), "after a failed one"); + } +} + +// The engine's state for a snapshot of the chip that carries it. Each timer +// is saved as the time it had left (see the IOC3's snapshot helpers). + +impl SerialDmaPort { + fn save_state(&self, now: Instant) -> toml::Value { + StateTable::new() + .u32s("reg", &self.reg) + .u8s("packer", &self.packer) + .u64("packed", self.packed as u64) + .u8s("tx_entry", &self.tx_entry) + .bool("tx_fetched", self.tx_fetched) + .u64("tx_slot", self.tx_slot as u64) + .bool("armed", self.armed) + .bool("drain_pending", self.drain_pending) + .bool("busy", self.busy) + .bool("paused", self.paused) + .u64("rx_due", until(self.rx_deadline, now)) + .u32("rx_idle_count", self.rx_idle_count) + .u64("rx_frozen", span(self.rx_frozen)) + .u64("tx_delay_due", until(self.tx_delay_deadline, now)) + .u64("tx_delay_frozen", span(self.tx_delay_frozen)) + .bool("tx_delay_irq", self.tx_delay_irq) + .bool("post_refused", self.post_refused) + .bool("halted", self.halted) + .u64("loop_due", until(self.loop_due, now)) + .build() + } + fn load_state(v: &toml::Value, now: Instant) -> Result { + Ok(Self { + reg: field_u32s(v, "reg")?, + packer: field_u8s(v, "packer")?, + packed: usize_field(v, "packed", ENTRY)?, + tx_entry: field_u8s(v, "tx_entry")?, + tx_fetched: field_bool(v, "tx_fetched")?, + tx_slot: usize_field(v, "tx_slot", ENTRY)?, + armed: field_bool(v, "armed")?, + drain_pending: field_bool(v, "drain_pending")?, + busy: field_bool(v, "busy")?, + paused: field_bool(v, "paused")?, + rx_deadline: due(field_u64(v, "rx_due")?, now), + rx_idle_count: field_u32(v, "rx_idle_count")?, + rx_frozen: unspan(field_u64(v, "rx_frozen")?), + tx_delay_deadline: due(field_u64(v, "tx_delay_due")?, now), + tx_delay_frozen: unspan(field_u64(v, "tx_delay_frozen")?), + tx_delay_irq: field_bool(v, "tx_delay_irq")?, + post_refused: field_bool(v, "post_refused")?, + halted: field_bool(v, "halted")?, + loop_due: due(field_u64(v, "loop_due")?, now), + }) + } +} + +impl SerialDma { + /// The ring bases and each port's state, timers as the time left after + /// `now`. + pub fn save_state(&self, now: Instant) -> toml::Value { + let mut t = StateTable::new(); + t.u32s("base_hi", &self.base_hi).u32s("base_lo", &self.base_lo); + for (i, port) in self.port.iter().enumerate() { + t.value(&format!("port{i}"), port.save_state(now)); + } + t.build() + } + + /// The engine `save_state` described, with this engine's parameters + /// and port count, timers set that far after `now`. + pub fn loaded(&self, v: &toml::Value, now: Instant) -> Result { + let port = (0..self.port.len()) + .map(|i| SerialDmaPort::load_state(field(v, &format!("port{i}"))?, now)) + .collect::, String>>()?; + // Transfers are under way only inside a call, never at a save. + Ok(SerialDma { params: self.params, base_hi: field_u32s(v, "base_hi")?, base_lo: field_u32s(v, "base_lo")?, in_flight: 0, port }) + } +} diff --git a/src/dev/isp1040.rs b/src/dev/isp1040.rs index 1dc60951f..ae655f888 100644 --- a/src/dev/isp1040.rs +++ b/src/dev/isp1040.rs @@ -50,8 +50,9 @@ //! again when the hold ends: an approximation, a chip that had fetched it //! would plausibly move on. A parked entry the guest rewrites meanwhile is //! answered incomplete. A snapshot carries each bus's remaining hold and -//! the parked positions. The ISP12160 would widen `BUSES` and decode the -//! bus from mailbox 2 and its request entries; the per-bus state is there. +//! the parked positions. Only the ISP1040's bus reset starts a hold-off: +//! the ISP12160 (two buses, per-bus state is there) keeps its own bus-reset +//! handling, and its hold-off is not modelled. //! * Only logical unit 0 of a present target exists. A command for another //! unit is passed to the target with that unit number, which answers as //! SCSI says (INQUIRY: no device at this unit; anything else: check @@ -81,7 +82,7 @@ use parking_lot::{Condvar, Mutex, MutexGuard}; use crate::devlog::LogModule; use crate::irq::IrqLine; -use crate::pci::{Bar, PciBusDevice, PciConfig, PciDma, PCI_INTA}; +use crate::pci::{Bar, PciBusDevice, PciConfig, PciConfigPolicy, PciDma, PciRegisterPolicy, PCI_INTA}; use crate::scsi::{scsi_cmd, ScsiBus, ScsiDataLength, ScsiIdentities, ScsiRequest, CDROM_IDENTITY, DISK_IDENTITY}; use crate::snapshot::{get_field, hex_u16, hex_u32, load_u16_slice, toml_u16, toml_u32, toml_u8, u16_slice_to_toml, u8_slice_to_toml}; @@ -115,7 +116,7 @@ pub mod reg { pub const WINDOW_SIZE: u32 = 0x100; } -const MAILBOX_COUNT: usize = 8; +pub(crate) const MAILBOX_COUNT: usize = 8; const MAILBOX_END: u32 = reg::MAILBOX_BASE + MAILBOX_COUNT as u32 * 2; const BANK_END: u32 = 0xc0; const BANK_WORDS: usize = ((BANK_END - reg::BANK_BASE) / 2) as usize; @@ -163,6 +164,8 @@ pub mod mbox { pub const INIT_REQ_QUEUE: u16 = 0x0010; pub const INIT_RSP_QUEUE: u16 = 0x0011; pub const ABORT_COMMAND: u16 = 0x0015; + pub const ABORT_DEVICE: u16 = 0x0016; + pub const ABORT_TARGET: u16 = 0x0017; pub const BUS_RESET: u16 = 0x0018; pub const GET_FIRMWARE_STATUS: u16 = 0x001f; pub const GET_TARGET_PARAMETERS: u16 = 0x0028; @@ -187,13 +190,14 @@ pub mod mbox { pub const INVALID: u16 = 0x4001; pub const HOST_INTERFACE_ERROR: u16 = 0x4002; pub const TEST_FAILED: u16 = 0x4003; + pub const COMMAND_ERROR: u16 = 0x4005; pub const PARAM_ERROR: u16 = 0x4006; } const RAM_WORDS: usize = 0x10000; /// The command budget the adapter announces in its firmware status. pub const MAX_REQUESTS: u16 = 512; -const FW_VERSION: [u16; 3] = [1, 0, 0]; +pub(crate) const FW_VERSION: [u16; 3] = [1, 0, 0]; pub const HOST_CMD_SHIFT: u32 = 12; pub const RISC_FLAG_HOST_INT: u16 = 0x0080; @@ -229,6 +233,7 @@ pub const SCSI_STATUS_SENSE_VALID: u16 = 0x0200; pub const IOCB_COMPLETE: u16 = 0x0000; pub const IOCB_INCOMPLETE: u16 = 0x0001; pub const IOCB_RESET: u16 = 0x0004; +pub const IOCB_ABORTED: u16 = 0x0005; pub const IOCB_DATA_UNDERRUN: u16 = 0x0015; pub const STATE_GOT_BUS: u16 = 0x0100; @@ -256,13 +261,14 @@ const DEFAULT_SELECTION_TIMEOUT: u16 = 250; /// source for a device, so it runs on the adapter's clock, host wall time /// less the time the machine was stopped, like the selection timeout. const BUS_RESET_DELAY_UNIT: Duration = Duration::from_secs(1); -/// The SCSI buses this part drives. The hold-off, the parked commands and the -/// selection are all kept per bus, so a two-bus part widens this and decodes -/// the bus from its own request entries and from mailbox 2. -const BUSES: usize = 1; +/// The most SCSI buses any part drives (the ISP12160's two; the ISP1040 has +/// one, `IspPart::channels`). The hold-off, the parked commands and the +/// selection are all kept per bus. Only the ISP1040's bus reset starts a +/// hold-off: the ISP12160 keeps its own bus-reset handling. +const BUSES: usize = 2; const CHECK_CONDITION: u8 = 0x02; -const MAX_TARGET: u8 = 15; +pub(crate) const MAX_TARGET: u8 = 15; const MAX_LUN: u8 = 7; /// A completion held for response-ring room. @@ -374,6 +380,49 @@ enum Prepared { Run(Job), } +/// Part-specific register and firmware behavior; the ring engine owns the state. +/// Defaults preserve the ISP1040 contract, including wrapping firmware RAM. +pub(crate) trait IspPart: Send + Sync { + fn name(&self) -> &'static str; + fn channels(&self) -> usize { 1 } + fn pci_config(&self, bars: [Bar; 6]) -> PciConfig { + PciConfig::new(PCI_VENDOR_QLOGIC, PCI_DEVICE_ISP1040, PCI_CLASS_SCSI, PCI_REVISION_ISP1040, bars, PCI_INTA) + } + fn firmware_version(&self) -> [u16; 3] { FW_VERSION } + fn active_bank(&self, cfg1: u16) -> Bank { + if cfg1 & CFG1_BANK_SELECT != 0 { Bank::Sxp } else { Bank::Risc } + } + fn checks_ram_bounds(&self) -> bool { false } + fn has_abort_commands(&self) -> bool { false } + fn target_on(&self, _channel: u8, target: u8) -> u8 { target } + fn decode_target(&self, target: u8) -> (usize, u8) { (0, target) } + fn id_high(&self) -> u16 { PCI_DEVICE_ISP1040 } + /// The ISP1040 register revision runs one ahead of its PCI revision. + fn bus_cfg0(&self) -> u16 { PCI_REVISION_ISP1040 as u16 + 1 } + fn cfg1_write_mask(&self) -> u16 { CFG1_WRITE_MASK } + fn unmodelled_dma_register(&self, _off: u32) -> bool { false } + fn completion_phases(&self, _moved: u64, _total: u64) -> u16 { STATE_ALL_PHASES & !STATE_GOT_SENSE } + fn mailbox_ready(&self, _semaphore: u16) -> bool { true } + /// Extra target-parameter results, beyond the common asynchronous/narrow reply. + fn clear_target_parameters(&self, _mailbox: &mut [u16; MAILBOX_COUNT]) {} + fn set_reset_delay(&self, st: &mut State) { st.reset_delay = st.mailbox[1]; } + /// Queue target notifications for the worker; the CPU never locks a bus. + fn reset_targets(&self, _st: &mut State) -> bool { false } + /// Handle extra mailbox commands, leaving result signalling to the engine. + fn execute_mailbox(&self, _core: &Core, _st: &mut MutexGuard<'_, State>) -> bool { false } + fn save_extra(&self, _st: &State, _t: &mut toml::map::Map) {} + fn load_extra(&self, _st: &mut State, _v: &toml::Value) {} +} + +struct Isp1040Part; + +impl IspPart for Isp1040Part { + fn name(&self) -> &'static str { "ISP1040" } +} + +/// The register window selected by BUS_CFG1, independent of its encoding. +pub(crate) enum Bank { Risc, Sxp, Sxp1, Dma } + /// A command waiting out its bus's post-reset hold-off. Only the ring /// position is kept: the entry, its data segments included, is read again /// from the guest's ring when the hold ends. @@ -392,7 +441,15 @@ struct Parked { started: bool, } -struct State { +pub(crate) struct State { + pub(crate) reset_targets: bool, + pub(crate) pending_target_resets: VecDeque<(u8, Option)>, + pub(crate) active: Option<(u8, u8, u32)>, + cancelled: bool, + pub(crate) outstanding: Vec<(u8, u8, u32)>, + pub(crate) sxp1_bank: [u16; BANK_WORDS], + pub(crate) dma_bank: [u16; BANK_WORDS], + pub(crate) reset_delay1: u16, ctl: Ctl, bus_cfg1: u16, bus_ctrl: u16, @@ -402,11 +459,11 @@ struct State { risc_flags: u16, dma_cmd_cfg: u16, dma_data_cfg: u16, - mailbox: [u16; MAILBOX_COUNT], + pub(crate) mailbox: [u16; MAILBOX_COUNT], risc_bank: [u16; BANK_WORDS], sxp_bank: [u16; BANK_WORDS], risc_ram: Vec, - reset_delay: u16, + pub(crate) reset_delay: u16, /// Mailbox command 0x0031's value, in [`SELECTION_TIMEOUT_UNIT`]s. selection_timeout: u16, rqst_base: u64, @@ -434,6 +491,14 @@ struct State { impl State { fn new() -> Self { let mut s = State { + reset_targets: false, + pending_target_resets: VecDeque::new(), + active: None, + cancelled: false, + outstanding: Vec::new(), + sxp1_bank: [0; BANK_WORDS], + dma_bank: [0; BANK_WORDS], + reset_delay1: 0, ctl: Ctl::default(), bus_cfg1: 0, bus_ctrl: 0, @@ -485,6 +550,12 @@ impl State { self.mailbox = [0, SIG1, SIG2, SIG3, SIG4, 0, 0, 0]; self.risc_bank = [0; BANK_WORDS]; self.sxp_bank = [0; BANK_WORDS]; + self.sxp1_bank = [0; BANK_WORDS]; + self.dma_bank = [0; BANK_WORDS]; + self.reset_delay1 = 0; + self.active = None; + self.cancelled = false; + self.outstanding.clear(); // The driver re-downloads firmware after every reset. self.risc_ram.iter_mut().for_each(|w| *w = 0); self.reset_delay = 0; @@ -499,6 +570,8 @@ impl State { self.resp_seen = 0; self.resp_cons = 0; self.pending.clear(); + self.reset_targets = false; + self.pending_target_resets.clear(); self.quiet_until = [None; BUSES]; self.parked.clear(); } @@ -524,9 +597,15 @@ impl State { self.resp_len != 0 && (u32::from(self.resp_prod) + 1) % u32::from(self.resp_len) != u32::from(self.resp_cons) } - fn active_bank(&mut self) -> &mut [u16; BANK_WORDS] { - if self.bus_cfg1 & CFG1_BANK_SELECT != 0 { &mut self.sxp_bank } else { &mut self.risc_bank } + fn active_bank(&mut self, bank: Bank) -> &mut [u16; BANK_WORDS] { + match bank { + Bank::Risc => &mut self.risc_bank, + Bank::Sxp => &mut self.sxp_bank, + Bank::Sxp1 => &mut self.sxp1_bank, + Bank::Dma => &mut self.dma_bank, + } } + } /// The adapter. Its state lives in a [`Core`] shared with the thread that @@ -543,8 +622,8 @@ impl std::ops::Deref for Isp1040 { /// Lock order: `line_order`, then `st`. Nothing else is taken with `st` /// held except the leaf locks (`notes`, `dma_misses`, `odd_accesses`): DMA, -/// the targets and `fence` are reached with it released. `fence` is held -/// across one guest-memory store at a time, and `targets` across one +/// either bus and `fence` are reached with it released. `fence` is held +/// across one guest-memory store at a time, and either bus across one /// target command; neither is held with the other. pub struct Core { st: Mutex, @@ -560,13 +639,17 @@ pub struct Core { wake: Condvar, /// Serialises driving the interrupt line; see [`Core::sync_irq`]. line_order: Mutex<()>, - /// The targets on the bus, by SCSI id. - bus: ScsiBus, - /// A bus reset the targets have not yet seen. `st` and `bus` are never - /// held together, so the reset (taken under `st`) is passed on by the - /// next command to take `bus`, before it runs: no command can see the - /// targets as they were. See [`Core::scsi`]. - targets_reset: AtomicBool, + /// Generation for commands cancelled by aborts without resetting the chip. + cancel: AtomicU64, + /// Immutable part policy, shared by register, worker and snapshot paths. + part: &'static dyn IspPart, + /// The controller's one bus, or both channels of the ISP12160. + buses: [ScsiBus; 2], + /// A bus reset the targets on that bus have not yet seen. `st` and the + /// buses are never held together, so the reset (taken under `st`) is + /// passed on by the next command to take the bus, before it runs: no + /// command can see the targets as they were. See [`Core::scsi`]. + targets_reset: [AtomicBool; 2], dma: OnceLock>, irq: OnceLock, /// Which of the Octane's two controllers this is (0 internal, 1 @@ -603,7 +686,11 @@ impl Isp1040 { /// The controller at `index` (0 internal, 1 external) on `bus`. pub fn new_at(index: u8, bus: ScsiBus) -> Self { - Self { core: Arc::new(Core::new_at(index, bus)), worker: Mutex::new(None) } + Self { core: Arc::new(Core::new_at(index, &Isp1040Part, [bus, Self::new_bus()])), worker: Mutex::new(None) } + } + + pub(crate) fn new_12160(bus0: ScsiBus, bus1: ScsiBus) -> Self { + Self { core: Arc::new(Core::new_at(0, &crate::dev::isp12160::Isp12160Part, [bus0, bus1])), worker: Mutex::new(None) } } /// Start the ring thread (idempotent) and let it take whatever was @@ -617,7 +704,7 @@ impl Isp1040 { let c = self.core.clone(); *worker = Some( std::thread::Builder::new() - .name(format!("ISP1040-{}", self.index)) + .name(format!("{}-{}", self.core.model_name(), self.index)) .spawn(move || c.run_ring()) .expect("spawn isp1040 thread"), ); @@ -641,13 +728,17 @@ impl Isp1040 { /// Name the adapter's long-lived locks in the lock registry, which /// holds only a weak reference to them. pub fn register_locks(&self) { - for (what, probe) in [ + let mut locks: Vec<(&str, fn(&Core) -> bool)> = vec![ ("st", (|c: &Core| c.st.is_locked()) as fn(&Core) -> bool), ("fence", |c| c.fence.is_locked()), - ("targets", |c| c.bus.is_locked()), - ] { + ("targets", |c| c.buses[0].is_locked()), + ]; + if self.core.part.channels() > 1 { + locks.push(("targets1", |c| c.buses[1].is_locked())); + } + for (what, probe) in locks { let core = Arc::downgrade(&self.core); - crate::locks::register_lock(format!("isp1040-{}::{what}", self.index), move || { + crate::locks::register_lock(format!("{}-{}::{what}", self.core.model_name().to_ascii_lowercase(), self.index), move || { core.upgrade().is_some_and(|c| probe(&c)) }); } @@ -662,27 +753,27 @@ impl Drop for Isp1040 { } impl Core { - fn new_at(index: u8, bus: ScsiBus) -> Self { + fn new_at(index: u8, part: &'static dyn IspPart, buses: [ScsiBus; 2]) -> Self { let st = State::new(); let gen = st.ctl.gen; + let bars = [Bar::Io(reg::WINDOW_SIZE), Bar::Mem(reg::WINDOW_SIZE), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused]; + let pci = part.pci_config(bars); Self { + part, + cancel: AtomicU64::new(0), st: Mutex::new(st), gen: AtomicU64::new(gen), fence: Mutex::new(()), wake: Condvar::new(), line_order: Mutex::new(()), - bus, - targets_reset: AtomicBool::new(false), + buses, + targets_reset: [AtomicBool::new(false), AtomicBool::new(false)], dma: OnceLock::new(), irq: OnceLock::new(), index, // BAR 0 is the I/O window and BAR 1 its memory-space alias, one // register file. - pci: PciConfig::new( - PCI_VENDOR_QLOGIC, PCI_DEVICE_ISP1040, PCI_CLASS_SCSI, PCI_REVISION_ISP1040, - [Bar::Io(reg::WINDOW_SIZE), Bar::Mem(reg::WINDOW_SIZE), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], - PCI_INTA, - ), + pci, notes: Mutex::new(Vec::new()), dma_misses: Mutex::new(std::collections::BTreeSet::new()), odd_accesses: Mutex::new(0), @@ -692,6 +783,8 @@ impl Core { } } + fn model_name(&self) -> &'static str { self.part.name() } + pub fn index(&self) -> u8 { self.index } /// A machine reset: the chip as at power-on (no firmware, no queues), @@ -702,6 +795,7 @@ impl Core { { let mut st = self.st.lock(); st.reset(); + if self.part.reset_targets(&mut st) { self.kick(&mut st); } self.publish_gen(&st); self.wake.notify_all(); } @@ -929,7 +1023,7 @@ impl Core { } /// Tell the ring thread the ring may have moved. - fn kick(&self, st: &mut State) { + pub(crate) fn kick(&self, st: &mut State) { st.ctl.work = true; self.wake.notify_all(); } @@ -957,11 +1051,16 @@ impl Core { /// The SCSI bus: attach targets to it and ask what is on it. pub fn bus(&self) -> &ScsiBus { - &self.bus + &self.buses[0] + } + + pub(crate) fn bus_for_channel(&self, channel: u8) -> Option<&ScsiBus> { + ((channel as usize) < self.part.channels()).then(|| &self.buses[channel as usize]) } - fn has_target(&self, id: u8) -> bool { - self.bus.has_target(id) + fn bus_for_target(&self, target: u8) -> (&ScsiBus, u8) { + let (channel, id) = self.part.decode_target(target); + (&self.buses[channel], id) } // ---- DMA --------------------------------------------------------- @@ -1045,12 +1144,18 @@ impl Core { /// A RISC RAM load or dump of up to 128 KiB, with the state unlocked /// for the DMA. The whole transfer is read before a word of it is kept, /// so a refused DMA leaves the array as it was. Words travel - /// little-endian. This part has no edge to its array: a transfer past - /// the end folds to the start. - fn ram_transfer(&self, st: &mut MutexGuard<'_, State>, load: bool) -> u16 { + /// little-endian. The ISP1040 wraps past the end; the ISP12160 rejects + /// transfers that would cross it. + pub(crate) fn ram_transfer(&self, st: &mut MutexGuard<'_, State>, load: bool) -> u16 { let base = st.mailbox[1]; let count = st.mailbox[4] as usize; - let host = Self::mbox_host_addr(&st.mailbox); + if self.part.checks_ram_bounds() && base as usize + count > RAM_WORDS { + return mbox::PARAM_ERROR; + } + let mut host = Self::mbox_host_addr(&st.mailbox); + if st.mailbox[0] == mbox::LOAD_RAM_A64 { + host |= ((st.mailbox[7] as u64) << 32) | ((st.mailbox[6] as u64) << 48); + } if count == 0 { return mbox::COMPLETE; } @@ -1077,6 +1182,20 @@ impl Core { /// Synchronous: the protocol has no way to leave a command pending. fn mailbox_command(&self, st: &mut MutexGuard<'_, State>) { let cmd = st.mailbox[0]; + if self.part.has_abort_commands() && matches!(cmd, mbox::ABORT_COMMAND | mbox::ABORT_DEVICE | mbox::ABORT_TARGET | mbox::BUS_RESET) { + self.with_dispatch_held(st, |st| self.execute_mailbox_command(st)); + } else { + self.execute_mailbox_command(st); + } + } + + fn execute_mailbox_command(&self, st: &mut MutexGuard<'_, State>) { + let cmd = st.mailbox[0]; + if self.part.execute_mailbox(self, st) { + st.semaphore = SEMAPHORE_LOCK; + st.bus_status |= STATUS_INT_REQ | STATUS_RISC_INT_REQ; + return; + } match cmd { mbox::NOP => st.mailbox[0] = mbox::COMPLETE, mbox::WRITE_RAM_WORD => { @@ -1098,7 +1217,19 @@ impl Core { // zero. mbox::VERIFY_CHECKSUM => { let start = st.mailbox[1]; + if self.part.checks_ram_bounds() && start as usize + 4 > RAM_WORDS { + st.mailbox[0] = mbox::PARAM_ERROR; + st.semaphore = SEMAPHORE_LOCK; + st.bus_status |= STATUS_INT_REQ | STATUS_RISC_INT_REQ; + return; + } let len = st.risc_ram[start.wrapping_add(3) as usize]; + if self.part.checks_ram_bounds() && start as usize + len as usize > RAM_WORDS { + st.mailbox[0] = mbox::PARAM_ERROR; + st.semaphore = SEMAPHORE_LOCK; + st.bus_status |= STATUS_INT_REQ | STATUS_RISC_INT_REQ; + return; + } let mut sum = 0u16; for i in 0..len { sum = sum.wrapping_add(st.risc_ram[start.wrapping_add(i) as usize]); @@ -1111,9 +1242,7 @@ impl Core { // Accepted, never executed. mbox::EXEC_FIRMWARE => st.mailbox[0] = mbox::COMPLETE, mbox::ABOUT_FIRMWARE => { - st.mailbox[1] = FW_VERSION[0]; - st.mailbox[2] = FW_VERSION[1]; - st.mailbox[3] = FW_VERSION[2]; + st.mailbox[1..4].copy_from_slice(&self.part.firmware_version()); st.mailbox[6] = 0; st.mailbox[0] = mbox::COMPLETE; } @@ -1128,6 +1257,7 @@ impl Core { mbox::GET_TARGET_PARAMETERS => { st.mailbox[2] = 0; st.mailbox[3] = 0; + self.part.clear_target_parameters(&mut st.mailbox); st.mailbox[0] = mbox::COMPLETE; } // Setters for state nothing reads back; answering invalid is a @@ -1149,7 +1279,7 @@ impl Core { st.mailbox[0] = mbox::COMPLETE; } mbox::SET_RESET_DELAY_PARAMS => { - st.reset_delay = st.mailbox[1]; + self.part.set_reset_delay(st); st.mailbox[0] = mbox::COMPLETE; } mbox::INIT_REQ_QUEUE | mbox::INIT_REQ_QUEUE_A64 | mbox::INIT_RSP_QUEUE | mbox::INIT_RSP_QUEUE_A64 => { @@ -1204,7 +1334,7 @@ impl Core { mbox::BUS_RESET => { let now = self.now_ns(st); let bus = usize::from(st.mailbox[2]); - if bus >= BUSES { + if bus >= self.part.channels() { // A bus this part does not have: nothing to quiet or // retire, and the datasheet's per-bus hold leaves the // real bus alone. Noted so a driver that leaves mailbox 2 @@ -1228,7 +1358,7 @@ impl Core { st.pending.push_back(Pending { handle: p.handle, completion: IOCB_RESET, ..Default::default() }); } } - self.targets_reset.store(true, Ordering::Release); + self.targets_reset[bus].store(true, Ordering::Release); let delay = BUS_RESET_DELAY_UNIT * u32::from(st.mailbox[1]); st.quiet_until[bus] = (!delay.is_zero()).then(|| now.saturating_add(delay.as_nanos() as i64)); self.kick(st); @@ -1245,6 +1375,67 @@ impl Core { st.bus_status |= STATUS_INT_REQ | STATUS_RISC_INT_REQ; } + /// Prevent the worker from taking another entry while a mailbox drops `st`. + fn with_dispatch_held(&self, st: &mut MutexGuard<'_, State>, f: impl FnOnce(&mut MutexGuard<'_, State>) -> R) -> R { + struct Hold<'a, 'b> { + core: &'a Core, + st: &'a mut MutexGuard<'b, State>, + } + impl Drop for Hold<'_, '_> { + fn drop(&mut self) { + self.st.ctl.hold -= 1; + self.core.kick(self.st); + } + } + st.ctl.hold += 1; + let held = Hold { core: self, st }; + f(held.st) + } + + /// A reset of `channel`'s bus: each disk on it puts its mode parameters back before the next + /// command, see [`ScsiTarget::reset_mode_parameters`]. A channel the part lacks is ignored. + pub(crate) fn note_bus_reset(&self, channel: usize) { + if let Some(flag) = self.targets_reset.get(channel) { flag.store(true, Ordering::Release); } + } + + /// Notify targets with `st` released; a blocked target must not stall register access. + fn reset_targets(&self, st: &mut MutexGuard<'_, State>, channel: u8, target: Option) { + let Some(bus) = self.bus_for_channel(channel) else { return }; + self.with_dispatch_held(st, |st| MutexGuard::unlocked(st, || bus.for_each(|id, dev| { + if target.is_none_or(|target| target as usize == id) { + dev.bus_reset(); + } + }))); + } + + pub(crate) fn retire_outstanding(&self, st: &mut State, matches: impl Fn((u8, u8, u32)) -> bool, completion: u16) -> usize { + let mut retired = 0; + let mut keep = Vec::new(); + for item in st.outstanding.drain(..) { + if matches(item) { + st.pending.push_back(Pending { handle: item.2, completion, ..Pending::default() }); + retired += 1; + } else { + keep.push(item); + } + } + st.outstanding = keep; + if retired != 0 { self.kick(st); } + retired + } + + /// Queue one cancellation reply, then fence its remaining guest-memory stores. + pub(crate) fn cancel_active(&self, st: &mut MutexGuard<'_, State>, completion: u16) { + if st.cancelled { return; } + if let Some((_, _, handle)) = st.active { + self.cancel.fetch_add(1, Ordering::AcqRel); + st.cancelled = true; + st.pending.push_back(Pending { handle, completion, ..Pending::default() }); + self.kick(st); + MutexGuard::unlocked(st, || self.retire_stores()); + } + } + fn host_command(&self, st: &mut MutexGuard<'_, State>, val: u16) { match val >> HOST_CMD_SHIFT { 0x0 => {} @@ -1255,6 +1446,10 @@ impl Core { 0x3 => st.risc_flags &= !(RISC_FLAG_PAUSED | RISC_FLAG_RESET), // "A mailbox command is ready": the trigger to process one. 0x5 => { + if !self.part.mailbox_ready(st.semaphore) { + self.note("mailbox submitted with an unacknowledged result".into()); + return; + } st.risc_flags |= RISC_FLAG_HOST_INT; self.mailbox_command(st); } @@ -1440,17 +1635,19 @@ impl Core { /// Run a CDB against the target; `data` is what a write sends it. None /// when a reset cancelled the command while it waited for the target. - fn scsi(&self, gen: u64, target: u8, cdb: &[u8], data: Option>) -> Option<(Vec, u8)> { - let mut targets = self.bus.lock(); - if self.targets_reset.swap(false, Ordering::AcqRel) { + fn scsi(&self, gen: u64, cancel: u64, target: u8, cdb: &[u8], data: Option>) -> Option<(Vec, u8)> { + let (channel, _) = self.part.decode_target(target); + let (bus, id) = self.bus_for_target(target); + let mut targets = bus.lock(); + if self.targets_reset[channel].swap(false, Ordering::AcqRel) { // The bus reset puts each target's mode parameters back; see // [`ScsiTarget::reset_mode_parameters`]. targets.iter_mut().flatten().for_each(|t| t.reset_mode_parameters()); } - if self.gen.load(Ordering::Acquire) != gen { + if self.gen.load(Ordering::Acquire) != gen || self.cancel.load(Ordering::Acquire) != cancel { return None; } - let Some(dev) = targets.get_mut(target as usize).and_then(|t| t.as_mut()) else { + let Some(dev) = targets.get_mut(id as usize).and_then(|t| t.as_mut()) else { return Some((Vec::new(), CHECK_CONDITION)); }; let req = ScsiRequest { cdb: cdb.to_vec(), data_len: ScsiDataLength::Unlimited, data_in: data }; @@ -1481,14 +1678,15 @@ impl Core { let lun = (device & 0xff) as u8; // A bus this part does not have. - if usize::from(channel) >= BUSES { + if channel as usize >= self.part.channels() { return Prepared::Answered(Self::incomplete(handle, 0, 0)); } // GOT_BUS alone is Linux's bad-target; the timeout bit beside it is // what lets the empty slot be treated as benign. - if target > MAX_TARGET || !self.has_target(target) { + if target > MAX_TARGET || !self.buses[channel as usize].has_target(target) { return Prepared::Absent(channel as u8, Self::incomplete(handle, STATE_GOT_BUS, STATUS_FLAG_TIMEOUT)); } + let id = target; if lun > MAX_LUN { return Prepared::Answered(Self::incomplete(handle, STATE_GOT_BUS, STATUS_FLAG_TIMEOUT)); } @@ -1497,9 +1695,9 @@ impl Core { cdb[1] = (cdb[1] & 0x1f) | (lun << 5); let cdb = &cdb[..(cdb_len as usize).clamp(1, CDB_FIELD_LEN)]; let block = self - .bus + .buses[channel as usize] .lock() - .get(target as usize) + .get(id as usize) .and_then(|t| t.as_ref()) .and_then(|t| t.block_size()) .map(|b| b as usize) @@ -1507,6 +1705,7 @@ impl Core { .unwrap_or(BLOCK_SIZE); let (xfer, to_dev) = Self::cdb_transfer(cdb, block); let (segs, described) = self.read_segments(ring, addr, entry_count, seg_count, xfer as u64, a64); + let target = self.part.target_on(channel as u8, target); Prepared::Run(Job { handle, target, cdb: cdb.to_vec(), xfer, to_dev, segs, described }) } @@ -1522,7 +1721,7 @@ impl Core { /// None too when a reset cancelled the command (`gen` is the generation /// it was read under): what it had not yet stored in host memory is not /// stored, and no completion is built. - fn run_job(&self, job: Job, gen: u64) -> Option { + fn run_job(&self, job: Job, gen: u64, cancel: u64) -> Option { let Job { handle, target, cdb, xfer, to_dev, segs, described } = job; // The bar has lights for SCSI ids 0-5. if let (Some(hb), 0..=5) = (self.heartbeat.get(), target) { @@ -1540,9 +1739,10 @@ impl Core { buf.extend_from_slice(&chunk); } moved = buf.len() as u64; - status = self.scsi(gen, target, &cdb, Some(buf))?.1; + if self.cancel.load(Ordering::Acquire) != cancel { return None; } + status = self.scsi(gen, cancel, target, &cdb, Some(buf))?.1; } else { - let (data, s) = self.scsi(gen, target, &cdb, None)?; + let (data, s) = self.scsi(gen, cancel, target, &cdb, None)?; status = s; // Offer the target's bytes to the descriptors in order. let take = (data.len() as u64).min(described).min(xfer as u64) as usize; @@ -1550,7 +1750,7 @@ impl Core { for &(a, l) in &segs { if off >= take { break; } let n = (l as usize).min(take - off); - if !self.fenced(gen, || self.dma_write(a, &data[off..off + n]))? { + if !self.fenced(gen, || self.cancel.load(Ordering::Acquire) == cancel && self.dma_write(a, &data[off..off + n]))? { self.note(format!("payload DMA for handle {handle:#x} refused by the fabric; command stopped")); return None; } @@ -1561,12 +1761,13 @@ impl Core { // Auto-sense runs only after a CHECK CONDITION. let mut sense = Vec::new(); + if self.cancel.load(Ordering::Acquire) != cancel { return None; } if status == CHECK_CONDITION { - sense = self.scsi(gen, target, &[scsi_cmd::REQUEST_SENSE, 0, 0, 0, SENSE_MAX as u8, 0], None)?.0; + sense = self.scsi(gen, cancel, target, &[scsi_cmd::REQUEST_SENSE, 0, 0, 0, SENSE_MAX as u8, 0], None)?.0; sense.truncate(SENSE_MAX); } let mut scsi_status = status as u16; - let mut state_flags = STATE_ALL_PHASES & !STATE_GOT_SENSE; + let mut state_flags = self.part.completion_phases(moved, described.max(xfer as u64)); if !sense.is_empty() { scsi_status |= SCSI_STATUS_SENSE_VALID; state_flags |= STATE_GOT_SENSE; @@ -1663,6 +1864,17 @@ impl Core { st.ctl.work = true; return st; } + if std::mem::take(&mut st.reset_targets) { + for channel in 0..self.part.channels() { + self.note_bus_reset(channel); + self.reset_targets(&mut st, channel as u8, None); + } + continue; + } + if let Some((channel, target)) = st.pending_target_resets.pop_front() { + self.reset_targets(&mut st, channel, target); + continue; + } // Completions held for room go first, in order, as the driver // frees it; while any is held the request ring waits. if !st.pending.is_empty() { @@ -1804,8 +2016,16 @@ impl Core { } Prepared::Run(job) => { st.total_cmds += 1; + st.active = Some((job.target, job.cdb.get(1).copied().unwrap_or(0) >> 5, job.handle)); + st.cancelled = false; st.ctl.busy = true; - let done = MutexGuard::unlocked(&mut st, || self.run_job(job, ring.gen)); + let cancel = self.cancel.load(Ordering::Acquire); + let mut done = MutexGuard::unlocked(&mut st, || self.run_job(job, ring.gen, cancel)); + if st.cancelled { done = None; } + else if self.part.has_abort_commands() && done.is_none() && st.ctl.gen == ring.gen { + if let Some(active) = st.active { st.outstanding.push(active); } + } + st.active = None; st.ctl.busy = false; self.wake.notify_all(); // A reset cancelled the command. @@ -1838,13 +2058,16 @@ impl Core { return st.mailbox[((off - reg::MAILBOX_BASE) / 2) as usize]; } if (reg::BANK_BASE..BANK_END).contains(&off) { - return st.active_bank()[((off - reg::BANK_BASE) / 2) as usize]; + let bank = self.part.active_bank(st.bus_cfg1); + return st.active_bank(bank)[((off - reg::BANK_BASE) / 2) as usize]; + } + if self.part.unmodelled_dma_register(off) { + return 0; } match off { reg::ID_LOW => PCI_VENDOR_QLOGIC, - reg::ID_HIGH => PCI_DEVICE_ISP1040, - // The revision field runs one ahead of the PCI byte. - reg::BUS_CFG0 => PCI_REVISION_ISP1040 as u16 + 1, + reg::ID_HIGH => self.part.id_high(), + reg::BUS_CFG0 => self.part.bus_cfg0(), reg::BUS_CFG1 => st.bus_cfg1, reg::BUS_CTRL => st.bus_ctrl, reg::BUS_STATUS => st.bus_status, @@ -1862,6 +2085,10 @@ impl Core { } fn reg_write(&self, off: u32, val: u16) { + if self.part.unmodelled_dma_register(off) { + self.note(format!("write to ISP12160 DMA register {off:#04x} is not modelled")); + return; + } let mut st = self.st.lock(); if off == reg::RESPONSE_QUEUE { // The consumer acknowledgement: recorded, never over the @@ -1884,15 +2111,17 @@ impl Core { return; } if (reg::BANK_BASE..BANK_END).contains(&off) { - st.active_bank()[((off - reg::BANK_BASE) / 2) as usize] = val; + let bank = self.part.active_bank(st.bus_cfg1); + st.active_bank(bank)[((off - reg::BANK_BASE) / 2) as usize] = val; return; } match off { reg::BUS_CFG1 => { - if val & !CFG1_WRITE_MASK != 0 { - self.note(format!("bus configuration 1 bits {:#06x} are not modelled", val & !CFG1_WRITE_MASK)); + let mask = self.part.cfg1_write_mask(); + if val & !mask != 0 { + self.note(format!("bus configuration 1 bits {:#06x} are not modelled", val & !mask)); } - st.bus_cfg1 = val & CFG1_WRITE_MASK; + st.bus_cfg1 = val & mask; } // Applied after the enables are stored, so the reset wins. reg::BUS_CTRL => { @@ -1900,6 +2129,7 @@ impl Core { // The command in flight is cancelled, not waited for. if val & CTRL_SOFT_RESET != 0 { st.reset(); + if self.part.reset_targets(&mut st) { self.kick(&mut st); } self.publish_gen(&st); self.wake.notify_all(); drop(st); @@ -2005,7 +2235,7 @@ impl crate::traits::Device for Isp1040 { impl crate::scsi_monitor::ScsiTargets for Isp1040 { fn scsi_bus(&self) -> &ScsiBus { - &self.bus + &self.buses[0] } } @@ -2041,14 +2271,15 @@ impl crate::traits::Saveable for Isp1040 { // A hold-off already over but not acted on still crosses: the // commands parked behind it start when the load re-arms it at 0. let quiet_left = st.quiet_until.map(|until| until.map(|u| u - now)); - Self::save_locked(st, left, quiet_left) + self.save_locked(st, left, quiet_left) }) } fn load_state(&self, v: &toml::Value) -> Result<(), String> { self.quiesced(|st| { let now = self.now_ns(st); - Self::load_locked(st, v, now); + self.load_locked(st, v, now); + self.cancel.store(0, Ordering::Release); self.publish_gen(st); }); self.sync_irq(); @@ -2057,7 +2288,7 @@ impl crate::traits::Saveable for Isp1040 { } impl Isp1040 { - fn save_locked(st: &State, selecting_left: Option, quiet_left: [Option; BUSES]) -> toml::Value { + fn save_locked(&self, st: &State, selecting_left: Option, quiet_left: [Option; BUSES]) -> toml::Value { let mut t = toml::map::Map::new(); for (k, v) in [ ("bus_cfg1", st.bus_cfg1), ("bus_ctrl", st.bus_ctrl), ("bus_status", st.bus_status), @@ -2130,16 +2361,18 @@ impl Isp1040 { m.insert("sense".into(), u8_slice_to_toml(&sel.done.sense)); t.insert("selecting".into(), toml::Value::Table(m)); } + self.part.save_extra(st, &mut t); toml::Value::Table(t) } /// Unprocessed request entries survive: the ring thread is kicked and /// runs them after the load. - fn load_locked(st: &mut State, v: &toml::Value, now: i64) { + fn load_locked(&self, st: &mut State, v: &toml::Value, now: i64) { { let ctl = std::mem::take(&mut st.ctl); *st = State::new(); st.ctl = ctl; + self.part.load_extra(st, v); st.ctl.gen += 1; st.ctl.rqst_epoch += 1; st.ctl.resp_epoch += 1; diff --git a/src/dev/isp1040/tests.rs b/src/dev/isp1040/tests.rs index 8a96b9d66..0bb7e084c 100644 --- a/src/dev/isp1040/tests.rs +++ b/src/dev/isp1040/tests.rs @@ -2,7 +2,8 @@ //! and ring contract. Host memory is a flat byte array behind a fake `PciDma`; the disk //! is a small raw image. //! -//! Ported from tests/qtest/qlogic-isp-test.c in jamesbraid/qemu-sgi-origami. +//! Ported from tests/qtest/qlogic-isp-test.c and tests/qtest/sn0-isp12160-test.c +//! in jamesbraid/qemu-sgi-origami. use super::*; use crate::irq::LevelProbe; @@ -59,15 +60,16 @@ fn image(name: &str, blocks: usize) -> std::path::PathBuf { } impl Rig { - /// A chip on fake memory, no disks. - fn bare() -> Rig { + fn on(isp: Isp1040) -> Rig { let mem = FakeMem::new(); - let isp = Isp1040::new(); Device::start(&isp); isp.connect_dma(mem.clone()); Rig { isp, mem } } + /// A chip on fake memory, no disks. + fn bare() -> Rig { Rig::on(Isp1040::new()) } + /// A chip with a disk of `blocks` blocks at target `id`. fn with_disk(name: &str, id: u8, blocks: usize) -> Rig { let r = Rig::bare(); @@ -1616,6 +1618,403 @@ fn a_controller_built_on_a_filled_bus_answers_for_its_target() { let _ = std::fs::remove_file(img); } + +struct MarkerTarget(u8); + +impl crate::scsi::ScsiTarget for MarkerTarget { + fn request(&mut self, _req: &crate::scsi::ScsiRequest) -> std::io::Result { + Ok(crate::scsi::ScsiResponse { status: 0, data: vec![self.0] }) + } +} + +/// Target 0 on channel 1 must stay distinct from target 0 on channel 0. +#[test] +fn the_isp12160_routes_ring_requests_to_separate_buses() { + let bus0 = Isp1040::new_bus(); + let bus1 = Isp1040::new_bus(); + bus0.attach(0, Box::new(MarkerTarget(0x10))).unwrap(); + bus1.attach(0, Box::new(MarkerTarget(0x20))).unwrap(); + let r = Rig::on(Isp1040::new_12160(bus0, bus1)); + r.init_rings(4, 4); + + r.put_entry(0, &request(0x1234, 0x80, &[0x12, 0, 0, 0, 36, 0], &[(BUF, 36)])); + r.doorbell(1); + assert_eq!(r.mem.get(BUF, 1), vec![0x20], "channel 1 reaches its target 0"); + + r.put_entry(1, &request(0x1235, 0, &[0x12, 0, 0, 0, 36, 0], &[(BUF + 64, 36)])); + r.doorbell(2); + assert_eq!(r.mem.get(BUF + 64, 1), vec![0x10], "channel 0 keeps its own target 0"); +} + +struct ResetProbe(Arc); + +impl crate::scsi::ScsiTarget for ResetProbe { + fn request(&mut self, _req: &crate::scsi::ScsiRequest) -> std::io::Result { + Ok(crate::scsi::ScsiResponse { status: 0, data: Vec::new() }) + } + + fn bus_reset(&mut self) { + self.0.fetch_add(1, Ordering::Relaxed); + } +} + +/// A snapshot retains both the chip-wide reset and the selected target reset, +/// and the resumed worker delivers both through the original channel buses. +#[test] +fn the_isp12160_preserves_queued_bus_and_target_resets_across_save_load() { + let src = Isp1040::new_12160(Isp1040::new_bus(), Isp1040::new_bus()); + { + let mut state = src.st.lock(); + state.reset_targets = true; + state.pending_target_resets.push_back((1, Some(2))); + } + let saved = src.save_state(); + + let resets0 = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let resets1 = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let selected_reset = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let bus0 = Isp1040::new_bus(); + let bus1 = Isp1040::new_bus(); + bus0.attach(1, Box::new(ResetProbe(resets0.clone()))).unwrap(); + bus1.attach(1, Box::new(ResetProbe(resets1.clone()))).unwrap(); + bus1.attach(2, Box::new(ResetProbe(selected_reset.clone()))).unwrap(); + let dst = Isp1040::new_12160(bus0, bus1); + dst.load_state(&saved).unwrap(); + Device::start(&dst); + dst.wait_idle(); + Device::stop(&dst); + + assert_eq!(resets0.load(Ordering::Relaxed), 1); + assert_eq!(resets1.load(Ordering::Relaxed), 1); + assert_eq!(selected_reset.load(Ordering::Relaxed), 2, "the full reset and queued target reset both run"); + assert_eq!(saved.get("reset_targets").and_then(toml::Value::as_bool), Some(true)); + assert_eq!(saved.get("target_resets").and_then(toml::Value::as_array).map(Vec::len), Some(1)); + let after = dst.save_state(); + assert_eq!(after.get("reset_targets").and_then(toml::Value::as_bool), Some(false)); + assert_eq!(after.get("target_resets").and_then(toml::Value::as_array).map(Vec::len), Some(0)); +} + +// ---- ISP12160 second channel and recovery commands ----------------------- + +fn rig12160() -> Rig { Rig::on(Isp1040::new_12160(Isp1040::new_bus(), Isp1040::new_bus())) } + +/// A 64-block image whose first block is `fill`, so a read says which disk answered. +fn filled_image(name: &str, fill: u8) -> std::path::PathBuf { + let p = image(name, 64); + let mut data = vec![0u8; 64 * BLOCK_SIZE]; + data[..BLOCK_SIZE].fill(fill); + std::fs::write(&p, data).expect("test image"); + p +} + +/// A 12160 with a different disk at target 1 on each channel. +fn disks_on_both_channels(name: &str) -> Rig { + let r = rig12160(); + for channel in 0..2u8 { + let img = filled_image(&format!("{name}{channel}"), 0xa0 + channel); + r.isp.bus_for_channel(channel).unwrap().attach_image(1, &img, false).expect("attach"); + } + r +} + +/// Issues entries one at a time on a 16-entry ring and returns each completion. +struct Seq<'a> { + r: &'a Rig, + n: u16, +} + +impl<'a> Seq<'a> { + fn new(r: &'a Rig) -> Self { + r.init_rings(16, 16); + Seq { r, n: 0 } + } + fn run(&mut self, handle: u32, target: u8, cdb: &[u8], segs: &[(u32, u32)]) -> [u8; ENTRY_SIZE] { + let slot = self.n; + self.r.put_entry(slot as u32, &request(handle, target, cdb, segs)); + self.n += 1; + self.r.doorbell(self.n); + self.r.status(slot as u32) + } +} + +const TUR: [u8; 6] = [0; 6]; + +/// Counts bus resets and answers any command with a full block pair, so a read +/// whose second buffer segment is unreachable stays outstanding. +struct CountingDisk(Arc); + +impl crate::scsi::ScsiTarget for CountingDisk { + fn request(&mut self, _req: &crate::scsi::ScsiRequest) -> std::io::Result { + Ok(crate::scsi::ScsiResponse { status: 0, data: vec![0; 2 * BLOCK_SIZE] }) + } + fn bus_reset(&mut self) { self.0.fetch_add(1, Ordering::Relaxed); } +} + +struct Held { + r: Rig, + /// Bus resets seen by the counting disks at target 1 on channel 0, target 1 on channel 1, + /// and target 2 on channel 1. + ch0: Arc, + ch1: Arc, + ch1_other: Arc, +} + +impl Held { + fn new() -> Held { + let counters: Vec<_> = (0..3).map(|_| Arc::new(std::sync::atomic::AtomicUsize::new(0))).collect(); + let (bus0, bus1) = (Isp1040::new_bus(), Isp1040::new_bus()); + bus0.attach(1, Box::new(CountingDisk(counters[0].clone()))).unwrap(); + bus1.attach(1, Box::new(CountingDisk(counters[1].clone()))).unwrap(); + bus1.attach(2, Box::new(CountingDisk(counters[2].clone()))).unwrap(); + let r = Rig::on(Isp1040::new_12160(bus0, bus1)); + r.init_rings(16, 16); + Held { r, ch0: counters[0].clone(), ch1: counters[1].clone(), ch1_other: counters[2].clone() } + } + /// A read whose payload store the fabric refuses: the adapter keeps it outstanding, which + /// is what the abort and reset commands act on. + fn hold(&self, slot: u32, handle: u32, target: u8) { + self.r.put_entry(slot, &request(handle, target, &cdb10(scsi_cmd::READ_10, 0, 2), + &[(BUF, BLOCK_SIZE as u32), (MEM as u32, BLOCK_SIZE as u32)])); + self.r.doorbell(slot as u16 + 1); + assert_eq!(g32(&self.r.status(slot), 4), 0, "handle {handle:#x} is held, not completed"); + } + fn command(&self, regs: &[u16]) -> u16 { + let status = self.r.cmd(regs); + self.r.isp.wait_idle(); + status + } + /// The completion posted in response slot `slot`: (handle, completion code). + fn answer(&self, slot: u32) -> (u32, u16) { + let s = self.r.status(slot); + (g32(&s, 4), g16(&s, 10)) + } +} + +const CH1: u16 = 0x8000; + +/// A driver sizing its queues and recognising the part reads these before any ring exists. +#[test] +fn the_isp12160_reports_its_own_identity_firmware_and_unmapped_registers() { + let r = rig12160(); + assert_eq!(r.r(reg::ID_HIGH), 0x1216); + assert_eq!(r.r(reg::BUS_CFG0), 0, "no revision field on this part"); + r.w(reg::DMA_CMD_CFG, 0xffff); + r.w(reg::DMA_DATA_CFG, 0xffff); + assert_eq!((r.r(reg::DMA_CMD_CFG), r.r(reg::DMA_DATA_CFG)), (0, 0), "the flat ISP1040 DMA blocks are not here"); + + // Mailbox 6 is seeded because an untouched output would also read zero. + r.w(r.mb(6), 0x1234); + r.w(r.mb(0), mbox::ABOUT_FIRMWARE); + r.w(reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT); + assert_eq!(r.r(r.mb(0)), mbox::COMPLETE); + assert_eq!([r.r(r.mb(1)), r.r(r.mb(2))], [10, 4]); + assert_eq!(r.r(r.mb(6)), 0, "the attribute word is cleared, not echoed"); + r.ack(); + r.w(r.mb(0), mbox::GET_FIRMWARE_STATUS); + r.w(reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT); + assert_eq!(r.r(r.mb(0)), mbox::COMPLETE); + assert_eq!([r.r(r.mb(1)), r.r(r.mb(2))], [0, MAX_REQUESTS]); + r.ack(); +} + +/// The driver switches register banks through bus configuration 1 bits 9:8; each of the +/// four selections must show its own words, and no other bit may stick. +#[test] +fn the_isp12160_selects_four_distinct_banks_and_keeps_only_the_bank_field() { + let r = rig12160(); + let selections = [0x0000, 0x0100, 0x0200, 0x0300]; + for sel in selections { + r.w(reg::BUS_CFG1, sel); + r.w(reg::BANK_BASE, 0x1000 + sel); + r.w(reg::BANK_BASE + 0x20, 0x2000 + sel); + } + for sel in selections { + r.w(reg::BUS_CFG1, sel); + assert_eq!(r.r(reg::BANK_BASE), 0x1000 + sel, "bank {sel:#x}"); + assert_eq!(r.r(reg::BANK_BASE + 0x20), 0x2000 + sel, "bank {sel:#x}"); + } + r.w(reg::BUS_CFG1, 0xb2c5); + assert_eq!(r.r(reg::BUS_CFG1), 0x0200); + + // A soft reset hands a re-initialising driver empty banks, not the previous run's words. + r.w(reg::BUS_CTRL, CTRL_SOFT_RESET); + assert_eq!(r.r(reg::BUS_CFG1), 0); + for sel in selections { + r.w(reg::BUS_CFG1, sel); + assert_eq!(r.r(reg::BANK_BASE), 0, "bank {sel:#x} after reset"); + } +} + +/// A second command submitted before the driver acknowledged the first result must not +/// run, or it would overwrite the answer the driver has not read. +#[test] +fn the_isp12160_ignores_a_mailbox_submission_while_the_result_is_unacknowledged() { + let r = rig12160(); + r.w(r.mb(0), mbox::NOP); + r.w(reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT); + assert_eq!(r.r(r.mb(0)), mbox::COMPLETE); + assert_eq!(r.r(reg::SEMAPHORE) & SEMAPHORE_LOCK, SEMAPHORE_LOCK); + + r.w(r.mb(1), 0x9999); + r.w(r.mb(0), mbox::ABOUT_FIRMWARE); + r.w(reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT); + assert_eq!(r.r(r.mb(0)), mbox::ABOUT_FIRMWARE, "the second submission did not run"); + assert_eq!(r.r(r.mb(1)), 0x9999); + + r.ack(); + r.w(reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT); + assert_eq!(r.r(r.mb(0)), mbox::COMPLETE); + assert_eq!(r.r(r.mb(1)), 10, "acknowledged, the same submission runs"); + r.ack(); +} + +/// An empty slot on either channel times out on its own bus; a driver probing channel 1 +/// must see an absent target, not a hang or the channel 0 answer. +#[test] +fn a_missing_target_times_out_on_each_channel() { + let r = rig12160(); + r.init_rings(16, 16); + r.clock(0); + // An absent target answers once its selection timeout has run out on + // the adapter's clock. + let mut now = 0; + for (slot, (handle, target)) in [(0x31, 5), (0x32, 0x80 | 5)].into_iter().enumerate() { + r.put_entry(slot as u32, &request(handle, target, &TUR, &[])); + r.doorbell(slot as u16 + 1); + assert_ne!(g32(&r.status(slot as u32), 4), handle, "nothing before the selection timeout"); + now += DEFAULT_SELECTION_TIMEOUT as i64 * MS; + r.clock(now); + let s = r.status(slot as u32); + assert_eq!(g32(&s, 4), handle); + assert_eq!(g16(&s, 10), IOCB_INCOMPLETE); + assert_eq!(g16(&s, 12), STATE_GOT_BUS); + assert_eq!(g16(&s, 14), STATUS_FLAG_TIMEOUT); + } +} + +/// A command refused on channel 1 must not disturb its neighbour on channel 0, and +/// a write must land on the disk of the channel it named. +#[test] +fn a_refused_command_on_channel_1_leaves_channel_0_and_the_media_alone() { + let r = disks_on_both_channels("refuse"); + let mut seq = Seq::new(&r); + // Collect each disk's power-on unit attention so the reads below see real answers. + seq.run(0xa0, 1, &TUR, &[]); + seq.run(0xa1, 0x81, &TUR, &[]); + + r.mem.put(BUF, &[0x5c; BLOCK_SIZE]); + let beyond_disk = seq.run(0xb1, 0x81, &cdb10(scsi_cmd::READ_10, 1000, 1), &[(BUF, BLOCK_SIZE as u32)]); + assert_eq!(g16(&beyond_disk, 8) & 0xff, CHECK_CONDITION as u16); + assert_eq!(r.mem.get(BUF, BLOCK_SIZE), vec![0x5c; BLOCK_SIZE], "the refused read stored nothing"); + + let good = seq.run(0xb0, 1, &cdb10(scsi_cmd::READ_10, 0, 1), &[(BUF + 0x1000, BLOCK_SIZE as u32)]); + assert_eq!(g16(&good, 10), IOCB_COMPLETE); + assert_eq!(r.mem.get(BUF + 0x1000, BLOCK_SIZE), vec![0xa0; BLOCK_SIZE], "channel 0 still reads its own disk"); + + r.mem.put(BUF + 0x2000, &[0x77; BLOCK_SIZE]); + seq.run(0xc0, 0x81, &cdb10(scsi_cmd::WRITE_10, 3, 1), &[(BUF + 0x2000, BLOCK_SIZE as u32)]); + seq.run(0xc1, 0x81, &cdb10(scsi_cmd::READ_10, 3, 1), &[(BUF + 0x3000, BLOCK_SIZE as u32)]); + seq.run(0xc2, 1, &cdb10(scsi_cmd::READ_10, 3, 1), &[(BUF + 0x4000, BLOCK_SIZE as u32)]); + assert_eq!(r.mem.get(BUF + 0x3000, BLOCK_SIZE), vec![0x77; BLOCK_SIZE], "channel 1 holds what was written"); + assert_eq!(r.mem.get(BUF + 0x4000, BLOCK_SIZE), vec![0; BLOCK_SIZE], "channel 0's block 3 is untouched"); +} + +/// Abort device names a channel in the top bit of its selector; the same target number +/// held on the other bus must stay outstanding. +#[test] +fn abort_device_retires_only_the_named_channels_commands() { + let h = Held::new(); + h.hold(0, 0x41, 1); + h.hold(1, 0x42, 0x81); + assert_eq!(h.command(&[mbox::ABORT_DEVICE, CH1 | 0x0100, 0, 0]), mbox::COMPLETE); + assert_eq!(h.answer(0), (0x42, IOCB_ABORTED)); + assert_eq!(h.answer(1), (0, 0), "channel 0's command is still held"); + // The survivor is still the adapter's to abort. + assert_eq!(h.command(&[mbox::ABORT_DEVICE, 0x0100, 0, 0]), mbox::COMPLETE); + assert_eq!(h.answer(1), (0x41, IOCB_ABORTED)); +} + +/// Abort command names one handle: two commands to the same channel, target and unit differ +/// only by it, and a selector that matches a held handle on the wrong channel is an error. +#[test] +fn abort_command_matches_the_handle_and_the_whole_selector() { + let h = Held::new(); + h.hold(0, 0x51, 0x81); + h.hold(1, 0x52, 0x81); + assert_eq!(h.command(&[mbox::ABORT_COMMAND, CH1 | 0x0100, 0, 0x52]), mbox::COMPLETE); + assert_eq!(h.answer(0), (0x52, IOCB_ABORTED)); + assert_eq!(h.answer(1), (0, 0), "its neighbour on the same target is untouched"); + + assert_eq!(h.command(&[mbox::ABORT_COMMAND, 0x0100, 0, 0x51]), mbox::COMMAND_ERROR, + "a held handle named on the wrong channel is not a match"); + assert_eq!(h.answer(1), (0, 0)); + assert_eq!(h.command(&[mbox::ABORT_COMMAND, CH1 | 0x0100, 0, 0x51]), mbox::COMPLETE); + assert_eq!(h.answer(1), (0x51, IOCB_ABORTED)); +} + +/// A driver clearing a device it already drained must read success, while naming a handle +/// nothing is using must read as an error rather than claim a retirement. +#[test] +fn aborting_nothing_is_complete_but_aborting_an_unknown_handle_is_refused() { + let h = Held::new(); + assert_eq!(h.command(&[mbox::ABORT_DEVICE, CH1 | 0x0100, 0, 0]), mbox::COMPLETE); + assert_eq!(h.command(&[mbox::ABORT_COMMAND, CH1 | 0x0100, 0, 0x77]), mbox::COMMAND_ERROR); + assert_eq!(h.answer(0), (0, 0), "neither posted a completion"); +} + +/// Target reset reaches one target on one channel; the same number on the other channel and +/// the neighbour on the same channel must not see it, and the reset target keeps serving. +#[test] +fn target_reset_is_scoped_to_one_target_on_one_channel() { + let h = Held::new(); + h.hold(0, 0x41, 1); + h.hold(1, 0x42, 0x81); + assert_eq!(h.command(&[mbox::ABORT_TARGET, CH1 | 0x0100, 0, 0]), mbox::COMPLETE); + assert_eq!(h.answer(0), (0x42, IOCB_RESET)); + assert_eq!(h.answer(1), (0, 0)); + assert_eq!([h.ch0.load(Ordering::Relaxed), h.ch1.load(Ordering::Relaxed), h.ch1_other.load(Ordering::Relaxed)], + [0, 1, 0]); + + h.r.put_entry(2, &request(0x43, 0x81, &TUR, &[])); + h.r.doorbell(3); + assert_eq!(h.answer(1), (0x43, IOCB_COMPLETE)); +} + +/// Bus reset takes its channel from mailbox 2; mailbox 1 is the reset delay and here holds a value +/// that is not even a channel. The whole named bus resets, the other one does not. +#[test] +fn bus_reset_takes_its_channel_from_mailbox_two_and_resets_every_target_on_it() { + let h = Held::new(); + h.hold(0, 0x41, 1); + h.hold(1, 0x42, 0x81); + assert_eq!(h.command(&[mbox::BUS_RESET, 3, 1]), mbox::COMPLETE); + assert_eq!(h.answer(0), (0x42, IOCB_RESET)); + assert_eq!(h.answer(1), (0, 0)); + assert_eq!([h.ch0.load(Ordering::Relaxed), h.ch1.load(Ordering::Relaxed), h.ch1_other.load(Ordering::Relaxed)], + [0, 1, 1]); + assert_eq!(h.command(&[mbox::BUS_RESET, 0, 2]), mbox::PARAM_ERROR, "there is no third bus"); +} + +/// A bus reset leaves a unit attention on the disks of that channel only; the driver's +/// first command afterwards is expected to collect it. +#[test] +fn a_bus_reset_leaves_a_unit_attention_on_the_reset_channel_only() { + let r = disks_on_both_channels("uareset"); + let mut seq = Seq::new(&r); + seq.run(0x10, 1, &TUR, &[]); + seq.run(0x11, 0x81, &TUR, &[]); + assert_eq!(g16(&seq.run(0x12, 0x81, &TUR, &[]), 8) & 0xff, 0, "attention was collected"); + + assert_eq!(r.cmd(&[mbox::BUS_RESET, 0, 1]), mbox::COMPLETE); + r.isp.wait_idle(); + let after = seq.run(0x13, 0x81, &TUR, &[]); + assert_eq!(g16(&after, 8) & 0xff, CHECK_CONDITION as u16); + assert_eq!(after[34] & 0x0f, 0x06, "unit attention"); + let other = seq.run(0x14, 1, &TUR, &[]); + assert_eq!(g16(&other, 8) & 0xff, 0, "channel 0 was not reset"); + assert_eq!(g16(&seq.run(0x15, 0x81, &TUR, &[]), 8) & 0xff, 0, "reported once"); +} + // ---- the hold-off after a bus reset --------------------------------------- // // Datasheet, "Bus-reset hold-off (0x0018)": after the reset the chip starts no @@ -1651,6 +2050,87 @@ fn a_bus_reset_puts_a_disks_write_cache_back() { assert_eq!(r.mem.get(BUF + 6, 1)[0] & 0x04, 0x04, "WCE is back"); } +/// A 12160 with a cached disk at target 1 on each channel whose guests cleared WCE on both. +fn cleared_write_cache_on_both_channels(name: &str) -> Rig { + let r = rig12160(); + for channel in 0..2u8 { + let bus = r.isp.bus_for_channel(channel).unwrap(); + bus.attach_cached_image(1, &image(&format!("{name}{channel}"), 64)).expect("attach"); + bus.select_write_cache(1, false); + assert!(!bus.write_cache_reported(1), "channel {channel}: the guest cleared it"); + } + r +} + +/// The write cache bit MODE SENSE(6) of the caching page reports for target 1 on `channel`. A TUR +/// goes first: it collects the reset's unit attention, and consumes the reset flag as a guest's +/// first command does, so the mode sense that follows returns its page. +fn wce_after_next_command(seq: &mut Seq, channel: u8, handle: u32) -> bool { + let target = channel << 7 | 1; + seq.run(handle, target, &TUR, &[]); + let done = seq.run(handle + 0x100, target, &[0x1a, 0x08, 0x08, 0, 24, 0], &[(BUF, 24)]); + assert_eq!(g16(&done, 10), IOCB_COMPLETE); + assert_eq!(seq.r.mem.get(BUF + 4, 1)[0], 0x08, "caching page"); + seq.r.mem.get(BUF + 6, 1)[0] & 0x04 != 0 +} + +/// Each channel's disks put their write cache back on that channel's reset and not on the +/// other's, so the flag has to be set for the channel the mailbox names. +#[test] +fn a_12160_bus_reset_of_channel_1_restores_write_cache_on_channel_1_only() { + let r = cleared_write_cache_on_both_channels("wce12160-ch1"); + let mut seq = Seq::new(&r); + assert_eq!(r.cmd(&[mbox::BUS_RESET, 0, 1]), mbox::COMPLETE); + r.isp.wait_idle(); + assert!(!wce_after_next_command(&mut seq, 0, 0x31), "channel 0 was not reset"); + assert!(wce_after_next_command(&mut seq, 1, 0x32), "channel 1 was reset"); +} + +#[test] +fn a_12160_bus_reset_of_channel_0_restores_write_cache_on_channel_0_only() { + let r = cleared_write_cache_on_both_channels("wce12160-ch0"); + let mut seq = Seq::new(&r); + assert_eq!(r.cmd(&[mbox::BUS_RESET, 0, 0]), mbox::COMPLETE); + r.isp.wait_idle(); + assert!(!wce_after_next_command(&mut seq, 1, 0x33), "channel 1 was not reset"); + assert!(wce_after_next_command(&mut seq, 0, 0x34), "channel 0 was reset"); +} + +/// Both reset the chip's SCSI buses, which the disks see as a bus reset on each channel. +#[test] +fn a_12160_chip_reset_restores_write_cache_on_both_channels() { + let r = cleared_write_cache_on_both_channels("wce12160-chip"); + r.isp.reset(); + r.isp.wait_idle(); + let mut seq = Seq::new(&r); + assert!(wce_after_next_command(&mut seq, 0, 0x35), "channel 0"); + assert!(wce_after_next_command(&mut seq, 1, 0x36), "channel 1"); +} + +#[test] +fn a_12160_soft_reset_restores_write_cache_on_both_channels() { + let r = cleared_write_cache_on_both_channels("wce12160-soft"); + r.w(reg::BUS_CTRL, CTRL_SOFT_RESET); + r.isp.wait_idle(); + let mut seq = Seq::new(&r); + assert!(wce_after_next_command(&mut seq, 0, 0x37), "channel 0"); + assert!(wce_after_next_command(&mut seq, 1, 0x38), "channel 1"); +} + +/// The single-bus part has no bus 1: a mailbox that names it is ignored, and the one bus it has +/// keeps its disks' write cache as the guest set it. +#[test] +fn an_isp1040_bus_reset_naming_bus_1_leaves_the_reset_flag_unset() { + let r = Rig::bare(); + r.isp.bus().attach_cached_image(1, &image("wce-bus1", 64)).expect("attach"); + r.init_rings(4, 4); + r.isp.bus().select_write_cache(1, false); + bus_reset(&r, 0, 1); + r.isp.wait_idle(); + assert!(!r.isp.targets_reset[0].load(Ordering::Acquire)); + assert!(!r.isp.targets_reset[1].load(Ordering::Acquire)); +} + /// A TUR for the disk at target 1. fn tur1(handle: u32) -> [u8; ENTRY_SIZE] { request(handle, 1, &[0, 0, 0, 0, 0, 0], &[]) diff --git a/src/dev/isp12160.rs b/src/dev/isp12160.rs new file mode 100644 index 000000000..9fcf55ab1 --- /dev/null +++ b/src/dev/isp12160.rs @@ -0,0 +1,338 @@ +//! The QLogic ISP12160: two SCSI buses behind one PCI function. +//! +//! The ISP12160 shares the ISP1040 ring, DMA, and mailbox mechanism, while +//! selecting targets from either channel. Each channel remains an explicit +//! [`ScsiBus`], including for monitor and run-time disc operations. +//! +//! Ported from hw/scsi/qlogic-isp.c and include/hw/scsi/qlogic-isp.h +//! in jamesbraid/qemu-sgi-origami. + +use std::collections::VecDeque; +use std::ops::Deref; + +use parking_lot::MutexGuard; + +use crate::dev::isp1040::{ + mbox, reg, Bank, Core, Isp1040, IspPart, State, CFG1_BANK_SELECT, FW_VERSION, IOCB_ABORTED, IOCB_RESET, MAILBOX_COUNT, + MAX_REQUESTS, MAX_TARGET, PCI_CLASS_SCSI, PCI_VENDOR_QLOGIC, SEMAPHORE_LOCK, STATE_ALL_PHASES, +}; +use crate::irq::IrqLine; +use crate::pci::{Bar, PciBusDevice, PciConfig, PciConfigPolicy, PciDma, PciRegisterPolicy, PCI_INTA}; +use crate::scsi::{ScsiBus, ScsiIdentities, ScsiIdentity}; +use crate::snapshot::{get_field, hex_u16, hex_u32, load_u16_slice, toml_u16, toml_u32, toml_u8, u16_slice_to_toml}; +use crate::traits::{Device, Resettable, Saveable}; +use crate::workstation::DiscChanger; + +const ISP_DISK: ScsiIdentity = ScsiIdentity::new(b"SGI ", b"IRIS EMULATED ", b"1.0 "); +const ISP_CDROM: ScsiIdentity = ISP_DISK; + +pub(crate) struct Isp12160Part; + +impl IspPart for Isp12160Part { + fn name(&self) -> &'static str { "ISP12160" } + fn channels(&self) -> usize { 2 } + fn pci_config(&self, bars: [Bar; 6]) -> PciConfig { + let registers = [ + PciRegisterPolicy { reg_dword: 13, initial: 0x44, writable_mask: 0 }, + PciRegisterPolicy { reg_dword: 17, initial: 0x0001_0001, writable_mask: 0 }, + PciRegisterPolicy { reg_dword: 18, initial: 0, writable_mask: 0x100 }, + ]; + PciConfig::new_with_policy( + PCI_VENDOR_QLOGIC, + 0x1216, + PCI_CLASS_SCSI, + 6, + bars, + PCI_INTA, + PciConfigPolicy { + status: 0x02b0, + status_w1c_mask: 0, + command_mask: 0x0156, + latency_mask: 0xff, + cache_line_mask: 0xff, + registers: ®isters, + }, + ) + } + fn firmware_version(&self) -> [u16; 3] { [10, 4, FW_VERSION[2]] } + fn active_bank(&self, cfg1: u16) -> Bank { + match (cfg1 >> 8) & 3 { + 1 => Bank::Sxp, + 2 => Bank::Sxp1, + 3 => Bank::Dma, + _ => Bank::Risc, + } + } + fn checks_ram_bounds(&self) -> bool { true } + fn has_abort_commands(&self) -> bool { true } + fn target_on(&self, channel: u8, target: u8) -> u8 { target | (channel << 4) } + fn decode_target(&self, target: u8) -> (usize, u8) { ((target >> 4) as usize, target & 0x0f) } + fn id_high(&self) -> u16 { 0x1216 } + fn bus_cfg0(&self) -> u16 { 0 } + fn cfg1_write_mask(&self) -> u16 { 0x0300 } + fn unmodelled_dma_register(&self, off: u32) -> bool { + matches!(off, reg::DMA_CMD_CFG | reg::DMA_CMD_CTRL | reg::DMA_DATA_CFG | reg::DMA_DATA_CTRL) + } + fn completion_phases(&self, moved: u64, total: u64) -> u16 { + let mut flags = 0x1700; + if moved != 0 { + flags |= 0x0800; + if moved == total { flags |= 0x4000; } + } + flags + } + fn mailbox_ready(&self, semaphore: u16) -> bool { semaphore & SEMAPHORE_LOCK == 0 } + fn clear_target_parameters(&self, mailbox: &mut [u16; MAILBOX_COUNT]) { mailbox[6] = 0; } + fn set_reset_delay(&self, st: &mut State) { + st.reset_delay = st.mailbox[1]; + st.reset_delay1 = st.mailbox[2]; + } + fn reset_targets(&self, st: &mut State) -> bool { + st.reset_targets = true; + true + } + fn execute_mailbox(&self, core: &Core, st: &mut MutexGuard<'_, State>) -> bool { + let cmd = st.mailbox[0]; + match cmd { + mbox::LOAD_RAM_A64 => st.mailbox[0] = core.ram_transfer(st, true), + mbox::ABORT_COMMAND | mbox::ABORT_DEVICE | mbox::ABORT_TARGET => { + let device = st.mailbox[1]; + let valid_target = (device >> 8) & 0x7f <= MAX_TARGET as u16; + let target = ((device >> 8) & 0x7f) as u8 | (((device >> 15) as u8) << 4); + let lun = device as u8; + let handle = ((st.mailbox[2] as u32) << 16) | st.mailbox[3] as u32; + let hit = valid_target && st.active.is_some_and(|(t, l, h)| t == target + && (cmd == mbox::ABORT_TARGET || l == lun) && (cmd != mbox::ABORT_COMMAND || h == handle)); + let completion = if cmd == mbox::ABORT_TARGET { IOCB_RESET } else { IOCB_ABORTED }; + if hit { core.cancel_active(st, completion); } + let retired = core.retire_outstanding(st, |(t, l, h)| valid_target && t == target + && (cmd == mbox::ABORT_TARGET || l == lun) && (cmd != mbox::ABORT_COMMAND || h == handle), completion); + if cmd == mbox::ABORT_TARGET && valid_target { + st.pending_target_resets.push_back((target >> 4, Some(target & 15))); + } + st.mailbox[0] = if cmd == mbox::ABORT_COMMAND && !hit && retired == 0 { mbox::COMMAND_ERROR } else { mbox::COMPLETE }; + } + // Mailbox 1 is the reset delay; mailbox 2 selects the bus. + mbox::BUS_RESET => { + let channel = st.mailbox[2]; + if channel > 1 { st.mailbox[0] = mbox::PARAM_ERROR; } + else { + if st.active.is_some_and(|(target, _, _)| target >> 4 == channel as u8) { + core.cancel_active(st, IOCB_RESET); + } + core.retire_outstanding(st, |(target, _, _)| target >> 4 == channel as u8, IOCB_RESET); + st.pending_target_resets.push_back((channel as u8, None)); + core.note_bus_reset(usize::from(channel)); + core.kick(st); + st.mailbox[0] = mbox::COMPLETE; + } + } + _ => return false, + } + true + } + fn save_extra(&self, st: &State, t: &mut toml::map::Map) { + t.insert("reset_targets".into(), toml::Value::Boolean(st.reset_targets)); + t.insert("target_resets".into(), toml::Value::Array(st.pending_target_resets.iter().map(|(channel, target)| { + toml::Value::Array(vec![hex_u16(*channel as u16), hex_u16(target.unwrap_or(0xff) as u16)]) + }).collect())); + t.insert("sxp1_bank".into(), u16_slice_to_toml(&st.sxp1_bank)); + t.insert("dma_bank".into(), u16_slice_to_toml(&st.dma_bank)); + t.insert("reset_delay1".into(), hex_u16(st.reset_delay1)); + t.insert("outstanding".into(), toml::Value::Array(st.outstanding.iter().map(|&(target, lun, handle)| { + toml::Value::Array(vec![hex_u16(target as u16), hex_u16(lun as u16), hex_u32(handle)]) + }).collect())); + } + fn load_extra(&self, st: &mut State, v: &toml::Value) { + st.reset_targets = get_field(v, "reset_targets").and_then(|x| x.as_bool()).unwrap_or(false); + if let Some(toml::Value::Array(items)) = get_field(v, "target_resets") { + for item in items.iter().take(64) { + if let toml::Value::Array(fields) = item { + if fields.len() == 2 { + if let (Some(channel), Some(target)) = (toml_u8(&fields[0]), toml_u8(&fields[1])) { + if channel < 2 && (target < 16 || target == 0xff) { + st.pending_target_resets.push_back((channel, (target < 16).then_some(target))); + } + } + } + } + } + } + let u16f = |k: &str| get_field(v, k).and_then(toml_u16).unwrap_or(0); + st.reset_delay1 = u16f("reset_delay1"); + if let Some(m) = get_field(v, "sxp1_bank") { load_u16_slice(m, &mut st.sxp1_bank); } + if let Some(m) = get_field(v, "dma_bank") { load_u16_slice(m, &mut st.dma_bank); } + if let Some(toml::Value::Array(items)) = get_field(v, "outstanding") { + for item in items.iter().take(MAX_REQUESTS as usize) { + if let toml::Value::Array(fields) = item { + if fields.len() == 3 { + if let (Some(target), Some(lun), Some(handle)) = + (toml_u8(&fields[0]), toml_u8(&fields[1]), toml_u32(&fields[2])) { + if target < 32 && lun < 8 { st.outstanding.push((target, lun, handle)); } + } + } + } + } + } + } +} + +/// The two-channel controller. The shared ISP engine owns both buses. +pub struct Isp12160(Isp1040); + +impl Default for Isp12160 { + fn default() -> Self { Self::new(Self::new_bus(), Self::new_bus()) } +} + +impl Isp12160 { + /// Light the status bar's SCSI activity bits in `heartbeat`. + pub fn set_heartbeat(&self, heartbeat: std::sync::Arc) { + self.0.set_heartbeat(heartbeat); + } + + /// A 16-id bus with the ISP controller's disk and CD-ROM identities. + pub fn new_bus() -> ScsiBus { + ScsiBus::new(16, ScsiIdentities { disk: ISP_DISK, cdrom: ISP_CDROM }) + } + + /// The controller with its two channel buses already populated. + pub fn new(bus0: ScsiBus, bus1: ScsiBus) -> Self { + Self(Isp1040::new_12160(bus0, bus1)) + } + + /// Borrow one channel for the shared monitor commands. + pub fn channel(&self, channel: u8) -> Option> { + (channel < 2).then_some(Channel { adapter: self, channel }) + } + + pub fn register_locks(&self) { self.0.register_locks(); } +} + +impl Deref for Isp12160 { + type Target = Core; + fn deref(&self) -> &Core { &self.0 } +} + +/// One channel, with SCSI ids 0–15 as the monitor presents them. +pub struct Channel<'a> { + adapter: &'a Isp12160, + channel: u8, +} + +impl crate::scsi_monitor::ScsiTargets for Channel<'_> { + fn scsi_bus(&self) -> &ScsiBus { + self.adapter.bus_for_channel(self.channel).expect("valid ISP12160 channel") + } +} + +/// The machine API flattens the ISP12160's channels to ids 0–31. +/// Monitor commands use a `Channel` instead, so their ids remain 0–15. +impl DiscChanger for Isp12160 { + fn disc_status(&self) -> Vec<(usize, String, Vec, u64, u64)> { + (0..2).flat_map(|channel| { + crate::scsi_monitor::disc_status(&self.channel(channel).expect("valid ISP12160 channel")) + .into_iter() + .map(move |(id, current, discs, physical, logical)| (id + channel as usize * 16, current, discs, physical, logical)) + }).collect() + } + + fn load_disc(&self, id: usize, path: String) -> Result { + let (channel, target) = Self::split_id(id)?; + crate::scsi_monitor::load_disc(&self.channel(channel).unwrap(), target, path) + } + + fn eject_disc(&self, id: usize) -> Result { + let (channel, target) = Self::split_id(id)?; + crate::scsi_monitor::eject_disc(&self.channel(channel).unwrap(), target) + } + + fn eject_to_empty(&self, id: usize) -> Result<(), String> { + let (channel, target) = Self::split_id(id)?; + crate::scsi_monitor::eject_to_empty(&self.channel(channel).unwrap(), target) + } +} + +impl Isp12160 { + fn split_id(id: usize) -> Result<(u8, usize), String> { + if id >= 32 { return Err(format!("SCSI ID {id} is out of range")); } + Ok(((id / 16) as u8, id % 16)) + } +} + +impl Device for Isp12160 { + fn step(&self, cycles: u64) { Device::step(&self.0, cycles) } + fn start(&self) { Device::start(&self.0) } + fn stop(&self) { Device::stop(&self.0) } + fn is_running(&self) -> bool { Device::is_running(&self.0) } + fn get_clock(&self) -> u64 { Device::get_clock(&self.0) } + + fn register_commands(&self) -> Vec<(String, String)> { + let usage = crate::scsi_monitor::SCSI_USAGE; + vec![ + ("scsi".into(), format!("SCSI (QLogic ISP12160, channel 0): {usage} | scsi notes")), + ("scsi0".into(), "Alias for `scsi` (ISP12160 channel 0)".into()), + ("cow".into(), "COW overlays on ISP12160 channel 0: cow status | cow commit [id] | cow reset [id]".into()), + ("scsi1".into(), format!("SCSI (QLogic ISP12160, channel 1): {usage}")), + ("cow1".into(), "COW overlays on ISP12160 channel 1: same subcommands as `cow`".into()), + ] + } + + fn execute_command(&self, cmd: &str, args: &[&str], mut w: Box) -> Result<(), String> { + use std::io::Write as _; + let (channel, cow) = match cmd { + "scsi" | "scsi0" | "cow" => (0, cmd.starts_with("cow")), + "scsi1" | "cow1" => (1, cmd.starts_with("cow")), + _ => return Err("Command not found".into()), + }; + let ctl = self.channel(channel).expect("valid ISP12160 channel"); + if cow { + return crate::scsi_monitor::cow_command(&ctl, args, &mut *w); + } + if let Some(done) = crate::scsi_monitor::scsi_subcommand(&ctl, args, &mut *w) { + return done; + } + if channel == 0 && args.first().copied() == Some("notes") { + let _ = writeln!(w, "QLogic ISP12160: {} commands", self.total_commands()); + for note in self.notes() { let _ = writeln!(w, " note: {note}"); } + return Ok(()); + } + Err(format!("Usage: {}", crate::scsi_monitor::SCSI_USAGE)) + } +} + +impl Resettable for Isp12160 { + fn power_on(&self) { + self.0.reset(); + PciBusDevice::config(&self.0).reset(); + } +} + +impl Saveable for Isp12160 { + fn save_state(&self) -> toml::Value { Saveable::save_state(&self.0) } + fn load_state(&self, v: &toml::Value) -> Result<(), String> { Saveable::load_state(&self.0, v) } +} + +impl PciBusDevice for Isp12160 { + fn config(&self) -> &PciConfig { PciBusDevice::config(&self.0) } + fn bar_read(&self, bar: usize, off: u32, width: usize) -> u32 { PciBusDevice::bar_read(&self.0, bar, off, width) } + fn bar_write(&self, bar: usize, off: u32, val: u32, width: usize) { PciBusDevice::bar_write(&self.0, bar, off, val, width) } + fn connect_interrupt(&self, line: IrqLine) { PciBusDevice::connect_interrupt(&self.0, line) } + fn connect_dma(&self, dma: std::sync::Arc) { PciBusDevice::connect_dma(&self.0, dma) } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// The machine disc API uses target 17 for channel 1, target 1. + #[test] + fn disc_status_flattens_the_second_channels_target_ids() { + let bus0 = Isp12160::new_bus(); + let bus1 = Isp12160::new_bus(); + bus1.attach_device(1, "", true, Vec::new(), false, None, false).unwrap(); + let isp = Isp12160::new(bus0, bus1); + + let status = DiscChanger::disc_status(&isp); + assert_eq!(status.iter().map(|drive| drive.0).collect::>(), vec![17]); + } +} diff --git a/src/dev/l1.rs b/src/dev/l1.rs new file mode 100644 index 000000000..a20535ade --- /dev/null +++ b/src/dev/l1.rs @@ -0,0 +1,460 @@ +//! Fuel's L1 system controller and its processor-side UART. +//! +//! The internal L1 connection is separate from IOC3's external serial ports. +//! The Ns16550 owns the UART registers, FIFO, timeout and interrupt state; +//! this device keeps the controller protocol and its eight-byte register stride. +//! Controller identities and power transitions are functional approximations. +//! Remote routing and controller diagnostics are absent. A configured router +//! interpreter answers the single-fragment `brick` request for the Chimera +//! brick types; other interpreter commands and multi-fragment messages are +//! refused and logged. +//! +//! Ported from hw/misc/sgi-l1.c in jamesbraid/qemu-sgi-origami. + +use crate::dev::ns16550::{com_reg, Ns16550}; +use crate::dev::serial_host::{null_serial_backend, SerialBackend}; +use crate::devlog::LogModule; +use crate::irq::IrqLine; +use crate::traits::{BusDevice, BusRead64, Device, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::Mutex; +use std::collections::VecDeque; +use std::io; +use std::sync::{ + atomic::{AtomicBool, AtomicU64, Ordering}, + Arc, +}; + +mod model; +use model::State; +pub use model::{parse_revision, L1Config, L1Event, DEFAULT_BRICK_SERIAL, DEFAULT_REVISION}; + +fn fcs(mut crc: u16, byte: u8) -> u16 { + crc ^= byte as u16; + for _ in 0..8 { + crc = (crc >> 1) ^ if crc & 1 != 0 { 0x8408 } else { 0 }; + } + crc +} + +fn encode(payload: &[u8]) -> Vec { + let crc = !payload.iter().fold(0xffff, |value, byte| fcs(value, *byte)); + let mut out = vec![0x7e]; + for byte in payload.iter().copied().chain([crc as u8, (crc >> 8) as u8]) { + if matches!(byte, 0x7e | 0x7d | 0x0d) { + out.extend_from_slice(&[0x7d, byte ^ 0x20]); + } else { + out.push(byte); + } + } + out.push(0x7e); + out +} + +const UART_BAUDBASE: u32 = 115200; +const UART_WINDOW_SIZE: u64 = 0x40; + +mod uart_offset { + use super::com_reg; + pub const DATA: u64 = 0x00; + pub const IER: u64 = 0x08; + pub const IIR_FCR: u64 = 0x10; + pub const LCR: u64 = 0x18; + pub const MCR: u64 = 0x20; + pub const LSR: u64 = 0x28; + pub const MSR: u64 = 0x30; + pub const SCR: u64 = 0x38; + + pub fn register(offset: u64) -> u32 { + match offset { + DATA => com_reg::THR, + IER => com_reg::IER, + IIR_FCR => com_reg::IIR_FCR, + LCR => com_reg::LCR, + MCR => com_reg::MCR, + LSR => com_reg::LSR, + MSR => com_reg::MSR, + SCR => com_reg::SCR, + _ => unreachable!("validated L1 UART offset"), + } + } +} + +type EventHandler = Arc; + +#[derive(Default)] +struct Dispatch { + active: bool, + pending: bool, + level: Option, +} + +/// The UART backend does not connect guest L1 commands to a host terminal. +/// Guest transmit bytes are handed to the protocol parser after the UART lock +/// is released; receive bytes come from the protocol reply queue. +struct ProtocolBackend { + guest_tx: Arc>>, + protocol_rx: Arc>>, +} + +impl SerialBackend for ProtocolBackend { + fn send_byte(&self, byte: u8) { + self.guest_tx.lock().push_back(byte); + } + + fn recv_byte(&self) -> io::Result { + self.protocol_rx + .lock() + .pop_front() + .ok_or_else(|| io::ErrorKind::WouldBlock.into()) + } +} + +/// The internal L1 connection is independent of IOC3's external serial ports. +/// Register offsets are relative to the 0x40-byte UART window. Only aligned +/// 64-bit accesses are decoded; register values occupy the low eight bits. +pub struct L1 { + // UART RX takes UART -> protocol queue. Model replies take state -> protocol + // queue. No path holds the protocol queue while acquiring either lock. + state: Arc>, + // Take dispatch before state when draining, release it before the UART or callbacks. + dispatch: Arc>, + uart: Arc>, + guest_tx: Arc>>, + backend: Arc>>, + irq: Arc>, + handler: Arc>>, + running: Arc, + clock: Arc, + worker: Mutex>>, + wake: Arc<(Mutex<()>, parking_lot::Condvar)>, +} +impl L1 { + pub fn new(config: L1Config) -> Result { + if config.rack > 0x3ff || config.bay > 0x3f || config.node_position > 3 { + return Err("L1 rack, bay or node position out of range".into()); + } + if config.port_speed_first > 1 { + return Err("L1 first port-speed value must be zero or one".into()); + } + if config.mac == [0; 6] || config.mac[0] & 1 != 0 { + return Err("L1 MAC must be nonzero unicast".into()); + } + if config + .brick_serial + .as_ref() + .is_some_and(|s| s.len() > 37 || s.as_bytes().contains(&0)) + { + return Err("L1 brick serial must fit a 37-byte text field".into()); + } + + let protocol_rx = Arc::new(Mutex::new(VecDeque::new())); + let state = Arc::new(Mutex::new(State::new(config, protocol_rx.clone()))); + let guest_tx = Arc::new(Mutex::new(VecDeque::new())); + let protocol_backend = Arc::new(ProtocolBackend { + guest_tx: guest_tx.clone(), + protocol_rx, + }); + let mut uart = Ns16550::new(UART_BAUDBASE); + uart.set_backend(protocol_backend); + + Ok(Self { + state, + dispatch: Arc::new(Mutex::new(Dispatch::default())), + uart: Arc::new(Mutex::new(uart)), + guest_tx, + backend: Arc::new(Mutex::new(null_serial_backend())), + irq: Arc::new(Mutex::new(IrqLine::unconnected())), + handler: Arc::new(Mutex::new(None)), + running: Arc::new(AtomicBool::new(false)), + clock: Arc::new(AtomicU64::new(0)), + worker: Mutex::new(None), + wake: Arc::new((Mutex::new(()), parking_lot::Condvar::new())), + }) + } + + /// Install the separate host-side L1 console connection. + pub fn set_serial_backend(&self, backend: Arc) { + *self.backend.lock() = backend; + } + + pub fn set_irq(&self, line: IrqLine) { + *self.irq.lock() = line; + self.dispatch.lock().level = None; + self.flush(); + } + + /// Power/reset requests reach the machine only after the reply is queued + /// and after the device state lock is released. + pub fn set_event_handler(&self, handler: EventHandler) { + *self.handler.lock() = Some(handler); + } + + #[cfg(test)] + pub fn node_powered(&self) -> bool { + self.state.lock().powered + } + + pub fn station_mac(&self) -> [u8; 6] { + self.state.lock().config.mac + } + + #[cfg(test)] + pub fn config(&self) -> L1Config { + self.state.lock().config.clone() + } + + #[cfg(test)] + pub fn display_line(&self) -> String { + self.state.lock().display.clone() + } + + #[cfg(test)] + pub fn record(&self, component: usize, area: u32) -> Option> { + let index = match area { + 1 => 0, + 2 => 1, + 4 => 2, + _ => return None, + }; + self.state.lock().records.get(component).and_then(|r| { + if r[index].is_empty() { + None + } else { + Some(r[index].clone()) + } + }) + } + + /// Install a raw area before execution. It replaces only the named area; + /// station identity remains the configured MAC, as in the Fuel model. + #[cfg(test)] + pub fn set_record(&self, component: usize, area: u32, bytes: Vec) -> Result<(), String> { + let index = match area { + 1 => 0, + 2 => 1, + 4 => 2, + _ => return Err("unsupported L1 area".into()), + }; + if component >= 16 || bytes.len() > 65535 { + return Err("L1 record out of range".into()); + } + self.state.lock().records[component][index] = bytes; + Ok(()) + } + + pub fn send_event(&self, code: u32, number: u32, text: &str) { + self.state.lock().event(code, number, text); + self.service_protocol_uart(); + self.flush(); + } + + /// Deterministic input path shared with the polled host serial backend. + pub fn receive_console(&self, bytes: &[u8]) { + for &byte in bytes { + self.state.lock().console_input(byte); + self.service_protocol_uart(); + self.flush(); + } + } + + pub fn poll_serial(&self) { + let backend = self.backend.lock().clone(); + for _ in 0..256 { + if self.state.lock().queue.lock().len() > 4088 { + break; + } + match backend.recv_byte() { + Ok(byte) => self.receive_console(&[byte]), + Err(error) => { + if error.kind() != io::ErrorKind::WouldBlock { + log::debug!("L1 serial input: {error}"); + } + break; + } + } + } + } + + /// Let the native UART pull reply bytes from the model's shared bounded queue. + fn service_protocol_uart(&self) { + let mut uart = self.uart.lock(); + uart.pump(); + uart.check_timeout(std::time::Instant::now()); + } + + /// Parse guest transmit only after the UART lock is released. The UART + /// callback only appends to this independent queue. + fn process_guest_transmit(&self) { + let bytes: Vec<_> = self.guest_tx.lock().drain(..).collect(); + if bytes.is_empty() { + return; + } + { + let mut state = self.state.lock(); + for byte in bytes { + state.parse(byte); + } + } + self.service_protocol_uart(); + } + + fn flush(&self) { + let mut dispatch = self.dispatch.lock(); + dispatch.pending = true; + if dispatch.active { + return; + } + dispatch.active = true; + loop { + dispatch.pending = false; + let (output, actions) = { + let mut state = self.state.lock(); + ( + std::mem::take(&mut state.output), + std::mem::take(&mut state.actions), + ) + }; + drop(dispatch); + let level = self.uart.lock().irq(); + + // Reentrant reads request another drain, but must not repeat an + // unchanged IRQ and recursively trigger the same machine callback. + dispatch = self.dispatch.lock(); + let changed = dispatch.level != Some(level); + dispatch.level = Some(level); + drop(dispatch); + + if changed { + let irq = self.irq.lock().clone(); + irq.set(level); + } + let backend = self.backend.lock().clone(); + for byte in output { + backend.send_byte(byte); + } + let handler = self.handler.lock().clone(); + for action in actions { + if let Some(handler) = &handler { + handler(action); + } else { + crate::dlog!(LogModule::L1, "unconnected L1 action {action:?}"); + } + } + + dispatch = self.dispatch.lock(); + if !dispatch.pending { + dispatch.active = false; + break; + } + } + } +} +impl BusDevice for L1 { + fn read64(&self, addr: u64) -> BusRead64 { + if addr >= UART_WINDOW_SIZE || addr & 7 != 0 { + return BusRead64::err(); + } + let value = self.uart.lock().read(uart_offset::register(addr)); + self.flush(); + BusRead64::ok(value as u64) + } + + fn write64(&self, addr: u64, value: u64) -> u32 { + if addr >= UART_WINDOW_SIZE || addr & 7 != 0 { + return BUS_ERR; + } + let register = uart_offset::register(addr); + let value = value as u8; + if register == com_reg::IIR_FCR && value & com_reg::FCR_RFR != 0 { + self.state.lock().queue.lock().clear(); + } + self.uart.lock().write(register, value); + self.process_guest_transmit(); + self.flush(); + BUS_OK + } +} +impl Resettable for L1 { + fn power_on(&self) { + self.state.lock().reset(); + self.guest_tx.lock().clear(); + self.uart.lock().reset(); + self.clock.store(0, Ordering::Relaxed); + self.flush(); + } +} +impl Device for L1 { + fn start(&self) { + let mut worker = self.worker.lock(); + if worker.is_some() { + return; + } + self.running.store(true, Ordering::Release); + let device = Self { + state: self.state.clone(), + dispatch: self.dispatch.clone(), + uart: self.uart.clone(), + guest_tx: self.guest_tx.clone(), + backend: self.backend.clone(), + irq: self.irq.clone(), + handler: self.handler.clone(), + running: self.running.clone(), + clock: self.clock.clone(), + worker: Mutex::new(None), + wake: self.wake.clone(), + }; + *worker = Some( + std::thread::Builder::new() + .name("l1-serial".into()) + .spawn(move || { + while device.is_running() { + device.poll_serial(); + let mut guard = device.wake.0.lock(); + if device.is_running() { + device + .wake + .1 + .wait_for(&mut guard, std::time::Duration::from_millis(2)); + } + } + }) + .expect("start L1 serial worker"), + ); + } + + fn stop(&self) { + let mut slot = self.worker.lock(); + { + let _guard = self.wake.0.lock(); + self.running.store(false, Ordering::Release); + self.wake.1.notify_all(); + } + let worker = slot.take(); + if let Some(worker) = worker { + if worker.thread().id() != std::thread::current().id() { + let _ = worker.join(); + } + } + } + + fn is_running(&self) -> bool { + self.running.load(Ordering::Acquire) + } + + fn get_clock(&self) -> u64 { + self.clock.load(Ordering::Relaxed) + } + + fn step(&self, cycles: u64) { + if self.is_running() { + self.clock.fetch_add(cycles, Ordering::Relaxed); + } + } +} +impl Drop for L1 { + fn drop(&mut self) { + self.stop(); + } +} +#[cfg(test)] +mod tests; diff --git a/src/dev/l1/model.rs b/src/dev/l1/model.rs new file mode 100644 index 000000000..478249211 --- /dev/null +++ b/src/dev/l1/model.rs @@ -0,0 +1,979 @@ +//! Fuel's L1 controller protocol parser and reply model. +//! +//! This state owns packet framing, controller identity records, console +//! commands and power events. The UART register file and receive pacing live +//! in the shared Ns16550 core. +//! +//! Ported from hw/misc/sgi-l1.c in jamesbraid/qemu-sgi-origami. + +use super::*; + +/// Firmware revision the model reports for request 0x0005. +/// +/// Model choice: 1.48.1 is the revision a retained real Fuel L1 session +/// prints (the Fuel contract's L1 section). The model reproduces that +/// firmware's replies, not its code, so the revision is a label. +pub const DEFAULT_REVISION: [u32; 3] = [1, 48, 1]; + +/// Brick serial the model reports for request 0x0004. +/// +/// Model choice: the marked synthetic text form, system-board letter `N`, +/// instance 0 (the component taxonomy's synthetic serials). The real format +/// is unknown, and the value must not pass as a real SGI serial. +pub const DEFAULT_BRICK_SERIAL: &str = "EMUN00"; + +/// Parse a "major.minor.patch" revision string, as a `[fuel] l1_revision` +/// override spells it. Each part is a decimal number. +pub fn parse_revision(text: &str) -> Result<[u32; 3], String> { + let bad = || format!("L1 revision \"{text}\" must be major.minor.patch, each a decimal number"); + let mut parts = text.split('.'); + let mut rev = [0u32; 3]; + for slot in &mut rev { + let part = parts.next().ok_or_else(bad)?; + if part.is_empty() || !part.bytes().all(|b| b.is_ascii_digit()) { + return Err(bad()); + } + *slot = part.parse().map_err(|_| bad())?; + } + if parts.next().is_some() { + return Err(bad()); + } + Ok(rev) +} + +#[derive(Clone, Debug)] +pub struct L1Config { + pub rack: u32, + pub bay: u32, + pub node_position: u8, + pub type_code: u32, + pub revision: Option<[u32; 3]>, + pub mac: [u8; 6], + pub brick_serial: Option, + pub auto_power: bool, + /// First integer of the port-speed reply (request 0xa). The PROM uses it + /// to select a display branch; it is not a link speed. Zero or one. + pub port_speed_first: u32, + /// Answer the router command-interpreter `brick` request. Only the brick + /// types in `router_type_name` have a reply. + pub router: bool, +} +impl Default for L1Config { + fn default() -> Self { + Self { + rack: 1, + bay: 1, + node_position: 0, + type_code: 0x41, + revision: Some(DEFAULT_REVISION), + mac: [8, 0, 0x69, 0x12, 0x34, 0x56], + brick_serial: Some(DEFAULT_BRICK_SERIAL.into()), + auto_power: true, + port_speed_first: 0, + router: false, + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum L1Event { + Power(bool), + Reset, + SoftReset, + Nmi, + Shutdown, +} + +pub(super) struct State { + pub config: L1Config, + pub queue: Arc>>, + frame: Vec, + in_frame: bool, + escape: bool, + overflow: bool, + pub display: String, + pub records: [[Vec; 3]; 16], + pub actions: Vec, + pub output: Vec, + pub powered: bool, + console: bool, + escaped: bool, + command: Vec, + port: u8, + debug: u32, + partition: u32, + two_bit: u32, + reasons: [u32; 2], + events: Option, + power_down_pending: bool, + /// One reassembly per subchannel. + router: [Option; 32], +} + +/// A router message being reassembled from 0x3001, 0x3002 and 0x3003 chunks. +struct RouterAssembly { + fragment_count: u8, + fragments_left: u8, + first_length: usize, + expected: usize, + received: usize, + data: Vec, +} +const ROUTER_MAX_FRAGMENT: usize = 0x400; +const ROUTER_CHUNK_DATA: usize = 113; + +/// Brick types for which the router interpreter reply exists. +fn router_type_name(type_code: u32) -> Option<&'static str> { + match type_code { + 0x37 => Some("Chimera Server"), + 0x39 => Some("Chimera WS"), + 0x51 => Some("Rackmount WS"), + _ => None, + } +} +fn text_field(out: &mut Vec, text: &[u8]) { + out.push(1); + out.extend_from_slice(text); + out.push(0); +} +pub(super) fn ints(values: &[u32]) -> Vec { + let mut out = Vec::new(); + for v in values { + out.push(0); + out.extend_from_slice(&v.to_be_bytes()); + } + out +} +fn word(p: &[u8]) -> u32 { + u32::from_be_bytes(p[..4].try_into().unwrap()) +} +fn arguments(mut p: &[u8], count: u8) -> Option> { + let mut fields = Vec::new(); + for _ in 0..count { + let tag = *p.first()?; + let len = match tag { + 0 => 5, + 1 => p[1..].iter().position(|b| *b == 0)? + 2, + t if t & 0x80 != 0 => 1 + (t & 0x7f) as usize, + _ => return None, + }; + fields.push(p.get(..len)?); + p = &p[len..]; + } + if p.is_empty() { + Some(fields) + } else { + None + } +} +fn area_finish(mut area: Vec) -> Vec { + while (area.len() + 1) % 8 != 0 { + area.push(0); + } + let sum = area.iter().fold(0u8, |s, b| s.wrapping_add(*b)); + area.push(sum.wrapping_neg()); + area +} +fn ia_text(area: &mut Vec, text: &str) { + area.push(0xc0 | text.len() as u8); + area.extend_from_slice(text.as_bytes()); +} +fn board(mac: Option<[u8; 6]>) -> Vec { + let mut b = vec![0; 6]; + for s in ["SYNTHETIC", "IP34", "EMUN00", "030-1707-003"] { + ia_text(&mut b, s); + } + b.push(0); + ia_text(&mut b, "-G"); + b.extend_from_slice(&[1, 0]); + ia_text(&mut b, "00"); + for _ in 0..3 { + b.extend_from_slice(&[4, 0, 0, 0, 0]); + } + if let Some(mac) = mac { + ia_text( + &mut b, + &mac.iter().map(|v| format!("{v:02X}")).collect::(), + ); + } + b.push(0xc1); + // The count includes the checksum and padding, not a separate header. + b[1] = (b.len() + 1).div_ceil(8) as u8; + area_finish(b) +} +impl State { + pub fn new(config: L1Config, queue: Arc>>) -> Self { + let mut records: [[Vec; 3]; 16] = + std::array::from_fn(|_| std::array::from_fn(|_| Vec::new())); + let chassis = area_finish(vec![0, 2, 0, 0xc2, b'N', b'A', 0xc2, b'N', b'A', 0xc1]); + for c in 0..2 { + records[c][0] = chassis.clone(); + records[c][1] = board(if c == 1 { Some(config.mac) } else { None }); + } + let powered = config.auto_power; + let port = config.node_position << 5; + Self { + config, + queue, + frame: Vec::new(), + in_frame: false, + escape: false, + overflow: false, + display: "L1 running".into(), + records, + actions: Vec::new(), + output: Vec::new(), + powered, + console: powered, + escaped: false, + command: Vec::new(), + port, + debug: 0, + partition: 0, + two_bit: 0, + reasons: [0; 2], + events: None, + power_down_pending: false, + router: std::array::from_fn(|_| None), + } + } + pub fn reset(&mut self) { + // Controller settings survive the controller's local boot. The UART and + // packet parser reset independently of those persistent settings. + let persistent = (self.debug, self.partition, self.two_bit, self.reasons); + let events = self.events.filter(|&channel| channel >= 4); + let records = std::mem::replace( + &mut self.records, + std::array::from_fn(|_| std::array::from_fn(|_| Vec::new())), + ); + let queue = self.queue.clone(); + queue.lock().clear(); + *self = Self::new(self.config.clone(), queue); + (self.debug, self.partition, self.two_bit, self.reasons) = persistent; + self.events = events; + self.records = records; + } + fn emit(&mut self, payload: &[u8]) { + if payload.len() > 128 { + crate::dlog!(LogModule::L1, "L1 reply exceeds frame capacity"); + return; + } + let bytes = encode(payload); + let mut queue = self.queue.lock(); + if queue.len() + bytes.len() <= 4096 { + queue.extend(bytes); + } else { + crate::dlog!(LogModule::L1, "L1 receive queue full"); + } + } + fn reply(&mut self, channel: u8, code: i32, count: u8, fields: &[u8]) { + let mut p = vec![0x20 | channel]; + p.extend_from_slice(&code.to_be_bytes()); + p.push(count); + p.extend_from_slice(fields); + self.emit(&p); + } + pub fn parse(&mut self, mut byte: u8) { + if byte == 0x7e { + if self.in_frame + && !self.escape + && !self.overflow + && self.frame.len() >= 3 + && self.frame.iter().fold(0xffff, |v, b| fcs(v, *b)) == 0xf0b8 + { + let payload = self.frame[..self.frame.len() - 2].to_vec(); + self.request(&payload); + } + self.in_frame = true; + self.escape = false; + self.overflow = false; + self.frame.clear(); + return; + } + if !self.in_frame || self.overflow { + return; + } + if byte == 0x7d { + self.escape = true; + return; + } + if self.escape { + byte ^= 0x20; + self.escape = false; + } + if self.frame.len() == 258 { + self.overflow = true; + } else { + self.frame.push(byte); + } + } + fn eeprom(&mut self, channel: u8, a: &[&[u8]]) { + if a.len() < 4 || a[..4].iter().any(|v| v[0] != 0) { + self.reply(channel, -100, 0, &[]); + return; + } + let component = word(&a[0][1..]) as usize; + let area = word(&a[1][1..]); + let offset = word(&a[2][1..]) as usize; + let requested = word(&a[3][1..]) as usize; + if component >= 16 || self.records[component].iter().all(Vec::is_empty) { + self.reply(channel, -104, 0, &[]); + return; + } + if area > 4 { + self.reply(channel, -105, 0, &[]); + return; + } + let index = match area { + 1 => 0, + 2 => 1, + 4 => 2, + _ => { + self.reply(channel, -104, 0, &[]); + return; + } + }; + let record = &self.records[component][index]; + if record.is_empty() { + self.reply(channel, -104, 0, &[]); + return; + } + let checksum_len = if area == 4 { + record.len().min(64) + } else { + record.len() + }; + if record[..checksum_len] + .iter() + .fold(0u8, |s, b| s.wrapping_add(*b)) + != 0 + { + self.reply(channel, -107, 0, &[]); + return; + } + if offset >= record.len() { + self.reply(channel, -105, 0, &[]); + return; + } + let len = requested.min(record.len() - offset); + if len > 116 { + crate::dlog!(LogModule::L1, "L1 EEPROM reply exceeds frame capacity"); + self.reply(channel, -105, 0, &[]); + return; + } + let mut fields = ints(&[(record.len() - offset - len) as u32]); + if requested != 0 { + fields.push(0x80 | len as u8); + fields.extend_from_slice(&record[offset..offset + len]); + } + self.reply(channel, 0, if requested == 0 { 1 } else { 2 }, &fields); + } + fn request(&mut self, p: &[u8]) { + let channel = p[0] & 31; + if self.config.router + && p.len() >= 7 + && p[0] & 0xe0 == 0 + && word(&p[1..]) == 0 + && p[5] == 0x30 + && matches!(p[6], 1..=3) + { + self.router_chunk(p); + return; + } + if p[0] & 0xe0 == 0x40 { + self.port = (self.port & 0x60) | channel; + if self.console && self.powered { + self.output.extend_from_slice(&p[1..]); + } else { + crate::dlog!(LogModule::L1, "L1 console output while operator is in L1 mode"); + } + return; + } + if p[0] & 0xe0 != 0 || p.len() < 7 { + return; + } + let address = word(&p[1..]); + let cmd = u16::from_be_bytes([p[5], p[6]]); + let key = address >> 12; + if key != 0xffff && key != ((self.config.rack << 6) | self.config.bay) { + crate::dlog!(LogModule::L1, "L1 unsupported destination {address:#x}"); + self.emit(&[0x20 | channel, 255, 255, 255, 255, 0, 0, 0, 8]); + return; + } + let count = if p.len() == 7 { 0 } else { p[7] }; + let raw = if p.len() > 8 { &p[8..] } else { &[] }; + let task = address & 31; + if task == 5 || task == 21 { + if self.powered { + self.emit(&[ + ((address >> 5) & 31) as u8, + raw.get(4).copied().unwrap_or(0), + ]); + } else { + let mut field = Vec::new(); + text_field( + &mut field, + format!( + "WARNING: power on {:03}c{:02} appears off!", + self.config.rack, self.config.bay + ) + .as_bytes(), + ); + self.reply(channel, -1, 1, &field); + } + return; + } + if task != 6 && task != 3 && task != 18 { + crate::dlog!(LogModule::L1, "L1 unsupported task {task}"); + return; + } + let Some(a) = arguments(raw, count) else { + self.reply(channel, -100, 0, &[]); + return; + }; + if task == 3 { + if a.len() != 1 || a[0][0] != 1 { + self.reply(channel, -100, 0, &[]); + return; + } + let text = String::from_utf8_lossy(&a[0][1..a[0].len() - 1]); + let (code, output) = self.run_line(&text, false); + let mut fields = Vec::new(); + let mut n = 0; + for line in output.split("\r\n").filter(|s| !s.is_empty()) { + if fields.len() + line.len() + 2 > 121 { + break; + } + text_field(&mut fields, line.as_bytes()); + n += 1; + } + self.reply(channel, code, n, &fields); + return; + } + if task == 18 { + let length = a.iter().take(3).map(|v| v.len()).sum::().min(121); + self.reply(channel, cmd as i32, count.min(3), &raw[..length]); + return; + } + let mut values = Vec::new(); + let mut fields = Vec::new(); + let mut field_count = 0; + let mut error = 0; + match cmd { + 1 => values.push(self.debug), + 2 => values.push((self.config.rack << 6) | self.config.bay), + 3 => { + if self.config.type_code == 0 { + error = -104; + } else { + values.extend([ + (self.config.rack << 6) | self.config.bay, + self.config.type_code, + ]); + } + } + 4 => { + if let Some(serial) = &self.config.brick_serial { + text_field(&mut fields, serial.as_bytes()); + field_count = 1; + } else { + error = -104; + } + } + 5 => { + if let Some(rev) = self.config.revision { + values.extend(rev); + } else { + crate::dlog!(LogModule::L1, "L1 revision metadata not configured"); + error = -104; + } + } + 6 => { + self.eeprom(channel, &a); + return; + } + 7 => error = if a.is_empty() { -100 } else { -104 }, + // The system serial is the station address as twelve uppercase + // hexadecimal digits in address order, no separators or OUI + // added: the text the stock PROM converts straight back into the + // Ethernet address (the Fuel contract's MAC section). The + // uppercase case is the contract's choice for the model. + 8 => { + let mac = self + .config + .mac + .iter() + .map(|b| format!("{b:02X}")) + .collect::(); + text_field(&mut fields, mac.as_bytes()); + field_count = 1; + } + 9 => values.push(self.partition), + 0xa => values.extend([self.config.port_speed_first, 0]), + 0xb => values.push(self.two_bit), + 0xc => values.extend([0; 8]), + 0xd => values.extend([0; 4]), + 0xe => values.push((self.port >> 5) as u32 & 3), + 0xf => values.extend(self.reasons), + 0x10 => values.extend([1, 1, 1, 1]), + 0x11 => { + error = -106; + values.push(1); + } + 0x1001 | 0x1002 | 0x1006 | 0x1007 | 0x1008 => { + if a.is_empty() { + error = -100; + } else if a[0][0] != 0 { + error = -105; + } else { + let v = word(&a[0][1..]); + match cmd { + 0x1001 => self.debug = v & 0xffff, + 0x1002 => self.port = ((self.port as u32 & 0x60) | v) as u8 & 0x7f, + 0x1006 => self.partition = v, + 0x1007 => self.events = Some(v as u8), + _ => self.two_bit = v & 3, + } + } + } + 0x1003 => {} + 0x1004 | 0x1005 => { + if a.is_empty() { + error = -100; + } else if a[0][0] != 1 { + error = -102; + } else { + self.display = a[0][1..a[0].len() - 1] + .iter() + .take(72) + .map(|b| { + if (0x20..=0x7e).contains(b) { + *b as char + } else { + '?' + } + }) + .collect(); + } + } + 0x1009 => { + if a.len() < 2 { + error = -100; + } else if a[0][0] != 0 || a[1][0] != 0 || word(&a[1][1..]) & 15 > 2 { + error = -105; + } + } + 0x100a => self.events = None, + 0x100b => { + if a.len() != 3 { + error = -100; + } else { + self.events = None; + } + } + 0x100c => { + if a.len() != 2 { + error = -100; + } + } + 0x2001 | 0x2005 | 0x2006 => { + self.reply(channel, 0, 0, &[]); + match cmd { + 0x2001 => self.reset_node(), + 0x2005 => self.power_down(true, false), + _ => { + self.power_down(false, false); + self.power_up(true); + } + } + return; + } + 0x2002..=0x2004 => error = -100, + 0x2007 | 0x2008 => error = -108, + 0x4000 => error = -104, + 0x4002 => { + self.reply(channel, 0, 0, &[]); + self.reboot(); + return; + } + _ => { + crate::dlog!(LogModule::L1, "L1 unsupported request {cmd:#x}"); + error = -101; + } + } + if !values.is_empty() { + field_count = values.len() as u8; + fields = ints(&values); + } + self.reply(channel, error, field_count, &fields); + } + /// The router replies are chunked and reassembled with the same three + /// commands; a malformed chunk drops the pending message. + fn router_chunk(&mut self, p: &[u8]) { + let channel = p[0] & 0x1f; + let slot = usize::from(channel); + if p.len() < 10 { + crate::dlog!(LogModule::L1, "L1 malformed router chunk"); + return; + } + let command = u16::from_be_bytes([p[5], p[6]]); + let payload = &p[10..]; + let fragment_length = u16::from_be_bytes([p[8], p[9]]) as usize; + match command { + 0x3001 => { + let fragments = p[7]; + if self.router[slot].take().is_some() { + crate::dlog!(LogModule::L1, "L1 replacing pending router request"); + } + if fragments == 0 + || fragments >= 128 + || fragment_length == 0 + || fragment_length > ROUTER_MAX_FRAGMENT + || payload.len() > fragment_length + { + crate::dlog!(LogModule::L1, "L1 unsupported router message start"); + return; + } + self.router[slot] = Some(RouterAssembly { + fragment_count: fragments, + fragments_left: fragments, + first_length: 0, + expected: fragment_length, + received: payload.len(), + data: payload.to_vec(), + }); + } + 0x3002 => { + let ok = self.router[slot].as_ref().is_some_and(|a| { + a.fragments_left != 0 + && a.received == a.expected + && fragment_length != 0 + && fragment_length <= ROUTER_MAX_FRAGMENT - a.data.len() + && payload.len() <= fragment_length + }); + let Some(a) = self.router[slot].as_mut().filter(|_| ok) else { + self.router[slot] = None; + crate::dlog!(LogModule::L1, "L1 invalid router next fragment"); + return; + }; + a.expected = fragment_length; + a.received = payload.len(); + a.data.extend_from_slice(payload); + } + _ => { + let ok = self.router[slot].as_ref().is_some_and(|a| { + a.received != a.expected + && payload.len() <= a.expected - a.received + && payload.len() <= ROUTER_MAX_FRAGMENT - a.data.len() + }); + let Some(a) = self.router[slot].as_mut().filter(|_| ok) else { + self.router[slot] = None; + crate::dlog!(LogModule::L1, "L1 router continuation without valid message"); + return; + }; + a.received += payload.len(); + a.data.extend_from_slice(payload); + } + } + self.router_complete(channel); + } + fn router_complete(&mut self, channel: u8) { + let slot = usize::from(channel); + let Some(a) = self.router[slot].as_mut() else { return }; + if a.received != a.expected { + return; + } + if a.first_length == 0 { + a.first_length = a.data.len(); + } + a.fragments_left -= 1; + if a.fragments_left != 0 { + return; + } + let a = self.router[slot].take().unwrap(); + // Only the single-fragment layout has a known descriptor mapping. + if a.fragment_count == 1 { + self.router_reply(channel, &a.data[..a.first_length], a.fragment_count); + } else { + crate::dlog!(LogModule::L1, "L1 multi-fragment router dispatch is unsupported"); + } + } + fn router_reply(&mut self, channel: u8, request: &[u8], fragment_count: u8) { + let Some(type_name) = router_type_name(self.config.type_code) else { + crate::dlog!(LogModule::L1, "L1 unsupported or malformed router request"); + return; + }; + let n = request.len(); + // Fixed header, an all-zero scalar block, then the string descriptor + // (kind 0x0a, 1) carrying the interpreter command. + let well_formed = n >= 37 + && usize::from(u16::from_be_bytes([request[0], request[1]])) == n + && request[2] == 2 + && request[3] == 0 + && request[5] == 0x80 | fragment_count + && request[6] == 0 + && request[7] == 0 + && word(&request[8..]) == 0x0fff_f003 + && request[16] == 2 + && request[20..28] == [0; 8] + && request[28] == 0x0a + && request[29] == 1 + && word(&request[32..]) == 52 + && usize::from(u16::from_be_bytes([request[30], request[31]])) == n - 36 + && request[n - 1] == 0; + if !well_formed { + crate::dlog!(LogModule::L1, "L1 unsupported or malformed router request"); + return; + } + // The captured form has one trailing space; the plain spelling is + // accepted too. Anything else is refused. + if !matches!(&request[36..], b"brick\0" | b"brick \0") { + crate::dlog!(LogModule::L1, "L1 router command interpreter operation refused"); + return; + } + let partition = match self.partition { + 0 => "none".to_string(), + 255 => "?".to_string(), + p => p.to_string(), + }; + let serial = self.config.brick_serial.as_deref().unwrap_or("(null)"); + let mut text = format!( + "rack: {:03}, slot: {:02}, partition: {partition}, type: {type_name} \ + [?MB flash], serial:{serial}, source: OTHER\n\n", + self.config.rack, self.config.bay + ) + .into_bytes(); + text.push(0); + let total = 36 + text.len(); + let node_id = (self.config.rack << 6) | self.config.bay; + let mut response = vec![0u8; 36]; + response[..2].copy_from_slice(&(total as u16).to_be_bytes()); + response[2] = 2; + response[4] = request[4]; + response[5] = 0x81; + response[6] = 1; + // The reply goes back to the requesting channel on this node. + response[8..12].copy_from_slice(&((node_id << 12) | (u32::from(channel) << 5) | 0x15).to_be_bytes()); + response[12..16].copy_from_slice(&((node_id << 12) | 3).to_be_bytes()); + response[16] = 2; + response[17] = request[17]; + response[18..20].copy_from_slice(&request[18..20]); + response[28] = 0x0a; + response[29] = 1; + response[30..32].copy_from_slice(&(text.len() as u16).to_be_bytes()); + response[32..36].copy_from_slice(&0x34u32.to_be_bytes()); + response.extend_from_slice(&text); + if total > ROUTER_MAX_FRAGMENT { + crate::dlog!(LogModule::L1, "L1 router brick response exceeds fragment limit"); + return; + } + for (i, part) in response.chunks(ROUTER_CHUNK_DATA).enumerate() { + let mut chunk = vec![0x20 | channel, 0, 0, 0, 0, 0x30, if i == 0 { 1 } else { 3 }]; + if i == 0 { + chunk.push(1); + chunk.extend_from_slice(&(total as u16).to_be_bytes()); + } else { + chunk.extend_from_slice(&[0, 0, 0]); + } + chunk.extend_from_slice(part); + self.emit(&chunk); + } + } + fn reason(&mut self) { + self.reasons[1] = self.reasons[0]; + self.reasons[0] = 3; + } + fn reset_node(&mut self) { + if self.powered { + self.reason(); + self.display = "Reset done".into(); + self.actions.push(L1Event::Reset); + } + } + fn power_down(&mut self, shutdown: bool, confirm: bool) { + if self.powered { + if confirm && !self.power_down_pending { + self.power_down_pending = true; + self.event( + 0x8000, + 0, + "Power Down: issue 'pwr d' again to immediately power down.", + ); + return; + } + self.power_down_pending = false; + self.reason(); + self.powered = false; + self.display = "Powered Down".into(); + self.actions.push(L1Event::Power(false)); + if shutdown { + self.actions.push(L1Event::Shutdown); + } + } + } + fn power_up(&mut self, restart: bool) { + self.reason(); + self.powered = true; + self.power_down_pending = false; + self.port &= 0x60; + self.console = true; + self.display = "Powered Up".into(); + self.actions.push(L1Event::Power(true)); + if restart { + self.actions.push(L1Event::Reset); + } + } + fn reboot(&mut self) { + self.power_down_pending = false; + self.console = false; + self.escaped = false; + self.command.clear(); + self.frame.clear(); + self.in_frame = false; + self.port = self.config.node_position << 5; + self.display = "L1 running".into(); + if self.events.is_some_and(|c| c < 4) { + self.events = None; + } + } + pub fn event(&mut self, code: u32, number: u32, text: &str) { + let Some(channel) = self.events else { + return; + }; + let mut p = vec![0x40 | (channel & 31)]; + p.extend_from_slice( + &((self.config.rack << 18) | (self.config.bay << 12) | 4).to_be_bytes(), + ); + p.extend_from_slice(&code.to_be_bytes()); + p.push(2); + p.extend(ints(&[number])); + text_field( + &mut p, + format!( + "\r\n{:03}c{:02} ATTN: {text}\r\n", + self.config.rack, self.config.bay + ) + .as_bytes(), + ); + if p.len() <= 127 { + self.emit(&p); + } + } + fn run_line(&mut self, line: &str, interactive: bool) -> (i32, String) { + let words = line.split_whitespace().collect::>(); + let local = if words.first() == Some(&"*") { + &words[1..] + } else { + &words[..] + }; + match local { + ["pwr" | "power", "d" | "down"] => { + self.power_down(true, interactive); + (0, String::new()) + } + ["pwr" | "power", "u" | "up"] => { + if !self.powered { + self.power_up(true); + } + (0, String::new()) + } + ["reset"] => { + self.reset_node(); + (0, String::new()) + } + ["softreset"] => { + self.actions.push(L1Event::SoftReset); + self.display = "Sft rst done".into(); + (0, String::new()) + } + ["nmi"] => { + self.actions.push(L1Event::Nmi); + self.display = "NMI done".into(); + (0, String::new()) + } + ["reboot_l1"] => { + self.reboot(); + (0, String::new()) + } + ["l1"] => { + self.escaped = false; + (0, String::new()) + } + ["version"] => { + if let Some(r) = self.config.revision { + ( + 0, + format!("L1 Controller: Revision {}.{}.{}\r\n", r[0], r[1], r[2]), + ) + } else { + ( + -104, + "L1 firmware revision metadata not configured\r\n".into(), + ) + } + } + ["debug"] if words.len() == 1 => ( + 0, + format!("debug switches set to 0x{:04x}\n\r\n", self.debug), + ), + ["debug", v] if words.len() == 2 => { + let n = if let Some(hex) = v.strip_prefix("0x") { + u32::from_str_radix(hex, 16) + } else { + v.parse() + }; + match n { + Ok(v) => { + self.debug = v & 0xffff; + ( + 0, + format!("debug switches set to 0x{:04x}\n\r\n", self.debug), + ) + } + Err(_) => (-1200, "ERROR: command not found.\n\r\n".into()), + } + } + _ => { + crate::dlog!(LogModule::L1, "L1 unsupported operator command {line:?}"); + (-1, "ERROR: command not found.\r\n".into()) + } + } + } + fn prompt(&mut self) { + self.output.extend_from_slice( + format!("{:03}c{:02}-L1>", self.config.rack, self.config.bay).as_bytes(), + ); + } + pub fn console_input(&mut self, byte: u8) { + if self.console { + if byte == 0x14 { + self.console = false; + self.escaped = true; + self.output + .extend_from_slice(b"escaping to L1 system controller\r\n"); + self.prompt(); + } else if self.powered { + self.emit(&[self.port & 31, byte]); + } else { + crate::dlog!(LogModule::L1, "L1 console input while powered off"); + } + } else if byte == 4 { + self.command.clear(); + self.console = true; + } else if byte == b'\r' || byte == b'\n' { + let line = String::from_utf8_lossy(&self.command).into_owned(); + self.command.clear(); + let (_, output) = self.run_line(&line, true); + self.output.extend_from_slice(output.as_bytes()); + if self.escaped { + self.console = true; + } else { + self.prompt(); + } + } else if byte == 8 || byte == 127 { + self.command.pop(); + } else if self.command.len() < 255 { + self.command.push(byte); + } + } +} diff --git a/src/dev/l1/tests.rs b/src/dev/l1/tests.rs new file mode 100644 index 000000000..f9e4b1a77 --- /dev/null +++ b/src/dev/l1/tests.rs @@ -0,0 +1,602 @@ +// Cases follow tests/qtest/sn1-l1-test.c in jamesbraid/qemu-sgi-origami: +// uart, frames, identity, eeprom, errors and reset, plus the port-speed and +// router brick-reply cases (test_port_speed, test_irouter_*). + +use super::*; + +fn send(l1: &L1, bytes: &[u8]) { + for &b in bytes { + assert_eq!(l1.write64(0, b as u64), BUS_OK); + } +} +fn drain(l1: &L1) -> Vec { + let mut out = Vec::new(); + while l1.read64(0x28).data & 1 != 0 { + out.push(l1.read64(0).data as u8); + } + out +} +fn request(l1: &L1, command: u16, args: &[u8], count: u8) -> Vec { + let mut p = vec![ + 0x10, + 0x0f, + 0xff, + 0xf0, + 6, + (command >> 8) as u8, + command as u8, + count, + ]; + p.extend_from_slice(args); + send(l1, &encode(&p)); + drain(l1) +} +fn integer_reply(values: &[u32]) -> Vec { + let mut p = vec![0x30, 0, 0, 0, 0, values.len() as u8]; + p.extend(model::ints(values)); + encode(&p) +} + +/// The PROM accesses the L1 UART through byte-sized register operations. +#[test] +fn the_l1_uart_frames_bytes_and_prioritizes_interrupts() { + let l1 = L1::new(Default::default()).unwrap(); + assert_eq!(l1.read64(1).status, BUS_ERR); + assert_eq!(l1.read8(0).status, BUS_ERR); + assert_eq!(l1.read32(0).status, BUS_ERR); + assert_eq!(l1.read64(0x28).data, 0x60); + send(&l1, &[0x7e, 0x10, 0x0f, 0xff, 0xf0, 6, 0, 0x0e, 0, 0xf9, 0x72, 0x7e]); + assert_eq!(drain(&l1), [0x7e, 0x30, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0x30, 0x98, 0x7e]); + let (irq, line) = crate::irq::LevelProbe::line(); + l1.set_irq(line); + l1.write64(8, 3); + assert!(irq.level()); + assert_eq!(l1.read64(0x10).data, 2); + assert!(!irq.level()); + l1.write64(0x18, 0x80); + l1.write64(0, 42); + l1.write64(8, 7); + assert_eq!(l1.read64(0).data, 42); + assert_eq!(l1.read64(8).data, 7); + l1.write64(0x18, 3); + send(&l1, &encode(&[0x10, 0x0f, 0xff, 0xf0, 6, 0, 0x0e, 0])); + assert_eq!(l1.read64(0x10).data, 4); + drain(&l1); + assert_eq!(l1.read64(0x10).data, 2); + assert!(!irq.level()); +} + +/// The guest reads controller identity and checks the EEPROM record checksum. +#[test] +fn the_l1_returns_identity_and_validated_eeprom_records() { + let l1 = L1::new(Default::default()).unwrap(); + assert_eq!(request(&l1, 3, &[], 0), integer_reply(&[65, 65])); + let board = l1.record(1, 2).unwrap(); + assert_eq!(board.iter().fold(0u8, |s, x| s.wrapping_add(*x)), 0); + for text in [b"IP34".as_slice(), b"EMUN00", b"080069123456"] { + assert!(board.windows(text.len()).any(|x| x == text)); + } + let mut expected = vec![0x30, 0, 0, 0, 0, 2]; + expected.extend(model::ints(&[0])); + expected.push(0x80 | board.len() as u8); + expected.extend(board); + assert_eq!(request(&l1, 6, &model::ints(&[1, 2, 0, 116]), 4), encode(&expected)); + for (args, count, error) in [([2, 5, 0, 1], 4, 152), ([1, 5, 0, 1], 4, 151), ([1, 2, 0, 1], 3, 156)] { + assert_eq!(request(&l1, 6, &model::ints(&args), count), encode(&[0x30, 255, 255, 255, error, 0])); + } + l1.power_on(); + assert_eq!(request(&l1, 5, &[], 0), integer_reply(&[1, 48, 1])); +} + +/// A successful reply carrying one text field. +fn text_reply(text: &str) -> Vec { + let mut p = vec![0x30, 0, 0, 0, 0, 1]; + p.push(1); + p.extend_from_slice(text.as_bytes()); + p.push(0); + encode(&p) +} + +/// With no configuration the controller reports the labelled model defaults: +/// firmware 1.48.1 and the synthetic brick serial. +#[test] +fn the_l1_reports_the_default_revision_and_brick_serial() { + let l1 = L1::new(Default::default()).unwrap(); + l1.power_on(); + assert_eq!(request(&l1, 5, &[], 0), integer_reply(&[1, 48, 1])); + assert_eq!(request(&l1, 4, &[], 0), text_reply("EMUN00")); +} + +/// Both identity values can be replaced. +#[test] +fn the_l1_revision_and_brick_serial_can_be_overridden() { + let l1 = L1::new(L1Config { + revision: Some(parse_revision("2.13.40").unwrap()), + brick_serial: Some("EMUN07".into()), + ..Default::default() + }) + .unwrap(); + l1.power_on(); + assert_eq!(request(&l1, 5, &[], 0), integer_reply(&[2, 13, 40])); + assert_eq!(request(&l1, 4, &[], 0), text_reply("EMUN07")); + assert_eq!( + L1::new(L1Config { brick_serial: Some("x".repeat(38)), ..Default::default() }).err().as_deref(), + Some("L1 brick serial must fit a 37-byte text field") + ); +} + +/// The revision override is exactly three decimal numbers. +#[test] +fn the_l1_revision_text_is_three_decimal_parts() { + assert_eq!(parse_revision("1.48.1"), Ok([1, 48, 1])); + for bad in ["", "1", "1.48", "1.48.1.0", "1..1", "a.b.c", "1.48.-1", "1.48.+1", " 1.48.1", "1.48.99999999999"] { + assert!(parse_revision(bad).is_err(), "{bad:?}"); + } +} + +/// Request 0x0008 is the configured station address as twelve uppercase +/// hexadecimal digits in address order, the text the PROM turns back into the +/// Ethernet address. +#[test] +fn the_l1_system_serial_is_the_station_address_in_hex() { + let l1 = L1::new(Default::default()).unwrap(); + l1.power_on(); + assert_eq!(request(&l1, 8, &[], 0), text_reply("080069123456")); + let l1 = L1::new(L1Config { mac: [0x00, 0xA0, 0xc9, 0x1f, 0x02, 0xbe], ..Default::default() }).unwrap(); + l1.power_on(); + assert_eq!(request(&l1, 8, &[], 0), text_reply("00A0C91F02BE")); +} + +/// The L1 parser recovers after malformed frames and preserves escaped console text. +#[test] +fn the_l1_resynchronizes_after_bad_frames_and_preserves_escaped_text() { + let l1 = L1::new(Default::default()).unwrap(); + send(&l1, &[0x7e, 1, 2, 3, 0x7e]); + send(&l1, &vec![0x41; 259]); + send(&l1, &[0x7e]); + assert!(drain(&l1).is_empty()); + assert_eq!(request(&l1, 0x2000, &[], 0), encode(&[0x30, 255, 255, 255, 155, 0])); + send(&l1, &encode(&[0x10, 0x4f, 0xff, 0xf0, 6, 0, 3, 0])); + assert_eq!(drain(&l1), encode(&[0x30, 255, 255, 255, 255, 0, 0, 0, 8])); + let mut text = vec![1]; + text.extend_from_slice(b"hi~}\r\0"); + assert_eq!(request(&l1, 0x1004, &text, 1), integer_reply(&[])); + assert_eq!(l1.display_line(), "hi~}?"); + text.truncate(1); + text.extend_from_slice(b"abcdefghijkl\x01\x7f"); + text.extend(std::iter::repeat_n(b'z', 70)); + text.push(0); + assert_eq!(request(&l1, 0x1004, &text, 1), integer_reply(&[])); + assert_eq!(l1.display_line(), format!("abcdefghijkl??{}", "z".repeat(58))); +} + +/// The PROM may reset L1 during boot; controller settings and channel configuration must remain available. +#[test] +fn l1_reset_clears_uart_state_but_preserves_settings_and_channels() { + for channel in [3, 23] { + let l1 = L1::new(Default::default()).unwrap(); + let (irq, line) = crate::irq::LevelProbe::line(); + l1.set_irq(line); + for (set, value) in [(0x1001, 0x1234), (0x1006, 42), (0x1008, 2), (0x1007, channel)] { + request(&l1, set, &model::ints(&[value]), 1); + } + request(&l1, 0x2001, &[], 0); + let reasons = request(&l1, 0xf, &[], 0); + assert_eq!(reasons, integer_reply(&[3, 0])); + l1.write64(8, 1); + l1.send_event(0x8000, 7, "before reset"); + assert!(irq.level()); + let stale = encode(&[0x10, 0x0f, 0xff, 0xf0, 6, 0, 9, 0]); + send(&l1, &stale[..4]); + l1.write64(0x18, 0x80); + l1.write64(0, 42); + l1.write64(8, 7); + l1.power_on(); + assert!(!irq.level()); + assert_eq!(l1.read64(0x18).data, 0); + assert_eq!(l1.read64(8).data, 0); + assert!(drain(&l1).is_empty()); + l1.write64(0x18, 0x80); + // Ns16550 resets its divisor latch to 0x000c for 9600 baud. + assert_eq!(l1.read64(0).data, 0x0c); + assert_eq!(l1.read64(8).data, 0); + l1.write64(0x18, 0); + send(&l1, &stale[4..]); + assert!(drain(&l1).is_empty()); + for (get, value) in [(1, 0x1234), (9, 42), (0xb, 2)] { + assert_eq!(request(&l1, get, &[], 0), integer_reply(&[value])); + } + assert_eq!(request(&l1, 0xf, &[], 0), reasons); + l1.write64(8, 1); + l1.send_event(0x8000, 7, "after reset"); + assert_eq!(irq.level(), channel == 23); + let emitted = drain(&l1); + assert_eq!(emitted.is_empty(), channel == 3); + assert!(!irq.level()); + } +} + +/// A local console power-down request requires a second confirmation command. +#[test] +fn interactive_power_down_waits_for_a_second_command() { + let l1 = Arc::new(L1::new(Default::default()).unwrap()); + let actions = Arc::new(Mutex::new(Vec::new())); + let observed = actions.clone(); + let weak = Arc::downgrade(&l1); + l1.set_event_handler(Arc::new(move |event| { + assert!(!weak.upgrade().unwrap().node_powered()); + observed.lock().push(event); + })); + request(&l1, 0x1007, &model::ints(&[23]), 1); + l1.receive_console(&[0x14]); + l1.receive_console(b"pwr d\r"); + assert!(l1.node_powered()); + assert!(actions.lock().is_empty()); + let mut warning = vec![0x57, 0, 4, 0x10, 4, 0, 0, 0x80, 0, 2]; + warning.extend(model::ints(&[0])); + warning.push(1); + warning.extend_from_slice(b"\r\n001c01 ATTN: Power Down: issue 'pwr d' again to immediately power down.\r\n\0"); + assert_eq!(drain(&l1), encode(&warning)); + l1.power_on(); + l1.receive_console(&[0x14]); + l1.receive_console(b"pwr d\r"); + assert!(l1.node_powered(), "reset clears the first confirmation"); + drain(&l1); + l1.receive_console(&[0x14]); + l1.receive_console(b"power down\r"); + assert!(!l1.node_powered()); + assert_eq!(*actions.lock(), [L1Event::Power(false), L1Event::Shutdown]); +} + +/// Protocol power-down actions must finish before the L1 request reply. +#[test] +fn protocol_and_task_three_power_down_commands_are_immediate() { + let protocol = vec![0x10, 0x0f, 0xff, 0xf0, 6, 0x20, 5, 0]; + let mut task_three = vec![0x10, 0x0f, 0xff, 0xf0, 3, 0, 0, 1, 1]; + task_three.extend_from_slice(b"pwr d\0"); + for (case, packet) in [ + ("protocol command", protocol), + ("task three console", task_three), + ] { + let l1 = L1::new(Default::default()).unwrap(); + let actions = Arc::new(Mutex::new(Vec::new())); + let observed = actions.clone(); + l1.set_event_handler(Arc::new(move |event| observed.lock().push(event))); + send(&l1, &encode(&packet)); + assert_eq!(drain(&l1), integer_reply(&[]), "{case}"); + assert!(!l1.node_powered(), "{case}"); + assert_eq!( + *actions.lock(), + [L1Event::Power(false), L1Event::Shutdown], + "{case}" + ); + } +} + +/// A CPU may read the UART in its interrupt callback while the worker updates the line. +#[test] +fn l1_worker_irq_delivery_stays_ordered_across_reentrant_cpu_reads() { + use std::sync::mpsc; + struct BlockingIrq { + entered: mpsc::SyncSender<()>, + resume: std::sync::Mutex>, + blocked: AtomicBool, + levels: Mutex>, + device: std::sync::Weak, + } + impl crate::irq::InterruptController for BlockingIrq { + fn set_line(&self, _: u32, level: bool) { + if level && !self.blocked.swap(true, Ordering::AcqRel) { + self.entered.send(()).unwrap(); + self.resume.lock().unwrap().recv().unwrap(); + // A machine interrupt handler may inspect the UART immediately. + assert_eq!(self.device.upgrade().unwrap().read64(0x28).data, 0x60); + } + self.levels.lock().push(level); + } + } + let l1 = Arc::new(L1::new(Default::default()).unwrap()); + let backend = Arc::new(crate::dev::serial_host::CiSerialBackend::new()); + let (entered_tx, entered_rx) = mpsc::sync_channel(1); + let (resume_tx, resume_rx) = mpsc::sync_channel(1); + let irq = Arc::new(BlockingIrq { + entered: entered_tx, + resume: std::sync::Mutex::new(resume_rx), + blocked: AtomicBool::new(false), + levels: Mutex::new(Vec::new()), + device: Arc::downgrade(&l1), + }); + l1.set_irq(IrqLine::new(irq.clone(), 0)); + l1.set_serial_backend(backend.clone()); + l1.write64(8, 1); + l1.start(); + backend.push_host(b"x"); + let entered = entered_rx.recv_timeout(std::time::Duration::from_secs(2)); + if entered.is_ok() { + l1.write64(0x10, 2); + } + resume_tx.send(()).unwrap(); + l1.stop(); + entered.unwrap(); + assert_eq!(*irq.levels.lock(), [false, true, false]); +} + +/// Concurrent worker output and platform actions must preserve their order. +#[test] +fn concurrent_l1_outputs_and_actions_are_delivered_in_order() { + use std::sync::mpsc; + struct BlockingBackend { + entered: mpsc::SyncSender<()>, + resume: std::sync::Mutex>, + output: Mutex>, + } + impl SerialBackend for BlockingBackend { + fn send_byte(&self, byte: u8) { + if byte == b'A' { + self.entered.send(()).unwrap(); + self.resume.lock().unwrap().recv().unwrap(); + } + self.output.lock().push(byte); + } + fn recv_byte(&self) -> std::io::Result { + Err(std::io::ErrorKind::WouldBlock.into()) + } + } + let l1 = Arc::new(L1::new(Default::default()).unwrap()); + let (entered_tx, entered_rx) = mpsc::sync_channel(1); + let (resume_tx, resume_rx) = mpsc::sync_channel(1); + let backend = Arc::new(BlockingBackend { + entered: entered_tx, + resume: std::sync::Mutex::new(resume_rx), + output: Mutex::new(Vec::new()), + }); + l1.set_serial_backend(backend.clone()); + let actions = Arc::new(Mutex::new(Vec::new())); + let captured = actions.clone(); + let weak = Arc::downgrade(&l1); + l1.set_event_handler(Arc::new(move |action| { + // Reentrant register reads must not recursively dispatch notifications. + assert_eq!(weak.upgrade().unwrap().read64(0x28).data, 0x60); + captured.lock().push(action); + })); + { + let mut s = l1.state.lock(); + s.output.extend(b"AB"); + s.actions.push(L1Event::Reset); + } + let worker_device = l1.clone(); + let worker = std::thread::spawn(move || worker_device.flush()); + let entered = entered_rx.recv_timeout(std::time::Duration::from_secs(2)); + if entered.is_ok() { + let mut s = l1.state.lock(); + s.output.extend(b"CD"); + s.actions.push(L1Event::Nmi); + drop(s); + l1.flush(); + } + resume_tx.send(()).unwrap(); + worker.join().unwrap(); + entered.unwrap(); + assert_eq!(*backend.output.lock(), b"ABCD"); + assert_eq!(*actions.lock(), [L1Event::Reset, L1Event::Nmi]); +} + +/// Request 0xa returns two integers; the first is a PROM display selector. +/// Ported from test_port_speed in tests/qtest/sn1-l1-test.c. +#[test] +fn the_port_speed_reply_carries_the_configured_first_integer() { + let l1 = L1::new(Default::default()).unwrap(); + assert_eq!(request(&l1, 0xa, &[], 0), integer_reply(&[0, 0])); + let l1 = L1::new(L1Config { port_speed_first: 1, ..Default::default() }).unwrap(); + assert_eq!(request(&l1, 0xa, &[], 0), integer_reply(&[1, 0])); + l1.power_on(); + drain(&l1); + assert_eq!(request(&l1, 0xa, &[], 0), integer_reply(&[1, 0])); + assert!(L1::new(L1Config { port_speed_first: 2, ..Default::default() }).is_err()); +} + +const BRICK: [u8; 53] = [ + 0x10, 0, 0, 0, 0, 0x30, 1, 1, 0, 0x2b, 0, 0x2b, 2, 0, 0, 0x81, 0, 0, 0x0f, 0xff, 0xf0, 3, 0x2f, + 0xff, 0xf0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0x0a, 1, 0, 7, 0, 0, 0, 0x34, b'b', b'r', + b'i', b'c', b'k', b' ', 0, +]; + +fn router_l1(type_code: u32, rack: u32, bay: u32, partition: u32, serial: Option<&str>) -> L1 { + let l1 = L1::new(L1Config { + router: true, + type_code, + rack, + bay, + brick_serial: serial.map(str::to_string), + ..Default::default() + }) + .unwrap(); + if partition != 0 { + request(&l1, 0x1006, &model::ints(&[partition]), 1); + } + l1 +} + +fn chunk(channel: u8, command: u16, count: u8, length: u16, data: &[u8]) -> Vec { + let mut p = vec![channel, 0, 0, 0, 0, (command >> 8) as u8, command as u8, count]; + p.extend_from_slice(&length.to_be_bytes()); + p.extend_from_slice(data); + p +} + +fn send_packet(l1: &L1, packet: &[u8]) { + send(l1, &encode(packet)); +} + +/// Collect one reply message from its chunks: the channel, and the inner bytes. +fn router_receive(l1: &L1) -> (u8, Vec) { + let wire = drain(l1); + let mut frames = wire.split(|b| *b == 0x7e).filter(|f| !f.is_empty()); + let mut decode = || -> Vec { + let raw = frames.next().expect("a reply frame"); + let mut out = Vec::new(); + let mut it = raw.iter(); + while let Some(&b) = it.next() { + out.push(if b == 0x7d { it.next().unwrap() ^ 0x20 } else { b }); + } + out.truncate(out.len() - 2); + out + }; + let first = decode(); + assert_eq!(first[1..5], [0; 4]); + assert_eq!(first[5..8], [0x30, 1, 1]); + let channel = first[0]; + let mut inner = first[10..].to_vec(); + while inner.len() < usize::from(u16::from_be_bytes([inner[0], inner[1]])) { + let next = decode(); + assert_eq!(next[0], channel); + assert_eq!(next[5..10], [0x30, 3, 0, 0, 0]); + inner.extend_from_slice(&next[10..]); + } + assert!(frames.next().is_none()); + (channel, inner) +} + +fn expect_brick(l1: &L1, channel: u8, destination: u32, source: u32, text: &str) { + let (got, inner) = router_receive(l1); + assert_eq!(got, channel); + assert_eq!(usize::from(u16::from_be_bytes([inner[0], inner[1]])), inner.len()); + assert_eq!(u32::from_be_bytes(inner[8..12].try_into().unwrap()), destination); + assert_eq!(u32::from_be_bytes(inner[12..16].try_into().unwrap()), source); + assert_eq!(&inner[36..], [text.as_bytes(), &[0]].concat()); +} + +const DEFAULT_TEXT: &str = "rack: 001, slot: 04, partition: none, type: Chimera Server \ + [?MB flash], serial:(null), source: OTHER\n\n"; + +/// Ported from test_irouter_brick_exact in tests/qtest/sn1-l1-test.c. +#[test] +fn the_router_brick_reply_matches_the_captured_layout() { + let l1 = router_l1(0x37, 1, 4, 0, None); + send_packet(&l1, &BRICK); + let mut inner = vec![ + 0, 0x8b, 2, 0, 0, 0x81, 1, 0, 0, 0x04, 0x42, 0x15, 0, 0x04, 0x40, 3, 2, 0, 0, 0, 0, 0, 0, + 0, 0, 0, 0, 0, 0x0a, 1, 0, 0x67, 0, 0, 0, 0x34, + ]; + inner.extend_from_slice(DEFAULT_TEXT.as_bytes()); + inner.push(0); + assert_eq!(inner.len(), 139); + let mut first = vec![0x30, 0, 0, 0, 0, 0x30, 1, 1, 0, 139]; + first.extend_from_slice(&inner[..113]); + let mut rest = vec![0x30, 0, 0, 0, 0, 0x30, 3, 0, 0, 0]; + rest.extend_from_slice(&inner[113..]); + let mut expected = encode(&first); + expected.extend(encode(&rest)); + assert_eq!(drain(&l1), expected); +} + +/// Subchannels are answered in place, and a request split across chunks +/// completes on the last continuation. Ported from +/// test_irouter_fragment_channel_transport (the single-transport part). +#[test] +fn the_router_reassembles_chunks_and_answers_on_the_request_subchannel() { + let l1 = router_l1(0x37, 1, 4, 0, None); + let rest = &BRICK[10..]; + send_packet(&l1, &chunk(0x10, 0x3001, 1, 43, &rest[..12])); + assert!(drain(&l1).is_empty()); + let mut other = BRICK; + other[0] = 0x11; + send_packet(&l1, &other); + expect_brick(&l1, 0x31, 0x0004_4235, 0x0004_4003, DEFAULT_TEXT); + send_packet(&l1, &chunk(0x10, 0x3003, 0, 0, &rest[12..])); + expect_brick(&l1, 0x30, 0x0004_4215, 0x0004_4003, DEFAULT_TEXT); +} + +/// Ported from test_irouter_reset_and_negative. +#[test] +fn a_reset_drops_a_pending_router_message_and_unsupported_types_stay_silent() { + let l1 = router_l1(0x37, 1, 4, 0, None); + let rest = &BRICK[10..]; + send_packet(&l1, &chunk(0x10, 0x3001, 1, 43, &rest[..12])); + l1.power_on(); + drain(&l1); + send_packet(&l1, &chunk(0x10, 0x3003, 0, 0, &rest[12..])); + assert!(drain(&l1).is_empty()); + send_packet(&l1, &BRICK); + expect_brick(&l1, 0x30, 0x0004_4215, 0x0004_4003, DEFAULT_TEXT); + + let off = L1::new(Default::default()).unwrap(); + send_packet(&off, &BRICK); + assert_eq!(drain(&off), encode(&[0x30, 255, 255, 255, 255, 0, 0, 0, 8])); + + let l1 = router_l1(0x37, 1, 4, 0, None); + send_packet(&l1, &BRICK[..7]); + assert!(drain(&l1).is_empty()); + send_packet(&l1, &chunk(0x10, 0x3001, 1, 43, &rest[..12])); + send_packet(&l1, &BRICK); + expect_brick(&l1, 0x30, 0x0004_4215, 0x0004_4003, DEFAULT_TEXT); + + let l1 = router_l1(0x34, 1, 4, 0, None); + send_packet(&l1, &BRICK); + send_packet(&l1, &chunk(0x10, 0x3003, 0, 0, rest)); + assert!(drain(&l1).is_empty()); + + let l1 = router_l1(0x37, 1, 4, 0, None); + let mut reset = BRICK; + reset[46..52].copy_from_slice(b"reset\0"); + send_packet(&l1, &reset); + assert!(drain(&l1).is_empty()); +} + +/// A new start replaces a pending message. Ported from +/// test_irouter_replace_pending. +#[test] +fn a_new_router_start_replaces_the_pending_message() { + let l1 = router_l1(0x37, 1, 4, 0, None); + let mut old = BRICK; + old[14] = 0x12; + old[27] = 0x34; + send_packet(&l1, &chunk(0x10, 0x3001, 1, 43, &old[10..22])); + send_packet(&l1, &BRICK); + expect_brick(&l1, 0x30, 0x0004_4215, 0x0004_4003, DEFAULT_TEXT); + send_packet(&l1, &chunk(0x10, 0x3003, 0, 0, &old[22..])); + assert!(drain(&l1).is_empty()); +} + +/// Messages of more than one fragment have no known descriptor layout. +/// Ported from test_irouter_multifragment_refusal. +#[test] +fn a_multi_fragment_router_message_is_refused_and_later_requests_work() { + let l1 = router_l1(0x37, 1, 4, 0, None); + let mut request = BRICK; + request[15] = 0x82; + send_packet(&l1, &chunk(0x10, 0x3001, 2, 43, &request[10..])); + assert!(drain(&l1).is_empty()); + send_packet(&l1, &chunk(0x10, 0x3002, 0, 4, &[0xde, 0xad, 0xbe, 0xef])); + assert!(drain(&l1).is_empty()); + request[0] = 0x11; + request[15] = 0x81; + send_packet(&l1, &request); + expect_brick(&l1, 0x31, 0x0004_4235, 0x0004_4003, DEFAULT_TEXT); +} + +/// Ported from test_irouter_identity_overrides; the type 0x39 case is the +/// Tezro identity. +#[test] +fn the_router_reply_reports_rack_slot_partition_type_and_serial() { + let l1 = router_l1(0x39, 2, 5, 3, Some("SERIAL-X")); + send_packet(&l1, &BRICK); + expect_brick( + &l1, + 0x30, + 0x0008_5215, + 0x0008_5003, + "rack: 002, slot: 05, partition: 3, type: Chimera WS [?MB flash], \ + serial:SERIAL-X, source: OTHER\n\n", + ); + let l1 = router_l1(0x51, 3, 2, 255, Some("")); + send_packet(&l1, &BRICK); + expect_brick( + &l1, + 0x30, + 0x000c_2215, + 0x000c_2003, + "rack: 003, slot: 02, partition: ?, type: Rackmount WS [?MB flash], \ + serial:, source: OTHER\n\n", + ); +} diff --git a/src/dev/mod.rs b/src/dev/mod.rs index fdd54e4b1..d5e6d794b 100644 --- a/src/dev/mod.rs +++ b/src/dev/mod.rs @@ -12,12 +12,17 @@ pub mod mace; pub mod mace_audio; pub mod mace_isa; pub mod crime; +pub mod deadline_timers; pub mod ds1687; +pub mod ds1744; pub mod heart; pub mod heart_mem; pub mod ioc3; +pub mod ioc_serial_dma; pub mod ioc3_eth; +pub mod ioc3_kbc; pub mod ioc3_port; +pub mod ioc4; pub mod isp1040; pub mod mace_id; pub mod mace_pci; @@ -56,7 +61,27 @@ pub mod seeq8003; pub mod testdev; pub mod timer; pub mod ultra64; +pub mod usb_hid; pub mod vino; pub mod virtio; pub mod wd33c93a; pub mod z85c30; +pub mod bedrock_flash; +pub mod bedrock_io; +pub mod bedrock_ni; +pub mod bedrock_xb; +pub mod bedrock_lb; +pub mod bedrock_md; +pub mod bedrock_mem; +pub mod bedrock_pi; +pub mod bcm5701; +pub mod dp83843; +pub mod ics1890; +pub mod isp12160; +pub mod l1; +pub mod ohci; +pub mod pic; +pub mod pxbow; +pub mod prom_image; +pub mod timekeeper; +pub mod xbridge; diff --git a/src/dev/ns16550.rs b/src/dev/ns16550.rs index 82a756d04..c442225be 100644 --- a/src/dev/ns16550.rs +++ b/src/dev/ns16550.rs @@ -38,9 +38,13 @@ //! //! * Transmit is instant. A write of the transmit register hands the byte to //! the host backend (or, in loopback, back to the receiver) before the -//! write returns, so the holding register and the transmitter always read -//! empty and QEMU's transmit timer, retry and hold-off machinery is not -//! here. Nothing is paced to the baud rate; the divisor and line control +//! write returns, so the holding register and the transmitter read empty. +//! The one exception is a backend that cannot take the byte (see +//! [`SerialBackend::try_send_byte`]): the byte is held, the transmitter +//! reads busy, and [`Ns16550::poll_tx`] retries it, gives up after +//! [`XMIT_RETRY_TIMEOUT`] and then treats the backend as disconnected until +//! it takes a byte again, as QEMU's transmit retry does. QEMU's transmit +//! timer is not here. Nothing is paced to the baud rate; the divisor and line control //! only set how long the receive timeout is. //! * The host end is a [`SerialBackend`] polled by [`Ns16550::pump`], where //! QEMU's chardev calls back. The receive timeout is a host-time deadline @@ -117,6 +121,8 @@ pub mod com_reg { /// LSR: a received character is waiting. pub const LSR_DR: u8 = 0x01; pub const LSR_OE: u8 = 0x02; + pub const LSR_PE: u8 = 0x04; + pub const LSR_FE: u8 = 0x08; pub const LSR_BI: u8 = 0x10; /// LSR: transmit holding register empty. pub const LSR_THRE: u8 = 0x20; @@ -180,8 +186,21 @@ pub struct Ns16550 { /// The last [`OUT_KEEP`] bytes sent, and how many were sent in all. out: VecDeque, sent: u64, + /// Bytes not yet sent, oldest first: the first is in the shift register + /// and the rest wait in the holding register or the transmit FIFO. Only a + /// backend that refuses a byte leaves any. The transmitter is busy while + /// there are any. + held: VecDeque, + held_since: Option, + /// A held byte was given up on: the backend counts as disconnected, and + /// bytes complete without waiting, until it takes one. + backend_dead: bool, } +/// How long a refused byte is retried before the backend counts as +/// disconnected. +pub const XMIT_RETRY_TIMEOUT: Duration = Duration::from_secs(2); + impl Ns16550 { /// A port in its reset state. `baudbase` is the reference clock over 16 /// (115200 for a 1.8432 MHz crystal): the divisor divides it. @@ -208,6 +227,9 @@ impl Ns16550 { typed: VecDeque::new(), out: VecDeque::new(), sent: 0, + held: VecDeque::new(), + held_since: None, + backend_dead: false, }; s.reset(); s @@ -233,6 +255,8 @@ impl Ns16550 { self.timeout_deadline = None; self.recv_fifo.clear(); self.thr_ipending = false; + self.held.clear(); + self.held_since = None; } pub fn set_backend(&mut self, backend: Arc) { @@ -261,6 +285,12 @@ impl Ns16550 { self.lcr & LCR_DLAB != 0 } + /// How long a character takes on the line at the programmed rate and + /// frame. + pub fn char_time(&self) -> Duration { + self.char_transmit_time + } + pub fn loopback(&self) -> bool { self.mcr & MCR_LOOP != 0 } @@ -271,6 +301,12 @@ impl Ns16550 { + if self.fcr & FCR_FE != 0 { self.recv_fifo.len() } else { (self.lsr & LSR_DR != 0) as usize } } + /// A byte is waiting to go out: only a backend that refused one leaves + /// any. + pub fn tx_pending(&self) -> bool { + !self.held.is_empty() + } + /// Type characters at the port, as if the host had sent them. They wait /// until the port will take them, as the backend's do. pub fn type_in(&mut self, bytes: &[u8]) { @@ -504,18 +540,89 @@ impl Ns16550 { if self.loopback() { self.receive(&[val]); } else { - if self.out.len() == OUT_KEEP { - self.out.pop_front(); + if self.fcr & FCR_FE != 0 { + // A full transmit FIFO drops its oldest byte, not the one + // in the shift register. + if self.held.len() > FIFO_LENGTH { + self.held.remove(1); + } + self.held.push_back(val); + } else if self.held.len() >= 2 { + // The holding register is overwritten. + *self.held.back_mut().expect("two held") = val; + } else { + self.held.push_back(val); } - self.out.push_back(val); - self.sent += 1; - self.backend.send_byte(val); + self.drain_tx(Instant::now()); + } + self.sync_tx_flags(); + } + + /// The holding register is empty once the shift register has taken its + /// byte, and the transmitter once nothing is held at all. + fn sync_tx_flags(&mut self) { + if self.held.len() <= 1 { + self.lsr |= LSR_THRE; + self.thr_ipending = true; + } + if self.held.is_empty() { + self.lsr |= LSR_TEMT; } - self.lsr |= LSR_THRE | LSR_TEMT; - self.thr_ipending = true; self.update_irq(); } + /// Hand held bytes to the backend in order. One it refuses stays, and + /// the transmitter stays busy, until it takes it or the retry time is + /// out. Past that the byte is dropped and the backend counts as + /// disconnected: further bytes are tried once and dropped if refused. + fn drain_tx(&mut self, now: Instant) { + while let Some(&b) = self.held.front() { + if self.backend.try_send_byte(b) { + if self.out.len() == OUT_KEEP { + self.out.pop_front(); + } + self.out.push_back(b); + self.sent += 1; + self.backend_dead = false; + } else if !self.backend_dead { + let since = *self.held_since.get_or_insert(now); + if now.saturating_duration_since(since) < XMIT_RETRY_TIMEOUT { + return; + } + self.backend_dead = true; + } + self.held.pop_front(); + self.held_since = None; + } + } + + /// Retry held bytes at `now`. Returns whether any went, which is when a + /// DMA engine waiting on the transmitter may go on. + pub fn poll_tx(&mut self, now: Instant) -> bool { + let before = self.held.len(); + if before == 0 { + return false; + } + self.drain_tx(now); + self.sync_tx_flags(); + self.held.len() < before + } + + /// When a refused byte stops being retried, if one is held. + pub fn tx_deadline(&self) -> Option { + self.held_since.map(|t| t + XMIT_RETRY_TIMEOUT) + } + + /// The port can take a byte from a DMA engine: the holding register is + /// empty, or with the FIFOs on the transmit FIFO is not full. + pub fn tx_ready(&self) -> bool { + if self.fcr & FCR_FE != 0 { + self.held.len() <= FIFO_LENGTH + } else { + self.lsr & LSR_THRE != 0 + } + } + // ----- receive ----- fn read_rbr(&mut self) -> u8 { @@ -949,4 +1056,109 @@ mod tests { assert_eq!((u.bytes_out(), u.output_tail(usize::MAX).len()), (OUT_KEEP as u64 + 6, OUT_KEEP)); assert!(u.output_tail(2).ends_with("z!")); } + + /// A backend that takes bytes only while its gate is open. + #[derive(Default)] + struct Gated { + open: std::sync::atomic::AtomicBool, + out: Mutex>, + } + + impl SerialBackend for Gated { + fn send_byte(&self, b: u8) { + self.out.lock().unwrap().push(b); + } + fn try_send_byte(&self, b: u8) -> bool { + let open = self.open.load(std::sync::atomic::Ordering::SeqCst); + if open { + self.send_byte(b); + } + open + } + fn recv_byte(&self) -> io::Result { + Err(io::Error::new(io::ErrorKind::WouldBlock, "empty")) + } + } + + fn gated() -> (Ns16550, Arc) { + let g = Arc::new(Gated::default()); + let mut u = Ns16550::new(115200); + u.set_backend(g.clone()); + (u, g) + } + + fn open(g: &Gated, yes: bool) { + g.open.store(yes, std::sync::atomic::Ordering::SeqCst); + } + + /// Ports the held-byte case of QEMU's serial transmit retry: a byte the + /// backend refuses keeps the transmitter busy and goes out, in order, + /// once the backend takes it. + #[test] + fn a_refused_byte_is_held_and_sent_in_order_when_the_backend_opens() { + let (mut u, g) = gated(); + u.write(IER, 0x02); + u.read(IIR_FCR); + u.write(THR, b'a'); + assert_eq!(u.read(LSR) & 0x60, 0x20, "the shift register is stuck, the holding register is free"); + assert!(u.tx_ready()); + u.write(THR, b'b'); + assert_eq!(u.read(LSR) & 0x60, 0, "busy"); + assert!(!u.tx_ready()); + assert!(u.tx_deadline().is_some()); + assert!(!u.poll_tx(Instant::now()), "still refused"); + open(&g, true); + assert!(u.poll_tx(Instant::now())); + assert_eq!(*g.out.lock().unwrap(), b"ab"); + assert_eq!(u.read(LSR) & 0x60, 0x60); + assert!(u.tx_ready() && u.irq()); + assert_eq!((u.bytes_out(), u.tx_deadline()), (2, None)); + } + + /// The hold is bounded: past two seconds the byte is dropped and the + /// backend counts as disconnected, so later bytes complete at once until + /// it takes one again. + #[test] + fn a_hold_gives_up_after_two_seconds_and_the_backend_counts_as_gone() { + let (mut u, g) = gated(); + let t0 = Instant::now(); + u.write(THR, b'x'); + assert!(!u.poll_tx(t0 + Duration::from_millis(1999))); + assert!(u.poll_tx(t0 + Duration::from_secs(3)), "given up on"); + assert!(g.out.lock().unwrap().is_empty()); + u.write(THR, b'y'); + assert_eq!(u.read(LSR) & 0x60, 0x60, "a dead backend completes without waiting"); + open(&g, true); + u.write(THR, b'z'); + assert_eq!(*g.out.lock().unwrap(), b"z"); + open(&g, false); + u.write(THR, b'w'); + assert_eq!(u.read(LSR) & 0x40, 0, "it took a byte, so the next is worth waiting for"); + } + + /// Without the FIFOs the shift register and the holding register take two + /// bytes, and a third overwrites the holding register. A transmit overrun + /// with the FIFOs on drops the oldest byte behind the shift register. + #[test] + fn held_bytes_overrun_as_the_part_does() { + let (mut u, g) = gated(); + for b in [1, 2, 3] { + u.write(THR, b); + } + open(&g, true); + assert!(u.poll_tx(Instant::now())); + assert_eq!(*g.out.lock().unwrap(), [1, 3], "the third write replaced the second"); + g.out.lock().unwrap().clear(); + open(&g, false); + u.write(IIR_FCR, 1); + for b in 0..FIFO_LENGTH as u8 + 3 { + u.write(THR, b); + } + assert!(!u.tx_ready()); + open(&g, true); + assert!(u.poll_tx(Instant::now())); + let mut want = vec![0]; + want.extend(3..FIFO_LENGTH as u8 + 3); + assert_eq!(*g.out.lock().unwrap(), want, "the byte in the shift register survives"); + } } diff --git a/src/dev/odyssey/board.rs b/src/dev/odyssey/board.rs index d0b72fe63..4314024e4 100644 --- a/src/dev/odyssey/board.rs +++ b/src/dev/odyssey/board.rs @@ -923,6 +923,7 @@ impl Board { // established, so it is zero. let identity = self.identity(); let mut f = self.front.lock(); + f.slot = self.slot; f.identity = identity; f.widget_control = self.slot & ODSY_CONTROL_WIDGET_MASK; f.xt_req_timeout = ODSY_XT_REQ_TIMEOUT_RESET; diff --git a/src/dev/odyssey/consts.rs b/src/dev/odyssey/consts.rs index 23d143938..96f2508f9 100644 --- a/src/dev/odyssey/consts.rs +++ b/src/dev/odyssey/consts.rs @@ -179,8 +179,9 @@ pub(super) mod reg { // the bit and raises the flag, and zero, as out of reset, is no threshold. // Neither the roles nor the link between bit 15, flag 25 and any 0x6bX value // is established. The level's unit (32-bit words, a doubleword store counting - // two) is a model choice, as is the FIFO's size (super::cfifo::WORDS), - // independent of the stored values, and so is the low-water threshold (super::cfifo::LOW_WATER). The model contract is pinned by + // two) is a model choice, as is the FIFO's size (super::cfifo::WORDS) and depth + // (super::cfifo::DEPTH), independent of the stored values, and so is the + // low-water threshold (super::cfifo::LOW_WATER). The model contract is pinned by // the_level_counts_words_against_the_model_fifo_and_the_6b2_threshold and // the_threshold_follows_6b2. pub const FIFO_SETUP_FIRST: u32 = 0x6b0; @@ -319,6 +320,28 @@ pub(super) mod cfifo { // deeper or shallower. Test, a model contract: // the_fifo_size_ignores_the_set_up_words. pub const WORDS: u32 = 32768; + + // The waiting stores whose credits have been returned, in stores (one + // credit per store). MODEL CHOICE: the FIFO's depth is UNKNOWN (see + // WORDS). It must exceed the host's credit window of 100 (IP35) or 256 + // (IP30) doublewords, and a credit is inferred to come back when the + // board takes the doubleword into its FIFO. Exceeding the window is not + // enough: the + // kernel's initialise call stores its start-up program, about 12,291 + // words, with no level check, before the low-water interrupt has a + // destination, and a credit owed during that load can leave the writer + // held until the 15-second FIFO timer. 16,384 is the smallest power of two + // that covers the whole program even at one word per store, so none of it + // waits on the board's thread for its credit. The figure is a model choice + // derived from the documented ~12,291-word start-up program, as evidence + // for a depth whose grade is unknown. Limitation: past this depth a credit + // waits for the board's thread again, the processor can run this many + // stores ahead of the board, and a writer that does is held until the + // backlog drains below the low-water mark. The stored 0x6bX values do not + // change it (fifo.rs). Tests: the_credit_depth_is_deeper_than_the_host_window, + // every_store_inside_the_credited_depth_is_credited_and_the_next_is_owed, + // the_fifo_set_up_does_not_widen_the_credit_depth (a model contract). + pub const DEPTH: u32 = 16384; } pub(super) const ODSY_FIFO_SETUP_7B5_INDEX: usize = 4; diff --git a/src/dev/odyssey/fifo.rs b/src/dev/odyssey/fifo.rs index 87b128b63..0813ac82c 100644 --- a/src/dev/odyssey/fifo.rs +++ b/src/dev/odyssey/fifo.rs @@ -40,11 +40,40 @@ //! //! The processor's stores are the producer; whoever runs the queue (the //! board's thread, or an access that drains it with that thread stopped) -//! is the consumer. +//! is the consumer. Graphics credits, which only machines whose processor +//! interface counts stores use, are settled as a store enters the FIFO's +//! first `cfifo::DEPTH` places: at the store when there is room, or +//! when the stores ahead of it have run. That depth (16,384 stores) is a +//! MODEL CHOICE, independent of the stored `0x6bX` values: the smallest +//! power of two that holds the kernel's whole start-up program (about +//! 12,291 words) even at one word per store, and far deeper than the +//! largest host credit window (256). A writer is then credited as its stores enter and does +//! not wait on the board's thread. That matters because IRIX's high-water +//! handler looks at the count for only about ten microseconds before it +//! holds the writer, and while the X server's initialise call is loading +//! that program, before the low-water interrupt has a destination, only +//! the kernel's 15-second FIFO timer wakes a writer held there. Limitation: a writer that runs more than the depth plus its +//! window ahead of the board waits for the board's thread again, and is +//! held until the level falls below the low-water mark. Whether a 32-bit +//! store costs the hardware a whole credit is UNKNOWN; the model credits +//! one per store. +//! +//! The deeper credited FIFO widens what the guest can see. A counted writer +//! can now leave up to the depth plus its window (about 16,640 stores) +//! waiting, where 64 plus the window was the bound before, so the level, +//! the unreturned-credit field of `0x106c` (which saturates), status bit 17 +//! (below the low-water mark) and flag 9 follow a backlog that long: bit 17 +//! stays clear, and flag 9 latches, only after it drains. Doubleword stores +//! that deep reach the `0x6b2` threshold the kernel stores, 32,192 words, +//! which the old bound never did, so status bit 15 and flag 25 can now +//! rise for counted stores too; the kernel leaves flag 25 masked and does +//! not read bit 15. The level the guest reads stops at the model's FIFO +//! size. use super::consts::{cfifo, intr, ODSY_WORK_QUEUE_MAX}; use super::interface::HostInterface; +use super::pi_gfx::{pi_gfx_owe_credits, pi_gfx_repay_credits}; use super::worker::GAP; use std::cell::Cell; use std::sync::atomic::{AtomicBool, AtomicU32, Ordering::{Relaxed, SeqCst}}; @@ -64,18 +93,28 @@ pub(super) struct QueuedStore { pub value: u64, pub addr: u32, pub size: u8, + /// The processor interface counted it: its credit is due. + pub counted: bool, + /// It waited outside the FIFO, so the node holds its credit as owed. + pub owed: bool, } // A ring entry is (address word, value). The address word carries the -// aperture offset in its low 24 bits and the store's size above them. +// aperture offset in its low 24 bits and the store's size and credit +// state above them. const ENTRY_OFFSET_MASK: u32 = 0x00ff_ffff; const ENTRY_SIZE_SHIFT: u32 = 24; const ENTRY_SIZE_MASK: u32 = 0xf; +const ENTRY_COUNTED: u32 = 1 << 28; +const ENTRY_OWED: u32 = 1 << 29; impl QueuedStore { fn entry(&self) -> u32 { debug_assert!(self.addr <= ENTRY_OFFSET_MASK && self.size as u32 <= ENTRY_SIZE_MASK); - self.addr | (self.size as u32) << ENTRY_SIZE_SHIFT + self.addr + | (self.size as u32) << ENTRY_SIZE_SHIFT + | if self.counted { ENTRY_COUNTED } else { 0 } + | if self.owed { ENTRY_OWED } else { 0 } } fn from_entry(addr: u32, value: u64) -> Self { @@ -83,6 +122,8 @@ impl QueuedStore { value, addr: addr & ENTRY_OFFSET_MASK, size: ((addr >> ENTRY_SIZE_SHIFT) & ENTRY_SIZE_MASK) as u8, + counted: addr & ENTRY_COUNTED != 0, + owed: addr & ENTRY_OWED != 0, } } @@ -108,7 +149,7 @@ pub(super) struct Run { impl Run { pub fn new() -> Self { - Run { stores: [QueuedStore { value: 0, addr: 0, size: 0 }; RUN_MAX], len: 0 } + Run { stores: [QueuedStore { value: 0, addr: 0, size: 0, counted: false, owed: false }; RUN_MAX], len: 0 } } /// The stores to run, oldest first. @@ -127,6 +168,8 @@ struct Levels { words: AtomicU32, /// Stores pushed and not yet taken out. stores: AtomicU32, + /// Counted stores waiting; while zero no store needs credit work. + counted: AtomicU32, } pub(super) struct CommandFifo { @@ -146,9 +189,9 @@ pub(super) struct CommandFifo { /// The board's thread is parked (`wait_for_work`): a producer unparks it /// only then, so a store reads a line nobody writes instead of locking. asleep: CachePadded, - /// The processor waits for the board's thread, for a refused access: its - /// next slice starts without the pause for a burst of stores, and a - /// pause under way ends. + /// The processor waits for the board's thread, for a refused access or + /// a counted store whose credit is owed: its next slice starts without + /// the pause for a burst of stores, and a pause under way ends. hurry: CachePadded, /// The board's thread is running stores. pub slice: CachePadded, @@ -168,7 +211,13 @@ pub(super) struct CommandFifo { /// counted after its last look at the hurry request. #[cfg(test)] pub test_parked: AtomicU32, - /// Tests: the next push calls this, once, as soon as its + /// Tests: pushes the full ring has refused. + #[cfg(test)] + pub test_refused: AtomicU32, + /// Tests: ends a long slice's hold (`Board::test_slice_sleep_ms`) early. + #[cfg(test)] + pub test_release: AtomicBool, + /// Tests: the next uncounted push calls this, once, as soon as its /// store is in the ring: a consumer that takes the store at that /// moment, before the push has returned. #[cfg(test)] @@ -182,7 +231,7 @@ impl Default for CommandFifo { // place because the ring is too big for the stack. ring: unsafe { Box::new_zeroed().assume_init() }, pop: Mutex::new(()), - levels: CachePadded::new(Levels { words: AtomicU32::new(0), stores: AtomicU32::new(0) }), + levels: CachePadded::new(Levels { words: AtomicU32::new(0), stores: AtomicU32::new(0), counted: AtomicU32::new(0) }), threshold_6b2: CachePadded::new(AtomicU32::new(0)), running: CachePadded::new(AtomicBool::new(false)), asleep: CachePadded::new(AtomicBool::new(false)), @@ -199,6 +248,10 @@ impl Default for CommandFifo { #[cfg(test)] test_parked: AtomicU32::new(0), #[cfg(test)] + test_refused: AtomicU32::new(0), + #[cfg(test)] + test_release: AtomicBool::new(false), + #[cfg(test)] test_after_ring_push: Mutex::new(None), } } @@ -251,9 +304,9 @@ impl CommandFifo { self.running.load(SeqCst) && !self.has_room() } - /// The processor waits for the board's thread to run what is queued, as - /// a refused access does. A thread already in its pause (`gap`) is woken - /// to run at once. + /// The processor waits for the board's thread to run what is queued: + /// a refused access, or a counted store whose credit is owed. A thread + /// already in its pause (`gap`) is woken to run at once. pub fn hurry(&self) { // A refused store retries in a loop; write the line only when it // changes, as IMPACT does for its asleep flag. @@ -310,20 +363,36 @@ impl CommandFifo { self.running.load(SeqCst) || self.pending() || self.held() } - /// Queue a store. False when the host ring has no room for it. - pub fn push(&self, front: &Mutex, addr: u64, value: u64, size: u32) -> bool { + /// The stores in the FIFO's first places, credited. + pub fn in_fifo(&self) -> u32 { + self.stores().min(cfifo::DEPTH) + } + + /// Queue a store. A counted store is credited at once when it lands + /// inside the FIFO's first places and owed otherwise. False when the + /// board's FIFO has no room for it. + pub fn push(&self, front: &Mutex, addr: u64, value: u64, size: u32, counted: bool) -> bool { let words = store_words(size); if !self.has_room() { + #[cfg(test)] + self.test_refused.fetch_add(1, SeqCst); return false; } - let q = QueuedStore { value, addr: addr as u32, size: size as u8 }; + let mut q = QueuedStore { value, addr: addr as u32, size: size as u8, counted, owed: false }; + // A counted store's place is settled against the consumer's pops. + let _c = counted.then(|| self.pop.lock()); // Count the store before the ring shows it. The board's thread can // take it and retire it the moment it is in the ring, and a retire // must find every store it takes already counted, or the level falls // below zero. Until the push lands the counts are a store high, // which reads as a store still waiting, never as one gone. - self.levels.stores.fetch_add(1, SeqCst); + if counted { + self.levels.counted.fetch_add(1, SeqCst); + } + let stores_before = self.levels.stores.fetch_add(1, SeqCst); let before = self.levels.words.fetch_add(words, SeqCst); + let credited = counted && stores_before < cfifo::DEPTH; + q.owed = counted && !credited; if !self.ring.try_push(q.entry(), q.value) { // With one processor this cannot happen: has_room counted the // ring's stores and nothing else pushes. With two, the other @@ -333,11 +402,32 @@ impl CommandFifo { // come back off. self.levels.words.fetch_sub(words, SeqCst); self.levels.stores.fetch_sub(1, SeqCst); + if counted { + self.levels.counted.fetch_sub(1, SeqCst); + } return false; } #[cfg(test)] - if let Some(consumer) = self.test_after_ring_push.lock().take() { - consumer(self, front); + if !counted { + if let Some(consumer) = self.test_after_ring_push.lock().take() { + consumer(self, front); + } + } + if counted { + let mut f = front.lock(); + if credited { + f.return_credit(); + } else { + if let Some(pi) = &f.pi { + pi_gfx_owe_credits(pi.as_ref(), f.slot, 1); + } + drop(f); + // The credit comes back only as the stores ahead retire, and + // the processor can be watching its credit count (a read the + // bus does not refuse), so the board's thread runs without + // its pause. + self.hurry(); + } } let after = before + words; let mark = self.threshold_6b2.load(SeqCst); @@ -353,17 +443,28 @@ impl CommandFifo { } /// Look at the oldest waiting stores, up to `RUN_MAX`, without taking - /// them out; the caller runs them, then `retire`s the ones it ran. + /// them out; the caller runs them, then `retire`s the ones it ran. While + /// stores wait past the credited depth, one of which may owe its credit, + /// the run is one store, so each owed credit is settled as the store + /// ahead of it leaves rather than after a whole run. Inside the depth + /// every counted store already has its credit, so the run is full. pub fn take_run(&self, run: &mut Run) { + // Only a hint for the run's length, so a stale count costs a longer + // run, never a lost credit (`retire` settles each one). The stores + // are read first and short-circuit it: a Relaxed read of `counted` + // after seeing more than DEPTH stores sees every counted push whose + // store count it read, since a push counts `counted` first. + let owing = self.stores() > cfifo::DEPTH && self.levels.counted.load(Relaxed) != 0; + let max = if owing { 1 } else { RUN_MAX }; let mut raw = [(0u32, 0u64); RUN_MAX]; - run.len = self.ring.peek_run_at(0, &mut raw); + run.len = self.ring.peek_run_at(0, &mut raw[..max]); for (store, (addr, value)) in run.stores.iter_mut().zip(&raw[..run.len]) { *store = QueuedStore::from_entry(*addr, *value); } } /// Take the first `n` stores of `run` out of the FIFO: they have run. The - /// level and the stores waiting move once for the run, not + /// level, the stores waiting and the credits move once for the run, not /// once per store, which is what the board's thread spends most of its /// time on when each store costs little to decode. A store the thread has /// run but not retired still counts in the level, so the level never @@ -379,6 +480,7 @@ impl CommandFifo { } let ran = &run.stores[..n]; let words: u32 = ran.iter().map(QueuedStore::words).sum(); + let counted_ran = ran.iter().filter(|q| q.counted).count() as u32; let _c = self.pop.lock(); self.ring.consume_n_at(0, n); if self.levels.words.load(SeqCst) < cfifo::LOW_WATER + words { @@ -393,17 +495,59 @@ impl CommandFifo { } else { self.levels.words.fetch_sub(words, SeqCst); } - self.levels.stores.fetch_sub(n as u32, SeqCst); + let stores_before = self.levels.stores.fetch_sub(n as u32, SeqCst); + if counted_ran != 0 { + self.levels.counted.fetch_sub(counted_ran, SeqCst); + } + // `pop` is held, and a counted push takes it to count `counted`, so + // this read needs no ordering of its own. Relaxed keeps the + // machines with no counted stores (every one but a linked + // processor interface) off a sequentially consistent load per run. + if self.levels.counted.load(Relaxed) == 0 { + return; + } + // The next waiting store moves into the FIFO's last credited place + // and is credited. + for k in 1..=n as u32 { + if stores_before - k < cfifo::DEPTH { + continue; + } + // The run is already out of the ring, so count back from its head. + let place = cfifo::DEPTH as usize - 1 - (n - k as usize); + if let Some((addr, value)) = self.ring.peek_nth_at(0, place) { + let entered = QueuedStore::from_entry(addr, value); + let mut f = front.lock(); + if entered.owed { + if let Some(pi) = &f.pi { + pi_gfx_repay_credits(pi.as_ref(), f.slot, 1); + } + } else if entered.counted { + f.return_credit(); + } + } + } } - /// Abandon every waiting store. - pub fn clear(&self) { + /// Abandon every waiting store; an owed one outside the FIFO gives its + /// owed credit back. + pub fn clear(&self, front: &Mutex) { let _c = self.pop.lock(); + let mut index = 0u32; while let Some((addr, value)) = self.ring.peek() { self.ring.consume(); let q = QueuedStore::from_entry(addr, value); self.levels.words.fetch_sub(q.words(), SeqCst); self.levels.stores.fetch_sub(1, SeqCst); + if q.counted { + self.levels.counted.fetch_sub(1, SeqCst); + } + if q.owed && index >= cfifo::DEPTH { + let f = front.lock(); + if let Some(pi) = &f.pi { + pi_gfx_owe_credits(pi.as_ref(), f.slot, -1); + } + } + index += 1; } } diff --git a/src/dev/odyssey/interface.rs b/src/dev/odyssey/interface.rs index 139129035..8790b8b5c 100644 --- a/src/dev/odyssey/interface.rs +++ b/src/dev/odyssey/interface.rs @@ -27,13 +27,22 @@ use super::consts::{ }; use super::fifo::CommandFifo; use super::mmio::{access_waits, command_store, fifo_port}; +use super::pi_gfx::{pi_gfx_return_credits, PiGfxLink}; use super::registers::reg; +use super::PacketSender; use parking_lot::Mutex; use std::sync::Arc; pub(super) struct HostInterface { - /// Where interrupt packets go (the iris glue's HEART). + /// The crosstalk widget number. + pub slot: u32, + /// Where interrupt packets go on the Octane (the iris glue's HEART). pub irq: Option>, + /// Where interrupt packets go on a machine whose fabric carries them to + /// any widget (the Fuel's XBridge crossbar); `irq` is unused then. + pub packets: Option, + /// The master node's processor interface, for graphics credits. + pub pi: Option>, /// The widget number of HEART, the packets' destination. pub heart_widget: u32, /// ODSY_IDENTITY, composed from the revision at reset. @@ -73,6 +82,9 @@ pub(super) struct HostInterface { /// The command FIFO (fifo.rs), whose level STATUS0 reports. pub fifo: Arc, + /// Tests: stores count as counted by a processor interface. + #[cfg(test)] + pub test_counted_stores: u8, /// The front accesses' own trace numbering; the board's accesses keep /// the state's. pub trace_sequence: u64, @@ -81,7 +93,10 @@ pub(super) struct HostInterface { impl Default for HostInterface { fn default() -> Self { HostInterface { + slot: 0, irq: None, + packets: None, + pi: None, heart_widget: 8, identity: 0, widget_control: 0, @@ -108,6 +123,8 @@ impl Default for HostInterface { displayed_buffer: 0, displayed_written: false, fifo: Default::default(), + #[cfg(test)] + test_counted_stores: 0, trace_sequence: 0, } } @@ -281,6 +298,15 @@ impl HostInterface { self.update_irq(); } } + + /// A counted store's credit, to the processor interface the node names. + pub fn return_credit(&mut self) { + if let Some(pi) = &self.pi { + pi_gfx_return_credits(pi.as_ref(), self.slot, 1); + } else { + crate::dlog!(LogModule::Odyssey, "sgi-odyssey: counted graphics store has no processor interface for its return"); + } + } } // The front registers. @@ -387,7 +413,7 @@ impl HostInterface { self.flctl_target_reported = true; crate::dlog!( LogModule::Odyssey, - "sgi-odyssey: flow-control interrupt target {} and credit target {} stored", + "sgi-odyssey: flow-control interrupt target {} and credit target {} stored; credits still return through the node interface", (value as u32) >> flctl::INTR_TARGET_SHIFT, ((value >> flctl::TARGET_SHIFT) as u32) & flctl::TARGET_MASK ); @@ -453,15 +479,20 @@ pub(super) fn front_mmio_read(f: &Mutex, addr: u64, size: u32) -> /// A processor's store through the front alone: true when it went to a /// front register or joined the command FIFO for the board's thread, false -/// when it needs the board's state. -pub(super) fn front_mmio_write(f: &Mutex, fifo: &CommandFifo, addr: u64, value: u64, size: u32) -> bool { +/// when it needs the board's state. `counted`: the processor interface counted +/// it. +pub(super) fn front_mmio_write(f: &Mutex, fifo: &CommandFifo, addr: u64, value: u64, size: u32, counted: bool) -> bool { if front_register(addr, size, true) { - f.lock().write(addr, value, size); + let mut f = f.lock(); + f.write(addr, value, size); + if counted { + f.return_credit(); + } return true; } // A queued store needs the board's thread to run it, and the queue to // have room; otherwise the caller runs it. - command_store(addr, size) && fifo.running.load(std::sync::atomic::Ordering::SeqCst) && fifo.push(f, addr, value, size) + command_store(addr, size) && fifo.running.load(std::sync::atomic::Ordering::SeqCst) && fifo.push(f, addr, value, size, counted) } /// The fabric packet's address bits; the widget sits above them. @@ -488,6 +519,12 @@ impl HostInterface { pub fn send_interrupt(&mut self, packet: u64, number: u8, set: bool) { let widget = (packet >> XIO_PACKET_ADDRESS_BITS) as u32; let address = packet & ((1u64 << XIO_PACKET_ADDRESS_BITS) - 1); + if let Some(send) = &self.packets { + if !send(widget as u8, address, number, set) { + crate::dlog!(LogModule::Odyssey, "odyssey: interrupt packet to widget {widget} address {address:#x} was refused"); + } + return; + } if widget != self.heart_widget || address != HEART_PACKET { crate::dlog!(LogModule::Odyssey, "odyssey: interrupt packet to widget {widget} address {address:#x} has no receiver"); return; diff --git a/src/dev/odyssey/mmio.rs b/src/dev/odyssey/mmio.rs index 59e3c7708..c009b1218 100644 --- a/src/dev/odyssey/mmio.rs +++ b/src/dev/odyssey/mmio.rs @@ -18,6 +18,7 @@ use super::consts::{ }; use super::fifo::CommandFifo; use super::interface::{front_mmio_read, front_mmio_write, front_register, HostInterface}; +use super::pi_gfx::pi_gfx_take_mark; use super::raster_unit::RasterQueue; use super::reader::DisplayShared; use super::registers::reg; @@ -132,6 +133,10 @@ impl Board { // see the drawing are answered busy until the queue is empty // (BoardLocks::bus_read) or, with no thread running (unit tests), run what // is queued first (Board::work_drain). + // + // The Octane has no PI link (only Hub and Bedrock machines have graphics + // credits), so the front's `pi` stays None there and every credit call is + // skipped. } @@ -221,13 +226,31 @@ impl Board { } } +/// Whether this store is counted: the processor interface's mark. +pub(super) fn store_counted(front: &parking_lot::Mutex) -> bool { + // The Octane has no PI link, so nothing sets the mark there. + #[cfg(test)] + if front.lock().test_counted_stores != 0 { + return true; + } + let _ = front; + pi_gfx_take_mark() +} + impl Board { /// A store to the board's state. pub fn mmio_write(&mut self, addr: u64, value: u64, size: u32) { + let counted = store_counted(&self.front); + self.mmio_write_counted(addr, value, size, counted); + } + + pub fn mmio_write_counted(&mut self, addr: u64, value: u64, size: u32, mut counted: bool) { let fifo = self.fifo.clone(); - let queued = command_store(addr, size) && fifo.queues() && fifo.push(&self.front, addr, value, size); + let queued = command_store(addr, size) && fifo.queues() && fifo.push(&self.front, addr, value, size, counted); if queued { - // With no board thread running the storing caller runs it. + // Credited as it enters the FIFO (CommandFifo::push). With no + // board thread running the storing caller runs it. + counted = false; if !fifo.running.load(std::sync::atomic::Ordering::SeqCst) && !fifo.held() { self.work_drain(addr); } @@ -242,6 +265,10 @@ impl Board { } self.write_dispatch(addr, value, size); } + if counted { + // It went through the FIFO inside the store. + self.front.lock().return_credit(); + } } } @@ -487,7 +514,8 @@ impl BoardLocks { /// answer busy, when it waits for queued stores the board's thread will /// run, or when it is a command store and the host ring is full while /// the board's thread runs: the processor retries the store, taking its - /// interrupts meanwhile, while the board's thread makes room. + /// interrupts meanwhile, while the board's thread makes room. The + /// processor interface's mark stays untaken until the store goes in. /// /// A store that waits is held back for the command FIFO only, not the /// raster queue: the raster unit never shared a lock with the board's @@ -524,25 +552,27 @@ impl BoardLocks { /// A command store the caller has classified, holding `submit`: it joins /// the FIFO for the board's thread, else, with no thread, runs here. fn write_command(&self, cpu: crate::locks::SpinGuard<'_, ()>, addr: u64, value: u64, size: u32) { + let counted = store_counted(&self.front); while self.fifo.running.load(std::sync::atomic::Ordering::SeqCst) { // The ring refuses a push that finds the last place taken between // the room check and the push: the board's thread owns the state, // so the store goes in behind it rather than around it. - if self.fifo.push(&self.front, addr, value, size) { + if self.fifo.push(&self.front, addr, value, size, counted) { return; } std::thread::yield_now(); } // SAFETY: no board thread is running. - unsafe { self.board(cpu) }.mmio_write(addr, value, size); + unsafe { self.board(cpu) }.mmio_write_counted(addr, value, size, counted); } /// A store to the board's state that is not a command store, holding /// `submit`. fn write_board(&self, cpu: crate::locks::SpinGuard<'_, ()>, addr: u64, value: u64, size: u32) { + let counted = store_counted(&self.front); // SAFETY: the waiting accesses have drained the FIFO, or the board's // thread is stopped. - unsafe { self.board(cpu) }.mmio_write(addr, value, size); + unsafe { self.board(cpu) }.mmio_write_counted(addr, value, size, counted); } /// A store for a caller other than the bus: through the front alone when it @@ -551,7 +581,8 @@ impl BoardLocks { /// for a store that waits. pub fn write(&self, addr: u64, value: u64, size: u32) { if front_register(addr, size, true) { - front_mmio_write(&self.front, &self.fifo, addr, value, size); + let counted = store_counted(&self.front); + front_mmio_write(&self.front, &self.fifo, addr, value, size, counted); return; } let cpu = self.submit.lock(); diff --git a/src/dev/odyssey/mod.rs b/src/dev/odyssey/mod.rs index c7edb4bf6..98e204abc 100644 --- a/src/dev/odyssey/mod.rs +++ b/src/dev/odyssey/mod.rs @@ -41,7 +41,8 @@ //! - `memory`, `scanout`, `reader`, `display`: board memory and planes; the //! marks writers make and the display thread's own reader, with the scanned //! frame and retrace; what reaches the window. -//! - `pixel_dma`: DMA between guest memory and planes. +//! - `pixel_dma`, `pi_gfx`: DMA between guest memory and planes, and graphics +//! credits for machines whose processor interface counts them. //! - `depth_word`: the board's 24-bit float depth word and its offset. //! - `consts`: register, command and field constants. //! @@ -73,6 +74,12 @@ mod interface; mod memory; mod mmio; mod objects; +mod pi_gfx; +pub use pi_gfx::{PiGfx, PI_GFX_REGS}; +pub(crate) use pi_gfx::{ + pi_gfx_mark_store, pi_gfx_unmark_store, PiGfxLink, PI_GFX_BIAS, PI_GFX_CREDIT, PI_GFX_CREDIT_MASK, + PI_GFX_INT_COUNT, PI_GFX_PAGE, PI_GFX_PAGE_ADDRESS, PI_GFX_PAGE_ENABLE, PI_GFX_SLOTS, PI_GFX_SLOT_REGS, +}; mod pixel_dma; mod pixel_write; mod raster; @@ -129,9 +136,15 @@ pub struct Odyssey { rec: Option, } +/// Carries a board interrupt packet into a machine's crosstalk fabric: the +/// destination widget, the address within it, the vector, and whether the +/// source rose or fell. False when nothing at the destination takes it. +pub type PacketSender = Arc bool + Send + Sync>; + impl Odyssey { - /// A board of `product` (v12, v10, v8, v6) at `widget`: its interrupt - /// packets reach `irq` (HEART), its DMA reaches `mem`. + /// A board of `product` (v12, v10, v8, v6) at `widget` on the Octane: + /// the processor reaches it through the IP30 widget windows, its + /// interrupt packets reach `irq` (HEART), its DMA reaches `mem`. pub fn new( widget: u8, product: &str, @@ -140,25 +153,62 @@ impl Odyssey { status: Arc, ) -> Result, String> { let mut s = Board::new(product, widget as u32)?; - let rec = match std::env::var("IRIS_ODSY_REC") { - Ok(path) => Some(record::Recorder::create(&path).map_err(|e| format!("IRIS_ODSY_REC {path}: {e}"))?), - Err(_) => None, - }; - let mem: Arc = match &rec { + let rec = Self::recorder()?; + s.guest = Some(Self::recorded_mem(mem, &rec)); + s.front.lock().irq = Some(irq); + Ok(Self::with_board(s, widget_small_base(widget), widget_large_base(widget), status, rec)) + } + + /// A board of `product` at `widget` on a machine that hands the board + /// widget-relative addresses, as Bedrock's I/O interface does, and + /// carries its interrupt packets through `packets`. + pub fn new_widget_relative( + widget: u8, + product: &str, + packets: PacketSender, + mem: Arc, + status: Arc, + ) -> Result, String> { + let mut s = Board::new(product, widget as u32)?; + let rec = Self::recorder()?; + s.guest = Some(Self::recorded_mem(mem, &rec)); + s.front.lock().packets = Some(packets); + // Every address is already an aperture offset: no large window. + Ok(Self::with_board(s, 0, u64::MAX, status, rec)) + } + + /// `IRIS_ODSY_REC`'s recorder, when it is set. + fn recorder() -> Result, String> { + match std::env::var("IRIS_ODSY_REC") { + Ok(path) => Ok(Some(record::Recorder::create(&path).map_err(|e| format!("IRIS_ODSY_REC {path}: {e}"))?)), + Err(_) => Ok(None), + } + } + + /// `mem`, recorded when a recording is running. + fn recorded_mem(mem: Arc, rec: &Option) -> Arc { + match rec { Some(r) => Arc::new(record::RecordingMem { inner: mem, rec: r.clone() }), None => mem, - }; - s.guest = Some(mem); - s.front.lock().irq = Some(irq); - Ok(Arc::new(Self { - small: widget_small_base(widget), - large: widget_large_base(widget), + } + } + + fn with_board( + s: Box, + small: u64, + large: u64, + status: Arc, + rec: Option, + ) -> Arc { + Arc::new(Self { + small, + large, state: BoardLocks::new(s), display: Arc::new(Display::new(status)), running: Arc::new(AtomicBool::new(false)), thread: Mutex::new(None), rec, - })) + }) } pub fn display(&self) -> Arc { @@ -167,6 +217,18 @@ impl Odyssey { self.display.clone() } + /// Link the board to the node whose processor interfaces count its + /// graphics credits, as a Bedrock node does. With none linked, as on + /// the Octane and the Fuel, every credit call is a no-op. + pub(crate) fn set_pi_link(&self, link: Arc) { + self.state.front.lock().pi = Some(link); + } + + /// The node link [`Self::set_pi_link`] set, if any. + pub(crate) fn pi_link(&self) -> Option> { + self.state.front.lock().pi.clone() + } + /// The aperture offset of `addr`, in either window. fn offset(&self, addr: u64) -> Option { let off = if addr >= self.large { addr - self.large } else { addr.wrapping_sub(self.small) }; diff --git a/src/dev/odyssey/pi_gfx.rs b/src/dev/odyssey/pi_gfx.rs new file mode 100644 index 000000000..d6a53ab0b --- /dev/null +++ b/src/dev/odyssey/pi_gfx.rs @@ -0,0 +1,210 @@ +//! Graphics flow control in the Hub and Bedrock processor interfaces. The +//! Octane has no processor-interface link (its board's `pi` stays `None`), +//! so on that machine the credit calls are no-ops. `PiGfx` is the register +//! block a node's processor interface embeds; its functions are pure state. +//! The board reaches the link through the `PiGfxLink` trait. +//! +//! Ported from hw/mips/sgi/pi-gfx.c in jamesbraid/qemu-sgi-origami. + +use crate::devlog::LogModule; +use std::cell::Cell; + +pub(crate) const PI_GFX_SLOTS: usize = 2; +pub const PI_GFX_REGS: u32 = 10; + +/// The five registers of one slot, in address order. +pub(crate) const PI_GFX_PAGE: u32 = 0; +pub(crate) const PI_GFX_CREDIT: u32 = 1; +pub(crate) const PI_GFX_BIAS: u32 = 2; +pub(crate) const PI_GFX_INT_COUNT: u32 = 3; +pub(crate) const PI_GFX_SLOT_REGS: u32 = 5; + +/// Page: bit 40 enables it, bits 39:17 name it; the rest is reserved. +pub(crate) const PI_GFX_PAGE_ENABLE: u64 = 1 << 40; +pub(crate) const PI_GFX_PAGE_ADDRESS: u64 = 0xfffffe0000; +pub(super) const PI_GFX_PAGE_MASK: u64 = PI_GFX_PAGE_ENABLE | PI_GFX_PAGE_ADDRESS; +pub(crate) const PI_GFX_CREDIT_MASK: u32 = 0xfff; +pub(super) const PI_GFX_BIAS_MASK: u32 = 0xfff; +pub(super) const PI_GFX_INT_MASK: u32 = 0x3ffffff; + +#[derive(Clone, Copy, Default)] +pub struct PiGfx { + pub page: [u64; PI_GFX_SLOTS], + pub credit: [u32; PI_GFX_SLOTS], + pub bias: [u32; PI_GFX_SLOTS], + pub int_count: [u32; PI_GFX_SLOTS], + pub int_compare: [u32; PI_GFX_SLOTS], + /// The slot's graphics interrupt has been raised for the counter falling + /// below the bias, and not yet re-armed by the counter rising above it. + pub starved: [bool; PI_GFX_SLOTS], + /// Host-only: the zero-length stall has been logged for this slot. + pub stall_logged: [bool; PI_GFX_SLOTS], + /// Bedrock: a counted store left the counter below the bias, and the + /// slot's next counted store waits until the processor has taken the + /// graphics interrupt or the counter has risen above the bias. + pub stopped: [bool; PI_GFX_SLOTS], + /// Bedrock: the interrupt counter runs (the credit counter is at zero). + pub int_running: [bool; PI_GFX_SLOTS], + /// The clock tick at which int_count held its value. + pub int_epoch: [u64; PI_GFX_SLOTS], +} + +/// The interface a node device implements for the graphics heads behind it +/// (return_credits, owe_credits and repay_credits). `widget` is the head's crosstalk widget number, the +/// value the driver writes into the node-widget map. +pub(crate) trait PiGfxLink: Send + Sync { + /// A graphics head took `n` counted stores off its port. + fn return_credits(&self, widget: u32, n: u32); + /// Credits a head owes for counted stores it has not yet taken off its + /// port; a negative count cancels owed credits, as a board reset that + /// abandons the words does. A node without a ledger ignores the debt. + fn owe_credits(&self, widget: u32, n: i32) { + let _ = (widget, n); + } + /// Owed credits returned as the head takes the words. A node without a + /// ledger routes a repayment as an ordinary return. + fn repay_credits(&self, widget: u32, n: u32) { + self.return_credits(widget, n); + } +} + +/// Return `credits` to `node` for stores a head took off its port. +pub(super) fn pi_gfx_return_credits(node: &dyn PiGfxLink, widget: u32, credits: u32) { + node.return_credits(widget, credits); +} + +/// Record `credits` a head owes `node` for stores it has not yet taken. +pub(super) fn pi_gfx_owe_credits(node: &dyn PiGfxLink, widget: u32, credits: i32) { + node.owe_credits(widget, credits); +} + +/// Repay `credits` that `node` held as owed. +pub(super) fn pi_gfx_repay_credits(node: &dyn PiGfxLink, widget: u32, credits: u32) { + node.repay_credits(widget, credits); +} + +impl PiGfx { + /// The counter above the bias re-arms the slot's graphics interrupt. + fn rearm(&mut self, slot: usize) { + if self.credit[slot] > self.bias[slot] { + self.starved[slot] = false; + } + } + + /// Register n of the block, 0 to PI_GFX_REGS - 1, counted in + /// doublewords from PI_GFX_BASE. + pub fn read(&self, n: u32) -> u64 { + let slot = (n / PI_GFX_SLOT_REGS) as usize; + + assert!(n < PI_GFX_REGS); + match n % PI_GFX_SLOT_REGS { + PI_GFX_PAGE => self.page[slot], + PI_GFX_CREDIT => self.credit[slot] as u64, + PI_GFX_BIAS => self.bias[slot] as u64, + PI_GFX_INT_COUNT => self.int_count[slot] as u64, + _ => self.int_compare[slot] as u64, + } + } + + /// A write keeps the field's bits; a write to the credit counter replaces + /// the count. + pub fn write(&mut self, n: u32, value: u64) { + let slot = (n / PI_GFX_SLOT_REGS) as usize; + + assert!(n < PI_GFX_REGS); + match n % PI_GFX_SLOT_REGS { + PI_GFX_PAGE => self.page[slot] = value & PI_GFX_PAGE_MASK, + PI_GFX_CREDIT => { + self.credit[slot] = (value as u32) & PI_GFX_CREDIT_MASK; + self.rearm(slot); + } + PI_GFX_BIAS => self.bias[slot] = (value as u32) & PI_GFX_BIAS_MASK, + PI_GFX_INT_COUNT => self.int_count[slot] = (value as u32) & PI_GFX_INT_MASK, + _ => self.int_compare[slot] = (value as u32) & PI_GFX_INT_MASK, + } + } + + /// Whether the slot's page is enabled and covers physical address `addr`. + pub fn covers(&self, slot: usize, addr: u64) -> bool { + assert!(slot < PI_GFX_SLOTS); + let page = self.page[slot]; + (page & PI_GFX_PAGE_ENABLE) != 0 + && (addr & PI_GFX_PAGE_ADDRESS) == (page & PI_GFX_PAGE_ADDRESS) + } + + /// An uncached I/O store by the processor in `slot` reached physical + /// address bits 39:0 `addr`. Charges one credit when the slot's page is + /// enabled and covers it, and returns whether it did. `who` names the + /// device in log messages. + pub fn count_store(&mut self, slot: usize, addr: u64, who: &str) -> bool { + if !self.covers(slot, addr) { + return false; + } + // Whether the twelve-bit counter wraps, saturates or goes negative is + // unknown. It wraps, and says so. + if self.credit[slot] == 0 { + crate::dlog!(LogModule::Odyssey, "{}: graphics credit counter {} wrapped below zero", who, (b'A' + slot as u8) as char); + } + self.credit[slot] = self.credit[slot].wrapping_sub(1) & PI_GFX_CREDIT_MASK; + true + } + + /// After a counted store: whether the slot's graphics interrupt must be + /// raised now. True once when the counter has fallen below the bias; the + /// counter rising above the bias again re-arms it. The stall that + /// accompanies it on the chip is modelled as zero length. + pub fn take_starved(&mut self, slot: usize, who: &str) -> bool { + assert!(slot < PI_GFX_SLOTS); + if self.starved[slot] || self.credit[slot] >= self.bias[slot] { + return false; + } + self.starved[slot] = true; + if !self.stall_logged[slot] { + self.stall_logged[slot] = true; + crate::dlog!( + LogModule::Odyssey, + "{}: graphics credit counter {} fell below its bias; interrupt raised at once, the write stall and the interrupt counter are not modelled", + who, + (b'A' + slot as u8) as char + ); + } + true + } + +} + +thread_local! { + /// The mark on the store a processor path is forwarding, per host + /// thread: the forward and the head's handler run on the issuing + /// vCPU's thread. + static PI_GFX_FORWARDING_COUNTED: Cell = const { Cell::new(false) }; +} + +/// A processor interface marks each store it forwards down its I/O path; +/// returns the previous mark, to be restored with pi_gfx_unmark_store. +pub(crate) fn pi_gfx_mark_store(counted: bool) -> bool { + PI_GFX_FORWARDING_COUNTED.with(|m| m.replace(counted)) +} + +pub(crate) fn pi_gfx_unmark_store(previous: bool) { + PI_GFX_FORWARDING_COUNTED.with(|m| m.set(previous)); +} + +/// The one-shot read: the call clears the mark. +pub(super) fn pi_gfx_take_mark() -> bool { + PI_GFX_FORWARDING_COUNTED.with(|m| m.replace(false)) +} + +impl PiGfx { + /// `credits` credits returned to `slot`. + pub fn add_credits(&mut self, slot: usize, credits: u32, who: &str) { + assert!(slot < PI_GFX_SLOTS); + let sum = self.credit[slot] as u64 + credits as u64; + if sum > PI_GFX_CREDIT_MASK as u64 { + crate::dlog!(LogModule::Odyssey, "{}: graphics credit counter {} wrapped above {:#x}", who, (b'A' + slot as u8) as char, PI_GFX_CREDIT_MASK); + } + self.credit[slot] = (sum & PI_GFX_CREDIT_MASK as u64) as u32; + self.rearm(slot); + } +} + diff --git a/src/dev/odyssey/tests/fifo.rs b/src/dev/odyssey/tests/fifo.rs index 1ba7f683b..7f088ebf0 100644 --- a/src/dev/odyssey/tests/fifo.rs +++ b/src/dev/odyssey/tests/fifo.rs @@ -1,4 +1,4 @@ -//! The command FIFO: queueing, the board's thread, status. +//! The command FIFO: queueing, the board's thread, status, credits. use crate::dev::odyssey::board::Board; use crate::dev::odyssey::command::{CFIFO_PREAMBLE, CfifoPreamble}; @@ -8,6 +8,10 @@ use crate::dev::odyssey::consts::{ use crate::dev::odyssey::fifo::{CommandFifo, RING_STORES, RUN_MAX, Run}; use crate::dev::odyssey::interface::HostInterface; use crate::dev::odyssey::mmio::BoardLocks; +use crate::dev::odyssey::pi_gfx::{ + pi_gfx_mark_store, pi_gfx_take_mark, pi_gfx_unmark_store, PI_GFX_BIAS, PI_GFX_CREDIT, + PI_GFX_PAGE, PI_GFX_PAGE_ENABLE, PiGfx, PiGfxLink, +}; use crate::dev::odyssey::tests::{ board, board_with_ram, cmd, cmd64, reg_read, reg_write, route_to_heart, select, FLAG_VECTOR, Ram, VPRO_WIDGET, @@ -210,41 +214,151 @@ fn a_waiting_bus_read_is_busy_until_the_board_thread_has_run_the_stores() { h.work_stop(); } +/// Records every call a board makes on its processor-interface link. +#[derive(Default)] +struct PiStub(Mutex>); + +impl PiGfxLink for PiStub { + fn return_credits(&self, widget: u32, n: u32) { + self.0.lock().push(("return", widget, n as i64)); + } + fn owe_credits(&self, widget: u32, n: i32) { + self.0.lock().push(("owe", widget, n as i64)); + } + fn repay_credits(&self, widget: u32, n: u32) { + self.0.lock().push(("repay", widget, n as i64)); + } +} + +impl PiStub { + fn count(&self, kind: &str) -> i64 { + self.0.lock().iter().filter(|c| c.0 == kind).map(|c| c.2).sum() + } + /// Credits given back so far, returned or repaid. + fn settled(&self) -> i64 { + self.count("return") + self.count("repay") + } +} + +/// A board whose stores are all counted and queued, with a stub link. +fn credit_board() -> (Box, Arc) { + let (s, _) = board("v12"); + let pi = Arc::new(PiStub::default()); + { + let mut f = s.front.lock(); + f.pi = Some(pi.clone()); + f.test_counted_stores = 1; + } + // No work thread runs: hold the queue and step by hand. + s.fifo.test_words.store(1, SeqCst); + (s, pi) +} + +/// Command work credits at FIFO entry: a counted store's credit +/// comes back when the board takes it into its FIFO, the oldest +/// `cfifo::DEPTH` waiting stores at once; each store behind them is owed, then +/// repaid as the work ahead runs and it moves up. A store that does +/// not wait is credited inside the store. +/// +/// QEMU: command-work-credits-at-fifo-entry. +#[test] +fn a_counted_store_is_credited_as_it_enters_the_fifo() { + let (mut s, pi) = credit_board(); + let stores = cfifo::DEPTH + 8; + + for _ in 0..stores { + s.mmio_write(consts::cmd::FIFO, 0, 8); + } + // Credits count stores; the level counts words, two a doubleword, and + // reads no more than the model's FIFO size. + assert_eq!(s.fifo.stores(), stores); + assert_eq!(s.fifo.level(), (2 * stores).min(cfifo::WORDS)); + assert_eq!(s.fifo.in_fifo(), cfifo::DEPTH); + assert_eq!(pi.count("return"), cfifo::DEPTH as i64); + assert_eq!(pi.count("owe"), (stores - cfifo::DEPTH) as i64); + assert_eq!(pi.count("repay"), 0); + assert!(pi.0.lock().iter().all(|c| c.1 == VPRO_WIDGET)); + + // One store runs per slice: the next one moves up and is repaid. + s.work_step(&mut Budget::stores(1)); + assert_eq!(pi.count("repay"), 1); + assert_eq!(pi.settled(), cfifo::DEPTH as i64 + 1); + + s.work_step(&mut Budget::unlimited()); + assert!(!s.fifo.pending()); + assert_eq!(pi.settled(), stores as i64); + + // A store that does not wait is credited inside the store. + s.fifo.test_words.store(0, SeqCst); + s.mmio_write(consts::cmd::FIFO, 0, 8); + assert!(!s.work_pending()); + assert_eq!(pi.settled(), stores as i64 + 1); + assert_eq!(pi.count("owe"), (stores - cfifo::DEPTH) as i64); +} + /// A run of stores the board's thread has looked at and then runs leaves -/// the FIFO together. The stores run in order; the level and the store count -/// end where taking the stores out one by one leaves them, including when -/// more stores join the queue after the run was taken. +/// the FIFO together. The stores run in order; the level, the store count +/// and the credits end where taking the stores out one by one leaves them, +/// including when a counted store joins the queue after the run was taken. #[test] -fn a_run_retires_with_the_levels_of_one_by_one() { +fn a_run_retires_with_the_levels_and_credits_of_one_by_one() { fn queue(s: &mut Board) { - for _ in 0..70 { + s.front.lock().test_counted_stores = 0; + for _ in 0..cfifo::DEPTH + 6 { s.mmio_write(consts::cmd::FIFO, 0, 8); } } - fn join(s: &mut Board) { + fn counted_join(s: &mut Board) { + s.front.lock().test_counted_stores = 1; for _ in 0..10 { s.mmio_write(consts::cmd::FIFO, 0, 8); } } - // One at a time: the store count and the level are the reference. - let (mut one, _, _) = queued_board(); + // One at a time: the store count and the credits are the reference. + let (mut one, pi_one) = credit_board(); queue(&mut one); - join(&mut one); + counted_join(&mut one); for _ in 0..RUN_MAX { step(&mut one); } - let (mut run, _, _) = queued_board(); + let (mut run, pi_run) = credit_board(); queue(&mut run); let mut taken = Run::new(); run.fifo.take_run(&mut taken); - assert_eq!(taken.stores().len(), RUN_MAX, "a full run"); - join(&mut run); - assert_eq!(run.fifo.stores(), 80); + assert_eq!(taken.stores().len(), RUN_MAX, "no counted store waits yet: a full run"); + counted_join(&mut run); + assert_eq!(run.fifo.stores(), cfifo::DEPTH + 16); run.fifo.retire(&run.front, &taken, RUN_MAX); assert_eq!(run.fifo.stores(), one.fifo.stores()); assert_eq!(run.fifo.level(), one.fifo.level()); + assert_eq!(pi_run.count("repay"), pi_one.count("repay")); + assert_eq!(pi_run.count("return"), pi_one.count("return")); + assert_eq!(pi_run.count("owe"), pi_one.count("owe")); + assert!(pi_run.count("repay") > 0, "the run moved owed stores up"); + // The counted stores left all sit inside the credited depth, so the + // next run is full. + assert!(run.fifo.stores() <= cfifo::DEPTH); + run.fifo.take_run(&mut taken); + assert_eq!(taken.stores().len(), RUN_MAX); +} + +/// The board's thread takes stores one at a time only while some wait past +/// the credited depth, where one may owe its credit; a backlog of counted +/// stores inside the depth, all credited, goes in full runs. +#[test] +fn a_run_is_one_store_only_past_the_credited_depth() { + let (mut s, _) = credit_board(); + for _ in 0..cfifo::DEPTH { + s.mmio_write(consts::cmd::FIFO, 0, 8); + } + let mut run = Run::new(); + s.fifo.take_run(&mut run); + assert_eq!(run.stores().len(), RUN_MAX, "every waiting store is credited"); + s.mmio_write(consts::cmd::FIFO, 0, 8); + s.fifo.take_run(&mut run); + assert_eq!(run.stores().len(), 1, "one store owes its credit"); } /// A slice cut short by its budget retires only the stores it ran: the rest @@ -265,6 +379,76 @@ fn a_slice_cut_short_leaves_the_rest_of_its_run_queued() { assert_eq!(ram.half(HOST_WRITE_BASE + 6), 0xa5a5); } +/// A reset abandons the waiting stores and cancels each owed credit +/// (one owe of -1 per owed store); stores not owed cancel nothing. +#[test] +fn a_reset_cancels_the_credits_owed_for_abandoned_stores() { + let (mut s, pi) = credit_board(); + + for _ in 0..cfifo::DEPTH + 8 { + s.mmio_write(consts::cmd::FIFO, 0, 8); + } + assert_eq!(pi.count("owe"), 8); + s.work_reset(); + assert!(!s.fifo.pending()); + assert_eq!(s.fifo.in_fifo(), 0); + assert_eq!(pi.count("owe"), 0); + assert_eq!(pi.0.lock().iter().filter(|c| c.0 == "owe" && c.2 == -1).count(), 8); + assert_eq!(pi.count("repay"), 0); +} + +/// The processor interface's mark makes a store counted, once, and a +/// counted store with no link logs and returns nothing. +#[test] +fn the_processor_mark_counts_exactly_one_store() { + let (mut s, pi) = credit_board(); + + { + let mut f = s.front.lock(); + f.test_counted_stores = 0; + } + s.fifo.test_words.store(0, SeqCst); + s.mmio_write(consts::cmd::FIFO, 0, 8); + assert_eq!(pi.settled(), 0); + + let previous = pi_gfx_mark_store(true); + s.mmio_write(consts::cmd::FIFO, 0, 8); + pi_gfx_unmark_store(previous); + assert_eq!(pi.0.lock().as_slice(), &[("return", VPRO_WIDGET, 1)]); + + // One-shot: the mark was taken by the first store. + s.mmio_write(consts::cmd::FIFO, 0, 8); + assert_eq!(pi.settled(), 1); + + // The Octane: no link, so the credit is dropped with a log line. + s.front.lock().pi = None; + pi_gfx_mark_store(true); + s.mmio_write(consts::cmd::FIFO, 0, 8); + assert_eq!(pi.settled(), 1); + assert!(!pi_gfx_take_mark()); +} + +/// The register block's pure state (PiGfx). +#[test] +fn the_credit_block_starves_a_widget_that_runs_out_of_credits() { + let mut g = PiGfx::default(); + + g.write(PI_GFX_PAGE, PI_GFX_PAGE_ENABLE | 0x2_0000); + g.write(PI_GFX_CREDIT, 3); + g.write(PI_GFX_BIAS, 2); + assert!(g.covers(0, 0x2_1000)); + assert!(!g.covers(0, 0x4_0000)); + assert!(g.count_store(0, 0x2_0000, "t")); + assert!(!g.take_starved(0, "t")); + assert!(g.count_store(0, 0x2_0000, "t")); + assert!(g.take_starved(0, "t")); + assert!(!g.take_starved(0, "t")); + g.add_credits(0, 5, "t"); + assert_eq!(g.read(PI_GFX_CREDIT), 6); + assert!(!g.starved[0]); +} + + /// The model's low-water threshold, in waiting words (a model choice). const LOW_WATER_MARK: u32 = cfifo::LOW_WATER; const HOST_WRITE_BASE: u64 = 0xb70000; @@ -508,6 +692,8 @@ fn the_threshold_follows_6b2() { /// MODEL CONTRACT ONLY: the FIFO's size is unknown and the model's size is /// its own choice. Whatever `0x6b1` the set-up stores, the level clamps at /// cfifo::WORDS however many stores wait. +/// +/// QEMU: command-work-credit-depth-ignores-setup covers the credit depth. #[test] fn the_fifo_size_ignores_the_set_up_words() { for word_6b1 in [1u32, 256, 0x7fff_ffff] { @@ -555,35 +741,129 @@ fn low_water_is_raised_as_the_queue_falls_below_the_mark() { assert_eq!(rec.events(FLAG_VECTOR), [true]); } -/// A hurry ends the board thread's pause before a slice. The pause is set -/// far longer than the wait below, so only the hurry runs the stores in -/// time; the stores wait while the thread sits in it. +/// MODEL CONTRACT ONLY: the FIFO's depth is unknown and the model's depth is +/// its own choice. The stored `0x6b1` and `0x6b2` values (256 and 32192, as +/// the kernels write them) do not widen the credits: only the oldest +/// `cfifo::DEPTH` waiting stores, the model's depth, have their credits back, so the +/// processor interface still holds back a writer that runs ahead of the slow +/// board. +/// +/// QEMU: command-work-credit-depth-ignores-setup. +#[test] +fn the_fifo_set_up_does_not_widen_the_credit_depth() { + let (mut s, pi) = credit_board(); + s.fifo.test_words.store(0, SeqCst); + cmd64(&mut s, 0x6b1, 256); + cmd64(&mut s, 0x6b3, 24144); + cmd64(&mut s, 0x6b2, 32192); + finish(&mut s); + pi.0.lock().clear(); + + s.fifo.test_words.store(1, SeqCst); + let stores = cfifo::DEPTH + 8; + for _ in 0..stores { + s.mmio_write(consts::cmd::FIFO, 0, 8); + } + assert_eq!(s.fifo.stores(), stores); + assert_eq!(pi.count("return"), cfifo::DEPTH as i64); + finish(&mut s); + assert_eq!(pi.settled(), stores as i64); +} + +/// MODEL CHOICE: the FIFO holds far more than the host's credit window, so +/// a writer that fills the whole window, 256 doublewords on the IP30 and at most 256 on +/// the IP35, gets every credit back as its stores enter, with the board's +/// thread taking none of them, and owes nothing. A credit that waited on +/// the board's thread could miss the high-water handler's ten-microsecond +/// poll. +#[test] +fn the_credit_depth_is_deeper_than_the_host_window() { + const LARGEST_HOST_WINDOW: u32 = 256; + let (mut s, pi) = credit_board(); + for _ in 0..=LARGEST_HOST_WINDOW { + s.mmio_write(consts::cmd::FIFO, 0, 8); + } + assert_eq!(s.fifo.stores(), LARGEST_HOST_WINDOW + 1, "the board took none"); + assert_eq!(pi.count("return"), LARGEST_HOST_WINDOW as i64 + 1); + assert_eq!(pi.count("owe"), 0); + finish(&mut s); +} + +/// MODEL CHOICE: the credited depth is 16,384 stores, the smallest power of +/// two that holds the kernel's start-up program of about 12,291 words at +/// one word per store. With the board taking nothing, every one of the +/// first 16,384 counted stores is credited as it enters, and +/// the next is owed until the board takes a store. #[test] -fn a_hurry_ends_the_board_threads_pause() { - let (s, _rec) = board("v12"); +fn every_store_inside_the_credited_depth_is_credited_and_the_next_is_owed() { + const DEPTH: u32 = 16_384; + let (mut s, pi) = credit_board(); + for _ in 0..DEPTH { + s.mmio_write(consts::cmd::FIFO, 0, 8); + } + assert_eq!(s.fifo.stores(), DEPTH, "the board took none"); + assert_eq!(pi.count("return"), DEPTH as i64); + assert_eq!(pi.count("owe"), 0); + s.mmio_write(consts::cmd::FIFO, 0, 8); + assert_eq!(pi.count("return"), DEPTH as i64); + assert_eq!(pi.count("owe"), 1, "the store past the depth owes its credit"); + finish(&mut s); + assert_eq!(pi.settled(), DEPTH as i64 + 1); +} + +/// A counted store whose credit is owed has the board's thread run without +/// its pause, so the credit comes back as soon as the stores ahead retire. +/// A guest that watches its credit count reads a register the bus does not +/// refuse, so nothing else hurries the thread; with the pause the kernel saw +/// its count fall below the bias in a burst of a few hundred stores and gave +/// the FIFO up as stuck. The stores inside the model's depth are credited +/// at once and do not hurry. +#[test] +fn a_counted_store_owed_its_credit_hurries_the_board_thread() { + let (mut s, pi) = credit_board(); + for _ in 0..cfifo::DEPTH { + s.mmio_write(consts::cmd::FIFO, 0, 8); + } + assert!(!s.fifo.take_hurry(), "a credited store leaves the pause alone"); + s.mmio_write(consts::cmd::FIFO, 0, 8); + assert_eq!(pi.count("owe"), 1); + assert!(s.fifo.take_hurry(), "an owed credit hurries the board's thread"); + finish(&mut s); + + // On the board's thread, with a pause far longer than the wait below: + // only the hurry runs the burst in time. + let stores = cfifo::DEPTH + 136; let h = BoardLocks::new(s); h.fifo.test_gap_ms.store(60_000, SeqCst); + // credit_board holds the queue for stepping by hand. + h.fifo.test_words.store(0, SeqCst); + pi.0.lock().clear(); h.work_start(); - // One store starts the thread, and the burst waits until the thread is - // in its pause: a burst that lands before it would find the request - // already made and prove nothing about waking a sleeper. + // One credited store starts the thread, and the burst waits until the + // thread is in its pause: a burst that lands before it would find the + // request already made and prove nothing about waking a sleeper. h.write(consts::cmd::FIFO, 0, 8); let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10); while h.fifo.test_parked.load(SeqCst) == 0 { assert!(std::time::Instant::now() < deadline, "the board's thread never reached its pause"); std::thread::yield_now(); } - for _ in 1..200 { + for _ in 1..stores { h.write(consts::cmd::FIFO, 0, 8); } - assert_eq!(h.fifo.stores(), 200, "the stores wait out the pause"); - h.fifo.hurry(); + // The last few stores may find the queue run dry and wait out a pause + // of their own, but inside the depth they are credited at the store. let deadline = std::time::Instant::now() + std::time::Duration::from_secs(10); - while h.fifo.stores() != 0 { - assert!(std::time::Instant::now() < deadline, "the board's thread kept its pause: {} stores wait", h.fifo.stores()); + while pi.settled() != stores as i64 { + assert!(std::time::Instant::now() < deadline, "the board's thread kept its pause: {} stores wait, {} of {stores} credits back", h.fifo.stores(), pi.settled()); std::thread::yield_now(); } + assert!(pi.count("owe") > 0, "the burst ran past the model's depth"); + assert_eq!(pi.count("repay"), pi.count("owe")); + // Cut a pause still under way short, so the stop does not wait it out. + h.fifo.hurry(); h.work_stop(); + assert_eq!(h.fifo.stores(), 0); } /// A stopping machine finishes the work waiting; a reset abandons it. @@ -628,7 +908,7 @@ fn two_processors_storing_to_the_fifo_lose_nothing_and_keep_each_ones_order() { let pushed = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { for seq in 0..PER { let size = if seq % 3 == 0 { 8 } else { 4 }; - while !h.fifo.push(&h.front, consts::cmd::FIFO, id << 32 | seq, size) { + while !h.fifo.push(&h.front, consts::cmd::FIFO, id << 32 | seq, size, false) { std::thread::yield_now(); } } @@ -676,7 +956,7 @@ fn a_store_run_before_its_push_returns_is_already_in_the_level() { let h = BoardLocks::new(s); h.front.lock().intr_flags &= !intr::FIFO_LOW_WATER; for seq in 0..20 { - assert!(h.fifo.push(&h.front, consts::cmd::FIFO, seq, 4)); + assert!(h.fifo.push(&h.front, consts::cmd::FIFO, seq, 4, false)); } fn run_everything(fifo: &CommandFifo, front: &Mutex) { let mut run = Run::new(); @@ -687,7 +967,7 @@ fn a_store_run_before_its_push_returns_is_already_in_the_level() { assert_eq!((fifo.stores(), fifo.level()), (0, 0), "the run took out what the level counted"); } *h.fifo.test_after_ring_push.lock() = Some(run_everything); - assert!(h.fifo.push(&h.front, consts::cmd::FIFO, 20, 8)); + assert!(h.fifo.push(&h.front, consts::cmd::FIFO, 20, 8, false)); assert!(h.fifo.test_after_ring_push.lock().is_none(), "the consumer ran inside the push"); assert_eq!((h.fifo.stores(), h.fifo.level()), (0, 0)); assert_ne!(h.front.lock().intr_flags & intr::FIFO_LOW_WATER, 0, "the level fell through the mark"); @@ -790,7 +1070,7 @@ fn two_processors_racing_for_the_last_place_in_the_ring_leave_one_store_in() { // The ring has room for one more store when it holds RING_STORES - 1. let fill = |h: &BoardLocks| { for _ in 0..RING_STORES - 1 { - assert!(h.fifo.push(&h.front, consts::cmd::FIFO, 0, 4)); + assert!(h.fifo.push(&h.front, consts::cmd::FIFO, 0, 4, false)); } }; fill(&h); @@ -805,7 +1085,7 @@ fn two_processors_racing_for_the_last_place_in_the_ring_leave_one_store_in() { for _ in 0..ROUNDS { start.wait(); // A panic in a push must fail the round, not leave the others waiting. - let pushed = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| h.fifo.push(&h.front, consts::cmd::FIFO, 0, 4))); + let pushed = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| h.fifo.push(&h.front, consts::cmd::FIFO, 0, 4, false))); match pushed { Ok(true) => { accepted.fetch_add(1, SeqCst); @@ -875,7 +1155,7 @@ fn processors_wait_for_each_other_and_the_board_thread_does_not_wait_for_them() // before the lock was held. Nothing is queued here, so queue through the // unlocked ring and see it drain. for w in &words[..4] { - assert!(h.fifo.push(&h.front, w.port, w.value, w.size as u32)); + assert!(h.fifo.push(&h.front, w.port, w.value, w.size as u32, false)); } let deadline = std::time::Instant::now() + std::time::Duration::from_secs(20); while h.fifo.pending() { @@ -922,11 +1202,14 @@ fn a_refused_command_push_retries_behind_the_board_thread() { let (h, _rec) = threaded_board(); let words = preamble_twice(); - start_long_slice(&h, &words, 1000); + // The slice holds the board's thread until the test releases it, so the + // ring stays full however long the host takes to get here. + start_long_slice(&h, &words, 600_000); while h.fifo.stores() < RING_STORES { assert!(h.bus_write(consts::cmd::FIFO, 0, 4)); } let stores = h.fifo.stores(); + let refused = h.fifo.test_refused.load(SeqCst); let done = Arc::new(std::sync::atomic::AtomicBool::new(false)); let store = { let (h, done) = (h.clone(), done.clone()); @@ -935,9 +1218,15 @@ fn a_refused_command_push_retries_behind_the_board_thread() { done.store(true, SeqCst); }) }; - std::thread::sleep(std::time::Duration::from_millis(150)); + // The store is refused and retries: wait until it has been. + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(20); + while h.fifo.test_refused.load(SeqCst) == refused { + assert!(std::time::Instant::now() < deadline, "the store was never refused"); + std::thread::yield_now(); + } assert!(!done.load(SeqCst), "the store was dropped or ran around the board's thread"); assert_eq!(h.fifo.stores(), stores); + h.fifo.test_release.store(true, SeqCst); store.join().unwrap(); assert!(h.fifo.level() <= cfifo::WORDS); h.fifo.test_words.store(0, SeqCst); diff --git a/src/dev/odyssey/tests/interrupts.rs b/src/dev/odyssey/tests/interrupts.rs index 6bbb67ca9..8f89604cf 100644 --- a/src/dev/odyssey/tests/interrupts.rs +++ b/src/dev/odyssey/tests/interrupts.rs @@ -437,3 +437,26 @@ fn a_held_flag_is_sent_with_the_source_enables_clear() { reg_write(&mut s, route::DEST_UPPER, HEART_UPPER); assert_eq!(rec.events(FLAG_VECTOR), [true]); } + +/// A board built for a machine whose fabric carries packets to any widget +/// hands them to that machine's sender, to the widget the routing names, +/// and never to HEART's line: the Fuel's widget 13 reaches Bedrock this way. +#[test] +fn a_packet_sender_takes_the_interrupt_to_the_routed_widget() { + let (mut s, rec) = board("v12"); + let seen = Arc::new(Mutex::new(Vec::new())); + let log = seen.clone(); + s.front.lock().packets = Some(Arc::new(move |widget, address, vector, level| { + log.lock().push((widget, address, vector, level)); + true + })); + s.mmio_write(ROUTE_DEST_UPPER, 1 << 25, 4); + s.mmio_write(ROUTE_DEST_LOWER, 0x1230, 4); + s.mmio_write(ROUTE_VECTORS, TEST_VECTOR as u64, 4); + s.mmio_write(INTR_ENABLE_SET, INTR_FIFO_LOW_WATER, 4); + assert_eq!(*seen.lock(), vec![(1, 0x1230, TEST_VECTOR as u8, true)]); + assert!(rec.silent(), "the packet did not go to HEART"); + + s.mmio_write(INTR_CLEAR, INTR_FIFO_LOW_WATER, 4); + assert_eq!(seen.lock().last(), Some(&(1, 0x1230, TEST_VECTOR as u8, false))); +} diff --git a/src/dev/odyssey/worker.rs b/src/dev/odyssey/worker.rs index 0bb30032c..8ac0a4677 100644 --- a/src/dev/odyssey/worker.rs +++ b/src/dev/odyssey/worker.rs @@ -24,7 +24,8 @@ pub(super) const ODYSSEY_WORK_THREAD: &str = "odyssey-work"; /// The pause before the board's thread runs a burst, so the stores of the /// burst land first and one run takes them together. A processor that /// waits on the thread cuts it short (`CommandFifo::hurry`): an access the -/// bus refused. +/// bus refused, or a counted store whose credit stays owed until the stores +/// ahead of it retire. pub(super) const GAP: Duration = Duration::from_micros(100); /// How much one run may do: for tests a count of stores. The budget is @@ -142,7 +143,11 @@ impl Board { // A run that takes as long as a long draw would. let sleep_ms = std::mem::take(&mut self.test_slice_sleep_ms); if sleep_ms != 0 { - std::thread::sleep(std::time::Duration::from_millis(sleep_ms as u64)); + // Until the time is up or a test releases the hold. + let end = std::time::Instant::now() + std::time::Duration::from_millis(sleep_ms as u64); + while std::time::Instant::now() < end && !self.fifo.test_release.load(SeqCst) { + std::thread::sleep(std::time::Duration::from_millis(1)); + } } } @@ -195,6 +200,6 @@ impl Board { /// Reset clears the FIFO and abandons the waiting stores. pub fn work_reset(&mut self) { - self.fifo.clear(); + self.fifo.clear(&self.front); } } diff --git a/src/dev/ohci.rs b/src/dev/ohci.rs new file mode 100644 index 000000000..2625c5b0f --- /dev/null +++ b/src/dev/ohci.rs @@ -0,0 +1,589 @@ +// Derived from MAME's xbox_usb.cpp and xbox_usb.h (the OHCI host controller), +// BSD-3-Clause, copyright Samuele Zannoli. + +//! The Lucent USS-312 USB controller: two root ports, OHCI registers and +//! descriptor processing through the PCI bridge's DMA interface. +//! +//! Functions plug into a root port with `attach`: the Fuel fits a keyboard and +//! a mouse (`usb_hid`), and the tests drive the descriptor and packet +//! scheduling engine through a test-only function. Isochronous +//! traffic, suspend/resume, external hubs, USB storage and legacy keyboard/SMI +//! routing are unsupported. Scheduling uses synthetic work units rather than +//! wire time. Register behavior follows OpenHCI 1.0a and the USS-312 revision 5 +//! data sheet. +//! +//! Ported from src/mame/shared/xbox_usb.cpp and xbox_usb.h in MAME, +//! revision 2ee9b1e30299b122b131cab0bd7912abc2b095ba. + +use crate::dev::heart::{host_clock, Clock}; +use crate::irq::IrqLine; +use crate::pci::{Bar, PciBusDevice, PciConfig, PciConfigPolicy, PciDma, PCI_INTA}; +use crate::traits::{Device, Resettable}; +use parking_lot::Mutex; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, OnceLock}; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[repr(u32)] +pub enum Pid { + Setup = 0, + Out = 1, + In = 2, +} +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct UsbPacket { + pub endpoint: u8, + pub pid: Pid, + pub toggle: bool, + /// IN requests name capacity; OUT and SETUP carry their bytes. + pub capacity: usize, + pub data: Vec, + pub time_ms: u64, +} +#[derive(Clone, Debug, PartialEq, Eq)] +pub enum UsbOutcome { + Ack(Vec), + Nak, + Stall, + NoResponse, + CrcError, + BitStuffError, +} +pub trait UsbFunction: Send + Sync { + fn reset(&self); + fn address(&self) -> u8; + fn low_speed(&self) -> bool { + false + } + fn transact(&self, packet: UsbPacket) -> UsbOutcome; +} + +mod engine; +mod schedule; + +const EVENTS: u32 = 0x4000_007f; +const WDH: u32 = 2; +const UE: u32 = 16; +const RHSC: u32 = 64; +const PPS: u32 = 0x100; +const CHANGES: u32 = 0x1f0000; +#[derive(Clone)] +struct State { + r: [u32; 23], + ports: [Option>; 2], + low_speed: [bool; 2], + reset_ms: [u8; 2], + done_delay: u8, + now_ms: u64, + active_fi: u32, + bit_credit: u64, +} +impl State { + fn new() -> Self { + let mut r = [0; 23]; + r[0] = 0x110; + r[13] = 0x2edf; + r[17] = 0x628; + r[18] = 0x1000_0902; + r[19] = 0x60000; + Self { + r, + ports: [None, None], + low_speed: [false; 2], + reset_ms: [0; 2], + done_delay: 7, + now_ms: 0, + active_fi: 0x2edf, + bit_credit: 0, + } + } + fn soft_reset(&mut self) { + let control = (self.r[1] & 0x300) | 0xc0; + let keep = [self.r[18], self.r[19], self.r[20], self.r[21], self.r[22]]; + let ports = self.ports.clone(); + let low_speed = self.low_speed; + let resets = self.reset_ms; + let time = self.now_ms; + *self = Self::new(); + self.r[1] = control; + self.r[18..23].copy_from_slice(&keep); + self.ports = ports; + self.low_speed = low_speed; + self.reset_ms = resets; + self.now_ms = time; + } + fn irq(&self) -> bool { + self.r[1] & 0x100 == 0 && self.r[4] & 0x80000000 != 0 && self.r[3] & self.r[4] & EVENTS != 0 + } + fn powered(&self, p: usize) -> bool { + self.r[18] & 0x200 != 0 || self.r[21 + p] & PPS != 0 + } + fn individual(&self, p: usize) -> bool { + self.r[18] & 0x100 != 0 && self.r[19] & (1 << (17 + p)) != 0 + } + fn power(&mut self, p: usize, on: bool) { + if self.r[18] & 0x200 != 0 || on { + self.r[21 + p] |= PPS; + if self.ports[p].is_some() && self.r[21 + p] & 1 == 0 { + self.r[21 + p] |= 1 | 0x10000 | if self.low_speed[p] { 0x200 } else { 0 }; + self.r[3] |= RHSC; + } + } else { + let previous = self.r[21 + p]; + self.r[21 + p] &= !(PPS | 0x17); + if previous & 1 != 0 { + self.r[21 + p] |= 0x10000; + } + if previous & 2 != 0 { + self.r[21 + p] |= 0x20000; + } + if previous & 3 != 0 { + self.r[3] |= RHSC; + } + self.reset_ms[p] = 0; + } + } +} + +pub struct Ohci { + config: PciConfig, + state: Mutex, + pass: Mutex<()>, + dma: Mutex>>, + irq: Mutex, + running: AtomicBool, + /// Source of the frame count: the machine's own clock, so the guest's + /// idea of a millisecond and the controller's agree. + time: OnceLock, + /// `time` reading at the last frame boundary delivered; `None` while stopped. + clock: Mutex>, +} +impl Ohci { + pub fn new() -> Arc { + let config = PciConfig::new_with_policy( + 0x11c1, + 0x5802, + 0x0c0310, + 0x10, + [ + Bar::Mem(0x1000), + Bar::Unused, + Bar::Unused, + Bar::Unused, + Bar::Unused, + Bar::Unused, + ], + PCI_INTA, + PciConfigPolicy { + status: 0x0200, + status_w1c_mask: 0x8000, + command_mask: 0x0006, + latency_mask: 0, + cache_line_mask: 0, + registers: &[], + }, + ); + config.set_subsystem_identity(0xffff_ffff); + Arc::new(Self { + config, + state: Mutex::new(State::new()), + pass: Mutex::new(()), + dma: Mutex::new(None), + irq: Mutex::new(IrqLine::unconnected()), + running: AtomicBool::new(false), + time: OnceLock::new(), + clock: Mutex::new(None), + }) + } + /// Use `clock` for frame timing. Call before `start`. + pub fn set_clock(&self, clock: Clock) { + assert!( + !self.running.load(Ordering::Acquire) && self.time.set(clock).is_ok(), + "connect the OHCI clock once, before start" + ); + } + fn now(&self) -> u64 { + (self.time.get_or_init(host_clock))() + } + /// Deliver every frame boundary the clock has passed. Every register + /// access calls this, so a read sees the frame count the guest's own + /// clock implies, however late the machine's poll or timer ran. + pub fn sync_clock(&self) { + // Held through delivery: a second caller must not find the boundary + // consumed and read the frame count before the first has applied it. + let mut clock = self.clock.lock(); + let Some(at) = clock.as_mut() else { return }; + let now = self.now(); + let elapsed = now.saturating_sub(*at) / 1_000_000; + *at += elapsed * 1_000_000; + self.advance_frames(elapsed); + } + fn drive_irq(&self) { + loop { + let level = self.state.lock().irq(); + let line = self.irq.lock().clone(); + line.set(level); + // A callback may acknowledge the interrupt or another writer may + // change it while delivery is outside the state lock. + if self.state.lock().irq() == level { + break; + } + } + } + pub fn attach(&self, port: usize, function: Arc) -> Result<(), String> { + if port >= 2 { + return Err("OHCI has two ports".into()); + } + if self.state.lock().ports[port].is_some() { + return Err("OHCI port occupied".into()); + } + let low_speed = function.low_speed(); + function.reset(); + { + let _pass = self.pass.lock(); + let mut s = self.state.lock(); + if s.ports[port].is_some() { + return Err("OHCI port occupied".into()); + } + if s.powered(port) { + s.r[21 + port] |= 1 | 0x10000; + } + s.low_speed[port] = low_speed; + if low_speed { + s.r[21 + port] |= 0x200; + } + s.ports[port] = Some(function); + s.r[3] |= RHSC; + } + self.drive_irq(); + Ok(()) + } + #[cfg(test)] + pub fn detach(&self, port: usize) -> Result<(), String> { + if port >= 2 { + return Err("OHCI has two ports".into()); + } + { + let _pass = self.pass.lock(); + let mut s = self.state.lock(); + let enabled = s.r[21 + port] & 2; + s.ports[port] = None; + s.low_speed[port] = false; + s.reset_ms[port] = 0; + s.r[21 + port] = + (s.r[21 + port] & PPS) | 0x10000 | if enabled != 0 { 0x20000 } else { 0 }; + s.r[3] |= RHSC; + } + self.drive_irq(); + Ok(()) + } + /// Deterministic millisecond ticks; `sync_clock` delivers the same ticks from the clock. + pub fn advance_frames(&self, count: u64) { + for _ in 0..count { + if !self.running.load(Ordering::Acquire) { + break; + } + { + let _pass = self.pass.lock(); + if !self.running.load(Ordering::Acquire) { + break; + } + let mut s = self.state.lock().clone(); + self.frame(&mut s); + *self.state.lock() = s; + } + self.drive_irq(); + } + } + fn frame(&self, s: &mut State) { + s.now_ms = s.now_ms.wrapping_add(1); + for p in 0..2 { + if s.reset_ms[p] != 0 { + s.reset_ms[p] -= 1; + if s.reset_ms[p] == 0 { + s.r[21 + p] = (s.r[21 + p] & !16) | 2 | (1 << 20); + s.r[3] |= RHSC; + } + } + } + if s.r[1] & 0xc0 != 0x80 || s.r[3] & UE != 0 { + return; + } + s.bit_credit += 12000; + while s.bit_credit >= u64::from((s.active_fi & 0x3fff) + 1) { + s.bit_credit -= u64::from((s.active_fi & 0x3fff) + 1); + s.active_fi = s.r[13]; + if self.schedule(s).is_err() { + s.r[3] |= UE; + s.bit_credit = 0; + self.config.set_status_bits(0x8000); + break; + } + } + } +} +impl PciBusDevice for Ohci { + fn config(&self) -> &PciConfig { + &self.config + } + fn connect_dma(&self, dma: Arc) { + *self.dma.lock() = Some(dma); + } + fn connect_interrupt(&self, line: IrqLine) { + *self.irq.lock() = line; + self.drive_irq(); + } + fn bar_read(&self, bar: usize, off: u32, width: usize) -> u32 { + if bar != 0 || width != 4 || off & 3 != 0 || off > 0x58 { + crate::dlog!( + crate::devlog::LogModule::Usb, + "ohci unsupported read at register offset {off:#x}, width {width}" + ); + return 0; + } + self.sync_clock(); + let s = self.state.lock(); + let i = (off / 4) as usize; + if i == 5 { + s.r[4] + } else if i == 21 || i == 22 { + s.r[i] | if s.r[18] & 0x200 != 0 { PPS } else { 0 } + } else { + s.r[i] + } + } + fn bar_write(&self, bar: usize, off: u32, val: u32, width: usize) { + if bar != 0 || width != 4 || off & 3 != 0 || off > 0x58 { + crate::dlog!( + crate::devlog::LogModule::Usb, + "ohci unsupported write at register offset {off:#x}, width {width}" + ); + return; + } + self.sync_clock(); + // Taken before `pass`, as `sync_clock` orders them. Only a control + // write can restart the frame phase. + let mut anchor = (off == 4).then(|| self.clock.lock()); + let mut reset = None; + { + let _pass = self.pass.lock(); + let mut s = self.state.lock(); + match off / 4 { + 1 => { + let old = s.r[1]; + s.r[1] = val & 0x7ff; + if val & 0xc0 == 0x80 && old & 0xc0 != 0x80 { + // OpenHCI 1.0a 6.2.1, 7.3.2, 7.3.3: entering + // UsbOperational loads HcFmRemaining from the frame + // interval and increments HcFmNumber, with no SOF at + // the load; the first follows one frame later. The + // free-running clock's phase therefore restarts here. + s.bit_credit = 0; + s.active_fi = s.r[13]; + s.r[14] = (s.r[13] & 0x8000_0000) | (s.r[13] & 0x3fff); + let old_frame = s.r[15] as u16; + let frame = old_frame.wrapping_add(1); + s.r[15] = u32::from(frame); + if (old_frame ^ frame) & 0x8000 != 0 { + s.r[3] |= 32; + } + if let Some(Some(at)) = anchor.as_deref_mut() { + *at = self.now(); + } + } + if val & 0xc0 == 0 && old & 0xc0 != 0 { + s.r[18] = 0x10000902; + for p in 0..2 { + s.power(p, false); + } + } + if val & 0xc0 == 0x40 || val & 0xc0 == 0xc0 { + crate::dlog!( + crate::devlog::LogModule::Usb, + "ohci suspend/resume timing unsupported at control register offset 0x04" + ); + } + } + 2 => { + if val & 1 != 0 { + s.soft_reset(); + } else { + s.r[2] |= val & 6; + if val & 8 != 0 { + s.r[3] |= 1 << 30; + } + } + } + 3 => s.r[3] &= !(val & EVENTS), + 4 => s.r[4] |= val & (EVENTS | 0x80000000), + 5 => s.r[4] &= !(val & (EVENTS | 0x80000000)), + 6 => s.r[6] = val & !255, + 7..=11 => s.r[(off / 4) as usize] = val & !15, + 13 => s.r[13] = val & 0xffff3fff, + 16 => s.r[16] = val & 0x3fff, + 17 => s.r[17] = val & 0xfff, + 18 => { + s.r[18] = (s.r[18] & !0xff001b00) | (val & 0xff001b00); + if val & 0x200 != 0 { + for p in 0..2 { + s.power(p, true); + } + } + } + 19 => s.r[19] = val & 0x60006, + 20 => { + if val & (1 << 15) != 0 { + s.r[20] |= 1 << 15; + } + if val & (1 << 31) != 0 { + s.r[20] &= !(1 << 15); + } + if val & (1 << 17) != 0 { + s.r[20] &= !(1 << 17); + } + if s.r[18] & 0x200 == 0 { + for p in 0..2 { + if !s.individual(p) { + if val & 1 != 0 { + s.power(p, false); + } + if val & (1 << 16) != 0 { + s.power(p, true); + } + } + } + } + } + 21 | 22 => { + let p = (off / 4 - 21) as usize; + let i = 21 + p; + s.r[i] &= !(val & CHANGES); + if val & 1 != 0 { + s.r[i] &= !2; + } + if val & 2 != 0 { + if s.r[i] & 1 != 0 && s.powered(p) { + s.r[i] |= 2; + } else { + s.r[i] |= 1 << 16; + } + } + if val & 12 != 0 { + crate::dlog!( + crate::devlog::LogModule::Usb, + "ohci port suspend/resume unsupported at port register offset {:#x}", + 0x54 + p * 4 + ); + } + if val & 16 != 0 { + if s.r[i] & 1 != 0 && s.powered(p) { + s.r[i] |= 16; + s.r[i] &= !2; + s.reset_ms[p] = 10; + reset = s.ports[p].clone(); + } else { + s.r[i] |= 1 << 16; + } + } + if s.individual(p) && s.r[18] & 0x200 == 0 { + if val & PPS != 0 { + s.power(p, true); + } + if val & 0x200 != 0 { + s.power(p, false); + } + } + if s.r[i] & CHANGES != 0 { + s.r[3] |= RHSC; + } + } + _ => { + crate::dlog!( + crate::devlog::LogModule::Usb, + "ohci write to read-only register offset {off:#x}" + ); + } + } + } + drop(anchor); + if let Some(f) = reset { + f.reset(); + } + self.drive_irq(); + } +} +impl Resettable for Ohci { + fn power_on(&self) { + let ports = { + let _pass = self.pass.lock(); + let mut s = self.state.lock(); + let ports = s.ports.clone(); + let low_speed = s.low_speed; + *s = State::new(); + s.ports = ports.clone(); + s.low_speed = low_speed; + self.config.reset(); + ports + }; + for f in ports.into_iter().flatten() { + f.reset(); + } + self.drive_irq(); + } +} +impl Device for Ohci { + fn start(&self) { + if self.running.swap(true, Ordering::AcqRel) { + return; + } + let now = self.now(); + *self.clock.lock() = Some(now); + } + fn stop(&self) { + self.sync_clock(); + self.running.store(false, Ordering::Release); + *self.clock.lock() = None; + drop(self.pass.lock()); + } + fn is_running(&self) -> bool { + self.running.load(Ordering::Acquire) + } + fn step(&self, _: u64) {} + fn get_clock(&self) -> u64 { + 0 + } +} + +#[cfg(test)] +mod register_tests { + use super::*; + use crate::pci::PciBusDevice; + use crate::traits::Resettable; + + #[test] + fn lucent_identity_and_reset_contract() { + let hc = Ohci::new(); + assert_eq!(hc.config().read(0), 0x5802_11c1); + assert_eq!(hc.config().read(2), 0x0c03_1010); + assert_eq!(hc.config().read(11), 0xffff_ffff); + assert_eq!(hc.config().read(1), 0x0200_0000); + assert_eq!(hc.bar_read(0, 0, 4), 0x110); + assert_eq!(hc.bar_read(0, 0x10, 4), 0); + assert_eq!(hc.bar_read(0, 0x34, 4), 0x2edf); + assert_eq!(hc.bar_read(0, 0x48, 4), 0x1000_0902); + assert_eq!(hc.bar_read(0, 0x4c, 4), 0x0006_0000); + for p in [0x54, 0x58] { + assert_eq!(hc.bar_read(0, p, 4), 0); + } + hc.bar_write(0, 0x10, 0xffff_ffff, 4); + hc.bar_write(0, 0x08, 1, 4); + assert_eq!(hc.bar_read(0, 0x10, 4), 0); + assert_eq!(hc.bar_read(0, 0x04, 4) & 0xc0, 0xc0); + hc.power_on(); + assert_eq!(hc.bar_read(0, 0x04, 4), 0); + } +} + +#[cfg(test)] +#[path = "ohci/tests.rs"] +mod tests; diff --git a/src/dev/ohci/engine.rs b/src/dev/ohci/engine.rs new file mode 100644 index 000000000..7b76cc646 --- /dev/null +++ b/src/dev/ohci/engine.rs @@ -0,0 +1,223 @@ +//! General TD packet processing; guest DMA byte slices are the endianness edge. +use super::*; + +fn read_words(dma: &dyn PciDma, at: u32) -> Result<[u32; 4], ()> { + let mut bytes = [0; 16]; + if !dma.dma_read(at as u64, &mut bytes).is_ok() { + return Err(()); + } + Ok(std::array::from_fn(|i| { + u32::from_le_bytes(bytes[i * 4..i * 4 + 4].try_into().unwrap()) + })) +} +fn write_words(dma: &dyn PciDma, at: u32, words: &[u32]) -> Result<(), ()> { + let bytes: Vec = words.iter().flat_map(|x| x.to_le_bytes()).collect(); + if dma.dma_write(at as u64, &bytes).is_ok() { + Ok(()) + } else { + Err(()) + } +} +fn remaining(cbp: u32, be: u32) -> Result { + if cbp == 0 { + return Ok(0); + } + if cbp & !0xfff == be & !0xfff { + be.checked_sub(cbp).map(|n| n as usize + 1).ok_or(()) + } else { + Ok(0x1000 - (cbp as usize & 0xfff) + (be as usize & 0xfff) + 1) + } +} +fn advance(cbp: u32, be: u32, n: usize) -> u32 { + let first = 0x1000 - (cbp as usize & 0xfff); + if n < first { + cbp.wrapping_add(n as u32) + } else { + (be & !0xfff).wrapping_add((n - first) as u32) + } +} +fn payload_read(dma: &dyn PciDma, cbp: u32, be: u32, len: usize) -> Result, ()> { + let mut bytes = vec![0; len]; + let first = len.min(0x1000 - (cbp as usize & 0xfff)); + if first != 0 && !dma.dma_read(cbp as u64, &mut bytes[..first]).is_ok() { + return Err(()); + } + if len > first + && !dma + .dma_read((be & !0xfff) as u64, &mut bytes[first..]) + .is_ok() + { + return Err(()); + } + Ok(bytes) +} +fn payload_write(dma: &dyn PciDma, cbp: u32, be: u32, bytes: &[u8]) -> Result<(), ()> { + let first = bytes.len().min(0x1000 - (cbp as usize & 0xfff)); + if first != 0 && !dma.dma_write(cbp as u64, &bytes[..first]).is_ok() { + return Err(()); + } + if bytes.len() > first && !dma.dma_write((be & !0xfff) as u64, &bytes[first..]).is_ok() { + return Err(()); + } + Ok(()) +} +impl Ohci { + /// Returns the next ED and whether this ED still holds queued general work. + pub(super) fn endpoint( + &self, + s: &mut State, + dma: &dyn PciDma, + at: u32, + ) -> Result<(u32, bool, bool), ()> { + let mut ed = read_words(dma, at)?; + let next = ed[3] & !15; + if ed[0] & (1 << 15) != 0 { + crate::dlog!(crate::devlog::LogModule::Usb, "ohci isochronous ED unsupported at guest address {at:#x}"); + return Ok((next, false, false)); + } + if ed[0] & (1 << 14) != 0 || ed[2] & 1 != 0 || ed[2] & !15 == ed[1] & !15 { + return Ok((next, false, false)); + } + let td_at = ed[2] & !15; + let mut td = read_words(dma, td_at)?; + let mps = ((ed[0] >> 16) & 0x7ff) as usize; + if mps == 0 || mps > 1023 { + crate::dlog!(crate::devlog::LogModule::Usb, "ohci invalid maximum packet size {mps} at ED address {at:#x}"); + return Ok((next, false, false)); + } + let Ok(n) = remaining(td[1], td[3]) else { + crate::dlog!(crate::devlog::LogModule::Usb, "ohci malformed TD buffer range at ED address {at:#x}"); + return Ok((next, false, false)); + }; + let direction = match (ed[0] >> 11) & 3 { + 1 => 1, + 2 => 2, + _ => (td[0] >> 19) & 3, + }; + let pid = match direction { + 0 => Pid::Setup, + 1 => Pid::Out, + 2 => Pid::In, + _ => { + crate::dlog!(crate::devlog::LogModule::Usb, "ohci reserved PID at ED address {at:#x}"); + return Ok((next, false, false)); + } + }; + let length = n.min(mps); + let toggle = if td[0] & (1 << 25) != 0 { + td[0] & (1 << 24) != 0 + } else { + ed[2] & 2 != 0 + }; + let outgoing = if pid != Pid::In { + payload_read(dma, td[1], td[3], length)? + } else { + vec![] + }; + let address = (ed[0] & 127) as u8; + let mut functions = (0..2) + .filter(|&p| s.powered(p) && s.r[21 + p] & 0x16 == 2) + .filter_map(|p| s.ports[p].clone()) + .filter(|f| f.address() == address); + // Multiple matching functions are an unsupported topology. Do not choose + // a recipient by port order or claim to model electrical collisions. + let first = functions.next(); + let function = if functions.next().is_none() { + first + } else { + None + }; + let reply = if let Some(f) = function { + f.transact(UsbPacket { + endpoint: ((ed[0] >> 7) & 15) as u8, + pid, + toggle, + capacity: if pid == Pid::In { mps } else { length }, + data: outgoing, + time_ms: s.now_ms, + }) + } else { + UsbOutcome::NoResponse + }; + let mut retire = false; + let mut cc = 0; + let mut halt = false; + match reply { + UsbOutcome::Nak => return Ok((next, true, true)), + UsbOutcome::Stall => { + cc = 4; + halt = true; + retire = true; + } + UsbOutcome::NoResponse | UsbOutcome::CrcError | UsbOutcome::BitStuffError => { + cc = match reply { + UsbOutcome::CrcError => 1, + UsbOutcome::BitStuffError => 2, + _ => 5, + }; + let ec = ((td[0] >> 26) & 3) + 1; + td[0] = (td[0] & !(3 << 26)) | ((ec.min(3)) << 26); + if ec >= 3 { + retire = true; + halt = true; + } + } + UsbOutcome::Ack(bytes) => { + let actual = if pid == Pid::In { bytes.len() } else { length }; + if pid == Pid::In { + payload_write(dma, td[1], td[3], &bytes[..actual.min(n).min(mps)])?; + } + td[0] = (td[0] & !((3 << 24) | (3 << 26))) | ((2 | u32::from(!toggle)) << 24); + if actual > length { + cc = 8; + halt = true; + retire = true; + } else { + td[1] = if actual == n { + 0 + } else { + advance(td[1], td[3], actual) + }; + if actual < length { + retire = true; + if td[0] & (1 << 18) == 0 { + cc = 9; + halt = true; + } + } else if actual == n { + retire = true; + } + } + } + } + td[0] = (td[0] & 0x0fff_ffff) | (cc << 28); + if retire { + let next_td = td[2] & !15; + td[2] = s.r[12]; + // Failure stops immediately. Earlier successful writes and USB side + // effects remain visible; descriptor writeback is not a transaction. + ed[2] = next_td | ((td[0] >> 23) & 2) | u32::from(halt); + write_words(dma, at + 8, &[ed[2]])?; + write_words(dma, td_at, &td)?; + s.r[12] = td_at; + let di = ((td[0] >> 21) & 7) as u8; + s.done_delay = s.done_delay.min(if cc != 0 { 0 } else { di }); + Ok((next, next_td != ed[1] & !15 && !halt, true)) + } else { + write_words(dma, td_at, &td)?; + Ok((next, true, true)) + } + } + pub(super) fn publish(&self, s: &mut State, dma: &dyn PciDma) -> Result<(), ()> { + if s.done_delay == 0 && s.r[3] & WDH == 0 && s.r[12] != 0 { + let other = u32::from(s.r[3] & s.r[4] & EVENTS & !WDH != 0); + write_words(dma, s.r[6] + 0x84, &[s.r[12] | other])?; + s.r[12] = 0; + s.done_delay = 7; + s.r[3] |= WDH; + } else if s.done_delay > 0 && s.done_delay < 7 { + s.done_delay -= 1; + } + Ok(()) + } +} diff --git a/src/dev/ohci/schedule.rs b/src/dev/ohci/schedule.rs new file mode 100644 index 000000000..b6c8c6255 --- /dev/null +++ b/src/dev/ohci/schedule.rs @@ -0,0 +1,98 @@ +//! Functional scheduling approximation: 256 synthetic units per ED visit. +//! Costs deliberately do not represent wire time. See the module's limitations. +use super::*; +const VISIT: u32 = 256; +impl Ohci { + fn list_ready(s: &mut State, head: usize, current: usize, filled: u32) -> bool { + if s.r[current] == 0 && s.r[2] & filled != 0 { + s.r[current] = s.r[head]; + s.r[2] &= !filled; + } + s.r[current] != 0 + } + fn nonperiodic( + &self, + s: &mut State, + dma: &dyn PciDma, + budget: &mut u32, + cutoff: u32, + credit: &mut u32, + ) -> Result<(), ()> { + let mut yielded: Option = None; + while *budget >= VISIT && *budget > cutoff { + let control = s.r[1] & 16 != 0 && Self::list_ready(s, 8, 9, 2); + let bulk = s.r[1] & 32 != 0 && Self::list_ready(s, 10, 11, 4); + if !control && !bulk { + break; + } + let choose_control = control + && (!bulk + || match yielded { + Some(was_control) => !was_control, + None => *credit > 0, + }); + let (current, filled) = if choose_control { (9, 2) } else { (11, 4) }; + let (next, queued, attempt) = self.endpoint(s, dma, s.r[current])?; + *budget -= VISIT; + s.r[current] = next; + if queued { + s.r[2] |= filled; + } + yielded = if attempt { None } else { Some(choose_control) }; + if attempt { + if choose_control { + *credit = credit.saturating_sub(1); + } else { + *credit = (s.r[1] & 3) + 1; + } + } + } + Ok(()) + } + pub(super) fn schedule(&self, s: &mut State) -> Result<(), ()> { + let old = s.r[15] as u16; + let frame = old.wrapping_add(1); + s.r[15] = u32::from(frame); + if (old ^ frame) & 0x8000 != 0 { + s.r[3] |= 32; + } + s.r[3] |= 4; + let mut budget = (s.active_fi & 0x3fff) + 1; + s.r[14] = (s.active_fi & 0x80000000) | budget.min(0x3fff); + if self.config.command() & 4 == 0 { + return Ok(()); + } + let Some(dma) = self.dma.lock().clone() else { + return Err(()); + }; + let frame_bytes = [frame as u8, (frame >> 8) as u8, 0, 0]; + if !dma.dma_write((s.r[6] + 0x80) as u64, &frame_bytes).is_ok() { + return Err(()); + } + let mut credit = (s.r[1] & 3) + 1; + let cutoff = s.r[16].min(budget); + self.nonperiodic(s, &*dma, &mut budget, cutoff, &mut credit)?; + if s.r[1] & 4 != 0 { + let mut bytes = [0; 4]; + if !dma + .dma_read((s.r[6] + u32::from(frame & 31) * 4) as u64, &mut bytes) + .is_ok() + { + return Err(()); + } + s.r[7] = u32::from_le_bytes(bytes) & !15; + while s.r[7] != 0 && budget >= VISIT { + let (next, _, _) = self.endpoint(s, &*dma, s.r[7])?; + s.r[7] = next; + budget -= VISIT; + } + if s.r[7] != 0 { + s.r[3] |= 1; + s.r[2] = (s.r[2] & !0x30000) | ((s.r[2].wrapping_add(0x10000)) & 0x30000); + } + } + self.nonperiodic(s, &*dma, &mut budget, 0, &mut credit)?; + s.r[14] = (s.active_fi & 0x80000000) | budget.min(0x3fff); + self.publish(s, &*dma) + } +} diff --git a/src/dev/ohci/tests.rs b/src/dev/ohci/tests.rs new file mode 100644 index 000000000..1d0733cd3 --- /dev/null +++ b/src/dev/ohci/tests.rs @@ -0,0 +1,554 @@ +// The register and clock cases follow tests/qtest/usb-lucent-ohci-test.c in +// jamesbraid/qemu-sgi-origami: root-hub-descriptor-a-is-writable and +// frame-number-tracks-clock-while-timer-starved. The descriptor engine cases +// drive a test-only function and have no qtest counterpart. +use super::*; +use crate::pci::{PciBusDevice, PciDmaStatus}; +use crate::traits::{Device, Resettable}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::Weak; + +struct Memory { + bytes: Mutex>, + refuse_read: Mutex>, + refuse_write: Mutex>, + writes: Mutex>, +} +impl Memory { + fn new() -> Arc { + Arc::new(Self { + bytes: Mutex::new(vec![0xa5; 0x10000]), + refuse_read: Mutex::new(None), + refuse_write: Mutex::new(None), + writes: Mutex::new(vec![]), + }) + } + fn words(&self, at: u32, words: &[u32]) { + for (i, w) in words.iter().enumerate() { + self.bytes.lock()[at as usize + i * 4..at as usize + i * 4 + 4] + .copy_from_slice(&w.to_le_bytes()); + } + } + fn word(&self, at: u32) -> u32 { + u32::from_le_bytes( + self.bytes.lock()[at as usize..at as usize + 4] + .try_into() + .unwrap(), + ) + } +} +impl PciDma for Memory { + fn dma_read(&self, addr: u64, dst: &mut [u8]) -> PciDmaStatus { + if self + .refuse_read + .lock() + .is_some_and(|x| (addr..addr + dst.len() as u64).contains(&(x as u64))) + { + return PciDmaStatus::MasterAbort; + } + let b = self.bytes.lock(); + let Some(s) = b.get(addr as usize..addr as usize + dst.len()) else { + return PciDmaStatus::MasterAbort; + }; + dst.copy_from_slice(s); + PciDmaStatus::Ok + } + fn dma_write(&self, addr: u64, src: &[u8]) -> PciDmaStatus { + if self + .refuse_write + .lock() + .is_some_and(|x| (addr..addr + src.len() as u64).contains(&(x as u64))) + { + return PciDmaStatus::MasterAbort; + } + let mut b = self.bytes.lock(); + let Some(s) = b.get_mut(addr as usize..addr as usize + src.len()) else { + return PciDmaStatus::MasterAbort; + }; + s.copy_from_slice(src); + self.writes.lock().push(addr); + PciDmaStatus::Ok + } +} +struct Function { + address: AtomicUsize, + calls: AtomicUsize, + reply: Mutex, + packets: Mutex>, +} +impl Function { + fn new(reply: UsbOutcome) -> Arc { + Arc::new(Self { + address: AtomicUsize::new(0), + calls: AtomicUsize::new(0), + reply: Mutex::new(reply), + packets: Mutex::new(vec![]), + }) + } +} +impl UsbFunction for Function { + fn reset(&self) {} + fn address(&self) -> u8 { + self.address.load(Ordering::SeqCst) as u8 + } + fn transact(&self, packet: UsbPacket) -> UsbOutcome { + self.calls.fetch_add(1, Ordering::SeqCst); + self.packets.lock().push(packet); + self.reply.lock().clone() + } +} +fn start_port_reset(h: &Ohci, port: u32) { + let status = 0x54 + port * 4; + h.bar_write(0, status, 0x100, 4); + h.bar_write(0, status, 0x10, 4); +} + +fn setup( + pid: Pid, + reply: UsbOutcome, + cbp: u32, + be: u32, + flags: u32, +) -> (Arc, Arc, Arc) { + let h = Ohci::new(); + let m = Memory::new(); + let f = Function::new(reply); + h.connect_dma(m.clone()); + h.attach(0, f.clone()).unwrap(); + h.start(); + start_port_reset(&h, 0); + h.advance_frames(10); + m.words(0x1000, &[0; 64]); + m.words(0x2000, &[(8 << 16), 0x3020, 0x3000, 0]); + m.words( + 0x3000, + &[0xf000_0000 | flags | ((pid as u32) << 19), cbp, 0x3020, be], + ); + h.config().write(1, 6); + h.bar_write(0, 0x18, 0x1000, 4); + h.bar_write(0, 0x20, 0x2000, 4); + h.bar_write(0, 0x08, 2, 4); + h.bar_write(0, 0x04, 0x90, 4); + (h, m, f) +} +/// The guest receives fatal status if the controller cannot access a descriptor. +#[test] +fn dma_refusal_preserves_descriptors_and_reports_a_fatal_error() { + for periodic in [false, true] { + for pid in [Pid::Out, Pid::In] { + let (h, m, f) = setup(pid, UsbOutcome::Ack(vec![5; 8]), 0x4000, 0x4007, 0); + if periodic { + m.words(0x1000, &[0x2000; 32]); + h.bar_write(0, 4, 0x84, 4); + } + if pid == Pid::In { + *m.refuse_write.lock() = Some(0x4000); + } else { + *m.refuse_read.lock() = Some(0x4000); + } + let before = m.bytes.lock()[0x3000..0x3010].to_vec(); + h.advance_frames(1); + let frame = h.bar_read(0, 0x3c, 4); + h.advance_frames(3); + assert_eq!(f.calls.load(Ordering::SeqCst), usize::from(pid == Pid::In)); + assert_eq!(&m.bytes.lock()[0x3000..0x3010], &before); + assert_eq!(m.word(0x2008), 0x3000); + assert_eq!(m.word(0x1084), 0); + assert_eq!(h.bar_read(0, 0x3c, 4), frame); + assert_ne!(h.bar_read(0, 0xc, 4) & UE, 0); + assert_eq!(h.config().read(1) >> 16, 0x8200); + h.config().write(1, 0x8000_0006); + assert_eq!(h.config().read(1) >> 16, 0x0200); + assert_ne!(h.bar_read(0, 0xc, 4) & UE, 0); + } + } +} + +/// Short and overlong USB replies leave bytes outside the packet untouched. +#[test] +fn short_or_overlong_in_packets_preserve_unwritten_bytes() { + for (name, payload, end, flags, bytes, current, condition) in [ + ("short", vec![1, 2, 3], 0x401f, (1 << 18) | (3 << 24), [1, 2, 3, 0xa5, 0xa5], 0x4003, 0), + ("overrun", vec![1; 8], 0x4003, 3 << 24, [1, 1, 1, 1, 0xa5], 0x4000, 8), + ] { + let (h, m, _) = setup(Pid::In, UsbOutcome::Ack(payload), 0x4000, end, flags); + h.advance_frames(1); + assert_eq!(&m.bytes.lock()[0x4000..0x4005], &bytes, "{name}"); + assert_eq!(m.word(0x3004), current, "{name}"); + assert_eq!(m.word(0x3000) >> 28, condition, "{name}"); + assert_eq!((m.word(0x3000) >> 24) & 3, 2, "{name}"); + if condition == 0 { + assert_eq!(m.word(0x2008) & !15, 0x3020); + assert_eq!(m.word(0x1084) & !15, 0x3000); + } + } +} + +/// The guest retries NAKs, stops on STALL, and retains consistent short-transfer state. +#[test] +fn nak_stall_and_short_in_completions_keep_the_endpoint_consistent() { + for (name, reply, flags, condition) in [ + ("NAK", UsbOutcome::Nak, 0, 15), + ("STALL", UsbOutcome::Stall, 3 << 24, 4), + ("no response", UsbOutcome::NoResponse, 0, 5), + ("short without rounding", UsbOutcome::Ack(vec![2]), 0, 9), + ] { + let (h, m, f) = setup(Pid::In, reply, 0x4000, 0x4007, flags); + h.bar_write(0, 0x34, 255, 4); + let before = m.bytes.lock()[0x3000..0x3010].to_vec(); + h.advance_frames(1); + if condition == 5 { + assert_eq!(f.calls.load(Ordering::SeqCst), 1); + assert_eq!((m.word(0x3000) >> 26) & 3, 1); + assert_eq!(m.word(0x2008), 0x3000); + h.advance_frames(2); + } + assert_eq!(m.word(0x3000) >> 28, condition, "{name}"); + if condition == 15 { + assert_eq!(&m.bytes.lock()[0x3000..0x3010], &before); + assert_eq!(m.word(0x2008), 0x3000); + } else { + assert_ne!(m.word(0x2008) & 1, 0, "{name}"); + } + if condition == 4 { + assert_eq!(m.word(0x3004), 0x4000); + assert_eq!((m.word(0x3000) >> 24) & 3, 3); + assert_eq!(m.word(0x2008) & 3, 3); + } + } +} + +/// Port and controller reset paths retain attached USB functions. +#[test] +fn attached_functions_survive_port_and_controller_resets() { + let h = Ohci::new(); + h.start(); + h.attach(1, Function::new(UsbOutcome::Nak)).unwrap(); + assert_eq!(h.bar_read(0, 0x58, 4) & 1, 0); + start_port_reset(&h, 1); + h.advance_frames(9); + assert_ne!(h.bar_read(0, 0x58, 4) & 16, 0); + h.advance_frames(1); + assert_eq!(h.bar_read(0, 0x58, 4) & 16, 0); + assert_ne!(h.bar_read(0, 0x58, 4) & (1 << 20), 0); + h.bar_write(0, 8, 1, 4); + assert_ne!(h.bar_read(0, 0x58, 4) & 1, 0); + h.bar_write(0, 0x58, 0x200, 4); + assert_eq!(h.bar_read(0, 0x58, 4) & 0x117, 0); + h.bar_write(0, 0x58, PPS, 4); + assert_ne!(h.bar_read(0, 0x58, 4) & 1, 0); + h.power_on(); + assert_eq!(h.bar_read(0, 0x58, 4), 0); + h.bar_write(0, 0x58, PPS, 4); + assert_ne!(h.bar_read(0, 0x58, 4) & 1, 0); +} + +/// The guest observes descriptor updates before the done bit and acknowledges them later. +#[test] +fn done_publication_orders_writes_and_waits_for_acknowledgement() { + let (h, m, _) = setup(Pid::In, UsbOutcome::Ack(vec![8; 8]), 0x4000, 0x4007, 0); + h.advance_frames(1); + { + let writes = m.writes.lock(); + assert!(writes.iter().position(|a| *a == 0x2008).unwrap() < writes.iter().position(|a| *a == 0x3000).unwrap()); + } + assert_eq!(m.word(0x1084) & !15, 0x3000); + m.words(0x3020, &[0xf0000000 | (2 << 19), 0x4100, 0x3040, 0x4107]); + m.words(0x2004, &[0x3040]); + h.bar_write(0, 8, 2, 4); + h.advance_frames(1); + assert_eq!(m.word(0x1084) & !15, 0x3000); + assert_eq!(h.bar_read(0, 0x30, 4), 0x3020); + h.bar_write(0, 0xc, 2, 4); + h.advance_frames(1); + assert_eq!(m.word(0x1084) & !15, 0x3020); + for delay in [0, 2, 7] { + let (h, m, _) = setup(Pid::In, UsbOutcome::Ack(vec![5; 8]), 0x4000, 0x4007, delay << 21); + h.advance_frames(1); + if delay != 0 { assert_eq!(m.word(0x1084), 0); } + h.advance_frames(u64::from(delay)); + assert_eq!(m.word(0x1084) & !15, if delay == 7 { 0 } else { 0x3000 }); + } + let (h, m, _) = setup(Pid::In, UsbOutcome::Ack(vec![1; 8]), 0x4000, 0x4007, 0); + *m.refuse_write.lock() = Some(0x1084); + h.advance_frames(1); + assert_ne!(h.bar_read(0, 0xc, 4) & UE, 0); + assert_eq!(h.bar_read(0, 0xc, 4) & WDH, 0); + assert_eq!(m.word(0x1084), 0); +} + +/// OHCI must yield to the other transfer list when a guest marks one list skipped. +#[test] +fn scheduler_shares_service_and_yields_past_skipped_lists() { + for (cbsr, skip_control, skip_bulk) in [(0, false, false), (3, false, false), (3, true, false), (0, false, true)] { + let (h, m, f) = setup(Pid::In, UsbOutcome::Nak, 0x4000, 0x4007, 0); + m.words(0x2000, &[(8 << 16) | (1 << 7) | (u32::from(skip_control) << 14), 0x3020, 0x3000, if skip_control { 0x2000 } else { 0 }]); + m.words(0x2100, &[(8 << 16) | (2 << 7) | (u32::from(skip_bulk) << 14), 0x3120, 0x3100, if skip_bulk { 0x2100 } else { 0 }]); + m.words(0x3100, &[0xf0100000, 0x4100, 0x3120, 0x4107]); + h.bar_write(0, 0x28, 0x2100, 4); + h.bar_write(0, 8, 6, 4); + h.bar_write(0, 4, 0xb0 | cbsr, 4); + h.advance_frames(1); + let packets = f.packets.lock(); + let control = packets.iter().filter(|p| p.endpoint == 1).count(); + let bulk = packets.iter().filter(|p| p.endpoint == 2).count(); + assert_eq!(control > 0, !skip_control); + assert_eq!(bulk > 0, !skip_bulk); + if skip_bulk { + assert!(control > 1, "control resumes after its credit is exhausted"); + } + if !skip_control && !skip_bulk { + assert!(control >= bulk * (cbsr as usize + 1)); + assert!(control <= bulk * (cbsr as usize + 1) + cbsr as usize + 1); + } + assert_eq!(h.bar_read(0, 0xc, 4) & UE, 0); + } +} + +/// The controller routes address zero only while one attached function can claim it. +#[test] +fn address_zero_is_routed_only_to_a_unique_eligible_function() { + for (name, address, peer_status, first_selected, peer_selected) in [ + ("ambiguous", 0, PPS | 1 | 2, false, false), + ("disabled peer", 0, PPS | 1, true, false), + ("assigned first", 5, PPS | 1 | 2, false, true), + ] { + let (h, m, f) = setup(Pid::In, UsbOutcome::Nak, 0x4000, 0x4007, 0); + let peer = Function::new(UsbOutcome::Nak); + h.attach(1, peer.clone()).unwrap(); + h.state.lock().r[22] = peer_status; + f.address.store(address, Ordering::SeqCst); + h.advance_frames(1); + assert_eq!(f.calls.load(Ordering::SeqCst) > 0, first_selected, "{name}"); + assert_eq!(peer.calls.load(Ordering::SeqCst) > 0, peer_selected, "{name}"); + if address != 0 { + let peer_calls = peer.calls.load(Ordering::SeqCst); + m.words(0x2000, &[(8 << 16) | 5]); + h.advance_frames(1); + assert!(f.calls.load(Ordering::SeqCst) > 0); + assert_eq!(peer.calls.load(Ordering::SeqCst), peer_calls); + } else if !first_selected { + assert_eq!(m.word(0x3000) >> 28, 5); + assert_eq!((m.word(0x3000) >> 26) & 3, 3); + } + } +} + + +/// A guest USB packet can cross physically nonadjacent DMA pages. +#[test] +fn packet_crosses_nonadjacent_pages() { + let (h, m, _) = setup( + Pid::In, + UsbOutcome::Ack(vec![1, 2, 3, 4, 5, 6, 7, 8]), + 0x4ffc, + 0x9003, + 0, + ); + h.advance_frames(1); + assert_eq!(&m.bytes.lock()[0x4ffc..0x5000], &[1, 2, 3, 4]); + assert_eq!(&m.bytes.lock()[0x9000..0x9004], &[5, 6, 7, 8]); + assert_eq!(m.bytes.lock()[0x5000], 0xa5); + assert_eq!(m.word(0x3004), 0); +} + +/// The frame counter catches up after its host timer was delayed. +#[test] +fn clock_read_catches_up_after_timer_starvation() { + let h = Ohci::new(); + let now = Arc::new(AtomicUsize::new(0)); + let t = now.clone(); + h.set_clock(Arc::new(move || t.load(Ordering::Relaxed) as u64)); + h.start(); + h.bar_write(0, 4, 0x80, 4); + now.store(5_000_000, Ordering::Relaxed); + assert!(h.bar_read(0, 0x3c, 4) >= 5); + h.stop(); + let frame = h.bar_read(0, 0x3c, 4); + h.advance_frames(10); + assert_eq!(h.bar_read(0, 0x3c, 4), frame); +} + +/// The frame count follows the guest's clock, not when the machine's poll or +/// a timer thread last ran: a guest that waits 1 ms after going operational +/// and reads HcFmNumber sees the frame, even if nothing else advanced the +/// controller in between (a slow or descheduled host thread). +#[test] +fn frame_number_follows_the_clock_with_no_poll_or_timer() { + let h = Ohci::new(); + let now = Arc::new(AtomicUsize::new(10_000_000)); + let t = now.clone(); + h.set_clock(Arc::new(move || t.load(Ordering::Relaxed) as u64)); + h.start(); + // Part-way into a frame period when the guest sets operational. + now.store(10_400_000, Ordering::Relaxed); + h.bar_write(0, 4, 0x80, 4); + // Entering Operational increments the frame number at once. + assert_eq!(h.bar_read(0, 0x3c, 4), 1); + // 1.1 ms on the guest's clock: the first boundary, 1 ms after the write, has passed. + now.store(11_500_000, Ordering::Relaxed); + assert_eq!(h.bar_read(0, 0x3c, 4), 2); + // A long guest-time gap is delivered in full at the next access. + now.store(31_500_000, Ordering::Relaxed); + assert_eq!(h.bar_read(0, 0x3c, 4), 22); + // Time that has not passed on the guest's clock delivers nothing. + assert_eq!(h.bar_read(0, 0x3c, 4), 22); + h.stop(); +} + +fn fake_clock_ohci(start_ns: usize) -> (Arc, Arc) { + let h = Ohci::new(); + let now = Arc::new(AtomicUsize::new(start_ns)); + let t = now.clone(); + h.set_clock(Arc::new(move || t.load(Ordering::Relaxed) as u64)); + h.start(); + (h, now) +} + +/// OpenHCI 1.0a sections 6.2.1 and 7.3.3: entering UsbOperational loads +/// HcFmRemaining from the frame interval, increments HcFmNumber, and sends the +/// first SOF one frame later, whatever the phase of the free-running clock. +fn operational_entry_restarts_the_frame(from_hcfs: Option) { + let (h, now) = fake_clock_ohci(10_000_000); + let m = Memory::new(); + h.connect_dma(m.clone()); + h.config().write(1, 6); + h.bar_write(0, 0x18, 0x1000, 4); + // Not operational: no frames count however much time passes. + if let Some(hcfs) = from_hcfs { + h.bar_write(0, 4, hcfs, 4); + } + now.store(15_000_000, Ordering::Relaxed); + assert_eq!(h.bar_read(0, 0x3c, 4), 0); + now.store(15_400_000, Ordering::Relaxed); + h.bar_write(0, 4, 0x80, 4); + assert_eq!(h.bar_read(0, 0x3c, 4), 1); + assert_eq!(h.bar_read(0, 0x38, 4), 0x2edf); + assert_eq!(h.bar_read(0, 0x0c, 4) & 4, 0, "no SOF at the load"); + assert_eq!(m.bytes.lock()[0x1080], 0xa5, "no HCCA write at the load"); + // Just short of one frame after the write: still nothing. + now.store(16_399_999, Ordering::Relaxed); + assert_eq!(h.bar_read(0, 0x3c, 4), 1); + assert_eq!(h.bar_read(0, 0x0c, 4) & 4, 0); + assert_eq!(m.bytes.lock()[0x1080], 0xa5); + // Exactly one frame after: the first SOF and its HCCA frame number. + now.store(16_400_000, Ordering::Relaxed); + assert_eq!(h.bar_read(0, 0x0c, 4) & 4, 4); + assert_eq!(h.bar_read(0, 0x3c, 4), 2); + assert_eq!(m.bytes.lock()[0x1080], 2); + h.stop(); +} + +#[test] +fn operational_entry_from_reset_restarts_the_frame() { + operational_entry_restarts_the_frame(None); +} + +#[test] +fn operational_entry_from_suspend_restarts_the_frame() { + // HCFS 11b. + operational_entry_restarts_the_frame(Some(0xc0)); +} + +#[test] +fn operational_entry_from_resume_restarts_the_frame() { + // HCFS 01b. + operational_entry_restarts_the_frame(Some(0x40)); +} + +/// The increment on entry carries into the frame-number overflow flag like a +/// frame's own, and HcFmRemaining takes the frame-interval toggle bit too. +#[test] +fn operational_entry_sets_overflow_and_loads_the_interval_toggle() { + for (before, after, overflow) in [(0x7fff, 0x8000, true), (0xffff, 0, true), (0x1234, 0x1235, false)] { + let (h, _now) = fake_clock_ohci(10_000_000); + h.state.lock().r[15] = before; + h.bar_write(0, 0x34, 0x8000_2edf, 4); + h.bar_write(0, 4, 0x80, 4); + assert_eq!(h.bar_read(0, 0x3c, 4), after, "from {before:#x}"); + assert_eq!(h.bar_read(0, 0x0c, 4) & 32 != 0, overflow, "from {before:#x}"); + assert_eq!(h.bar_read(0, 0x38, 4), 0x8000_2edf); + h.stop(); + } +} + +/// An interrupt handler can acknowledge a line while the controller raises it. +#[test] +fn reentrant_irq_acknowledgement_leaves_final_line_clear() { + struct Ack { + hc: Weak, + level: AtomicBool, + once: AtomicBool, + } + impl crate::irq::InterruptController for Ack { + fn set_line(&self, _: u32, level: bool) { + if level && !self.once.swap(true, Ordering::SeqCst) { + self.hc.upgrade().unwrap().bar_write(0, 0xc, RHSC, 4); + } + self.level.store(level, Ordering::SeqCst); + } + } + let h = Ohci::new(); + let ack = Arc::new(Ack { + hc: Arc::downgrade(&h), + level: AtomicBool::new(false), + once: AtomicBool::new(false), + }); + h.connect_interrupt(IrqLine::new(ack.clone(), 0)); + h.bar_write(0, 0x10, 0x80000040, 4); + h.attach(0, Function::new(UsbOutcome::Nak)).unwrap(); + assert!(!ack.level.load(Ordering::SeqCst)); +} + +/// The guest sees the documented write masks, HCR exemptions, and ownership bit. +#[test] +fn register_masks_preserve_hcr_exempt_fields_and_ownership_status() { + let h = Ohci::new(); + h.bar_write(0, 0x18, 0x123456ff, 4); + assert_eq!(h.bar_read(0, 0x18, 4), 0x12345600); + h.bar_write(0, 0x20, 0x123456ff, 4); + assert_eq!(h.bar_read(0, 0x20, 4), 0x123456f0); + h.bar_write(0, 0x10, u32::MAX, 4); + assert_eq!(h.bar_read(0, 0x10, 4), 0xc000007f); + h.bar_write(0, 8, 8, 4); + assert_eq!(h.bar_read(0, 0xc, 4), 1 << 30); + h.bar_write(0, 0xc, 1 << 31, 4); + assert_eq!(h.bar_read(0, 0xc, 4), 1 << 30); + h.bar_write(0, 0xc, 1 << 30, 4); + assert_eq!(h.bar_read(0, 0xc, 4), 0); + h.bar_write(0, 0x48, u32::MAX, 4); + assert_eq!(h.bar_read(0, 0x48, 4), 0xff001b02); + h.bar_write(0, 4, 0x700, 4); + h.bar_write(0, 8, 1, 4); + assert_eq!(h.bar_read(0, 4, 4), 0x3c0); + assert_eq!(h.bar_read(0, 0x48, 4), 0xff001b02); + assert_eq!(h.bar_read(0, 0x10, 4), 0); + h.bar_write(0, 0x3c, 0xffff, 4); + assert_eq!(h.bar_read(0, 0x3c, 4), 0); + h.bar_write(0, 0x18, 1, 1); + assert_eq!(h.bar_read(0, 0x18, 4), 0); +} + +/// The periodic phase controls how the guest control and bulk lists share service. +#[test] +fn periodic_phase_preserves_control_and_bulk_credit() { + let (h, m, f) = setup(Pid::In, UsbOutcome::Nak, 0x4000, 0x4007, 0); + m.words(0x2000, &[(8 << 16) | (1 << 7), 0x3020, 0x3000, 0]); + m.words(0x2100, &[(8 << 16) | (2 << 7), 0x3120, 0x3100, 0]); + m.words(0x3100, &[0xf0100000, 0x4100, 0x3120, 0x4107]); + m.words(0x2200, &[(8 << 16) | (3 << 7), 0x3220, 0x3200, 0]); + m.words(0x3200, &[0xf0100000, 0x4200, 0x3220, 0x4207]); + for i in 0..32 { + m.words(0x1000 + i * 4, &[0x2200]); + } + let mut s = h.state.lock().clone(); + s.active_fi = 5 * 256 - 1; + s.r[16] = 3 * 256; + s.r[1] = 0xb5; + s.r[10] = 0x2100; + s.r[2] |= 4; + h.schedule(&mut s).unwrap(); + let order: Vec<_> = f.packets.lock().iter().map(|p| p.endpoint).collect(); + assert_eq!(order, [1, 1, 3, 2, 1]); + assert_eq!(s.r[14], 0); +} diff --git a/src/dev/pic.rs b/src/dev/pic.rs new file mode 100644 index 000000000..4716f74e7 --- /dev/null +++ b/src/dev/pic.rs @@ -0,0 +1,1314 @@ +//! The PIC, the PCI host bridge on the interface board of the Origin 350 and +//! Tezro (IP53 family). One chip sits in one XIO widget and carries two +//! independent bridge heads, at widget offset 0 and `0x800000`, each with its +//! own register block and PCI bus. Widget 15 holds the first PIC; the +//! two- and four-processor Tezro board adds a second at widget 11. +//! +//! A [`Pic`] is self-contained: the machine builds one per widget, hands the +//! processor's view of it to Bedrock IO ([`Pic::widget`]), gives it a +//! [`PicPeer`] for the packets it originates (interrupts and DMA), connects +//! the Bedrock PIs it raises interrupts on, and attaches PCI functions to the +//! bus slots. The bus slots are hooks only: IOC4 at device 1 (PIC input 0), +//! the ISP12160 at device 3 (input 2) and the BCM5701 at device 4 (input 3) +//! are the board's choice, made by the machine. +//! +//! Widget 0 of the board is the PXBOW of the PIC that carries IO9 +//! ([`Pic::enable_pxbow`]); the other PIC has none of its own and is +//! attached behind that one ([`Pxbow::attach_endpoint`]). +//! +//! The register names are the ones the Bridge family shares +//! ([`xbridge::reg`](super::xbridge::reg)); the PIC leaves out the ATE, the +//! alias windows and the second and third write-request banks' meaning. +//! +//! # What is approximated here +//! +//! The model is the compatibility profile QEMU selects for Tezro: revision 3 +//! with identity bit 0 set, `DIR_MAP` bits 0 to 16 and 20 to 23, the control +//! and Device masks, and the reset values below are emulator choices that let +//! the PROM and IRIX run, not measured reset state. The CPU word view (a +//! 32-bit access to the low half of an eight-byte slot) and the processor's +//! byte-lane placement for PCI cycles are inferred from the public drivers. +//! The one-millisecond reassertion of a still-high PCI input after the +//! processor clears its interrupt bit is the Bedrock PI's +//! ([`BedrockPi::set_level_source`](super::bedrock_pi::BedrockPi::set_level_source)); +//! this chip tells it which inputs are active. Translated (ATE) DMA, PCI-X +//! errors and the bridge's own PCI function are unsupported and observable +//! ([`Pic::unsupported`]). The packet face a peer could read the heads +//! through has no user and is not modeled. All transfers are synchronous. +//! +//! A configuration read that no function claims is a PCI master timeout: +//! the processor gets a bus error back through Bedrock, the error view +//! records bit 11 whatever the enable, and interrupt status records it only +//! while enable bit 11 is set (the gating is inferred). Status clears one +//! bit at a time; any reset-register write clears the view. Device 0, which +//! the PIC keeps for itself, reads as unclaimed: an approximation, since +//! what the chip answers there is unknown. Which bit a bus in PCI-X mode +//! sets (11 or 34) is unknown; the model sets 11. An unclaimed +//! configuration write goes nowhere and records nothing. +//! +//! Ported from hw/pci-host/sgi-pic.c in jamesbraid/qemu-sgi-origami. + +use super::bedrock_pi::BedrockPi; +use super::heart::Clock; +use super::pxbow::{AttachError, Pxbow}; +use super::xbridge::reg; +use crate::devlog::LogModule; +use crate::irq::{InterruptController, IrqLine}; +use crate::pci::{PciBusDevice, PciDma, PciDmaStatus, Space}; +use crate::traits::{BusDevice, BusRead16, BusRead32, BusRead64, BusRead8, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::{Mutex, RwLock}; +use std::sync::atomic::{AtomicUsize, Ordering}; +use std::sync::{Arc, OnceLock, Weak}; + +/// Where the second head sits in the widget. +pub const HEAD_OFFSET: u64 = 0x80_0000; +/// The widget's address space. +pub const WIDGET_SIZE: u64 = 1 << 32; +/// Interrupt inputs per head. +pub const PINS: usize = 8; +/// PCI devices behind a head: configuration devices 1 to 4. Device 0 is the +/// bridge itself, and no PCI identity for it is established. +pub const DEVICES: u8 = 4; + +const REVISION: u32 = 3; +const MANUFACTURER: u32 = 0x24; +const PARTS: [u32; 2] = [0xd102, 0xd112]; + +const REGISTER_BYTES: usize = 0x640; +const REQUEST_BANKS: usize = 3; +const REQUEST_ENTRIES: usize = 256; +const REQUEST_BASE: usize = 0x18000; +const REQUEST_END: usize = 0x19800; +const REQUEST_BANK_SIZE: usize = 0x800; +const CONFIG_BASE: u64 = 0x20000; +const CONFIG_TYPE1: u64 = 0x28000; +const CONFIG_END: u64 = 0x29000; +const PCI_WINDOW: [u64; 2] = [0x4000_0000, 0xc000_0000]; +const PCI_WINDOW_SIZE: u64 = 1 << 30; +const SHORT_BASE: u64 = 0x20_0000; +/// The packet window of the 48-bit XIO address: bits 63:60 of a direct64 DMA +/// address name the destination widget, and a payload keeps 47 bits. +const PAYLOAD_BITS: u32 = 48; +const UNSUPPORTED_LIMIT: usize = 32; + +/// The widget flush register (read to order earlier writes). +const FLUSH: usize = 0x50; +/// The link-side error address slot, whose low-word view the PROM writes. +const ERROR_ADDRESS_WORD: u64 = 0x7c; + +const CONTROL_MASK: u64 = 0x0f00_f20f; +const CONTROL_RESET: u64 = 0x0f00_320f; +const REQUEST_TIMEOUT_RESET: u64 = 0xfffff; +const DIRECT_MAP_SUPPORTED: u64 = 0x1ffff | (0xf << 20); +const DEVICE_MASK: u64 = 0x1858_3fff; +const DEVICE_RESET: [u64; DEVICES as usize] = [0x1800_1000, 0x1800_1002, 0x1800_1004, 0x1800_1005]; +const DEVICE_SWAP: u64 = 1 << 19; +const DEVICE_STREAM: u64 = 1 << 13; +const DEVICE_WIDE: u64 = 1 << 22; +const DEVICE_MEMORY: u64 = 1 << 12; +const INVALID_ADDRESS: u64 = 1 << 24; +/// PCI master timeout: a cycle no device claimed. The same bit in the +/// interrupt status, enable and error view registers. +const MASTER_TIMEOUT: u64 = 1 << 11; +/// The error view: the error bits, whether or not they are enabled. +/// Read-only; any write to the reset register clears it. +const ERROR_VIEW: usize = reg::INTERRUPT_MULTIPLE; +const SET: u64 = 0x100; +const LEVELS: std::ops::RangeInclusive = 7..=127; + +/// The widget identity word of a head: revision, part and manufacturer +/// packed as the common widget header lays them out, with bit 0 set. +pub const fn identity(head: usize) -> u32 { + (REVISION << 28) | (PARTS[head] << 12) | (MANUFACTURER << 1) | 1 +} + +/// A configuration cycle no device claimed: a PCI master timeout. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +struct Unclaimed; + +/// What the machine's side of the XIO link refused: no target, a permission +/// check, or an address with nothing behind it. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct Refused; + +/// The far end of the link: where the packets this chip originates go. On +/// Tezro that is Bedrock IO. Bytes are in address order, as the XIO packet +/// carries them. +pub trait PicPeer: Send + Sync { + /// An interrupt packet: `payload` is the vector, with bit 8 for assert, + /// written to the 48-bit destination `address`. + fn interrupt(&self, source_widget: u8, address: u64, payload: u64) -> Result<(), Refused>; + + /// A DMA read of `data.len()` bytes at the 48-bit `address` of `widget`. + fn dma_read(&self, source_widget: u8, widget: u8, address: u64, data: &mut [u8]) -> Result<(), Refused>; + + fn dma_write(&self, source_widget: u8, widget: u8, address: u64, data: &[u8]) -> Result<(), Refused>; +} + +/// How the board uses a PCI function's slot. +#[derive(Clone, Copy, Debug, Default)] +pub struct Function { + /// The function's registers keep the processor's byte sequence (IOC4), + /// so the bridge leaves the address and value of a processor access + /// alone instead of placing the bytes for a little-endian adapter. + pub self_ordered_pio: bool, + /// The slot drives REQ64# and the function takes 64-bit data phases: + /// its swap-clear direct64 DMA reverses each doubleword, not each word. + pub req64: bool, +} + +/// The peer was already connected. +#[derive(Debug, PartialEq, Eq)] +pub struct AlreadyConnected; + +/// Wiring set once before the machine runs. +struct Wiring { + peer: OnceLock>, + pi: [OnceLock>; 2], + /// The first PI level source this chip owns; head h, pin p is + /// `source_base + h * PINS + p`. + source_base: AtomicUsize, +} + +struct State { + reg: [u64; REGISTER_BYTES / 8], + request: [[u64; REQUEST_ENTRIES]; REQUEST_BANKS], + pins: u8, + set_requests: [u32; PINS], + clear_requests: [u32; PINS], +} + +impl State { + fn new() -> Self { + let mut s = Self { + reg: [0; REGISTER_BYTES / 8], + request: [[0; REQUEST_ENTRIES]; REQUEST_BANKS], + pins: 0, + set_requests: [0; PINS], + clear_requests: [0; PINS], + }; + s.reset(); + s + } + + fn r(&self, off: usize) -> u64 { + self.reg[off / 8] + } + + fn set(&mut self, off: usize, value: u64) { + self.reg[off / 8] = value; + } + + fn reset(&mut self) { + *self = Self { + reg: [0; REGISTER_BYTES / 8], + request: [[0; REQUEST_ENTRIES]; REQUEST_BANKS], + pins: 0, + set_requests: [0; PINS], + clear_requests: [0; PINS], + }; + self.set(reg::REQUEST_TIMEOUT, REQUEST_TIMEOUT_RESET); + self.set(reg::CONTROL, CONTROL_RESET); + for (i, v) in DEVICE_RESET.iter().enumerate() { + self.set(reg::DEVICE_BASE + i * 8, *v); + } + } +} + +struct Slot { + function: Arc, + options: Function, +} + +/// When each BAR window was last put on the bus. PCI space is a set of +/// overlapping mappings in which the one mapped most recently answers, so a +/// window that is enabled or moved takes its range from whatever else decodes +/// it. +#[derive(Default)] +struct Mappings { + counter: u64, + /// (device slot, space, BAR, base, size, mapping order) + table: Vec<(usize, Space, usize, u32, u32, u64)>, +} + +/// A PCI cycle's responder: the slot and the offset within one of its BARs. +struct Claim { + function: Arc, + options: Function, + bar: usize, + offset: u32, +} + +fn short_window(addr: u64) -> Option<(usize, u64, u64)> { + match addr { + 0x20_0000..=0x3f_ffff => Some((0, addr - 0x20_0000, 0x20_0000)), + 0x40_0000..=0x5f_ffff => Some((1, addr - 0x40_0000, 0x20_0000)), + 0x60_0000..=0x6f_ffff => Some((2, addr - 0x60_0000, 0x10_0000)), + 0x70_0000..=0x7f_ffff => Some((3, addr - 0x70_0000, 0x10_0000)), + _ => None, + } +} + +fn defined(off: usize) -> bool { + matches!( + off, + reg::IDENTIFICATION + | reg::CONTROL + | reg::REQUEST_TIMEOUT + | reg::INT_DEST_UPPER + | reg::INT_DEST_LOWER + | FLUSH + | reg::DIRECT_MAP + | reg::PCI_TYPE1_CONFIG + | reg::INTERRUPT_STATUS + | reg::INTERRUPT_ENABLE + | reg::INTERRUPT_RESET + | reg::INTERRUPT_MODE + | reg::INTERRUPT_DEVICE + | reg::INTERRUPT_ERROR_HOST + | reg::INTERRUPT_HOST_BASE..=reg::INTERRUPT_HOST_LAST + | ERROR_VIEW + | reg::DEVICE_BASE..=0x218 + ) +} + +fn request_slot(off: usize) -> (usize, usize) { + let rel = off - REQUEST_BASE; + (rel / REQUEST_BANK_SIZE, (rel % REQUEST_BANK_SIZE) / 8) +} + +/// The largest naturally aligned access of at most eight bytes that starts +/// at `address` and fits in `remaining`. +fn natural(address: u64, remaining: usize) -> usize { + let mut n = 8; + while n > remaining || !address.is_multiple_of(n as u64) { + n /= 2; + } + n +} + +fn le_bytes(size: usize, value: u64) -> [u8; 8] { + let mut b = [0; 8]; + b[..size].copy_from_slice(&value.to_le_bytes()[..size]); + b +} + +fn be_bytes(size: usize, value: u64) -> [u8; 8] { + let mut b = [0; 8]; + b[..size].copy_from_slice(&value.to_be_bytes()[8 - size..]); + b +} + +fn le_value(bytes: &[u8]) -> u64 { + bytes.iter().rev().fold(0, |v, &b| (v << 8) | b as u64) +} + +fn be_value(bytes: &[u8]) -> u64 { + bytes.iter().fold(0, |v, &b| (v << 8) | b as u64) +} + +/// One bridge head: registers, one PCI bus, and its interrupt inputs. +// Lock order: `order`, then `st`. Packets go to the peer and the PIs with +// `st` dropped; `order` keeps two threads' packets in the order decided. +struct Head { + index: usize, + wiring: Arc, + st: Mutex, + order: Mutex<()>, + slots: RwLock<[Option; DEVICES as usize]>, + pci_outputs: RwLock<[Option; 32]>, + mappings: Mutex, + unsupported: Mutex>, +} + +struct Pin { + head: Weak, + pci: bool, +} + +impl InterruptController for Pin { + fn set_line(&self, line: u32, level: bool) { + let Some(head) = self.head.upgrade() else { return }; + if !self.pci { + head.pin(line as usize, level); + return; + } + let output = head.pci_outputs.read().get(line as usize).and_then(Clone::clone); + match output { + Some(output) => output.set(level), + None if level => { + crate::dlog!(LogModule::Bedrock, "pic: head {} PCI interrupt source {line} has no board wiring", head.index) + } + None => {} + } + } +} + +struct SlotDma { + head: Weak, + device: u8, +} + +impl PciDma for SlotDma { + fn dma_read(&self, address: u64, buf: &mut [u8]) -> PciDmaStatus { + match self.head.upgrade() { + Some(head) => head.dma(self.device, address, buf, false), + None => { + buf.fill(0); + PciDmaStatus::MasterAbort + } + } + } + + fn dma_write(&self, address: u64, data: &[u8]) -> PciDmaStatus { + match self.head.upgrade() { + Some(head) => head.dma(self.device, address, &mut data.to_vec(), true), + None => PciDmaStatus::MasterAbort, + } + } +} + +impl Head { + fn new(index: usize, wiring: Arc) -> Arc { + Arc::new(Self { + index, + wiring, + st: Mutex::new(State::new()), + order: Mutex::new(()), + slots: RwLock::new(std::array::from_fn(|_| None)), + pci_outputs: RwLock::new(std::array::from_fn(|_| None)), + mappings: Mutex::new(Mappings::default()), + unsupported: Mutex::new(Vec::new()), + }) + } + + fn note_unsupported(&self, addr: u64, size: usize, write: bool, value: u64) { + let mut seen = self.unsupported.lock(); + if seen.len() >= UNSUPPORTED_LIMIT { + return; + } + seen.push((self.index, write, addr, size)); + crate::dlog!( + LogModule::Bedrock, + "pic: head {} {} offset {addr:#x} size {size} value {value:#x} unsupported{}", + self.index, + if write { "write" } else { "read" }, + if seen.len() == UNSUPPORTED_LIMIT { "; further accesses suppressed" } else { "" } + ); + } + + fn source_widget(&self) -> u8 { + (self.st.lock().r(reg::CONTROL) & 0xf) as u8 + } + + // ---- interrupts ---- + + /// Send pin `pin`'s set or clear packet. True when the peer took it. + fn packet(&self, pin: usize, set: bool) -> bool { + let (destination, source) = { + let mut st = self.st.lock(); + if set { + st.set_requests[pin] += 1; + } else { + st.clear_requests[pin] += 1; + } + (st.r(reg::INTERRUPT_HOST_BASE + pin * 8), (st.r(reg::CONTROL) & 0xf) as u8) + }; + let level = destination >> 48; + if !LEVELS.contains(&level) { + crate::dlog!(LogModule::Bedrock, "pic: head {} pin {pin} rejected interrupt level {level:#x}", self.index); + return false; + } + let address = destination & ((1 << PAYLOAD_BITS) - 1); + let payload = level | if set { SET } else { 0 }; + let sent = self.send_interrupt(source, address, payload); + if !sent { + crate::dlog!(LogModule::Bedrock, "pic: head {} interrupt destination {destination:#x} refused", self.index); + } + sent + } + + fn send_interrupt(&self, source: u8, address: u64, payload: u64) -> bool { + match self.wiring.peer.get() { + Some(peer) => peer.interrupt(source, address, payload).is_ok(), + None => { + crate::dlog!(LogModule::Bedrock, "pic: head {} interrupt has no peer", self.index); + false + } + } + } + + /// Tell the Bedrock PI whether this pin's input is active at its + /// destination's level, so it can reassert a level the processor + /// cleared while the input stays high. + fn update_level_source(&self, pin: usize, delivered: bool) { + let (destination, active_pin) = { + let st = self.st.lock(); + let bit = 1u64 << pin; + (st.r(reg::INTERRUPT_HOST_BASE + pin * 8), st.pins as u64 & bit != 0 && st.r(reg::INTERRUPT_ENABLE) & bit != 0) + }; + let level = destination >> 48; + let target = usize::from(destination & (1 << 21) != 0); + let active = delivered && active_pin && LEVELS.contains(&level); + let source = self.wiring.source_base.load(Ordering::Acquire) + self.index * PINS + pin; + for (interface, pi) in self.wiring.pi.iter().enumerate() { + if let Some(pi) = pi.get() { + pi.set_level_source(source, level as u8, active && interface == target); + } + } + } + + fn pin(&self, pin: usize, level: bool) { + if pin >= PINS { + return; + } + let _order = self.order.lock(); + let bit = 1u8 << pin; + let (was_high, enabled, mode) = { + let mut st = self.st.lock(); + let was_high = st.pins & bit != 0; + st.pins = (st.pins & !bit) | if level { bit } else { 0 }; + if level && !was_high { + let status = st.r(reg::INTERRUPT_STATUS) | bit as u64; + st.set(reg::INTERRUPT_STATUS, status); + } + (was_high, st.r(reg::INTERRUPT_ENABLE) & bit as u64 != 0, st.r(reg::INTERRUPT_MODE) & bit as u64 != 0) + }; + if !level { + self.update_level_source(pin, false); + } + if level && !was_high && enabled { + let delivered = self.packet(pin, true); + self.update_level_source(pin, delivered); + } + if was_high && !level && mode { + self.packet(pin, false); + } + } + + /// The PROM's write to the link-side error address slot latches the + /// invalid-address condition and, when enabled, sends one packet. + fn invalid_address(&self) { + let _order = self.order.lock(); + let enabled = { + let mut st = self.st.lock(); + let status = st.r(reg::INTERRUPT_STATUS) | INVALID_ADDRESS; + st.set(reg::INTERRUPT_STATUS, status); + st.r(reg::INTERRUPT_ENABLE) & INVALID_ADDRESS != 0 + }; + if enabled { + self.error_interrupt(); + } + } + + /// An unclaimed configuration read: the error view always records the + /// master timeout, and interrupt status only while it is enabled, when + /// it also sends the error interrupt. + fn master_timeout(&self) { + let _order = self.order.lock(); + let enabled = { + let mut st = self.st.lock(); + let view = st.r(ERROR_VIEW) | MASTER_TIMEOUT; + st.set(ERROR_VIEW, view); + let enabled = st.r(reg::INTERRUPT_ENABLE) & MASTER_TIMEOUT != 0; + if enabled { + let status = st.r(reg::INTERRUPT_STATUS) | MASTER_TIMEOUT; + st.set(reg::INTERRUPT_STATUS, status); + } + enabled + }; + if enabled { + self.error_interrupt(); + } + } + + /// One set packet for the error interrupt, to the widget destination. + /// With `order` held. + fn error_interrupt(&self) { + let number = self.st.lock().r(reg::INTERRUPT_ERROR_HOST); + if number > 0x7f { + crate::dlog!(LogModule::Bedrock, "pic: head {} refused error interrupt number {number:#x}", self.index); + return; + } + let Some(address) = self.error_destination() else { return }; + let source = self.source_widget(); + if !self.send_interrupt(source, address, number | SET) { + crate::dlog!(LogModule::Bedrock, "pic: head {} error destination {address:#x} refused", self.index); + } + } + + /// The error interrupt's address from the widget destination pair: a + /// lower register that uses bits 47:32 is the address (the upper half + /// may repeat them), otherwise the pair joins into 48 bits. + fn error_destination(&self) -> Option { + let (upper, lower) = { + let st = self.st.lock(); + (st.r(reg::INT_DEST_UPPER), st.r(reg::INT_DEST_LOWER)) + }; + let upper_address = upper & 0xffff; + let lower_high = lower >> 32; + let refuse = |why: &str| { + crate::dlog!(LogModule::Bedrock, "pic: head {} error destination {why} (upper {upper:#x}, lower {lower:#x})", self.index); + None + }; + if upper >> 32 != 0 { + return refuse("has unsupported high bits"); + } + if lower >> 48 != 0 { + return refuse("exceeds 48 bits"); + } + if upper_address != 0 && lower_high != 0 && upper_address != lower_high { + return refuse("has conflicting address fields"); + } + let address = if lower_high != 0 { lower } else { (upper_address << 32) | (lower & 0xffff_ffff) }; + if address == 0 { + return refuse("is zero"); + } + Some(address) + } + + // ---- registers ---- + + fn identity(&self) -> u64 { + identity(self.index) as u64 + } + + /// `None` is a read the processor gets back as a bus error: only an + /// unclaimed configuration read fails that way. + fn read(&self, addr: u64, size: usize) -> Option { + if short_window(addr).is_some() { + return Some(self.short(addr, size, None).unwrap_or(0)); + } + if (CONFIG_BASE..CONFIG_END).contains(&addr) && size <= 4 { + return self.config(addr, size, None).ok(); + } + let off = addr as usize; + if size == 4 && off % 8 == 4 { + return Some(self.word_read(off)); + } + if size == 8 && off == 0 { + return Some(self.identity()); + } + if size == 8 && off % 8 == 0 && defined(off) { + return Some(self.st.lock().r(off)); + } + if size == 8 && off % 8 == 0 && (REQUEST_BASE..REQUEST_END).contains(&off) { + let (bank, index) = request_slot(off); + return Some(self.st.lock().request[bank][index]); + } + self.note_unsupported(addr, size, false, 0); + Some(0) + } + + fn write(&self, addr: u64, size: usize, value: u64) { + if short_window(addr).is_some() { + self.short(addr, size, Some(value)); + return; + } + if addr == ERROR_ADDRESS_WORD && size == 4 { + self.invalid_address(); + return; + } + if (CONFIG_BASE..CONFIG_END).contains(&addr) && size <= 4 { + let _ = self.config(addr, size, Some(value)); + return; + } + let off = addr as usize; + if size == 4 && off % 8 == 4 { + self.word_write(off, value as u32); + return; + } + self.register_write(off, size, value); + } + + /// A 32-bit access to the low half of an eight-byte slot. + fn word_slot(off: usize) -> Option { + let slot = off.checked_sub(4).filter(|slot| slot % 8 == 0)?; + ((REQUEST_BASE..REQUEST_END).contains(&slot) || (defined(slot) && slot < REGISTER_BYTES)).then_some(slot) + } + + fn word_read(&self, off: usize) -> u64 { + match Self::word_slot(off) { + Some(slot) if slot != FLUSH => { + if slot == reg::IDENTIFICATION { + return self.identity(); + } + let st = self.st.lock(); + let full = if slot >= REQUEST_BASE { + let (bank, index) = request_slot(slot); + st.request[bank][index] + } else { + st.r(slot) + }; + full as u32 as u64 + } + _ => { + self.note_unsupported(off as u64, 4, false, 0); + 0 + } + } + } + + fn word_write(&self, off: usize, value: u32) { + let slot = match Self::word_slot(off) { + Some(slot) if !matches!(slot, reg::IDENTIFICATION | FLUSH | reg::INTERRUPT_STATUS | reg::INTERRUPT_DEVICE | ERROR_VIEW) => slot, + _ => { + self.note_unsupported(off as u64, 4, true, value as u64); + return; + } + }; + if slot >= REQUEST_BASE { + let (bank, index) = request_slot(slot); + let entry = &mut self.st.lock().request[bank][index]; + *entry = (*entry & 0xffff_ffff_0000_0000) | value as u64; + } else if slot == reg::INTERRUPT_RESET { + self.register_write(slot, 8, value as u64); + } else { + let merged = (self.st.lock().r(slot) & 0xffff_ffff_0000_0000) | value as u64; + self.register_write(slot, 8, merged); + } + } + + fn register_write(&self, off: usize, size: usize, value: u64) { + if size != 8 || off % 8 != 0 { + self.note_unsupported(off as u64, size, true, value); + return; + } + if (REQUEST_BASE..REQUEST_END).contains(&off) { + let (bank, index) = request_slot(off); + self.st.lock().request[bank][index] = value; + return; + } + if !defined(off) || off >= REGISTER_BYTES { + self.note_unsupported(off as u64, size, true, value); + return; + } + match off { + reg::IDENTIFICATION | FLUSH | reg::INTERRUPT_STATUS | ERROR_VIEW => { + self.note_unsupported(off as u64, size, true, value) + } + reg::CONTROL => { + if value & !CONTROL_MASK != 0 { + self.note_unsupported(off as u64, size, true, value); + } + self.st.lock().set(off, value & CONTROL_MASK); + } + reg::DIRECT_MAP => { + if value & !DIRECT_MAP_SUPPORTED != 0 { + self.note_unsupported(off as u64, size, true, value); + } + self.st.lock().set(off, value & DIRECT_MAP_SUPPORTED); + } + reg::INTERRUPT_ENABLE => { + let _order = self.order.lock(); + let (newly, pins, status) = { + let mut st = self.st.lock(); + let newly = value & !st.r(off); + st.set(off, value); + (newly, st.pins as u64, st.r(reg::INTERRUPT_STATUS)) + }; + for pin in 0..PINS { + if newly & status & pins & (1 << pin) != 0 { + let delivered = self.packet(pin, true); + self.update_level_source(pin, delivered); + } else if value & (1 << pin) == 0 { + self.update_level_source(pin, false); + } + } + } + reg::INTERRUPT_RESET => { + let _order = self.order.lock(); + let (pins, enable) = { + let mut st = self.st.lock(); + // Each error bit clears on its own one; any write + // clears the view. + let status = (st.r(reg::INTERRUPT_STATUS) & !value) | st.pins as u64; + st.set(reg::INTERRUPT_STATUS, status); + st.set(ERROR_VIEW, 0); + (st.pins as u64, st.r(reg::INTERRUPT_ENABLE)) + }; + for pin in 0..PINS { + if value & pins & enable & (1 << pin) != 0 { + let delivered = self.packet(pin, true); + self.update_level_source(pin, delivered); + } + } + } + reg::INTERRUPT_DEVICE => { + // The device-to-pin map has no effect here; the board wires + // each device to its pin. + self.note_unsupported(off as u64, size, true, value); + self.st.lock().set(off, value); + } + _ => { + let device = (reg::DEVICE_BASE..=0x218).contains(&off); + let stored = if device { + if value & !DEVICE_MASK != 0 { + self.note_unsupported(off as u64, size, true, value); + } + value & DEVICE_MASK + } else { + value + }; + self.st.lock().set(off, stored); + if (reg::INTERRUPT_HOST_BASE..=reg::INTERRUPT_HOST_LAST).contains(&off) { + let _order = self.order.lock(); + self.update_level_source((off - reg::INTERRUPT_HOST_BASE) / 8, false); + } + } + } + } + + // ---- PCI configuration ---- + + /// A processor configuration cycle. Bus 0 devices sit at `0x1000` + /// strides from `0x20000`; the type-1 window addresses the bus and + /// device `PCI_TYPE1_CONFIG` names. A read no function claims, device 0 + /// (the bridge's own) and every function but 0 included, is a master + /// timeout and fails; a write nobody claims goes nowhere. + fn config(&self, addr: u64, size: usize, value: Option) -> Result { + let mask = u32::MAX >> (32 - size * 8); + let function = (addr >> 8) & 7; + let (bus, device) = if addr < CONFIG_TYPE1 { + (0, ((addr - CONFIG_BASE) >> 12) as usize) + } else { + let t1 = self.st.lock().r(reg::PCI_TYPE1_CONFIG); + (((t1 >> 16) & 0xff) as usize, ((t1 >> 11) & 0x1f) as usize) + }; + let register = ((addr & 0xff) as usize) ^ (4 - size); + let found = (bus == 0 && function == 0 && (1..=DEVICES as usize).contains(&device)) + .then(|| self.slots.read()[device - 1].as_ref().map(|slot| slot.function.clone())) + .flatten(); + let Some(function) = found else { + crate::dlog!( + LogModule::Bedrock, + "pic: head {} configuration {} miss at {addr:#x}", + self.index, + if value.is_some() { "write" } else { "read" } + ); + if value.is_some() { + return Ok(0); + } + self.master_timeout(); + return Err(Unclaimed); + }; + let shift = (register & 3) * 8; + match value { + Some(value) => { + function.config().write_masked(register / 4, (value as u32 & mask) << shift, mask << shift); + self.note_mappings(); + Ok(0) + } + None => Ok(function.config().read_sized(register, size) as u64), + } + } + + // ---- PCI memory and I/O cycles ---- + + /// Record which BAR windows are new or moved since the last look, so a + /// later claim can tell the most recently mapped of two overlapping + /// windows. Called after every configuration write. + fn note_mappings(&self) { + let slots = self.slots.read(); + let mut mapped = self.mappings.lock(); + let mut counter = mapped.counter; + let mut table = Vec::new(); + for (i, slot) in slots.iter().enumerate() { + let Some(slot) = slot else { continue }; + for (space, bar, base, size) in slot.function.config().windows() { + let known = mapped.table.iter().find(|e| (e.0, e.1, e.2, e.3, e.4) == (i, space, bar, base, size)); + let order = known.map_or_else( + || { + counter += 1; + counter + }, + |e| e.5, + ); + table.push((i, space, bar, base, size, order)); + } + } + mapped.counter = counter; + mapped.table = table; + } + + /// The function that decodes `len` bytes at `address` in `space`, whole + /// within one BAR. Where windows overlap, the one mapped last answers. + fn claim(&self, space: Space, address: u64, len: usize) -> Option { + let first = u32::try_from(address).ok()?; + let last = u32::try_from(address.checked_add(len as u64)? - 1).ok()?; + let slots = self.slots.read(); + let mapped = self.mappings.lock(); + let (_, slot, bar, base) = slots + .iter() + .enumerate() + .filter_map(|(i, slot)| Some((i, slot.as_ref()?))) + .flat_map(|(i, slot)| { + slot.function.config().windows().into_iter().map(move |(space, bar, base, size)| (i, slot, space, bar, base, size)) + }) + .filter(|&(_, _, s, _, base, size)| s == space && first >= base && (last as u64) < base as u64 + size as u64) + .map(|(i, slot, space, bar, base, size)| { + let order = mapped.table.iter().find(|e| (e.0, e.1, e.2, e.3, e.4) == (i, space, bar, base, size)).map_or(0, |e| e.5); + (order, slot, bar, base) + }) + .max_by_key(|&(order, ..)| order)?; + Some(Claim { function: slot.function.clone(), options: slot.options, bar, offset: first - base }) + } + + /// Move `bytes` (little-endian PCI byte order, in address order) across + /// a PCI cycle. False when no function decodes the whole run. + fn pci_cycle(&self, space: Space, address: u64, bytes: &mut [u8], write: bool) -> bool { + let Some(claim) = self.claim(space, address, bytes.len()) else { + return false; + }; + for (i, part) in bytes.chunks_mut(4).enumerate() { + let offset = claim.offset + i as u32 * 4; + if write { + claim.function.bar_write(claim.bar, offset, le_value(part) as u32, part.len()); + } else { + let v = claim.function.bar_read(claim.bar, offset, part.len()); + part.copy_from_slice(&v.to_le_bytes()[..part.len()]); + } + } + true + } + + /// A processor access to PCI space. Aligned bytes, halfwords and words + /// reach a conventional adapter at the address with its low two bits + /// corrected, carrying the numeric value; a self-ordered function gets + /// the processor's byte sequence at the address as it is. Other widths + /// pass through without placement. + fn cpu_pci(&self, space: Space, mut address: u64, size: usize, value: Option) -> Option { + let placed = matches!(size, 1 | 2 | 4) && address.is_multiple_of(size as u64); + let native = placed + && self.claim(space, address, size).is_some_and(|claim| claim.options.self_ordered_pio); + if placed && !native && size < 4 { + address ^= 4 - size as u64; + } + let mut bytes = [0u8; 8]; + let ok = if let Some(value) = value { + bytes = if native { be_bytes(size, value) } else { le_bytes(size, value) }; + self.pci_cycle(space, address, &mut bytes[..size], true) + } else { + self.pci_cycle(space, address, &mut bytes[..size], false) + }; + ok.then(|| if native { be_value(&bytes[..size]) } else { le_value(&bytes[..size]) }) + } + + /// The short device-I/O windows: the Device register of the window + /// picks PCI memory or I/O space and the megabyte of it. + fn short(&self, addr: u64, size: usize, value: Option) -> Option { + let write = value.is_some(); + let (window, offset, window_size) = short_window(addr)?; + if size as u64 > window_size - offset { + crate::dlog!(LogModule::Bedrock, "pic: head {} short window {addr:#x} rejects boundary-crossing {size}-byte access", self.index); + return None; + } + let device = self.st.lock().r(reg::DEVICE_BASE + window * 8); + let refused = device & (DEVICE_STREAM | DEVICE_WIDE); + if refused != 0 { + crate::dlog!( + LogModule::Bedrock, + "pic: head {} short window {addr:#x} refuses unsupported Device attributes {refused:#x}", + self.index + ); + return None; + } + let memory = device & DEVICE_MEMORY != 0; + let mut base = device & if memory { 0x3ff } else { 0xfff }; + if window < 2 { + base &= !1; + } + let result = self.cpu_pci(if memory { Space::Mem } else { Space::Io }, (base << 20) + offset, size, value); + if result.is_none() { + crate::dlog!( + LogModule::Bedrock, + "pic: head {} short window {addr:#x} PCI {} failed for {size} bytes", + self.index, + if write { "write" } else { "read" } + ); + } + result + } + + /// The large PCI memory window of this head. + fn pci_window(&self, offset: u64, size: usize, value: Option) -> u64 { + let result = self.cpu_pci(Space::Mem, offset, size, value); + if result.is_none() { + self.note_unsupported(PCI_WINDOW[0] + offset, size, value.is_some(), value.unwrap_or(0)); + } + result.unwrap_or(0) + } + + // ---- DMA ---- + + fn dma(&self, device: u8, address: u64, bytes: &mut [u8], write: bool) -> PciDmaStatus { + if !(1..=DEVICES).contains(&device) { + crate::dlog!(LogModule::Bedrock, "pic: head {} unsupported requester device {device}", self.index); + if !write { + bytes.fill(0); + } + return PciDmaStatus::MasterAbort; + } + let mut done = 0; + while done < bytes.len() { + let at = address.wrapping_add(done as u64); + let n = (0x1000 - (at as usize & 0xfff)).min(bytes.len() - done); + let ok = address.checked_add(done as u64).is_some() && self.dma_page(device, at, &mut bytes[done..done + n], write); + if !ok { + if !write { + bytes[done..].fill(0); + } + return PciDmaStatus::MasterAbort; + } + done += n; + } + PciDmaStatus::Ok + } + + fn dma_page(&self, device: u8, at: u64, chunk: &mut [u8], write: bool) -> bool { + if at > u32::MAX as u64 { + return self.direct64(device, at, chunk, write); + } + if at < 0x8000_0000 { + crate::dlog!(LogModule::Bedrock, "pic: head {} unsupported DMA {at:#x} from device {device}", self.index); + return false; + } + self.direct32(device, at, chunk, write) + } + + fn send_dma(&self, source: u8, widget: u8, address: u64, data: &mut [u8], write: bool) -> bool { + let Some(peer) = self.wiring.peer.get() else { + crate::dlog!(LogModule::Bedrock, "pic: head {} DMA has no peer", self.index); + return false; + }; + if write { + peer.dma_write(source, widget, address, data).is_ok() + } else { + peer.dma_read(source, widget, address, data).is_ok() + } + } + + /// The direct map: PCI `0x80000000` and up reach the widget `DIR_MAP` + /// names, at the address its high bits and the PCI offset make. + fn direct32(&self, device: u8, at: u64, chunk: &mut [u8], write: bool) -> bool { + let (direct, attributes, source) = { + let st = self.st.lock(); + (st.r(reg::DIRECT_MAP), st.r(reg::DEVICE_BASE + (device as usize - 1) * 8), (st.r(reg::CONTROL) & 0xf) as u8) + }; + let widget = ((direct >> 20) & 0xf) as u8; + let target = ((direct & 0x1ffff) << 31) | (at & 0x7fff_ffff); + if attributes & DEVICE_SWAP != 0 { + // A swapped slot moves single bytes, each at its address with + // the low two bits flipped. + return chunk.iter_mut().enumerate().all(|(i, byte)| { + self.send_dma(source, widget, (target + i as u64) ^ 3, std::slice::from_mut(byte), write) + }); + } + let mut done = 0; + while done < chunk.len() { + let n = natural(target + done as u64, chunk.len() - done); + if !self.send_dma(source, widget, target + done as u64, &mut chunk[done..done + n], write) { + return false; + } + done += n; + } + true + } + + /// Why a direct64 request cannot be carried, if it cannot. It is + /// checked in full before any packet leaves. + fn direct64_refusal(address: u64, size: usize) -> Option<&'static str> { + let payload = address & ((1 << PAYLOAD_BITS) - 1); + if address >> 60 == 0 { + Some("target-zero") + } else if address & ((1 << 54) | (0x3f << 48)) != 0 { + Some("reserved-bits") + } else if payload & (1 << 47) != 0 { + Some("register-space") + } else if size as u64 > (1 << 47) - payload { + Some("range") + } else { + None + } + } + + /// A PCI address above 32 bits carries the target widget in bits 63:60 + /// and the XIO payload below. Bit 55 selects an ordered byte stream; + /// clear, the bytes keep 32-bit numeric values (the byte address low + /// bits flip by 3, or by 7 for a REQ64 slot's 64-bit requester). + fn direct64(&self, device: u8, at: u64, chunk: &mut [u8], write: bool) -> bool { + let wide = self.slots.read()[device as usize - 1].as_ref().is_some_and(|slot| slot.options.req64); + let source = self.source_widget(); + let check = |address: u64, size: usize| match Self::direct64_refusal(address, size) { + Some(why) => { + crate::dlog!(LogModule::Bedrock, "pic: head {} direct64 {why} address {address:#018x} from device {device}", self.index); + false + } + None => true, + }; + if !check(at, 1) { + return false; + } + let widget = (at >> 60) as u8; + let payload = at & ((1 << PAYLOAD_BITS) - 1); + let mask = if wide { 7 } else { 3 }; + let mut done = 0; + while done < chunk.len() { + let here = at + done as u64; + if at & (1 << 55) != 0 { + let n = 8.min(chunk.len() - done); + if !check(here, n) || !self.send_dma(source, widget, payload + done as u64, &mut chunk[done..done + n], write) { + return false; + } + done += n; + } else { + if !check(here, 1) || !self.send_dma(source, widget, (payload + done as u64) ^ mask, &mut chunk[done..done + 1], write) { + return false; + } + done += 1; + } + } + true + } + + fn attach(self: &Arc, device: u8, function: Arc, options: Function) { + assert!((1..=DEVICES).contains(&device), "PIC devices are 1 to {DEVICES}"); + self.slots.write()[device as usize - 1] = Some(Slot { function: function.clone(), options }); + self.note_mappings(); + function.connect_dma(Arc::new(SlotDma { head: Arc::downgrade(self), device })); + let pin = function.config().interrupt_pin(); + if (1..=4).contains(&pin) { + function.connect_interrupt(IrqLine::new( + Arc::new(Pin { head: Arc::downgrade(self), pci: true }), + device as u32 * 4 + pin as u32 - 1, + )); + } + function.req64_sampled(options.req64); + } + + fn reset(&self) { + { + let _order = self.order.lock(); + self.st.lock().reset(); + } + for pin in 0..PINS { + self.update_level_source(pin, false); + } + let outputs = self.pci_outputs.read().clone(); + for line in outputs.into_iter().flatten() { + line.set(false); + } + for slot in self.slots.read().iter().flatten() { + slot.function.config().reset(); + slot.function.req64_sampled(slot.options.req64); + } + self.mappings.lock().table.clear(); + self.unsupported.lock().clear(); + } +} + +/// The PCI host bridge in one XIO widget. +pub struct Pic { + widget: u8, + wiring: Arc, + heads: [Arc; 2], + pxbow: OnceLock>, +} + +impl Pic { + /// A PIC that answers as XIO widget `widget`. + pub fn new(widget: u8) -> Arc { + let wiring = Arc::new(Wiring { + peer: OnceLock::new(), + pi: [OnceLock::new(), OnceLock::new()], + source_base: AtomicUsize::new(0), + }); + Arc::new(Self { + widget, + heads: [Head::new(0, wiring.clone()), Head::new(1, wiring.clone())], + wiring, + pxbow: OnceLock::new(), + }) + } + + pub fn widget_number(&self) -> u8 { + self.widget + } + + /// The widget as the processor reaches it: both heads, the short + /// device-I/O windows and the PCI memory windows. Bedrock IO takes this. + pub fn widget(self: &Arc) -> Arc { + Arc::new(CpuWidget { heads: self.heads.clone() }) + } + + /// Give the chip its PXBOW, the widget-0 face of the board that owns + /// it, reached from `host_widget`. On Tezro the PIC that carries IO9 + /// supplies widget 0; a second PIC gets none. The PIC's own widget is + /// attached behind it, and widget 0's interrupt goes to the peer when it + /// targets the host. Once, before the machine runs. + pub fn enable_pxbow(self: &Arc, host_widget: u8, clock: Clock) -> Result, AttachError> { + let pxbow = Pxbow::new(host_widget, self.widget, clock)?; + pxbow.attach_endpoint(self.widget, self.widget())?; + let wiring = self.wiring.clone(); + pxbow.connect_packets(Box::new(move |widget, address, vector, set| { + widget == host_widget + && wiring + .peer + .get() + .is_some_and(|peer| peer.interrupt(0, address, vector as u64 | if set { SET } else { 0 }).is_ok()) + })); + self.pxbow.set(pxbow.clone()).map_err(|_| AttachError::Occupied(self.widget))?; + Ok(pxbow) + } + + /// The chip's PXBOW, if it has one. + pub fn pxbow(&self) -> Option<&Arc> { + self.pxbow.get() + } + + /// Where interrupts and DMA this chip originates go. Once, before the + /// machine runs. + pub fn connect_peer(&self, peer: Arc) -> Result<(), AlreadyConnected> { + self.wiring.peer.set(peer).map_err(|_| AlreadyConnected) + } + + /// Connect Bedrock PI `interface` (0 or 1) as a destination of this + /// chip's interrupts. `source_base` is the first of the 16 level + /// sources the chip owns on the PI; two PICs on one node need different + /// bases. + pub fn connect_pi(&self, interface: usize, pi: Arc, source_base: usize) { + self.wiring.source_base.store(source_base, Ordering::Release); + let _ = self.wiring.pi[interface].set(pi); + } + + /// Put a PCI function at configuration `device` (1 to 4) of `head`'s + /// bus. It gets its DMA path and its INTA to INTD lines; the board + /// routes those with [`Pic::route_pci_interrupt`] or [`Pic::wire_inta`]. + pub fn attach(&self, head: usize, device: u8, function: Arc, options: Function) { + self.heads[head].attach(device, function, options); + } + + /// Interrupt input `pin` (0 to 7) of `head`, for a device whose + /// interrupt the board wires to it directly. + pub fn pin_line(&self, head: usize, pin: usize) -> IrqLine { + assert!(pin < PINS); + IrqLine::new(Arc::new(Pin { head: Arc::downgrade(&self.heads[head]), pci: false }), pin as u32) + } + + /// Board wiring of a device's interrupt pin (`intx` 0 for INTA) to + /// `output`. + pub fn route_pci_interrupt(&self, head: usize, device: u8, intx: u8, output: IrqLine) { + assert!((1..=DEVICES).contains(&device) && intx < 4); + self.heads[head].pci_outputs.write()[(device * 4 + intx) as usize] = Some(output); + } + + /// Route `device`'s INTA to the input numbered one less than the + /// device, the convention the public drivers use. + pub fn wire_inta(&self, head: usize, device: u8) { + self.route_pci_interrupt(head, device, 0, self.pin_line(head, device as usize - 1)); + } + + /// Set interrupt input `pin` of `head`. + pub fn set_pin(&self, head: usize, pin: usize, level: bool) { + self.heads[head].pin(pin, level); + } + + /// The set and clear packets `head` has requested for `pin`, delivered + /// or not. + pub fn request_counts(&self, head: usize, pin: usize) -> (u32, u32) { + let st = self.heads[head].st.lock(); + (st.set_requests[pin], st.clear_requests[pin]) + } + + /// Accesses the model refused or could not carry out, as (head, write, + /// offset, size); the first 32 per head. + pub fn unsupported(&self) -> Vec<(usize, bool, u64, usize)> { + self.heads.iter().flat_map(|head| head.unsupported.lock().clone()).collect() + } + + pub fn register_locks(self: &Arc) { + use crate::locks::register_lock_fn; + for head in &self.heads { + for (what, probe) in [ + ("state", (|h: &Head| h.st.is_locked()) as fn(&Head) -> bool), + ("irq_order", |h| h.order.is_locked()), + ] { + let h = Arc::downgrade(head); + register_lock_fn(format!("pic{:x}::head{}::{what}", self.widget, head.index), move || { + h.upgrade().is_some_and(|h| probe(&h)) + }); + } + } + if let Some(pxbow) = self.pxbow.get() { + pxbow.xbow().register_locks(); + } + } +} + +impl Resettable for Pic { + fn power_on(&self) { + if let Some(pxbow) = self.pxbow.get() { + pxbow.xbow().power_on(); + } + for head in &self.heads { + head.reset(); + } + } +} + +/// The processor's view of the widget. +struct CpuWidget { + heads: [Arc; 2], +} + +impl CpuWidget { + /// `None` is a bus error: a malformed access or an unclaimed + /// configuration read. Accesses the PIC cannot make sense of that are + /// not malformed (unsupported registers, refused windows) read zero and + /// write nowhere without one. + fn access(&self, addr: u64, size: usize, value: Option) -> Option { + let write = value.is_some(); + let short = addr < 2 * HEAD_OFFSET && short_window(addr % HEAD_OFFSET).is_some(); + if !short && !addr.is_multiple_of(size as u64) { + return None; + } + if addr >= 2 * HEAD_OFFSET { + if addr < 2 * HEAD_OFFSET + 8 { + crate::dlog!(LogModule::Bedrock, "pic: short-window guard {} at {addr:#x} refused for {size} bytes", if write { "write" } else { "read" }); + return Some(0); + } + let window = PCI_WINDOW.iter().position(|&base| (base..base + PCI_WINDOW_SIZE).contains(&addr))?; + return Some(self.heads[window].pci_window(addr - PCI_WINDOW[window], size, value)); + } + let head = &self.heads[(addr / HEAD_OFFSET) as usize]; + let rel = addr % HEAD_OFFSET; + match value { + Some(v) => { + head.write(rel, size, v); + Some(0) + } + None => head.read(rel, size), + } + } +} + +macro_rules! cpu_access { + ($r:ident, $w:ident, $t:ty, $out:ident, $n:expr) => { + fn $r(&self, addr: u64) -> $out { + self.access(addr, $n, None).map_or_else($out::err, |v| $out::ok(v as $t)) + } + fn $w(&self, addr: u64, value: $t) -> u32 { + if self.access(addr, $n, Some(value as u64)).is_some() { BUS_OK } else { BUS_ERR } + } + }; +} + +impl BusDevice for CpuWidget { + cpu_access!(read8, write8, u8, BusRead8, 1); + cpu_access!(read16, write16, u16, BusRead16, 2); + cpu_access!(read32, write32, u32, BusRead32, 4); + cpu_access!(read64, write64, u64, BusRead64, 8); +} + +#[cfg(test)] +mod tests; diff --git a/src/dev/pic/tests.rs b/src/dev/pic/tests.rs new file mode 100644 index 000000000..589adb9c7 --- /dev/null +++ b/src/dev/pic/tests.rs @@ -0,0 +1,1043 @@ +// Cases follow tests/qtest/sgi-pic-test.c in jamesbraid/qemu-sgi-origami, for +// the Tezro runtime profile only: identity-fields, runtime-identity-bit, +// heads-and-widths, runtime-reset-compatibility-trial, widget-registers, +// cpu-word-view-registers, runtime-host-error-field, +// cpu-pio-overlapping-bars-use-active-function, +// runtime-dir-map-unsupported-fields, configuration, slot-bounds, +// runtime-profile-reserved-device-and-dma, type1-configuration, +// cpu-pio-conventional-lanes, cpu-pio-actual-ioc4, +// short-device-io-windows, short-device-io-attributes-boundaries, +// direct-dma, direct-dma-head-targets, direct64-card-write, +// direct64-requester-width, direct64-rejects-unsupported-addresses, +// direct64-range-boundary-is-split, direct64-target-error-preserves-data, +// pin-and-reset, remaining-pin-edges, bedrock-interrupt-level-bounds, +// runtime-profile-bedrock-level-mirror, runtime-profile-bedrock-rearm-one-level, +// runtime-profile-bedrock-rearm-staggered-levels, +// runtime-invalid-address-error-trial and unwired-pci-interrupt. The qtest +// fixtures' packet face, translated DMA, migration streams and profile +// selection have no counterpart here. +use super::*; +use crate::dev::bedrock_pi::{reg as pi_reg, BedrockPiConfig}; +use crate::irq::LevelProbe; +use crate::traits::Device; +use crate::pci::{Bar, PciConfig, PCI_INTA}; +use std::collections::HashMap; +use std::sync::atomic::{AtomicBool, AtomicU64}; + +const HEAD1: u64 = HEAD_OFFSET; +const BAR_BASE: u32 = 0x0100_0000; +const MEMORY_AND_MASTER: u32 = 6; + +/// A PCI function with one 4 KiB memory BAR that stores what it is given. +struct Card { + config: PciConfig, + bar: Mutex<[u8; 0x1000]>, + dma: OnceLock>, + line: OnceLock, + /// Each REQ64# level the bridge reported, in order. + req64: Mutex>, +} + +impl Card { + fn new(device: u16) -> Arc { + let bars = [Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused]; + Arc::new(Self { + config: PciConfig::new(0x1234, device, 0x028000, 1, bars, PCI_INTA), + bar: Mutex::new([0; 0x1000]), + dma: OnceLock::new(), + line: OnceLock::new(), + req64: Mutex::new(Vec::new()), + }) + } + + fn dma(&self) -> &Arc { + self.dma.get().expect("attached to a bridge") + } + + fn raise(&self, level: bool) { + self.line.get().expect("attached to a bridge").set(level); + } +} + +impl PciBusDevice for Card { + fn config(&self) -> &PciConfig { + &self.config + } + + fn bar_read(&self, _bar: usize, off: u32, width: usize) -> u32 { + le_value(&self.bar.lock()[off as usize..off as usize + width]) as u32 + } + + fn bar_write(&self, _bar: usize, off: u32, val: u32, width: usize) { + self.bar.lock()[off as usize..off as usize + width].copy_from_slice(&val.to_le_bytes()[..width]); + } + + fn req64_sampled(&self, asserted: bool) { + self.req64.lock().push(asserted); + } + + fn connect_interrupt(&self, line: IrqLine) { + let _ = self.line.set(line); + } + + fn connect_dma(&self, dma: Arc) { + let _ = self.dma.set(dma); + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +struct Dma { + source: u8, + widget: u8, + address: u64, + len: usize, + write: bool, +} + +/// The link's far end: sparse memory per widget, and a log of what arrived. +#[derive(Default)] +struct Peer { + interrupts: Mutex>, + dma: Mutex>, + memory: Mutex>, + refuse: AtomicBool, +} + +impl Peer { + fn byte(&self, widget: u8, address: u64) -> u8 { + self.memory.lock().get(&(widget, address)).copied().unwrap_or(0) + } + + fn bytes(&self, widget: u8, address: u64, len: usize) -> Vec { + (0..len as u64).map(|i| self.byte(widget, address + i)).collect() + } + + fn put(&self, widget: u8, address: u64, data: &[u8]) { + for (i, b) in data.iter().enumerate() { + self.memory.lock().insert((widget, address + i as u64), *b); + } + } +} + +impl PicPeer for Peer { + fn interrupt(&self, source_widget: u8, address: u64, payload: u64) -> Result<(), Refused> { + if self.refuse.load(Ordering::SeqCst) { + return Err(Refused); + } + self.interrupts.lock().push((source_widget, address, payload)); + Ok(()) + } + + fn dma_read(&self, source: u8, widget: u8, address: u64, data: &mut [u8]) -> Result<(), Refused> { + self.dma.lock().push(Dma { source, widget, address, len: data.len(), write: false }); + if self.refuse.load(Ordering::SeqCst) { + return Err(Refused); + } + for (i, b) in data.iter_mut().enumerate() { + *b = self.byte(widget, address + i as u64); + } + Ok(()) + } + + fn dma_write(&self, source: u8, widget: u8, address: u64, data: &[u8]) -> Result<(), Refused> { + self.dma.lock().push(Dma { source, widget, address, len: data.len(), write: true }); + if self.refuse.load(Ordering::SeqCst) { + return Err(Refused); + } + self.put(widget, address, data); + Ok(()) + } +} + +struct Rig { + pic: Arc, + bus: Arc, + peer: Arc, +} + +fn rig() -> Rig { + let pic = Pic::new(15); + let peer = Arc::new(Peer::default()); + pic.connect_peer(peer.clone()).unwrap(); + Rig { bus: pic.widget(), pic, peer } +} + +impl Rig { + fn r64(&self, addr: u64) -> u64 { + let r = self.bus.read64(addr); + assert!(r.is_ok(), "read64 {addr:#x}"); + r.data + } + + fn r32(&self, addr: u64) -> u32 { + let r = self.bus.read32(addr); + assert!(r.is_ok(), "read32 {addr:#x}"); + r.data + } + + fn w64(&self, addr: u64, v: u64) { + assert_eq!(self.bus.write64(addr, v), BUS_OK, "write64 {addr:#x}"); + } + + fn w32(&self, addr: u64, v: u32) { + assert_eq!(self.bus.write32(addr, v), BUS_OK, "write32 {addr:#x}"); + } + + /// Fit `card` at `device` of `head` and map its BAR at `BAR_BASE`. + fn fit(&self, head: usize, device: u8, card: &Arc, options: Function) { + self.pic.attach(head, device, card.clone(), options); + let cfg = config(head, device as u64); + self.w32(cfg + 0x10, BAR_BASE); + self.w32(cfg + 4, MEMORY_AND_MASTER); + } +} + +fn head_base(head: usize) -> u64 { + head as u64 * HEAD_OFFSET +} + +fn config(head: usize, device: u64) -> u64 { + head_base(head) + CONFIG_BASE + device * 0x1000 +} + +/// The processor address of byte `off` of a card's BAR through the large +/// PCI memory window. +fn window(head: usize, off: u64) -> u64 { + PCI_WINDOW[head] + BAR_BASE as u64 + off +} + +fn decode(word: u64, head: usize) { + assert_eq!((word >> 28) & 0xf, 3, "revision"); + assert_eq!((word >> 12) & 0xffff, [0xd102, 0xd112][head], "part"); + assert_eq!((word & 0xffe) >> 1, 0x24, "manufacturer, packed one bit up"); + assert_eq!(word & 1, 1, "bit 0, which the PROM's identity check needs"); +} + +/// A Bedrock PI bank the PIC raises its levels on. +fn pi_with_clock() -> (Arc, Arc) { + let now = Arc::new(AtomicU64::new(0)); + let clock = now.clone(); + let pi = BedrockPi::new(BedrockPiConfig::default(), Arc::new(move || clock.load(Ordering::SeqCst))); + pi.start(); + (pi, now) +} + +/// The destination a pin's packet goes to: `level` in bits 63:48 and an +/// address in the PI's window. +fn destination(level: u64) -> u64 { + (level << 48) | 0x1800090 +} + +/// The PIC's identity is the one thing the PROM checks before it trusts the +/// bridge, and each head has its own. +#[test] +fn each_head_has_its_own_identity_at_both_read_widths() { + let r = rig(); + for head in 0..2 { + decode(r.r64(head_base(head)), head); + assert_eq!(r.r32(head_base(head) + 4) as u64, r.r64(head_base(head))); + assert_eq!(identity(head) as u64, r.r64(head_base(head))); + } + assert_ne!(r.r64(0), r.r64(HEAD1)); + r.w64(0x20, 5); + r.pic.power_on(); + decode(r.r64(0), 0); + assert_eq!(r.r64(0x20), 0x0f00_320f, "reset restores control but not identity edits"); +} + +#[test] +fn reset_restores_the_selected_control_and_device_values() { + let r = rig(); + for head in 0..2 { + assert_eq!(r.r64(head_base(head) + 0x20), 0x0f00_320f); + for (i, v) in [0x18001000, 0x18001002, 0x18001004, 0x18001005].into_iter().enumerate() { + assert_eq!(r.r64(head_base(head) + 0x200 + i as u64 * 8), v); + } + assert_eq!(r.r64(head_base(head) + 0x28), 0xfffff); + } + // The low word of control merges into the register, per head. + r.w32(0x24, 0x0f00_420f); + assert_eq!(r.r32(0x24), 0x0f00_420f); + assert_eq!(r.r32(HEAD1 + 0x24), 0x0f00_320f); + r.w64(HEAD1 + 0x210, 0x1858_1007); + assert_eq!(r.r64(HEAD1 + 0x210), 0x1858_1007); + for head in 0..2 { + for i in 0..4 { + r.w64(head_base(head) + 0x200 + i * 8, 0xdead_0000 + head as u64 * 0x100 + i); + } + } + r.pic.power_on(); + for head in 0..2 { + assert_eq!(r.r64(head_base(head) + 0x20), 0x0f00_320f); + assert_eq!(r.r64(head_base(head) + 0x200), 0x18001000); + assert_eq!(r.r64(head_base(head) + 0x218), 0x18001005); + } +} + +/// The PROM writes the widget registers a word at a time through the low +/// half of each slot; a full doubleword write is the other form. +#[test] +fn widget_registers_take_full_and_low_word_writes_on_each_head() { + let r = rig(); + for head in 0..2 { + let base = head_base(head); + for (n, seed) in [0xfffff_u64, 0, 0].into_iter().enumerate() { + let slot = base + 0x28 + n as u64 * 8; + let initial = (0x1234_5678_0000_0000 | seed) ^ ((head as u64) << 40); + let low = if n == 1 && head == 1 { 0x5a5a_5a5a } else { 0xa5a5_a5a5 }; + assert_eq!(r.r64(slot), seed); + r.w64(slot, initial); + assert_eq!(r.r64(slot), initial); + r.w32(slot + 4, low); + assert_eq!(r.r32(slot + 4), low); + assert_eq!(r.r64(slot), (initial & 0xffff_ffff_0000_0000) | low as u64); + r.w64(slot, initial ^ 0x55aa_55aa); + // Other widths are refused and change nothing. + assert_eq!(r.bus.write8(slot, 0x7e), BUS_OK); + assert_eq!(r.bus.write16(slot, 0x7e5a), BUS_OK); + r.w32(slot, 0x7e5a_3c1d); + assert_eq!(r.bus.read8(slot).data, 0); + assert_eq!(r.bus.read16(slot).data, 0); + assert_eq!(r.r32(slot), 0); + assert_eq!(r.r64(slot), initial ^ 0x55aa_55aa); + } + } + assert!(r.pic.unsupported().iter().any(|&(_, write, off, size)| write && off == 0x28 && size == 1)); + r.pic.power_on(); + for head in 0..2 { + assert_eq!(r.r64(head_base(head) + 0x28), 0xfffff); + assert_eq!(r.r64(head_base(head) + 0x30), 0); + assert_eq!(r.r32(head_base(head) + 0x2c), 0xfffff); + } +} + +#[test] +fn the_host_error_field_has_a_low_word_view() { + let r = rig(); + for head in 0..2u64 { + let base = head * HEAD_OFFSET; + assert_eq!(r.r64(base + 0x128), 0); + r.w64(base + 0x128, 0x1234_5678_0000_0000 | (head + 1)); + assert_eq!(r.r32(base + 0x12c) as u64, head + 1); + r.w32(base + 0x12c, 0x55 + head as u32); + assert_eq!(r.r64(base + 0x128), 0x1234_5678_0000_0000 | (0x55 + head)); + assert_eq!(r.r64(base + 0x100), 0, "the write raised no status"); + } + r.pic.power_on(); + assert_eq!(r.r64(0x128), 0); +} + +#[test] +fn dir_map_keeps_only_the_implemented_bits() { + let r = rig(); + for head in 0..2 { + let at = head_base(head) + 0x80; + r.w64(at, 0x1ffff | (0xf << 20)); + assert_eq!(r.r64(at), 0x1ffff | (0xf << 20)); + let before = r.pic.unsupported().len(); + r.w64(at, (1 << 17) | (1 << 40) | 0x1234); + assert_eq!(r.r64(at), 0x1234, "unsupported bits clear and the rest stay"); + assert!(r.pic.unsupported().len() > before, "the refusal is observable"); + } +} + +/// Device 0 is the bridge and answers nothing; the function at device `n` +/// is `n` windows up, and every miss reads all ones. +#[test] +fn configuration_cycles_reach_the_function_one_window_above_its_device() { + let r = rig(); + let a = Card::new(0x5001); + let b = Card::new(0x5002); + r.pic.attach(0, 1, a.clone(), Function::default()); + r.pic.attach(0, 3, b.clone(), Function::default()); + assert_eq!(r.r32(config(0, 1)), 0x5001_1234); + assert_eq!(r.r32(config(0, 3)), 0x5002_1234); + assert!(!r.bus.read32(config(1, 1)).is_ok(), "head 1 has its own bus"); + for device in [0, 2, 4, 5, 7] { + assert!(!r.bus.read32(config(0, device)).is_ok(), "device {device}"); + r.w32(config(0, device) + 4, 7); + } + // Bytes and halfwords of the big-endian processor land on the right + // little-endian bytes of the register. + assert_eq!(r.bus.read16(config(0, 1)).data, 0x5001, "offset 0 is the register's upper half"); + assert_eq!(r.bus.read16(config(0, 1) + 2).data, 0x1234); + assert_eq!(r.bus.read8(config(0, 1)).data, 0x50); + assert_eq!(r.bus.read8(config(0, 1) + 3).data, 0x34); + // Only function 0 answers. + assert!(!r.bus.read32(config(0, 1) + 0x100).is_ok()); + r.w32(config(0, 1) + 4, MEMORY_AND_MASTER); + assert_eq!(r.r32(config(0, 1) + 4) & 0xffff, MEMORY_AND_MASTER); + r.w32(config(0, 1) + 0x10, BAR_BASE); + assert_eq!(r.r32(config(0, 1) + 0x10), BAR_BASE); + assert_eq!(a.config.bar_base(0), BAR_BASE); +} + +#[test] +fn type_one_configuration_selects_the_bus_and_device_through_the_register() { + let r = rig(); + let a = Card::new(0x5001); + r.pic.attach(0, 2, a, Function::default()); + r.w64(0xc8, 2 << 11); + assert_eq!(r.r32(CONFIG_TYPE1), 0x5001_1234); + r.w64(0xc8, (1 << 16) | (2 << 11)); + assert!(!r.bus.read32(CONFIG_TYPE1).is_ok(), "a subordinate bus has nothing"); + r.w64(0xc8, 3 << 11); + assert!(!r.bus.read32(CONFIG_TYPE1).is_ok()); + r.w64(0xc8, 2 << 11); + assert_eq!(r.r32(CONFIG_TYPE1 + 8), 0x0280_0001, "class and revision"); +} + +/// The processor's byte lanes: an adapter wired little-endian sees the +/// numeric value of a word, and a halfword at offset 2 reaches offset 0. +#[test] +fn a_conventional_function_sees_cpu_pio_with_its_address_corrected() { + let r = rig(); + let card = Card::new(0x6001); + r.fit(0, 1, &card, Function::default()); + r.w32(window(0, 0x10), 0x1234_5678); + assert_eq!(&card.bar.lock()[0x10..0x14], [0x78, 0x56, 0x34, 0x12]); + assert_eq!(r.r32(window(0, 0x10)), 0x1234_5678); + assert_eq!(r.bus.write16(window(0, 0x22), 0x9abc), BUS_OK); + assert_eq!(&card.bar.lock()[0x20..0x22], [0xbc, 0x9a], "offset 2 reaches offset 0"); + assert_eq!(r.bus.read16(window(0, 0x22)).data, 0x9abc); + assert_eq!(r.bus.write8(window(0, 0x33), 0x5a), BUS_OK); + assert_eq!(card.bar.lock()[0x30], 0x5a, "byte offset 3 reaches offset 0"); + assert_eq!(r.bus.read8(window(0, 0x33)).data, 0x5a); + assert_eq!(r.bus.read8(window(0, 0x30)).data, 0, "offset 0 is a different byte"); + // The same card through the short window of head 0, memory space. + r.w64(0x200, (1 << 12) | (BAR_BASE >> 20) as u64); + r.w32(SHORT_BASE + 0x40, 0xcafe_f00d); + assert_eq!(&card.bar.lock()[0x40..0x44], [0x0d, 0xf0, 0xfe, 0xca]); + assert_eq!(r.r32(SHORT_BASE + 0x40), 0xcafe_f00d); +} + +/// IOC4 keeps the processor's byte sequence and the address as written. +#[test] +fn a_self_ordered_function_keeps_the_processors_byte_sequence() { + let r = rig(); + let card = Card::new(0x6002); + r.fit(1, 1, &card, Function { self_ordered_pio: true, req64: false }); + r.w32(window(1, 0x10), 0x1234_5678); + assert_eq!(&card.bar.lock()[0x10..0x14], [0x12, 0x34, 0x56, 0x78]); + assert_eq!(r.r32(window(1, 0x10)), 0x1234_5678); + assert_eq!(r.bus.write8(window(1, 0x21), 0xa5), BUS_OK); + assert_eq!(card.bar.lock()[0x21], 0xa5, "no address correction"); + assert_eq!(r.bus.read8(window(1, 0x21)).data, 0xa5); + assert_eq!(r.bus.write16(window(1, 0x22), 0x9abc), BUS_OK); + assert_eq!(&card.bar.lock()[0x22..0x24], [0x9a, 0xbc]); +} + +/// A short window follows its Device register: space, megabyte and size +/// limit, and it refuses the attributes it does not model. +#[test] +fn short_windows_follow_their_device_registers() { + let r = rig(); + let card = Card::new(0x6003); + r.fit(0, 1, &card, Function::default()); + // Window 0 and 1 are 2 MiB and ignore the low bit of the base; windows + // 2 and 3 are 1 MiB. + for (window_index, cpu) in [(0u64, 0x20_0000u64), (1, 0x40_0000), (2, 0x60_0000), (3, 0x70_0000)] { + // The two big windows ignore the base's low bit; the small ones keep it. + let base = (BAR_BASE >> 20) as u64 | if window_index < 2 { 1 } else { 0 }; + r.w64(0x200 + window_index * 8, (1 << 12) | base); + card.bar.lock().fill(0); + r.w32(cpu + 0x80, 0x1111_1111 * (window_index as u32 + 1)); + assert_eq!(le_value(&card.bar.lock()[0x80..0x84]), 0x1111_1111 * (window_index + 1), "window {window_index}"); + } + // The last word of a window fits; a doubleword across its end does not. + r.w32(0x70_0000 + 0xf_fffc, 7); + let before = r.pic.unsupported().len(); + assert_eq!(r.bus.write64(0x70_0000 + 0xf_fffc, 7), BUS_OK); + assert_eq!(r.r64(0x70_0000 + 0xf_fffc), 0, "a boundary-crossing access moves nothing"); + assert_eq!(r.pic.unsupported().len(), before, "it is logged, not listed as unsupported"); + // Swapped and wide programmed I/O are refused rather than guessed. + r.w64(0x200, (1 << 12) | (BAR_BASE >> 20) as u64 | (1 << 13)); + card.bar.lock().fill(0); + r.w32(SHORT_BASE + 0x40, 0xffff_ffff); + assert_eq!(card.bar.lock()[0x40], 0); + r.w64(0x200, (1 << 12) | (BAR_BASE >> 20) as u64 | (1 << 22)); + assert_eq!(r.r32(SHORT_BASE + 0x40), 0); + assert_eq!(card.bar.lock()[0x40], 0); +} + +#[test] +fn the_guard_after_the_second_head_reads_zero_and_the_gaps_fault() { + let r = rig(); + assert_eq!(r.r64(2 * HEAD_OFFSET), 0); + assert_eq!(r.bus.write64(2 * HEAD_OFFSET, 1), BUS_OK); + assert!(!r.bus.read64(2 * HEAD_OFFSET + 8).is_ok()); + assert!(!r.bus.read64(0x8000_0000).is_ok(), "the 64-bit alias window is not this chip's"); + assert!(!r.bus.read32(0x1_0000_0000).is_ok()); + assert!(r.bus.read32(0x20_0001).is_ok(), "short windows take unaligned accesses"); + assert!(!r.bus.read32(0x1002).is_ok(), "registers need natural alignment"); +} + +/// Both heads send direct-map DMA to the widget and address their own +/// `DIR_MAP` names. +#[test] +fn direct_map_dma_goes_where_each_heads_direct_map_says() { + let r = rig(); + let a = Card::new(0x7001); + let b = Card::new(0x7002); + r.fit(0, 1, &a, Function::default()); + r.fit(1, 1, &b, Function::default()); + r.w64(0x80, 10 << 20); + r.w64(HEAD1 + 0x80, (11 << 20) | 1); + let data: Vec = (0x11..0x19).collect(); + assert!(a.dma().dma_write(0x8000_1000, &data).is_ok()); + let other: Vec = (0x21..0x29).collect(); + assert!(b.dma().dma_write(0x8000_2000, &other).is_ok()); + assert_eq!(r.peer.bytes(10, 0x1000, 8), data); + assert_eq!(r.peer.bytes(11, (1 << 31) | 0x2000, 8), other); + assert_eq!(r.peer.bytes(10, 0x2000, 8), vec![0; 8], "the other head's target is untouched"); + let log = r.peer.dma.lock().clone(); + assert_eq!(log[0], Dma { source: 15, widget: 10, address: 0x1000, len: 8, write: true }); + // Reads come back through the same path. + let mut back = [0u8; 8]; + assert!(a.dma().dma_read(0x8000_1000, &mut back).is_ok()); + assert_eq!(back.to_vec(), data); + // A transfer at an odd address is cut into naturally aligned pieces, and + // one that crosses a page is translated again at the page. + r.peer.dma.lock().clear(); + assert!(a.dma().dma_write(0x8000_1ffd, &[1, 2, 3, 4, 5, 6]).is_ok()); + let sizes: Vec<_> = r.peer.dma.lock().iter().map(|p| (p.address, p.len)).collect(); + assert_eq!(sizes, [(0x1ffd, 1), (0x1ffe, 2), (0x2000, 2), (0x2002, 1)].map(|(a, l)| (a as u64, l))); +} + +/// A slot whose Device register sets the swap bit moves single bytes with +/// the low two address bits flipped. +#[test] +fn a_swapping_slot_flips_the_byte_address_within_each_word() { + let r = rig(); + let card = Card::new(0x7003); + r.fit(0, 1, &card, Function::default()); + r.w64(0x80, 10 << 20); + r.w64(0x200, 0x18001000 | (1 << 19)); + assert!(card.dma().dma_write(0x8000_0100, &[1, 2, 3, 4, 5, 6, 7, 8]).is_ok()); + assert_eq!(r.peer.bytes(10, 0x100, 8), [4, 3, 2, 1, 8, 7, 6, 5]); + let mut back = [0u8; 8]; + assert!(card.dma().dma_read(0x8000_0100, &mut back).is_ok()); + assert_eq!(back, [1, 2, 3, 4, 5, 6, 7, 8]); + assert!(r.peer.dma.lock().iter().all(|p| p.len == 1)); +} + +#[test] +fn dma_outside_the_direct_window_is_refused_and_a_read_comes_back_zero() { + let r = rig(); + let card = Card::new(0x7004); + r.fit(0, 1, &card, Function::default()); + r.peer.put(0, 0, &[0xaa; 16]); + assert_eq!(card.dma().dma_write(0x7fff_f000, &[1; 8]), PciDmaStatus::MasterAbort, "below the window"); + let mut buf = [0xffu8; 8]; + assert_eq!(card.dma().dma_read(0x4000_0000, &mut buf), PciDmaStatus::MasterAbort, "translated DMA is unsupported"); + assert_eq!(buf, [0; 8]); + assert!(r.peer.dma.lock().is_empty(), "nothing reached the peer"); + // A peer that refuses ends the transfer and zeroes the unread rest. + r.peer.refuse.store(true, Ordering::SeqCst); + let mut buf = [0xffu8; 8]; + assert_eq!(card.dma().dma_read(0x8000_0000, &mut buf), PciDmaStatus::MasterAbort); + assert_eq!(buf, [0; 8]); + // Device 0 and devices past 4 have no DMA path. + r.peer.refuse.store(false, Ordering::SeqCst); + assert_eq!(r.pic.heads[0].dma(0, 0x8000_0000, &mut [0; 4], false), PciDmaStatus::MasterAbort); + assert_eq!(r.pic.heads[0].dma(5, 0x8000_0000, &mut [0; 4], false), PciDmaStatus::MasterAbort); +} + +fn direct64(widget: u64, flags: u64, payload: u64) -> u64 { + (widget << 60) | flags | payload +} + +/// Bit 55 set is an ordered byte stream; clear, each 32-bit word's bytes +/// swap, or each doubleword's for a 64-bit requester. +#[test] +fn direct64_dma_keeps_a_stream_and_swaps_the_words_of_a_swap_clear_request() { + let r = rig(); + let narrow = Card::new(0x7101); + let wide = Card::new(0x7102); + r.fit(0, 1, &narrow, Function::default()); + r.fit(0, 4, &wide, Function { self_ordered_pio: false, req64: true }); + let data = [0x12u8, 0x34, 0x56, 0x78, 0x9a, 0xbc, 0xde, 0xf0]; + assert!(narrow.dma().dma_write(direct64(0xa, 1 << 55, 0x3000), &data).is_ok()); + assert_eq!(r.peer.bytes(0xa, 0x3000, 8), data, "ordered bytes unchanged"); + assert!(narrow.dma().dma_write(direct64(0xa, 0, 0x4000), &data).is_ok()); + assert_eq!(r.peer.bytes(0xa, 0x4000, 8), [0x78, 0x56, 0x34, 0x12, 0xf0, 0xde, 0xbc, 0x9a]); + assert!(wide.dma().dma_write(direct64(0xa, 0, 0x5000), &data).is_ok()); + assert_eq!(r.peer.bytes(0xa, 0x5000, 8), [0xf0, 0xde, 0xbc, 0x9a, 0x78, 0x56, 0x34, 0x12]); + assert!(wide.dma().dma_write(direct64(0xa, 1 << 55, 0x6000), &data).is_ok()); + assert_eq!(r.peer.bytes(0xa, 0x6000, 8), data, "a stream is a stream at either width"); + let mut back = [0u8; 8]; + assert!(narrow.dma().dma_read(direct64(0xa, 0, 0x4000), &mut back).is_ok()); + assert_eq!(back, data); + // Prefetch, precise, virtual channel 1 and barrier are recognized and + // ignored; DIR_MAP does not apply. + r.w64(0x80, 3 << 20); + for bits in [1 << 59, 1 << 58, 1 << 57, 1 << 56] { + assert!(narrow.dma().dma_write(direct64(0xa, bits | (1 << 55), 0x7000), &data).is_ok()); + } + assert!(r.peer.dma.lock().iter().all(|p| p.widget == 0xa)); +} + +/// A function samples REQ64# when it is fitted and again as each reset of +/// its head ends, high only in a slot the board wires for it, as the QEMU +/// head's reset exit reports its req64-slots mask to the bus. +#[test] +fn each_function_samples_its_slots_req64_at_attach_and_after_reset() { + let r = rig(); + let narrow = Card::new(0x7111); + let wide = Card::new(0x7112); + r.fit(0, 1, &narrow, Function::default()); + r.fit(0, 4, &wide, Function { self_ordered_pio: false, req64: true }); + assert_eq!(*narrow.req64.lock(), [false]); + assert_eq!(*wide.req64.lock(), [true]); + r.pic.power_on(); + assert_eq!(*narrow.req64.lock(), [false, false]); + assert_eq!(*wide.req64.lock(), [true, true]); +} + +#[test] +fn direct64_refuses_what_it_cannot_carry_before_any_packet_leaves() { + let r = rig(); + let card = Card::new(0x7103); + r.fit(1, 2, &card, Function::default()); + let sentinel = [0xa5u8; 8]; + r.peer.put(0xa, 0x1000, &sentinel); + for bit in 48..=54 { + assert_eq!(card.dma().dma_write(direct64(0xa, 1 << bit, 0x1000), &[1; 4]), PciDmaStatus::MasterAbort, "bit {bit}"); + } + assert_eq!(card.dma().dma_write(direct64(0xa, 0, (1 << 47) | 0x1000), &[1; 4]), PciDmaStatus::MasterAbort, "register space"); + assert_eq!(card.dma().dma_write(1 << 32, &[1; 4]), PciDmaStatus::MasterAbort, "target zero"); + assert!(r.peer.dma.lock().is_empty()); + assert_eq!(r.peer.bytes(0xa, 0x1000, 8), sentinel); +} + +/// A transfer that runs into the register-space boundary delivers the part +/// below it and fails at the page that crosses. +#[test] +fn direct64_splits_at_the_register_space_boundary() { + let r = rig(); + let card = Card::new(0x7104); + r.fit(0, 1, &card, Function::default()); + let boundary = 1u64 << 47; + let status = card.dma().dma_write(direct64(0xa, 1 << 55, boundary - 2), &[0x90, 0x91, 0x92, 0x93]); + assert_eq!(status, PciDmaStatus::MasterAbort); + assert_eq!(r.peer.bytes(0xa, boundary - 2, 2), [0x90, 0x91]); + assert_eq!(r.peer.dma.lock().len(), 1); +} + +#[test] +fn a_refused_direct64_write_leaves_the_target_as_it_was() { + let r = rig(); + let card = Card::new(0x7105); + r.fit(0, 1, &card, Function::default()); + r.peer.put(0xa, 0x3000, &[0xa5; 8]); + r.peer.refuse.store(true, Ordering::SeqCst); + for flags in [0, 1 << 55, 1 << 58, (1 << 56) | (1 << 55)] { + assert_eq!(card.dma().dma_write(direct64(0xa, flags, 0x3000), &[1; 8]), PciDmaStatus::MasterAbort); + } + assert_eq!(r.peer.bytes(0xa, 0x3000, 8), [0xa5; 8]); + assert!(r.peer.dma.lock().len() >= 4, "each attempt reached the peer"); +} + +/// An input that rises sets its status and, once enabled, sends the set +/// packet to its destination; a falling edge sends a clear if its mode bit +/// asks for one. +#[test] +fn a_rising_enabled_pin_sends_a_set_packet_and_a_falling_one_a_clear() { + let r = rig(); + for pin in 0..PINS as u64 { + r.w64(0x130 + pin * 8, destination(0x20 + pin)); + } + r.pic.set_pin(0, 3, true); + assert_eq!(r.r64(0x100), 1 << 3); + assert!(r.peer.interrupts.lock().is_empty(), "disabled"); + r.w64(0x108, 1 << 3); + assert_eq!(r.peer.interrupts.lock().as_slice(), [(15, 0x1800090, 0x123)], "enabling a pending input sends it"); + assert_eq!(r.pic.request_counts(0, 3), (1, 0)); + r.pic.set_pin(0, 3, false); + assert_eq!(r.pic.request_counts(0, 3), (1, 0), "no clear without the mode bit"); + r.w64(0x118, 1 << 3); + r.pic.set_pin(0, 3, true); + r.pic.set_pin(0, 3, false); + assert_eq!(r.pic.request_counts(0, 3), (2, 1)); + assert_eq!(r.peer.interrupts.lock().last().copied(), Some((15, 0x1800090, 0x23))); + // The status bit stays until software clears it, and a pin that is + // still high comes back. + assert_eq!(r.r64(0x100), 1 << 3); + r.w64(0x110, 1 << 3); + assert_eq!(r.r64(0x100), 0); + r.pic.set_pin(1, 0, true); + assert_eq!(r.r64(HEAD1 + 0x100), 1, "the heads' pins are separate"); + assert_eq!(r.r64(0x100), 0); + // Reset drops pins and status. + r.pic.power_on(); + assert_eq!(r.r64(0x100), 0); + r.pic.set_pin(0, 3, true); + assert_eq!(r.pic.request_counts(0, 3), (0, 0), "counts restart and the enable is gone"); +} + +#[test] +fn a_clear_write_resends_a_pin_that_is_still_high_and_enabled() { + let r = rig(); + r.w64(0x130, destination(9)); + r.w64(0x108, 1); + r.pic.set_pin(0, 0, true); + assert_eq!(r.pic.request_counts(0, 0), (1, 0)); + r.w64(0x110, 1); + assert_eq!(r.pic.request_counts(0, 0), (2, 0), "the status bit comes back with the level, and so does the packet"); + assert_eq!(r.r64(0x100), 1); + // A masked pin stays quiet on a clear write. + r.w64(0x108, 0); + r.w64(0x110, 1); + assert_eq!(r.pic.request_counts(0, 0), (2, 0)); +} + +#[test] +fn only_levels_a_pi_can_take_leave_the_chip() { + let r = rig(); + r.pic.set_pin(0, 0, true); + for level in [0, 6, 128, 0x7fff] { + r.w64(0x130, destination(level)); + r.w64(0x108, 0); + r.w64(0x108, 1); + assert!(r.peer.interrupts.lock().is_empty(), "level {level}"); + } + assert_eq!(r.pic.request_counts(0, 0).0, 4, "each attempt was counted"); + for level in [7, 127] { + r.w64(0x130, destination(level)); + r.w64(0x108, 0); + r.w64(0x108, 1); + } + let sent: Vec<_> = r.peer.interrupts.lock().iter().map(|p| p.2).collect(); + assert_eq!(sent, [0x107, 0x17f]); + // A peer that refuses is not an error the guest sees, but it is counted. + r.peer.refuse.store(true, Ordering::SeqCst); + r.w64(0x108, 0); + r.w64(0x108, 1); + assert_eq!(r.peer.interrupts.lock().len(), 2); +} + +/// A PCI function's INTA reaches the input the board wired it to, and a +/// function nobody wired is reported, not routed. +#[test] +fn pci_interrupts_reach_the_input_the_board_wired() { + let r = rig(); + let card = Card::new(0x7201); + let unwired = Card::new(0x7202); + r.pic.attach(0, 3, card.clone(), Function::default()); + r.pic.attach(0, 4, unwired.clone(), Function::default()); + r.pic.wire_inta(0, 3); + r.w64(0x130 + 2 * 8, destination(0x31)); + r.w64(0x108, 1 << 2); + card.raise(true); + assert_eq!(r.r64(0x100), 1 << 2, "device 3 is input 2"); + assert_eq!(r.peer.interrupts.lock().as_slice(), [(15, 0x1800090, 0x131)]); + card.raise(false); + unwired.raise(true); + assert_eq!(r.r64(0x100), 1 << 2, "device 4 leads nowhere"); + // A board routes a pin to another line when it wants. + let (probe, line) = LevelProbe::line(); + r.pic.route_pci_interrupt(0, 4, 0, line); + unwired.raise(false); + unwired.raise(true); + assert!(probe.level()); + r.pic.power_on(); + assert!(!probe.level(), "reset lowers the routed lines"); +} + +/// Both PIs, told apart the way the node does: by address bit 21. +struct ToPi([Arc; 2]); +impl PicPeer for ToPi { + fn interrupt(&self, _: u8, address: u64, payload: u64) -> Result<(), Refused> { + self.0[(address >> 21) as usize & 1].write64(pi_reg::PENDING_MODIFY, payload); + Ok(()) + } + fn dma_read(&self, _: u8, _: u8, _: u64, _: &mut [u8]) -> Result<(), Refused> { + Err(Refused) + } + fn dma_write(&self, _: u8, _: u8, _: u64, _: &[u8]) -> Result<(), Refused> { + Err(Refused) + } +} + +/// The PI is told which inputs are active, so it can raise a level again +/// after the processor clears it while the line stays high. +#[test] +fn the_pi_reasserts_a_level_whose_input_is_still_active() { + let (pi, now) = pi_with_clock(); + let pic = Pic::new(15); + pic.connect_peer(Arc::new(ToPi([pi.clone(), pi_with_clock().0]))).unwrap(); + pic.connect_pi(0, pi.clone(), 0); + let bus = pic.widget(); + bus.write64(0x130, destination(12)); + bus.write64(0x108, 1); + pic.set_pin(0, 0, true); + assert_eq!(pi.read64(pi_reg::PENDING_STATUS_A).data, 1 << 12); + // The processor clears its pending bit; the input is still high. + pi.write64(pi_reg::PENDING_MODIFY, 12); + assert_eq!(pi.read64(pi_reg::PENDING_STATUS_A).data, 0); + now.store(999_999, Ordering::SeqCst); + assert_eq!(pi.read64(pi_reg::PENDING_STATUS_A).data, 0, "not before a millisecond"); + now.store(1_000_000, Ordering::SeqCst); + assert_eq!(pi.read64(pi_reg::PENDING_STATUS_A).data, 1 << 12); + // Dropping the input ends the reassertion. + pi.write64(pi_reg::PENDING_MODIFY, 12); + pic.set_pin(0, 0, false); + now.store(5_000_000, Ordering::SeqCst); + assert_eq!(pi.read64(pi_reg::PENDING_STATUS_A).data, 0); +} + +/// Each level keeps its own deadline, a masked input is not reasserted and +/// a destination in the other PI's window leaves this one alone. +#[test] +fn levels_reassert_independently_and_only_while_enabled() { + let (pi, now) = pi_with_clock(); + let (other, other_now) = pi_with_clock(); + let pic = Pic::new(15); + pic.connect_peer(Arc::new(ToPi([pi.clone(), other.clone()]))).unwrap(); + pic.connect_pi(0, pi.clone(), 0); + pic.connect_pi(1, other.clone(), 0); + let bus = pic.widget(); + for (pin, level) in [(0u64, 12u64), (1, 13)] { + bus.write64(0x130 + pin * 8, destination(level)); + } + // Pin 2 targets the second PI (destination bit 21). + bus.write64(0x140, destination(14) | (1 << 21)); + bus.write64(0x108, 0b111); + for pin in 0..3 { + pic.set_pin(0, pin, true); + } + pi.write64(pi_reg::PENDING_MODIFY, 12); + now.store(400_000, Ordering::SeqCst); + pi.write64(pi_reg::PENDING_MODIFY, 13); + now.store(1_000_000, Ordering::SeqCst); + let pending = pi.read64(pi_reg::PENDING_STATUS_A).data; + assert_eq!(pending & (1 << 12), 1 << 12, "level 12 is due"); + assert_eq!(pending & (1 << 13), 0, "level 13 is not"); + now.store(1_400_000, Ordering::SeqCst); + assert_eq!(pi.read64(pi_reg::PENDING_STATUS_A).data & (1 << 13), 1 << 13); + // Masking an input stops its level from coming back. + pi.write64(pi_reg::PENDING_MODIFY, 12); + bus.write64(0x108, 0b110); + now.store(3_000_000, Ordering::SeqCst); + assert_eq!(pi.read64(pi_reg::PENDING_STATUS_A).data & (1 << 12), 0); + // The destination of pin 2 is the other PI's, so this one never held it. + assert_eq!(pi.read64(pi_reg::PENDING_STATUS_A).data & (1 << 14), 0); + assert_eq!(other.read64(pi_reg::PENDING_STATUS_A).data & (1 << 14), 1 << 14); + other.write64(pi_reg::PENDING_MODIFY, 14); + other_now.store(1_000_000, Ordering::SeqCst); + assert_eq!(other.read64(pi_reg::PENDING_STATUS_A).data & (1 << 14), 1 << 14, "the other PI reasserts it"); +} + +/// The PROM writes the link-side error address slot's low word; the chip +/// latches the invalid-address status and, if enabled, sends one packet to +/// the destination the widget registers name. +#[test] +fn an_invalid_address_write_latches_status_and_sends_one_packet() { + let r = rig(); + let invalid = 1u64 << 24; + // Other widths and the doubleword form of the slot do nothing. + r.bus.write8(0x7c, 0x11); + r.bus.write16(0x7c, 0x2233); + r.bus.write32(0x78, 0x4455_6677); + r.w64(0x78, 0x8899_aabb_ccdd_eeff); + assert_eq!(r.r64(0x100), 0); + r.w32(0x7c, 0xdead_beef); + assert_eq!(r.r32(0x104) as u64, invalid); + assert_eq!(r.r32(HEAD1 + 0x104), 0, "per head"); + assert!(r.peer.interrupts.lock().is_empty(), "not enabled"); + r.w32(0x114, invalid as u32); + assert_eq!(r.r32(0x104), 0, "written one to clear"); + let target = 0x1_0000_3000u64; + r.w64(0x30, 0x3400_0000); + r.w64(0x38, target); + r.w32(0x10c, invalid as u32); + for bad in [0x80u64, 0x100] { + r.w64(0x128, bad); + r.w32(0x7c, 0); + assert!(r.peer.interrupts.lock().is_empty(), "interrupt number {bad:#x}"); + } + r.w64(0x128, 0x65); + r.w32(0x7c, 0); + assert_eq!(r.peer.interrupts.lock().as_slice(), [(15, target, 0x165)]); + r.w32(0x7c, 0); + assert_eq!(r.peer.interrupts.lock().len(), 2, "every request sends while the status stays latched"); + // A repeat of the lower register's bits 47:32 in the upper register is + // accepted; a different one, or a lone upper word, is not. + r.w64(0x30, 1); + r.w32(0x7c, 0); + assert_eq!(r.peer.interrupts.lock().len(), 3); + r.w64(0x30, 2); + r.w32(0x7c, 0); + r.w64(0x30, 0x3400_0000); + r.w64(0x38, 0); + r.w32(0x7c, 0); + r.w64(0x30, 0x1_0000_0001); + r.w64(0x38, target); + r.w32(0x7c, 0); + r.w64(0x30, 0x3400_0000); + r.w64(0x38, 0x000f_0001_0000_3000); + r.w32(0x7c, 0); + assert_eq!(r.peer.interrupts.lock().len(), 3, "conflicts, zero and overflow are refused"); + // The PROM's two word writes join into the address. + r.w64(0x30, 0); + r.w64(0x38, 0); + r.w32(0x34, 1); + r.w32(0x3c, 0x3000); + r.w32(0x7c, 0); + assert_eq!(r.peer.interrupts.lock().last().map(|p| p.1), Some(target)); +} + +#[test] +fn two_pics_are_independent_devices() { + let a = Pic::new(11); + let b = Pic::new(15); + let peer_a = Arc::new(Peer::default()); + let peer_b = Arc::new(Peer::default()); + a.connect_peer(peer_a.clone()).unwrap(); + b.connect_peer(peer_b.clone()).unwrap(); + assert_eq!(a.connect_peer(peer_b.clone()), Err(AlreadyConnected)); + assert_eq!((a.widget_number(), b.widget_number()), (11, 15)); + let (bus_a, bus_b) = (a.widget(), b.widget()); + bus_a.write64(0x20, 0x0f00_3205); + assert_eq!(bus_b.read64(0x20).data, 0x0f00_320f, "state is per chip"); + bus_a.write64(0x130, destination(9)); + bus_a.write64(0x108, 1); + a.set_pin(0, 0, true); + assert_eq!(peer_a.interrupts.lock().as_slice(), [(5, 0x1800090, 0x109)], "the packet carries the control widget field"); + assert!(peer_b.interrupts.lock().is_empty()); + let card = Card::new(1); + a.attach(0, 1, card.clone(), Function::default()); + assert!(!bus_b.read32(config(0, 1)).is_ok(), "the other chip's slot stays empty"); +} + +// An empty slot's configuration read fails as a master timeout. QEMU's +// sgi-pic.c returns all ones with no fault and has no counterpart case. + +const MASTER_TIMEOUT_BIT: u64 = 1 << 11; +const STATUS: u64 = 0x100; +const ENABLE: u64 = 0x108; +const RESET: u64 = 0x110; +const VIEW: u64 = 0x170; + +/// With every enable clear, as the PROM probes: an empty device fails on +/// either head, function 0 or 1, for every width the processor uses. +/// Error view bit 11 records it and status bit 11 stays clear. +#[test] +fn an_empty_slot_read_faults_and_only_the_error_view_records_it() { + let r = rig(); + for head in [0, 1] { + let base = head_base(head); + for (device, function) in [(2u64, 0u64), (2, 1), (0, 0), (5, 1)] { + let at = config(head, device) + function * 0x100; + assert!(!r.bus.read32(at).is_ok(), "head {head} device {device}.{function}"); + assert!(!r.bus.read16(at + 2).is_ok()); + assert!(!r.bus.read8(at + 3).is_ok()); + assert_eq!(r.r64(base + VIEW) & MASTER_TIMEOUT_BIT, MASTER_TIMEOUT_BIT, "head {head}"); + assert_eq!(r.r32(base + VIEW + 4) as u64 & MASTER_TIMEOUT_BIT, MASTER_TIMEOUT_BIT, "low-word view"); + assert_eq!(r.r64(base + STATUS) & MASTER_TIMEOUT_BIT, 0, "status needs the enable"); + assert_eq!(r.r32(base + STATUS + 4) as u64 & MASTER_TIMEOUT_BIT, 0); + // The PROM's low-word write of 0x41 clears the view. + r.w32(base + RESET + 4, 0x41); + assert_eq!(r.r64(base + VIEW), 0); + } + // The other head's view is its own. + assert!(!r.bus.read32(config(head, 2)).is_ok()); + assert_eq!(r.r64(head_base(1 - head) + VIEW), 0); + r.w64(base + RESET, 0x41); + assert_eq!(r.r64(base + VIEW), 0, "an aligned write clears it too"); + } + // The view is read-only and an unclaimed write records nothing. + r.w64(VIEW, MASTER_TIMEOUT_BIT); + r.w32(config(0, 2) + 4, 7); + assert_eq!(r.r64(VIEW), 0); + assert!(r.pic.unsupported().iter().any(|&(_, write, off, _)| write && off == VIEW)); + assert!(r.peer.interrupts.lock().is_empty()); +} + +/// With enable bit 11 set, status bit 11 sets too and the error interrupt +/// goes out. Status clears one bit at a time: the PROM's 0x41 leaves it set, +/// a one at bit 11 clears it. +#[test] +fn an_enabled_master_timeout_sets_status_until_bit_11_is_written() { + let r = rig(); + r.w64(0x38, 0x1800090); + r.w64(0x128, 9); + r.w64(ENABLE, MASTER_TIMEOUT_BIT); + assert!(!r.bus.read32(config(0, 2)).is_ok()); + assert_eq!(r.r64(STATUS) & MASTER_TIMEOUT_BIT, MASTER_TIMEOUT_BIT); + assert_eq!(r.r64(VIEW) & MASTER_TIMEOUT_BIT, MASTER_TIMEOUT_BIT); + assert_eq!(r.peer.interrupts.lock().len(), 1, "one error interrupt"); + r.w32(RESET + 4, 0x41); + assert_eq!(r.r64(STATUS) & MASTER_TIMEOUT_BIT, MASTER_TIMEOUT_BIT, "0x41 leaves status bit 11"); + assert_eq!(r.r64(VIEW), 0, "but clears the view"); + r.w64(RESET, MASTER_TIMEOUT_BIT); + assert_eq!(r.r64(STATUS) & MASTER_TIMEOUT_BIT, 0); + assert_eq!(r.r64(HEAD1 + STATUS) & MASTER_TIMEOUT_BIT, 0, "head 1 untouched"); +} + +/// A populated device after an empty one still reads present once the +/// PROM's 0x41 has been written, as widget 15's second head lists device 3 +/// after an empty device 2 on a real Tezro. A present single-function card +/// has header-type bit 7 clear, so the PROM never asks for its function 1, +/// which would fault. +#[test] +fn a_device_after_an_empty_one_is_still_found() { + let r = rig(); + let card = Card::new(0x5003); + r.pic.attach(1, 3, card, Function::default()); + let probe = |device: u64| { + r.w32(HEAD1 + RESET + 4, 0x41); + let read = r.bus.read32(config(1, device)); + let absent = !read.is_ok() || r.r32(HEAD1 + STATUS + 4) as u64 & MASTER_TIMEOUT_BIT != 0; + (!absent).then_some(read.data) + }; + assert_eq!(probe(1), None); + assert_eq!(probe(2), None); + assert_eq!(probe(3), Some(0x5003_1234)); + assert_eq!(probe(4), None); + assert_eq!(r.r32(config(1, 3) + 0xc) >> 16 & 0x80, 0, "not multifunction"); + assert!(!r.bus.read32(config(1, 3) + 0x100).is_ok(), "its function 1 is unclaimed"); +} + +/// A reset clears the view with the rest of the head. +#[test] +fn reset_clears_the_error_view() { + let r = rig(); + assert!(!r.bus.read32(config(0, 2)).is_ok()); + assert_ne!(r.r64(VIEW), 0); + r.pic.power_on(); + assert_eq!(r.r64(VIEW), 0); +} + +/// Two functions that decode the same range: the window mapped last answers, +/// and the other answers again when it goes away. +/// (cpu-pio-overlapping-bars-use-active-function) +#[test] +fn overlapping_bars_go_to_the_function_mapped_last() { + let r = rig(); + let first = Card::new(0x8001); + let second = Card::new(0x8002); + r.fit(0, 1, &first, Function::default()); + r.fit(0, 2, &second, Function::default()); + let at = window(0, 0x10); + r.w32(at, 0x40); + assert_eq!(le_value(&second.bar.lock()[0x10..0x14]), 0x40, "device 2 was mapped after device 1"); + assert_eq!(le_value(&first.bar.lock()[0x10..0x14]), 0); + // Moving the first window away and back maps it again, now last. + r.w32(config(0, 1) + 0x10, 0x0200_0000); + r.w32(config(0, 1) + 0x10, BAR_BASE); + r.w32(at, 0x20); + assert_eq!(le_value(&first.bar.lock()[0x10..0x14]), 0x20); + assert_eq!(le_value(&second.bar.lock()[0x10..0x14]), 0x40); + assert_eq!(r.r32(at), 0x20); + // Switching the winner off leaves the other one. + r.w32(config(0, 1) + 4, 0); + assert_eq!(r.r32(at), 0x40); + // A configuration write that changes nothing about a window does not + // remap it. + r.w32(config(0, 2) + 0x3c, 0x0102); + r.w32(config(0, 1) + 4, MEMORY_AND_MASTER); + assert_eq!(r.r32(at), 0x20, "enabling device 1 again maps it last"); + r.w32(config(0, 2) + 0x3c, 0x0103); + assert_eq!(r.r32(at), 0x20); +} diff --git a/src/dev/prom_image.rs b/src/dev/prom_image.rs new file mode 100644 index 000000000..1a50f89b9 --- /dev/null +++ b/src/dev/prom_image.rs @@ -0,0 +1,210 @@ +//! SGI PROM image loading: version-six single-segment containers and raw +//! payloads. A recognized but malformed container is rejected rather than +//! retried as a raw image. +//! +//! Ported from hw/mips/sgi/prom-image.c in jamesbraid/qemu-sgi-origami. + +use std::io::{self, Read}; + +fn invalid(message: &str) -> io::Error { + io::Error::new(io::ErrorKind::InvalidData, message) +} +fn word(data: &[u8], off: usize) -> u64 { + u64::from_be_bytes(data[off..off + 8].try_into().unwrap()) +} +fn byte_sum(data: &[u8]) -> u64 { + data.iter().map(|b| *b as u64).sum() +} + +/// Read a PROM file and decode its payload, retaining the path in errors. +pub fn load(path: &std::path::Path, payload_limit: usize) -> io::Result> { + let with_path = |error: io::Error| { + io::Error::new(error.kind(), format!("PROM image {}: {error}", path.display())) + }; + let file = std::fs::File::open(path).map_err(with_path)?; + // Bound the encoded file by zlib's compression bound, the container + // header, gzip framing and the accepted zero tail. + let size = payload_limit as u64; + let file_limit = size + .saturating_add(size >> 12) + .saturating_add(size >> 14) + .saturating_add(size >> 25) + .saturating_add(13 + 4096 + 12 + 7); + if file.metadata().map_err(with_path)?.len() > file_limit { + return Err(with_path(invalid( + "encoded PROM image exceeds flash capacity", + ))); + } + let mut bytes = Vec::new(); + file.take(file_limit.saturating_add(1)) + .read_to_end(&mut bytes) + .map_err(with_path)?; + if bytes.len() as u64 > file_limit { + return Err(with_path(invalid( + "encoded PROM image exceeds flash capacity", + ))); + } + decode(&bytes, payload_limit).map_err(with_path) +} + +/// Accept raw payload bytes or the supported stored/gzip container subset. +/// Gzip output is bounded before decoding, and exactly one member is required. +pub fn decode(file: &[u8], payload_limit: usize) -> io::Result> { + if file.len() < 0x48 || word(file, 0x40) != 0x4a464b535743534d { + if file.len() > payload_limit { + return Err(invalid("PROM image exceeds flash capacity")); + } + return Ok(file.to_vec()); + } + if file.len() < 4096 { + return Err(invalid("truncated SGI PROM container header")); + } + if word(file, 0x48) != 6 { + return Err(invalid("unsupported SGI PROM container version")); + } + if word(file, 0x50) != file.len() as u64 { + return Err(invalid("SGI PROM declared end does not match file size")); + } + if u32::from_be_bytes(file[0x7c..0x80].try_into().unwrap()) != 1 { + return Err(invalid("unsupported SGI PROM segment count")); + } + if word(file, 0x98) != 4096 { + return Err(invalid("unsupported SGI PROM payload offset")); + } + let flags = word(file, 0x90); + if flags != 0x10 && flags != 0x13 { + return Err(invalid("unsupported SGI PROM flags")); + } + let length = word(file, 0xb0); + if length == 0 || length > payload_limit as u64 { + return Err(invalid( + "SGI PROM decoded length is zero or exceeds capacity", + )); + } + let payload = if flags == 0x10 { + if word(file, 0xb8) != 0 || word(file, 0xc8) != 0 || word(file, 0xd0) != length { + return Err(invalid("unsupported SGI PROM uncompressed metadata")); + } + if length != (file.len() - 4096) as u64 { + return Err(invalid("invalid SGI PROM stored length")); + } + file[4096..].to_vec() + } else { + let stored = word(file, 0xb8); + if stored == 0 || stored > (file.len() - 4096) as u64 { + return Err(invalid("invalid SGI PROM stored length")); + } + let end = 4096 + stored as usize; + if file.len() - end > 7 || file[end..].iter().any(|b| *b != 0) { + return Err(invalid("unsupported SGI PROM trailing bytes")); + } + if byte_sum(&file[4096..end]) != word(file, 0xc8) { + return Err(invalid("SGI PROM stored checksum mismatch")); + } + let mut decoder = flate2::bufread::GzDecoder::new(&file[4096..end]); + let mut decoded = Vec::new(); + (&mut decoder).take(length + 1).read_to_end(&mut decoded)?; + if decoded.len() as u64 != length || !decoder.get_ref().is_empty() { + return Err(invalid("invalid SGI PROM gzip segment or decoded length")); + } + decoded + }; + if byte_sum(&payload) != word(file, 0xc0) { + return Err(invalid("SGI PROM decoded checksum mismatch")); + } + Ok(payload) +} + +#[cfg(test)] +mod tests { + // Cases follow tests/unit/test-sgi-prom-image.c in + // jamesbraid/qemu-sgi-origami: wrapped, gzip, raw, malformed and files. + use super::*; + use std::io::Write; + fn put(v: &mut [u8], off: usize, n: u64) { + v[off..off + 8].copy_from_slice(&n.to_be_bytes()); + } + fn container(payload: &[u8], gzip: bool) -> Vec { + let stored = if gzip { + let mut enc = flate2::write::GzEncoder::new(Vec::new(), flate2::Compression::default()); + enc.write_all(payload).unwrap(); + enc.finish().unwrap() + } else { + payload.to_vec() + }; + let mut v = vec![0; 4096]; + put(&mut v, 0x40, 0x4a464b535743534d); + put(&mut v, 0x48, 6); + put(&mut v, 0x50, (4096 + stored.len()) as u64); + v[0x7c..0x80].copy_from_slice(&1u32.to_be_bytes()); + put(&mut v, 0x90, if gzip { 0x13 } else { 0x10 }); + put(&mut v, 0x98, 4096); + put(&mut v, 0xb0, payload.len() as u64); + put(&mut v, 0xc0, byte_sum(payload)); + if gzip { + put(&mut v, 0xb8, stored.len() as u64); + put(&mut v, 0xc8, byte_sum(&stored)); + } else { + put(&mut v, 0xd0, payload.len() as u64); + } + v.extend(stored); + v + } + /// Equivalent PROM payloads must boot from each supported image format. + #[test] + fn raw_stored_and_gzip_images_load_with_the_same_capacity_limit() { + let path = std::env::temp_dir().join(format!("iris-prom-load-{}", std::process::id())); + for file in [vec![9, 8, 7, 6], container(&[9, 8, 7, 6], false), container(&[9, 8, 7, 6], true)] { + std::fs::write(&path, file).unwrap(); + assert_eq!(load(&path, 4).unwrap(), [9, 8, 7, 6]); + assert!(load(&path, 3).is_err(), "decoded payload exceeds capacity"); + } + std::fs::write(&path, []).unwrap(); + assert!(load(&path, 4).unwrap().is_empty()); + std::fs::remove_file(path).unwrap(); + } + + /// An invalid PROM must identify its input file instead of executing a + /// malformed container. + #[test] + fn invalid_or_unavailable_images_report_their_path() { + let path = std::env::temp_dir().join(format!("iris-prom-invalid-{}", std::process::id())); + let error = load(&path, 4).unwrap_err(); + assert_eq!(error.kind(), io::ErrorKind::NotFound); + assert!(error.to_string().contains(&path.display().to_string())); + let reject = |file: &[u8]| { + std::fs::write(&path, file).unwrap(); + let error = load(&path, 4).unwrap_err(); + assert_eq!(error.kind(), io::ErrorKind::InvalidData); + assert!(error.to_string().contains(&path.display().to_string())); + }; + reject(&vec![0; 5000]); // encoded file bound + let original = container(&[4, 3, 2, 1], false); + reject(&original[..100]); // recognized but truncated header + for (off, n) in [(0x48, 7), (0x50, 4096), (0x90, 0x14), (0x98, 4095), (0xb0, 0), (0xd0, 3)] { + let mut file = original.clone(); + put(&mut file, off, n); + reject(&file); + } + for gzip in [false, true] { + let mut file = container(&[4, 3, 2, 1], gzip); + file[0xc7] ^= 1; + reject(&file); + } + let gzip = container(&[4, 3, 2, 1], true); + for tail in [7, 8] { + let mut file = gzip.clone(); + file.extend(vec![0; tail]); + let size = file.len(); + put(&mut file, 0x50, size as u64); + if tail == 7 { + std::fs::write(&path, &file).unwrap(); + assert_eq!(load(&path, 4).unwrap(), [4, 3, 2, 1]); + } else { + reject(&file); + } + } + std::fs::remove_file(path).unwrap(); + } + +} diff --git a/src/dev/ps2.rs b/src/dev/ps2.rs index fa24f7b6e..068e70084 100644 --- a/src/dev/ps2.rs +++ b/src/dev/ps2.rs @@ -57,8 +57,25 @@ struct Ps2State { const MOUSE_MAX: i32 = 127; /// Combined PS/2 Keyboard and Mouse Controller +/// Another input device on the same host keyboard and mouse: a machine whose +/// guest may use USB instead of PS/2 (the Fuel) fits one beside the 8042 or +/// IOC3 controller, and host input reaches both. Each device drops what its +/// guest has not enabled, as the PS/2 side does. +pub trait HostInputSink: Send + Sync { + fn key(&self, key: KeyCode, pressed: bool); + /// Buttons in PS/2 order (left 1, right 2, middle 4); `dy` is positive + /// down the screen. + fn mouse(&self, buttons: u8, dx: i32, dy: i32, dz: i32); + /// Whether the guest has enabled this mouse. + fn mouse_enabled(&self) -> bool; + /// Mouse movement queued that the guest has not read yet. + fn mouse_pending(&self) -> usize; +} + pub struct Ps2Controller { state: Mutex, + /// Other devices fed from the same host input. + sinks: Mutex>>, /// The controller's interrupt: up while a byte waits to be read. irq: Option, running: AtomicBool, @@ -91,6 +108,7 @@ impl Ps2Controller { held_keys: [false; 256], }), irq, + sinks: Mutex::new(Vec::new()), running: AtomicBool::new(false), layout: AtomicU8::new(KeyboardLayout::Ansi as u8), } @@ -526,17 +544,27 @@ impl Ps2Controller { /// clicks are dropped. pub fn mouse_enabled(&self) -> bool { let state = self.state.lock(); - state.mouse_enabled && state.config & 0x20 == 0 + (state.mouse_enabled && state.config & 0x20 == 0) + || self.sinks.lock().iter().any(|s| s.mouse_enabled()) + } + + /// Feed host keyboard and mouse input to `sink` as well. + pub fn add_sink(&self, sink: Arc) { + self.sinks.lock().push(sink); } /// Mouse bytes queued that the guest has not read yet. pub fn mouse_pending(&self) -> usize { - self.state.lock().mouse_queue_bytes + let own = self.state.lock().mouse_queue_bytes; + self.sinks.lock().iter().fold(own, |n, s| n.max(s.mouse_pending())) } /// Push a scancode byte to the keyboard queue (called from UI) pub fn push_kb(&self, key: KeyCode, pressed: bool) { if !self.running.load(Ordering::Relaxed) { return; } + for sink in self.sinks.lock().iter() { + sink.key(key, pressed); + } let mut state = self.state.lock(); if !state.scanning_enabled || state.config & 0x10 != 0 { return; @@ -968,6 +996,9 @@ impl Ps2Controller { /// - `dz`: scroll wheel delta (positive = scroll up/away from user) pub fn push_mouse_input(&self, buttons: u8, dx: i32, dy: i32, dz: i32) { if !self.running.load(Ordering::Relaxed) { return; } + for sink in self.sinks.lock().iter() { + sink.mouse(buttons, dx, dy, dz); + } let mut state = self.state.lock(); if !state.mouse_enabled || state.config & 0x20 != 0 { return; } @@ -1348,6 +1379,41 @@ impl Saveable for Ps2Controller { mod tests { use super::*; + /// A sink sees host keys and mouse moves even while the guest has not + /// enabled the PS/2 devices, and its mouse counts toward `mouse_enabled` + /// and `mouse_pending`. + #[test] + fn a_sink_shares_the_host_input() { + #[derive(Default)] + struct Sink(Mutex>); + impl HostInputSink for Sink { + fn key(&self, key: KeyCode, pressed: bool) { + self.0.lock().push(format!("{key:?} {pressed}")); + } + fn mouse(&self, buttons: u8, dx: i32, dy: i32, dz: i32) { + self.0.lock().push(format!("mouse {buttons} {dx} {dy} {dz}")); + } + fn mouse_enabled(&self) -> bool { + true + } + fn mouse_pending(&self) -> usize { + 5 + } + } + let p = Ps2Controller::new(None); + let sink = Arc::new(Sink::default()); + p.add_sink(sink.clone()); + p.push_kb(KeyCode::KeyA, true); + assert!(sink.0.lock().is_empty(), "a stopped controller passes nothing on"); + p.start(); + assert!(p.mouse_enabled(), "the sink's mouse is on though the PS/2 mouse is not"); + assert_eq!(p.mouse_pending(), 5); + p.push_kb(KeyCode::KeyA, true); + p.push_mouse_input(2, 3, -4, 1); + assert_eq!(*sink.0.lock(), ["KeyA true", "mouse 2 3 -4 1"]); + assert_eq!(p.pop_from(false), None, "and the PS/2 keyboard, not enabled, took nothing"); + } + /// `type_char` sends what the keys themselves would: Shift around a /// shifted character, Enter for a newline, and nothing for a character /// with no key. diff --git a/src/dev/pxbow.rs b/src/dev/pxbow.rs new file mode 100644 index 000000000..913a40e30 --- /dev/null +++ b/src/dev/pxbow.rs @@ -0,0 +1,181 @@ +//! PXBOW, the crossbar-like widget 0 that sits behind the PIC on Tezro and +//! the Origin 350, and the forwarding of processor accesses through it. +//! +//! The model reuses the Crossbow ([`Xbow`]) at revision 5 behaviour with +//! PXBOW's identity word `0x3d100001` (revision 3, part `0xd100`, +//! manufacturer 0), and a source field in its +//! status reads. That is a software choice, not a claim that the part is a +//! Crossbow 2.0 or that the processor's route to widget 0 passes a real +//! switch: its physical route, fitted revision and register set beyond the +//! reused ones are unknown and stay unsupported and observable +//! ([`Xbow::unsupported`]). +//! +//! The chip knows two links when it is built: the host's widget (Bedrock +//! IO, 10) and the PIC's own. A further widget, such as the second PIC at +//! widget 11 or the optional V12 at widget 12 on the two- and +//! four-processor Tezro, becomes reachable when the machine attaches it +//! with [`Pxbow::attach_endpoint`]; until then its link is absent and an +//! access to it is a bus error. Each processor access to a widget through +//! [`Pxbow::route`] needs both the host's link and the target's to be +//! alive, so a link reset takes the PIC out of reach for the 20 ns of its +//! pulse. +//! +//! Ported from hw/pci-host/sgi-pic.c and hw/misc/sgi-xbow.c in +//! jamesbraid/qemu-sgi-origami. + +use super::heart::Clock; +use super::xbow::{PacketSink, Xbow}; +use crate::traits::{BusDevice, BusRead16, BusRead32, BusRead64, BusRead8, BUS_ERR}; +use parking_lot::RwLock; +use std::sync::Arc; + +/// The Crossbow revision the PIC's PXBOW selects for behaviour. +pub const REVISION: u32 = 5; +/// The revision its identification word reports. Established from real +/// Tezro, Origin 350 and Onyx 4 PROM inventories (revision 3). +pub const REPORTED_REVISION: u32 = 3; +// Part and manufacturer are INFERRED; a 2009 OpenBSD dmesg names the part +// Bedrock, a conflict that stays open. +pub const PART: u32 = 0xd100; +pub const MANUFACTURER: u32 = 0; + +/// Why a widget could not be attached. +#[derive(Debug, PartialEq, Eq)] +pub enum AttachError { + /// Links carry widgets 8 to 15 only. + NotALink(u8), + /// The host's own widget has no endpoint behind the switch. + Host(u8), + /// The widget already has an endpoint. + Occupied(u8), +} + +pub struct Pxbow { + xbow: Arc, + host: u8, + pic: u8, + endpoints: RwLock<[Option>; 16]>, +} + +impl Pxbow { + /// A PXBOW reached from `host` (the widget Bedrock IO answers as) with + /// its PIC at `pic`. Both links are present and alive. `clock` is the + /// machine's virtual time in nanoseconds, which times link resets. + pub fn new(host: u8, pic: u8, clock: Clock) -> Result, AttachError> { + for widget in [host, pic] { + if !(8..16).contains(&widget) { + return Err(AttachError::NotALink(widget)); + } + } + if host == pic { + return Err(AttachError::Occupied(pic)); + } + let xbow = Xbow::new(REVISION, (1 << host) | (1 << pic)) + .with_identity(PART, MANUFACTURER) + .with_reported_revision(REPORTED_REVISION) + .with_status_source(host) + // PROM 6.210's crossbar check writes the read-only lower error + // address on purpose. + .with_posted_error_address_writes() + .with_clock(clock); + Ok(Arc::new(Self { + xbow: Arc::new(xbow), + host, + pic, + endpoints: RwLock::new(std::array::from_fn(|_| None)), + })) + } + + pub fn host_widget(&self) -> u8 { + self.host + } + + pub fn pic_widget(&self) -> u8 { + self.pic + } + + /// The register block the processor reaches as widget 0. + pub fn xbow(&self) -> &Arc { + &self.xbow + } + + /// Widget 0 as a bus device, for Bedrock IO's widget 0. + pub fn widget_zero(&self) -> Arc { + self.xbow.clone() + } + + /// Where widget 0's own interrupt goes when it targets a widget other + /// than HEART. Once, before the machine runs. + pub fn connect_packets(&self, sink: PacketSink) { + self.xbow.connect_packets(sink); + } + + /// Put `device` behind the switch as `widget` and bring its link up. + /// The PIC's own endpoint is set this way too. + pub fn attach_endpoint(&self, widget: u8, device: Arc) -> Result<(), AttachError> { + if !(8..16).contains(&widget) { + return Err(AttachError::NotALink(widget)); + } + if widget == self.host { + return Err(AttachError::Host(widget)); + } + let mut endpoints = self.endpoints.write(); + if endpoints[widget as usize].is_some() { + return Err(AttachError::Occupied(widget)); + } + endpoints[widget as usize] = Some(device); + drop(endpoints); + self.xbow.attach(widget); + Ok(()) + } + + /// Widgets with an endpoint, as a mask with bit n for widget n. + pub fn endpoint_widgets(&self) -> u16 { + self.endpoints.read().iter().enumerate().fold(0, |m, (w, e)| m | (u16::from(e.is_some()) << w)) + } + + /// `widget` as the processor reaches it through the switch: forwarded + /// to the endpoint while the host's link and the widget's are alive, a + /// bus error otherwise. + pub fn route(self: &Arc, widget: u8) -> Arc { + assert!((8..16).contains(&widget)); + Arc::new(Routed { pxbow: self.clone(), widget }) + } +} + +struct Routed { + pxbow: Arc, + widget: u8, +} + +impl Routed { + /// The endpoint, if the route is open. + fn open(&self) -> Option> { + let p = &self.pxbow; + if !p.xbow.carries(p.host) || !p.xbow.carries(self.widget) { + return None; + } + p.endpoints.read()[self.widget as usize].clone() + } +} + +macro_rules! forward { + ($r:ident, $w:ident, $t:ty, $out:ident) => { + fn $r(&self, addr: u64) -> $out { + self.open().map_or_else($out::err, |d| d.$r(addr)) + } + fn $w(&self, addr: u64, value: $t) -> u32 { + self.open().map_or(BUS_ERR, |d| d.$w(addr, value)) + } + }; +} + +impl BusDevice for Routed { + forward!(read8, write8, u8, BusRead8); + forward!(read16, write16, u16, BusRead16); + forward!(read32, write32, u32, BusRead32); + forward!(read64, write64, u64, BusRead64); +} + +#[cfg(test)] +mod tests; diff --git a/src/dev/pxbow/tests.rs b/src/dev/pxbow/tests.rs new file mode 100644 index 000000000..d7e026387 --- /dev/null +++ b/src/dev/pxbow/tests.rs @@ -0,0 +1,239 @@ +// Cases follow tests/qtest/sn-builder-test.c in jamesbraid/qemu-sgi-origami: +// ip53-widget-zero-pxbow-identity, ip53-widget-zero-status-source and +// ip53-pxbow-route-fences-and-interrupt, and the widget-12 absence check of +// tests/qtest/sgi-graphics-pipes-test.c (ip53-v12). The qtest's machine +// properties, link-pulse timing and migration have no counterpart here. +use super::*; +use crate::dev::pic::{PicPeer, Refused, Pic}; +use crate::traits::{BusRead32, BUS_OK}; +use parking_lot::Mutex; +use std::sync::atomic::{AtomicU64, Ordering}; + +const STATUS: u64 = 0x0c; +const STATUS_CLEAR: u64 = 0x54; +const LINK_STATUS_ALIVE: u32 = 1 << 31; +const AUX_PRESENT: u32 = 1 << 5; + +/// A clock the test advances, in nanoseconds. +fn clock() -> (Clock, Arc) { + let now = Arc::new(AtomicU64::new(0)); + let c = now.clone(); + (Arc::new(move || c.load(Ordering::SeqCst)), now) +} + +fn link(widget: u64, register: u64) -> u64 { + 0x100 + (widget - 8) * 0x40 + register + 4 +} + +/// A widget that answers one word and remembers a write. +struct Stub(Mutex); +impl BusDevice for Stub { + fn read32(&self, addr: u64) -> BusRead32 { + if addr == 0x40 { BusRead32::ok(*self.0.lock()) } else { BusRead32::err() } + } + fn write32(&self, addr: u64, value: u32) -> u32 { + if addr != 0x40 { + return BUS_ERR; + } + *self.0.lock() = value; + BUS_OK + } +} + +#[derive(Default)] +struct Peer(Mutex>); +impl PicPeer for Peer { + fn interrupt(&self, source: u8, address: u64, payload: u64) -> Result<(), Refused> { + self.0.lock().push((source, address, payload)); + Ok(()) + } + fn dma_read(&self, _: u8, _: u8, _: u64, _: &mut [u8]) -> Result<(), Refused> { + Err(Refused) + } + fn dma_write(&self, _: u8, _: u8, _: u64, _: &[u8]) -> Result<(), Refused> { + Err(Refused) + } +} + +/// Widget zero names itself PXBOW, not Crossbow and not the PIC, and shows +/// the host's and the PIC's links up. +#[test] +fn widget_zero_identifies_as_pxbow_with_the_host_and_pic_links_up() { + let p = Pxbow::new(10, 15, clock().0).unwrap(); + let w = p.widget_zero(); + let id = w.read32(4).data; + assert_eq!(id, 0x3d10_0001); + assert_eq!((id >> 28) & 0xf, 3); + assert_eq!((id >> 12) & 0xffff, 0xd100); + assert_eq!((id >> 1) & 0x7ff, 0); + assert_eq!(id & 1, 1); + for widget in [10, 15] { + assert_ne!(w.read32(link(widget, 0x10)).data & LINK_STATUS_ALIVE, 0, "widget {widget} alive"); + assert_ne!(w.read32(link(widget, 0x38)).data & AUX_PRESENT, 0); + } + for widget in [8, 9, 11, 12, 13, 14] { + assert_eq!(w.read32(link(widget, 0x10)).data & LINK_STATUS_ALIVE, 0, "widget {widget} absent"); + assert_eq!(w.read32(link(widget, 0x38)).data & AUX_PRESENT, 0); + } + assert_eq!(p.endpoint_widgets(), 0); +} + +#[test] +fn status_reads_carry_the_hosts_widget_as_their_source() { + let p = Pxbow::new(10, 15, clock().0).unwrap(); + let w = p.widget_zero(); + // Seed the status with an access error first. + let _ = w.read32(0x1004); + let status = w.read32(STATUS).data; + assert_eq!((status >> 6) & 0xf, 10); + assert_ne!(status & (1 << 5), 0, "the seeded access error shows"); + assert_eq!((w.read32(STATUS_CLEAR).data >> 6) & 0xf, 10); + assert_eq!(w.read32(STATUS).data & (1 << 5), 0, "the alias acknowledged it"); + let other = Pxbow::new(11, 14, clock().0).unwrap(); + assert_eq!((other.widget_zero().read32(STATUS).data >> 6) & 0xf, 11); +} + +/// PROM 6.210's crossbar check on the Tezro writes 0xdeadbeef to the +/// read-only lower error address and 0 to the upper one. The store +/// completes, the registers keep their value, and widget 0's status reports +/// the access error (NEC U10278EJ4V0UM p.104). +#[test] +fn the_proms_error_address_writes_complete_and_are_reported() { + let p = Pxbow::new(10, 15, clock().0).unwrap(); + let w = p.widget_zero(); + assert_eq!(w.write32(0x1c, 0xdead_beef), BUS_OK); + assert_eq!(w.write32(0x14, 0), BUS_OK); + assert_eq!(w.read32(0x1c).data, 0); + assert_ne!(w.read32(STATUS).data & (1 << 5), 0, "the access error shows"); + assert!(p.xbow().unsupported().contains(&(0x1c, 4, true)), "the write stays observable"); + assert_ne!(w.write32(0x04, 0), BUS_OK, "the identification register still refuses"); +} + +/// Revision 5 decodes the fences and the lock; the lock takes a write only +/// while it reads zero and any write to its clear alias releases it. +#[test] +fn fences_and_the_lock_are_stored_and_read_back() { + let p = Pxbow::new(10, 15, clock().0).unwrap(); + let w = p.widget_zero(); + for (off, v) in [(0x78u64, 0xff_u32), (0x80, 0xff), (0xb8, 0xff), (0xc0, 0), (0xc8, 0)] { + assert_eq!(w.read32(off + 4).data, v, "reset value at {off:#x}"); + } + for (off, v) in [(0x78u64, 0x5a), (0x90, 0xa5), (0xb8, 0x3c)] { + assert_eq!(w.write32(off + 4, v), BUS_OK); + assert_eq!(w.read32(off + 4).data, v); + } + assert_eq!(w.read32(0x78 + 4).data, 0x5a); + assert_eq!(w.write32(0xc0 + 4, 0x11), BUS_OK); + assert_eq!(w.write32(0xc0 + 4, 0x22), BUS_OK); + assert_eq!(w.read32(0xc0 + 4).data, 0x11, "a held lock ignores a second write"); + assert_eq!(w.read32(0xc8 + 4).data, 0x11, "both offsets read the one value"); + assert_eq!(w.write32(0xc8 + 4, 0), BUS_OK); + assert_eq!(w.read32(0xc0 + 4).data, 0); + assert_eq!(w.write32(0xc0 + 4, 0x22), BUS_OK); + assert_eq!(w.read32(0xc0 + 4).data, 0x22); + crate::traits::Resettable::power_on(&*p.xbow().clone()); + assert_eq!(w.read32(0x78 + 4).data, 0xff, "reset restores the fences"); + assert_eq!(w.read32(0xc0 + 4).data, 0, "and releases the lock"); + // The upper word of a slot and an eight-byte access are still refused. + assert!(!w.read32(0x78).is_ok()); + assert!(!w.read64(0x78).is_ok()); +} + +/// Widget zero's own error interrupt is a packet to the host, with the +/// vector and address its destination registers hold. +#[test] +fn a_widget_zero_error_sends_an_interrupt_packet_to_the_host() { + let pic = Pic::new(15); + let peer = Arc::new(Peer::default()); + pic.connect_peer(peer.clone()).unwrap(); + let p = pic.enable_pxbow(10, clock().0).unwrap(); + let w = p.widget_zero(); + let address = 0x1_0180_0090u64; + assert_eq!(w.write32(0x34, (0x34 << 24) | (10 << 16) | ((address >> 32) as u32 & 0xffff)), BUS_OK); + assert_eq!(w.write32(0x3c, address as u32), BUS_OK); + assert_eq!(w.write32(0x24, 1 << 5), BUS_OK, "enable the access-error condition"); + assert!(peer.0.lock().is_empty()); + let _ = w.read32(0x1004); + assert_ne!(w.read32(STATUS).data & (1 << 23), 0, "the request bit is up"); + assert_eq!(peer.0.lock().as_slice(), [(0, address, 0x134)]); + let _ = w.read32(STATUS_CLEAR); + assert_eq!(peer.0.lock().last().copied(), Some((0, address, 0x34)), "acknowledging lowers it"); + // A target that is not the host goes nowhere. + assert_eq!(w.write32(0x34, (0x34 << 24) | (11 << 16)), BUS_OK); + let _ = w.read32(0x1004); + assert_eq!(peer.0.lock().len(), 2); +} + +/// The V12's widget is absent until the machine fits it, then forwards to +/// the endpoint with the widget-relative address unchanged. +#[test] +fn a_widget_is_unreachable_until_an_endpoint_is_attached() { + let p = Pxbow::new(10, 15, clock().0).unwrap(); + let w12 = p.route(12); + assert!(!w12.read32(0x40).is_ok()); + assert_eq!(w12.write32(0x40, 1), BUS_ERR); + let stub = Arc::new(Stub(Mutex::new(0))); + p.attach_endpoint(12, stub.clone()).unwrap(); + assert_eq!(p.endpoint_widgets(), 1 << 12); + let wz = p.widget_zero(); + assert_ne!(wz.read32(link(12, 0x10)).data & LINK_STATUS_ALIVE, 0, "its link comes up"); + assert_ne!(wz.read32(link(12, 0x38)).data & AUX_PRESENT, 0); + assert_eq!(w12.write32(0x40, 0xcafe), BUS_OK); + assert_eq!(*stub.0.lock(), 0xcafe); + assert_eq!(w12.read32(0x40).data, 0xcafe); + assert!(!w12.read32(0x44).is_ok(), "the endpoint's own refusal passes through"); + assert!(!p.route(11).read32(0x40).is_ok(), "other widgets stay absent"); +} + +#[test] +fn attachment_refuses_the_host_a_taken_widget_and_a_non_link() { + let p = Pxbow::new(10, 15, clock().0).unwrap(); + let stub: Arc = Arc::new(Stub(Mutex::new(0))); + assert_eq!(p.attach_endpoint(10, stub.clone()), Err(AttachError::Host(10))); + assert_eq!(p.attach_endpoint(3, stub.clone()), Err(AttachError::NotALink(3))); + assert_eq!(p.attach_endpoint(12, stub.clone()), Ok(())); + assert_eq!(p.attach_endpoint(12, stub.clone()), Err(AttachError::Occupied(12))); + assert!(Pxbow::new(10, 10, clock().0).is_err()); + assert!(Pxbow::new(10, 2, clock().0).is_err()); +} + +/// The PIC that owns IO9 supplies widget 0; the other PIC does not, and +/// the PIC's own widget is reached through the switch. +#[test] +fn only_the_pic_that_enables_it_has_a_pxbow_and_it_forwards_to_the_pic() { + let with = Pic::new(15); + let without = Pic::new(11); + assert!(with.pxbow().is_none()); + let p = with.enable_pxbow(10, clock().0).unwrap(); + assert!(without.pxbow().is_none()); + assert_eq!(p.pic_widget(), 15); + assert_eq!(p.endpoint_widgets(), 1 << 15); + assert!(Arc::ptr_eq(with.pxbow().unwrap(), &p)); + assert!(with.enable_pxbow(10, clock().0).is_err(), "once"); + let via_switch = p.route(15); + let id = via_switch.read64(0).data; + assert_eq!((id >> 28) & 0xf, 3, "the PIC's head 0 answers through widget 15"); + assert_eq!(via_switch.read64(crate::dev::pic::HEAD_OFFSET).data >> 12 & 0xffff, 0xd112); + assert!(Pic::new(15).enable_pxbow(15, clock().0).is_err(), "host and PIC differ"); +} + +/// A link reset on the host's link or the PIC's takes the PIC out of reach +/// for the pulse and brings it back, as the qtest's 20 ns clock step does. +#[test] +fn a_link_reset_makes_the_pic_unreachable_for_the_pulse() { + let (c, now) = clock(); + let pic = Pic::new(15); + let p = pic.enable_pxbow(10, c).unwrap(); + let via = p.route(15); + let wz = p.widget_zero(); + assert!(via.read64(0).is_ok()); + for (widget, reset) in [(10u64, 0x1b4u64), (15, 0x2f4)] { + now.fetch_add(100, Ordering::SeqCst); + assert_eq!(wz.write32(reset, 0), BUS_OK, "widget {widget}'s link reset"); + assert!(!via.read64(0).is_ok(), "down during the pulse"); + assert_eq!(via.write64(0x20, 0), BUS_ERR); + now.fetch_add(20, Ordering::SeqCst); + assert!(via.read64(0).is_ok(), "back after 20 ns"); + assert_eq!(via.read64(0x20).data, 0x0f00_320f, "the PIC kept its registers"); + } +} diff --git a/src/dev/serial_host.rs b/src/dev/serial_host.rs index 21d2a1ade..9b9fef830 100644 --- a/src/dev/serial_host.rs +++ b/src/dev/serial_host.rs @@ -23,6 +23,14 @@ use crate::dev::serial_log::SerialLog; pub trait SerialBackend: Send + Sync { fn send_byte(&self, byte: u8); + + /// Send a byte if the host end can take it now. A backend that can be + /// full (a socket nobody reads) says so here and the 16550 holds the byte + /// and retries; the default takes every byte, as the others always did. + fn try_send_byte(&self, byte: u8) -> bool { + self.send_byte(byte); + true + } fn recv_byte(&self) -> io::Result; } diff --git a/src/dev/sflash.rs b/src/dev/sflash.rs index 6d0e75ea6..5d35bfe91 100644 --- a/src/dev/sflash.rs +++ b/src/dev/sflash.rs @@ -30,8 +30,7 @@ //! [`Bridge::attach_flash`]: crate::dev::bridge::Bridge::attach_flash use std::cell::Cell; -use std::fs::{File, OpenOptions}; -use std::io::{self, Read, Seek, SeekFrom, Write}; +use std::io; use std::path::Path; use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; @@ -40,6 +39,10 @@ use parking_lot::Mutex; use crate::snapshot::{field_u8, StateTable}; use crate::traits::{BusDevice, BusRead16, Saveable, BUS_ERR, BUS_OK}; +pub(crate) mod storage; +pub(crate) mod amd; +use self::storage::FlashBacking; + /// The Octane's part: 1 MiB. pub const FLASH_SIZE: usize = 1 << 20; /// Erase block size. @@ -78,7 +81,7 @@ struct State { data: Vec, mode: Mode, status: u8, - file: Option, + backing: Option, } pub struct CfiFlash { @@ -114,7 +117,7 @@ impl CfiFlash { assert!(image.len() <= size, "image must fit the part"); let mut data = vec![0xff; size]; data[..image.len()].copy_from_slice(image); - Self { st: Mutex::new(State { data, mode: Mode::Array, status: SR_READY, file: None }), gen: AtomicU64::new(0) } + Self { st: Mutex::new(State { data, mode: Mode::Array, status: SR_READY, backing: None }), gen: AtomicU64::new(0) } } /// A reset: back to read-array mode with the status clear. The array @@ -132,23 +135,10 @@ impl CfiFlash { /// untouched, since it is probably some other file. pub fn with_file(image: &[u8], size: usize, path: impl AsRef) -> io::Result { let flash = Self::new(image, size); - let path = path.as_ref(); - let mut file = OpenOptions::new().read(true).write(true).create(true).truncate(false).open(path)?; - { - let mut st = flash.st.lock(); - let len = file.metadata()?.len(); - if len == size as u64 { - file.read_exact(&mut st.data)?; - } else if len == 0 { - file.write_all(&st.data)?; - } else { - return Err(io::Error::new( - io::ErrorKind::InvalidData, - format!("flash file {} is {len} bytes, expected {size} (or empty); not overwriting it", path.display()), - )); - } - st.file = Some(file); - } + let mut st = flash.st.lock(); + let backing = FlashBacking::open(path, &mut st.data)?; + st.backing = Some(backing); + drop(st); Ok(flash) } @@ -168,10 +158,9 @@ impl CfiFlash { } impl State { - fn sync(&self, off: usize, len: usize) { - if let Some(mut f) = self.file.as_ref() { - // `&File` seeks and writes without needing `&mut self`. - let res = f.seek(SeekFrom::Start(off as u64)).and_then(|_| f.write_all(&self.data[off..off + len])); + fn sync(&mut self, off: usize, len: usize) { + if let Some(backing) = self.backing.as_mut() { + let res = backing.write_range(off, &self.data[off..off + len]); if let Err(e) = res { // The guest sees a completed cycle; the file is only the backing store. log::warn!("flash: backing file write at {off:#x} failed: {e}"); diff --git a/src/dev/sflash/amd.rs b/src/dev/sflash/amd.rs new file mode 100644 index 000000000..41c4cdff9 --- /dev/null +++ b/src/dev/sflash/amd.rs @@ -0,0 +1,313 @@ +//! The AMD flash command set: unlock, word program, sector erase and CFI +//! query state, over storage supplied by the caller. +//! +//! Array bytes and bus translation belong to the enclosing device. Fuel uses +//! this 8 MiB command engine for each boot-flash array. The CFI data and timing +//! are compatibility values; they do not identify the fitted flash chip. +//! Buffered writes, protection, bypass and erase suspend are absent. +//! +//! Ported from MAME's intelfsh.cpp AMD command path, revision +//! 2ee9b1e30299b122b131cab0bd7912abc2b095ba (BSD-3-Clause). + +// Copyright (c) Aaron Giles + +pub const SECTOR_SIZE: usize = 64 * 1024; +const SECTORS: usize = 128; +pub const PART_SIZE: usize = SECTOR_SIZE * SECTORS; +pub const ERASE_SETUP_NS: u64 = 50_000; +pub const ERASE_NS: u64 = 512_000_000; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Mode { + Array, + Unlock1, + Unlock2, + Program, + Erase1, + Erase2, + Erase3, + Query, + Identify, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum Effect { + None, + Program { offset: usize, value: u16 }, +} + +pub struct AmdCore { + mode: Mode, + selected: [bool; SECTORS], + /// Remaining time includes the sector-selection interval. + remaining_ns: u64, + setup_ns: u64, + toggle: bool, + failed: bool, +} + +impl Default for AmdCore { + fn default() -> Self { + Self { + mode: Mode::Array, + selected: [false; SECTORS], + remaining_ns: 0, + setup_ns: 0, + toggle: false, + failed: false, + } + } +} + +impl AmdCore { + pub fn reset(&mut self) { + *self = Self::default(); + } + pub fn busy(&self) -> bool { + self.remaining_ns != 0 || self.failed + } + pub fn selecting(&self) -> bool { + self.setup_ns != 0 + } + pub fn array_reads(&self) -> bool { + !self.busy() && self.mode != Mode::Query && self.mode != Mode::Identify + } + /// None selects ordinary backing bytes. Polling never advances time. + pub fn read16(&mut self, offset: usize) -> Option { + if self.busy() { + let selected = self.selected[offset / SECTOR_SIZE]; + self.toggle = !self.toggle; + return Some( + (if self.toggle { + 0x40 | if selected { 4 } else { 0 } + } else { + 0 + }) | if self.setup_ns == 0 { 8 } else { 0 } + | if self.failed { 0x20 } else { 0 }, + ); + } + match self.mode { + Mode::Query => Some(query(offset / 2) as u16), + // IDs remain unspecified until a part/profile is selected. + Mode::Identify => Some(0), + _ => None, + } + } + pub fn write16(&mut self, offset: usize, value: u16) -> Result { + if offset & 1 != 0 || offset >= PART_SIZE { + return Err(()); + } + let cmd = value as u8; + let word = offset / 2; + if self.busy() { + if (self.failed || self.setup_ns != 0) && cmd == 0xf0 { + self.reset(); + } else if self.setup_ns != 0 && cmd == 0x30 { + self.selected[offset / SECTOR_SIZE] = true; + self.setup_ns = ERASE_SETUP_NS; + self.remaining_ns = ERASE_SETUP_NS + ERASE_NS; + } else if self.setup_ns != 0 { + self.reset(); + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "amd-flash: sector selection cancelled by command {cmd:#x} at word offset {word:#x}" + ); + return Err(()); + } else if cmd != 0xf0 { + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "amd-flash: unsupported command {cmd:#x} while erasing at word offset {word:#x}" + ); + return Err(()); + } + return Ok(Effect::None); + } + // The pending program data cycle consumes every bit, including F0. + if self.mode == Mode::Program { + self.mode = Mode::Array; + return Ok(Effect::Program { offset, value }); + } + if cmd == 0xf0 { + self.reset(); + return Ok(Effect::None); + } + let next = match (self.mode, word, cmd) { + (Mode::Array, 0x55, 0x98) => Mode::Query, + (Mode::Array, 0x555, 0xaa) => Mode::Unlock1, + (Mode::Unlock1, 0x2aa, 0x55) => Mode::Unlock2, + (Mode::Unlock2, 0x555, 0xa0) => Mode::Program, + (Mode::Unlock2, 0x555, 0x80) => Mode::Erase1, + (Mode::Unlock2, 0x555, 0x90) => Mode::Identify, + (Mode::Erase1, 0x555, 0xaa) => Mode::Erase2, + (Mode::Erase2, 0x2aa, 0x55) => Mode::Erase3, + (Mode::Erase3, _, 0x30) => { + self.selected[offset / SECTOR_SIZE] = true; + self.setup_ns = ERASE_SETUP_NS; + self.remaining_ns = ERASE_SETUP_NS + ERASE_NS; + Mode::Array + } + _ => { + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "amd-flash: unsupported cycle {value:#06x} at byte offset {offset:#x} in {:?}", + self.mode + ); + self.mode = Mode::Array; + return Err(()); + } + }; + self.mode = next; + Ok(Effect::None) + } + /// The owner persists and applies selected sectors before calling finish. + pub fn advance_ns(&mut self, ns: u64) -> bool { + if self.remaining_ns == 0 || self.failed { + return false; + } + self.setup_ns = self.setup_ns.saturating_sub(ns); + self.remaining_ns = self.remaining_ns.saturating_sub(ns); + self.remaining_ns == 0 + } + pub fn selected_sectors(&self) -> impl Iterator + '_ { + self.selected + .iter() + .enumerate() + .filter_map(|(i, selected)| selected.then_some(i)) + } + pub fn finish_erase(&mut self, success: bool) { + if success { + self.reset(); + } else { + self.failed = true; + } + } +} + +// Generic model response; voltage/timing do not identify Fuel's fitted part. +// Field positions/units: AMD publication 21444 rev E+2, pp14-15. +// Region encoding also follows Renode AMDCFIFlash.cs (MIT), revision +// 5448105bc7875aa6c5305761d97a371dc640dd7f. No Renode mutation code is used. +fn query(index: usize) -> u8 { + const PROFILE: [u8; 33] = [ + b'Q', b'R', b'Y', 2, 0, 0, 0, 0, 0, 0, 0, 0x27, 0x36, 0, 0, 3, 0, 9, 0, 4, 0, 4, 0, 23, 1, + 0, 0, 0, 1, 0x7f, 0, 0, 1, + ]; + index + .checked_sub(0x10) + .and_then(|i| PROFILE.get(i)) + .copied() + .unwrap_or(0) +} + +#[cfg(test)] +mod tests { + // Cases ported from upstream QEMU's tests/qtest/pflash-cfi02-test.c. + use super::*; + + // Word addresses from the command sequences, as byte offsets on the 16-bit bus. + const UNLOCK0: usize = 0x555 * 2; + const UNLOCK1: usize = 0x2aa * 2; + const CFI: usize = 0x55 * 2; + + fn cmd(f: &mut AmdCore, offset: usize, value: u16) -> Effect { + f.write16(offset, value).unwrap() + } + + fn unlock(f: &mut AmdCore) { + cmd(f, UNLOCK0, 0xaa); + cmd(f, UNLOCK1, 0x55); + } + + fn start_sector_erase(f: &mut AmdCore, offset: usize) { + unlock(f); + cmd(f, UNLOCK0, 0x80); + unlock(f); + cmd(f, offset, 0x30); + } + + /// The guest's erase loop polls DQ3, DQ6 and DQ7; get the phases wrong and + /// firmware either times out or reads the sector before it is blank. + #[test] + fn a_sector_erase_reports_toggle_status_until_it_completes() { + let mut f = AmdCore::default(); + let sector = 5 * SECTOR_SIZE; + start_sector_erase(&mut f, sector); + assert!(f.busy() && f.selecting()); + assert!(!f.array_reads()); + // DQ3 clear: still inside the window where more sectors may be added. + let first = f.read16(sector).unwrap(); + assert_eq!(first & 0x08, 0); + assert_eq!(first & 0x80, 0); + f.advance_ns(ERASE_SETUP_NS); + assert!(!f.selecting()); + let a = f.read16(sector).unwrap(); + let b = f.read16(sector).unwrap(); + assert_eq!(a & 0x08, 0x08); + assert_eq!(a & 0x80, 0); + assert_eq!(a & 0x40, !b & 0x40, "DQ6 toggles on successive reads"); + assert!(f.advance_ns(ERASE_NS)); + assert_eq!(f.selected_sectors().collect::>(), [5]); + f.finish_erase(true); + assert!(f.array_reads()); + assert_eq!(f.read16(sector), None); + } + + /// Firmware erases several sectors with one command; each further 0x30 + /// inside the selection window must join the set, not start a second erase. + #[test] + fn extra_sectors_join_a_pending_erase_inside_the_selection_window() { + let mut f = AmdCore::default(); + start_sector_erase(&mut f, 0); + cmd(&mut f, 3 * SECTOR_SIZE, 0x30); + assert_eq!(f.selected_sectors().collect::>(), [0, 3]); + // The window restarts on each selection. + f.advance_ns(ERASE_SETUP_NS - 1); + assert!(f.selecting()); + // A non-erase command in the window cancels the whole operation. + assert!(f.write16(0, 0x55).is_err()); + assert!(!f.busy()); + assert_eq!(f.selected_sectors().count(), 0); + } + + /// Reset (F0) is how firmware backs out of a half-finished command or + /// leaves CFI mode, so it must always return the part to array reads. + #[test] + fn a_reset_command_abandons_a_partial_unlock_and_leaves_cfi_mode() { + let mut f = AmdCore::default(); + cmd(&mut f, UNLOCK0, 0xaa); + cmd(&mut f, 0, 0xf0); + // The unlock is forgotten: a program setup now is not a valid cycle. + assert!(f.write16(UNLOCK1, 0x55).is_err()); + + cmd(&mut f, CFI, 0x98); + assert!(!f.array_reads()); + cmd(&mut f, 0, 0xf0); + assert!(f.array_reads()); + assert_eq!(f.read16(0x10 * 2), None); + } + + /// Firmware that probes the part reads the CFI "QRY" signature and the + /// device-size exponent to learn how much flash it has. + #[test] + fn the_cfi_query_returns_the_qry_signature_and_the_size_exponent() { + let mut f = AmdCore::default(); + cmd(&mut f, CFI, 0x98); + let q = |f: &mut AmdCore, word: usize| f.read16(word * 2).unwrap(); + assert_eq!([q(&mut f, 0x10), q(&mut f, 0x11), q(&mut f, 0x12)], [b'Q' as u16, b'R' as u16, b'Y' as u16]); + assert_eq!(1usize << q(&mut f, 0x27), PART_SIZE); + } + + /// Word program is the unlock sequence, A0, then one data cycle; the data + /// cycle goes to the owner as an effect and is not decoded as a command. + #[test] + fn a_program_sequence_hands_the_data_word_to_the_owner() { + let mut f = AmdCore::default(); + unlock(&mut f); + cmd(&mut f, UNLOCK0, 0xa0); + // 0xf0 as data must not be taken for a reset. + assert_eq!(cmd(&mut f, 0x1234 * 2, 0xa5f0), Effect::Program { offset: 0x1234 * 2, value: 0xa5f0 }); + assert!(f.array_reads()); + // Odd byte offsets are not word cycles. + assert!(f.write16(UNLOCK0 + 1, 0xaa).is_err()); + } +} diff --git a/src/dev/sflash/storage.rs b/src/dev/sflash/storage.rs new file mode 100644 index 000000000..815e1aa6d --- /dev/null +++ b/src/dev/sflash/storage.rs @@ -0,0 +1,62 @@ +//! Exact-size file backing shared by flash command-set models. + +use std::fs::{File, OpenOptions}; +use std::io::{self, Read, Seek, SeekFrom, Write}; +use std::path::Path; + +pub(crate) struct FlashBacking { + file: File, + size: usize, +} + +impl FlashBacking { + #[cfg(test)] + pub(crate) fn from_file(file: File, size: usize) -> Self { + Self { file, size } + } + + /// Open a flash store, loading its bytes or seeding a new empty file. + /// A non-empty file must match the supplied array size and is left intact + /// when its size is refused. + pub(crate) fn open(path: impl AsRef, data: &mut [u8]) -> io::Result { + let path = path.as_ref(); + let mut file = OpenOptions::new() + .read(true) + .write(true) + .create(true) + .truncate(false) + .open(path)?; + let len = file.metadata()?.len(); + if len == data.len() as u64 { + file.read_exact(data)?; + } else if len == 0 { + file.write_all(data)?; + } else { + return Err(io::Error::new( + io::ErrorKind::InvalidData, + format!( + "flash file {} is {len} bytes, expected {} (or empty); not overwriting it", + path.display(), + data.len() + ), + )); + } + Ok(Self { + file, + size: data.len(), + }) + } + + /// Write one byte range through to the backing file. + pub(crate) fn write_range(&mut self, offset: usize, bytes: &[u8]) -> io::Result<()> { + if offset > self.size || bytes.len() > self.size - offset { + return Err(io::Error::new( + io::ErrorKind::InvalidInput, + "flash backing range is outside the array", + )); + } + self.file.seek(SeekFrom::Start(offset as u64))?; + self.file.write_all(bytes)?; + self.file.flush() + } +} diff --git a/src/dev/timekeeper.rs b/src/dev/timekeeper.rs new file mode 100644 index 000000000..95a2557a5 --- /dev/null +++ b/src/dev/timekeeper.rs @@ -0,0 +1,701 @@ +//! The Fuel M48T35 Timekeeper exposes 32 KiB of byte-wide NVRAM and calendar +//! registers at the end of that window. The guest reads the clock independently +//! of processor byte order, and software can stage clock changes through the +//! control register before committing them. +//! +//! # What is approximated here +//! +//! Host time and a persisted sidecar stand in for the battery-backed clock. +//! The model does not reproduce the physical oscillator or battery behavior. +//! +//! Ported from hw/rtc/m48t35.c in jamesbraid/qemu-sgi-origami. + +use crate::traits::{BusDevice, BusRead8, Device, Resettable, BUS_ERR, BUS_OK}; +use parking_lot::Mutex; +use std::path::{Path, PathBuf}; +use std::sync::Arc; +use std::time::{Instant, SystemTime, UNIX_EPOCH}; + +pub const SIZE: usize = 0x8000; +pub mod reg { + pub const CONTROL: usize = 0x7ff8; + pub const SECONDS: usize = 0x7ff9; + pub const MINUTES: usize = 0x7ffa; + pub const HOURS: usize = 0x7ffb; + pub const CLOCK_WRITE: usize = 0x7ffc; + pub const DAY: usize = 0x7ffd; + pub const MONTH: usize = 0x7ffe; + pub const YEAR: usize = 0x7fff; +} +pub const CONTROL: usize = reg::CONTROL; +const CENTURY: i64 = 36525 * 86400; +const MAGIC: &[u8; 8] = b"M48TREF1"; +fn bcd(v: i64) -> u8 { + ((v / 10) * 16 + v % 10) as u8 +} +fn unbcd(v: u8) -> i64 { + i64::from((v >> 4) * 10 + (v & 15)) +} +fn month_days(y: i64, m: i64) -> i64 { + match m { + 2 => 28 + i64::from(y % 4 == 0), + 4 | 6 | 9 | 11 => 30, + _ => 31, + } +} +#[derive(Clone, Copy)] +struct Calendar { + year: i64, + month: i64, + day: i64, + hour: i64, + minute: i64, + second: i64, +} +impl Calendar { + fn seconds(self) -> i64 { + let year = (self.year + (self.month - 1).div_euclid(12)).rem_euclid(100); + let month = (self.month - 1).rem_euclid(12) + 1; + let mut days = year * 365 + (year + 3) / 4 + self.day - 1; + for m in 1..month { + days += month_days(year, m); + } + (days * 86400 + self.hour * 3600 + self.minute * 60 + self.second).rem_euclid(CENTURY) + } + fn from_seconds(t: i64) -> Self { + let t = t.rem_euclid(CENTURY); + let mut days = t / 86400; + let mut year = 0; + while days >= 365 + i64::from(year % 4 == 0) { + days -= 365 + i64::from(year % 4 == 0); + year += 1; + } + let mut month = 1; + while days >= month_days(year, month) { + days -= month_days(year, month); + month += 1; + } + Self { + year, + month, + day: days + 1, + hour: t / 3600 % 24, + minute: t / 60 % 60, + second: t % 60, + } + } +} +fn saved_calendar(image: &[u8]) -> Option { + if image.len() != SIZE { + return None; + } + let values = [ + image[0x7ff9] & 0x7f, + image[0x7ffa], + image[0x7ffb], + image[0x7ffd], + image[0x7ffe], + image[0x7fff], + ]; + let limits = [(0, 59), (0, 59), (0, 23), (1, 31), (1, 12), (0, 99)]; + let valid = values + .iter() + .zip(limits) + .all(|(&v, (lo, hi))| v & 15 <= 9 && v >> 4 <= 9 && (lo..=hi).contains(&unbcd(v))); + if !valid { + return None; + } + Some(Calendar { + second: unbcd(values[0]), + minute: unbcd(values[1]), + hour: unbcd(values[2]), + day: unbcd(values[3]), + month: unbcd(values[4]), + year: unbcd(values[5]), + }) +} +struct State { + cells: Vec, + offset: i64, + stopped: Option, + frozen: Option, + staged: Option<(Calendar, bool)>, + file: Option, + #[cfg(test)] + extra: i64, +} +impl State { + fn running(&self, host: i64) -> i64 { + self.stopped + .unwrap_or_else(|| (host + self.offset).rem_euclid(CENTURY)) + } + fn calendar(&self, host: i64) -> Calendar { + Calendar::from_seconds(self.frozen.unwrap_or_else(|| self.running(host))) + } +} + +/// Fuel's 1968-based clock. Ordinary storage starts blank; the guest owns it. +/// The lower candidate clock addresses are independent RAM, with no aliases. +pub struct Timekeeper { + initial_epoch: i64, + st: Mutex, + clock: Arc i64 + Send + Sync>, +} +impl Timekeeper { + pub fn fuel() -> Result { + let host = SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_err(|e| e.to_string())? + .as_secs() as i64; + // Gregorian host date, used only to initialize the part's own calendar. + let mut days = host / 86400; + let mut year = 1970; + loop { + let n = 365 + i64::from(year % 4 == 0 && (year % 100 != 0 || year % 400 == 0)); + if days < n { + break; + } + days -= n; + year += 1; + } + let mut month = 1; + while days >= month_days(year, month) { + days -= month_days(year, month); + month += 1; + } + let t = Self::fuel_at( + year, + month, + days + 1, + host / 3600 % 24, + host / 60 % 60, + host % 60, + )?; + Ok(t) + } + /// Open authoritative saved clock state before applying new-store year limits. + pub fn fuel_with_nvram_file(path: &Path) -> Result { + Self::backed(path, Self::fuel()) + } + fn backed(path: &Path, initial: Result) -> Result { + let saved = match std::fs::read(path) { + Ok(v) => v, + Err(e) if e.kind() == std::io::ErrorKind::NotFound => Vec::new(), + Err(e) => return Err(e.to_string()), + }; + let clock = if saved_calendar(&saved).is_some() { + // This seed is replaced by the authoritative image before exposure. + Self::fuel_at(1968, 1, 1, 0, 0, 0)? + } else { + initial? + }; + clock.set_nvram_file(path)?; + Ok(clock) + } + /// A deterministic initial date; subsequent seconds follow monotonic host time. + pub fn fuel_at( + year: i64, + month: i64, + day: i64, + hour: i64, + minute: i64, + second: i64, + ) -> Result { + if !(1968..=2067).contains(&year) { + return Err("Timekeeper: initial year is outside 1968-2067".into()); + } + if !(1..=12).contains(&month) + || !(1..=month_days(year, month)).contains(&day) + || !(0..24).contains(&hour) + || !(0..60).contains(&minute) + || !(0..60).contains(&second) + { + return Err("Timekeeper: invalid initial calendar".into()); + } + let epoch = Calendar { + year: year - 1968, + month, + day, + hour, + minute, + second, + } + .seconds(); + let at = Instant::now(); + Ok(Self { + initial_epoch: epoch, + st: Mutex::new(State { + cells: vec![0; SIZE], + offset: epoch, + stopped: None, + frozen: None, + staged: None, + file: None, + #[cfg(test)] + extra: 0, + }), + clock: Arc::new(move || at.elapsed().as_secs() as i64), + }) + } + fn host(&self, st: &State) -> i64 { + #[cfg(not(test))] + let _ = st; + (self.clock)() + { + #[cfg(test)] + { + st.extra + } + #[cfg(not(test))] + { + 0 + } + } + } + #[cfg(test)] + fn advance_for_test(&self, seconds: i64) { + self.st.lock().extra += seconds; + } + + fn image(st: &State, host: i64, saved_at: i64) -> (Vec, [u8; 16]) { + let c = Calendar::from_seconds(st.running(host)); + let mut image = st.cells.clone(); + image[CONTROL + 1] = bcd(c.second) | if st.stopped.is_some() { 0x80 } else { 0 }; + image[CONTROL + 2] = bcd(c.minute); + image[CONTROL + 3] = bcd(c.hour); + image[CONTROL + 5] = bcd(c.day); + image[CONTROL + 6] = bcd(c.month); + image[CONTROL + 7] = bcd(c.year); + let mut reference = [0; 16]; + reference[..8].copy_from_slice(MAGIC); + reference[8..].copy_from_slice(&saved_at.to_le_bytes()); + (image, reference) + } + /// Export the exact raw image and companion clock reference, without changing RAM. + pub fn persistence_image_at(&self, saved_at: i64) -> (Vec, [u8; 16]) { + let st = self.st.lock(); + Self::image(&st, self.host(&st), saved_at) + } + /// Restore validated storage. Invalid/unwritten clock cells keep the initial clock. + /// A missing/invalid reference adds no elapsed time, matching the model's policy. + pub fn restore_image_at( + &self, + image: &[u8], + reference: Option<&[u8]>, + now: i64, + ) -> Result<(), String> { + if image.len() != SIZE { + return Err(format!( + "Timekeeper: backing image has {} bytes, expected {SIZE}", + image.len() + )); + } + if reference.is_some_and(|r| r.len() < 16) { + return Err("Timekeeper: clock reference is shorter than 16 bytes".into()); + } + let mut st = self.st.lock(); + st.cells.copy_from_slice(image); + st.frozen = None; + st.staged = None; + st.stopped = None; + if let Some(c) = saved_calendar(image) { + let stopped = image[0x7ff9] & 0x80 != 0; + let elapsed = reference.filter(|r| &r[..8] == MAGIC).map_or(0, |r| { + now.saturating_sub(i64::from_le_bytes(r[8..16].try_into().unwrap())) + .max(0) + }); + let epoch = (c.seconds() + + if stopped { + 0 + } else { + elapsed.rem_euclid(CENTURY) + }) + .rem_euclid(CENTURY); + st.offset = epoch - self.host(&st); + if stopped { + st.stopped = Some(epoch); + } + } else { + st.offset = self.initial_epoch; + crate::dlog!(crate::devlog::LogModule::Rtc, "Timekeeper: unwritten or invalid clock cells; following initial clock"); + } + Ok(()) + } + /// Bind a raw store and its `.clock` companion. Validate both before creating either. + pub fn set_nvram_file(&self, path: &Path) -> Result<(), String> { + let clock_path = PathBuf::from(format!("{}.clock", path.display())); + let image = match std::fs::read(path) { + Ok(v) => Some(v), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => None, + Err(e) => return Err(e.to_string()), + }; + if image.as_ref().is_some_and(|v| v.len() != SIZE) { + return Err("Timekeeper: backing image must contain exactly 32768 bytes".into()); + } + let reference = match std::fs::read(&clock_path) { + Ok(v) => Some(v), + Err(e) if e.kind() == std::io::ErrorKind::NotFound => None, + Err(e) => return Err(e.to_string()), + }; + if reference.as_ref().is_some_and(|v| v.len() < 16) { + return Err("Timekeeper: clock reference is shorter than 16 bytes".into()); + } + if let Some(image) = image.as_ref() { + self.restore_image_at(image, reference.as_deref(), unix_now())?; + } + use std::io::Write; + if image.is_none() { + std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(path) + .and_then(|mut f| f.write_all(&vec![0; SIZE])) + .map_err(|e| e.to_string())?; + } + if reference.is_none() { + std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(clock_path) + .and_then(|mut f| f.write_all(&[0; 16])) + .map_err(|e| e.to_string())?; + } + self.st.lock().file = Some(path.to_path_buf()); + Ok(()) + } + pub fn flush_nvram(&self) -> Result<(), String> { + let st = self.st.lock(); + if let Some(path) = &st.file { + let (image, reference) = Self::image(&st, self.host(&st), unix_now()); + std::fs::write(path, image).map_err(|e| e.to_string())?; + use std::io::Write; + std::fs::OpenOptions::new() + .write(true) + .open(format!("{}.clock", path.display())) + .and_then(|mut f| f.write_all(&reference)) + .map_err(|e| e.to_string())?; + } + Ok(()) + } +} +fn unix_now() -> i64 { + SystemTime::now() + .duration_since(UNIX_EPOCH) + .map_or(0, |d| d.as_secs() as i64) +} +impl BusDevice for Timekeeper { + fn read8(&self, addr: u64) -> BusRead8 { + let Ok(a) = usize::try_from(addr) else { + return BusRead8::err(); + }; + if a >= SIZE { + return BusRead8::err(); + } + let st = self.st.lock(); + let c = st.calendar(self.host(&st)); + let data = match a { + reg::SECONDS => bcd(c.second) | if st.stopped.is_some() { 0x80 } else { 0 }, + reg::MINUTES => bcd(c.minute), + reg::HOURS => bcd(c.hour), + reg::DAY => bcd(c.day), + reg::MONTH => bcd(c.month), + reg::YEAR => bcd(c.year), + _ => st.cells[a], + }; + BusRead8 { + status: BUS_OK, + data, + } + } + fn write8(&self, addr: u64, val: u8) -> u32 { + let Ok(a) = usize::try_from(addr) else { + return BUS_ERR; + }; + if a >= SIZE { + return BUS_ERR; + } + let mut st = self.st.lock(); + let host = self.host(&st); + let old_control = st.cells[CONTROL]; + let transfer = a == CONTROL && st.cells[CONTROL] & 0x80 != 0 && val & 0x80 == 0; + if a == CONTROL { + let old = st.cells[a]; + if val & 0x40 != 0 && old & 0x40 == 0 { + st.frozen = Some(st.running(host)); + } else if val & 0x40 == 0 { + st.frozen = None; + } + if val & 0x80 != 0 && old & 0x80 == 0 { + st.staged = Some((st.calendar(host), st.stopped.is_some())); + } else if val & 0x80 == 0 && old & 0x80 != 0 { + if let Some((c, stop)) = st.staged.take() { + let t = c.seconds(); + st.offset = t - host; + st.stopped = if stop { Some(t) } else { None }; + } + } + st.cells[a] = val; + } else if a > CONTROL { + if a == reg::CLOCK_WRITE && st.staged.is_some() { + st.cells[a] = val; + } else if let Some((c, stop)) = &mut st.staged { + match a { + reg::SECONDS => { + c.second = unbcd(val & 0x7f); + *stop = val & 0x80 != 0; + } + reg::MINUTES => c.minute = unbcd(val), + reg::HOURS => c.hour = unbcd(val), + reg::DAY => c.day = unbcd(val), + reg::MONTH => c.month = unbcd(val), + reg::YEAR => c.year = unbcd(val), + _ => {} + } + } else { + crate::dlog!(crate::devlog::LogModule::Rtc, "Timekeeper: clock write at {a:#x} without write enable"); + } + } else { + st.cells[a] = val; + } + if !transfer + && (a < CONTROL + || a == CONTROL && (old_control ^ val) & !0xc0 != 0 + || a == reg::CLOCK_WRITE && st.staged.is_some()) + { + // Ordinary storage writes must not save or replace the clock cells. + if let Some(path) = &st.file { + use std::io::{Seek, SeekFrom, Write}; + let result = + std::fs::OpenOptions::new() + .write(true) + .open(path) + .and_then(|mut f| { + f.seek(SeekFrom::Start(a as u64))?; + f.write_all(&[st.cells[a]]) + }); + if let Err(e) = result { + log::error!("Timekeeper: backing write failed: {e}"); + } + } + } + drop(st); + if transfer { + if let Err(e) = self.flush_nvram() { + log::error!("Timekeeper: clock save failed: {e}"); + } + } + BUS_OK + } +} +impl Resettable for Timekeeper { + fn power_on(&self) { + let mut st = self.st.lock(); + st.frozen = None; + st.staged = None; + st.cells[CONTROL] &= !0xc0; + } +} +impl Device for Timekeeper { + fn start(&self) {} + fn step(&self, _cycles: u64) {} + fn is_running(&self) -> bool { + false + } + fn get_clock(&self) -> u64 { + 0 + } + fn stop(&self) { + if let Err(e) = self.flush_nvram() { + log::error!("Timekeeper: saving failed: {e}"); + } + } +} +impl Drop for Timekeeper { + fn drop(&mut self) { + Device::stop(self); + } +} +#[cfg(test)] +mod tests { + // Cases ported from tests/qtest/sn1-fuel-test.c in jamesbraid/qemu-sgi-origami. + use super::*; + use crate::traits::{BusDevice, Resettable}; + + /// The guest can freeze, stop, stage, and resume the M48T35 calendar. + #[test] + fn clock_control_freezes_stops_and_resets_the_calendar() { + let t = Timekeeper::fuel_at(2067, 12, 31, 23, 59, 59).unwrap(); + t.write8(0x7f8, 0x42); + t.write8(CONTROL as u64, 0x40); + t.advance_for_test(2); + assert_eq!(t.read8(reg::YEAR as u64).data, 0x99); + t.power_on(); + assert_eq!(t.read8(CONTROL as u64).data, 0); + assert_eq!(t.read8(reg::YEAR as u64).data, 0); + assert_eq!(t.read8(reg::MONTH as u64).data, 1); + assert_eq!(t.read8(reg::DAY as u64).data, 1); + assert_eq!(t.read8(0x7f8).data, 0x42); + t.write8(CONTROL as u64, 0x80); + t.write8(reg::SECONDS as u64, 0x89); + t.write8(CONTROL as u64, 0); + t.advance_for_test(60); + assert_eq!(t.read8(reg::SECONDS as u64).data, 0x89); + } + + /// A restored guest clock advances by host elapsed time unless its stop bit was saved. + #[test] + fn restored_clock_accounts_for_elapsed_time_unless_stopped() { + for stopped in [false, true] { + let t = Timekeeper::fuel_at(2024, 2, 28, 23, 59, 59).unwrap(); + t.write8(0x7ff0, 0xa5); + if stopped { + t.write8(CONTROL as u64, 0x80); + t.write8(reg::SECONDS as u64, 0x85); + t.write8(CONTROL as u64, 0); + } + let (image, reference) = t.persistence_image_at(100); + let restored = Timekeeper::fuel_at(2026, 1, 1, 0, 0, 0).unwrap(); + restored.restore_image_at(&image, Some(&reference), 102).unwrap(); + assert_eq!(restored.read8(reg::DAY as u64).data, if stopped { 0x28 } else { 0x29 }); + assert_eq!(restored.read8(reg::SECONDS as u64).data, if stopped { 0x85 } else { 1 }); + assert_eq!(restored.read8(0x7ff0).data, 0xa5); + assert!(restored.restore_image_at(&image[..2048], None, 102).is_err()); + } + } + + /// NVRAM persists across emulator restarts, and malformed backing images are rejected. + #[test] + fn backed_nvram_preserves_saved_state_and_rejects_invalid_images() { + let dir = std::env::current_exe().unwrap().parent().unwrap() + .join(format!("timekeeper-test-{}", std::process::id())); + std::fs::create_dir(&dir).unwrap(); + let path = dir.join("nvram"); + let t = Timekeeper::fuel_at(2026, 1, 1, 0, 0, 0).unwrap(); + t.set_nvram_file(&path).unwrap(); + assert_eq!(std::fs::read(&path).unwrap(), vec![0; SIZE]); + t.write8(0x7f8, 0x42); + assert_eq!(std::fs::read(&path).unwrap()[0x7f8], 0x42); + t.flush_nvram().unwrap(); + drop(t); + // Saved clock state is authoritative even if the host date is out of range. + let restored = Timekeeper::backed(&path, Timekeeper::fuel_at(2068, 1, 1, 0, 0, 0)).unwrap(); + restored.power_on(); + assert_eq!(restored.read8(0x7f8).data, 0x42); + assert_eq!(restored.read8(reg::YEAR as u64).data, 0x58); + drop(restored); + std::fs::remove_file(dir.join("nvram.clock")).unwrap(); + std::fs::write(&path, [0xa5; 2048]).unwrap(); + assert!(Timekeeper::backed(&path, Timekeeper::fuel_at(2026, 1, 1, 0, 0, 0)).is_err()); + assert_eq!(std::fs::read(&path).unwrap(), vec![0xa5; 2048]); + assert!(!dir.join("nvram.clock").exists()); + std::fs::remove_file(&path).unwrap(); + assert!(Timekeeper::backed(&path, Timekeeper::fuel_at(2068, 1, 1, 0, 0, 0)).is_err()); + assert!(!path.exists()); + std::fs::remove_dir(dir).unwrap(); + } + + const WEEKDAY: u64 = reg::CLOCK_WRITE as u64; + + fn clock_cells(t: &Timekeeper) -> [u8; 7] { + std::array::from_fn(|i| t.read8(reg::SECONDS as u64 + i as u64).data) + } + + fn set_clock(t: &Timekeeper, cells: [u8; 7]) { + t.write8(CONTROL as u64, 0x80); + for (i, v) in cells.into_iter().enumerate() { + t.write8(reg::SECONDS as u64 + i as u64, v); + } + t.write8(CONTROL as u64, 0); + } + + // Cells are seconds, minutes, hours, weekday, day, month, year, all BCD. + + /// The PROM and kernel derive file timestamps from this calendar, so + /// 2024 must have a 29 February and 2026 must not. + #[test] + fn the_calendar_rolls_over_february_in_leap_years_only() { + let t = Timekeeper::fuel_at(2026, 1, 1, 0, 0, 0).unwrap(); + set_clock(&t, [0x59, 0x59, 0x23, 0x05, 0x28, 0x02, 0x56]); + t.advance_for_test(1); + assert_eq!(clock_cells(&t), [0, 0, 0, 0x05, 0x29, 0x02, 0x56]); + set_clock(&t, [0x59, 0x59, 0x23, 0x05, 0x28, 0x02, 0x58]); + t.advance_for_test(1); + assert_eq!(clock_cells(&t), [0, 0, 0, 0x05, 0x01, 0x03, 0x58]); + } + + /// The part counts a two-digit year from 1968, so the calendar wraps after + /// 2067; the guest must see year 00 again, not garbage. + #[test] + fn the_year_wraps_from_2067_to_1968() { + let t = Timekeeper::fuel_at(2026, 1, 1, 0, 0, 0).unwrap(); + set_clock(&t, [0x59, 0x59, 0x23, 0x05, 0x31, 0x12, 0x99]); + assert_eq!(t.read8(reg::YEAR as u64).data, 0x99); + t.advance_for_test(1); + assert_eq!(clock_cells(&t), [0, 0, 0, 0x05, 0x01, 0x01, 0]); + } + + /// A guest reading the clock one byte at a time needs the read bit to + /// freeze every counter at once, or a rollover tears the timestamp. + #[test] + fn the_read_bit_freezes_all_counters_until_it_is_cleared() { + let t = Timekeeper::fuel_at(2026, 1, 1, 0, 0, 0).unwrap(); + let end_of_century = [0x59, 0x59, 0x23, 0x05, 0x31, 0x12, 0x99]; + set_clock(&t, end_of_century); + t.write8(CONTROL as u64, 0x40); + assert_eq!(t.read8(CONTROL as u64).data, 0x40); + t.advance_for_test(2); + assert_eq!(clock_cells(&t), end_of_century); + t.write8(CONTROL as u64, 0); + assert_eq!(clock_cells(&t), [0x01, 0, 0, 0x05, 0x01, 0x01, 0]); + } + + /// Writes in the write-enable window are staged: the guest sets six + /// counters and the weekday, and nothing but the weekday is visible until + /// the window closes. + #[test] + fn staged_clock_writes_apply_only_when_the_write_bit_clears() { + let t = Timekeeper::fuel_at(2026, 1, 2, 3, 4, 5).unwrap(); + let before = clock_cells(&t); + t.write8(CONTROL as u64, 0x80); + for (i, v) in [0x59u8, 0x58, 0x17, 0x03, 0x15, 0x06, 0x30].into_iter().enumerate() { + t.write8(reg::SECONDS as u64 + i as u64, v); + } + let during = clock_cells(&t); + for i in (0..7).filter(|&i| i != 3) { + assert_eq!(during[i], before[i], "cell {i}"); + } + assert_eq!(t.read8(WEEKDAY).data, 0x03); + t.write8(CONTROL as u64, 0); + assert_eq!(clock_cells(&t), [0x59, 0x58, 0x17, 0x03, 0x15, 0x06, 0x30]); + } + + /// The stop bit is how a guest parks the oscillator while it sets the time; + /// the clock must hold until the guest clears it, then continue from there. + #[test] + fn the_stop_bit_holds_the_seconds_until_it_is_cleared() { + let t = Timekeeper::fuel_at(2026, 1, 1, 0, 0, 0).unwrap(); + t.write8(CONTROL as u64, 0x80); + t.write8(reg::SECONDS as u64, 0x82); + t.write8(CONTROL as u64, 0); + t.advance_for_test(3); + assert_eq!(t.read8(reg::SECONDS as u64).data, 0x82); + t.write8(CONTROL as u64, 0x80); + t.write8(reg::SECONDS as u64, 0x02); + t.write8(CONTROL as u64, 0); + t.advance_for_test(1); + assert_eq!(t.read8(reg::SECONDS as u64).data, 0x03); + } + + /// The 32 KiB window ends at the year register; an access past it must + /// fault, and ordinary storage must keep its contents across clock writes. + #[test] + fn accesses_past_the_window_fault_and_ordinary_storage_is_untouched() { + let t = Timekeeper::fuel_at(2026, 1, 1, 0, 0, 0).unwrap(); + t.write8(0, 0x5a); + t.write8(0x7ff7, 0xa5); + assert_eq!(t.write8(SIZE as u64, 0x69), BUS_ERR); + assert_ne!(t.read8(SIZE as u64).status, BUS_OK); + set_clock(&t, [0, 0, 0, 0x05, 0x01, 0x03, 0x58]); + assert_eq!(t.read8(0).data, 0x5a); + assert_eq!(t.read8(0x7ff7).data, 0xa5); + } +} diff --git a/src/dev/usb_hid.rs b/src/dev/usb_hid.rs new file mode 100644 index 000000000..8f417c05b --- /dev/null +++ b/src/dev/usb_hid.rs @@ -0,0 +1,609 @@ +//! USB HID boot-protocol keyboard and mouse: low-speed functions that plug +//! into an OHCI root port and are fed from the host's keyboard and mouse. +//! +//! Written from the public specifications: USB 1.1 (chapter 9 requests and +//! descriptors), Device Class Definition for HID 1.11 (class requests and +//! Appendix B, the boot interface descriptors and report formats) and the HID +//! Usage Tables (keyboard/keypad page). Each device has one configuration, +//! one boot interface and one interrupt-IN endpoint. +//! +//! The control pipe follows USB 1.1 section 8.5.2: a SETUP packet, an optional +//! data stage and a status stage. SET_ADDRESS takes effect once its status +//! stage completes. Interrupt reports queue one per host-side change so a key +//! tapped and released between two polls is still seen twice. Input that +//! arrives before the guest has selected a configuration is dropped, as an +//! unconfigured device would not report it. +//! +//! The data toggle on the wire is not tracked: the controller model keeps +//! each endpoint's toggle and a lost transfer cannot be replayed here. +//! Remote wakeup, the wheel, and the host keyboard layout are not modelled. + +use crate::dev::ohci::{Pid, UsbFunction, UsbOutcome, UsbPacket}; +use crate::dev::ps2::HostInputSink; +use parking_lot::Mutex; +use std::collections::VecDeque; +use std::sync::Arc; +use winit::keyboard::KeyCode; + +/// A vendor and product pair from the pid.codes open test allocation, since +/// the devices are not products of any vendor. +const VENDOR_ID: u16 = 0x1209; +const PRODUCT_ID: u16 = 0x0001; + +// Standard request codes (USB 1.1 table 9-4) and descriptor types (table 9-5). +const GET_STATUS: u8 = 0; +const CLEAR_FEATURE: u8 = 1; +const SET_FEATURE: u8 = 3; +const SET_ADDRESS: u8 = 5; +const GET_DESCRIPTOR: u8 = 6; +const GET_CONFIGURATION: u8 = 8; +const SET_CONFIGURATION: u8 = 9; +const GET_INTERFACE: u8 = 10; +const SET_INTERFACE: u8 = 11; +const DESC_DEVICE: u8 = 1; +const DESC_CONFIGURATION: u8 = 2; +const DESC_STRING: u8 = 3; +const DESC_HID: u8 = 0x21; +const DESC_REPORT: u8 = 0x22; +// HID class requests (HID 1.11 section 7.2). +const GET_REPORT: u8 = 1; +const GET_IDLE: u8 = 2; +const GET_PROTOCOL: u8 = 3; +const SET_REPORT: u8 = 9; +const SET_IDLE: u8 = 10; +const SET_PROTOCOL: u8 = 11; + +/// HID 1.11 appendix B.1: the boot keyboard's report descriptor. Input: a +/// modifier bit per key and six key usages; output: five LED bits. +pub const KEYBOARD_REPORT_DESCRIPTOR: [u8; 63] = [ + 0x05, 0x01, 0x09, 0x06, 0xa1, 0x01, 0x05, 0x07, 0x19, 0xe0, 0x29, 0xe7, 0x15, 0x00, 0x25, 0x01, + 0x75, 0x01, 0x95, 0x08, 0x81, 0x02, 0x95, 0x01, 0x75, 0x08, 0x81, 0x01, 0x95, 0x05, 0x75, 0x01, + 0x05, 0x08, 0x19, 0x01, 0x29, 0x05, 0x91, 0x02, 0x95, 0x01, 0x75, 0x03, 0x91, 0x01, 0x95, 0x06, + 0x75, 0x08, 0x15, 0x00, 0x25, 0x65, 0x05, 0x07, 0x19, 0x00, 0x29, 0x65, 0x81, 0x00, 0xc0, +]; +/// HID 1.11 appendix B.2: the boot mouse's report descriptor. Three buttons +/// and relative X and Y. +pub const MOUSE_REPORT_DESCRIPTOR: [u8; 50] = [ + 0x05, 0x01, 0x09, 0x02, 0xa1, 0x01, 0x09, 0x01, 0xa1, 0x00, 0x05, 0x09, 0x19, 0x01, 0x29, 0x03, + 0x15, 0x00, 0x25, 0x01, 0x95, 0x03, 0x75, 0x01, 0x81, 0x02, 0x95, 0x01, 0x75, 0x05, 0x81, 0x01, + 0x05, 0x01, 0x09, 0x30, 0x09, 0x31, 0x15, 0x81, 0x25, 0x7f, 0x75, 0x08, 0x95, 0x02, 0x81, 0x06, + 0xc0, 0xc0, +]; + +/// Reports held for the guest before the oldest is dropped. +const QUEUE_LIMIT: usize = 64; + +/// What differs between the two devices: the report they send and how host +/// input becomes one. +pub trait Boot: Send { + const PRODUCT: &'static str; + /// bInterfaceProtocol: 1 keyboard, 2 mouse. + const PROTOCOL: u8; + const REPORT_DESCRIPTOR: &'static [u8]; + const MAX_PACKET: u8; + /// The idle rate at reset, in 4 ms units (HID 1.11 section 7.2.4). + const DEFAULT_IDLE: u8; + fn reset(&mut self); + /// The report GET_REPORT returns now. + fn current(&self) -> Vec; + /// The next queued report, if any. + fn next_report(&mut self) -> Option>; + /// An output report; false when the device has none. + fn set_output(&mut self, _data: &[u8]) -> bool { + false + } + /// Host input waiting for the guest, in reports. + fn backlog(&self) -> usize; +} + +enum Stage { + Idle, + /// Data stage toward the host, then an OUT status. + In { data: Vec, sent: usize }, + /// Data stage from the host for a SET_REPORT of `length` bytes. + Out { length: usize, data: Vec }, + /// Status stage of a request with no data toward the host: the next IN + /// answers with an empty packet and the request takes effect. + StatusIn(Effect), +} + +#[derive(Clone, Copy)] +enum Effect { + None, + Address(u8), + Configuration(u8), + Idle(u8), + Protocol(u8), + Output, +} + +struct Inner { + dev: B, + address: u8, + configured: bool, + /// 0 boot, 1 report; a device starts in report protocol. + protocol: u8, + idle: u8, + remote_wakeup: bool, + stage: Stage, + last_report_ms: u64, +} + +pub struct UsbHid { + inner: Mutex>, +} + +impl UsbHid { + fn with(dev: B) -> Arc { + Arc::new(Self { + inner: Mutex::new(Inner { + dev, + address: 0, + configured: false, + protocol: 1, + idle: B::DEFAULT_IDLE, + remote_wakeup: false, + stage: Stage::Idle, + last_report_ms: 0, + }), + }) + } +} + +fn string_descriptor(s: &str) -> Vec { + let units: Vec = s.encode_utf16().flat_map(u16::to_le_bytes).collect(); + let mut d = vec![2 + units.len() as u8, DESC_STRING]; + d.extend(units); + d +} + +impl Inner { + fn device_descriptor() -> Vec { + let mut d = vec![18, DESC_DEVICE, 0x10, 0x01, 0, 0, 0, 8]; + d.extend(VENDOR_ID.to_le_bytes()); + d.extend(PRODUCT_ID.to_le_bytes()); + // bcdDevice, manufacturer and product strings, no serial, one configuration. + d.extend([0x00, 0x01, 1, 2, 0, 1]); + d + } + + fn hid_descriptor() -> Vec { + let n = B::REPORT_DESCRIPTOR.len() as u16; + let mut d = vec![9, DESC_HID, 0x11, 0x01, 0, 1, DESC_REPORT]; + d.extend(n.to_le_bytes()); + d + } + + /// The configuration descriptor followed by its interface, HID and + /// endpoint descriptors, as GET_DESCRIPTOR(configuration) returns them. + fn configuration_descriptor() -> Vec { + let interface = [9, 4, 0, 0, 1, 3, 1, B::PROTOCOL, 0]; + let hid = Self::hid_descriptor(); + // Interrupt IN endpoint 1, polled every 10 ms. + let endpoint = [7, 5, 0x81, 3, B::MAX_PACKET, 0, 10]; + let total = (9 + interface.len() + hid.len() + endpoint.len()) as u16; + let mut d = vec![9, DESC_CONFIGURATION]; + d.extend(total.to_le_bytes()); + // One interface, configuration 1, no string, bus powered with remote + // wakeup, 100 mA. + d.extend([1, 1, 0, 0xa0, 50]); + d.extend(interface); + d.extend(hid); + d.extend(endpoint); + d + } + + fn string(index: u8) -> Option> { + match index { + 0 => Some(vec![4, DESC_STRING, 0x09, 0x04]), + 1 => Some(string_descriptor("iris")), + 2 => Some(string_descriptor(B::PRODUCT)), + _ => None, + } + } + + fn reset(&mut self) { + self.address = 0; + self.configured = false; + self.protocol = 1; + self.idle = B::DEFAULT_IDLE; + self.remote_wakeup = false; + self.stage = Stage::Idle; + self.dev.reset(); + } + + /// Decode a SETUP packet (USB 1.1 table 9-2). None is a request this + /// device does not have, answered with a STALL. + fn setup(&mut self, p: &[u8]) -> Option<()> { + let [rt, req, vl, vh, il, _ih, ll, lh] = <[u8; 8]>::try_from(p).ok()?; + let value = u16::from_le_bytes([vl, vh]); + let length = u16::from_le_bytes([ll, lh]) as usize; + let reply = |data: Vec| Stage::In { sent: 0, data: data[..data.len().min(length)].to_vec() }; + let (kind, recipient) = ((rt >> 5) & 3, rt & 0x1f); + let to_host = rt & 0x80 != 0; + self.stage = match (kind, recipient, to_host, req) { + // Standard requests (USB 1.1 section 9.4). + (0, 0, true, GET_STATUS) => reply(vec![u8::from(self.remote_wakeup) << 1, 0]), + (0, 1 | 2, true, GET_STATUS) if value == 0 => reply(vec![0, 0]), + (0, 0, false, CLEAR_FEATURE) if value == 1 => { + self.remote_wakeup = false; + Stage::StatusIn(Effect::None) + } + (0, 0, false, SET_FEATURE) if value == 1 => { + self.remote_wakeup = true; + Stage::StatusIn(Effect::None) + } + // Clearing a halt on the interrupt endpoint is accepted; nothing halts it. + (0, 2, false, CLEAR_FEATURE) if value == 0 && il & 0x8f == 0x81 => Stage::StatusIn(Effect::None), + (0, 0, false, SET_ADDRESS) if value <= 127 => Stage::StatusIn(Effect::Address(value as u8)), + (0, 0, true, GET_DESCRIPTOR) => match (vh, vl) { + (DESC_DEVICE, 0) => reply(Self::device_descriptor()), + (DESC_CONFIGURATION, 0) => reply(Self::configuration_descriptor()), + (DESC_STRING, n) => reply(Self::string(n)?), + _ => return None, + }, + (0, 0, true, GET_CONFIGURATION) => reply(vec![u8::from(self.configured)]), + (0, 0, false, SET_CONFIGURATION) if value <= 1 => Stage::StatusIn(Effect::Configuration(value as u8)), + (0, 1, true, GET_INTERFACE) if il == 0 => reply(vec![0]), + (0, 1, false, SET_INTERFACE) if il == 0 && value == 0 => Stage::StatusIn(Effect::None), + // The HID descriptor and report descriptor belong to the interface. + (0, 1, true, GET_DESCRIPTOR) if il == 0 => match (vh, vl) { + (DESC_HID, 0) => reply(Self::hid_descriptor()), + (DESC_REPORT, 0) => reply(B::REPORT_DESCRIPTOR.to_vec()), + _ => return None, + }, + // HID class requests (HID 1.11 section 7.2), interface 0. + (1, 1, true, GET_REPORT) if il == 0 && vh == 1 && vl == 0 => reply(self.dev.current()), + (1, 1, false, SET_REPORT) if il == 0 && vh == 2 && vl == 0 && length > 0 => { + Stage::Out { length, data: Vec::new() } + } + (1, 1, true, GET_IDLE) if il == 0 => reply(vec![self.idle]), + (1, 1, false, SET_IDLE) if il == 0 => Stage::StatusIn(Effect::Idle(vh)), + (1, 1, true, GET_PROTOCOL) if il == 0 => reply(vec![self.protocol]), + (1, 1, false, SET_PROTOCOL) if il == 0 && value <= 1 => Stage::StatusIn(Effect::Protocol(value as u8)), + _ => return None, + }; + Some(()) + } + + fn apply(&mut self, effect: Effect, data: &[u8], now_ms: u64) -> bool { + match effect { + Effect::None => {} + Effect::Address(a) => self.address = a, + Effect::Configuration(c) => { + self.configured = c != 0; + self.dev.reset(); + self.last_report_ms = now_ms; + } + Effect::Idle(rate) => self.idle = rate, + Effect::Protocol(p) => self.protocol = p, + Effect::Output => return self.dev.set_output(data), + } + true + } + + fn control(&mut self, p: UsbPacket) -> UsbOutcome { + match p.pid { + Pid::Setup => match self.setup(&p.data) { + Some(()) => UsbOutcome::Ack(Vec::new()), + None => { + self.stage = Stage::Idle; + UsbOutcome::Stall + } + }, + Pid::In => match &mut self.stage { + Stage::In { data, sent } => { + let n = p.capacity.min(data.len() - *sent); + let chunk = data[*sent..*sent + n].to_vec(); + *sent += n; + UsbOutcome::Ack(chunk) + } + Stage::StatusIn(effect) => { + let effect = *effect; + self.stage = Stage::Idle; + if self.apply(effect, &[], p.time_ms) { UsbOutcome::Ack(Vec::new()) } else { UsbOutcome::Stall } + } + _ => UsbOutcome::Stall, + }, + Pid::Out => match &mut self.stage { + // The status stage of a request that sent data to the host. + Stage::In { .. } if p.data.is_empty() => { + self.stage = Stage::Idle; + UsbOutcome::Ack(Vec::new()) + } + Stage::Out { length, data } => { + data.extend(&p.data); + if data.len() >= *length { + let data = std::mem::take(data); + let ok = self.apply(Effect::Output, &data, p.time_ms); + self.stage = if ok { Stage::StatusIn(Effect::None) } else { Stage::Idle }; + if !ok { + return UsbOutcome::Stall; + } + } + UsbOutcome::Ack(Vec::new()) + } + _ => UsbOutcome::Stall, + }, + } + } + + /// Interrupt IN endpoint 1 (USB 1.1 section 5.7): a report when there is + /// one, otherwise NAK. A nonzero idle rate repeats the last state once the + /// period passes without a change. + fn interrupt(&mut self, p: UsbPacket) -> UsbOutcome { + if !self.configured { + return UsbOutcome::Stall; + } + if let Some(r) = self.dev.next_report() { + self.last_report_ms = p.time_ms; + return UsbOutcome::Ack(r); + } + if self.idle != 0 && p.time_ms.saturating_sub(self.last_report_ms) >= u64::from(self.idle) * 4 { + self.last_report_ms = p.time_ms; + return UsbOutcome::Ack(self.dev.current()); + } + UsbOutcome::Nak + } +} + +impl UsbFunction for UsbHid { + fn reset(&self) { + self.inner.lock().reset(); + } + fn address(&self) -> u8 { + self.inner.lock().address + } + fn low_speed(&self) -> bool { + true + } + fn transact(&self, packet: UsbPacket) -> UsbOutcome { + let mut s = self.inner.lock(); + match packet.endpoint { + 0 => s.control(packet), + 1 if packet.pid == Pid::In => s.interrupt(packet), + _ => UsbOutcome::Stall, + } + } +} + +// --------------------------------------------------------------------------- +// Keyboard +// --------------------------------------------------------------------------- + +/// The HID usage (keyboard/keypad page 0x07) of a host key, or None for a +/// key the page has no entry for. +pub fn usage(key: KeyCode) -> Option { + use KeyCode::*; + Some(match key { + KeyA => 0x04, KeyB => 0x05, KeyC => 0x06, KeyD => 0x07, KeyE => 0x08, KeyF => 0x09, + KeyG => 0x0a, KeyH => 0x0b, KeyI => 0x0c, KeyJ => 0x0d, KeyK => 0x0e, KeyL => 0x0f, + KeyM => 0x10, KeyN => 0x11, KeyO => 0x12, KeyP => 0x13, KeyQ => 0x14, KeyR => 0x15, + KeyS => 0x16, KeyT => 0x17, KeyU => 0x18, KeyV => 0x19, KeyW => 0x1a, KeyX => 0x1b, + KeyY => 0x1c, KeyZ => 0x1d, + Digit1 => 0x1e, Digit2 => 0x1f, Digit3 => 0x20, Digit4 => 0x21, Digit5 => 0x22, + Digit6 => 0x23, Digit7 => 0x24, Digit8 => 0x25, Digit9 => 0x26, Digit0 => 0x27, + Enter => 0x28, Escape => 0x29, Backspace => 0x2a, Tab => 0x2b, Space => 0x2c, + Minus => 0x2d, Equal => 0x2e, BracketLeft => 0x2f, BracketRight => 0x30, + Backslash => 0x31, Semicolon => 0x33, Quote => 0x34, Backquote => 0x35, + Comma => 0x36, Period => 0x37, Slash => 0x38, CapsLock => 0x39, + F1 => 0x3a, F2 => 0x3b, F3 => 0x3c, F4 => 0x3d, F5 => 0x3e, F6 => 0x3f, + F7 => 0x40, F8 => 0x41, F9 => 0x42, F10 => 0x43, F11 => 0x44, F12 => 0x45, + PrintScreen => 0x46, ScrollLock => 0x47, Pause => 0x48, Insert => 0x49, + Home => 0x4a, PageUp => 0x4b, Delete => 0x4c, End => 0x4d, PageDown => 0x4e, + ArrowRight => 0x4f, ArrowLeft => 0x50, ArrowDown => 0x51, ArrowUp => 0x52, + NumLock => 0x53, NumpadDivide => 0x54, NumpadMultiply => 0x55, + NumpadSubtract => 0x56, NumpadAdd => 0x57, NumpadEnter => 0x58, + Numpad1 => 0x59, Numpad2 => 0x5a, Numpad3 => 0x5b, Numpad4 => 0x5c, + Numpad5 => 0x5d, Numpad6 => 0x5e, Numpad7 => 0x5f, Numpad8 => 0x60, + Numpad9 => 0x61, Numpad0 => 0x62, NumpadDecimal => 0x63, + IntlBackslash => 0x64, ContextMenu => 0x65, + _ => return None, + }) +} + +/// The bit of a modifier key in the report's first byte (usages 0xe0 to 0xe7). +fn modifier(key: KeyCode) -> Option { + use KeyCode::*; + Some(match key { + ControlLeft => 0x01, ShiftLeft => 0x02, AltLeft => 0x04, SuperLeft => 0x08, + ControlRight => 0x10, ShiftRight => 0x20, AltRight => 0x40, SuperRight => 0x80, + _ => return None, + }) +} + +#[derive(Default)] +pub struct Keyboard { + modifiers: u8, + /// Pressed keys in the order they went down. + keys: Vec, + queue: VecDeque>, + leds: u8, +} + +impl Keyboard { + /// The 8-byte boot report (HID 1.11 B.1): modifiers, reserved, six keys. + /// More than six keys at once is the phantom state, ErrorRollOver in every slot. + fn report(&self) -> Vec { + let mut r = vec![self.modifiers, 0, 0, 0, 0, 0, 0, 0]; + if self.keys.len() > 6 { + r[2..].fill(0x01); + } else { + r[2..2 + self.keys.len()].copy_from_slice(&self.keys); + } + r + } + + fn push(&mut self) { + if self.queue.len() >= QUEUE_LIMIT { + self.queue.pop_front(); + } + let report = self.report(); + self.queue.push_back(report); + } +} + +impl Boot for Keyboard { + const PRODUCT: &'static str = "USB Keyboard"; + const PROTOCOL: u8 = 1; + const REPORT_DESCRIPTOR: &'static [u8] = &KEYBOARD_REPORT_DESCRIPTOR; + const MAX_PACKET: u8 = 8; + // 500 ms, the recommended keyboard default. + const DEFAULT_IDLE: u8 = 125; + fn reset(&mut self) { + *self = Keyboard::default(); + } + fn current(&self) -> Vec { + self.report() + } + fn next_report(&mut self) -> Option> { + self.queue.pop_front() + } + /// The LED output report (HID 1.11 B.1): num lock, caps lock, scroll + /// lock, compose, kana in bits 0 to 4. + fn set_output(&mut self, data: &[u8]) -> bool { + self.leds = data.first().copied().unwrap_or(0) & 0x1f; + true + } + fn backlog(&self) -> usize { + self.queue.len() + } +} + +pub type UsbKeyboard = UsbHid; + +impl UsbKeyboard { + pub fn new() -> Arc { + Self::with(Keyboard::default()) + } + + /// The LEDs the guest last asked for. + pub fn leds(&self) -> u8 { + self.inner.lock().dev.leds + } +} + +impl HostInputSink for UsbKeyboard { + fn key(&self, key: KeyCode, pressed: bool) { + let mut s = self.inner.lock(); + if !s.configured { + return; + } + let k = &mut s.dev; + if let Some(bit) = modifier(key) { + let before = k.modifiers; + if pressed { k.modifiers |= bit } else { k.modifiers &= !bit } + if k.modifiers != before { + k.push(); + } + } else if let Some(u) = usage(key) { + let at = k.keys.iter().position(|&x| x == u); + match (pressed, at) { + (true, None) => k.keys.push(u), + (false, Some(i)) => { + k.keys.remove(i); + } + _ => return, + } + k.push(); + } + } + fn mouse(&self, _: u8, _: i32, _: i32, _: i32) {} + fn mouse_enabled(&self) -> bool { + false + } + fn mouse_pending(&self) -> usize { + 0 + } +} + +// --------------------------------------------------------------------------- +// Mouse +// --------------------------------------------------------------------------- + +#[derive(Default)] +pub struct Mouse { + buttons: u8, + /// Movement not yet in a report. + dx: i32, + dy: i32, + queue: VecDeque>, +} + +impl Mouse { + /// Move accumulated motion into reports, at most a signed byte on each + /// axis apiece (HID 1.11 B.2: buttons, X, Y). `force` sends one even + /// with no motion, for a button change. + fn flush(&mut self, mut force: bool) { + while (force || self.dx != 0 || self.dy != 0) && self.queue.len() < QUEUE_LIMIT { + let (x, y) = (self.dx.clamp(-127, 127), self.dy.clamp(-127, 127)); + self.dx -= x; + self.dy -= y; + self.queue.push_back(vec![self.buttons, x as i8 as u8, y as i8 as u8]); + force = false; + } + } +} + +impl Boot for Mouse { + const PRODUCT: &'static str = "USB Mouse"; + const PROTOCOL: u8 = 2; + const REPORT_DESCRIPTOR: &'static [u8] = &MOUSE_REPORT_DESCRIPTOR; + const MAX_PACKET: u8 = 4; + // A mouse reports only on change. + const DEFAULT_IDLE: u8 = 0; + fn reset(&mut self) { + *self = Mouse::default(); + } + fn current(&self) -> Vec { + vec![self.buttons, 0, 0] + } + fn next_report(&mut self) -> Option> { + if self.queue.is_empty() { + self.flush(false); + } + self.queue.pop_front() + } + fn backlog(&self) -> usize { + self.queue.len() + usize::from(self.dx != 0 || self.dy != 0) + } +} + +pub type UsbMouse = UsbHid; + +impl UsbMouse { + pub fn new() -> Arc { + Self::with(Mouse::default()) + } +} + +impl HostInputSink for UsbMouse { + fn key(&self, _: KeyCode, _: bool) {} + fn mouse(&self, buttons: u8, dx: i32, dy: i32, _dz: i32) { + let mut s = self.inner.lock(); + if !s.configured { + return; + } + let m = &mut s.dev; + m.dx = m.dx.saturating_add(dx); + m.dy = m.dy.saturating_add(dy); + let buttons = buttons & 7; + // Motion up to now travels with the old button state. + if buttons != m.buttons { + m.flush(false); + m.buttons = buttons; + m.flush(true); + } else { + m.flush(false); + } + } + fn mouse_enabled(&self) -> bool { + self.inner.lock().configured + } + fn mouse_pending(&self) -> usize { + self.inner.lock().dev.backlog() + } +} + +#[cfg(test)] +mod tests; diff --git a/src/dev/usb_hid/tests.rs b/src/dev/usb_hid/tests.rs new file mode 100644 index 000000000..5efe1a8b2 --- /dev/null +++ b/src/dev/usb_hid/tests.rs @@ -0,0 +1,448 @@ +// The cases follow USB 1.1 chapter 9 and HID 1.11: descriptor layout and +// lengths, the control pipe's stages, and the boot report formats. The last +// ones put a device behind the OHCI model to show an interrupt transfer +// returns a report through guest memory. +use super::*; +use crate::dev::ohci::Ohci; +use crate::pci::{PciBusDevice, PciDma, PciDmaStatus}; +use crate::traits::Device; + +fn packet(endpoint: u8, pid: Pid, capacity: usize, data: &[u8], time_ms: u64) -> UsbPacket { + UsbPacket { endpoint, pid, toggle: false, capacity, data: data.to_vec(), time_ms } +} + +fn ack(o: UsbOutcome) -> Vec { + match o { + UsbOutcome::Ack(b) => b, + other => panic!("expected an ACK, got {other:?}"), + } +} + +/// A control transfer toward the host: SETUP, IN packets of 8 until short, +/// then the OUT status. +fn control_in(f: &dyn UsbFunction, rt: u8, req: u8, value: u16, index: u16, length: u16) -> Result, UsbOutcome> { + let mut setup = vec![rt, req]; + setup.extend(value.to_le_bytes()); + setup.extend(index.to_le_bytes()); + setup.extend(length.to_le_bytes()); + match f.transact(packet(0, Pid::Setup, 8, &setup, 0)) { + UsbOutcome::Ack(_) => {} + other => return Err(other), + } + let mut data = Vec::new(); + loop { + let chunk = ack(f.transact(packet(0, Pid::In, 8, &[], 0))); + let short = chunk.len() < 8; + data.extend(chunk); + if short || data.len() >= length as usize { + break; + } + } + ack(f.transact(packet(0, Pid::Out, 8, &[], 0))); + Ok(data) +} + +/// A control transfer with no data stage: SETUP then the IN status. +fn control_nodata(f: &dyn UsbFunction, rt: u8, req: u8, value: u16, index: u16) -> UsbOutcome { + let mut setup = vec![rt, req]; + setup.extend(value.to_le_bytes()); + setup.extend(index.to_le_bytes()); + setup.extend(0u16.to_le_bytes()); + match f.transact(packet(0, Pid::Setup, 8, &setup, 0)) { + UsbOutcome::Ack(_) => f.transact(packet(0, Pid::In, 8, &[], 0)), + other => other, + } +} + +fn enumerate(f: &dyn UsbFunction, address: u8) { + f.reset(); + assert!(ack(control_nodata(f, 0x00, SET_ADDRESS, address as u16, 0)).is_empty()); + assert_eq!(f.address(), address); + assert!(ack(control_nodata(f, 0x00, SET_CONFIGURATION, 1, 0)).is_empty()); +} + +fn interrupt(f: &dyn UsbFunction, time_ms: u64) -> UsbOutcome { + f.transact(packet(1, Pid::In, 8, &[], time_ms)) +} + +#[test] +fn the_device_descriptor_names_a_low_speed_hid_function() { + for f in [&*UsbKeyboard::new() as &dyn UsbFunction, &*UsbMouse::new()] { + assert!(f.low_speed()); + let d = control_in(f, 0x80, GET_DESCRIPTOR, 0x0100, 0, 64).unwrap(); + assert_eq!(d.len(), 18); + assert_eq!(d[0], 18, "bLength"); + assert_eq!(d[1], 1, "bDescriptorType"); + assert_eq!(u16::from_le_bytes([d[2], d[3]]), 0x0110, "USB 1.1"); + assert_eq!(d[4], 0, "the class is in the interface"); + assert_eq!(d[7], 8, "a low-speed control pipe carries 8 bytes"); + assert_eq!(d[17], 1, "one configuration"); + // The guest may read just the first 8 bytes before addressing. + assert_eq!(control_in(f, 0x80, GET_DESCRIPTOR, 0x0100, 0, 8).unwrap(), d[..8]); + } +} + +/// A configuration, its interface, the HID descriptor and the endpoint come +/// back in one block whose wTotalLength covers them (USB 1.1 9.6.2). +#[test] +fn the_configuration_block_describes_one_boot_interface_and_one_interrupt_endpoint() { + for (f, protocol, report_len, packet_size) in [ + (&*UsbKeyboard::new() as &dyn UsbFunction, 1u8, KEYBOARD_REPORT_DESCRIPTOR.len(), 8u8), + (&*UsbMouse::new(), 2, MOUSE_REPORT_DESCRIPTOR.len(), 4), + ] { + let c = control_in(f, 0x80, GET_DESCRIPTOR, 0x0200, 0, 255).unwrap(); + assert_eq!(c.len(), 34); + assert_eq!(&c[..2], [9, 2]); + assert_eq!(u16::from_le_bytes([c[2], c[3]]) as usize, c.len(), "wTotalLength"); + assert_eq!((c[4], c[5]), (1, 1), "one interface; configuration value 1"); + assert_eq!(&c[9..14], [9, 4, 0, 0, 1], "interface 0, no alternate, one endpoint"); + assert_eq!(&c[14..17], [3, 1, protocol], "HID class, boot subclass, keyboard or mouse"); + assert_eq!(&c[18..20], [9, 0x21], "HID descriptor"); + assert_eq!(u16::from_le_bytes([c[20], c[21]]), 0x0111, "bcdHID 1.11"); + assert_eq!(&c[24..26], [0x22, report_len as u8], "one report descriptor of its length"); + assert_eq!(&c[27..], [7, 5, 0x81, 3, packet_size, 0, 10], "interrupt IN 1, polled at 10 ms"); + // The HID descriptor is also readable on its own through the interface. + assert_eq!(control_in(f, 0x81, GET_DESCRIPTOR, 0x2100, 0, 9).unwrap(), c[18..27]); + } +} + +#[test] +fn the_report_descriptors_are_the_boot_descriptors() { + let k = UsbKeyboard::new(); + let r = control_in(&*k, 0x81, GET_DESCRIPTOR, 0x2200, 0, 255).unwrap(); + assert_eq!(r, KEYBOARD_REPORT_DESCRIPTOR); + // Usage page generic desktop, usage keyboard, application collection. + assert_eq!(&r[..6], [0x05, 0x01, 0x09, 0x06, 0xa1, 0x01]); + assert_eq!(r.last(), Some(&0xc0), "collection ends"); + let m = UsbMouse::new(); + let r = control_in(&*m, 0x81, GET_DESCRIPTOR, 0x2200, 0, 255).unwrap(); + assert_eq!(r, MOUSE_REPORT_DESCRIPTOR); + assert_eq!(&r[..6], [0x05, 0x01, 0x09, 0x02, 0xa1, 0x01]); +} + +#[test] +fn strings_are_utf16_with_a_language_table() { + let k = UsbKeyboard::new(); + assert_eq!(control_in(&*k, 0x80, GET_DESCRIPTOR, 0x0300, 0, 255).unwrap(), [4, 3, 0x09, 0x04]); + let s = control_in(&*k, 0x80, GET_DESCRIPTOR, 0x0302, 0x0409, 255).unwrap(); + assert_eq!(s[0] as usize, s.len()); + let text: Vec = s[2..].chunks(2).map(|c| u16::from_le_bytes([c[0], c[1]])).collect(); + assert_eq!(String::from_utf16(&text).unwrap(), "USB Keyboard"); + assert!(control_in(&*k, 0x80, GET_DESCRIPTOR, 0x0309, 0x0409, 255).is_err(), "no such string"); +} + +/// SET_ADDRESS takes effect after its status stage, not before (USB 1.1 9.4.6). +#[test] +fn set_address_applies_after_the_status_stage() { + let m = UsbMouse::new(); + let mut setup = vec![0x00, SET_ADDRESS, 9, 0, 0, 0, 0, 0]; + assert!(ack(m.transact(packet(0, Pid::Setup, 8, &setup, 0))).is_empty()); + assert_eq!(m.address(), 0, "still the default address during the status stage"); + assert!(ack(m.transact(packet(0, Pid::In, 8, &[], 0))).is_empty()); + assert_eq!(m.address(), 9); + // An address above 127 is not one. + setup[2] = 200; + assert_eq!(m.transact(packet(0, Pid::Setup, 8, &setup, 0)), UsbOutcome::Stall); + // A bus reset returns the device to the default address. + m.reset(); + assert_eq!(m.address(), 0); +} + +#[test] +fn configuration_state_is_readable_and_unknown_requests_stall() { + let k = UsbKeyboard::new(); + assert_eq!(control_in(&*k, 0x80, GET_CONFIGURATION, 0, 0, 1).unwrap(), [0]); + assert!(ack(control_nodata(&*k, 0x00, SET_CONFIGURATION, 1, 0)).is_empty()); + assert_eq!(control_in(&*k, 0x80, GET_CONFIGURATION, 0, 0, 1).unwrap(), [1]); + assert_eq!(control_nodata(&*k, 0x00, SET_CONFIGURATION, 2, 0), UsbOutcome::Stall); + assert_eq!(control_in(&*k, 0x81, GET_INTERFACE, 0, 0, 1).unwrap(), [0]); + assert_eq!(control_in(&*k, 0x80, GET_STATUS, 0, 0, 2).unwrap(), [0, 0]); + // No device qualifier on a low-speed-only device; no such request. + assert!(control_in(&*k, 0x80, GET_DESCRIPTOR, 0x0600, 0, 10).is_err()); + assert_eq!(control_nodata(&*k, 0x00, 0x77, 0, 0), UsbOutcome::Stall); + // A stalled request leaves the pipe usable for the next SETUP. + assert_eq!(control_in(&*k, 0x80, GET_CONFIGURATION, 0, 0, 1).unwrap(), [1]); + // The interrupt endpoint is not there before a configuration is selected. + let fresh = UsbKeyboard::new(); + assert_eq!(interrupt(&*fresh, 0), UsbOutcome::Stall); +} + +#[test] +fn hid_class_requests_set_and_report_idle_and_protocol() { + let k = UsbKeyboard::new(); + // A device starts in report protocol with the keyboard's 500 ms idle. + assert_eq!(control_in(&*k, 0xa1, GET_PROTOCOL, 0, 0, 1).unwrap(), [1]); + assert_eq!(control_in(&*k, 0xa1, GET_IDLE, 0, 0, 1).unwrap(), [125]); + assert!(ack(control_nodata(&*k, 0x21, SET_PROTOCOL, 0, 0)).is_empty()); + assert_eq!(control_in(&*k, 0xa1, GET_PROTOCOL, 0, 0, 1).unwrap(), [0]); + // SET_IDLE carries the duration in the high byte of wValue. + assert!(ack(control_nodata(&*k, 0x21, SET_IDLE, 0, 0)).is_empty()); + assert_eq!(control_in(&*k, 0xa1, GET_IDLE, 0, 0, 1).unwrap(), [0]); + assert_eq!(control_nodata(&*k, 0x21, SET_PROTOCOL, 2, 0), UsbOutcome::Stall); + // A mouse starts with no idle repeat. + let m = UsbMouse::new(); + assert_eq!(control_in(&*m, 0xa1, GET_IDLE, 0, 0, 1).unwrap(), [0]); +} + +#[test] +fn set_report_takes_the_keyboards_leds_and_a_mouse_has_none() { + let k = UsbKeyboard::new(); + let setup = [0x21, SET_REPORT, 0, 2, 0, 0, 1, 0]; + assert!(ack(k.transact(packet(0, Pid::Setup, 8, &setup, 0))).is_empty()); + assert!(ack(k.transact(packet(0, Pid::Out, 8, &[0x1f | 0x80], 0))).is_empty()); + assert!(ack(k.transact(packet(0, Pid::In, 8, &[], 0))).is_empty(), "status stage"); + assert_eq!(k.leds(), 0x1f, "five LED bits"); + let m = UsbMouse::new(); + assert!(ack(m.transact(packet(0, Pid::Setup, 8, &setup, 0))).is_empty()); + assert_eq!(m.transact(packet(0, Pid::Out, 8, &[1], 0)), UsbOutcome::Stall); +} + +#[test] +fn keyboard_reports_carry_modifiers_and_keys_in_press_order() { + let k = UsbKeyboard::new(); + enumerate(&*k, 3); + assert_eq!(interrupt(&*k, 1), UsbOutcome::Nak, "nothing to report yet"); + k.key(KeyCode::ShiftLeft, true); + k.key(KeyCode::KeyA, true); + k.key(KeyCode::KeyB, true); + k.key(KeyCode::KeyA, false); + k.key(KeyCode::ShiftLeft, false); + k.key(KeyCode::KeyB, false); + let reports: Vec> = (0..6).map(|i| ack(interrupt(&*k, i))).collect(); + assert_eq!(reports, [ + vec![0x02, 0, 0, 0, 0, 0, 0, 0], + vec![0x02, 0, 0x04, 0, 0, 0, 0, 0], + vec![0x02, 0, 0x04, 0x05, 0, 0, 0, 0], + vec![0x02, 0, 0x05, 0, 0, 0, 0, 0], + vec![0x00, 0, 0x05, 0, 0, 0, 0, 0], + vec![0x00, 0, 0, 0, 0, 0, 0, 0], + ], "a tap between polls still reports both edges"); + assert_eq!(interrupt(&*k, 7), UsbOutcome::Nak); +} + +#[test] +fn keyboard_usages_follow_the_usage_table() { + assert_eq!(usage(KeyCode::KeyA), Some(0x04)); + assert_eq!(usage(KeyCode::KeyZ), Some(0x1d)); + assert_eq!(usage(KeyCode::Digit1), Some(0x1e)); + assert_eq!(usage(KeyCode::Digit0), Some(0x27)); + assert_eq!(usage(KeyCode::Enter), Some(0x28)); + assert_eq!(usage(KeyCode::Escape), Some(0x29)); + assert_eq!(usage(KeyCode::Space), Some(0x2c)); + assert_eq!(usage(KeyCode::F1), Some(0x3a)); + assert_eq!(usage(KeyCode::F12), Some(0x45)); + assert_eq!(usage(KeyCode::ArrowRight), Some(0x4f)); + assert_eq!(usage(KeyCode::ArrowUp), Some(0x52)); + assert_eq!(usage(KeyCode::Numpad0), Some(0x62)); + assert_eq!(usage(KeyCode::ControlLeft), None, "modifiers have a report bit, not a usage slot"); + assert_eq!(modifier(KeyCode::ControlLeft), Some(0x01)); + assert_eq!(modifier(KeyCode::SuperRight), Some(0x80)); +} + +#[test] +fn a_seventh_key_reports_rollover_and_release_recovers() { + let k = UsbKeyboard::new(); + enumerate(&*k, 1); + let keys = [KeyCode::KeyA, KeyCode::KeyB, KeyCode::KeyC, KeyCode::KeyD, KeyCode::KeyE, KeyCode::KeyF, KeyCode::KeyG]; + for key in keys { + k.key(key, true); + } + let reports: Vec> = (0..7).map(|i| ack(interrupt(&*k, i))).collect(); + assert_eq!(reports[5], [0, 0, 4, 5, 6, 7, 8, 9]); + assert_eq!(reports[6], [0, 0, 1, 1, 1, 1, 1, 1], "ErrorRollOver"); + k.key(KeyCode::KeyG, false); + assert_eq!(ack(interrupt(&*k, 8)), [0, 0, 4, 5, 6, 7, 8, 9]); + // A repeated press of a held key is not a new edge. + k.key(KeyCode::KeyA, true); + assert_eq!(interrupt(&*k, 9), UsbOutcome::Nak); +} + +#[test] +fn input_before_configuration_is_dropped_and_get_report_reads_the_state() { + let k = UsbKeyboard::new(); + k.key(KeyCode::KeyA, true); + enumerate(&*k, 2); + assert_eq!(interrupt(&*k, 1), UsbOutcome::Nak, "the early key never reached the report queue"); + k.key(KeyCode::KeyQ, true); + assert_eq!(control_in(&*k, 0xa1, GET_REPORT, 0x0100, 0, 8).unwrap(), [0, 0, 0x14, 0, 0, 0, 0, 0]); +} + +/// With a nonzero idle rate the keyboard repeats its state when the period +/// passes without a change (HID 1.11 7.2.4), and with zero it stays quiet. +#[test] +fn the_idle_rate_repeats_the_last_report() { + let k = UsbKeyboard::new(); + enumerate(&*k, 2); + k.key(KeyCode::KeyA, true); + assert_eq!(ack(interrupt(&*k, 10)), [0, 0, 4, 0, 0, 0, 0, 0]); + assert_eq!(interrupt(&*k, 509), UsbOutcome::Nak, "500 ms not yet passed"); + assert_eq!(ack(interrupt(&*k, 510)), [0, 0, 4, 0, 0, 0, 0, 0]); + assert!(ack(control_nodata(&*k, 0x21, SET_IDLE, 0, 0)).is_empty()); + assert_eq!(interrupt(&*k, 5000), UsbOutcome::Nak); +} + +#[test] +fn mouse_reports_carry_buttons_and_signed_motion() { + let m = UsbMouse::new(); + enumerate(&*m, 4); + assert!(m.mouse_enabled()); + assert_eq!(interrupt(&*m, 1), UsbOutcome::Nak); + // Host dy is positive down the screen, as HID Y is. + m.mouse(0, 10, -20, 0); + assert_eq!(ack(interrupt(&*m, 2)), [0, 10, (-20i8) as u8]); + // A press travels with its own report, then the release. + m.mouse(1, 0, 0, 0); + m.mouse(0, 0, 0, 0); + assert_eq!(ack(interrupt(&*m, 3)), [1, 0, 0]); + assert_eq!(ack(interrupt(&*m, 4)), [0, 0, 0]); + // Buttons in PS/2 order: right is 2, middle is 4. + m.mouse(2 | 4, 0, 0, 0); + assert_eq!(ack(interrupt(&*m, 5)), [6, 0, 0]); + assert_eq!(control_in(&*m, 0xa1, GET_REPORT, 0x0100, 0, 3).unwrap(), [6, 0, 0]); + // Boot protocol has no wheel byte: the report stays 3 bytes. + m.mouse(6, 0, 0, 3); + assert_eq!(interrupt(&*m, 6), UsbOutcome::Nak); +} + +#[test] +fn large_motion_splits_across_reports_without_loss() { + let m = UsbMouse::new(); + enumerate(&*m, 4); + m.mouse(0, 300, -200, 0); + assert_eq!(m.mouse_pending(), 3); + let mut sum = (0i32, 0i32); + let mut n = 0; + while let UsbOutcome::Ack(r) = interrupt(&*m, n) { + sum.0 += r[1] as i8 as i32; + sum.1 += r[2] as i8 as i32; + assert!(r[1] as i8 != -128 && r[2] as i8 != -128); + n += 1; + } + assert_eq!(sum, (300, -200)); + assert_eq!(m.mouse_pending(), 0); +} + +#[test] +fn a_mouse_before_configuration_is_not_enabled_and_drops_motion() { + let m = UsbMouse::new(); + assert!(!m.mouse_enabled()); + m.mouse(1, 5, 5, 0); + assert_eq!(m.mouse_pending(), 0); + enumerate(&*m, 1); + assert_eq!(interrupt(&*m, 1), UsbOutcome::Nak); +} + +/// Host input arrives through the PS/2 controller's sinks, so the one +/// interface the GUI, VNC and CI use feeds both kinds of device. +#[test] +fn host_input_reaches_usb_devices_through_the_ps2_controller() { + use crate::dev::ps2::Ps2Controller; + let ps2 = Ps2Controller::new(None); + Device::start(&ps2); + let (k, m) = (UsbKeyboard::new(), UsbMouse::new()); + ps2.add_sink(k.clone()); + ps2.add_sink(m.clone()); + enumerate(&*k, 1); + enumerate(&*m, 2); + assert!(ps2.mouse_enabled(), "the USB mouse counts: the PS/2 mouse is still off"); + ps2.push_kb(KeyCode::KeyA, true); + ps2.type_char('b'); + ps2.push_mouse_input(1, 3, 4, 0); + assert_eq!(ps2.mouse_pending(), 2, "a move and a button change wait for the guest"); + assert_eq!(ack(interrupt(&*k, 1)), [0, 0, 4, 0, 0, 0, 0, 0]); + assert_eq!(ack(interrupt(&*k, 2)), [0, 0, 4, 5, 0, 0, 0, 0]); + assert_eq!(ack(interrupt(&*m, 1)), [0, 3, 4]); + assert_eq!(ack(interrupt(&*m, 2)), [1, 0, 0]); +} + +// ---- Through the OHCI model ------------------------------------------------ + +struct Memory(Mutex>); +impl Memory { + fn new() -> Arc { + Arc::new(Self(Mutex::new(vec![0xa5; 0x10000]))) + } + fn words(&self, at: usize, words: &[u32]) { + for (i, w) in words.iter().enumerate() { + self.0.lock()[at + i * 4..at + i * 4 + 4].copy_from_slice(&w.to_le_bytes()); + } + } + fn word(&self, at: usize) -> u32 { + u32::from_le_bytes(self.0.lock()[at..at + 4].try_into().unwrap()) + } +} +impl PciDma for Memory { + fn dma_read(&self, addr: u64, dst: &mut [u8]) -> PciDmaStatus { + match self.0.lock().get(addr as usize..addr as usize + dst.len()) { + Some(s) => { + dst.copy_from_slice(s); + PciDmaStatus::Ok + } + None => PciDmaStatus::MasterAbort, + } + } + fn dma_write(&self, addr: u64, src: &[u8]) -> PciDmaStatus { + match self.0.lock().get_mut(addr as usize..addr as usize + src.len()) { + Some(s) => { + s.copy_from_slice(src); + PciDmaStatus::Ok + } + None => PciDmaStatus::MasterAbort, + } + } +} + +/// An OHCI controller with the device on port 0, reset and enabled, with +/// the periodic list holding one interrupt-IN endpoint at `address`. +fn ohci_with(f: Arc, address: u8) -> (Arc, Arc) { + let h = Ohci::new(); + let m = Memory::new(); + h.connect_dma(m.clone()); + h.attach(0, f.clone()).unwrap(); + h.start(); + h.bar_write(0, 0x54, 0x100, 4); + h.bar_write(0, 0x54, 0x10, 4); + h.advance_frames(10); + // The port reset returned the device to its default state: enumerate it. + enumerate(&*f, address); + m.words(0x1000, &[0; 64]); + // Interrupt table: every slot points at the endpoint. ED: address, endpoint 1, + // direction from the TD, low speed, max packet 8; TD queue 0x3000 .. tail 0x3020. + m.words(0x1000, &[0x2000; 32]); + m.words(0x2000, &[u32::from(address) | (1 << 7) | (1 << 13) | (8 << 16), 0x3020, 0x3000, 0]); + // TD: direction IN (2), buffer 0x4000..0x4007, buffer rounding allowed, interrupt delay 0. + m.words(0x3000, &[0xf000_0000 | (2 << 19) | (1 << 18), 0x4000, 0x3020, 0x4007]); + h.config().write(1, 6); + h.bar_write(0, 0x18, 0x1000, 4); + h.bar_write(0, 0x04, 0x84, 4); + (h, m) +} + +/// A report waiting in the device is carried by an interrupt TD into guest +/// memory, with the TD retired as a success. +#[test] +fn an_interrupt_transfer_returns_a_keyboard_report_through_ohci() { + let k = UsbKeyboard::new(); + let (h, m) = ohci_with(k.clone(), 5); + k.key(KeyCode::KeyA, true); + h.advance_frames(2); + let report = m.0.lock()[0x4000..0x4009].to_vec(); + // The done queue is published once the delay for a low-priority completion passes. + h.advance_frames(10); + assert_eq!(report, vec![0, 0, 4, 0, 0, 0, 0, 0, 0xa5], "the report, and nothing past its end"); + assert_eq!(m.word(0x3000) >> 28, 0, "completed without error"); + assert_eq!(m.word(0x1084) & !15, 0x3000, "the TD is on the done queue"); +} + +/// With nothing to report the device NAKs and the transfer stays queued. +#[test] +fn an_interrupt_transfer_with_no_report_stays_pending() { + let m = UsbMouse::new(); + let (h, mem) = ohci_with(m.clone(), 5); + h.advance_frames(3); + assert_eq!(mem.word(0x3000) >> 28, 0xf, "not accessed"); + m.mouse(2, 1, 1, 0); + h.advance_frames(2); + assert_eq!(&mem.0.lock()[0x4000..0x4003], [0, 1, 1]); + assert_eq!(mem.word(0x3000) >> 28, 0); +} diff --git a/src/dev/xbow.rs b/src/dev/xbow.rs index 85f395255..7ffadb503 100644 --- a/src/dev/xbow.rs +++ b/src/dev/xbow.rs @@ -13,22 +13,33 @@ //! lock, with the registers that describe them. Interrupt packets and the //! callbacks (link change, link reset) go out after that lock is dropped. //! -//! Revision code 5 (the Octane2's part) adds the reset fences and the -//! resource lock; the fences are kept but gate nothing, since no warm reset -//! travels through the modelled switch. -//! //! Two approximations: a link reset on a fitted link brings the link back //! alive as it was before the reset (link training is not modelled, and the //! stock PROM resets each link it finds and expects it back), and register //! accesses are 32-bit at the data half of each eight-byte slot only. //! +//! Revision 5 and later add the reset fences and the resource lock, and the +//! chip can carry a part number and manufacturer, a source field in its +//! status reads and an interrupt that goes to a packet sink instead of +//! HEART: what [`Pxbow`](super::pxbow::Pxbow) needs behind a PIC. +//! +//! With a clock ([`Xbow::with_clock`]) a link reset is the two-core-cycle +//! pulse the QEMU model has: the link is down for 20 ns of the clock, and +//! the chip ends the pulse the next time anything reads its registers or +//! asks [`Xbow::carries`], so what the guest sees follows the clock and not +//! host scheduling. Without one the pulse is over at once. [`Xbow::warm_reset`] +//! takes a warm reset from a link through the revision-5 fences. No board in +//! this tree sends one yet: the Hub's link control is the only sender the +//! QEMU model names. +//! //! Ported from hw/misc/sgi-xbow.c in jamesbraid/qemu-sgi-origami. -use std::sync::atomic::{AtomicBool, AtomicU8, Ordering}; +use std::sync::atomic::{AtomicBool, AtomicU64, AtomicU8, Ordering}; use std::sync::{Arc, OnceLock}; use parking_lot::Mutex; +use super::heart::Clock; use crate::irq::InterruptController; use crate::snapshot::{field_bool, field_u32, field_u32s, field_u8, StateTable}; use crate::traits::{BusDevice, BusRead16, BusRead32, BusRead64, BusRead8, BUS_ERR, BUS_OK, Saveable}; @@ -39,51 +50,62 @@ pub const fn widget_id(rev: u32, part: u32, mfgr: u32) -> u32 { } /// The widget HEART answers as, where widget 0's interrupt packets go. -const WIDGET_HEART: u8 = 8; +pub(crate) const WIDGET_HEART: u8 = 8; -const FIRST_WIDGET: u8 = 8; -const LINKS: usize = 8; +pub(crate) const FIRST_WIDGET: u8 = 8; +pub(crate) const LINKS: usize = 8; /// The register block's span in widget 0's window. pub const REG_SPAN: u32 = 0x0100_0000; -const IDENTIFICATION: u32 = 0x00; -const STATUS: u32 = 0x08; -const ERR_UPPER: u32 = 0x10; -const ERR_LOWER: u32 = 0x18; -const CONTROL: u32 = 0x20; -const PACKET_TIMEOUT: u32 = 0x28; -const INT_UPPER: u32 = 0x30; -const INT_LOWER: u32 = 0x38; -const ERR_CMD: u32 = 0x40; -const LLP_CONTROL: u32 = 0x48; -const STATUS_CLEAR: u32 = 0x50; -const ARB_RELOAD: u32 = 0x58; -const PERF_A: u32 = 0x60; -const PERF_B: u32 = 0x68; -const NIC: u32 = 0x70; -/// Revision code 5 and later: widget 0's reset fence, then links 8 to 15's -/// at a stride of 8, then the resource lock and its clear. -const FENCE_BASE: u32 = 0x78; -const FENCES: usize = 1 + LINKS; -const FENCE_END: u32 = FENCE_BASE + 8 * FENCES as u32; -const LOCK: u32 = 0xc0; -const LOCK_CLEAR: u32 = 0xc8; -const FENCE_MASK: u32 = 0xff; -/// The first revision code with fences and the lock (Crossbow 2.0 in the -/// software tables; an Octane2's code-5 part shows as 1.4 in its inventory). -const REVISION_FENCES: u32 = 5; -const LINK_BASE: u32 = 0x100; -const LINK_STRIDE: u32 = 0x40; -const LINK_END: u32 = LINK_BASE + LINK_STRIDE * LINKS as u32; -const LINK_FLUSH: u32 = 0x00; -const LINK_CONTROL: u32 = 0x08; -const LINK_STATUS: u32 = 0x10; -const LINK_ARB_UPPER: u32 = 0x18; -const LINK_ARB_LOWER: u32 = 0x20; -const LINK_STATUS_CLEAR: u32 = 0x28; -const LINK_RESET: u32 = 0x30; -const LINK_AUX_STATUS: u32 = 0x38; +pub(crate) const IDENTIFICATION: u32 = 0x00; +pub(crate) const STATUS: u32 = 0x08; +pub(crate) const ERR_UPPER: u32 = 0x10; +pub(crate) const ERR_LOWER: u32 = 0x18; +pub(crate) const CONTROL: u32 = 0x20; +pub(crate) const PACKET_TIMEOUT: u32 = 0x28; +pub(crate) const INT_UPPER: u32 = 0x30; +pub(crate) const INT_LOWER: u32 = 0x38; +pub(crate) const ERR_CMD: u32 = 0x40; +pub(crate) const LLP_CONTROL: u32 = 0x48; +pub(crate) const STATUS_CLEAR: u32 = 0x50; +pub(crate) const ARB_RELOAD: u32 = 0x58; +pub(crate) const PERF_A: u32 = 0x60; +pub(crate) const PERF_B: u32 = 0x68; +pub(crate) const NIC: u32 = 0x70; +const CORE_CYCLE_NS: u64 = 10; +/// How long a link reset holds the link down: two core cycles. +const LINK_RESET_NS: u64 = 2 * CORE_CYCLE_NS; +/// A warm reset arriving this many cycles after another is ignored. +const WARM_RESET_QUIET_CYCLES: u64 = 1024; +/// Revision 5 ("Crossbow 2.0") and later: a fence mask for widget 0's core, +/// one per link, and the resource lock. +const FENCED_REVISION: u32 = 5; +pub(crate) const FENCE: u32 = 0x78; +pub(crate) const LINK_FENCE_BASE: u32 = 0x80; +pub(crate) const LINK_FENCE_END: u32 = LINK_FENCE_BASE + 8 * LINKS as u32; +/// The fence block as the other crossbars' models name it: widget 0's +/// fence through link 15's. +pub(crate) const FENCE_BASE: u32 = FENCE; +pub(crate) const FENCE_END: u32 = LINK_FENCE_END; +const FENCE_RESET: u8 = 0xff; +pub(crate) const LOCK: u32 = 0xc0; +pub(crate) const LOCK_CLEAR: u32 = 0xc8; +/// Where a status read carries the widget that asked, when the chip is told +/// to. +const STATUS_SOURCE_SHIFT: u32 = 6; +const STATUS_SOURCE_MASK: u32 = 0xf << STATUS_SOURCE_SHIFT; +pub(crate) const LINK_BASE: u32 = 0x100; +pub(crate) const LINK_STRIDE: u32 = 0x40; +pub(crate) const LINK_END: u32 = LINK_BASE + LINK_STRIDE * LINKS as u32; +pub(crate) const LINK_FLUSH: u32 = 0x00; +pub(crate) const LINK_CONTROL: u32 = 0x08; +pub(crate) const LINK_STATUS: u32 = 0x10; +pub(crate) const LINK_ARB_UPPER: u32 = 0x18; +pub(crate) const LINK_ARB_LOWER: u32 = 0x20; +pub(crate) const LINK_STATUS_CLEAR: u32 = 0x28; +pub(crate) const LINK_RESET: u32 = 0x30; +pub(crate) const LINK_AUX_STATUS: u32 = 0x38; const STATUS_REQUEST: u32 = 1 << 23; const STATUS_LINK_REQUESTS: u32 = 0xff00_0000; @@ -136,6 +158,9 @@ struct Regs { perf_a: u32, perf_b: u32, nic: u32, + fence: u8, + link_fence: [u8; LINKS], + lock: u8, link_control: [u32; LINKS], link_status: [u32; LINKS], link_arb_upper: [u32; LINKS], @@ -148,12 +173,12 @@ struct Regs { initializing: u8, eight_bit: u8, interrupt_asserted: bool, - /// Revision code 5 and later: which ports' warm resets may reach the - /// core (fence 0) and each link (fences 1 to 8). Kept; iris sends no - /// warm resets through the switch. - fences: [u32; FENCES], - /// The resource lock's byte. - lock: u32, + /// When each link's reset pulse ends, in clock ns; 0 when none runs. + reset_due: [u64; LINKS], + /// Links that were alive when their pulse began and return alive. + reset_restore: u8, + /// A warm reset before this time is ignored. + warm_quiet_until: u64, } impl Regs { @@ -173,18 +198,22 @@ impl Regs { perf_a: PERF_RESET, perf_b: PERF_RESET, nic: NIC_RESET, + fence: FENCE_RESET, + link_fence: [FENCE_RESET; LINKS], + lock: 0, link_control: [LINK_CREDIT_RESET; LINKS], link_status: [0; LINKS], link_arb_upper: [0; LINKS], link_arb_lower: [0; LINKS], link_aux_status: [0; LINKS], link_event: [0; LINKS], - fences: [FENCE_MASK; FENCES], - lock: 0, came_up: startup, initializing: 0, eight_bit: 0, interrupt_asserted: false, + reset_due: [0; LINKS], + reset_restore: 0, + warm_quiet_until: 0, } } } @@ -194,13 +223,31 @@ impl Regs { struct Out { /// Widget 0's interrupt packets to HEART: (vector, set). packets: Vec<(u8, bool)>, + /// Packets to a widget other than HEART: widget, address, vector, set. + xio: Vec<(u8, u64, u8, bool)>, /// A link reset's far-end widget. - link_reset: Option, + link_resets: Vec, } +/// Where widget 0's interrupt goes when its target is not HEART: the target +/// widget, the 48-bit address, the vector and whether it raises. False is a +/// refusal. +pub type PacketSink = Box bool + Send + Sync>; + pub struct Xbow { regs: Mutex, revision: u32, + /// Revision field of the identification word when it differs from the + /// revision that selects behaviour. + reported_revision: Option, + part: u32, + manufacturer: u32, + /// The widget a status read reports as its source, if the chip does. + status_source: Option, + packets: OnceLock, + clock: Option, + /// The earliest running pulse's end, or `u64::MAX`. Written under `regs`. + reset_due: AtomicU64, /// Links with something attached, bit n for widget 8 + n. present: AtomicU8, /// Links passing traffic, as `present`. Written under `regs`. @@ -220,6 +267,9 @@ pub struct Xbow { unsupported: Mutex>, /// `unsupported` holds 64: a further access costs one relaxed load. unsupported_full: AtomicBool, + /// A write to an error address register completes; see + /// [`Xbow::with_posted_error_address_writes`]. + posted_error_address_writes: bool, } fn link_bit(widget: u8) -> Option { @@ -234,6 +284,13 @@ impl Xbow { Self { regs: Mutex::new(Regs::reset(present)), revision, + reported_revision: None, + part: 0, + manufacturer: 0, + status_source: None, + packets: OnceLock::new(), + clock: None, + reset_due: AtomicU64::new(u64::MAX), present: AtomicU8::new(present), alive: AtomicU8::new(present), irq_order: Mutex::new(()), @@ -242,9 +299,57 @@ impl Xbow { on_link_reset: OnceLock::new(), unsupported: Mutex::new(Vec::new()), unsupported_full: AtomicBool::new(false), + posted_error_address_writes: false, } } + /// Complete a write to either error address register instead of + /// refusing it. The registers are read-only and the write still changes + /// nothing, sets the register-access error and is listed in + /// [`Xbow::unsupported`]: the specification makes a write to a read-only + /// register an error condition that widget 0's status reports. What the processor sees is the other half: an + /// uncached store is a write request with no completion response (NEC + /// U10278EJ4V0UM p.104, table 6-1), so no error can come back to it. + /// A stock IP35 PROM writes the lower register on purpose in its crossbar + /// check. QEMU's `complete-posted-error-address-writes` property. + pub fn with_posted_error_address_writes(mut self) -> Self { + self.posted_error_address_writes = true; + self + } + + /// Answer to `part` and `manufacturer` in the identification word + /// instead of the Crossbow's zeros. + pub fn with_identity(mut self, part: u32, manufacturer: u32) -> Self { + self.part = part; + self.manufacturer = manufacturer; + self + } + + /// Report `revision` in the identification word while behaviour stays + /// that of the revision given to [`Xbow::new`]. + pub fn with_reported_revision(mut self, revision: u32) -> Self { + self.reported_revision = Some(revision); + self + } + + /// Time link resets and warm resets by `clock`, in nanoseconds. + pub fn with_clock(mut self, clock: Clock) -> Self { + self.clock = Some(clock); + self + } + + /// Report `widget` in bits 9:6 of every status read. + pub fn with_status_source(mut self, widget: u8) -> Self { + self.status_source = Some(widget); + self + } + + /// Send widget 0's interrupt to `sink` when it targets a widget other + /// than HEART. Once, before the machine runs. + pub fn connect_packets(&self, sink: PacketSink) { + let _ = self.packets.set(sink); + } + /// Where widget 0's interrupt packets go: HEART, whose line is the /// vector the packet names. Once, before the machine runs. pub fn connect_heart(&self, heart: Arc) { @@ -278,12 +383,26 @@ impl Xbow { let order = self.irq_order.lock(); let before = self.alive.load(Ordering::Acquire); let mut out = Out::default(); - let ret = f(&mut self.regs.lock(), &mut out); + let ret = { + let mut r = self.regs.lock(); + self.end_pulses(&mut r, &mut out); + let ret = f(&mut r, &mut out); + let due = r.reset_due.iter().copied().filter(|&d| d != 0).min().unwrap_or(u64::MAX); + self.reset_due.store(due, Ordering::Release); + ret + }; if let Some(h) = self.heart.get() { for (vector, set) in out.packets { h.set_line(vector as u32, set); } } + if let Some(sink) = self.packets.get() { + for (widget, address, vector, set) in out.xio { + if !sink(widget, address, vector, set) { + crate::dlog!(crate::devlog::LogModule::Bedrock, "xbow: interrupt packet to widget {widget} refused"); + } + } + } drop(order); let changed = before ^ self.alive.load(Ordering::Acquire); if let Some(cb) = self.on_link_change.get() { @@ -291,8 +410,10 @@ impl Xbow { cb(FIRST_WIDGET + link, self.alive.load(Ordering::Acquire) >> link & 1 != 0); } } - if let (Some(w), Some(cb)) = (out.link_reset, self.on_link_reset.get()) { - cb(w); + if let Some(cb) = self.on_link_reset.get() { + for w in out.link_resets { + cb(w); + } } ret } @@ -310,6 +431,17 @@ impl Xbow { /// Whether the switch passes traffic to and from `widget`. Lock-free. pub fn carries(&self, widget: u8) -> bool { + if let Some(clock) = &self.clock { + if clock() >= self.reset_due.load(Ordering::Acquire) { + self.locked(|_, _| {}); + } + } + self.carries_now(widget) + } + + /// [`Self::carries`] without ending a pulse that is over, for use with + /// the register lock held. + fn carries_now(&self, widget: u8) -> bool { link_bit(widget).is_some_and(|b| self.alive.load(Ordering::Acquire) & b != 0) } @@ -318,8 +450,16 @@ impl Xbow { } fn identification(&self) -> u32 { - // Part number and manufacturer are zero on Crossbow. - (self.revision << 28) | 1 + (self.reported_revision.unwrap_or(self.revision) << 28) | (self.part << 12) | (self.manufacturer << 1) | 1 + } + + /// A status word as a read reports it: with the asking widget in the + /// source field, when the chip has one. + fn reported_status(&self, status: u32) -> u32 { + match self.status_source { + Some(widget) => (status & !STATUS_SOURCE_MASK) | ((widget as u32) << STATUS_SOURCE_SHIFT), + None => status, + } } fn link_state(&self, r: &Regs, link: usize) -> u32 { @@ -380,10 +520,21 @@ impl Xbow { } let target = ((r.int_upper >> 16) & 0xf) as u8; let vector = (r.int_upper >> 24) as u8; - if target != WIDGET_HEART || !self.carries(target) || self.heart.get().is_none() { + if !self.carries_now(target) { return; } - out.packets.push((vector, pending)); + if target == WIDGET_HEART { + if self.heart.get().is_none() { + return; + } + out.packets.push((vector, pending)); + } else { + if self.packets.get().is_none() { + return; + } + let address = ((r.int_upper as u64 & 0xffff) << 32) | r.int_lower as u64; + out.xio.push((target, address, vector, pending)); + } r.interrupt_asserted = pending; } @@ -409,13 +560,12 @@ impl Xbow { } fn accessible(&self, slot: u32, write: bool) -> bool { - if self.revision >= REVISION_FENCES && ((FENCE_BASE..FENCE_END).contains(&slot) || slot == LOCK || slot == LOCK_CLEAR) { - return slot % 8 == 0; - } match slot { IDENTIFICATION | STATUS | ERR_UPPER | ERR_LOWER | STATUS_CLEAR => !write, CONTROL | PACKET_TIMEOUT | INT_UPPER | INT_LOWER | ERR_CMD | LLP_CONTROL | ARB_RELOAD | PERF_A | PERF_B | NIC => true, + FENCE | LOCK | LOCK_CLEAR => self.revision >= FENCED_REVISION, + s if (LINK_FENCE_BASE..LINK_FENCE_END).contains(&s) => self.revision >= FENCED_REVISION, s if (LINK_BASE..LINK_END).contains(&s) => match (s - LINK_BASE) % LINK_STRIDE { LINK_FLUSH | LINK_STATUS | LINK_STATUS_CLEAR | LINK_AUX_STATUS => !write, LINK_CONTROL | LINK_ARB_UPPER | LINK_ARB_LOWER => true, @@ -450,7 +600,7 @@ impl Xbow { let slot = self.decode(r, out, off, size, false)?; let v = match slot { IDENTIFICATION => self.identification(), - STATUS => r.status, + STATUS => self.reported_status(r.status), ERR_UPPER => r.err_upper, ERR_LOWER => r.err_lower, CONTROL => r.control, @@ -462,7 +612,7 @@ impl Xbow { STATUS_CLEAR => { // Acknowledges widget 0's own errors and request; each // link's request is acknowledged at its own alias. - let v = r.status; + let v = self.reported_status(r.status); r.status &= !(STATUS_WIDGET0_ERRORS | STATUS_REQUEST); self.update_interrupt(r, out); v @@ -471,8 +621,11 @@ impl Xbow { PERF_A => r.perf_a, PERF_B => r.perf_b, NIC => r.nic, - s if (FENCE_BASE..FENCE_END).contains(&s) => r.fences[((s - FENCE_BASE) / 8) as usize], - LOCK | LOCK_CLEAR => r.lock, + FENCE => r.fence as u32, + LOCK | LOCK_CLEAR => r.lock as u32, + s if (LINK_FENCE_BASE..LINK_FENCE_END).contains(&s) => { + r.link_fence[((s - LINK_FENCE_BASE) / 8) as usize] as u32 + } s => { let link = ((s - LINK_BASE) / LINK_STRIDE) as usize; match (s - LINK_BASE) % LINK_STRIDE { @@ -502,7 +655,13 @@ impl Xbow { /// A processor write; false is a bus error. pub fn write(&self, off: u32, size: usize, val: u64) -> bool { self.locked(|r, out| { - let Some(slot) = self.decode(r, out, off, size, true) else { return false }; + let Some(slot) = self.decode(r, out, off, size, true) else { + // Refused and recorded either way; a posted error address + // write completes all the same. + return self.posted_error_address_writes + && size == 4 + && matches!(off.wrapping_sub(4), ERR_UPPER | ERR_LOWER); + }; let val = val as u32; match slot { CONTROL => { @@ -524,14 +683,17 @@ impl Xbow { PERF_A => r.perf_a = val & PERF_SELECT_MASK, PERF_B => r.perf_b = val & PERF_SELECT_MASK, NIC => r.nic = val & NIC_MASK, - s if (FENCE_BASE..FENCE_END).contains(&s) => r.fences[((s - FENCE_BASE) / 8) as usize] = val & FENCE_MASK, - // Test-and-set: stored only while the lock is free. + FENCE => r.fence = val as u8, + // A test-and-set: the write only takes while the lock is free. LOCK => { if r.lock == 0 { - r.lock = val & 0xff; + r.lock = val as u8; } } LOCK_CLEAR => r.lock = 0, + s if (LINK_FENCE_BASE..LINK_FENCE_END).contains(&s) => { + r.link_fence[((s - LINK_FENCE_BASE) / 8) as usize] = val as u8 + } s => { let link = ((s - LINK_BASE) / LINK_STRIDE) as usize; match (s - LINK_BASE) % LINK_STRIDE { @@ -546,7 +708,7 @@ impl Xbow { self.link_reset(r, link, out); // After the lock: the widget's reset may ask // the switch about its links. - out.link_reset = Some(FIRST_WIDGET + link as u8); + out.link_resets.push(FIRST_WIDGET + link as u8); } _ => unreachable!(), } @@ -563,6 +725,10 @@ impl Xbow { /// initializing; an empty port fails initialization. fn link_reset(&self, r: &mut Regs, link: usize, out: &mut Out) { let bit = 1u8 << link; + if let Some(clock) = &self.clock { + self.begin_pulse(r, link, clock(), out); + return; + } let was_alive = self.alive.load(Ordering::Acquire) & bit != 0; r.eight_bit &= !bit; if self.present.load(Ordering::Acquire) & bit != 0 { @@ -585,6 +751,96 @@ impl Xbow { } } +impl Xbow { + /// A link reset with a clock: the link drops now and the pulse ends + /// two core cycles on. A fitted link that was alive comes back alive + /// then, with an alive event; one that was down comes out of + /// initialization still down; an empty port fails initialization and + /// stays failed. + fn begin_pulse(&self, r: &mut Regs, link: usize, now: u64, out: &mut Out) { + let bit = 1u8 << link; + let present = self.present.load(Ordering::Acquire) & bit != 0; + if present && self.alive.load(Ordering::Acquire) & bit != 0 { + r.reset_restore |= bit; + } else { + r.reset_restore &= !bit; + } + self.alive.fetch_and(!bit, Ordering::AcqRel); + r.eight_bit &= !bit; + if present { + r.initializing |= bit; + } else { + r.initializing &= !bit; + r.came_up &= !bit; + } + r.reset_due[link] = now + LINK_RESET_NS; + self.update_interrupt(r, out); + } + + /// End every pulse the clock says is over. + fn end_pulses(&self, r: &mut Regs, out: &mut Out) { + let Some(clock) = &self.clock else { return }; + if r.reset_due.iter().all(|&d| d == 0) { + return; + } + let now = clock(); + for link in 0..LINKS { + let bit = 1u8 << link; + if r.reset_due[link] == 0 || r.reset_due[link] > now { + continue; + } + r.reset_due[link] = 0; + if self.present.load(Ordering::Acquire) & bit == 0 { + continue; + } + r.initializing &= !bit; + if r.reset_restore & bit != 0 { + self.alive.fetch_or(bit, Ordering::AcqRel); + r.came_up |= bit; + r.link_event[link] |= LINK_ALIVE; + if r.link_control[link] & LINK_EIGHT_BIT_FORCE != 0 { + r.eight_bit |= bit; + } + } + r.reset_restore &= !bit; + self.update_interrupt(r, out); + } + } + + /// A warm reset arriving on the link of `source` widget, at `now` ns. + /// The first in 1024 core cycles is taken and later ones are ignored. + /// On revision 5 and later each destination has a fence naming the + /// sources it admits: the core resets only if its fence admits the + /// source, and each link whose fence does gets a reset pulse out to + /// its far end ([`Self::on_link_reset`]). Fences keep their programmed + /// value; the lock clears with the core. Earlier revisions admit every + /// source. + pub fn warm_reset(&self, source: u8, now: u64) { + let Some(port) = link_bit(source).map(|b| b.trailing_zeros() as usize) else { return }; + self.locked(|r, out| { + if now < r.warm_quiet_until { + crate::dlog!(crate::devlog::LogModule::Bedrock, "xbow: warm reset from widget {source} inside the quiet window; ignored"); + return; + } + let (fence, link_fence) = (r.fence, r.link_fence); + if fence >> port & 1 != 0 { + let present = self.present.load(Ordering::Acquire); + *r = Regs::reset(present); + self.alive.store(present, Ordering::Release); + r.fence = fence; + r.link_fence = link_fence; + self.update_interrupt(r, out); + } + r.warm_quiet_until = now + WARM_RESET_QUIET_CYCLES * CORE_CYCLE_NS; + for (link, fence) in link_fence.iter().enumerate() { + if fence >> port & 1 != 0 { + out.link_resets.push(FIRST_WIDGET + link as u8); + } + } + }); + } +} + impl crate::traits::Resettable for Xbow { fn power_on(&self) { self.locked(|r, _| { @@ -633,6 +889,17 @@ impl BusDevice for Xbow { } } +/// Widget 0's fence and the links' fences, in the order the saved state +/// lists them. +fn fences_of(r: &Regs) -> [u32; 1 + LINKS] { + let mut f = [0u32; 1 + LINKS]; + f[0] = u32::from(r.fence); + for (dst, link) in f[1..].iter_mut().zip(&r.link_fence) { + *dst = u32::from(*link); + } + f +} + // The crossbow's state for a snapshot: its registers and each link's // state. Which links are fitted is the board's, and a load onto other // links is refused. The interrupt the crossbow drives into HEART is @@ -655,8 +922,8 @@ impl Saveable for Xbow { .u8("initializing", r.initializing) .u8("eight_bit", r.eight_bit) .bool("interrupt_asserted", r.interrupt_asserted) - .u32s("fences", &r.fences) - .u32("lock", r.lock) + .u32s("fences", &fences_of(&r)) + .u32("lock", u32::from(r.lock)) .u32("revision", self.revision) .u8("present", self.present.load(Ordering::Acquire)) .u8("alive", self.alive.load(Ordering::Acquire)) @@ -672,7 +939,7 @@ impl Saveable for Xbow { } let [status, err_upper, err_lower, control, packet_timeout, int_upper, int_lower, err_cmd, llp_control, arb_reload, perf_a, perf_b, nic] = field_u32s::<13>(v, "words")?; - let n = Regs { + let mut n = Regs { status, err_upper, err_lower, control, packet_timeout, int_upper, int_lower, err_cmd, llp_control, arb_reload, perf_a, perf_b, nic, link_control: field_u32s(v, "link_control")?, @@ -685,10 +952,23 @@ impl Saveable for Xbow { initializing: field_u8(v, "initializing")?, eight_bit: field_u8(v, "eight_bit")?, interrupt_asserted: field_bool(v, "interrupt_asserted")?, - fences: field_u32s(v, "fences")?, - lock: field_u32(v, "lock")?, + fence: 0, + link_fence: [0; LINKS], + lock: field_u32(v, "lock")? as u8, + // A link reset lasts two core cycles and the warm-reset quiet + // time is counted on the board clock; a snapshot carries + // neither, so a restored crossbow has none in progress. + reset_due: [0; LINKS], + reset_restore: 0, + warm_quiet_until: 0, }; + let fences: [u32; 1 + LINKS] = field_u32s(v, "fences")?; + n.fence = fences[0] as u8; + for (link, f) in n.link_fence.iter_mut().zip(&fences[1..]) { + *link = *f as u8; + } *self.regs.lock() = n; + self.reset_due.store(u64::MAX, Ordering::Release); self.alive.store(field_u8(v, "alive")?, Ordering::Release); Ok(()) } @@ -802,6 +1082,167 @@ mod tests { assert_eq!(x.write32(0x1000_0000 + IDENTIFICATION as u64 + 4, 0), BUS_ERR); } + /// A write to a read-only error address register is an access error and + /// changes nothing. With posted error + /// address writes the store still completes, as QEMU's crossbar does with + /// `complete-posted-error-address-writes` (hw/misc/sgi-xbow.c, asserted + /// set on the Origin 2000 by sn0-origin2000-test's xio-owner case); other + /// read-only registers are refused as before. + #[test] + fn a_posted_error_address_write_completes_and_still_records_the_access_error() { + let refusing = octane(); + assert_eq!(refusing.write32(ERR_LOWER as u64 + 4, 0xdead_beef), BUS_ERR); + let x = octane().with_posted_error_address_writes(); + assert_eq!(x.write32(ERR_LOWER as u64 + 4, 0xdead_beef), BUS_OK); + assert_eq!(x.read(ERR_LOWER + 4, 4), Some(0), "the register keeps its value"); + assert_eq!(x.read(STATUS + 4, 4).unwrap() as u32 & STATUS_ACCESS_ERROR, STATUS_ACCESS_ERROR); + assert_eq!(x.write32(ERR_UPPER as u64 + 4, 0), BUS_OK); + assert_eq!(x.read(STATUS + 4, 4).unwrap() as u32 & STATUS_MULTIPLE_ERROR, STATUS_MULTIPLE_ERROR); + assert_eq!(x.unsupported(), vec![(ERR_LOWER + 4, 4, true), (ERR_UPPER + 4, 4, true)]); + assert_eq!(x.write32(IDENTIFICATION as u64 + 4, 0), BUS_ERR, "only the error address registers"); + assert_eq!(x.write64(ERR_LOWER as u64, 0), BUS_ERR, "nor an eight-byte write"); + } + + /// A Crossbow built without the posted-write option, as the Octane and + /// any other standalone one is, refuses both error address registers and + /// still records the access error; only the PXBOW completes them. + #[test] + fn a_standalone_crossbow_refuses_both_error_address_writes() { + for revision in [4, 5] { + let x = Xbow::new(revision, 0x8400); + assert_eq!(x.write32(ERR_LOWER as u64 + 4, 0xdead_beef), BUS_ERR, "revision {revision}"); + assert_eq!(x.write32(ERR_UPPER as u64 + 4, 0), BUS_ERR, "revision {revision}"); + assert_eq!(x.read(STATUS + 4, 4).unwrap() as u32 & STATUS_ACCESS_ERROR, STATUS_ACCESS_ERROR); + } + } + + /// The fences, the lock and a non-zero part number belong to revision 5 + /// and later; the Octane's revision 4 leaves them reserved. + #[test] + fn fences_the_lock_and_the_identity_fields_need_the_revision_that_has_them() { + let x = octane(); + for slot in [FENCE, LINK_FENCE_BASE, LOCK, LOCK_CLEAR] { + assert_eq!(x.read(slot + 4, 4), None, "slot {slot:#x} is reserved on revision 4"); + assert!(!x.write(slot + 4, 4, 1)); + } + assert_eq!(x.read(IDENTIFICATION + 4, 4), Some(0x4000_0001)); + let y = Xbow::new(5, 0x8400).with_identity(0xd100, 0); + assert_eq!(y.read(IDENTIFICATION + 4, 4), Some(0x5d10_0001)); + assert_eq!(y.read(FENCE + 4, 4), Some(0xff)); + assert!(y.write(FENCE + 4, 4, 0x1ff)); + assert_eq!(y.read(FENCE + 4, 4), Some(0xff), "a fence is eight bits"); + } + + fn timed(revision: u32) -> (Xbow, Arc) { + let now = Arc::new(AtomicU64::new(0)); + let clock = now.clone(); + let x = Xbow::new(revision, 0x8100).with_clock(Arc::new(move || clock.load(Ordering::SeqCst))); + (x, now) + } + + /// Link reset is a pulse of two core cycles: the link is down and not + /// initializing-failed while it lasts, then a fitted live link returns + /// with an alive event. (sn0-xbow-test link-reset-pulse) + #[test] + fn a_link_reset_holds_the_link_down_for_twenty_nanoseconds() { + let (x, now) = timed(4); + let status = |link| LINK_BASE + LINK_STRIDE * link + LINK_STATUS + 4; + now.store(1000, Ordering::SeqCst); + assert!(x.write(LINK_BASE + LINK_STRIDE * 3 + LINK_RESET + 4, 4, 0), "widget 11 has nothing fitted"); + assert!(x.write(LINK_BASE + LINK_STRIDE * 7 + LINK_RESET + 4, 4, 0)); + assert!(!x.carries(15)); + assert_eq!(x.read(status(7), 4), Some(0), "initializing shows none of the three bits"); + assert_eq!(x.read(LINK_BASE + LINK_STRIDE * 7 + LINK_AUX_STATUS + 4, 4), Some(AUX_PRESENT as u64)); + now.store(1019, Ordering::SeqCst); + assert!(!x.carries(15)); + now.store(1020, Ordering::SeqCst); + assert!(x.carries(15), "the pulse is over by the clock without any register access"); + assert_eq!(x.read(status(7), 4).unwrap() as u32 & LINK_ALIVE, LINK_ALIVE); + assert!(!x.carries(11), "an empty port stays down"); + assert_eq!(x.read(status(3), 4), Some(LINK_MAX_RETRY as u64)); + assert!(x.carries(8), "other links were not touched"); + } + + /// A register read ends a finished pulse before it answers. + #[test] + fn a_register_read_after_the_pulse_sees_the_link_back() { + let (x, now) = timed(4); + assert!(x.write(LINK_BASE + LINK_STRIDE * 7 + LINK_RESET + 4, 4, 0)); + now.store(20, Ordering::SeqCst); + let aux = x.read(LINK_BASE + LINK_STRIDE * 7 + LINK_AUX_STATUS + 4, 4).unwrap() as u32; + assert_eq!(aux & (AUX_PRESENT | AUX_FAILURE_MODE), AUX_PRESENT); + } + + fn resets(x: &Xbow) -> Arc>> { + let seen = Arc::new(Mutex::new(Vec::new())); + let s = seen.clone(); + x.on_link_reset(Box::new(move |w| s.lock().push(w))); + seen + } + + /// Fences gate a warm reset per destination, an accepted one leaves + /// them as programmed, and a machine reset restores them. + /// (sn0-xbow-test fence-persists-across-warm-reset, + /// fence-independence-link-outlives-core-rejection) + #[test] + fn a_warm_reset_obeys_each_destinations_fence_and_keeps_the_fences() { + let (x, now) = timed(5); + let seen = resets(&x); + assert!(x.write(CONTROL + 4, 4, 0x26)); + assert!(x.write(FENCE + 4, 4, 0xbf)); + assert!(x.write(LINK_FENCE_BASE + 8 * 7 + 4, 4, 0x3c)); + assert!(x.write(LINK_FENCE_BASE + 4, 4, 0xfd)); + x.warm_reset(15, 0); + assert_eq!(x.read(CONTROL + 4, 4), Some(0), "the core admitted widget 15 and reset"); + assert_eq!(x.read(FENCE + 4, 4), Some(0xbf)); + assert_eq!(x.read(LINK_FENCE_BASE + 8 * 7 + 4, 4), Some(0x3c)); + // Link 7's fence refuses port 15 (bit 7 clear); the rest admit it. + assert_eq!(*seen.lock(), (8..15).collect::>()); + // A core that refuses the source is left alone while links still admit. + assert!(x.write(CONTROL + 4, 4, 0x26)); + assert!(x.write(FENCE + 4, 4, 0xbf & !(1 << 7))); + now.store(20_000, Ordering::SeqCst); + x.warm_reset(15, 20_000); + assert_eq!(x.read(CONTROL + 4, 4), Some(0x26)); + crate::traits::Resettable::power_on(&x); + assert_eq!(x.read(FENCE + 4, 4), Some(0xff)); + assert_eq!(x.read(LINK_FENCE_BASE + 8 * 7 + 4, 4), Some(0xff)); + } + + /// A second warm reset inside 1024 core cycles is the first one's echo. + /// The lock clears with the core only. (sn0-xbow-test + /// lock-clears-with-core-reset-only) + #[test] + fn a_warm_reset_inside_the_quiet_window_is_ignored_and_the_lock_follows_the_core() { + let (x, _) = timed(5); + let seen = resets(&x); + assert!(x.write(FENCE + 4, 4, 0xff & !(1 << 7))); + assert!(x.write(LOCK + 4, 4, 0x66)); + x.warm_reset(15, 100); + assert_eq!(x.read(LOCK + 4, 4), Some(0x66), "fenced out: the core is untouched"); + assert_eq!(seen.lock().len(), 8, "every link admitted the source"); + assert!(x.write(FENCE + 4, 4, 0xff)); + x.warm_reset(15, 100 + 10_239); + assert_eq!(x.read(LOCK + 4, 4), Some(0x66), "still inside the window"); + assert_eq!(seen.lock().len(), 8); + x.warm_reset(15, 100 + 10_240); + assert_eq!(x.read(LOCK + 4, 4), Some(0)); + assert_eq!(seen.lock().len(), 16); + } + + /// Revision 4 has no fences, so a warm reset always reaches the core and + /// every link. (sn0-xbow-test warm-reset-revision-4-unaffected-by-fence) + #[test] + fn revision_four_admits_every_warm_reset() { + let (x, _) = timed(4); + let seen = resets(&x); + assert!(x.write(CONTROL + 4, 4, 0x26)); + assert!(!x.write(FENCE + 4, 4, 0), "reserved on this revision"); + x.warm_reset(15, 0); + assert_eq!(x.read(CONTROL + 4, 4), Some(0)); + assert_eq!(seen.lock().len(), 8); + } + #[test] fn link_changes_reach_the_callback_outside_the_lock() { let x = Arc::new(octane()); @@ -843,8 +1284,8 @@ mod tests { #[test] fn revision_five_has_reset_fences_and_a_lock() { let x = Xbow::new(5, 0x8100); - for fence in 0..FENCES as u32 { - let off = FENCE_BASE + 8 * fence + 4; + for fence in 0..1 + LINKS as u32 { + let off = FENCE + 8 * fence + 4; assert_eq!(x.read(off, 4), Some(0xff), "fence {fence} resets to all ones"); assert!(x.write(off, 4, 0x1f7)); assert_eq!(x.read(off, 4), Some(0xf7)); @@ -858,7 +1299,7 @@ mod tests { assert_eq!(x.read(STATUS + 4, 4).unwrap() as u32 & STATUS_ACCESS_ERROR, 0); let old = octane(); - assert_eq!(old.read(FENCE_BASE + 4, 4), None); + assert_eq!(old.read(FENCE + 4, 4), None); assert_ne!(old.read(STATUS + 4, 4).unwrap() as u32 & STATUS_ACCESS_ERROR, 0); } diff --git a/src/dev/xbridge.rs b/src/dev/xbridge.rs new file mode 100644 index 000000000..424542d23 --- /dev/null +++ b/src/dev/xbridge.rs @@ -0,0 +1,1600 @@ +//! Fuel's XBridge revision 2 contains two independent PCI heads and their +//! shared crossbar. Each head has its own 1024-entry ATE, device windows and +//! interrupt state. Revision 2 places aliases differently from the Octane's +//! single-head Bridge, uses different PCI slot data-phase widths, and sends +//! host DMA through widget 0 with source-aware packets. +//! +//! This remains separate from `bridge.rs` because its two-head topology, +//! alias map and host packet path are externally visible parts of this chip. +//! Generic PCI BAR-run and lane helpers are shared; the ATE and interrupt +//! dispatch remain tied to the XBridge head layout. Sorted BAR runs decode PCI +//! PIO without a `Physical` table; the machine's RAM still uses one. +//! +//! # What is approximated here +//! +//! Slot widths, immediate link initialization, synchronous flushes and +//! revision-2 alias placement follow the project's BSD QEMU model, not a +//! silicon measurement. Timing, external SRAM, flash and diagnostic engines +//! are absent. Undefined accesses latch an error and remain observable. +//! +//! Ported from hw/pci-host/sgi-xbridge.c and hw/misc/sgi-xbridge-xbow.c +//! in jamesbraid/qemu-sgi-origami, revision +//! fe128635b6dfe2511f1bd451684cb30007686df1. + +use crate::dev::w1; +use crate::irq::{InterruptController, IrqLine}; +use crate::pci::{bar_runs, BarRun, PciBar, PciBusDevice, PciDma, PciDmaStatus, PciLanes, Space}; +use crate::physical::{read_bytes, write_bytes}; +use crate::traits::{BusDevice, BusRead16, BusRead32, BusRead64, BusRead8, Resettable, BUS_OK}; +use parking_lot::{Mutex, RwLock}; +use std::collections::HashMap; +use std::sync::{atomic::{AtomicU64, Ordering}, Arc, Weak}; + +pub const HOST_WIDGET: u8 = 10; +pub const HEAD_E: u8 = 14; +pub const HEAD_F: u8 = 15; +pub const ATE_ENTRIES: usize = 1024; +pub const ATE_MAPPED: usize = 512; + +/// Offsets of the PCI head register words, at eight-byte strides. +pub mod reg { + pub const IDENTIFICATION: usize = 0x0; + pub const STATUS: usize = 0x8; + pub const ERR_UPPER: usize = 0x10; + pub const ERR_LOWER: usize = 0x18; + pub const CONTROL: usize = 0x20; + pub const REQUEST_TIMEOUT: usize = 0x28; + pub const INT_DEST_UPPER: usize = 0x30; + pub const INT_DEST_LOWER: usize = 0x38; + pub const LINK_CONFIG: usize = 0x48; + pub const TEST_PIN_CONTROL: usize = 0x70; + pub const DIRECT_MAP: usize = 0x80; + pub const SSRAM_PARITY_ERR: usize = 0x90; + pub const ARBITRATION: usize = 0xa0; + pub const NIC: usize = 0xb0; + pub const BUS_TIMEOUT: usize = 0xc0; + pub const PCI_TYPE1_CONFIG: usize = 0xc8; + pub const PCI_ERR_LOWER: usize = 0xd8; + pub const INTERRUPT_STATUS: usize = 0x100; + pub const INTERRUPT_ENABLE: usize = 0x108; + pub const INTERRUPT_RESET: usize = 0x110; + pub const INTERRUPT_MODE: usize = 0x118; + pub const INTERRUPT_DEVICE: usize = 0x120; + pub const INTERRUPT_ERROR_HOST: usize = 0x128; + pub const INTERRUPT_HOST_BASE: usize = 0x130; + pub const INTERRUPT_HOST_LAST: usize = 0x168; + pub const INTERRUPT_MULTIPLE: usize = 0x170; + pub const FORCE_INTERRUPT_BASE: usize = 0x180; + pub const FORCE_PIN_BASE: usize = 0x1c0; + pub const FORCE_PIN_LAST: usize = 0x1f8; + pub const DEVICE_BASE: usize = 0x200; + pub const DEVICE_LAST: usize = 0x238; + pub const RRB_EVEN: usize = 0x280; + pub const RRB_ODD: usize = 0x288; + pub const RRB_CLEAR: usize = 0x298; + pub const ATE_UPPER_BASE: u64 = 0x10000; + pub const ATE_LOWER_BASE: u64 = 0x12000; + pub const ATE_END: u64 = 0x14000; + +} + +/// The crossbar has the Crossbow's widget 0 register map, which `xbow.rs` +/// owns; the register offsets here are its constants. The access rules and +/// reset values below are this chip's own (the Octane's Crossbow model differs +/// in status, link state, wide accesses and where its interrupts go). +use crate::dev::xbow; + +/// Fences run from widget 0's to link 15's. +const FENCE_LAST: u32 = xbow::FENCE_END - 8; +/// Read-only registers between the resource lock and the first link. +const READ_ONLY_BASE: u32 = xbow::LOCK_CLEAR + 8; +const READ_ONLY_LAST: u32 = xbow::LINK_BASE - 8; +fn ix(off: u32) -> usize { + (off / 8) as usize +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct InterruptPacket { + pub source_widget: u8, + pub widget: u8, + pub address: u64, + pub vector: u8, + pub level: bool, +} +pub type InterruptSender = Arc bool + Send + Sync>; + +/// Source identity and placement hints retained for a host's inbound packet path. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub struct DmaPacket { + pub source_widget: u8, + pub slot: u8, + pub widget: u8, + pub address: u64, + pub write: bool, + pub half_write_requested: bool, + pub half_read_requested: bool, +} +/// Bytes are in XIO address order after bridge placement. A false return +/// refuses that access. The host applies the requested exchange of doubleword halves. +pub type DmaSender = Arc bool + Send + Sync>; + +struct Routes { + targets: RwLock<[Option>; 16]>, + interrupts: RwLock>, + dma: RwLock>, +} +impl Routes { + fn interrupt(&self, p: InterruptPacket) -> bool { + let sender = self.interrupts.read().clone(); + let accepted = if let Some(send) = sender { + send(p) + } else { + let target = self.targets.read()[p.widget as usize].clone(); + target.is_some_and(|t| { + t.write64( + p.address, + p.vector as u64 | if p.level { 1 << 8 } else { 0 }, + ) == BUS_OK + }) + }; + if !accepted { + crate::dlog!(crate::devlog::LogModule::Bedrock, "xbridge: refused interrupt {p:?}"); + } + accepted + } +} + +/// One chip owns both heads and widget 0; views share routing and reset state. +pub struct XBridge { + routes: Arc, + heads: [Arc; 2], + xbow: Arc, +} +impl XBridge { + pub fn new(memory: Arc) -> Arc { + let routes = Arc::new(Routes { + targets: RwLock::new(std::array::from_fn(|i| { + if i == HOST_WIDGET as usize { + Some(memory.clone()) + } else { + None + } + })), + interrupts: RwLock::new(None), + dma: RwLock::new(None), + }); + let heads = [ + XBridgeHead::new(HEAD_E, 1, routes.clone()), + XBridgeHead::new(HEAD_F, 0, routes.clone()), + ]; + let xbow = Arc::new(Crossbar { + st: Mutex::new(XbowState::new()), + order: Mutex::new(()), + routes: routes.clone(), + heads: heads.each_ref().map(Arc::downgrade), + reset_output: RwLock::new(None), + }); + Arc::new(Self { + routes, + heads, + xbow, + }) + } + pub fn register_locks(self: &Arc) { + use crate::locks::register_lock_fn; + for head in &self.heads { + for (what, probe) in [ + ("state", (|h: &XBridgeHead| h.st.is_locked()) as fn(&XBridgeHead) -> bool), + ("irq_order", |h| h.order.is_locked()), + ("microlan", |h| h.microlan.is_locked()), + ("pio_map", |h| h.pio_map.is_locked()), + ] { + let h = Arc::downgrade(head); + let name = format!("xbridge::head{:x}::{what}", head.widget); + register_lock_fn(name, move || h.upgrade().is_some_and(|h| probe(&h))); + } + } + for (what, probe) in [ + ("state", (|x: &Crossbar| x.st.is_locked()) as fn(&Crossbar) -> bool), + ("irq_order", |x| x.order.is_locked()), + ] { + let x = Arc::downgrade(&self.xbow); + register_lock_fn(format!("xbridge::crossbar::{what}"), move || { + x.upgrade().is_some_and(|x| probe(&x)) + }); + } + } + fn pci_head(&self, widget: u8) -> &Arc { + assert!((HEAD_E..=HEAD_F).contains(&widget)); + &self.heads[(widget - HEAD_E) as usize] + } + pub fn head(&self, widget: u8) -> Arc { + self.pci_head(widget).clone() + } + pub fn crossbar(&self) -> Arc { + self.xbow.clone() + } + /// A reader-specific widget-0 view; the host reads port 10 by default. + pub fn crossbar_for_port(&self, port: u8) -> Arc { + assert!((8..=15).contains(&port)); + Arc::new(CrossbarView { + xbow: self.xbow.clone(), + port, + }) + } + /// Host-port processor projection uses the XBridge widget-0 alias. + pub fn widget(&self, widget: u8) -> Option> { + match widget { + 0 | HOST_WIDGET => Some(self.crossbar()), + HEAD_E | HEAD_F => Some(self.head(widget)), + _ => self + .routes + .targets + .read() + .get(widget as usize) + .and_then(Clone::clone), + } + } + pub fn attach(&self, widget: u8, slot: u8, device: Arc) { + self.pci_head(widget).attach(slot, device); + } + pub fn dma(&self, widget: u8, slot: u8) -> Arc { + assert!(slot < 8); + Arc::new(SlotDma { + head: Arc::downgrade(self.pci_head(widget)), + slot, + }) + } + pub fn pin_line(&self, widget: u8, pin: u8) -> IrqLine { + assert!(pin < 8); + IrqLine::new( + Arc::new(Pin { + head: Arc::downgrade(self.pci_head(widget)), + pci: false, + }), + pin as u32, + ) + } + /// Raw PCI INTx is board wiring, independent of the eight head pins. + pub fn set_pci_irq(&self, widget: u8, slot: u8, pin: u8, output: IrqLine) { + assert!(slot < 8 && pin < 4); + self.pci_head(widget).pci_outputs.write()[(slot * 4 + pin) as usize] = Some(output); + } + pub fn set_interrupt_sender(&self, send: InterruptSender) { + *self.routes.interrupts.write() = Some(send); + } + pub fn set_dma_sender(&self, send: DmaSender) { + *self.routes.dma.write() = Some(send); + } + pub fn set_dma_64bit(&self, widget: u8, slot: u8, wide: bool) { + assert!(slot < 8); + let head = self.pci_head(widget); + { + let mut st = head.st.lock(); + st.dma_64bit = (st.dma_64bit & !(1 << slot)) | ((wide as u8) << slot); + } + head.publish_bars(); + } + pub fn set_wide_slot_mask(&self, widget: u8, mask: u8) { + self.pci_head(widget).st.lock().wide_slots = mask; + } + pub fn set_microlan(&self, widget: u8, bus: w1::Bus) { + *self.pci_head(widget).microlan.lock() = Some(bus); + } + /// An interrupt packet from the widget attached at `p.source_widget`, + /// routed as the heads' packets are. False when nothing takes it. + pub fn send_interrupt(&self, p: InterruptPacket) -> bool { + assert!((8..14).contains(&p.source_widget) && p.source_widget != HOST_WIDGET); + self.routes.interrupt(p) + } + pub fn attach_widget(&self, widget: u8, target: Arc) { + assert!((8..14).contains(&widget) && widget != HOST_WIDGET); + self.routes.targets.write()[widget as usize] = Some(target); + self.set_link_present(widget, true); + } + pub fn set_link_present(&self, widget: u8, present: bool) { + assert!((8..=15).contains(&widget)); + let mut st = self.xbow.st.lock(); + st.present = (st.present & !(1 << (widget - 8))) | ((present as u8) << (widget - 8)); + } + pub fn set_link_8bit(&self, widget: u8, enabled: bool) { + assert!((8..=15).contains(&widget)); + let mut st = self.xbow.st.lock(); + let bit = 1 << (widget - 8); + st.eight_bit = (st.eight_bit & !bit) | if enabled { bit } else { 0 }; + } + /// A pulse for each reset port, outside chip locks. A head's registers, + /// routed PCI lines and function headers are already reset. The board + /// must reset that port's attached engines through this callback. + pub fn set_reset_output(&self, output: Arc) { + *self.xbow.reset_output.write() = Some(output); + } + /// Time is supplied by the machine's virtual clock, in nanoseconds. + pub fn warm_reset(&self, source_port: u8, now_ns: u64) { + self.xbow.warm_reset(source_port, now_ns); + } + pub fn reset(&self) { + for h in &self.heads { + h.reset(); + } + let mut st = self.xbow.st.lock(); + st.reset(); + st.quiet_until = 0; + } +} +impl Resettable for XBridge { + fn power_on(&self) { + self.reset(); + } +} + +struct HeadState { + reg: [u32; 0x300 / 8], + ate: [u64; ATE_ENTRIES], + pins: u8, + dma_64bit: u8, + wide_slots: u8, +} +impl HeadState { + fn r(&self, off: usize) -> u32 { + self.reg[off / 8] + } + fn set(&mut self, off: usize, value: u32) { + self.reg[off / 8] = value; + } + fn reset(&mut self, speed: u8) { + self.reg.fill(0); + self.ate.fill(0); + self.pins = 0; + for (off, value) in [ + (0, 0x2d002049), + (8, 0x20), + (0x20, 0x0f00320f | ((speed as u32) << 4)), + (0x28, 0xfffff), + (reg::LINK_CONFIG, 0x03ff1810), + (0x80, 0x20000), + (0xa0, 0x20008), + (0xc0, 0x10000), + ] { + self.set(off, value); + } + for slot in 0..8 { + self.set( + 0x200 + slot * 8, + 0x18001000 + | if slot < 2 { + slot as u32 * 2 + } else { + slot as u32 + 2 + }, + ); + } + } + fn packet(&self, pin: Option, level: bool) -> InterruptPacket { + let upper = self.r(reg::INT_DEST_UPPER); + let mut address = ((upper as u64 & 0xffff) << 32) | self.r(reg::INT_DEST_LOWER) as u64; + let vector = if let Some(pin) = pin { + let host = self.r(reg::INTERRUPT_HOST_BASE + pin * 8); + address = (address & !(0x3ff << 38)) | (((host >> 8) as u64 & 0x3ff) << 38); + host as u8 + } else { + self.r(reg::INTERRUPT_ERROR_HOST) as u8 + }; + InterruptPacket { + source_widget: (self.r(reg::CONTROL) & 15) as u8, + widget: ((upper >> 16) & 15) as u8, + address, + vector, + level, + } + } +} +// Lock order: order, then microlan (NIC only), then st. +struct XBridgeHead { + widget: u8, + speed: u8, + st: Mutex, + order: Mutex<()>, + routes: Arc, + slots: RwLock<[Option>; 8]>, + pci_outputs: RwLock<[Option; 32]>, + microlan: Mutex>, + pio: Arc, + pio_map: Mutex, +} + +#[derive(Default)] +struct PioMap { + entries: HashMap<(u8, usize, usize), MappedBar>, + runs: Vec<(u64, u64)>, +} + +struct MappedBar { + function: Arc, + lanes: Arc, + device: Arc, +} + +/// Whole BAR-page runs are few and carry no RAM or native fault-handler state. +/// Readers clone one endpoint and release the map lock before calling a device. +#[derive(Default)] +struct PioDecode { + runs: RwLock>, +} +impl PioDecode { + fn publish(&self, mut runs: Vec) -> Vec { + runs.sort_unstable_by_key(|run| run.start); + debug_assert!(runs.windows(2).all(|pair| pair[0].end <= pair[1].start)); + std::mem::replace(&mut *self.runs.write(), runs) + } + fn target(&self, address: u64, size: usize) -> Option> { + let runs = self.runs.read(); + let i = runs.partition_point(|run| run.start <= address).checked_sub(1)?; + let run = &runs[i]; + (address.checked_add(size as u64)? <= run.end).then(|| run.dev.clone()) + } +} + +struct Lanes { + // BAR-relative CPU base, data-phase width and stream/swap policy. + policy: AtomicU64, + alias: bool, + size: u64, +} + +const PIO_INVALID: u64 = 1 << 63; + +impl Lanes { + fn access(&self, f: &dyn PciBusDevice, bar: usize, off: u32, size: usize, value: Option) -> u64 { + let policy = self.policy.load(Ordering::Acquire); + if policy & PIO_INVALID != 0 { + return if value.is_none() { u64::MAX >> (64 - size * 8) } else { 0 }; + } + self.cycle(f, bar, off.wrapping_sub(policy as u32), size, value, policy) + } + + fn cycle(&self, f: &dyn PciBusDevice, bar: usize, off: u32, size: usize, value: Option, policy: u64) -> u64 { + let width = if policy & (1 << 32) != 0 { 8 } else { 4 }; + if size > width { + let hi = self.cycle(f, bar, off, 4, value.map(|v| v >> 32), policy); + let lo = self.cycle(f, bar, off + 4, 4, value.map(|v| v as u32 as u64), policy); + return (hi << 32) | lo; + } + let selected = policy & (1 << 33) != 0; + let reverse = if self.alias { !selected } else { selected }; + let placed = if selected { off } else { off ^ (width - size) as u32 }; + // A reader may have obtained the old table entry before relocation. + // Its new policy must not send the stale address outside the BAR. + if placed as u64 + size as u64 > self.size { + return if value.is_none() { u64::MAX >> (64 - size * 8) } else { 0 }; + } + let mut bytes = [0; 8]; + if let Some(v) = value { + for (i, b) in bytes[..size].iter_mut().enumerate() { + let shift = if self.alias { (size - 1 - i) * 8 } else { i * 8 }; + *b = (v >> shift) as u8; + } + if reverse { bytes[..size].reverse(); } + } + XBridgeHead::bar_cycle(f, bar, placed, &mut bytes[..size], value.is_some()); + if reverse && value.is_none() { bytes[..size].reverse(); } + if self.alias { + bytes[..size].iter().fold(0, |v, &b| (v << 8) | b as u64) + } else { + bytes[..size].iter().enumerate().fold(0, |v, (i, &b)| v | ((b as u64) << (i * 8))) + } + } +} + +impl PciLanes for Lanes { + fn read(&self, f: &dyn PciBusDevice, bar: usize, off: u32, size: usize) -> u64 { + self.access(f, bar, off, size, None) + } + fn write(&self, f: &dyn PciBusDevice, bar: usize, off: u32, size: usize, value: u64) { + self.access(f, bar, off, size, Some(value)); + } +} +struct Pin { + head: Weak, + pci: bool, +} +impl InterruptController for Pin { + fn set_line(&self, line: u32, level: bool) { + if let Some(head) = self.head.upgrade() { + if self.pci { + let output = head + .pci_outputs + .read() + .get(line as usize) + .and_then(Clone::clone); + if let Some(output) = output { + output.set(level); + } else if level { + crate::dlog!(crate::devlog::LogModule::Bedrock, + "xbridge: head {} PCI line {line} has no board wiring", + head.widget + ); + } + } else { + head.pin(line as usize, level); + } + } + } +} +struct SlotDma { + head: Weak, + slot: u8, +} +impl PciDma for SlotDma { + fn dma_read(&self, address: u64, bytes: &mut [u8]) -> PciDmaStatus { + if let Some(h) = self.head.upgrade() { + h.dma(self.slot, address, bytes, false) + } else { + bytes.fill(0); + PciDmaStatus::MasterAbort + } + } + fn dma_write(&self, address: u64, bytes: &[u8]) -> PciDmaStatus { + self.head.upgrade().map_or(PciDmaStatus::MasterAbort, |h| { + h.dma(self.slot, address, &mut bytes.to_vec(), true) + }) + } +} +impl XBridgeHead { + fn new(widget: u8, speed: u8, routes: Arc) -> Arc { + let mut st = HeadState { + reg: [0; 96], + ate: [0; ATE_ENTRIES], + pins: 0, + dma_64bit: 0, + wide_slots: 0, + }; + st.reset(speed); + Arc::new(Self { + widget, + speed, + st: Mutex::new(st), + order: Mutex::new(()), + routes, + slots: RwLock::new(std::array::from_fn(|_| None)), + pci_outputs: RwLock::new(std::array::from_fn(|_| None)), + microlan: Mutex::new(None), + pio: Arc::new(PioDecode::default()), + pio_map: Mutex::new(PioMap::default()), + }) + } + fn attach(self: &Arc, slot: u8, device: Arc) { + assert!(slot < 8); + self.slots.write()[slot as usize] = Some(device.clone()); + device.connect_dma(Arc::new(SlotDma { + head: Arc::downgrade(self), + slot, + })); + let pin = device.config().interrupt_pin(); + if (1..=4).contains(&pin) { + device.connect_interrupt(IrqLine::new( + Arc::new(Pin { + head: Arc::downgrade(self), + pci: true, + }), + slot as u32 * 4 + pin as u32 - 1, + )); + } + self.publish_bars(); + } + fn pin(&self, pin: usize, level: bool) { + if pin >= 8 { + return; + } + let _order = self.order.lock(); + let packet = { + let mut st = self.st.lock(); + let bit = 1 << pin; + let old = st.pins & bit != 0; + st.pins = (st.pins & !bit) | if level { bit } else { 0 }; + let pins = st.pins as u32; + let status = (st.r(reg::INTERRUPT_STATUS) & !255) | pins; + st.set(reg::INTERRUPT_STATUS, status); + (old != level + && st.r(reg::INTERRUPT_ENABLE) & bit as u32 != 0 + && (level || st.r(reg::INTERRUPT_MODE) & bit as u32 != 0)) + .then(|| st.packet(Some(pin), level)) + }; + if let Some(packet) = packet { + self.routes.interrupt(packet); + } + } + fn reset(&self) { + { + let _order = self.order.lock(); + self.st.lock().reset(self.speed); + } + let outputs = self.pci_outputs.read().clone(); + for line in outputs.into_iter().flatten() { + line.set(false); + } + let devices = self.slots.read().clone(); + for d in devices.into_iter().flatten() { + d.config().reset(); + } + self.publish_bars(); + } + fn invalid(&self, address: u64) { + let _order = self.order.lock(); + let packet = { + let mut st = self.st.lock(); + let first = st.r(reg::INTERRUPT_STATUS) & (1 << 24) == 0; + let status = st.r(reg::INTERRUPT_STATUS) | (1 << 24); + st.set(reg::INTERRUPT_STATUS, status); + let multi = st.r(reg::INTERRUPT_MULTIPLE) | (1 << 24); + st.set(reg::INTERRUPT_MULTIPLE, multi); + if first { + st.set(reg::ERR_UPPER, (address >> 32) as u32); + st.set(reg::ERR_LOWER, address as u32); + } + (first && st.r(reg::INTERRUPT_ENABLE) & (1 << 24) != 0).then(|| st.packet(None, true)) + }; + if let Some(p) = packet { + self.routes.interrupt(p); + } + crate::dlog!(crate::devlog::LogModule::Bedrock, + "xbridge: head {} invalid access at {address:#x}", + self.widget + ); + } + fn defined(off: usize) -> bool { + matches!(off, reg::IDENTIFICATION..=reg::TEST_PIN_CONTROL | reg::DIRECT_MAP | reg::SSRAM_PARITY_ERR | reg::ARBITRATION | reg::NIC | reg::BUS_TIMEOUT..=reg::PCI_ERR_LOWER | reg::INTERRUPT_STATUS..=reg::FORCE_PIN_LAST | reg::DEVICE_BASE..=reg::RRB_CLEAR) + } + fn deferred(address: u64) -> bool { + (0x300..0x600).contains(&address) + || (0x30000..0x30008).contains(&address) + || (0xc00000..0x1000000).contains(&address) + } + fn config(&self, address: u64, size: usize, value: Option) -> u64 { + if size == 8 { + let hi = self.config(address, 4, value.map(|v| v >> 32)); + return (hi << 32) | self.config(address + 4, 4, value.map(|v| v & 0xffffffff)); + } + let reg = ((address & 255) as usize) ^ (4 - size); + let slot = if address < 0x28000 { + ((address - 0x20000) >> 12) as usize + } else { + ((self.st.lock().r(reg::PCI_TYPE1_CONFIG) >> 11) & 31) as usize + }; + let bus = if address < 0x28000 { + 0 + } else { + (self.st.lock().r(reg::PCI_TYPE1_CONFIG) >> 16) & 255 + }; + let device = if bus == 0 && address & 0x700 == 0 { + self.slots.read().get(slot).and_then(Clone::clone) + } else { + None + }; + let mask = u32::MAX >> (32 - size * 8); + let shift = (reg & 3) * 8; + if let Some(dev) = device { + if let Some(value) = value { + dev.config().write_masked( + reg / 4, + (value as u32 & mask) << shift, + mask << shift, + ); + self.publish_bars(); + 0 + } else { + dev.config().read_sized(reg, size) as u64 + } + } else { + if value.is_none() { + let mut st = self.st.lock(); + let multi = st.r(reg::INTERRUPT_MULTIPLE) | (1 << 11); + st.set(reg::INTERRUPT_MULTIPLE, multi); + } + mask as u64 + } + } + fn reg_write(&self, address: u64, value: u32) { + let off = address as usize & !7; + let _order = self.order.lock(); + let mut packets = Vec::new(); + if off == 0xb0 { + let pulse = (value >> 10) & 0x3ff; + let sample = (value >> 2) & 255; + let mut bus = self.microlan.lock(); + if let Some(bus) = bus.as_mut() { + let level = w1::slot::slot(bus, pulse, sample); + self.st.lock().set(reg::NIC, 2 | level as u32); + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "xbridge: head {} NIC has no board", self.widget); + } + return; + } + { + let mut st = self.st.lock(); + let masked = match off { + reg::CONTROL => (value & 0x0fe1f20f) | ((self.speed as u32) << 4), + reg::REQUEST_TIMEOUT | reg::INT_DEST_UPPER => value & 0xfffff, + reg::INT_DEST_LOWER | reg::RRB_EVEN | reg::RRB_ODD => value, + reg::LINK_CONFIG => value & 0x03ffffff, + reg::TEST_PIN_CONTROL => value & 0xffffff, + reg::DIRECT_MAP => value & 0xf3ffff, + reg::ARBITRATION => value & 0x3ff7f, + reg::BUS_TIMEOUT => value & 0x1f03ff, + reg::PCI_TYPE1_CONFIG => value & 0xfff800, + reg::INTERRUPT_DEVICE => value & 0xffffff, + reg::INTERRUPT_ERROR_HOST | reg::INTERRUPT_MODE => value & 255, + reg::INTERRUPT_ENABLE => { + let value = value & 0x7fffffff; + let newly = value & !st.r(reg::INTERRUPT_ENABLE) & st.pins as u32; + st.set(off, value); + for pin in 0..8 { + if newly & (1 << pin) != 0 { + packets.push(st.packet(Some(pin), true)); + } + } + value + } + reg::INTERRUPT_RESET => { + let clear = (if value & 2 != 0 { 1 << 30 } else { 0 }) + | (if value & 8 != 0 { 1 << 24 } else { 0 }); + let status = st.r(reg::INTERRUPT_STATUS) & !clear; + st.set(reg::INTERRUPT_STATUS, status); + let multi = if value & 64 != 0 { + 0 + } else { + st.r(reg::INTERRUPT_MULTIPLE) & !clear + }; + st.set(reg::INTERRUPT_MULTIPLE, multi); + return; + } + reg::FORCE_INTERRUPT_BASE..=reg::FORCE_PIN_LAST => { + let pin = (off - reg::FORCE_INTERRUPT_BASE) / 8 & 7; + if st.r(reg::INTERRUPT_ENABLE) & (1 << pin) == 0 { + crate::dlog!(crate::devlog::LogModule::Bedrock, "xbridge: masked force semantics unresolved"); + } else if off < reg::FORCE_PIN_BASE || st.pins & (1 << pin) != 0 { + packets.push(st.packet(Some(pin), true)); + } + 0 + } + reg::INTERRUPT_HOST_BASE..=reg::INTERRUPT_HOST_LAST => value & 0x3ffff, + reg::DEVICE_BASE..=reg::DEVICE_LAST => value & 0x1eefffff, + _ => { + crate::dlog!(crate::devlog::LogModule::Bedrock, + "xbridge: head {} ignored register write {off:#x}", + self.widget + ); + return; + } + }; + if !(reg::FORCE_INTERRUPT_BASE..=reg::FORCE_PIN_LAST).contains(&off) { + st.set(off, masked); + } + } + for p in packets { + self.routes.interrupt(p); + } + drop(_order); + if off == reg::CONTROL || (reg::DEVICE_BASE..=reg::DEVICE_LAST).contains(&off) { + self.publish_bars(); + } + } + + fn publish_bars(&self) { + // Keep endpoint identities stable across BAR relocation. Publish the + // complete small decoder after dropping state and attachment locks. + let mut map = self.pio_map.lock(); + let (regs, control, wide) = { + let st = self.st.lock(); + let regs: [u32; 8] = std::array::from_fn(|i| st.r(reg::DEVICE_BASE + i * 8)); + (regs, st.r(reg::CONTROL), st.dma_64bit) + }; + let funcs: Vec<_> = self.slots.read().iter().enumerate() + .filter_map(|(slot, f)| Some((slot, f.clone()?))).collect(); + let windows: Vec<_> = funcs.iter().flat_map(|(slot, f)| { + f.config().windows().into_iter().map(move |(space, bar, base, size)| (*slot, bar, space, base as u64, base as u64 + size as u64)) + }).collect(); + map.runs.clear(); + let mut decoded = Vec::new(); + let mut retired = Vec::new(); + // Host replacement changes an endpoint's function. Guest configuration + // writes only update policy and never allocate another table endpoint. + map.entries.retain(|&(_, slot, _), entry| { + let keep = funcs.iter().any(|(s, f)| *s == slot && Arc::ptr_eq(f, &entry.function)); + if !keep { + entry.lanes.policy.fetch_or(PIO_INVALID, Ordering::AcqRel); + retired.push(entry.device.clone()); + } + keep + }); + for kind in 0u8..11 { + let (cpu, len, space, pci, width, selected) = if kind < 8 { + let slot = kind as usize; + let dev = regs[slot]; + let memory = dev & (1 << 12) != 0; + let mut base = dev & if memory { 0x3ff } else { 0xfff }; + if slot < 2 { base &= !1; } + let (cpu, len) = if slot < 2 { + (0x200000 + slot as u64 * 0x200000, 0x200000) + } else { + (0x600000 + (slot as u64 - 2) * 0x100000, 0x100000) + }; + (cpu, len, if memory { Space::Mem } else { Space::Io }, base << 20, + if dev & (1 << 22) != 0 { 8 } else { 4 }, dev & (1 << 13) != 0) + } else { + let (cpu, len, space, width) = match kind { + 8 => (0x40000000, 0x40000000, Space::Mem, 4), + 9 => (0x80000000, 0x40000000, Space::Mem, 8), + _ => (0x100000000, 0x100000000, Space::Io, 4), + }; + (cpu, len, space, 0, width, control & (1 << if space == Space::Mem { 22 } else { 23 }) != 0) + }; + let runs = bar_runs(&funcs, space, cpu, pci, len, |b| { + let width = if kind == 9 && wide & (1 << b.slot) == 0 { 4 } else { width }; + let policy = b.cpu_base as u32 as u64 | ((width == 8) as u64) << 32 | (selected as u64) << 33; + let entry = map.entries.entry((kind, b.slot, b.bar)).or_insert_with(|| { + let size = windows.iter().find(|&&(slot, bar, _, _, _)| slot == b.slot && bar == b.bar) + .map(|&(_, _, _, lo, hi)| hi - lo).unwrap(); + let lanes = Arc::new(Lanes { policy: AtomicU64::new(policy), alias: kind >= 8, size }); + let device = Arc::new(PciBar::new(lanes.clone(), b.dev.clone(), b.bar, u32::MAX as u64, 0)); + MappedBar { function: b.dev.clone(), lanes, device } + }); + entry.lanes.policy.store(policy, Ordering::Release); + entry.device.clone() + }); + for run in runs { + let lo = run.start - cpu + pci as u64; + let hi = run.end - cpu + pci as u64; + // bar_runs rounds to whole pages. A subpage BAR can still + // overlap one of those pages, so retain the slow decoder there. + let claims = windows.iter().filter(|&&(_, _, s, l, h)| s == space && l < hi && lo < h).count(); + if claims != 1 { continue; } + map.runs.push((run.start, run.end)); + decoded.push(run); + } + } + let previous = self.pio.publish(decoded); + drop(map); + // Replaced functions may run shutdown callbacks from their destructor. + drop(previous); + drop(retired); + } + + fn bar_cycle(dev: &dyn PciBusDevice, bar: usize, off: u32, bytes: &mut [u8], write: bool) { + for (i, part) in bytes.chunks_mut(4).enumerate() { + if write { + let v = part.iter().enumerate().fold(0, |v, (j, &b)| v | ((b as u32) << (j * 8))); + dev.bar_write(bar, off + i as u32 * 4, v, part.len()); + } else { + let v = dev.bar_read(bar, off + i as u32 * 4, part.len()); + for (j, b) in part.iter_mut().enumerate() { *b = (v >> (j * 8)) as u8; } + } + } + } + /// PCI functions see little-endian values; bytes here remain in address order. + fn pci_cycle(&self, space: Space, address: u32, bytes: &mut [u8], write: bool) -> bool { + let claim = self.slots.read().iter().flatten().find_map(|dev| { + let (bar, off) = dev.config().decode(space, address)?; + let (end_bar, end) = dev + .config() + .decode(space, address.checked_add(bytes.len() as u32 - 1)?)?; + (end_bar == bar && end == off + bytes.len() as u32 - 1).then(|| (dev.clone(), bar, off)) + }); + let Some((dev, bar, off)) = claim else { + crate::dlog!(crate::devlog::LogModule::Bedrock, + "xbridge: head {} no PCI BAR for {space:?} {address:#x}", + self.widget + ); + if !write { + bytes.fill(0xff); + } + return false; + }; + Self::bar_cycle(&*dev, bar, off, bytes, write); + true + } + fn window_cycle(&self, slot: usize, off: u32, bytes: &mut [u8], write: bool) { + let dev = self.st.lock().r(reg::DEVICE_BASE + slot * 8); + let width = if dev & (1 << 22) != 0 { 8 } else { 4 }; + if bytes.len() > width { + // Device windows split a doubleword high word first. + let (low, high) = bytes.split_at_mut(4); + self.window_cycle(slot, off, high, write); + self.window_cycle(slot, off + 4, low, write); + return; + } + let memory = dev & (1 << 12) != 0; + let mut base = dev & if memory { 0x3ff } else { 0xfff }; + if slot < 2 { + base &= !1; + } + let stream = dev & (1 << 13) != 0; + let placed = if stream { + off + } else { + off ^ (width - bytes.len()) as u32 + }; + if stream && write { + bytes.reverse(); + } + self.pci_cycle( + if memory { Space::Mem } else { Space::Io }, + (base << 20).wrapping_add(placed), + bytes, + write, + ); + if stream && !write { + bytes.reverse(); + } + } + fn alias_cycle(&self, space: Space, address: u32, width: usize, bytes: &mut [u8], write: bool) { + let width = if width == 8 { + let slots = self.slots.read().clone(); + let access_len = bytes.len(); + let matches: Vec<_> = slots + .iter() + .enumerate() + .flat_map(|(slot, dev)| { + dev.as_ref() + .map(|d| d.config().windows()) + .unwrap_or_default() + .into_iter() + .filter_map(move |(s, _, base, size)| { + (s == space + && address >= base + && (address as u64 + access_len as u64) + <= base as u64 + size as u64) + .then_some(slot) + }) + }) + .collect(); + if matches.len() == 1 && self.st.lock().dma_64bit & (1 << matches[0]) == 0 { + 4 + } else { + 8 + } + } else { + width + }; + if bytes.len() > width { + let (first, last) = bytes.split_at_mut(width); + self.alias_cycle(space, address, width, first, write); + self.alias_cycle(space, address + width as u32, width, last, write); + return; + } + let swap = self.st.lock().r(reg::CONTROL) & (1 << if space == Space::Mem { 22 } else { 23 }) != 0; + let address = if swap { + address + } else { + address ^ (width - bytes.len()) as u32 + }; + if !swap && write { + bytes.reverse(); + } + self.pci_cycle(space, address, bytes, write); + if !swap && !write { + bytes.reverse(); + } + } + fn slow_pio(&self, address: u64, size: usize, value: Option) -> u64 { + let write = value.is_some(); + let mut bytes = [0u8; 8]; + if (0x200000..0xc00000).contains(&address) { + let (slot, off) = if address < 0x600000 { + (((address - 0x200000) >> 21) as usize, address & 0x1fffff) + } else { + (2 + ((address - 0x600000) >> 20) as usize, address & 0xfffff) + }; + if let Some(v) = value { + for (i, b) in bytes[..size].iter_mut().enumerate() { + *b = (v >> (i * 8)) as u8; + } + } + self.window_cycle(slot, off as u32, &mut bytes[..size], write); + return bytes[..size] + .iter() + .enumerate() + .fold(0, |v, (i, &b)| v | ((b as u64) << (i * 8))); + } + let alias = if (0x40000000..0x80000000).contains(&address) { + Some((Space::Mem, address - 0x40000000, 4)) + } else if (0x80000000..0xc0000000).contains(&address) { + Some((Space::Mem, address - 0x80000000, 8)) + } else if (0x100000000..0x200000000).contains(&address) { + Some((Space::Io, address - 0x100000000, 4)) + } else { + None + }; + if let Some((space, off, width)) = alias { + if let Some(v) = value { + for (i, b) in bytes[..size].iter_mut().enumerate() { + *b = (v >> ((size - 1 - i) * 8)) as u8; + } + } + self.alias_cycle(space, off as u32, width, &mut bytes[..size], write); + return bytes[..size].iter().fold(0, |v, &b| (v << 8) | b as u64); + } + 0 + } + fn access(&self, address: u64, size: usize, value: Option) -> u64 { + let write = value.is_some(); + if (0x20000..0x29000).contains(&address) { + return self.config(address, size, value); + } + if (0x200000..0xc00000).contains(&address) + || (0x40000000..0xc0000000).contains(&address) + || (0x100000000..0x200000000).contains(&address) + { + // An unaligned access can straddle a BAR's final mapped page. + if address & 0xfff > 0x1000 - size as u64 { + return self.slow_pio(address, size, value); + } + let Some(endpoint) = self.pio.target(address, size) else { + return self.slow_pio(address, size, value); + }; + return if let Some(v) = value { + match size { + 1 => { endpoint.write8(address, v as u8); } + 2 => { endpoint.write16(address, v as u16); } + 4 => { endpoint.write32(address, v as u32); } + _ => { endpoint.write64(address, v); } + } + 0 + } else { + match size { + 1 => endpoint.read8(address).data as u64, + 2 => endpoint.read16(address).data as u64, + 4 => endpoint.read32(address).data as u64, + _ => endpoint.read64(address).data, + } + }; + } + if size == 8 && address & 7 == 0 && (reg::ATE_UPPER_BASE..reg::ATE_LOWER_BASE).contains(&address) && write { + self.st.lock().ate[((address - reg::ATE_UPPER_BASE) / 8) as usize] = + value.unwrap() & 0x0000ffffffffffff; + return 0; + } + if size == 4 && address & 7 == 4 { + if !write { + if (reg::ATE_UPPER_BASE..reg::ATE_LOWER_BASE).contains(&address) { + return self.st.lock().ate[((address - reg::ATE_UPPER_BASE) / 8) as usize] >> 32; + } + if (reg::ATE_LOWER_BASE..reg::ATE_END).contains(&address) { + return self.st.lock().ate[((address - reg::ATE_LOWER_BASE) / 8) as usize] as u32 as u64; + } + } + if address < 0x300 && Self::defined(address as usize & !7) { + if let Some(value) = value { + self.reg_write(address, value as u32); + return 0; + } + return self.st.lock().r(address as usize & !7) as u64; + } + } + if Self::deferred(address) { + crate::dlog!(crate::devlog::LogModule::Bedrock, + "xbridge: head {} deferred access at {address:#x}", + self.widget + ); + } else { + self.invalid(address); + } + 0 + } + fn translate( + &self, + slot: u8, + address: u64, + write: bool, + packets: bool, + ) -> Option<(u8, u64, u64, bool)> { + let mut st = self.st.lock(); + let dev = st.r(reg::DEVICE_BASE + slot as usize * 8); + let wide = st.dma_64bit & (1 << slot) != 0; + let device_swap = dev & (1 << 19) != 0; + let (widget, target, swap, direct, page, wide64) = if (0x40000000..0x40800000) + .contains(&address) + || (0x60000000..0x60800000).contains(&address) + { + let shift = if st.r(reg::CONTROL) & (1 << 21) != 0 { 14 } else { 12 }; + let index = ((address & 0x1fffffff) >> shift) as usize; + if index >= ATE_MAPPED || st.ate[index] & 1 == 0 { + let v = st.r(reg::INTERRUPT_STATUS) | (1 << 30); + st.set(reg::INTERRUPT_STATUS, v); + let v = st.r(reg::INTERRUPT_MULTIPLE) | (1 << 30); + st.set(reg::INTERRUPT_MULTIPLE, v); + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "xbridge: invalid page DMA {address:#x}, entry {index}" + ); + return None; + } + let ate = st.ate[index]; + let mask = (1 << shift) - 1; + ( + ((ate >> 8) & 15) as u8, + (ate & 0x0000fffffffff000 & !mask) | (address & mask), + ate & 32 != 0 || address & (1 << 29) != 0, + false, + true, + false, + ) + } else if address >= 0x1000000000000000 { + ( + (address >> 60) as u8, + address & 0x0000ffffffffffff, + address & (1 << 55) != 0, + false, + false, + true, + ) + } else if (0x80000000..=0xffffffff).contains(&address) { + let direct = st.r(reg::DIRECT_MAP); + let mut target = ((direct as u64 & 0x1ffff) << 31) | (address & 0x7fffffff); + if direct & 0x1ffff == 0 && direct & (1 << 17) != 0 { + target += 0x20000000; + } + ( + ((direct >> 20) & 15) as u8, + target, + device_swap, + true, + false, + false, + ) + } else { + crate::dlog!(crate::devlog::LogModule::Bedrock, "xbridge: unsupported DMA {address:#x}"); + return None; + }; + let half = packets + && swap + && st.wide_slots & (1 << slot) != 0 + && if write { + direct || page || (wide64 && device_swap) + } else { + direct || (wide64 && wide && device_swap) || (page && address & (1 << 29) != 0) + }; + let mask = if swap || half { + 0 + } else if wide { + 7 + } else { + 3 + }; + Some((widget, target, mask, half)) + } + // Retain the existing memory path's probe before issuing a chunk's writes. + // The neutral helper alone can commit bytes before reporting a refusal. + fn write_memory(mem: &dyn BusDevice, address: u64, data: &[u8]) -> bool { + if data.is_empty() { + return true; + } + let first = address & !7; + let Some(end) = address + .checked_add(data.len() as u64) + .and_then(|a| a.checked_add(7)) + else { + return false; + }; + let mut probe = vec![0; ((end & !7) - first) as usize]; + read_bytes(mem, first, &mut probe) == BUS_OK && write_bytes(mem, address, data) == BUS_OK + } + fn dma(&self, slot: u8, address: u64, bytes: &mut [u8], write: bool) -> PciDmaStatus { + let sender = self.routes.dma.read().clone(); + let mut good = true; + let mut done = 0; + while done < bytes.len() { + let Some(at) = address.checked_add(done as u64) else { + if !write { + bytes[done..].fill(0); + } + return PciDmaStatus::MasterAbort; + }; + let n = (0x1000 - (at as usize & 0xfff)).min(bytes.len() - done); + if (0x3fff0000..0x40000000).contains(&at) { + if !write { + bytes[done..done + n].fill(0); + } + done += n; + continue; + } + if !write { + bytes[done..done + n].fill(0); + } + let translated = self.translate(slot, at, write, sender.is_some()); + let accepted = if let Some((widget, target, mask, half)) = translated { + if let Some(send) = &sender { + // Packet legs use naturally aligned accesses, retaining source identity. + let mut offset = 0; + let mut ok = true; + while offset < n { + let p = target + offset as u64; + let mut size = 8usize; + while size > n - offset || p & (size as u64 - 1) != 0 { + size /= 2; + } + let mut data = [0u8; 8]; + if write { + for i in 0..size { + data[i] = bytes[done + + offset + + if half { + i + } else { + i ^ (mask as usize & (size - 1)) + }]; + } + } + let placed = if half { + p + } else { + p ^ (mask & !(size as u64 - 1)) + }; + let packet = DmaPacket { + source_widget: (self.st.lock().r(reg::CONTROL) & 15) as u8, + slot, + widget, + address: placed, + write, + half_write_requested: half && write && size == 8, + half_read_requested: half && !write && size == 8, + }; + let part = send(packet, &mut data[..size]); + ok &= part; + if !write { + for i in 0..size { + bytes[done + offset + i] = if part { + data[if half { + i + } else { + i ^ (mask as usize & (size - 1)) + }] + } else { + 0 + }; + } + } + offset += size; + } + ok + } else { + let target_device = self.routes.targets.read()[widget as usize].clone(); + target_device.is_some_and(|mem| { + let low = target & !mask; + let high = (target + n as u64 + mask) & !mask; + let mut data = vec![0; (high - low) as usize]; + if write { + for i in 0..n { + data[(((target + i as u64) ^ mask) - low) as usize] = + bytes[done + i]; + } + if low == target && high == target + n as u64 { + Self::write_memory(&*mem, low, &data) + } else { + let mut ok = true; + for i in 0..n { + let p = (target + i as u64) ^ mask; + ok &= Self::write_memory( + &*mem, + p, + &data[(p - low) as usize..(p - low) as usize + 1], + ); + } + ok + } + } else { + // Preserve the old helper's covering-doubleword probe + // and zero the whole failed physical chunk, even if + // the neutral helper returned part of its contents. + let first = low & !7; + let mut fetched = vec![0; (((high + 7) & !7) - first) as usize]; + let ok = read_bytes(&*mem, first, &mut fetched) == BUS_OK; + for i in 0..n { + bytes[done + i] = if ok { + fetched[(((target + i as u64) ^ mask) - first) as usize] + } else { + 0 + }; + } + ok + } + }) + } + } else { + false + }; + if !accepted { + good = false; + } + done += n; + } + if good { + PciDmaStatus::Ok + } else { + PciDmaStatus::MasterAbort + } + } +} + +struct XbowState { + reg: [u32; 96], + status: u32, + present: u8, + eight_bit: u8, + quiet_until: u64, +} +impl XbowState { + fn new() -> Self { + let mut st = Self { + reg: [0; 96], + status: 0, + present: (1 << 2) | (1 << 6) | (1 << 7), + eight_bit: 0, + quiet_until: 0, + }; + st.reset(); + st + } + fn reset(&mut self) { + self.reg.fill(0); + self.status = 0; + for (off, value) in [ + (xbow::IDENTIFICATION, 0x2d000001), + (xbow::CONTROL, 0x80), + (xbow::PACKET_TIMEOUT, 0xfffff), + (xbow::INT_UPPER, 0x80000), + (xbow::LLP_CONTROL, 0x03ff1810), + (xbow::ARB_RELOAD, 2), + (xbow::NIC, 2), + (xbow::FENCE_BASE, 255), + ] { + self.reg[ix(off)] = value; + } + for i in 0..8 { + self.reg[ix(xbow::FENCE_BASE + 8 + i * 8)] = 255; + self.reg[ix(xbow::LINK_BASE + xbow::LINK_CONTROL + i * xbow::LINK_STRIDE)] = 0x80000; + } + } + fn packet(&self, level: bool) -> InterruptPacket { + let upper = self.reg[ix(xbow::INT_UPPER)]; + InterruptPacket { + source_widget: 0, + widget: ((upper >> 16) & 15) as u8, + address: ((upper as u64 & 0xffff) << 32) | self.reg[ix(xbow::INT_LOWER)] as u64, + vector: (upper >> 24) as u8, + level, + } + } +} +// Lock order: order, then st. Reset outputs run with both released. +struct Crossbar { + st: Mutex, + order: Mutex<()>, + routes: Arc, + heads: [Weak; 2], + reset_output: RwLock>>, +} +impl Crossbar { + fn reset_port(&self, port: u8) { + if (14..=15).contains(&port) { + if let Some(h) = self.heads[(port - 14) as usize].upgrade() { + h.reset(); + } + } + let out = self.reset_output.read().clone(); + if let Some(out) = out { + out(port); + } + } + fn warm_reset(&self, source: u8, now: u64) { + assert!((8..=15).contains(&source)); + let ports = { + let mut st = self.st.lock(); + if now < st.quiet_until { + return; + } + st.quiet_until = now.saturating_add(10240); + if st.reg[ix(xbow::FENCE_BASE)] & (1 << (source - 8)) == 0 { + return; + } + let ports: Vec<_> = (0..8) + .filter(|&i| st.reg[ix(xbow::FENCE_BASE + 8 + i * 8)] & (1 << (source - 8)) != 0) + .map(|i| i as u8 + 8) + .collect(); + st.reset(); + ports + }; + for port in ports { + self.reset_port(port); + } + } + fn access(&self, port: u8, address: u64, size: usize, value: Option) -> u64 { + let _order = self.order.lock(); + let mut packet = None; + let mut reset = None; + let mut access_error = false; + let result = { + let mut st = self.st.lock(); + let off = address as u32 & !7; + if size != 4 || address & 7 != 4 || address >= xbow::LINK_END as u64 || (READ_ONLY_BASE..=READ_ONLY_LAST).contains(&off) { + st.status |= 32; + access_error = true; + if st.reg[ix(xbow::CONTROL)] & 32 != 0 && st.status & (1 << 23) == 0 { + st.status |= 1 << 23; + packet = Some(st.packet(true)); + } + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "xbridge-xbow: invalid access {address:#x} width {size}" + ); + 0 + } else if let Some(v) = value { + let v = v as u32; + if off >= xbow::LINK_BASE { + match (off - xbow::LINK_BASE) % xbow::LINK_STRIDE { + xbow::LINK_CONTROL => st.reg[ix(off)] = v & 0xbf7f01ff, + xbow::LINK_ARB_UPPER | xbow::LINK_ARB_LOWER => st.reg[ix(off)] = v, + xbow::LINK_RESET => reset = Some((8 + (off - xbow::LINK_BASE) / xbow::LINK_STRIDE) as u8), + _ => { + st.status |= 32; + access_error = true; + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "xbridge-xbow: read-only link register offset {off:#x}" + ); + } + } + } else { + match off { + xbow::CONTROL => st.reg[ix(off)] = v & 0xfe, + xbow::PACKET_TIMEOUT => st.reg[ix(off)] = v & 0xfffff, + xbow::INT_UPPER => st.reg[ix(off)] = (v & 0xff07ffff) | 0x80000, + xbow::INT_LOWER => st.reg[ix(off)] = v, + xbow::ERR_CMD => { + st.reg[ix(xbow::ERR_CMD)] = 0; + st.reg[ix(xbow::ERR_UPPER)] = 0; + st.reg[ix(xbow::ERR_LOWER)] = 0; + } + xbow::LLP_CONTROL => st.reg[ix(off)] = v & 0x3ffffff, + xbow::ARB_RELOAD => st.reg[ix(off)] = v & 0x3f, + xbow::PERF_A | xbow::PERF_B => st.reg[ix(off)] = v & 0x700000, + xbow::NIC => { + st.reg[ix(off)] = (v & 0xffffc) | 2; + if v & 0xffc00 != 0 { + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "xbridge-xbow: one-wire pulse unimplemented at register offset 0x70" + ); + } + } + xbow::FENCE_BASE..=FENCE_LAST => st.reg[ix(off)] = v & 255, + xbow::LOCK => { + if st.reg[ix(off)] == 0 { + st.reg[ix(off)] = v & 255; + } + } + xbow::LOCK_CLEAR => st.reg[ix(xbow::LOCK)] = 0, + _ => { + st.status |= 32; + access_error = true; + crate::dlog!( + crate::devlog::LogModule::Bedrock, + "xbridge-xbow: read-only register offset {off:#x}" + ); + } + } + } + 0 + } else if off >= xbow::LINK_BASE { + let link = (off - xbow::LINK_BASE) / xbow::LINK_STRIDE; + let present = st.present & (1 << link) != 0; + match (off - xbow::LINK_BASE) % xbow::LINK_STRIDE { + xbow::LINK_FLUSH => 0, + xbow::LINK_STATUS | xbow::LINK_STATUS_CLEAR => { + if present { + 1 << 31 + } else { + 1 << 5 + } + } + xbow::LINK_AUX_STATUS => { + (if present { 1 << 5 } else { 1 << 6 }) + | (if st.eight_bit & (1 << link) != 0 { + 1 << 4 + } else { + 0 + }) + } + xbow::LINK_RESET => { + st.status |= 32; + access_error = true; + 0 + } + _ => st.reg[ix(off)] as u64, + } + } else { + match off { + xbow::STATUS => (st.status | ((port as u32) << 6)) as u64, + xbow::STATUS_CLEAR => { + let v = st.status | ((port as u32) << 6); + if st.status & (1 << 23) != 0 { + packet = Some(st.packet(false)); + } + st.status &= !((1 << 23) | 0x3f); + v as u64 + } + xbow::LOCK_CLEAR => st.reg[ix(xbow::LOCK)] as u64, + _ => st.reg[ix(off)] as u64, + } + } + }; + if access_error && packet.is_none() { + let mut st = self.st.lock(); + if st.reg[ix(xbow::CONTROL)] & 32 != 0 && st.status & (1 << 23) == 0 { + st.status |= 1 << 23; + packet = Some(st.packet(true)); + } + } + if let Some(p) = packet { + self.routes.interrupt(p); + } + // Reset outputs may call back into the chip, so release ordering too. + drop(_order); + if let Some(port) = reset { + self.reset_port(port); + } + result + } +} +struct CrossbarView { + xbow: Arc, + port: u8, +} + +macro_rules! bus_view { + ($ty:ty, $read:expr) => { + impl BusDevice for $ty { + fn read8(&self, a: u64) -> BusRead8 { + BusRead8::ok(($read)(self, a, 1, None) as u8) + } + fn read16(&self, a: u64) -> BusRead16 { + BusRead16::ok(($read)(self, a, 2, None) as u16) + } + fn read32(&self, a: u64) -> BusRead32 { + BusRead32::ok(($read)(self, a, 4, None) as u32) + } + fn read64(&self, a: u64) -> BusRead64 { + BusRead64::ok(($read)(self, a, 8, None)) + } + fn write8(&self, a: u64, v: u8) -> u32 { + ($read)(self, a, 1, Some(v as u64)); + BUS_OK + } + fn write16(&self, a: u64, v: u16) -> u32 { + ($read)(self, a, 2, Some(v as u64)); + BUS_OK + } + fn write32(&self, a: u64, v: u32) -> u32 { + ($read)(self, a, 4, Some(v as u64)); + BUS_OK + } + fn write64(&self, a: u64, v: u64) -> u32 { + ($read)(self, a, 8, Some(v)); + BUS_OK + } + } + }; +} +bus_view!(XBridgeHead, |s: &XBridgeHead, a, n, v| s.access(a, n, v)); +bus_view!(Crossbar, |s: &Crossbar, a, n, v| s.access( + HOST_WIDGET, + a, + n, + v +)); +bus_view!(CrossbarView, |s: &CrossbarView, a, n, v| s + .xbow + .access(s.port, a, n, v)); + +#[cfg(test)] +mod tests; diff --git a/src/dev/xbridge/tests.rs b/src/dev/xbridge/tests.rs new file mode 100644 index 000000000..1caeff8ce --- /dev/null +++ b/src/dev/xbridge/tests.rs @@ -0,0 +1,738 @@ +// Cases follow tests/qtest/sn1-xbridge-test.c (page-dma, interrupt-packets, +// access-errors), sn1-xbridge-chip-test.c (head-dma-reaches-the-host, +// warm-reset-reaches-the-heads) and sn1-xbridge-xbow-test.c (reset-values, +// lock, fences, link-reset-one-shot) in jamesbraid/qemu-sgi-origami. +use super::*; +use crate::traits::BUS_ERR; +struct Memory(Mutex>); +impl BusDevice for Memory { + fn read32(&self, a: u64) -> BusRead32 { + let mem = self.0.lock(); + if a as usize + 4 > mem.len() { + return BusRead32::err(); + } + BusRead32::ok( + mem[a as usize..a as usize + 4] + .iter() + .fold(0, |v, &b| (v << 8) | b as u32), + ) + } + fn write32(&self, a: u64, v: u32) -> u32 { + let mut mem = self.0.lock(); + if a as usize + 4 > mem.len() { + return BUS_ERR; + } + for i in 0..4 { + mem[a as usize + i] = (v >> (24 - i * 8)) as u8; + } + BUS_OK + } + + fn read64(&self, a: u64) -> BusRead64 { + let mem = self.0.lock(); + if a as usize + 8 > mem.len() { + return BusRead64::err(); + } + BusRead64::ok( + mem[a as usize..a as usize + 8] + .iter() + .fold(0, |v, &b| (v << 8) | b as u64), + ) + } + fn write64(&self, a: u64, v: u64) -> u32 { + let mut mem = self.0.lock(); + if a as usize + 8 > mem.len() { + return BUS_ERR; + } + for i in 0..8 { + mem[a as usize + i] = (v >> (56 - i * 8)) as u8; + } + BUS_OK + } + fn write64_masked(&self, a: u64, v: u64, mask: u64) -> u32 { + let old = self.read64(a); + if old.status != BUS_OK { + return BUS_ERR; + } + self.write64(a, (old.data & !mask) | (v & mask)) + } + + fn read8(&self, a: u64) -> BusRead8 { + match self.0.lock().get(a as usize) { + Some(&value) => BusRead8 { + data: value, + status: BUS_OK, + }, + None => BusRead8 { + data: 0, + status: BUS_ERR, + }, + } + } + fn write8(&self, a: u64, v: u8) -> u32 { + match self.0.lock().get_mut(a as usize) { + Some(b) => { + *b = v; + BUS_OK + } + None => BUS_ERR, + } + } +} +fn chip() -> (Arc, Arc) { + let mem = Arc::new(Memory(Mutex::new(vec![0; 0x8000]))); + (XBridge::new(mem.clone()), mem) +} + +/// The machine lock registry must not extend the lifetime of a chip or attached function. +#[test] +fn lock_registry_does_not_keep_the_chip_or_its_devices_alive() { + let memory = Arc::new(Memory(Mutex::new(vec![0; 0x8000]))); + let memory_ref = Arc::downgrade(&memory); + let chip = XBridge::new(memory); + let chip_ref = Arc::downgrade(&chip); + let heads = chip.heads.each_ref().map(Arc::downgrade); + let pio = chip.heads.each_ref().map(|head| Arc::downgrade(&head.pio)); + let crossbar = Arc::downgrade(&chip.xbow); + let card = Card::with_bar(crate::pci::Bar::Mem(0x1000), 0); + let card_ref = Arc::downgrade(&card); + chip.attach(15, 1, card.clone()); + chip.set_reset_output(Arc::new(move |_| card.config.reset())); + chip.head(15).write32(0x21010, 0x02000000); + chip.head(15).write32(0x21004, 2); + assert!(!chip.pci_head(15).pio_map.lock().runs.is_empty()); + chip.register_locks(); + + assert!(heads.iter().all(|head| head.upgrade().is_some())); + assert!(crossbar.upgrade().is_some()); + assert!(card_ref.upgrade().is_some()); + assert!(memory_ref.upgrade().is_some()); + drop(chip); + assert!(chip_ref.upgrade().is_none()); + assert!(heads.iter().all(|head| head.upgrade().is_none())); + assert!(pio.iter().all(|view| view.upgrade().is_none())); + assert!(crossbar.upgrade().is_none()); + assert!(card_ref.upgrade().is_none()); + assert!(memory_ref.upgrade().is_none()); +} +/// A DMA packet reports failure without losing the valid prefix already transferred. +#[test] +fn page_dma_keeps_valid_prefix_and_refuses_invalid_suffix() { + physical_chunk_refusal_does_not_publish_a_prefix(); + let (c, mem) = chip(); + let f = c.head(15); + f.write64(reg::ATE_UPPER_BASE as u64, 0x2000 | (10 << 8) | 0x21); + let dma = c.dma(15, 2); + let bytes = [1, 2, 3, 4, 5, 6, 7, 8]; + assert_eq!(dma.dma_write(0x40000ffc, &bytes), PciDmaStatus::MasterAbort); + assert_eq!(&mem.0.lock()[0x2ffc..0x3000], &[1, 2, 3, 4]); + assert_eq!(&mem.0.lock()[0x3000..0x3004], &[0; 4]); + let mut read = [0xff; 8]; + assert_eq!( + dma.dma_read(0x40000ffc, &mut read), + PciDmaStatus::MasterAbort + ); + assert_eq!(read, [1, 2, 3, 4, 0, 0, 0, 0]); + assert_ne!(f.read32((reg::INTERRUPT_STATUS + 4) as u64).data & (1 << 30), 0); + f.write32((reg::INTERRUPT_RESET + 4) as u64, 2); + assert_eq!(f.read32((reg::INTERRUPT_STATUS + 4) as u64).data & (1 << 30), 0); + f.write64((reg::ATE_UPPER_BASE + 800 * 8) as u64, 0xffff123456789abc); + assert_eq!(f.read32((reg::ATE_UPPER_BASE + 4 + 800 * 8) as u64).data, 0x1234); + assert_eq!(f.read32((reg::ATE_LOWER_BASE + 4 + 800 * 8) as u64).data, 0x56789abc); + f.write64((reg::ATE_UPPER_BASE + 512 * 8) as u64, 0x2a21); + f.write32((reg::DIRECT_MAP + 4) as u64, 9 << 20); + for (address, status) in [(0x80002000, PciDmaStatus::MasterAbort), + (0x3fff8000, PciDmaStatus::Ok), (0x40800000, PciDmaStatus::MasterAbort), + (0x40200000, PciDmaStatus::MasterAbort)] { + let mut bytes = [0xff; 4]; + assert_eq!(dma.dma_read(address, &mut bytes), status); + assert_eq!(bytes, [0; 4]); + } +} +// A mapped memory range whose middle word refuses accesses. +struct FaultMemory(Arc); +impl BusDevice for FaultMemory { + fn read8(&self, a: u64) -> BusRead8 { + if (0x2004..0x2008).contains(&a) { + BusRead8::err() + } else { + self.0.read8(a) + } + } + fn write8(&self, a: u64, v: u8) -> u32 { + if (0x2004..0x2008).contains(&a) { + BUS_ERR + } else { + self.0.write8(a, v) + } + } + fn read32(&self, a: u64) -> BusRead32 { + if (0x2004..0x2008).contains(&a) { + BusRead32::err() + } else { + self.0.read32(a) + } + } + fn write32(&self, a: u64, v: u32) -> u32 { + if (0x2004..0x2008).contains(&a) { + BUS_ERR + } else { + self.0.write32(a, v) + } + } +} +fn physical_chunk_refusal_does_not_publish_a_prefix() { + let mem = Arc::new(Memory(Mutex::new(vec![0xaa; 0x8000]))); + let c = XBridge::new(Arc::new(FaultMemory(mem.clone()))); + let f = c.head(15); + f.write32((reg::DIRECT_MAP + 4) as u64, 10 << 20); + f.write32((reg::DEVICE_BASE + 4) as u64, 1 << 19); + let dma = c.dma(15, 0); + let mut bytes = [0xff; 8]; + assert_eq!( + dma.dma_read(0x80002002, &mut bytes), + PciDmaStatus::MasterAbort + ); + assert_eq!( + bytes, [0; 8], + "a failed physical chunk zeroes its readable prefix and suffix" + ); + assert_eq!( + dma.dma_write(0x80002002, &[1; 8]), + PciDmaStatus::MasterAbort + ); + assert_eq!( + &mem.0.lock()[0x2000..0x2010], + &[0xaa; 16], + "the probe prevents writes before or after the refused word" + ); +} +/// Enabling an asserted guest interrupt pin delivers its programmed destination. +#[test] +fn interrupt_keeps_destination_and_enabling_a_live_pin_asserts() { + let (c, _) = chip(); + let f = c.head(15); + let packets = Arc::new(Mutex::new(Vec::new())); + let capture = packets.clone(); + c.set_interrupt_sender(Arc::new(move |p| { + capture.lock().push(p); + true + })); + f.write32((reg::INT_DEST_UPPER + 4) as u64, 0xa0001); + f.write32((reg::INT_DEST_LOWER + 4) as u64, 0x23450080); + f.write32((reg::INTERRUPT_HOST_BASE + 4) as u64, (0x155 << 8) | 0x42); + c.pin_line(15, 0).set(true); + assert!(packets.lock().is_empty()); + f.write32((reg::INTERRUPT_ENABLE + 4) as u64, 1); + f.write32((reg::INTERRUPT_MODE + 4) as u64, 1); + c.pin_line(15, 0).set(false); + let p = packets.lock(); + assert_eq!(p.len(), 2); + assert_eq!(p[0].source_widget, 15); + assert_eq!(p[0].widget, 10); + assert_eq!(p[0].vector, 0x42); + assert_eq!(p[0].address, (0x123450080 & !(0x3ff << 38)) | (0x155 << 38)); + assert!(p[0].level); + assert!(!p[1].level); + drop(p); + let card = Card::new(); + c.attach(15, 2, card.clone()); + let (probe, line) = crate::irq::LevelProbe::line(); + c.set_pci_irq(15, 2, 0, line); + card.irq.lock().as_ref().unwrap().set(true); + assert!(probe.level()); + assert_eq!(f.read32((reg::INTERRUPT_STATUS + 4) as u64).data & 255, 0); + c.reset(); + assert!(!probe.level()); +} + +/// The guest receives host DMA packets with their original widget and address placement. +#[test] +fn host_dma_places_packets_and_rejects_invalid_addresses() { + let (c, mem) = chip(); + let f = c.head(15); + let d = c.dma(15, 0); + f.write32((reg::DIRECT_MAP + 4) as u64, 10 << 20); + f.write64(reg::ATE_UPPER_BASE as u64, 0x3000 | (10 << 8) | 1); + for (wide, address, target, expected) in [ + (false, 0x80002000, 0x2000, [3, 2, 1, 0, 7, 6, 5, 4]), + (true, 0x80002000, 0x2000, [7, 6, 5, 4, 3, 2, 1, 0]), + (true, 0x60000000, 0x3000, [0, 1, 2, 3, 4, 5, 6, 7]), + (true, (10 << 60) | (1 << 55) | 0x2000, 0x2000, [0, 1, 2, 3, 4, 5, 6, 7]), + ] { + c.set_dma_64bit(15, 0, wide); + assert_eq!(d.dma_write(address, &[0, 1, 2, 3, 4, 5, 6, 7]), PciDmaStatus::Ok); + assert_eq!(&mem.0.lock()[target..target + 8], &expected); + } + let records = Arc::new(Mutex::new(Vec::new())); + let out = records.clone(); + c.set_dma_sender(Arc::new(move |p, b| { + out.lock().push((p, b.to_vec())); + true + })); + c.set_wide_slot_mask(15, 1 << 3); + c.set_dma_64bit(15, 3, true); + f.write32((reg::CONTROL + 4) as u64, 8); + f.write32((reg::DIRECT_MAP + 4) as u64, 10 << 20); + f.write32((reg::DEVICE_BASE + 3 * 8 + 4) as u64, 1 << 19); + assert_eq!( + c.dma(15, 3) + .dma_write(0x80002000, &[0, 1, 2, 3, 4, 5, 6, 7]), + PciDmaStatus::Ok + ); + let r = records.lock(); + assert_eq!(r.len(), 1); + assert_eq!(r[0].0.source_widget, 8); + assert_eq!(r[0].0.slot, 3); + assert_eq!(r[0].0.widget, 10); + assert_eq!(r[0].0.address, 0x2000); + assert!(r[0].0.half_write_requested); + assert_eq!(r[0].1, [0, 1, 2, 3, 4, 5, 6, 7]); + drop(r); + c.set_dma_sender(Arc::new(|p, b| { + if p.address >= 0x2008 { + return false; + } + b.fill(0x42); + true + })); + let mut bytes = [0xff; 16]; + assert_eq!( + c.dma(15, 0).dma_read(0x80002000, &mut bytes), + PciDmaStatus::MasterAbort + ); + assert_eq!( + bytes, + [0x42, 0x42, 0x42, 0x42, 0x42, 0x42, 0x42, 0x42, 0, 0, 0, 0, 0, 0, 0, 0] + ); +} +/// A PCI function may inspect XBridge from its link-reset callback. +#[test] +fn link_reset_callback_resets_a_function_and_can_reenter_the_chip() { + use crate::dev::isp1040::{mbox, reg as isp_reg, CTRL_INT_ENABLE, CTRL_RISC_INT_ENABLE, HOST_CMD_SHIFT}; + use crate::dev::isp12160::Isp12160; + + let (c, _) = chip(); + let isp = Arc::new(Isp12160::new(Isp12160::new_bus(), Isp12160::new_bus())); + c.attach(15, 1, isp.clone()); + let (probe, line) = crate::irq::LevelProbe::line(); + c.set_pci_irq(15, 1, 0, line); + isp.config().write(1, 6); + isp.config().write(5, 0x02000000); + isp.bar_write(1, isp_reg::BUS_CTRL, u32::from(CTRL_INT_ENABLE | CTRL_RISC_INT_ENABLE), 2); + isp.bar_write(1, isp_reg::MAILBOX_BASE, u32::from(mbox::NOP), 2); + isp.bar_write(1, isp_reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT, 2); + assert!(probe.level()); + assert_eq!(isp.bar_read(1, isp_reg::MAILBOX_BASE, 2), u32::from(mbox::COMPLETE)); + + let weak = Arc::downgrade(&c); + let reset_isp = isp.clone(); + c.set_reset_output(Arc::new(move |port| { + assert_eq!(port, 15); + let chip = weak.upgrade().unwrap(); + // Callbacks own attached engines and may inspect or rewire the chip. + assert_eq!(chip.head(port).read32((reg::DIRECT_MAP + 4) as u64).data, 0x20000); + assert_eq!(chip.crossbar().read32((xbow::IDENTIFICATION + 4) as u64).data, 0x2d000001); + assert_eq!(reset_isp.config().command(), 0); + reset_isp.power_on(); + chip.set_reset_output(Arc::new(|_| {})); + })); + c.head(15).write32((reg::DIRECT_MAP + 4) as u64, 0xa00000); + c.crossbar().write32((xbow::LINK_BASE + 7 * xbow::LINK_STRIDE + xbow::LINK_RESET + 4) as u64, 1); + assert!(!probe.level()); + assert_eq!(isp.config().bar_base(1), 0); + assert_eq!(isp.bar_read(1, isp_reg::BUS_CTRL, 2), 0); + assert_eq!(isp.bar_read(1, isp_reg::BUS_STATUS, 2), 0); + assert_eq!(isp.bar_read(1, isp_reg::MAILBOX_BASE, 2), 0); + assert_eq!(isp.bar_read(1, isp_reg::MAILBOX_BASE + 2, 2), 0x4953); +} + +/// Only admitted guest ports receive warm-reset notifications after both heads reset. +#[test] +fn warm_reset_notifies_only_admitted_ports_after_head_state_is_reset() { + let (c, _) = chip(); + let ports = Arc::new(Mutex::new(Vec::new())); + let output = ports.clone(); + let weak = Arc::downgrade(&c); + c.set_reset_output(Arc::new(move |port| { + if port >= 14 { + assert_eq!(weak.upgrade().unwrap().head(port).read32((reg::DIRECT_MAP + 4) as u64).data, 0x20000); + } + output.lock().push(port); + })); + let x = c.crossbar(); + x.write32((xbow::FENCE_BASE + 8 + 6 * 8 + 4) as u64, 0xff & !(1 << 2)); + c.head(14).write32((reg::DIRECT_MAP + 4) as u64, 0xa00000); + c.head(15).write32((reg::DIRECT_MAP + 4) as u64, 0xa00000); + c.warm_reset(10, 0); + assert_eq!(*ports.lock(), vec![8, 9, 10, 11, 12, 13, 15]); + assert_eq!(c.head(14).read32((reg::DIRECT_MAP + 4) as u64).data, 0xa00000); + ports.lock().clear(); + c.warm_reset(10, 10239); + assert!(ports.lock().is_empty()); + c.warm_reset(10, 10240); + assert_eq!(*ports.lock(), vec![8, 9, 10, 11, 12, 13, 14, 15]); +} +/// The crossbar reports the first error once and retains its programmed destination. +#[test] +fn crossbar_errors_deliver_once_then_clear_the_original_destination() { + let (c, _) = chip(); + let x = c.crossbar(); + let packets = Arc::new(Mutex::new(Vec::new())); + let out = packets.clone(); + c.set_interrupt_sender(Arc::new(move |p| { + out.lock().push(p); + true + })); + x.write32((xbow::INT_UPPER + 4) as u64, 0x550a0000); + x.write32((xbow::INT_LOWER + 4) as u64, 0x80); + x.write32((xbow::CONTROL + 4) as u64, 0x20); + x.read64(xbow::IDENTIFICATION as u64); + x.write32((xbow::IDENTIFICATION + 4) as u64, 1); + assert_eq!(packets.lock().len(), 1); + assert_eq!(x.read32((xbow::STATUS_CLEAR + 4) as u64).data & ((1 << 23) | 32), (1 << 23) | 32); + let p = packets.lock(); + assert_eq!(p.len(), 2); + assert_eq!(p[0].vector, 0x55); + assert_eq!(p[0].widget, 10); + assert_eq!(p[0].address, 0x80); + assert!(p[0].level); + assert!(!p[1].level); +} + +struct Card { + config: crate::pci::PciConfig, + bytes: Mutex>, + irq: Mutex>, +} +impl Card { + fn new() -> Arc { + Self::with_status_events(0) + } + fn with_status_events(mask: u16) -> Arc { + Self::with_bar(crate::pci::Bar::Mem(32), mask) + } + fn with_bar(bar: crate::pci::Bar, mask: u16) -> Arc { + use crate::pci::{Bar, PciConfig}; + let size = match bar { Bar::Mem(size) | Bar::Io(size) => size as usize, Bar::Unused => 0 }; + Arc::new(Self { + config: PciConfig::new_with_policy( + 0x1234, + 0x5678, + 0, + 0, + [ + bar, + Bar::Unused, + Bar::Unused, + Bar::Unused, + Bar::Unused, + Bar::Unused, + ], + 1, + crate::pci::PciConfigPolicy { status_w1c_mask: mask, ..Default::default() }, + ), + bytes: Mutex::new((0..size).map(|i| i as u8).collect()), + irq: Mutex::new(None), + }) + } +} +impl PciBusDevice for Card { + fn config(&self) -> &crate::pci::PciConfig { + &self.config + } + fn bar_read(&self, _: usize, off: u32, width: usize) -> u32 { + let b = self.bytes.lock(); + (0..width).fold(0, |v, i| v | ((b[off as usize + i] as u32) << (i * 8))) + } + fn bar_write(&self, _: usize, off: u32, value: u32, width: usize) { + let mut b = self.bytes.lock(); + for i in 0..width { + b[off as usize + i] = (value >> (i * 8)) as u8; + } + } + fn connect_interrupt(&self, line: IrqLine) { + *self.irq.lock() = Some(line); + } +} +/// The guest can identify the first invalid head access and its error destination. +#[test] +fn invalid_register_captures_first_address_and_reports_error_destination() { + let (c, _) = chip(); + let f = c.head(15); + let packets = Arc::new(Mutex::new(Vec::new())); + let out = packets.clone(); + c.set_interrupt_sender(Arc::new(move |p| { + out.lock().push(p); + true + })); + f.write32((reg::INT_DEST_UPPER + 4) as u64, 0xa0000); + f.write32((reg::INT_DEST_LOWER + 4) as u64, 0x80); + f.write32((reg::INTERRUPT_ERROR_HOST + 4) as u64, 73); + f.write32((reg::INTERRUPT_ENABLE + 4) as u64, 1 << 24); + f.read32(0x1000004); + f.read32(0x2000004); + assert_eq!(f.read32((reg::ERR_LOWER + 4) as u64).data, 0x1000004); + assert_eq!(packets.lock().len(), 1); + assert_eq!(packets.lock()[0].vector, 73); + f.write32((reg::INTERRUPT_RESET + 4) as u64, 8); + assert_eq!(f.read32((reg::INTERRUPT_STATUS + 4) as u64).data & (1 << 24), 0); +} +/// Narrow guest configuration writes preserve status bits outside the written lanes. +#[test] +fn partial_configuration_writes_preserve_unwritten_status_events() { + let (c, _) = chip(); + let f = c.head(15); + let card = Card::with_status_events(0xc000); + c.attach(15, 0, card.clone()); + for (bus, expected) in [(0, 0x56781234), (1, u32::MAX)] { + f.write32((reg::PCI_TYPE1_CONFIG + 4) as u64, bus << 16); + assert_eq!(f.read32(0x28000).data, expected); + } + card.config.set_status_bits(0xc000); + assert_eq!(f.write16(0x20006, 0x0102), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0xc200_0102); + assert_eq!(f.write8(0x20007, 3), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0xc200_0103); + assert_eq!(f.write8(0x20006, 2), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0xc200_0203); + assert_eq!(f.write8(0x20005, 0xff), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0xc200_0203); + assert_eq!(f.write8(0x20004, 0x80), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0x4200_0203); + assert_eq!(f.write16(0x20004, 0x4000), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0x0200_0203); + card.config.set_status_bits(0xc000); + assert_eq!(f.write32(0x20004, 0x8000_0002), BUS_OK); + assert_eq!(f.read32(0x20004).data, 0x4200_0002); +} + +fn assert_pio_matches_slow(head: &XBridgeHead, card: &Card, address: u64, mask: usize) { + for size in [1, 2, 4, 8] { + for offset in (0..32).step_by(size) { + let at = address + offset as u64; + let before = card.bytes.lock().clone(); + let expected = (0..size).fold(0u64, |v, i| (v << 8) | before[(offset + i) ^ mask] as u64); + assert_eq!(head.access(at, size, None), expected, "read {at:#x} width {size}"); + assert_eq!(head.slow_pio(at, size, None), expected); + head.access(at, size, Some(0x0123456789abcdef)); + let fast = card.bytes.lock().clone(); + *card.bytes.lock() = before.clone(); + head.slow_pio(at, size, Some(0x0123456789abcdef)); + assert_eq!(*card.bytes.lock(), fast, "write {at:#x} width {size}"); + *card.bytes.lock() = before; + } + } +} + +/// Native BAR dispatch preserves bridge data lanes at every supported access width. +#[test] +fn native_bar_pages_preserve_all_pio_lanes() { + for memory in [false, true] { + let (chip, _) = chip(); + let head = chip.pci_head(15); + let bar = if memory { crate::pci::Bar::Mem(0x2000) } else { crate::pci::Bar::Io(0x2000) }; + let card = Card::with_bar(bar, 0); + // Device window 0 reaches a function owned by slot 3. + chip.attach(15, 3, card.clone()); + head.write32(0x23010, 0x02000000 | u32::from(!memory)); + head.write32(0x23004, if memory { 2 } else { 1 }); + for wide in [false, true] { + chip.set_dma_64bit(15, 3, wide); + for stream in [false, true] { + head.write32((reg::DEVICE_BASE + 4) as u64, 0x20 | if memory { 1 << 12 } else { 0 } + | if wide { 1 << 22 } else { 0 } | if stream { 1 << 13 } else { 0 }); + assert!(head.pio_map.lock().runs.contains(&(0x200000, 0x202000))); + assert_pio_matches_slow(head, &card, 0x200000, if stream { 0 } else if wide { 7 } else { 3 }); + for swap in [false, true] { + head.write32((reg::CONTROL + 4) as u64, if swap { 1 << if memory { 22 } else { 23 } } else { 0 }); + let aliases = if memory { + vec![(0x42000000, false), (0x82000000, wide)] + } else { vec![(0x102000000, false)] }; + for (address, wide) in aliases { + assert_pio_matches_slow(head, &card, address, if swap { 0 } else if wide { 7 } else { 3 }); + } + } + } + } + } +} + +/// A moved or reset BAR stops reaching its former endpoint. +#[test] +fn bar_moves_disable_and_reset_revoke_native_pages_without_new_endpoints() { + let (chip, _) = chip(); + let head = chip.pci_head(15); + let card = Card::with_bar(crate::pci::Bar::Mem(0x1000), 0); + let old_ref = Arc::downgrade(&card); + chip.attach(15, 0, card); + head.write32(0x20010, 0x02000000); + head.write32(0x20004, 2); + head.write32((reg::DEVICE_BASE + 4) as u64, 0x1020); + let original: Vec<_> = head.pio_map.lock().entries.iter() + .map(|(key, entry)| (*key, entry.device.clone())).collect(); + let stale = head.pio_map.lock().entries[&(8, 0, 0)].device.clone(); + head.write32((reg::DEVICE_BASE + 4) as u64, 0x1022); + assert_eq!(head.read32(0x200000).data, u32::MAX); + head.write32((reg::DEVICE_BASE + 4) as u64, 0x1020); + assert_eq!(head.read32(0x200000).data, 0x03020100); + for i in 0..128 { + let base = 0x02000000 + i * 0x1000; + head.write32(0x20010, base); + assert_eq!(head.read32(0x40000000 + base as u64).data, 0x03020100); + if i != 0 { + assert_eq!(head.read32(0x40000000 + base as u64 - 0x1000).data, u32::MAX); + } + } + assert_eq!(stale.read32(0x42000000).data, u32::MAX); + let entries = head.pio_map.lock(); + assert_eq!(entries.entries.len(), original.len()); + for (key, device) in &original { assert!(Arc::ptr_eq(device, &entries.entries[key].device)); } + drop(entries); + head.write8(0x20007, 0); + assert!(head.pio_map.lock().runs.is_empty()); + assert_eq!(head.read32(0x4207f000).data, u32::MAX); + head.write8(0x20007, 2); + assert!(!head.pio_map.lock().runs.is_empty()); + chip.crossbar().write32((xbow::LINK_BASE + 7 * xbow::LINK_STRIDE + xbow::LINK_RESET + 4) as u64, 1); + assert!(head.pio_map.lock().runs.is_empty()); + assert_eq!(head.read32(0x4207f000).data, u32::MAX); + head.write32(0x20010, 0x02000000); + head.write32(0x20004, 2); + let stale = head.pio_map.lock().entries[&(8, 0, 0)].device.clone(); + let replacement = Card::with_bar(crate::pci::Bar::Mem(0x1000), 0); + replacement.bytes.lock().fill(0x5a); + chip.attach(15, 0, replacement); + assert_eq!(head.read32(0x42000000).data, u32::MAX); + assert_eq!(stale.read32(0x42000000).data, u32::MAX); + stale.write32(0x42000000, 0xdeadbeef); + assert_eq!(old_ref.upgrade().unwrap().bytes.lock()[..4], [0, 1, 2, 3]); + head.write32(0x20010, 0x02000000); + head.write32(0x20004, 2); + assert_eq!(head.read32(0x42000000).data, 0x5a5a5a5a); +} + +/// A BAR page shared with another mapping must retain the slow decoder. +#[test] +fn partial_or_overlapping_bar_pages_use_the_slow_decoder() { + let (chip, _) = chip(); + let head = chip.pci_head(15); + let small = Card::new(); + let large = Card::with_bar(crate::pci::Bar::Mem(0x2000), 0); + small.bytes.lock().fill(0xa5); + chip.attach(15, 0, small); + chip.attach(15, 1, large.clone()); + for slot in 0..2 { + head.write32(0x20010 + slot * 0x1000, 0x02000000); + head.write32(0x20004 + slot * 0x1000, 2); + } + head.write32((reg::DEVICE_BASE + 4) as u64, 0x1020); + assert!(head.pio_map.lock().runs.is_empty()); + assert_eq!(head.read32(0x42000000).data, 0xa5a5a5a5); + assert_eq!(head.read32(0x42000020).data, 0x23222120); + head.write32(0x20004, 0); + assert!(!head.pio_map.lock().runs.is_empty()); + assert_pio_matches_slow(head, &large, 0x42000000, 3); + head.write32(0x21004, 0); + head.write32(0x20004, 2); + assert!(head.pio_map.lock().runs.is_empty()); + assert_eq!(head.read32(0x42000000).data, 0xa5a5a5a5); +} + +/// BAR holes and cross-boundary cycles must use the original slow decoder, +/// including after the host publishes a replacement set of BAR runs. +#[test] +fn sparse_bar_lookup_rejects_holes_and_cross_boundary_cycles() { + let decode = PioDecode::default(); + let first: Arc = Arc::new(Memory(Mutex::new(vec![0; 0x4000]))); + let second: Arc = Arc::new(Memory(Mutex::new(vec![0; 0x4000]))); + drop(decode.publish(vec![ + BarRun { start: 0x3000, end: 0x4000, dev: second.clone() }, + BarRun { start: 0x1000, end: 0x2000, dev: first.clone() }, + ])); + assert!(Arc::ptr_eq(&decode.target(0x1000, 8).unwrap(), &first)); + assert!(Arc::ptr_eq(&decode.target(0x1ff8, 8).unwrap(), &first)); + assert!(decode.target(0x1ffc, 8).is_none()); + assert!(decode.target(0x2000, 8).is_none()); + assert!(decode.target(0x2ff8, 8).is_none()); + assert!(Arc::ptr_eq(&decode.target(0x3000, 8).unwrap(), &second)); + assert!(decode.target(0x4000, 8).is_none()); + assert!(decode.target(u64::MAX, 8).is_none()); + drop(decode.publish(vec![BarRun { start: 0x2000, end: 0x3000, dev: first.clone() }])); + assert!(decode.target(0x1000, 8).is_none()); + assert!(Arc::ptr_eq(&decode.target(0x2000, 8).unwrap(), &first)); + assert!(decode.target(0x3000, 8).is_none()); +} + +fn xbow_read(x: &Arc, off: u32) -> u32 { + x.read32((off + 4) as u64).data +} + +/// Software probing the crossbar after reset sees these values. Each +/// link's control word resets to 0x80000; that value belongs in the control +/// word, not in the link reset register, which no read can see. +#[test] +fn the_crossbar_resets_to_its_documented_values() { + let (c, _) = chip(); + let x = c.crossbar(); + for (off, value) in [ + (xbow::ERR_UPPER, 0), + (xbow::ERR_LOWER, 0), + (xbow::CONTROL, 0x80), + (xbow::PACKET_TIMEOUT, 0xfffff), + (xbow::INT_UPPER, 0x80000), + (xbow::INT_LOWER, 0), + (xbow::ERR_CMD, 0), + (xbow::LLP_CONTROL, 0x03ff1810), + (xbow::ARB_RELOAD, 2), + (xbow::PERF_A, 0), + (xbow::PERF_B, 0), + (xbow::NIC, 2), + (xbow::FENCE_BASE, 0xff), + (xbow::LOCK, 0), + ] { + assert_eq!(xbow_read(&x, off), value, "{off:#x}"); + } + for link in 0..8 { + let base = xbow::LINK_BASE + link * xbow::LINK_STRIDE; + assert_eq!(xbow_read(&x, xbow::FENCE_BASE + 8 + link * 8), 0xff); + assert_eq!(xbow_read(&x, base + xbow::LINK_CONTROL), 0x80000, "link {}", link + 8); + assert_eq!(xbow_read(&x, base + xbow::LINK_ARB_UPPER), 0); + assert_eq!(xbow_read(&x, base + xbow::LINK_ARB_LOWER), 0); + } +} + +/// Software shares the crossbar between processors with the resource lock: +/// a write takes it only while it is free, so a second holder is refused +/// until the first writes the clear register. +#[test] +fn the_resource_lock_admits_one_holder_until_cleared() { + let (c, _) = chip(); + let x = c.crossbar(); + let other = c.crossbar_for_port(HEAD_E); + x.write32((xbow::LOCK + 4) as u64, 0x5a); + assert_eq!(xbow_read(&x, xbow::LOCK), 0x5a); + other.write32((xbow::LOCK + 4) as u64, 0x3c); + assert_eq!(xbow_read(&x, xbow::LOCK), 0x5a); + assert_eq!(xbow_read(&x, xbow::LOCK_CLEAR), 0x5a); + x.write32((xbow::LOCK_CLEAR + 4) as u64, 0); + assert_eq!(xbow_read(&x, xbow::LOCK), 0); + other.write32((xbow::LOCK + 4) as u64, 0x3c); + assert_eq!(xbow_read(&x, xbow::LOCK), 0x3c); +} + +/// Every reset fence bit is read and write, a link's own bit included, and +/// only the low eight bits exist. +#[test] +fn reset_fences_keep_their_low_eight_bits() { + let (c, _) = chip(); + let x = c.crossbar(); + for off in (xbow::FENCE_BASE..=FENCE_LAST).step_by(8) { + x.write32((off + 4) as u64, 0); + assert_eq!(xbow_read(&x, off), 0); + x.write32((off + 4) as u64, off as u32); + assert_eq!(xbow_read(&x, off), off as u32 & 0xff); + x.write32((off + 4) as u64, u32::MAX); + assert_eq!(xbow_read(&x, off), 0xff); + } +} diff --git a/src/devlog.rs b/src/devlog.rs index d082e7ba1..045cd6f4f 100644 --- a/src/devlog.rs +++ b/src/devlog.rs @@ -55,6 +55,9 @@ pub enum LogModule { /// The Octane's VPro (Odyssey) graphics board: guest accesses the /// model does not support. Odyssey = 28, + Bedrock = 29, + L1 = 30, + Usb = 31, } // Mask bits shared by L1i / L1d / L2c @@ -63,7 +66,7 @@ pub const CACHE_LOG_MISS: u32 = 0x2; // log misses and fills pub const CACHE_LOG_OP: u32 = 0x4; // log CACHE instruction ops impl LogModule { - pub const COUNT: usize = 29; + pub const COUNT: usize = 32; pub fn from_str(s: &str) -> Option { match s { @@ -96,6 +99,9 @@ impl LogModule { "gr2" => Some(Self::Gr2), "rad1" => Some(Self::Rad1), "odyssey" => Some(Self::Odyssey), + "bedrock" => Some(Self::Bedrock), + "l1" => Some(Self::L1), + "usb" => Some(Self::Usb), _ => None, } } @@ -131,6 +137,9 @@ impl LogModule { Self::Gr2 => "gr2", Self::Rad1 => "rad1", Self::Odyssey => "odyssey", + Self::Bedrock => "bedrock", + Self::L1 => "l1", + Self::Usb => "usb", } } @@ -141,7 +150,7 @@ impl LogModule { Self::Vino, Self::Dcb, Self::Vc2, Self::Cmap, Self::Xmap, Self::Bt445, Self::Scc, Self::Ps2, Self::Rtc, Self::Eeprom, Self::L1i, Self::L1d, Self::L2c, Self::Ultra, Self::Scsi1, - Self::Nveeprom, Self::Gr2, Self::Rad1, Self::Odyssey, + Self::Nveeprom, Self::Gr2, Self::Rad1, Self::Odyssey, Self::Bedrock, Self::L1, Self::Usb, ] } @@ -283,6 +292,9 @@ impl DevLog { ModuleLog::new(), // Gr2 ModuleLog::new(), // Rad1 ModuleLog::new(), // Odyssey + ModuleLog::new(), // Bedrock + ModuleLog::new(), // L1 + ModuleLog::new(), // Usb ], } } diff --git a/src/ip30_machine.rs b/src/ip30_machine.rs index cdaa19b16..5a0f5e0d7 100644 --- a/src/ip30_machine.rs +++ b/src/ip30_machine.rs @@ -16,7 +16,8 @@ use std::sync::atomic::{AtomicU64, Ordering}; use std::sync::Arc; use crate::config::{CpuModel, MachineConfig, MachineProfile}; -use crate::cpu::mips_cache_shadow::R10000ShadowCache; +use crate::cpu::mips_cache_v2::MipsCache; +use crate::cpu::mips_cache_shadow::{R10000ShadowCache, R12000ShadowCache, R14000_PRID}; use crate::cpu::smp::World; use crate::cpu::mips_exec::{CpuDevice, MipsCpu, MipsCpuConfig, MipsExecutor}; use crate::cpu::mips_tlb::MipsTlb; @@ -49,9 +50,29 @@ pub const OCTANE_PROM_FILE: &str = "IP30prom.bin"; /// The 195 MHz R10000 of SGI's standard 128 MB Octane configurations (the /// OCTANE Technical Report): Count at half that. pub const OCTANE_COUNT_HZ: u64 = 97_500_000; +/// Count for the 360 MHz R12000 (revision 3.5, 2 MB) that SGI sold in the +/// Octane: half the pipeline clock. +pub const OCTANE_R12000_COUNT_HZ: u64 = 180_000_000; +/// Count for the 600 MHz R14000 (revision 2.4, 2 MB) that SGI sold in the +/// Octane: half the pipeline clock. +pub const OCTANE_R14000_COUNT_HZ: u64 = 300_000_000; + /// R10000 revision 2.7, as an Octane's hinv reports it; the raw PRId is /// inferred from the R10000 manual's encoding. pub const OCTANE_PRID: u32 = 0x0927; + +/// CP0 Count for a part when `[clock] fixed_mhz` is unset: half the clock of +/// one speed that part shipped at in an Octane, so a plain `cpu = "r12000"` +/// or `"r14000"` reports a clock its tuple had. The PROM measures Count +/// against the HEART timer, so `hinv` prints twice this. +fn octane_default_count_hz(cpu: CpuModel) -> u64 { + match cpu { + CpuModel::R12000 => OCTANE_R12000_COUNT_HZ, + CpuModel::R14000 => OCTANE_R14000_COUNT_HZ, + _ => OCTANE_COUNT_HZ, + } +} + /// The Crossbow on the Octane reports revision code 4. const XBOW_REVISION: u32 = 4; /// The Octane2's Crossbow: revision code 5, which IRIX's inventory names @@ -169,7 +190,7 @@ impl OctaneMachine { if prom.len() > FLASH_SIZE { return Err(format!("Octane PROM {prom_path} is {} bytes; the flash holds {FLASH_SIZE}", prom.len())); } - let cpu_count = cfg.machine.cpus; + let cpu_count = cfg.machine.cpus(); MachineProfile::OctaneIp30.accepts_cpus(cpu_count)?; let banks = heart_banks(cfg); let kind = cfg.graphics_board(); @@ -353,7 +374,7 @@ impl OctaneMachine { if has_vpro { kind.label() } else { board.map_or("no graphics", |b| b.label()) } ); - let count_hz = cfg.clock.fixed_mhz.map_or(OCTANE_COUNT_HZ, |mhz| (mhz * 1_000_000.0) as u64); + let count_hz = cfg.clock.fixed_mhz.map_or(octane_default_count_hz(cfg.machine.cpu), |mhz| (mhz * 1_000_000.0) as u64); let flash_lines: Vec<_> = (0..cpu_count) .map(|_| { let bus: Arc = phys.phys.clone(); @@ -365,7 +386,7 @@ impl OctaneMachine { ticks.extend((1..cpu_count).map(|_| Arc::new(AtomicU64::new(0)))); let memory = Arc::new(crate::ip30::memory_map(&cfg.banks[..4])); let cpus: Vec> = (0..cpu_count) - .map(|id| Self::build_cpu(id, cpu_count, memory.clone(), phys.phys.clone(), flash_lines[id as usize].clone(), &timer_manager, count_hz.max(1), ticks[id as usize].clone())) + .map(|id| Self::build_cpu_for(cfg.machine.cpu, id, cpu_count, memory.clone(), phys.phys.clone(), flash_lines[id as usize].clone(), &timer_manager, count_hz.max(1), ticks[id as usize].clone())) .collect(); let cpu = cpus[0].clone(); world.set_slots(cpus.iter().map(|c| c.park_slot()).collect()); @@ -575,8 +596,12 @@ impl OctaneMachine { } } + /// The configured part, as an executor over its cache model. The R10000 + /// takes the Octane's own revision (`OCTANE_PRID`) and the R14000 is the + /// R12000's 2 MB model with its PRId; the R12000 keeps its model's. #[allow(clippy::too_many_arguments)] - fn build_cpu( + fn build_cpu_for( + model: CpuModel, id: u8, cpu_count: u8, memory: Arc, @@ -586,25 +611,56 @@ impl OctaneMachine { count_hz: u64, fasttick: Arc, ) -> Arc { - let cfg = MipsCpuConfig::for_model::(memory); + macro_rules! build { + ($c:ty, $prid:expr) => { + Self::build_cpu::<$c>(id, cpu_count, memory, sysad, cached, tm, count_hz, fasttick, $prid) + }; + } + match model { + CpuModel::R12000 => build!(R12000ShadowCache, None), + CpuModel::R14000 => build!(R12000ShadowCache, Some(R14000_PRID)), + _ => build!(R10000ShadowCache, Some(OCTANE_PRID)), + } + } + + #[allow(clippy::too_many_arguments)] + fn build_cpu( + id: u8, + cpu_count: u8, + memory: Arc, + sysad: Arc, + cached: Arc, + tm: &Arc, + count_hz: u64, + fasttick: Arc, + prid: Option, + ) -> Arc + where + C: crate::cpu::mips_cache_v2::CpuModel + MipsCache + From> + 'static, + MipsCpu: CpuDevice, + { + let cfg = MipsCpuConfig::for_model::(memory); let tlb = MipsTlb::new(cfg.tlb_entries); - let mut exec: MipsExecutor = MipsExecutor::new(sysad, tlb, &cfg); + let mut exec: MipsExecutor = MipsExecutor::new(sysad, tlb, &cfg); // Cached accesses see the PROM's dirty flash lines; see `CachedFlashLines`. - exec.cache = R10000ShadowCache::from(cached as Arc); + exec.cache = C::from(cached as Arc); exec.core.set_timer_manager(tm.clone()); exec.core.set_count_hz(count_hz); exec.core.fasttick_count = fasttick; - exec.core.reset_prid = OCTANE_PRID; - exec.core.cp0_prid = OCTANE_PRID; + if let Some(prid) = prid { + exec.core.reset_prid = prid; + exec.core.cp0_prid = prid; + } // IRIX zeroes Count and arms Compare a few instructions later; see // `MipsCore::count_write_samples`. exec.core.count_write_samples = true; - // The R10000 drives 40 physical address bits; HEART decodes them - // all, and the attribute bits above fold away. + // The R10000 drives 40 physical address bits; HEART decodes them all. + // EntryLo bits 33:32 are physical 39:38; a guest that keeps software + // flags there clears them with FrameMask, so no narrowing is needed. exec.phys_mask = (1 << 40) - 1; exec.core.cpu_id = id; exec.core.smp = cpu_count > 1; - exec.batch_hook = Some(Box::new(move |exec: &mut MipsExecutor| { + exec.batch_hook = Some(Box::new(move |exec: &mut MipsExecutor| { LAST_PC[id as usize].store(exec.core.pc, Ordering::Relaxed); })); Arc::new(MipsCpu::new(exec)) @@ -1468,3 +1524,18 @@ mod snapshot_tests { }).unwrap().join().unwrap(); } } + +#[cfg(test)] +mod cpu_clock_tests { + use super::*; + + /// A plain `cpu = ...` reports a clock that part shipped at in an Octane: + /// 195 MHz for the R10000, 360 MHz for the R12000 (revision 3.5) and + /// 600 MHz for the R14000 (revision 2.4). Count runs at half the clock. + #[test] + fn each_octane_part_defaults_to_a_shipped_clock() { + for (cpu, mhz) in [(CpuModel::R10000, 195), (CpuModel::R12000, 360), (CpuModel::R14000, 600)] { + assert_eq!(octane_default_count_hz(cpu) * 2, mhz * 1_000_000, "{cpu:?}"); + } + } +} diff --git a/src/ip32_machine.rs b/src/ip32_machine.rs index a7a42a6d4..cc42d6ffb 100644 --- a/src/ip32_machine.rs +++ b/src/ip32_machine.rs @@ -27,7 +27,7 @@ use parking_lot::Mutex; use crate::config::{CpuModel, MachineConfig, MachineProfile}; use crate::cpu::mips_core::CyclesPtr; use crate::cpu::mips_exec::{CpuDevice, MipsCpu, MipsCpuConfig, MipsExecutor}; -use crate::cpu::mips_cache_shadow::{R10000O2ShadowCache, R5000ShadowCache}; +use crate::cpu::mips_cache_shadow::{R10000O2ShadowCache, R12000O2ShadowCache, R5000ShadowCache}; use crate::cpu::mips_cache_v2::MipsCache; use crate::cpu::mips_tlb::MipsTlb; use crate::dev::crime_mem::O2Memory; @@ -157,8 +157,23 @@ pub const O2_PROM_FILE: &str = "ip32prom.rev4.18.bin"; /// CP0 Count when `[clock] fixed_mhz` is unset: half of a 180 MHz CPU. pub const O2_COUNT_MHZ: f64 = 90.0; +/// CP0 Count for the O2's R12000 when `[clock] fixed_mhz` is unset: half of +/// 400 MHz, the speed O2 listings show for revision 3.5. +pub const O2_R12000_COUNT_MHZ: f64 = 200.0; + +/// CP0 Count for a part when `[clock] fixed_mhz` is unset. +fn o2_default_count_mhz(cpu: CpuModel) -> f64 { + match cpu { + CpuModel::R12000 => O2_R12000_COUNT_MHZ, + _ => O2_COUNT_MHZ, + } +} + impl O2Machine { pub fn new(cfg: &MachineConfig) -> Result { + if !matches!(cfg.machine.cpu, CpuModel::R5000 | CpuModel::R10000 | CpuModel::R12000) { + return Err("an O2 has an R5000, R10000 or R12000 ([machine] cpu = \"r5000\", \"r10000\" or \"r12000\")".into()); + } let o2 = &cfg.o2; // `prom`, or the last release's file when `prom` is the shared default // and that file is not there: there is no embedded O2 PROM. @@ -235,14 +250,15 @@ impl O2Machine { let arcs_fpu = o2.arcs_fpu; // `[clock] fixed_mhz` is CP0 Count; the CPU clock the PROM measures // and reports is twice it. - let count_mhz = cfg.clock.fixed_mhz.unwrap_or(O2_COUNT_MHZ); + let count_mhz = cfg.clock.fixed_mhz.unwrap_or(o2_default_count_mhz(cfg.machine.cpu)); let cpu_mhz = (count_mhz * 2.0).round().max(1.0) as u32; let boot = Arc::new(BootState::new()); let memory = Arc::new(crate::ip32::memory_map()); let cpu: Arc = match cfg.machine.cpu { CpuModel::R10000 => Self::build_cpu::(memory.clone(), sysad, &timer_manager, arcs_fpu, cpu_mhz, boot.clone(), bus.ram.clone()), CpuModel::R5000 => Self::build_cpu::(memory.clone(), sysad, &timer_manager, arcs_fpu, cpu_mhz, boot.clone(), bus.ram.clone()), - CpuModel::R4400 => return Err("an O2 has an R5000 or an R10000 ([machine] cpu)".into()), + CpuModel::R12000 => Self::build_cpu::(memory.clone(), sysad, &timer_manager, arcs_fpu, cpu_mhz, boot.clone(), bus.ram.clone()), + CpuModel::R4400 | CpuModel::R14000 | CpuModel::R16000 | CpuModel::R16000Mhz700 => unreachable!("refused at the top of `new`"), }; // ppmem: the CPU's cache and the JIT reach RAM through the memory's @@ -1192,3 +1208,19 @@ fn control_command(bus: &Ip32Bus, cmd: &str, arg: &str, rgba: &mut [u8]) -> Stri } } + +#[cfg(test)] +mod cpu_clock_tests { + use super::*; + + /// A plain `cpu = "r12000"` reports the 400 MHz the revision 3.5 part + /// shipped at; the others keep the 180 MHz default. Count runs at half + /// the clock. + #[test] + fn the_o2_r12000_defaults_to_400_mhz() { + assert_eq!(o2_default_count_mhz(CpuModel::R12000) * 2.0, 400.0); + for cpu in [CpuModel::R5000, CpuModel::R10000] { + assert_eq!(o2_default_count_mhz(cpu) * 2.0, 180.0, "{cpu:?}"); + } + } +} diff --git a/src/ip34_machine.rs b/src/ip34_machine.rs new file mode 100644 index 000000000..9bc0d061f --- /dev/null +++ b/src/ip34_machine.rs @@ -0,0 +1,832 @@ +//! The Fuel (IP34 board, IP35 software platform): an R14000 on Bedrock, +//! XBridge and the base I/O devices, with Iris's CPU thread, monitor and serial console. +//! +//! Selected with `[machine] profile = "fuel_ip34"`. Shared keys select the +//! PROM, NVRAM, memory and SCSI devices; `[fuel]` selects the serial console +//! and writable flash. `ip35::Ip35Bus` maps the processor views over the +//! same Physical memory banks. The IOC3 carries PS/2 keyboard and mouse ports +//! (default) and the OHCI controller can take a USB pair; without graphics the +//! machine runs on a serial console. With `[graphics] board = "v12"` (or +//! `"v10"`) a VPro board answers at crossbar widget 13, the Fuel's graphics +//! slot. +//! +//! During PROM cache-as-memory bootstrap, cached startup bytes live in a +//! Fuel cache adapter until a successful cached fetch leaves the PROM homes. +//! This retirement rule is an approximation, not an established hardware event. +//! +//! The Bedrock node itself, and its approximations, are in `ip35::node`. +//! +//! The processor interfaces' graphics flow-control registers store what +//! IRIX writes, which is what its VPro driver checks, but count no stores: +//! no uncached store into the graphics page takes a credit and the board +//! returns none, so the credit-starvation interrupt never fires. The board's +//! DMA reaches memory through Bedrock's inbound path as widget 13. +//! +//! Board composition ported from hw/mips/sgi/sn1-node.c +//! in jamesbraid/qemu-sgi-origami. + +use crate::config::{scsi_label, FuelConsole, GraphicsBoard, MachineConfig}; +use crate::cpu::mips_exec::{CpuDevice, MipsCpu, MipsCpuConfig, MipsExecutor}; +use crate::cpu::mips_tlb::MipsTlb; +use crate::dev::bedrock_flash::{Flash, FirmwareRecord, PART_SIZE}; +use crate::dev::bedrock_io::{BedrockIo, SIZE as IO_SIZE}; +use crate::dev::bedrock_lb::{BedrockLb, SIZE as LB_SIZE}; +use crate::dev::bedrock_md::{BedrockMd, MemoryError}; +use crate::dev::bedrock_ni::{BedrockNi, SIZE as NI_SIZE}; +use crate::dev::bedrock_pi::{BedrockPi, BedrockPiConfig, Population, SIZE as PI_SIZE}; +use crate::dev::bedrock_xb::{BedrockXb, SIZE as XB_SIZE}; +use crate::dev::heart::{host_clock, Clock}; +use crate::dev::ioc3::{Ioc3, Ioc3Config}; +use crate::dev::odyssey::{Odyssey, PacketSender, PiGfx, PI_GFX_REGS}; +use crate::dev::ioc3_eth::Ioc3Eth; +use crate::dev::isp12160::Isp12160; +use crate::dev::l1::{L1, L1Config, L1Event}; +use crate::dev::ohci::Ohci; +use crate::dev::usb_hid::{UsbKeyboard, UsbMouse}; +use crate::dev::prom_image; +use crate::dev::ps2::Ps2Controller; +use crate::dev::serial_host::{null_serial_backend, open_log, tcp_serial_backend, CiSerialBackend, LogMode, SerialBackend, TeeBackend}; +use crate::dev::timekeeper::Timekeeper; +use crate::dev::xbridge::{InterruptPacket, XBridge}; +use crate::devlog::{init_devlog, LogModule}; +use crate::gdb_stub::CpuDebug; +use crate::gfx_display::GfxDisplay; +use crate::hptimer::TimerManager; +use crate::cpu::mips_cache_shadow::R14000FuelShadowCache; +use crate::cpu::mips_cache_v2::MipsCache; +use crate::ip35::node::{connect_cpu, connect_serial_ports, Node, NodeConfig, SerialPort, WidgetDma}; +#[cfg(test)] +pub(crate) use crate::ip35::node::PendingIrq; +use crate::ip35::{Ip35Bus, CPU_PHYS_MASK}; +use crate::irq::{InterruptController, IrqLine}; +use crate::machine_state::GuestState; +use crate::monitor::{Monitor, DEFAULT_PORT as DEFAULT_MONITOR_PORT}; +use crate::net::{GatewayConfig, Link, NatControl}; +use crate::perf_monitor::PerfMonitor; +use crate::ppmem::devmap::SLOT_SHIFT; +use crate::scsi::{attach_from_config, ScsiBus}; +use crate::traits::{BusDevice, BusRead8, BusRead16, BusRead32, BusRead64, Device, MachineEvent, Resettable, Signal, BUS_ERR, BUS_OK}; +use crate::workstation::{register_system_controller, DiscChanger, MachineRef, MachineSlot, Workstation}; +use parking_lot::Mutex; +use std::net::{Ipv4Addr, SocketAddr}; +use std::path::{Path, PathBuf}; +use std::sync::mpsc::{channel, Receiver}; +use std::sync::{atomic::{AtomicBool, AtomicU64, Ordering}, Arc}; + +pub struct FuelMachine { + board: Arc, + cpu: Arc, + timers: Arc, + monitor: Arc, + monitor_port: u16, + ci_serial: Option>, + console: usize, + ci: bool, + scratch: Option, + timekeeper: Arc, + nvram: String, + frames: Option>, + nat: Option<( + Arc, + (Ipv4Addr, Ipv4Addr, Ipv4Addr), + )>, + events: Option>, + slot: Option, + lifecycle: GuestState, +} + +impl FuelMachine { + pub fn new(cfg: &MachineConfig) -> Result { + cfg.validate()?; + let prom = prom_image::load(Path::new(&cfg.prom), 8 * 1024 * 1024) + .map_err(|e| format!("Fuel {e}"))?; + let timekeeper = Arc::new(Timekeeper::fuel_with_nvram_file( + Path::new(&cfg.nvram), + )?); + let board = Arc::new(FuelBoard::with_config(FuelBoardConfig { + payload: &prom, + clock: host_clock(), + flash_path: cfg.fuel.flash.as_deref().map(Path::new), + station_mac: cfg.network().mac, + timekeeper: timekeeper.clone(), + ps2_input: cfg.fuel.ps2_input, + usb_input: cfg.fuel.usb_input, + l1: L1Identity { + revision: cfg.fuel.l1_revision.as_deref().map(crate::dev::l1::parse_revision).transpose()?, + brick_serial: cfg.fuel.brick_serial.clone(), + }, + graphics: cfg.graphics_board(), + })?); + let timers = Arc::new(TimerManager::new()); + let mut exec = board.build_cpu(); + // The CPU's Compare matches blink the graphics status bar's LED. + exec.core.fasttick_count = board.fasttick.clone(); + exec.core.set_timer_manager(timers.clone()); + exec.core.set_count_hz( + cfg.clock + .fixed_mhz + .map_or(250_000_000, |mhz| (mhz * 1_000_000.0) as u64) + .max(1), + ); + exec.core.count_write_samples = true; + let poll_board = board.clone(); + exec.batch_hook = Some(Box::new(move |_| poll_board.poll())); + let cpu: Arc = Arc::new(MipsCpu::<_, R14000FuelShadowCache>::new(*exec)); + board.status.set_cpu_cycles(cpu.cycles_ptr()); + board.status.set_count_hz(cpu.count_hz_atomic()); + let rx = connect_cpu("Fuel", &board.md, &board.pi[0], &board.lb, &board.ni, &board.l1, &cpu); + // After the windows above: each compile worker takes the cache's + // geometry and the CPU model's policies once, as it starts, and + // without the windows the geometry has no inline load or store path. + // Workers read plain physical code addresses through the view the CPU + // fetched each page by: the cached plane for cached pages, the + // uncached attribute-0 view for the reset PROM's uncached fetches. + // Decoding plain addresses in the uncached views for cached pages + // would reach Hub registers. + #[cfg(feature = "jitv2")] + { + let cpu_device: Arc = cpu.clone().as_device(); + let jit = cpu.jitv2(); + let mut g = jit.lock(); + g.compile_queue.set_thread_count(cfg.jitv2.threads.max(1)); + g.compile_queue.set_cpu(Arc::downgrade(&cpu_device)); + g.compile_queue.set_owner(Arc::downgrade(&jit)); + // Not under lockstep, as on the other machines: lockstep compiles + // inline on the CPU thread, and a running pool both owns the + // Codegen inline needs and deadlocks the CPU thread's flush. + #[cfg(not(feature = "jitv2_lockstep"))] + { + let stats = g.stats.clone(); + g.compile_queue.start(board.code_bus(), stats); + } + // The compile pool keeps only a weak handle on the CPU. + std::mem::forget(cpu_device); + } + let scratch = Self::attach_scsi(cfg, &board)?; + let console = match cfg.fuel.console { + FuelConsole::L1 => 0, + FuelConsole::Ioc3A => 1, + FuelConsole::Ioc3B => 2, + }; + let ci_serial = cfg.ci.then(|| board.serial[console].clone()); + Self::connect_serial(cfg, &board, console)?; + let monitor = Monitor::new(); + monitor.register_device(init_devlog()); + monitor.register_device(Arc::new(crate::cpu::cpu_monitor::CpuMonitor::new(vec![cpu.clone()]))); + monitor.register_device(PerfMonitor::new( + cpu.running_flag(), cpu.cycles_ptr(), cpu.fasttick_count(), None, None, + )); + monitor.register_device(board.md.physical().clone()); + monitor.register_device(board.l1.clone()); + monitor.register_device(board.scsi.clone()); + // `ps2 type` and friends: the keyboard and mouse from the monitor. + monitor.register_device(board.ioc3.ps2()); + cpu.register_locks(); + board.scsi.register_locks(); + board.ioc3.register_locks(); + board.xbridge.register_locks(); + board.flash.register_locks(); + board.md.register_locks(); + let (mut nat, mut frames) = (None, None); + let net = GatewayConfig::from_network(&cfg.network()); + if let Some(link) = Link::with_gateway(net.clone())? { + frames = Some(link.frames()); + nat = Some((link.nat(), (net.client_ip, net.gateway_ip, net.netmask))); + board.eth.attach(link); + } + eprintln!( + "iris: Fuel IP34, R14000 500 MHz, 512 MiB, {}, serial console {:?}", + if board.vpro.is_some() { cfg.graphics_board().label() } else { "no graphics" }, + cfg.fuel.console + ); + Ok(Self { + board, + cpu, + timers, + monitor: Arc::new(monitor), + monitor_port: cfg.monitor_port.unwrap_or(DEFAULT_MONITOR_PORT), + ci_serial, + console, + ci: cfg.ci, + scratch, + timekeeper, + nvram: cfg.nvram.clone(), + nat, + frames, + events: Some(rx), + slot: None, + lifecycle: GuestState::Idle, + }) + } + + fn attach_scsi(cfg: &MachineConfig, board: &FuelBoard) -> Result, String> { + let attached = attach_from_config(cfg, Some("fuel"), &|channel| { + board.scsi.bus_for_channel(channel) + .ok_or_else(|| format!("controller {channel} (a Fuel has channels 0 and 1)")) + })?; + if let Some(device) = attached.daynaports.first() { + return Err(format!("{}: kind = \"daynaport\" is not available on the Fuel", + scsi_label(device.controller, device.id))); + } + Ok(attached.scratch) + } + + fn connect_serial( + cfg: &MachineConfig, + board: &FuelBoard, + console: usize, + ) -> Result<(), String> { + let [a, b] = cfg.serial_ports(); + let ports = [ + SerialPort { + label: "L1", + port: cfg.fuel.l1_port.unwrap_or(8882), + setting: "fuel.l1_port", + ci: &board.serial[0], + log: cfg.fuel.l1_log.clone(), + connect: Box::new(|backend| board.l1.set_serial_backend(backend)), + }, + SerialPort { + label: "IOC3 A", + port: a, + setting: "serial_port_a", + ci: &board.serial[1], + log: None, + connect: Box::new(|backend| board.ioc3.set_serial_backend(0, backend)), + }, + SerialPort { + label: "IOC3 B", + port: b, + setting: "serial_port_b", + ci: &board.serial[2], + log: None, + connect: Box::new(|backend| board.ioc3.set_serial_backend(1, backend)), + }, + ]; + connect_serial_ports(cfg, "Fuel", &ports, console) + } + + #[cfg(test)] + pub(crate) fn connect_serial_for_test(cfg: &MachineConfig, board: &FuelBoard, console: usize) -> Result<(), String> { + Self::connect_serial(cfg, board, console) + } + +} +impl Workstation for FuelMachine { + fn start(&mut self) { + self.lifecycle = GuestState::Starting; + self.monitor + .clone() + .start_server(format!("127.0.0.1:{}", self.monitor_port)); + self.restart_peripherals(); + if !self.ci { + self.cpu_start(); + } + } + fn stop(&mut self) { + self.cpu.clone().as_device().stop(); + self.board.stop(); + if let Err(e) = self.timekeeper.flush_nvram() { + dlog!(LogModule::Bedrock, "Fuel NVRAM: {e}"); + } + if self.lifecycle != GuestState::Idle { + self.lifecycle = GuestState::Halted; + } + } + fn reset(&mut self) { + self.stop(); + self.cpu.power_on(); + for pi in &self.board.pi { + pi.power_on(); + } + self.board.md.power_on(); + self.board.lb.power_on(); + self.board.io.power_on(); + self.board.ni.power_on(); + self.board.xb.power_on(); + self.board.l1.power_on(); + self.board.flash.power_on(); + self.board.xbridge.power_on(); + self.board.ioc3.power_on(); + self.board.eth.power_on(); + self.board.scsi.power_on(); + self.board.usb.power_on(); + if let Some(vpro) = &self.board.vpro { + vpro.power_on(); + } + self.timekeeper.power_on(); + self.restart_peripherals(); + } + fn restart_peripherals(&mut self) { + self.board.start(); + } + fn cpu_start(&self) { + self.cpu.clone().as_device().start(); + } + fn cpu_is_running(&self) -> bool { + self.cpu.clone().as_device().is_running() + } + fn machine_slot(&mut self) -> MachineSlot { + let ptr = self as *mut Self as *mut dyn Workstation; + self.slot + .get_or_insert_with(|| Arc::new(Mutex::new(MachineRef(ptr)))) + .clone() + } + fn register_system_controller(&mut self) { + let slot = self.machine_slot(); + let events = self + .events + .take() + .expect("Fuel system controller registered twice"); + register_system_controller(&self.monitor, slot, Some(events)); + } + fn monitor_addr(&self) -> Option { + self.monitor.bound_addr() + } + fn lifecycle(&self) -> GuestState { + self.lifecycle + } + fn set_lifecycle(&mut self, state: GuestState) { + self.lifecycle = state; + } + fn get_display(&self) -> Option> { + self.board.vpro.as_ref().map(|vpro| vpro.display()) + } + fn register_monitor_device(&self, device: Arc) { + self.monitor.register_device(device); + } + fn get_ps2(&self) -> Arc { + self.board.ioc3.ps2() + } + fn get_timer_manager(&self) -> Arc { + self.timers.clone() + } + fn get_cpu_debug(&self) -> Arc { + self.cpu.clone().debug_adapter() + } + fn fasttick_count(&self) -> Arc { + self.cpu.fasttick_count() + } + fn get_ci_serial(&self) -> Option> { + self.ci_serial.clone() + } + fn inject_serial_console(&self, bytes: &[u8]) { + self.board.serial[self.console].push_host(bytes); + } + fn as_any_mut(&mut self) -> &mut dyn std::any::Any { + self + } + fn save_nvram(&self, path: Option<&str>) -> Result { + if path.is_some_and(|p| p != self.nvram) { + return Err("Fuel NVRAM save uses the configured path".into()); + } + self.timekeeper.flush_nvram()?; + Ok(self.nvram.clone()) + } + fn scratch_path(&self) -> Option<&Path> { + self.scratch.as_deref() + } + fn disc_changer(&self) -> Option> { + Some(self.board.scsi.clone() as Arc) + } + fn pending_chd_sync_count(&self) -> usize { + (0..2).map(|channel| self.board.scsi.bus_for_channel(channel).unwrap().pending_chd_sync_count()).sum() + } + fn flush_disks(&self) -> std::io::Result<()> { + self.board.flush_disks() + } + fn sync_chd_disks( + &self, + only: Option, + progress: &mut dyn FnMut(usize, usize, f32), + cancel: &dyn Fn() -> bool, + ) -> std::io::Result { + let buses: Vec<_> = (0..2).map(|channel| self.board.scsi.bus_for_channel(channel).unwrap()).collect(); + ScsiBus::sync_chd_disks(&buses, only, progress, cancel) + } + fn net_frames(&self) -> Option> { self.frames.clone() } + fn nat_control(&self) -> Option> { + self.nat.as_ref().map(|(n, _)| n.clone()) + } + fn nat_expected(&self) -> Option<(Ipv4Addr, Ipv4Addr, Ipv4Addr)> { + self.nat.as_ref().map(|(_, a)| *a) + } +} + +impl Drop for FuelMachine { + fn drop(&mut self) { + if let Some(slot) = self.slot.take() { + slot.lock().0 = MachineRef::null(); + } + self.stop(); + self.monitor.shutdown(); + } +} + +pub(crate) type FuelCpu = MipsExecutor; + +const FUEL_BOARD_ID: u64 = 0x4004; +/// Two 256 MiB banks, locational banks 0 and 1. +const FUEL_BANKS_MIB: [u32; 2] = [256, 256]; + +/// Select the named R14000 model and the Fuel PROM boot-mode inputs. +pub(crate) fn build_cpu(bus: Arc) -> Box { + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip35::memory_map(&crate::dev::bedrock_mem::RamExtent::new(&FUEL_BANKS_MIB)))); + let mut cpu = Box::new(MipsExecutor::new( + bus as Arc, + MipsTlb::new(cfg.tlb_entries), + &cfg, + )); + cpu.phys_mask = CPU_PHYS_MASK; + // Fuel's 500 MHz/2 MiB flash record selects these boot-mode inputs. + // A mismatch enters the PROM's error routine before GP is initialized. + cpu.core.cp0_config = FirmwareRecord::FUEL.reset_config0(); + cpu +} + +/// Shared device composition for a single-processor Fuel. +/// Graphics is absent; the serial workstation shares these base I/O devices. +pub(crate) struct FuelBoard { + pub bus: Arc, + pub pi: [Arc; 2], + pub md: Arc, + pub l1: Arc, + pub flash: Arc, + pub lb: Arc, + pub io: Arc, + pub ni: Arc, + pub xb: Arc, + pub xbridge: Arc, + pub ioc3: Arc, + pub eth: Arc, + pub scsi: Arc, + pub usb: Arc, + /// The VPro board in the graphics slot, widget 13. + pub vpro: Option>, + /// The status bar's activity bits and the CPU's Compare matches, which + /// the graphics board shows. + pub heartbeat: Arc, + pub fasttick: Arc, + /// What the graphics board's status bar shows: the activity bits, the + /// Compare matches and the CPU's cycle count and Count rate. + pub status: Arc, + pub serial: [Arc; 3], + #[cfg(test)] + pub irq: Arc, + clock: Clock, + last_clock: AtomicU64, + running: AtomicBool, +} + +pub(crate) struct FuelBoardConfig<'a> { + pub payload: &'a [u8], + pub clock: Clock, + pub flash_path: Option<&'a Path>, + pub station_mac: [u8; 6], + pub timekeeper: Arc, + /// The IOC3's PS/2 keyboard and mouse controller. + pub ps2_input: bool, + /// A USB keyboard and mouse on the OHCI root ports. + pub usb_input: bool, + /// Replacements for the L1's reported identity; unset keeps its defaults. + pub l1: L1Identity, + /// `None`, `V12` or `V10`; validation refuses the others. + pub graphics: GraphicsBoard, +} + +/// What a Fuel config may replace in the L1's identity replies. +#[derive(Clone, Debug, Default)] +pub(crate) struct L1Identity { + pub revision: Option<[u32; 3]>, + pub brick_serial: Option, +} + +impl FuelBoard { + /// Flush the disks on both channels of the storage controller. + pub fn flush_disks(&self) -> std::io::Result<()> { + (0..2).try_for_each(|channel| self.scsi.bus_for_channel(channel).unwrap().flush()) + } + + #[cfg(test)] + pub fn new(payload: &[u8], clock: Clock) -> Result { + Self::with_config(FuelBoardConfig { + payload, clock, flash_path: None, + station_mac: L1Config::default().mac, + timekeeper: Arc::new(Timekeeper::fuel()?), + ps2_input: true, + usb_input: false, + l1: L1Identity::default(), + graphics: GraphicsBoard::None, + }) + } + + pub fn with_config(config: FuelBoardConfig<'_>) -> Result { + let FuelBoardConfig { payload, clock, flash_path, station_mac, timekeeper, ps2_input, usb_input, l1: l1_identity, graphics } = config; + let mut l1_config = L1Config { mac: station_mac, ..Default::default() }; + if let Some(revision) = l1_identity.revision { + l1_config.revision = Some(revision); + } + if let Some(serial) = l1_identity.brick_serial { + l1_config.brick_serial = Some(serial); + } + let node = Node::build(NodeConfig { + name: "Fuel", + payload, + clock: clock.clone(), + flash_path, + banks_mib: &FUEL_BANKS_MIB, + population: Population::SINGLE, + running: Population::SINGLE, + firmware: FirmwareRecord::FUEL, + flash_records: Vec::new(), + board_id: FUEL_BOARD_ID, + l1: l1_config, + })?; + let (md, io) = (&node.md, &node.io); + let serial = std::array::from_fn(|_| Arc::new(CiSerialBackend::new())); + node.l1.set_serial_backend(serial[0].clone()); + let xbridge = XBridge::new(md.dma_view()); + for widget in [0, 10, 14, 15] { + io.connect_widget(widget, xbridge.widget(widget).unwrap()); + } + let weak_io = Arc::downgrade(io); + xbridge.set_interrupt_sender(Arc::new(move |packet| { + if packet.widget != 10 { + return false; + } + weak_io.upgrade().is_some_and(|io| { + io.inbound_write( + packet.source_widget, + None, + packet.address, + 8, + packet.vector as u64 | if packet.level { 1 << 8 } else { 0 }, + ) == BUS_OK + }) + })); + let weak_io = Arc::downgrade(io); + xbridge.set_dma_sender(Arc::new(move |packet, bytes| { + if packet.widget != 10 || packet.half_read_requested || packet.half_write_requested { + dlog!(LogModule::Bedrock, + "Fuel: unsupported XBridge DMA destination or halfword placement {packet:?}" + ); + return false; + } + let Some(io) = weak_io.upgrade() else { + return false; + }; + // Preserve the width of XBridge's naturally aligned packet legs, + // including Hub registers whose narrower accesses are refused. + let size = bytes.len(); + if matches!(size, 1 | 2 | 4 | 8) + && packet.address & (size as u64 - 1) == 0 + { + if packet.write { + let value = bytes.iter().fold(0u64, |value, byte| (value << 8) | *byte as u64); + return io.inbound_write( + packet.source_widget, Some(packet.slot), packet.address, size as u8, value, + ) == BUS_OK; + } + let value = io.inbound_read( + packet.source_widget, Some(packet.slot), packet.address, size as u8, + ); + if value.status != BUS_OK { + return false; + } + bytes.copy_from_slice(&value.data.to_be_bytes()[8 - size..]); + return true; + } + for (i, byte) in bytes.iter_mut().enumerate() { + let a = packet.address + i as u64; + if packet.write { + if io.inbound_write(packet.source_widget, Some(packet.slot), a, 1, *byte as u64) + != BUS_OK + { + return false; + } + } else { + let value = io.inbound_read(packet.source_widget, Some(packet.slot), a, 1); + if value.status != BUS_OK { + return false; + } + *byte = value.data as u8; + } + } + true + })); + let mac = node.l1.station_mac(); + let mut ioc3_config = Ioc3Config::fuel(mac); + ioc3_config.keyboard_mouse = ps2_input; + let ioc3 = Arc::new(Ioc3::new(ioc3_config)); + let eth = Ioc3Eth::new_fuel(mac); + ioc3.attach_eth(eth.clone()); + ioc3.attach_timekeeper(timekeeper); + ioc3.set_serial_backend(0, serial[1].clone()); + ioc3.set_serial_backend(1, serial[2].clone()); + let scsi = Arc::new(Isp12160::new( + Isp12160::new_bus(), + Isp12160::new_bus(), + )); + let heartbeat = Arc::new(AtomicU64::new(0)); + let fasttick = Arc::new(AtomicU64::new(0)); + scsi.set_heartbeat(heartbeat.clone()); + ioc3.set_heartbeat(heartbeat.clone()); + eth.set_heartbeat(heartbeat.clone()); + let status = crate::disp::StatusSource::new(heartbeat.clone(), fasttick.clone()); + let vpro = if graphics == GraphicsBoard::None { + None + } else { + // Interrupt packets leave through the crossbar as the heads' do; + // DMA enters Bedrock as this widget's inbound traffic. + let crossbar = Arc::downgrade(&xbridge); + let packets: PacketSender = Arc::new(move |widget, address, vector, level| { + crossbar.upgrade().is_some_and(|x| x.send_interrupt(InterruptPacket { + source_widget: WIDGET_VPRO, + widget, + address, + vector, + level, + })) + }); + let dma = Arc::new(WidgetDma { io: Arc::downgrade(&io), widget: WIDGET_VPRO }); + let board = Odyssey::new_widget_relative( + WIDGET_VPRO, graphics.as_str(), packets, dma, status.clone(), + )?; + xbridge.attach_widget(WIDGET_VPRO, board.clone()); + io.connect_widget(WIDGET_VPRO, xbridge.widget(WIDGET_VPRO).expect("attached widget")); + Some(board) + }; + // Each processor interface keeps its own graphics registers. + for pi in &node.pi { + pi.set_node_hooks(Arc::new(PiGraphics::default())); + } + xbridge.attach(15, 1, scsi.clone()); + xbridge.attach(15, 4, ioc3.clone()); + let usb = Ohci::new(); + usb.set_clock(clock.clone()); + if usb_input { + // Host input reaches the USB devices through the same controller + // object the GUI, VNC and CI already drive; each device drops + // what its guest has not enabled. + let keyboard = UsbKeyboard::new(); + let mouse = UsbMouse::new(); + usb.attach(0, keyboard.clone())?; + usb.attach(1, mouse.clone())?; + ioc3.ps2().add_sink(keyboard); + ioc3.ps2().add_sink(mouse); + } + xbridge.attach(15, 5, usb.clone()); + for slot in [1, 4, 5] { + xbridge.set_pci_irq(15, slot, 0, xbridge.pin_line(15, slot)); + } + ioc3.connect_superio_interrupt(xbridge.pin_line(15, 0)); + let reset_ioc3 = ioc3.clone(); + let reset_eth = eth.clone(); + let reset_scsi = scsi.clone(); + let reset_usb = usb.clone(); + let reset_vpro = vpro.as_ref().map(Arc::downgrade); + xbridge.set_reset_output(Arc::new(move |port| { + if port == 15 { + reset_ioc3.power_on(); + reset_eth.power_on(); + reset_scsi.power_on(); + reset_usb.power_on(); + } + // A link reset of the graphics port resets the board behind it. + if port == WIDGET_VPRO { + if let Some(vpro) = reset_vpro.as_ref().and_then(|v| v.upgrade()) { + vpro.power_on(); + } + } + })); + Ok(Self { + bus: node.bus, + pi: node.pi, + md: node.md, + l1: node.l1, + flash: node.flash, + lb: node.lb, + io: node.io, + ni: node.ni, + xb: node.xb, + xbridge, + ioc3, + eth, + scsi, + usb, + vpro, + heartbeat, + fasttick, + status, + serial, + #[cfg(test)] + irq: node.irq, + last_clock: AtomicU64::new(clock()), + clock, + running: AtomicBool::new(false), + }) + } + + pub fn cached_bus(&self) -> Arc { + self.bus.cached_view(self.md.dma_view(), self.md.ram_extent()) + } + + /// The bus the JIT compiles code through; see [`Ip35Bus::code_view`]. + #[cfg(feature = "jitv2")] + pub fn code_bus(&self) -> Arc { + self.bus.code_view(self.cached_bus()) + } + + pub fn build_cpu(&self) -> Box { + let mut cpu = build_cpu(self.bus.clone()); + cpu.cache = R14000FuelShadowCache::new(self.cached_bus()); + // The PROM fetches its own code uncached; with memory over its + // window the cached and uncached views of it differ. + #[cfg(feature = "jitv2")] + cpu.enable_jitv2_uncached_views(self.code_bus()); + // The PROM keeps its stack in cache lines it tags for a window with no + // memory behind it yet; the controller answers that window while they stand. + let memory = self.md.clone(); + cpu.cache.set_pd_tag_store_hook(Box::new(move |tag| memory.startup_tag_store(tag))); + cpu + } + + pub fn start(&self) { + self.last_clock + .store((self.clock)(), Ordering::Release); + self.md.start_clock(); + self.running + .store(true, Ordering::Release); + for pi in &self.pi { + pi.start(); + } + self.l1.start(); + self.ioc3.start(); + self.eth.start(); + self.scsi.start(); + self.usb.start(); + if let Some(vpro) = &self.vpro { + vpro.start(); + } + } + pub fn poll(&self) { + if self.running.load(Ordering::Acquire) { + let now = (self.clock)(); + let previous = self + .last_clock + .swap(now, Ordering::AcqRel); + self.flash.advance_ns(now.saturating_sub(previous)); + self.usb.sync_clock(); + for pi in &self.pi { + pi.poll(); + } + } + } + pub fn stop(&self) { + self.poll(); + self.running + .store(false, Ordering::Release); + self.md.stop_clock(); + if let Some(vpro) = &self.vpro { + vpro.stop(); + } + self.usb.stop(); + self.scsi.stop(); + self.eth.stop(); + self.ioc3.stop(); + self.l1.stop(); + for pi in &self.pi { + pi.stop(); + } + } +} + +/// The crossbar widget of the Fuel's graphics slot: a Fuel IRIX inventory +/// lists its VPro under xtalk/13, and OpenBSD finds Odyssey at widget 13. +pub const WIDGET_VPRO: u8 = 13; + +/// One processor interface's graphics flow-control registers (0x300 to +/// 0x348): stored as written, with no store counting (see the module notes). +#[derive(Default)] +struct PiGraphics(Mutex); +impl crate::dev::bedrock_pi::NodeHooks for PiGraphics { + fn gfx_read(&self, index: usize) -> Option { + (index < PI_GFX_REGS as usize).then(|| self.0.lock().read(index as u32)) + } + fn gfx_write(&self, index: usize, value: u64) -> bool { + if index >= PI_GFX_REGS as usize { + return false; + } + self.0.lock().write(index as u32, value); + true + } + fn gfx_reset(&self) { + *self.0.lock() = PiGfx::default(); + } +} + +impl Drop for FuelBoard { + fn drop(&mut self) { + self.stop(); + } +} diff --git a/src/ip35.rs b/src/ip35.rs new file mode 100644 index 000000000..c5cdbd69e --- /dev/null +++ b/src/ip35.rs @@ -0,0 +1,441 @@ +//! The IP35 processor bus: Bedrock's four XKPHYS access-space views. +//! +//! Raw address bits 58:57 select Hub-special, I/O, fetch operations or uncached +//! RAM. Each view has a small fixed set of MMIO apertures, decoded by sorted +//! windows rather than a full physical-address table. Maps replace whole 4 KiB +//! pages as Physical does, while each endpoint still rejects padding and +//! enforces its cell permission. +//! +//! Cached RAM alone uses Physical, including its direct bank pointers, ppmem +//! and generation mappings. The MMIO views and register sub-bus need none of +//! those RAM facilities. Giving each its own full table would consume several +//! entries in the process-wide fault-handler registry per machine. +//! +//! Ported from hw/mips/sgi/sn1-node.c in jamesbraid/qemu-sgi-origami. + +use crate::dev::bedrock_mem::RamExtent; +use crate::physical::Physical; +use crate::traits::*; +#[cfg(feature = "jitv2")] +use crate::cpu::jitv2::jitv2::UNCACHED_VIEW_ADDR; +#[cfg(feature = "jitv2")] +use std::sync::{atomic::AtomicU64, OnceLock}; +use std::sync::Arc; + +pub(crate) const CPU_PHYS_MASK: u64 = ((1u64 << 40) - 1) | (3u64 << 57); +const PA_MASK: u64 = (1u64 << 40) - 1; + +/// Where a Bedrock node has memory: each 1 GiB software slot `extent` says +/// can hold RAM, as far as its largest bank reaches, and the reset PROM. +/// Which bank lands in which slot of a group of four depends on how the PROM +/// programs the memory controller; the Fuel's two 256 MiB banks can be in any +/// of the first four. Uncached fetches of the PROM's own code land in its +/// range. A slot 0 larger than 508 MiB reaches the PROM's range, which the +/// node maps over it, so its RAM is listed on either side. +pub fn memory_map(extent: &RamExtent) -> crate::memory_map::MemoryMap { + use crate::dev::bedrock_mem::{BANK_SPAN, SLOTS}; + use crate::memory_map::{MemKind, MemRange, MemoryMap}; + const NAMES: [&str; SLOTS] = [ + "ram slot 0", "ram slot 1", "ram slot 2", "ram slot 3", + "ram slot 4", "ram slot 5", "ram slot 6", "ram slot 7", + ]; + const PROM: (u64, u64) = (0x1fc0_0000, 0x2000_0000); + let mut ranges = vec![MemRange::new("prom", MemKind::Rom, PROM.0, PROM.1)]; + for (slot, &name) in NAMES.iter().enumerate() { + let (base, end) = (slot as u64 * BANK_SPAN, slot as u64 * BANK_SPAN + extent.prefix(slot)); + if base == end { + continue; + } + if base < PROM.1 && end > PROM.0 { + ranges.push(MemRange::new(name, MemKind::Ram, base, PROM.0)); + if end > PROM.1 { + ranges.push(MemRange::new("ram slot 0 above the prom", MemKind::Ram, PROM.1, end)); + } + } else { + ranges.push(MemRange::new(name, MemKind::Ram, base, end)); + } + } + MemoryMap::new(ranges) +} + +struct Unclaimed; +impl BusDevice for Unclaimed {} + +/// An exact-bounds chip-relative window, optionally restricted by a cell permission. +/// Physical rounds maps to pages, so the adapter also refuses their padding. +pub(crate) struct Window { + base: u64, + size: u64, + device: Arc, + permission: Option bool + Send + Sync>>, +} +impl Window { + pub(crate) fn new(base: u64, size: u64, device: Arc) -> Self { + Self { base, size, device, permission: None } + } + pub(crate) fn permitted(mut self, permission: Arc bool + Send + Sync>) -> Self { + self.permission = Some(permission); + self + } + pub(crate) fn contains(&self, a: u64, n: u64) -> bool { self.local(a, n).is_some() } + fn local(&self, a: u64, n: u64) -> Option { + let local = a.checked_sub(self.base)?; + (local.checked_add(n)? <= self.size + && self.permission.as_ref().is_none_or(|allowed| allowed())) + .then_some(local) + } + fn write_masked_bytes(&self, a: u64, val: u64, mask: u64) -> u32 { + for i in 0..8usize { + let shift = (7 - i) * 8; + if (mask >> shift) & 0xff == 0 { + continue; + } + let Some(local) = self.local(a + i as u64, 1) else { + return BUS_ERR; + }; + let status = self.device.write8(local, (val >> shift) as u8); + if status != BUS_OK { + return status; + } + } + BUS_OK + } +} +macro_rules! window_access { + ($r:ident,$w:ident,$t:ty,$out:ident,$n:expr) => { + fn $r(&self, a: u64) -> $out { + self.local(a, $n).map(|at| self.device.$r(at)).unwrap_or_else($out::err) + } + fn $w(&self, a: u64, v: $t) -> u32 { + self.local(a, $n).map(|at| self.device.$w(at, v)).unwrap_or(BUS_ERR) + } + }; +} +impl BusDevice for Window { + #[cfg(feature = "jitv2")] + fn gen_ptr(&self, address: u64) -> *const AtomicU64 { + self.local(address, 1).map(|local| self.device.gen_ptr(local)).unwrap_or(std::ptr::null()) + } + fn write64_masked(&self, a: u64, val: u64, mask: u64) -> u32 { + if a & 7 != 0 { + return BUS_ERR; + } + if mask == 0 { + return BUS_OK; + } + if let Some(local) = self.local(a, 8) { + self.device.write64_masked(local, val, mask) + } else { + self.write_masked_bytes(a, val, mask) + } + } + fn write_narrow_from(&self, a: u64, size: usize, gpr: u64) -> u32 { + self.local(a, size as u64) + .map(|local| self.device.write_narrow_from(local, size, gpr)) + .unwrap_or(BUS_ERR) + } + window_access!(read8, write8, u8, BusRead8, 1); + window_access!(read16, write16, u16, BusRead16, 2); + window_access!(read32, write32, u32, BusRead32, 4); + window_access!(read64, write64, u64, BusRead64, 8); +} +pub(crate) fn map_device( + view: &Physical, + base: u64, + size: u64, + device: Arc, + name: &str, +) { + assert!(size > 0 && base.checked_add(size).is_some_and(|end| end <= PA_MASK + 1)); + view.map(base, base + size, Arc::new(Window::new(base, size, device)), name); +} + +struct DecodeRange { + start: u64, + end: u64, + window: Arc, +} + +/// Small MMIO maps have no RAM, direct pointers or host fault-handler entries. +/// Last map wins over whole pages, including padding rejected by its Window. +#[derive(Default)] +pub(crate) struct SparseWindowMap { + ranges: Vec, +} +impl SparseWindowMap { + pub(crate) fn map(&mut self, base: u64, size: u64, device: Arc) { + const PAGE: u64 = 1 << crate::ppmem::devmap::SLOT_SHIFT; + let end = base.checked_add(size).filter(|&end| size != 0 && end <= PA_MASK + 1) + .expect("IP35 window exceeds physical address space"); + let start = base & !(PAGE - 1); + let end = end.next_multiple_of(PAGE); + let window = Arc::new(Window::new(base, size, device)); + let mut ranges = Vec::with_capacity(self.ranges.len() + 2); + for old in self.ranges.drain(..) { + if old.end <= start || old.start >= end { + ranges.push(old); + } else { + if old.start < start { + ranges.push(DecodeRange { start: old.start, end: start, window: old.window.clone() }); + } + if old.end > end { + ranges.push(DecodeRange { start: end, end: old.end, window: old.window }); + } + } + } + ranges.push(DecodeRange { start, end, window }); + ranges.sort_unstable_by_key(|range| range.start); + self.ranges = ranges; + } + fn target(&self, address: u64) -> Option<&Window> { + let i = self.ranges.partition_point(|range| range.start <= address).checked_sub(1)?; + let range = &self.ranges[i]; + (address < range.end).then_some(range.window.as_ref()) + } +} +macro_rules! sparse_access { + ($read:ident,$write:ident,$ty:ty,$result:ident) => { + fn $read(&self, address: u64) -> $result { + self.target(address).map(|window| window.$read(address)).unwrap_or_else($result::err) + } + fn $write(&self, address: u64, value: $ty) -> u32 { + self.target(address).map(|window| window.$write(address, value)).unwrap_or(BUS_ERR) + } + }; +} +impl BusDevice for SparseWindowMap { + #[cfg(feature = "jitv2")] + fn gen_ptr(&self, address: u64) -> *const AtomicU64 { + self.target(address).map(|window| window.gen_ptr(address)).unwrap_or(std::ptr::null()) + } + fn write64_masked(&self, address: u64, value: u64, mask: u64) -> u32 { + self.target(address).map(|window| window.write64_masked(address, value, mask)) + .unwrap_or_else(|| Unclaimed.write64_masked(address, value, mask)) + } + fn write_narrow_from(&self, address: u64, size: usize, gpr: u64) -> u32 { + self.target(address).map(|window| window.write_narrow_from(address, size, gpr)) + .unwrap_or(BUS_ERR) + } + sparse_access!(read8, write8, u8, BusRead8); + sparse_access!(read16, write16, u16, BusRead16); + sparse_access!(read32, write32, u32, BusRead32); + sparse_access!(read64, write64, u64, BusRead64); +} + +/// The four uncached processor views decode MMIO separately from cached RAM. +pub struct Ip35Bus { + pub phys: Arc, + views: [SparseWindowMap; 4], + #[cfg(feature = "jitv2")] + ram_generations: OnceLock>, +} +impl Ip35Bus { + pub fn new(phys: Arc) -> Self { + phys.set_default(Arc::new(Unclaimed), "IP35 unclaimed cached address"); + Self { + phys, + views: std::array::from_fn(|_| SparseWindowMap::default()), + #[cfg(feature = "jitv2")] + ram_generations: OnceLock::new(), + } + } + /// Code pages use stable logical RAM generations across bank placement and + /// diagnostic remaps; non-RAM pages keep their own device generations. + #[cfg(feature = "jitv2")] + pub(crate) fn set_ram_generations(&self, memory: Arc) { + assert!(self.ram_generations.set(memory).is_ok(), "IP35 RAM generations already wired"); + } + /// Raw XKPHYS bits 58:57 select Hub-special, I/O, fetchop, or uncached RAM. + pub fn map(&mut self, raw_uc: u8, base: u64, size: u64, device: Arc) { + assert!(raw_uc < 4); + self.views[raw_uc as usize].map(base, size, device); + } + pub fn set_fallback(&mut self, raw_uc: u8, end: u64, device: Arc) { + self.map(raw_uc, 0, end, device); + } + fn target(&self, a: u64, n: u64) -> Option<(&SparseWindowMap, u64)> { + let pa = plain_address(a, n)?; + Some((&self.views[(a >> 57) as usize], pa)) + } + /// The bus the JIT compiles code through: a page the CPU fetched cached + /// reads the cached view, one it fetched uncached reads the uncached view + /// the CPU's own fetch used (`UNCACHED_VIEW_ADDR`). Bedrock decodes the + /// two differently at the reset PROM once memory covers it. + #[cfg(feature = "jitv2")] + pub fn code_view(self: &Arc, cached: Arc) -> Arc { + Arc::new(CodeViews { cached, uncached: self.clone() }) + } + /// Cached RAM, with `plain_memory` answering wherever `extent` says + /// fitted RAM can be. + pub fn cached_view(&self, plain_memory: Arc, extent: RamExtent) -> Arc { + Arc::new(CachedPhysical { + phys: self.phys.clone(), + plain_memory, + extent, + }) + } +} +fn plain_address(a: u64, n: u64) -> Option { + if a & !CPU_PHYS_MASK != 0 { + return None; + } + let pa = a & PA_MASK; + (pa.checked_add(n)? <= PA_MASK + 1).then_some(pa) +} +/// See [`Ip35Bus::code_view`]. Compiles read instruction words only. +#[cfg(feature = "jitv2")] +struct CodeViews { + cached: Arc, + uncached: Arc, +} +#[cfg(feature = "jitv2")] +impl BusDevice for CodeViews { + fn read32(&self, a: u64) -> BusRead32 { + if a & UNCACHED_VIEW_ADDR != 0 { + self.uncached.read32(a & !UNCACHED_VIEW_ADDR) + } else { + self.cached.read32(a) + } + } + fn gen_ptr(&self, a: u64) -> *const AtomicU64 { + if a & UNCACHED_VIEW_ADDR != 0 { self.uncached.gen_ptr(a) } else { self.cached.gen_ptr(a) } + } +} +struct CachedPhysical { + phys: Arc, + plain_memory: Arc, + extent: RamExtent, +} +impl CachedPhysical { + fn memory_target(&self, a: u64, n: u64) -> Option<(&dyn BusDevice, u64)> { + self.target(a, n).map(|(physical, pa)| { + // MD owns every guest write to a possible fitted-RAM prefix, so + // all paths update its stable logical generation counters. PROM + // and other devices retain the cached Physical connection. + let memory: &dyn BusDevice = if self.extent.contains(pa) { + self.plain_memory.as_ref() + } else { physical }; + (memory, pa) + }) + } + fn target(&self, a: u64, n: u64) -> Option<(&Physical, u64)> { + // UC selectors belong to the uncached SysAd connection. Reject them + // here rather than turning a malformed cached request into an alias. + if a & !PA_MASK != 0 || a & (n - 1) != 0 { + return None; + } + Some((self.phys.as_ref(), plain_address(a, n)?)) + } +} +macro_rules! processor_access { + ($r:ident,$w:ident,$t:ty,$out:ident,$n:expr,$writes:ident) => { + fn $r(&self, a: u64) -> $out { + self + .target(a, $n) + .map(|(v, pa)| v.$r(pa)) + .unwrap_or_else($out::err) + } + fn $w(&self, a: u64, v: $t) -> u32 { + self + .$writes(a, $n) + .map(|(view, pa)| view.$w(pa, v)) + .unwrap_or(BUS_ERR) + } + }; +} +impl BusDevice for Ip35Bus { + #[cfg(feature = "jitv2")] + fn gen_ptr(&self, address: u64) -> *const AtomicU64 { + if address & UNCACHED_VIEW_ADDR != 0 { + // A page fetched uncached: its bytes come from the uncached view + // at that address (attribute 0, the reset PROM and node memory), + // so its generation is that device's. Writes through the cached + // view change other bytes where the views differ, and bump the + // same RAM counter where they coincide. + // The PROM view's counter covers the whole flash, so any flash + // write invalidates every PROM page (over-invalidation, safe). + let pa = address & !UNCACHED_VIEW_ADDR; + return if pa & !PA_MASK == 0 { self.views[0].gen_ptr(pa) } else { std::ptr::null() }; + } + if address & !PA_MASK == 0 { + // JIT page lookup passes plain physical code addresses, without + // a cache attribute. Raw UC0 therefore cannot select the MMIO map + // here: high cached RAM and the reset PROM use the cached plane. + if let Some(memory) = self.ram_generations.get() { + let generation = memory.gen_ptr(address); + if !generation.is_null() { return generation; } + } + return self.phys.gen_ptr(address); + } + self.target(address, 1).map(|(view, pa)| view.gen_ptr(pa)).unwrap_or(std::ptr::null()) + } + fn write64_masked(&self, a: u64, val: u64, mask: u64) -> u32 { + if a & 7 != 0 { + return BUS_ERR; + } + self.target(a, 8) + .map(|(view, pa)| view.write64_masked(pa, val, mask)) + .unwrap_or(BUS_ERR) + } + fn write_narrow_from(&self, a: u64, size: usize, gpr: u64) -> u32 { + self.target(a, size as u64) + .map(|(view, pa)| view.write_narrow_from(pa, size, gpr)) + .unwrap_or(BUS_ERR) + } + processor_access!(read8, write8, u8, BusRead8, 1, target); + processor_access!(read16, write16, u16, BusRead16, 2, target); + processor_access!(read32, write32, u32, BusRead32, 4, target); + processor_access!(read64, write64, u64, BusRead64, 8, target); +} +impl BusDevice for CachedPhysical { + fn write64_masked(&self, address: u64, value: u64, mask: u64) -> u32 { + self.memory_target(address, 8).map(|(view, pa)| view.write64_masked(pa, value, mask)) + .unwrap_or(BUS_ERR) + } + fn mem_ptr(&self, a: u64) -> Option<*const u64> { + self.target(a, 8).and_then(|(view, pa)| view.mem_ptr(pa)) + } + #[cfg(feature = "jitv2")] + fn gen_ptr(&self, a: u64) -> *const AtomicU64 { + self.target(a, 1).map(|(view, pa)| { + let generation = self.plain_memory.gen_ptr(pa); + if generation.is_null() { view.gen_ptr(pa) } else { generation } + }).unwrap_or(std::ptr::null()) + } + fn read_block(&self, a: u64, out: &mut [u64]) -> u32 { + self.target(a, 8) + .map(|(view, pa)| view.read_block(pa, out)).unwrap_or(BUS_ERR) + } + fn write_block(&self, a: u64, data: &[u64]) -> u32 { + self.memory_target(a, 8) + .map(|(view, pa)| view.write_block(pa, data)).unwrap_or(BUS_ERR) + } + // A processor's SC to RAM the shadow cache cannot reach directly, such + // as banks above the direct window: the swap the store makes atomic. + fn cas32(&self, a: u64, old: u32, new: u32) -> Option { + self.memory_target(a, 4).and_then(|(view, pa)| view.cas32(pa, old, new)) + } + fn cas64(&self, a: u64, old: u64, new: u64) -> Option { + self.memory_target(a, 8).and_then(|(view, pa)| view.cas64(pa, old, new)) + } + processor_access!(read8, write8, u8, BusRead8, 1, memory_target); + processor_access!(read16, write16, u16, BusRead16, 2, memory_target); + processor_access!(read32, write32, u32, BusRead32, 4, memory_target); + processor_access!(read64, write64, u64, BusRead64, 8, memory_target); + fn dma_read64(&self, a: u64) -> BusRead64 { + plain_address(a, 8) + .map(|pa| self.plain_memory.dma_read64(pa)) + .unwrap_or_else(BusRead64::err) + } + fn dma_write64(&self, a: u64, v: u64) -> u32 { + plain_address(a, 8) + .map(|pa| self.plain_memory.dma_write64(pa, v)) + .unwrap_or(BUS_ERR) + } +} + +pub(crate) mod node; +pub(crate) mod rendezvous; + +#[cfg(test)] +mod tests; diff --git a/src/ip35/node.rs b/src/ip35/node.rs new file mode 100644 index 000000000..42414d32a --- /dev/null +++ b/src/ip35/node.rs @@ -0,0 +1,799 @@ +//! The Bedrock node every IP35 board shares: the processor bus with its four +//! views, the two PIs, MD and its memory, the L1, the boot flash and the +//! node's I/O, local-bus, network and crossbar blocks, wired together and to +//! the running CPU, and the board's serial ports. A board names its inputs in +//! [`NodeConfig`] and adds its own XIO widgets. +//! +//! # What is approximated here +//! +//! The two Hub-special head windows return zero on reads and acknowledge +//! writes without side effects. PROM 6.210 writes the 64-bit value 0x2000 +//! at relative offsets 0 and 0x40 before DRAM discovery. Their hardware +//! meaning is unknown; Bedrock logging records accesses to these windows. +//! +//! Node composition ported from hw/mips/sgi/sn1-node.c in +//! jamesbraid/qemu-sgi-origami. + +use super::{map_device, Ip35Bus, SparseWindowMap, Window}; +use crate::config::MachineConfig; +use crate::cpu::mips_exec::CpuDevice; +use crate::dev::bedrock_flash::{FirmwareRecord, Flash, PART_SIZE}; +use crate::dev::bedrock_io::{BedrockIo, SIZE as IO_SIZE}; +use crate::dev::bedrock_lb::{BedrockLb, SIZE as LB_SIZE}; +use crate::dev::bedrock_md::{BedrockMd, MemoryError}; +use crate::dev::bedrock_ni::{BedrockNi, SIZE as NI_SIZE}; +use crate::dev::bedrock_pi::{BedrockPi, BedrockPiConfig, Population, SIZE as PI_SIZE}; +use crate::dev::bedrock_xb::{BedrockXb, SIZE as XB_SIZE}; +use crate::dev::heart::Clock; +use crate::dev::l1::{L1, L1Config, L1Event}; +use crate::dev::odyssey::{pi_gfx_mark_store, pi_gfx_unmark_store}; +use crate::dev::serial_host::{null_serial_backend, open_log, tcp_serial_backend, CiSerialBackend, LogMode, SerialBackend, TeeBackend}; +use crate::devlog::LogModule; +use crate::irq::{InterruptController, IrqLine}; +use crate::ppmem::devmap::SLOT_SHIFT; +use crate::traits::{BusDevice, BusRead16, BusRead32, BusRead64, BusRead8, MachineEvent, Signal, BUS_BUSY, BUS_ERR, BUS_OK}; +use parking_lot::Mutex; +use std::path::Path; +use std::sync::mpsc::{channel, Receiver}; +use std::sync::{Arc, OnceLock}; +#[cfg(test)] +use std::sync::atomic::{AtomicU64, Ordering}; + +/// Give a node's memory and devices to its running CPU: the direct-access +/// window, interrupts from PI0 slot A, and the resets and power events the +/// local bus, network block and L1 raise. The events arrive on the receiver +/// the system controller takes. +pub(crate) fn connect_cpu( + name: &'static str, + md: &Arc, + pi0: &BedrockPi, + lb: &BedrockLb, + ni: &BedrockNi, + l1: &L1, + cpu: &Arc, +) -> Receiver { + connect(name, md, &[pi0], lb, ni, l1, Population::SINGLE, std::slice::from_ref(cpu)) +} + +/// Give a node's memory and devices to the processors the board runs, CPU 0 +/// first, each in the slot `running` gives it: the direct-access window and +/// its page bitmaps for every one, each PI's interrupt outputs to the +/// processors in its slots, and the node's resets and power events, an L1 +/// reset reaching every processor. +pub(crate) fn connect_cpus(name: &'static str, node: &Node, running: Population, cpus: &[Arc]) -> Receiver { + connect(name, &node.md, &[node.pi[0].as_ref(), node.pi[1].as_ref()], &node.lb, &node.ni, &node.l1, running, cpus) +} + +/// [`connect_cpus`] over the PIs given, PI0 first. +#[allow(clippy::too_many_arguments)] +fn connect( + name: &'static str, + md: &Arc, + pis: &[&BedrockPi], + lb: &BedrockLb, + ni: &BedrockNi, + l1: &L1, + running: Population, + cpus: &[Arc], +) -> Receiver { + if let Some(space) = md.physical().ppmem_space() { + use crate::ppmem::MappedMemory; + unsafe { + for (id, cpu) in cpus.iter().enumerate() { + if id == 0 { + space.set_bitmap_sink(cpu.ppmem_bitmap_ptr()); + } else { + space.add_bitmap_sink(cpu.ppmem_bitmap_ptr()); + } + cpu.set_tcache_window(space.window_base()); + let bitmap = cpu.tcache_bitmap_ptr(); + if !bitmap.is_null() { + if id == 0 { space.set_bitmap_sink2(bitmap) } else { space.add_bitmap_sink(bitmap) } + } + #[cfg(feature = "jitv2")] + cpu.set_tcache_gen_window(md.generation_base()); + } + } + md.physical().resync_ppmem_bitmap(); + md.rebuild_memory(); + } + for (interface, pi) in pis.iter().enumerate() { + let slot = |slot| running.cpu_in(interface as u8, slot).and_then(|cpu| cpus.get(cpu)).map(Arc::downgrade); + pi.set_interrupt_controller(Arc::new(SlotIrq([slot(0), slot(1)]))); + } + node_events(name, lb, ni, l1, cpus.iter().map(Arc::downgrade).collect()) +} + +/// The resets and power events a node's local bus, network block and L1 +/// raise, on the receiver the system controller takes. An L1 soft reset +/// reaches each of `cpus`. +fn node_events(name: &'static str, lb: &BedrockLb, ni: &BedrockNi, l1: &L1, cpus: Vec>) -> Receiver { + let (tx, rx) = channel(); + let reset_tx = tx.clone(); + lb.connect_soft_reset(Arc::new(move || { + dlog!(LogModule::Bedrock, "{name}: LB soft reset from CPU {:?}; restarting the node", crate::cpu::smp::on_cpu_thread()); + let _ = reset_tx.send(MachineEvent::HardReset); + })); + let reset_tx = tx.clone(); + ni.connect_local_reset(Arc::new(move || { + dlog!(LogModule::Bedrock, "{name}: NI local reset from CPU {:?}; restarting the node", crate::cpu::smp::on_cpu_thread()); + let _ = reset_tx.send(MachineEvent::HardReset); + })); + l1.set_event_handler(Arc::new(move |event| { + match event { + L1Event::SoftReset => { + for cpu in cpus.iter().filter_map(|cpu| cpu.upgrade()) { + cpu.as_device().signal(Signal::Reset(true)); + } + } + L1Event::Nmi => dlog!(LogModule::L1, "{name}: NMI entry is not implemented by the CPU"), + _ => { + dlog!(LogModule::L1, "{name}: L1 event {event:?}"); + if let Some(action) = machine_action(event) { + let _ = tx.send(action); + } + } + } + })); + rx +} + +/// What the machine does for an L1 power or reset event. A power change +/// alone does nothing: the shutdown or restart that the L1 reports with it +/// carries the action. So a power cycle, which the PROM asks for to retry +/// a processor, restarts the node rather than powering the machine off, as +/// in the QEMU L1, where only a shutdown stops the machine. +fn machine_action(event: L1Event) -> Option { + match event { + L1Event::Shutdown => Some(MachineEvent::PowerOff), + L1Event::Reset => Some(MachineEvent::HardReset), + L1Event::Power(_) | L1Event::SoftReset | L1Event::Nmi => None, + } +} + +/// One of a board's serial connections: what it is called, the TCP port and +/// the setting that names the port, the CI backend it uses when it is the +/// console, the log it keeps when it is not, and where the host end goes. +pub(crate) struct SerialPort<'a> { + pub label: &'static str, + pub port: u16, + pub setting: &'static str, + pub ci: &'a Arc, + pub log: Option, + pub connect: Box) + 'a>, +} + +/// Connect a board's serial ports to the host. The console takes +/// `serial_log` and, under CI, the in-process backend; the others listen on +/// their TCP ports, and a port that is taken leaves its stream unconnected +/// rather than stopping the machine. +pub(crate) fn connect_serial_ports( + cfg: &MachineConfig, + name: &str, + ports: &[SerialPort<'_>], + console: usize, +) -> Result<(), String> { + for (i, a) in ports.iter().enumerate() { + if let Some(b) = ports[i + 1..].iter().find(|b| b.port == a.port) { + return Err(format!("{name} {} and {} serial ports must be distinct", a.label, b.label)); + } + } + let mut log_config = cfg.clone(); + for (index, port) in ports.iter().enumerate() { + log_config.serial_log = if index == console { cfg.serial_log.clone() } else { port.log.clone() }; + let tag = format!("{name} {}", port.label); + let log = open_log(&log_config, &tag, LogMode::Fresh); + let backend: Arc = if cfg.ci && index == console { + if let Some(log) = log { + port.ci.set_log(log); + } + port.ci.clone() + } else { + let mut backend = null_serial_backend(); + match tcp_serial_backend(port.port) { + Some((socket, actual_port)) => { + eprintln!("iris: {tag} on 127.0.0.1:{actual_port}"); + backend = socket; + } + None => eprintln!( + "iris: {tag}: port {} unavailable; is another iris using it? ({})", + port.port, port.setting + ), + } + match log { + Some(log) => Arc::new(TeeBackend::new(backend, log)), + None => backend, + } + }; + (port.connect)(backend); + } + Ok(()) +} + +/// The local node register blocks projected into the processor's I/O view. +pub mod reg { + pub const PI1_BASE: u64 = 0x20_0000; + pub const IO_BASE: u64 = 0x40_0000; + pub const LB_BASE: u64 = 0x60_0000; + pub const NI_BASE: u64 = 0x68_0000; + pub const XB_BASE: u64 = 0x70_0000; + pub const MD_BASE: u64 = 0x78_0000; + pub const MD_SIZE: u64 = 0x8_0000; +} + +/// What a board gives its Bedrock node. +pub(crate) struct NodeConfig<'a> { + /// The board's name in logs and map names. + pub name: &'static str, + pub payload: &'a [u8], + pub clock: Clock, + pub flash_path: Option<&'a Path>, + /// The memory banks in MiB, locational bank 0 upward. + pub banks_mib: &'a [u32], + /// The processor slots the board fits. + pub population: Population, + /// The fitted slots whose processor runs. + pub running: Population, + pub firmware: FirmwareRecord, + /// Environment records a fresh boot flash starts with, at their offsets. + pub flash_records: Vec<(usize, Vec)>, + /// The word the board reads back at the Hub-special board-id window. + pub board_id: u64, + pub l1: L1Config, +} + +/// One Bedrock node as the IP35 boards share it: the processor bus with its +/// four views, the two PIs, MD with its memory, the L1, the boot flash and +/// the node's I/O, local-bus, network and crossbar blocks, wired together. +/// The board adds its XIO widgets to `io` and its serial backends to `l1`. +pub(crate) struct Node { + pub bus: Arc, + pub pi: [Arc; 2], + pub md: Arc, + pub l1: Arc, + pub flash: Arc, + pub lb: Arc, + pub io: Arc, + pub ni: Arc, + pub xb: Arc, + /// Set once by a board that counts graphics stores. + io_store_counter: Arc>>, + #[cfg(test)] + pub irq: Arc, +} + +impl Node { + /// Count the processor's uncached I/O stores with `counter`, as a board + /// with graphics credits does. Once, before the processor runs. + pub fn count_io_stores(&self, counter: Box) { + assert!(self.io_store_counter.set(counter).is_ok(), "node I/O store counter already set"); + } + + pub fn build(config: NodeConfig<'_>) -> Result { + let NodeConfig { name, payload, clock, flash_path, banks_mib, population, running, firmware, flash_records, board_id, l1 } = config; + let md = BedrockMd::with_clock(banks_mib, clock.clone())?; + let phys = md.physical().clone(); + let pi: [Arc; 2] = std::array::from_fn(|interface| { + BedrockPi::new( + BedrockPiConfig { + interface: interface as u8, + request_region: 0, + population, + running, + }, + clock.clone(), + ) + }); + let l1 = Arc::new(L1::new(l1)?); + let flash = Arc::new(match flash_path { + Some(path) => Flash::with_file_seeded(payload, path, firmware, &flash_records), + None => Flash::seeded(payload, firmware, &flash_records), + }.map_err(|e| e.to_string())?); + let lb = Arc::new(BedrockLb::new(clock.clone())); + let io = Arc::new(BedrockIo::new()); + let ni = Arc::new(BedrockNi::new()); + let xb = Arc::new(BedrockXb::new()); + // A processor on PI1 finds its own interface at the local form's + // offset 0, as QEMU's node overlays it there (OpenBSD expects the + // local form to select the requester's interface). That the chip + // answers PI1 there is an approximation pending a hardware read. + let local_pi: Arc = if (0..2).any(|slot| running.fitted(1, slot)) { + Arc::new(LocalPi { pi: pi.clone(), running }) + } else { + pi[0].clone() + }; + let mut registers = SparseWindowMap::default(); + registers.map( + 0, + PI_SIZE, + pi[0].clone(), + ); + registers.map( + reg::PI1_BASE, + PI_SIZE, + pi[1].clone(), + ); + registers.map( + reg::IO_BASE, + IO_SIZE, + io.clone(), + ); + registers.map( + reg::LB_BASE, + LB_SIZE, + lb.clone(), + ); + registers.map( + reg::NI_BASE, + NI_SIZE, + ni.clone(), + ); + registers.map( + reg::XB_BASE, + XB_SIZE, + xb.clone(), + ); + registers.map( + reg::MD_BASE, + reg::MD_SIZE, + md.md_bus(), + ); + let mut local = SparseWindowMap::default(); + local.map(0, PI_SIZE, local_pi); + local.map(reg::PI1_BASE, PI_SIZE, pi[1].clone()); + local.map(reg::IO_BASE, IO_SIZE, io.clone()); + local.map(reg::LB_BASE, LB_SIZE, lb.clone()); + local.map(reg::NI_BASE, NI_SIZE, ni.clone()); + local.map(reg::XB_BASE, XB_SIZE, xb.clone()); + local.map(reg::MD_BASE, reg::MD_SIZE, md.md_bus()); + let registers = Arc::new(registers); + let local = Arc::new(local); + io.connect_hub(Arc::new(IoHub(pi.clone()))); + io.connect_memory(md.dma_view()); + let p = pi.clone(); + lb.connect_error(Arc::new(move |level| { + for pi in &p { + pi.set_lb_error(level); + } + })); + let p = pi.clone(); + ni.connect_error(Arc::new(move |level| { + for pi in &p { + pi.set_ni_error(level); + } + })); + let p = pi.clone(); + xb.connect_error(Arc::new(move |level| { + for pi in &p { + pi.set_xb_error(level); + } + })); + let io_node = Arc::downgrade(&io); + lb.connect_node_changed(Arc::new(move |node| { + if let Some(io) = io_node.upgrade() { + io.set_node_number(node); + } + })); + let p = pi.clone(); + md.on_error(Arc::new(move |kind| { + for pi in &p { + match kind { + MemoryError::Correctable => pi.set_pending(1, 61), + MemoryError::Uncorrectable => { + pi.set_memory_uncorrectable(); + } + } + } + })); + #[cfg(test)] + let irq = Arc::new(PendingIrq::default()); + #[cfg(test)] + pi[0].set_interrupt_controller(irq.clone()); + l1.set_irq(IrqLine::new(Arc::new(UartIrq(pi.clone())), 0)); + let mut bus = Ip35Bus::new(phys.clone()); + #[cfg(feature = "jitv2")] + bus.set_ram_generations(md.dma_view()); + md.rebuild_memory(); + let io_store_counter = Arc::new(OnceLock::new()); + bus.set_fallback(1, 1 << 32, Arc::new(IoPio { io: io.clone(), counter: io_store_counter.clone() })); + // The two XIO register forms select the requester's PI0 interface. + let permission_md = Arc::downgrade(&md); + let permission: Arc bool + Send + Sync> = Arc::new(move || { + permission_md.upgrade().is_some_and(|md| md.current_cell() & 1 != 0) + }); + let protected_regs = Arc::new(Window::new(0, 0x80_0000, registers.clone()).permitted(permission.clone())); + let protected_local = Arc::new(Window::new(0, 0x80_0000, local.clone()).permitted(permission.clone())); + bus.map(1, 0x0100_0000, 0x80_0000, protected_local); + bus.map(1, 0x0180_0000, 0x80_0000, protected_regs.clone()); + // Cached RAM uses MD's direct bank mappings; diagnostic views stay separate. + bus.map(3, 0, 1 << 33, md.uncached_view()); + // Hub-special low memory: its first 256 MiB is the node memory view. + bus.map(0, 0, 0x1000_0000, md.uncached_view()); + let prom = Arc::new( + flash + .array_view(0, PART_SIZE) + .map_err(|e| e.to_string())?, + ); + let boot = Arc::new(flash.boot_view()); + for base in [0x1800_0000, 0x3800_0000] { + bus.map(0, base, 0x800_0000, boot.clone()); + } + // Reset execution always sees stored bytes, independently of commands. + bus.map(0, 0x1fc0_0000, 0x40_0000, prom.clone()); + if !payload.is_empty() { + map_device( + &phys, + 0x1fc0_0000, + payload.len() as u64, + prom.clone(), + &format!("{name} cached reset PROM"), + ); + } + let leds = Arc::new(Leds { + md: md.clone(), + values: Mutex::new([0; 4]), + }); + // The three windows occupy the same processor register page. + let special = Arc::new(HubSpecialPage { + windows: [ + Window::new(0, 0x80, Arc::new(HubHead(name))).permitted(permission.clone()), + Window::new(0x80, 0x40, l1.clone()).permitted(permission.clone()), + Window::new(0xc0, 0x20, leds).permitted(permission.clone()), + ], + }); + for base in [0x1000_0000, 0x3000_0000] { + bus.map(0, base, 1 << SLOT_SHIFT, special.clone()); + bus.map(0, base + 0x400_0000, 8, Arc::new(BoardId { md: md.clone(), word: board_id })); + } + bus.map(0, 0x1_c000_0000, 1 << 30, md.directory_bus()); + bus.map(0, 0x1_4000_0000, 1 << 30, md.ecc_bus()); + bus.map(0, 0x1_0000_0000, 1 << 30, md.ecc_scrub_bus()); + bus.map(2, 0, 1 << 33, md.fetchop_bus()); + Ok(Self { + bus: Arc::new(bus), + io_store_counter, + pi, + md, + l1, + flash, + lb, + io, + ni, + xb, + #[cfg(test)] + irq, + }) + } +} + +/// The local register form's PI: the requester's own interface. A reader +/// that is no processor reaches PI0. +struct LocalPi { + pi: [Arc; 2], + running: Population, +} +impl LocalPi { + fn own(&self) -> &BedrockPi { + let interface = crate::cpu::smp::on_cpu_thread().and_then(|cpu| self.running.slot_of(cpu)).map_or(0, |(i, _)| i); + &self.pi[usize::from(interface)] + } +} +impl BusDevice for LocalPi { + fn read64(&self, a: u64) -> BusRead64 { + self.own().read64(a) + } + fn read32(&self, a: u64) -> BusRead32 { + self.own().read32(a) + } + fn write64(&self, a: u64, v: u64) -> u32 { + self.own().write64(a, v) + } +} + +/// A PI's interrupt outputs to the processors in its two slots: lines 0..4 +/// are slot A's IP2..IP6, lines 5..9 slot B's, each to the processor the +/// board put there. +pub(crate) struct SlotIrq(pub [Option>; 2]); +impl InterruptController for SlotIrq { + fn set_line(&self, line: u32, level: bool) { + if let Some(cpu) = self.0.get(line as usize / 5).and_then(|c| c.as_ref()).and_then(|c| c.upgrade()) { + cpu.as_device().signal(Signal::Interrupt(line % 5 + 2, level)); + } + } +} + +// XIO register packets reach only the two PI interrupt-modify registers. +// The processor's register projection has a separate, wider interface. +struct IoHub([Arc; 2]); +impl BusDevice for IoHub { + fn write64(&self, a: u64, v: u64) -> u32 { + if matches!(a, 0x90 | 0x20_0090) && self.0[(a >> 21) as usize].io_modify(v, 0) { + BUS_OK + } else { + BUS_ERR + } + } +} + +struct HubSpecialPage { + windows: [Window; 3], +} +impl HubSpecialPage { + fn target(&self, a: u64, n: u64) -> Option<&Window> { + self.windows.iter().find(|window| window.contains(a, n)) + } +} +macro_rules! hub_page_access { + ($r:ident,$w:ident,$t:ty,$out:ident,$n:expr) => { + fn $r(&self, a: u64) -> $out { + self.target(a, $n).map(|window| window.$r(a)).unwrap_or_else($out::err) + } + fn $w(&self, a: u64, value: $t) -> u32 { + self.target(a, $n).map(|window| window.$w(a, value)).unwrap_or(BUS_ERR) + } + }; +} +impl BusDevice for HubSpecialPage { + hub_page_access!(read8, write8, u8, BusRead8, 1); + hub_page_access!(read16, write16, u16, BusRead16, 2); + hub_page_access!(read32, write32, u32, BusRead32, 4); + hub_page_access!(read64, write64, u64, BusRead64, 8); +} + +struct BoardId { + md: Arc, + word: u64, +} +impl BusDevice for BoardId { + fn read64(&self, a: u64) -> BusRead64 { + if a == 0 && self.md.current_cell() & 1 != 0 { + BusRead64::ok(self.word) + } else { + BusRead64::err() + } + } +} +// Compatibility head windows: their register contract is not established. +// PROM 6.210 writes 0x2000 at offsets 0 and 0x40 before DRAM discovery. +// Retain the existing zero-read/acknowledged-write model, with accesses +// observable through the Bedrock log, until the register behavior is known. +/// The board name for the log. +struct HubHead(&'static str); +macro_rules! head_access { + ($r:ident,$w:ident,$ty:ty,$result:ident) => { + fn $r(&self, a: u64) -> $result { + dlog!(LogModule::Bedrock, + "{}: unidentified Hub-special read {} bytes at {a:#x}", + self.0, + std::mem::size_of::<$ty>() + ); + $result::ok(0) + } + fn $w(&self, a: u64, v: $ty) -> u32 { + dlog!(LogModule::Bedrock, + "{}: unidentified Hub-special write {} bytes at {a:#x} value {v:#x}", + self.0, + std::mem::size_of::<$ty>() + ); + BUS_OK + } + }; +} +impl BusDevice for HubHead { + head_access!(read8, write8, u8, BusRead8); + head_access!(read16, write16, u16, BusRead16); + head_access!(read32, write32, u32, BusRead32); + head_access!(read64, write64, u64, BusRead64); +} +struct Leds { + md: Arc, + values: Mutex<[u8; 4]>, +} +impl BusDevice for Leds { + fn read64(&self, a: u64) -> BusRead64 { + if a < 32 && a & 7 == 0 && self.md.current_cell() & 1 != 0 { + BusRead64::ok(self.values.lock()[a as usize / 8] as u64) + } else { + BusRead64::err() + } + } + fn write64(&self, a: u64, v: u64) -> u32 { + if a < 32 && a & 7 == 0 && self.md.current_cell() & 1 != 0 { + self.values.lock()[a as usize / 8] = v as u8; + BUS_OK + } else { + BUS_ERR + } + } +} + +/// A widget's DMA as Bedrock's inbound path takes it: crosstalk addresses, +/// with the widget's inbound permissions applied. +pub(crate) struct WidgetDma { + pub io: std::sync::Weak, + pub widget: u8, +} +macro_rules! widget_dma_access { + ($r:ident,$w:ident,$ty:ty,$result:ident,$size:expr) => { + fn $r(&self, a: u64) -> $result { + let Some(io) = self.io.upgrade() else { return $result::err() }; + let r = io.inbound_read(self.widget, None, a, $size); + $result { data: r.data as $ty, status: r.status } + } + fn $w(&self, a: u64, v: $ty) -> u32 { + self.io.upgrade().map_or(BUS_ERR, |io| io.inbound_write(self.widget, None, a, $size, v as u64)) + } + }; +} +impl BusDevice for WidgetDma { + widget_dma_access!(read8, write8, u8, BusRead8, 1); + widget_dma_access!(read16, write16, u16, BusRead16, 2); + widget_dma_access!(read32, write32, u32, BusRead32, 4); + widget_dma_access!(read64, write64, u64, BusRead64, 8); +} + +/// What the processor interface does with a processor's uncached store. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum Store { + /// Outside the slot's enabled graphics page. + Uncounted, + /// Counted; `below_bias` asks for the slot's graphics interrupt. + Counted { below_bias: bool }, + /// The slot is stopped: the store waits, uncounted. + Held, +} + +/// The processor interface's count of uncached I/O stores against a +/// graphics page (see [`crate::dev::odyssey::PiGfx`]). +pub(crate) trait StoreCounter: Send + Sync { + /// The store at node I/O offset `address`: uncounted, counted, or held + /// and answered busy without being forwarded. + fn count(&self, address: u64) -> Store; + /// A counted store the bus answered busy did not happen: the processor + /// will make it again, and pays for it then. + fn refund(&self, address: u64); +} + +/// The processor's uncached I/O view of the node. A board with graphics +/// counts its stores; the store is forwarded marked as counted or not. +struct IoPio { + io: Arc, + counter: Arc>>, +} +macro_rules! pio_access { + ($r:ident,$w:ident,$ty:ty,$result:ident,$size:expr) => { + fn $r(&self, a: u64) -> $result { + let r = self.io.pio_read(a, $size); + $result { + data: r.data as $ty, + status: r.status, + } + } + fn $w(&self, a: u64, v: $ty) -> u32 { + let counter = self.counter.get(); + let counted = match counter.map_or(Store::Uncounted, |c| c.count(a)) { + Store::Held => return BUS_BUSY, + Store::Counted { .. } => true, + Store::Uncounted => false, + }; + let previous = pi_gfx_mark_store(counted); + let status = self.io.pio_write(a, $size, v as u64); + pi_gfx_unmark_store(previous); + if let (true, BUS_BUSY, Some(c)) = (counted, status, counter) { + c.refund(a); + } + status + } + }; +} +impl BusDevice for IoPio { + pio_access!(read8, write8, u8, BusRead8, 1); + pio_access!(read16, write16, u16, BusRead16, 2); + pio_access!(read32, write32, u32, BusRead32, 4); + pio_access!(read64, write64, u64, BusRead64, 8); +} +#[cfg(test)] +#[derive(Default)] +pub(crate) struct PendingIrq(AtomicU64); +#[cfg(test)] +impl PendingIrq { + pub fn bits(&self) -> u64 { + self.0.load(Ordering::Acquire) + } +} +#[cfg(test)] +impl InterruptController for PendingIrq { + fn set_line(&self, line: u32, level: bool) { + if line >= 5 { + return; + } + let mask = 1 << (10 + line); + if level { + self.0.fetch_or(mask, Ordering::Release); + } else { + self.0 + .fetch_and(!mask, Ordering::Release); + } + } +} +struct UartIrq([Arc; 2]); +impl InterruptController for UartIrq { + fn set_line(&self, _: u32, level: bool) { + for pi in &self.0 { + pi.set_uart_irq(level); + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + /// A test node with four processors fitted and `running` of them run. + fn node(running: usize) -> Node { + let population = Population::fitted_cpus(4).unwrap(); + Node::build(NodeConfig { + name: "test", + payload: &[], + clock: Arc::new(|| 0), + flash_path: None, + banks_mib: &[512], + population, + running: Population::fitted_cpus(running).unwrap(), + firmware: FirmwareRecord::TEZRO, + flash_records: Vec::new(), + board_id: 0, + l1: L1Config::default(), + }) + .unwrap() + } + + /// The CPU-number register as processor `cpu` reads it through the XIO + /// register form at `window`. + fn cpu_number(node: &Arc, cpu: Option, window: u64) -> u64 { + let node = node.clone(); + std::thread::spawn(move || { + if let Some(cpu) = cpu { + crate::cpu::smp::set_current_cpu(cpu); + } + node.bus.read64((1 << 57) | window | 0x20).data + }) + .join() + .unwrap() + } + + /// Through the local form a processor reaches its own interface at + /// offset 0, so every processor of four reads its own slice there; the + /// node form keeps PI0 at 0 for everyone, and a reader that is no + /// processor reaches PI0. Follows sn1-cpu-identity-test + /// four-processors-know-their-slots, which reads the local form. + #[test] + fn the_local_form_answers_the_requesters_own_interface() { + let node = Arc::new(node(4)); + let local: Vec = (0..4).map(|cpu| cpu_number(&node, Some(cpu), 0x0100_0000)).collect(); + assert_eq!(local, [0, 2, 1, 3]); + let node_form: Vec = (0..4).map(|cpu| cpu_number(&node, Some(cpu), 0x0180_0000)).collect(); + assert_eq!(node_form, [0, 0, 1, 0], "PI0's register: only PI0's processors read their slot"); + assert_eq!(cpu_number(&node, None, 0x0100_0000), 0); + // PI1 stays at its own offset in both forms. + let pi1 = node.bus.read64((1 << 57) | 0x0100_0000 | reg::PI1_BASE | 0x20).data; + assert_eq!(pi1, 2); + // With PI0/A alone running, as on the Fuel, the local form is PI0. + assert_eq!(cpu_number(&Arc::new(self::node(1)), Some(0), 0x0100_0000), 0); + } + + /// The L1 reports a power cycle as power down, power up and a restart, + /// and a shutdown as power down and a shutdown: one restart for the + /// first and one power-off for the second, never a power-off on the way + /// through a cycle. + #[test] + fn a_power_cycle_restarts_and_only_a_shutdown_powers_off() { + let actions = |events: &[L1Event]| { + format!("{:?}", events.iter().filter_map(|&e| machine_action(e)).collect::>()) + }; + assert_eq!(actions(&[L1Event::Power(false), L1Event::Power(true), L1Event::Reset]), "[HardReset]"); + assert_eq!(actions(&[L1Event::Power(false), L1Event::Shutdown]), "[PowerOff]"); + assert_eq!(actions(&[L1Event::Power(true), L1Event::Reset]), "[HardReset]"); + assert_eq!(actions(&[L1Event::Reset]), "[HardReset]"); + } +} diff --git a/src/ip35/rendezvous.rs b/src/ip35/rendezvous.rs new file mode 100644 index 000000000..48719baf2 --- /dev/null +++ b/src/ip35/rendezvous.rs @@ -0,0 +1,521 @@ +//! The IP35 node's boot rendezvous: where the PROM assumes its processors +//! run in lockstep, the model holds the one that arrived early. +//! +//! The IP35 PROM brings a node's processors up on the assumption that they +//! left reset on the same clock edge and have run continuously since. Four +//! of its early handshakes order one processor's stores before another's +//! reads by nothing more than both running at hardware speed. Here each +//! processor is a host thread that can fall far behind the others, so at +//! each of those points the processor that arrived early is held +//! ([`HOLD_RENDEZVOUS`]) until the ones it must follow catch up, or until +//! they have run [`HOLD_CYCLES`] without doing so. The four, in boot order: +//! +//! - Reset barrier. Every processor clears LB scratch 1 on its reset path, +//! and the boot master then posts its release there, which the others +//! poll for. A processor's second or later store there is withheld, and +//! the processor held, until every other running processor has made its +//! first; the withheld value lands when the hold ends, before the +//! processor runs again. +//! - Pair liveness. Each processor of a pair stores a status value to its +//! own real-time compare register and polls its partner's. The first of +//! a pair to post is held until its partner posts. +//! - Presence publisher. The lowest running processor of each interface +//! publishes its interface's processors in II scratch 0 under the hub +//! lock. A publisher is held at its next release of the lock (a store +//! leaving LB scratch 3 clear) until every other populated interface has +//! published. +//! - Partner wait. The other processor of an interface is held at its next +//! real-time counter read, once both of the pair have posted, until its +//! publisher has released the lock after publishing. +//! +//! The bound counts the cycles the processors being waited for have +//! retired, not host time. A host thread the scheduler leaves off a core +//! for seconds retires nothing, so it cannot run a hold out: the processor +//! that arrived early waits however long the host takes, and the PROM +//! counts every processor. Host time only backs the bound as a deadlock +//! safety net ([`SAFETY_NS`]), logged when it fires. QEMU bounds the same +//! holds at 2 s of its virtual clock, which under multi-threaded TCG +//! follows the host clock, so a starved vCPU thread can run its holds out +//! there too. +//! +//! Each is an approximation with no hardware counterpart: a hold changes +//! when a processor next runs, never a value it reads. A held processor +//! stops at its next batch boundary rather than at once, so it may run a +//! few more instructions than QEMU's, which stops at the end of its +//! translation block. A publisher's or partner's hold that runs out ends +//! the join until the next reset; a liveness or reset-barrier hold that +//! runs out ends only itself, and the PROM reports the processor that never +//! came. Nothing holds with fewer than two processors running. QEMU hooks +//! the publisher's hold on LB scratch 3, as here: the PROM publishes in II +//! scratch 0 and takes scratch 3 as its console lock, and the hub lock being +//! scratch 3's read-set twin is inferred. +//! +//! Ported from hw/misc/sgi-bedrock.c in jamesbraid/qemu-sgi-origami. + +use crate::cpu::mips_exec::CpuDevice; +use crate::cpu::smp::{on_cpu_thread, ParkSlot, HOLD_RENDEZVOUS}; +use crate::dev::bedrock_lb::{BedrockLb, ScratchHooks}; +use crate::dev::bedrock_pi::Population; +use crate::dev::heart::Clock; +use crate::devlog::LogModule; +use crate::hptimer::{TimerId, TimerManager, TimerReturn}; +use parking_lot::Mutex; +use std::sync::atomic::{AtomicBool, Ordering}; +use std::sync::{Arc, OnceLock, Weak}; +use std::time::Duration; + +/// How far the processors a hold waits for may run, in their retired +/// cycles, before the hold gives up: 2 s of a 1 GHz processor, the order of +/// QEMU's 2 s bound. +pub(crate) const HOLD_CYCLES: u64 = 2_000_000_000; +/// The longest any hold lasts in board nanoseconds whatever the processors +/// do: a deadlock safety net, far past any host stall. +pub(crate) const SAFETY_NS: u64 = 120_000_000_000; +/// How often the holds' bounds are checked, in host time. +const POLL: Duration = Duration::from_millis(50); +const SLOTS: usize = 4; +/// Processor slots per interface. +const PER_PI: usize = 2; + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum Kind { + Scratch1, + Liveness, + Presence, + Partner, +} + +#[derive(Default)] +struct State { + scratch1_seen: [bool; SLOTS], + /// A processor's withheld scratch-1 store and its order among them. + withheld: [Option<(u64, u64)>; SLOTS], + seq: u64, + scratch1_released: bool, + joined: [bool; SLOTS], + posted: [bool; SLOTS], + presence_seen: [bool; 2], + presence_released: [bool; 2], + abandoned: bool, + park: [Option; SLOTS], +} + +/// One held slot: what it waits at, when it started in board nanoseconds, +/// and how many cycles the processors it waits for had retired by then. +#[derive(Clone, Copy)] +struct Hold { + kind: Kind, + since: u64, + progress: u64, +} + +/// One processor the rendezvous may hold, and whether it runs. +pub(crate) struct Cpu { + pub slot: Arc, + /// Cycles this processor has retired; frozen while the host does not run it. + pub progress: Box u64 + Send + Sync>, + pub running: Box bool + Send + Sync>, +} + +/// The node's rendezvous state. Slots are numbered `interface * 2 + slot`: +/// A, B on PI0, C, D on PI1. +pub(crate) struct Rendezvous { + running: Population, + clock: Clock, + lb: Weak, + cpus: OnceLock>, + timers: OnceLock>, + timer: Mutex>, + state: Mutex, + /// Every populated interface has released its hub lock after publishing, + /// or the join was abandoned, and nothing is held: the joins are over + /// until the next reset. Checked before any lock, so the PROM's and the + /// kernel's later register traffic costs one load. + done: AtomicBool, + weak: Weak, +} + +impl Rendezvous { + pub fn new(running: Population, clock: Clock, lb: &Arc) -> Arc { + Arc::new_cyclic(|weak| Self { + running, + clock, + lb: Arc::downgrade(lb), + cpus: OnceLock::new(), + timers: OnceLock::new(), + timer: Mutex::new(None), + state: Mutex::new(State::default()), + done: AtomicBool::new(false), + weak: weak.clone(), + }) + } + + /// The processors, CPU 0 first, in the order the board's population + /// numbers them. Once. + pub fn set_cpus(&self, cpus: &[Arc]) { + self.set_processors(cpus.iter().map(|c| { + let device: Weak = Arc::downgrade(c); + let cycles = c.cycles_ptr(); + Cpu { + slot: c.park_slot(), + progress: Box::new(move || cycles.retired()), + running: Box::new(move || device.upgrade().is_some_and(|c| c.as_device().is_running())), + } + }).collect()); + } + + pub(crate) fn set_processors(&self, cpus: Vec) { + assert!(self.cpus.set(cpus).is_ok(), "rendezvous processors set twice"); + } + + pub fn set_timer_manager(&self, timers: Arc) { + let _ = self.timers.set(timers); + } + + fn cpu(&self, slot: usize) -> Option<&Cpu> { + let cpu = self.running.cpu_in((slot / PER_PI) as u8, slot % PER_PI)?; + self.cpus.get()?.get(cpu) + } + + /// The slot the calling thread's processor sits in, if it is one here. + fn current_slot(&self) -> Option { + let (interface, slot) = self.running.slot_of(on_cpu_thread()?)?; + Some(usize::from(interface) * PER_PI + slot) + } + + /// Fitted, running and built; a processor this holds still counts. + fn slot_running(&self, slot: usize) -> bool { + self.cpu(slot).is_some_and(|c| (c.running)()) + } + + fn applicable(&self) -> bool { + (0..SLOTS).filter(|&s| self.slot_running(s)).count() >= 2 + } + + fn interface_populated(&self, interface: usize) -> bool { + (0..PER_PI).any(|p| self.slot_running(interface * PER_PI + p)) + } + + /// The interface's publisher: its lowest running processor. + fn publisher(&self, interface: usize) -> Option { + (0..PER_PI).map(|p| interface * PER_PI + p).find(|&s| self.slot_running(s)) + } + + fn other_pending(&self, s: &State, slot: usize) -> bool { + let own = slot / PER_PI; + !s.abandoned && (0..2).any(|i| i != own && self.interface_populated(i) && !s.presence_seen[i]) + } + + /// The running slots a hold at `kind` waits for. + fn awaited(&self, s: &State, slot: usize, kind: Kind) -> Vec { + let own = slot / PER_PI; + (0..SLOTS) + .filter(|&p| p != slot && self.slot_running(p)) + .filter(|&p| match kind { + Kind::Scratch1 => !s.scratch1_seen[p], + Kind::Liveness => p == slot ^ 1, + Kind::Presence => p / PER_PI != own, + Kind::Partner => self.publisher(own) == Some(p), + }) + .collect() + } + + /// Cycles the slots a hold waits for have retired between them. + fn progress(&self, s: &State, slot: usize, kind: Kind) -> u64 { + self.awaited(s, slot, kind) + .into_iter() + .filter_map(|p| self.cpu(p)) + .fold(0u64, |sum, c| sum.saturating_add((c.progress)())) + } + + fn scratch1_ready(&self, s: &State) -> bool { + (0..SLOTS).all(|p| !self.slot_running(p) || s.scratch1_seen[p]) + } + + fn ready(&self, s: &State, slot: usize, kind: Kind) -> bool { + match kind { + Kind::Presence => !self.other_pending(s, slot), + Kind::Partner => s.abandoned || s.presence_released[slot / PER_PI], + Kind::Liveness => s.posted[slot ^ 1] || !self.slot_running(slot ^ 1), + Kind::Scratch1 => self.scratch1_ready(s), + } + } + + /// Land `slot`'s withheld scratch-1 store, if one waits. + fn flush(&self, s: &mut State, slot: usize) { + if let Some((value, _)) = s.withheld[slot].take() { + if let Some(lb) = self.lb.upgrade() { + lb.set_scratch1(value); + } + s.scratch1_released = true; + } + } + + /// An access to scratch 1 from outside the node's processors: every + /// withheld store lands first, in the order its processor made it. + fn external(&self, s: &mut State) { + while let Some((oldest, _)) = (0..SLOTS).filter_map(|p| s.withheld[p].map(|(_, seq)| (p, seq))).min_by_key(|&(_, seq)| seq) { + self.flush(s, oldest); + } + } + + /// The calling processor's slot, if it may be held now. + fn park_slot(&self, s: &State) -> Option { + let slot = self.current_slot()?; + (self.applicable() && s.park[slot].is_none()).then_some(slot) + } + + fn start(&self, s: &mut State, slot: usize, kind: Kind) { + s.park[slot] = Some(Hold { kind, since: (self.clock)(), progress: self.progress(s, slot, kind) }); + dlog!(LogModule::Bedrock, "rendezvous: holding slot {} at the {kind:?} join", slot_name(slot)); + if let Some(cpu) = self.cpu(slot) { + cpu.slot.hold(HOLD_RENDEZVOUS); + } + self.arm(s); + } + + fn release(&self, s: &mut State, slot: usize, outcome: &str) { + let kind = s.park[slot].take().map(|h| h.kind); + // A withheld store lands before the processor runs again. + self.flush(s, slot); + dlog!(LogModule::Bedrock, "rendezvous: slot {} {kind:?} hold {outcome}", slot_name(slot)); + if let Some(cpu) = self.cpu(slot) { + cpu.slot.release(HOLD_RENDEZVOUS); + } + } + + fn recheck(&self, s: &mut State) { + for slot in 0..SLOTS { + if let Some(Hold { kind, .. }) = s.park[slot] { + if self.ready(s, slot, kind) { + self.release(s, slot, "satisfied"); + } + } + } + self.arm(s); + if self.finished(s) { + self.done.store(true, Ordering::Relaxed); + dlog!(LogModule::Bedrock, "rendezvous: the joins are over"); + } + } + + /// Every populated interface released its hub lock after publishing, or + /// the join was abandoned, with nothing held or withheld. + fn finished(&self, s: &State) -> bool { + let interfaces = (0..2).filter(|&i| self.interface_populated(i)).all(|i| s.abandoned || s.presence_released[i]); + interfaces && s.park.iter().all(Option::is_none) && s.withheld.iter().all(Option::is_none) + } + + /// Whether the joins are over; see [`Self::done`]. + fn over(&self) -> bool { + self.done.load(Ordering::Relaxed) + } + + /// While any slot is held, a timer checks the bounds every [`POLL`]. + fn arm(&self, s: &State) { + let Some(timers) = self.timers.get() else { return }; + let mut timer = self.timer.lock(); + if let Some(id) = timer.take() { + timers.remove(id); + } + if s.park.iter().all(Option::is_none) { + return; + } + *timer = Some(timers.add_one_shot(POLL, self.weak.clone(), |weak| { + if let Some(r) = weak.upgrade() { + r.expire(); + } + TimerReturn::Delete + })); + } + + /// Every hold past its bound ends: the processors it waits for have run + /// [`HOLD_CYCLES`] without coming, or [`SAFETY_NS`] of host time has + /// passed. A publisher's or partner's ends the join for this reset. + pub(crate) fn expire(&self) { + let now = (self.clock)(); + let mut s = self.state.lock(); + let mut abandon = false; + for slot in 0..SLOTS { + if let Some(hold) = s.park[slot] { + let ran = self.progress(&s, slot, hold.kind).saturating_sub(hold.progress); + let safety = now.saturating_sub(hold.since) >= SAFETY_NS; + if ran >= HOLD_CYCLES || safety { + abandon |= matches!(hold.kind, Kind::Presence | Kind::Partner); + let why = if safety { "timed out at the safety net" } else { "timed out" }; + dlog!(LogModule::Bedrock, "rendezvous: slot {} waited for {ran} cycles of {:?}", slot_name(slot), self.awaited(&s, slot, hold.kind).iter().map(|&p| slot_name(p)).collect::>()); + self.release(&mut s, slot, why); + } + } + } + if abandon { + s.abandoned = true; + } + self.recheck(&mut s); + } + + /// The PROM's write of II scratch 0, its interface's processors. + pub fn presence_published(&self) { + if self.over() { + return; + } + let Some(slot) = self.current_slot() else { return }; + dlog!(LogModule::Bedrock, "rendezvous: slot {} published its interface's processors", slot_name(slot)); + let mut s = self.state.lock(); + s.presence_seen[slot / PER_PI] = true; + self.recheck(&mut s); + } + + /// A store to `slot` of `interface`'s real-time compare register. + pub fn rt_compare_written(&self, interface: u8, slot: usize) { + if self.over() { + return; + } + let Some(cslot) = self.current_slot() else { return }; + let mut s = self.state.lock(); + // Only a processor's own register counts as its post, and only its + // first store after the reset barrier: earlier ones are progress codes. + if cslot != usize::from(interface) * PER_PI + slot || !s.joined[cslot] || s.posted[cslot] { + return; + } + s.posted[cslot] = true; + dlog!(LogModule::Bedrock, "rendezvous: slot {} posted its liveness", slot_name(cslot)); + self.recheck(&mut s); + if self.park_slot(&s) == Some(cslot) && !self.ready(&s, cslot, Kind::Liveness) { + self.start(&mut s, cslot, Kind::Liveness); + } + } + + /// A read of the real-time counter. + pub fn rt_counter_read(&self) { + if self.over() { + return; + } + let mut s = self.state.lock(); + let Some(cslot) = self.park_slot(&s) else { return }; + let interface = cslot / PER_PI; + let Some(publisher) = self.publisher(interface) else { return }; + if publisher == cslot || s.abandoned || s.presence_released[interface] || !s.posted[cslot] || !s.posted[publisher] { + return; + } + self.start(&mut s, cslot, Kind::Partner); + } + + /// Every hold released and every first store forgotten: a fresh start. + pub fn reset(&self) { + dlog!(LogModule::Bedrock, "rendezvous: reset"); + let mut s = self.state.lock(); + for slot in 0..SLOTS { + if s.park[slot].take().is_some() { + if let Some(cpu) = self.cpu(slot) { + cpu.slot.release(HOLD_RENDEZVOUS); + } + } + } + *s = State::default(); + self.done.store(false, Ordering::Relaxed); + if let (Some(timers), Some(id)) = (self.timers.get(), self.timer.lock().take()) { + timers.remove(id); + } + } + + /// Whether the joins are over, for tests. + #[cfg(test)] + pub(crate) fn is_over(&self) -> bool { + self.over() + } + + /// The slots held now, with their kinds, for tests. + #[cfg(test)] + pub(crate) fn held(&self) -> Vec<(usize, Kind)> { + let s = self.state.lock(); + (0..SLOTS).filter_map(|slot| s.park[slot].map(|h| (slot, h.kind))).collect() + } +} + +impl ScratchHooks for Rendezvous { + fn scratch1_write(&self, value: u64) { + if self.over() { + if let Some(lb) = self.lb.upgrade() { + lb.set_scratch1(value); + } + return; + } + let mut s = self.state.lock(); + let Some(slot) = self.current_slot() else { + // A test, a peer or a monitor: plain storage, after every + // withheld local store. + self.external(&mut s); + if let Some(lb) = self.lb.upgrade() { + lb.set_scratch1(value); + } + return; + }; + self.flush(&mut s, slot); + let store = |s: &mut State| { + if let Some(lb) = self.lb.upgrade() { + lb.set_scratch1(value); + } + s.scratch1_released = true; + }; + if !s.scratch1_seen[slot] { + if let Some(lb) = self.lb.upgrade() { + lb.set_scratch1(value); + } + s.scratch1_seen[slot] = true; + dlog!(LogModule::Bedrock, "rendezvous: slot {} made its first scratch 1 store", slot_name(slot)); + self.recheck(&mut s); + return; + } + // A second store before the barrier is over is the master's release + // of the others; every processor's join is its store after it. + if s.scratch1_released { + s.joined[slot] = true; + } + if self.park_slot(&s) != Some(slot) || self.scratch1_ready(&s) { + store(&mut s); + return; + } + s.seq += 1; + s.withheld[slot] = Some((value, s.seq)); + self.start(&mut s, slot, Kind::Scratch1); + } + + fn scratch1_read(&self) { + if !self.over() && self.current_slot().is_none() { + let mut s = self.state.lock(); + self.external(&mut s); + } + } + + fn scratch3_released(&self) { + if self.over() { + return; + } + let mut s = self.state.lock(); + // A publisher releasing the lock after its interface published: its + // partner, if held, may now read the bits. + if let Some(cslot) = self.current_slot() { + let interface = cslot / PER_PI; + if self.publisher(interface) == Some(cslot) && s.presence_seen[interface] && !s.presence_released[interface] { + s.presence_released[interface] = true; + dlog!(LogModule::Bedrock, "rendezvous: slot {} released the hub lock after publishing", slot_name(cslot)); + self.recheck(&mut s); + } + } + // The publisher waits here, with the lock free, for the other + // interface's publication. + let Some(cslot) = self.park_slot(&s) else { return }; + if self.publisher(cslot / PER_PI) != Some(cslot) || !s.presence_seen[cslot / PER_PI] || !self.other_pending(&s, cslot) { + return; + } + self.start(&mut s, cslot, Kind::Presence); + } +} + +fn slot_name(slot: usize) -> char { + (b'A' + slot as u8) as char +} + +#[cfg(test)] +mod tests; diff --git a/src/ip35/rendezvous/tests.rs b/src/ip35/rendezvous/tests.rs new file mode 100644 index 000000000..cc9d610df --- /dev/null +++ b/src/ip35/rendezvous/tests.rs @@ -0,0 +1,284 @@ +// Cases follow hw/misc/sgi-bedrock.c's holds and tests/qtest/sn1-bedrock-lb-test.c +// scratch1-second-write-no-hold in jamesbraid/qemu-sgi-origami. Each hold +// case fails with that hold removed. + +use super::*; +use crate::cpu::smp::{set_current_cpu, HOST_PARK_BIT}; +use crate::traits::BusDevice; +use std::sync::atomic::{AtomicU64, Ordering}; + +const SCRATCH1: u64 = 0x8008; +const SCRATCH3: u64 = 0x8018; + +struct Node { + r: Arc, + lb: Arc, + words: Vec>, + now: Arc, + /// Cycles each processor has retired. + cycles: Vec>, +} + +/// A node running `count` processors, with nothing between it and the LB. +fn node(count: usize) -> Node { + let now = Arc::new(AtomicU64::new(0)); + let clock: Clock = { + let now = now.clone(); + Arc::new(move || now.load(Ordering::SeqCst)) + }; + let lb = Arc::new(BedrockLb::new(clock.clone())); + let r = Rendezvous::new(Population::fitted_cpus(count).unwrap(), clock, &lb); + let words: Vec<_> = (0..count).map(|_| Arc::new(AtomicU64::new(0))).collect(); + let cycles: Vec<_> = (0..count).map(|_| Arc::new(AtomicU64::new(0))).collect(); + r.set_processors( + words + .iter() + .zip(&cycles) + .map(|(w, c)| { + let c = c.clone(); + Cpu { + slot: Arc::new(ParkSlot::new(Arc::as_ptr(w))), + progress: Box::new(move || c.load(Ordering::SeqCst)), + running: Box::new(|| true), + } + }) + .collect(), + ); + lb.connect_scratch_hooks(r.clone()); + Node { r, lb, words, now, cycles } +} + +impl Node { + /// Run `f` as processor `cpu`. + fn on(&self, cpu: u8, f: impl FnOnce(&Arc, &Arc) -> T + Send + 'static) -> T { + let (r, lb) = (self.r.clone(), self.lb.clone()); + std::thread::spawn(move || { + set_current_cpu(cpu); + f(&r, &lb) + }) + .join() + .unwrap() + } + fn store(&self, cpu: u8, offset: u64, value: u64) { + self.on(cpu, move |_, lb| lb.write64(offset, value)); + } + fn held(&self, cpu: usize) -> bool { + self.words[cpu].load(Ordering::SeqCst) & HOST_PARK_BIT != 0 + } + /// Scratch 1 as processor 1 reads it, which lands nothing withheld. + fn scratch1(&self) -> u64 { + self.on(1, |_, lb| lb.read64(SCRATCH1).data) + } + /// Every processor through the reset barrier: each clears scratch 1, + /// CPU 0 posts the release, and each makes the store after it that joins. + fn join(&self, count: u8) { + for cpu in 0..count { + self.store(cpu, SCRATCH1, 0); + } + self.store(0, SCRATCH1, 0x7f8_0000); + for cpu in 0..count { + self.store(cpu, SCRATCH1, 0x100 + u64::from(cpu)); + } + assert!(self.r.held().is_empty()); + } +} + +/// A store from no processor, as a qtest's, is stored at once however many +/// times it comes. Follows scratch1-second-write-no-hold. +#[test] +fn a_store_from_no_processor_is_never_withheld() { + let n = node(2); + for value in [0x7f8_0000, 0, 0x7f8_0000] { + n.lb.write64(SCRATCH1, value); + assert_eq!(n.scratch1(), value); + } + assert!(n.r.held().is_empty()); +} + +/// One processor alone is never held. +#[test] +fn one_processor_is_never_held() { + let n = node(1); + n.store(0, SCRATCH1, 0); + n.store(0, SCRATCH1, 0x7f8_0000); + assert_eq!(n.scratch1(), 0x7f8_0000); + assert!(n.r.held().is_empty() && !n.held(0)); +} + +/// The master's release posted before the other processor's clear is +/// withheld, and its writer held, until the clear lands; then the release +/// lands over it, so the clear cannot erase it. +#[test] +fn the_reset_barrier_lands_the_release_after_every_clear() { + let n = node(2); + n.store(0, SCRATCH1, 0); + n.store(0, SCRATCH1, 0x7f8_0000); + assert_eq!(n.r.held(), [(0, Kind::Scratch1)]); + assert!(n.held(0)); + assert_eq!(n.scratch1(), 0, "withheld"); + // CPU 1 (slot C) makes its first store, its clear. + n.store(1, SCRATCH1, 0); + assert!(n.r.held().is_empty() && !n.held(0)); + assert_eq!(n.scratch1(), 0x7f8_0000, "the release survives the clear"); +} + +/// A read from no processor sees every withheld store first. +#[test] +fn an_outside_read_lands_the_withheld_stores() { + let n = node(2); + n.store(0, SCRATCH1, 0); + n.store(0, SCRATCH1, 0x55); + assert_eq!(n.scratch1(), 0, "withheld from the other processor"); + assert_eq!(n.lb.read64(SCRATCH1).data, 0x55); + assert!(n.held(0), "the processor stays held"); +} + +/// The first of a pair to post its liveness status in its own compare +/// register is held until its partner posts. +#[test] +fn the_first_of_a_pair_to_post_waits_for_its_partner() { + let n = node(4); + n.join(4); + // CPU 0 (A) posts; its partner is CPU 2 (B). + n.on(0, |r, _| r.rt_compare_written(0, 0)); + assert_eq!(n.r.held(), [(0, Kind::Liveness)]); + // Another processor's register is not its post. + n.on(1, |r, _| r.rt_compare_written(0, 1)); + assert_eq!(n.r.held(), [(0, Kind::Liveness)]); + n.on(2, |r, _| r.rt_compare_written(0, 1)); + assert!(n.r.held().is_empty() && !n.held(0)); +} + +/// A store to the compare register before the barrier is a progress code, +/// not a post, and holds nothing. +#[test] +fn a_compare_store_before_the_join_is_not_a_post() { + let n = node(4); + n.on(0, |r, _| r.rt_compare_written(0, 0)); + assert!(n.r.held().is_empty()); +} + +/// Once both of a pair have posted, the partner's counter read holds it +/// until its publisher has published and released the hub lock. +#[test] +fn the_partner_waits_at_the_counter_for_its_publisher() { + let n = node(4); + n.join(4); + n.on(0, |r, _| r.rt_compare_written(0, 0)); + n.on(2, |r, _| r.rt_compare_written(0, 1)); + // The publisher's own counter read holds nothing. + n.on(0, |r, _| r.rt_counter_read()); + assert!(n.r.held().is_empty()); + n.on(2, |r, _| r.rt_counter_read()); + assert_eq!(n.r.held(), [(1, Kind::Partner)]); + n.on(0, |r, _| r.presence_published()); + assert_eq!(n.r.held(), [(1, Kind::Partner)], "published, lock still held"); + n.store(0, SCRATCH3, 0); + // The publisher now waits for PI1's publication, and the partner goes. + assert_eq!(n.r.held(), [(0, Kind::Presence)]); + assert!(n.held(0) && !n.held(2)); +} + +/// A publisher releasing the hub lock after its own publication is held +/// until the other populated interface has published. +#[test] +fn a_publisher_waits_for_the_other_interface() { + let n = node(2); + n.join(2); + n.on(0, |r, _| r.presence_published()); + n.store(0, SCRATCH3, 0); + assert_eq!(n.r.held(), [(0, Kind::Presence)]); + n.on(1, |r, _| r.presence_published()); + assert!(n.r.held().is_empty() && !n.held(0)); + // CPU 1 releases with nothing left to wait for. + n.store(1, SCRATCH3, 0); + assert!(n.r.held().is_empty()); +} + +/// The joins are over once every populated interface has released its hub +/// lock after publishing: from then on a scratch 1 store is plain storage and +/// nothing is held, until a reset starts them again. +#[test] +fn the_joins_end_once_every_interface_has_released_its_lock() { + let n = node(2); + n.join(2); + n.on(0, |r, _| r.presence_published()); + n.store(0, SCRATCH3, 0); + n.on(1, |r, _| r.presence_published()); + assert!(!n.r.is_over(), "PI1 still holds its lock"); + n.store(1, SCRATCH3, 0); + assert!(n.r.is_over()); + n.store(1, SCRATCH1, 0x55); + assert_eq!(n.scratch1(), 0x55, "a store after the joins lands at once"); + n.on(1, |r, _| r.rt_counter_read()); + assert!(n.r.held().is_empty()); + n.r.reset(); + assert!(!n.r.is_over(), "a reset starts the joins again"); +} + +/// A hold ends once the processors it waits for have run their bound +/// without coming; a publisher's that runs out ends the join, so later +/// releases hold nothing. +#[test] +fn every_hold_ends_at_its_bound() { + let n = node(2); + n.join(2); + n.on(0, |r, _| r.presence_published()); + n.store(0, SCRATCH3, 0); + // CPU 0 is a publisher; CPU 1 sits on the other interface. + n.cycles[1].store(HOLD_CYCLES - 1, Ordering::SeqCst); + n.r.expire(); + assert!(n.held(0)); + n.cycles[1].store(HOLD_CYCLES, Ordering::SeqCst); + n.r.expire(); + assert!(n.r.held().is_empty() && !n.held(0)); + n.store(0, SCRATCH3, 0); + assert!(n.r.held().is_empty(), "the join is over"); +} + +/// A processor the host does not run retires nothing, so no amount of host +/// time up to the safety net ends a hold waiting for it. Fails with a bound +/// in host time, which let a starved thread drop out of the PROM's count. +#[test] +fn a_starved_processor_cannot_run_a_hold_out() { + let n = node(2); + n.join(2); + n.on(0, |r, _| r.presence_published()); + n.store(0, SCRATCH3, 0); + n.now.store(10 * 2_000_000_000, Ordering::SeqCst); + n.r.expire(); + assert!(n.held(0), "ten seconds of host time with no progress is no timeout"); + n.now.store(SAFETY_NS, Ordering::SeqCst); + n.r.expire(); + assert!(!n.held(0), "the safety net still ends a hold"); +} + +/// Cycles run by processors the hold does not wait for do not count. +#[test] +fn only_the_processors_waited_for_count() { + let n = node(4); + n.join(4); + n.on(0, |r, _| r.rt_compare_written(0, 0)); + assert_eq!(n.r.held(), [(0, Kind::Liveness)]); + // Slot A waits for its partner B (CPU 2) alone. + n.cycles[1].store(HOLD_CYCLES * 2, Ordering::SeqCst); + n.cycles[3].store(HOLD_CYCLES * 2, Ordering::SeqCst); + n.r.expire(); + assert!(n.held(0)); + n.cycles[2].store(HOLD_CYCLES, Ordering::SeqCst); + n.r.expire(); + assert!(!n.held(0)); +} + +/// A reset lets every held processor go and forgets the barrier. +#[test] +fn a_reset_releases_every_hold() { + let n = node(2); + n.store(0, SCRATCH1, 0); + n.store(0, SCRATCH1, 1); + assert!(n.held(0)); + n.r.reset(); + assert!(!n.held(0) && n.r.held().is_empty()); + n.store(0, SCRATCH1, 0); + assert_eq!(n.scratch1(), 0, "a first store again"); +} diff --git a/src/ip35/tests.rs b/src/ip35/tests.rs new file mode 100644 index 000000000..29b9d1809 --- /dev/null +++ b/src/ip35/tests.rs @@ -0,0 +1,999 @@ +use super::*; +use crate::traits::{BusDevice, BUS_OK}; + +struct Word(u64); +impl BusDevice for Word { + fn read64(&self, _: u64) -> BusRead64 { + BusRead64::ok(self.0) + } +} + +#[derive(Clone, Debug, PartialEq, Eq)] +enum NarrowWriteCall { + Byte(u64, u8), + Half(u64, u16), + From(usize, u64, u64), +} +/// Records every narrow write it sees. With `sees_register` it overrides +/// `write_narrow_from` and keeps the whole source register, as Bedrock's ECC +/// view does; otherwise it takes the trait's default. +#[derive(Default)] +struct NarrowWriteSpy { + sees_register: bool, + calls: std::sync::Mutex>, +} +impl NarrowWriteSpy { + fn new(sees_register: bool) -> Arc { + Arc::new(Self { sees_register, ..Default::default() }) + } + fn calls(&self) -> Vec { + self.calls.lock().unwrap().clone() + } +} +impl BusDevice for NarrowWriteSpy { + fn write8(&self, address: u64, value: u8) -> u32 { + self.calls.lock().unwrap().push(NarrowWriteCall::Byte(address, value)); + BUS_OK + } + fn write16(&self, address: u64, value: u16) -> u32 { + self.calls.lock().unwrap().push(NarrowWriteCall::Half(address, value)); + BUS_OK + } + fn write_narrow_from(&self, address: u64, size: usize, gpr: u64) -> u32 { + if !self.sees_register { + return if size == 1 { self.write8(address, gpr as u8) } else { self.write16(address, gpr as u16) }; + } + self.calls.lock().unwrap().push(NarrowWriteCall::From(size, address, gpr)); + BUS_OK + } +} + +/// An uncached SB or SH reaches a device with the whole source register, at the +/// device-local address; only a device that overrides the call sees more than +/// the low bits, and a window outside every map is a bus error. +/// The Fuel's processor is the R14000 at revision 2.3, with the Octane +/// R14000's caches; the Octane's own part is revision 2.4. +#[test] +fn the_fuel_runs_the_r14000_at_revision_2_3() { + use crate::cpu::mips_cache_shadow::{R14000FuelShadowCache, R14000_PRID}; + use crate::cpu::mips_cache_v2::{CpuModel, MipsCache}; + assert_eq!(::PRID, 0x0f23); + assert_eq!(R14000_PRID, 0x0f24); + assert_eq!(::L2_SIZE, 2 << 20); + // Typed R12000, it takes the R10000 family's behaviour: reserved + // secondary-tag bits read as zero, and a refused uncached store posts. + assert!(::R10K_FAMILY); + assert_ne!(::SECONDARY_TAG_MASK, u64::MAX); + assert!(::POSTS_UNCACHED_STORES); + assert!(::PARTIAL_LWR_KEEPS_UPPER); +} + +#[test] +fn narrow_stores_carry_the_register_to_devices_that_ask_for_it() { + let mut bus = Ip35Bus::new(Physical::new(Vec::new())); + let plain = NarrowWriteSpy::new(false); + let ecc = NarrowWriteSpy::new(true); + bus.map(3, 0x1000, 0x1000, plain.clone()); + bus.map(3, 0x2000, 0x1000, ecc.clone()); + + let source = 0x1122_3344_5566_77a5u64; + let uc = 3u64 << 57; + assert_eq!(bus.write_narrow_from(uc | 0x1003, 1, source), BUS_OK); + assert_eq!(bus.write_narrow_from(uc | 0x1004, 2, source), BUS_OK); + assert_eq!( + plain.calls(), + vec![NarrowWriteCall::Byte(3, source as u8), NarrowWriteCall::Half(4, source as u16)], + "an ordinary window gets the architectural low bits", + ); + + assert_eq!(bus.write_narrow_from(uc | 0x2004, 2, source), BUS_OK); + assert_eq!(ecc.calls(), vec![NarrowWriteCall::From(2, 4, source)]); + + assert_eq!(bus.write_narrow_from(uc | 0x8000, 2, source), BUS_ERR, "no window there"); +} + +/// Equal physical offsets in different XKPHYS access spaces reach different devices. +#[test] +fn uc_spaces_keep_equal_offsets_distinct() { + let mut bus = Ip35Bus::new(Physical::new(Vec::new())); + for space in 0..4 { + bus.map( + space, + 0x0100_0000, + 0x1000, + Arc::new(Word(0x100 + space as u64)), + ); + } + for space in 0..4 { + let value = bus.read64(((space as u64) << 57) | 0x0100_0000); + assert_eq!(value.status, BUS_OK); + assert_eq!(value.data, 0x100 + space as u64); + } +} + +/// The processor must retain both XKPHYS access-space bits through translation. +#[test] +fn xkphys_uc_bits_survive_the_board_cpu_mask() { + // MipsExecutor and the TLB's inline arrays exceed the test worker's stack. + std::thread::Builder::new() + .stack_size(4 * 1024 * 1024) + .spawn(|| { + let bus = Arc::new(Ip35Bus::new(Physical::new(Vec::new()))); + let mut cpu = crate::ip34_machine::build_cpu(bus); + cpu.core.write_cp0( + 12, + (cpu.core.cp0_status | crate::cpu::mips_core::STATUS_KX) as u64, + ); + for raw_uc in 0..4u64 { + let va = 0x9000_0000_0100_0020 | (raw_uc << 57); + for _ in 0..2 { + let translated = cpu.debug_translate(va); + assert_eq!(translated.status, crate::cpu::mips_exec::TR_UNCACHED); + assert_eq!(translated.phys, (raw_uc << 57) | 0x0100_0020); + } + } + }) + .expect("spawn") + .join() + .expect("thread panicked"); +} + +/// RAM placed above the first GiB must remain shared by cached, uncached and DMA paths. +#[test] +fn cached_and_uncached_memory_keep_high_addresses() { + let board = crate::ip34_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + assert_eq!(board.md.md_bus().write64(8, 0x33), BUS_OK); + let at = (1 << 30) + 0x1000; + assert_eq!(board.md.memory_view().write64(at, 0x12345678), BUS_OK); + assert_eq!(board.cached_bus().read64(at).data, 0x12345678); + assert_eq!(board.bus.read64((3 << 57) | at).data, 0x12345678); + assert_eq!(board.bus.read64(at).status, BUS_ERR); + assert_eq!(board.md.dma_view().read64(at).data, 0x12345678); +} + +/// Memory diagnostics must notify both processor interfaces and refuse bad uncached data. +#[test] +fn memory_errors_reach_both_interfaces_and_uncached_reads_fail() { + let board = crate::ip34_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + let correctable = 1 << 61; + board.pi[0].write64(0xb0, correctable); + board.md.md_bus().write64(8, 0x33); + for directory in [false, true] { + if directory { + board.md.md_bus().write64(0x40000, 2); + board.md.directory_bus().write64(0, 1 << 16); + board.md.md_bus().write64(0x40000, 0); + } else { + board.md.ecc_bus().write8(7, 1); + } + assert_eq!(board.bus.read64(0).status, BUS_OK); + for pi in &board.pi { + assert_eq!(pi.read64(0xa0).data & correctable, correctable); + } + assert_eq!(board.irq.bits() & (1 << 11), 1 << 11); + for pi in &board.pi { + pi.write64(0x90, 64 + 61); + assert_eq!(pi.read64(0xa0).data & correctable, 0); + } + assert_eq!(board.irq.bits() & (1 << 11), 0); + } + + let board = crate::ip34_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + board.md.md_bus().write64(8, 0x33); + board.md.ecc_bus().write8(7, 3); + for selector in [0, 3] { + assert_eq!(board.bus.read64(selector << 57).status, BUS_ERR); + } + assert_eq!(board.cached_bus().read64(0).status, BUS_OK); + assert_eq!(board.md.dma_view().read64(0).status, BUS_OK); +} + +/// Firmware resets the PCI link before reinitializing the attached storage controller. +#[test] +fn fuel_link_reset_resets_the_attached_scsi_engine() { + use crate::dev::isp1040::{mbox, reg, CTRL_INT_ENABLE, CTRL_RISC_INT_ENABLE, HOST_CMD_SHIFT}; + use crate::pci::PciBusDevice; + let board = crate::ip34_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + let isp = &board.scsi; + isp.config().write(1, 6); + isp.config().write(5, 0x02000000); + isp.bar_write(1, reg::BUS_CTRL, u32::from(CTRL_INT_ENABLE | CTRL_RISC_INT_ENABLE), 2); + isp.bar_write(1, reg::MAILBOX_BASE, u32::from(mbox::NOP), 2); + isp.bar_write(1, reg::HOST_CMD, 0x5 << HOST_CMD_SHIFT, 2); + assert_eq!(isp.bar_read(1, reg::MAILBOX_BASE, 2), u32::from(mbox::COMPLETE)); + assert_ne!(isp.bar_read(1, reg::BUS_CTRL, 2), 0); + assert_eq!(board.xbridge.widget(0).unwrap().write32(0x2f4, 1), BUS_OK); + assert_eq!(isp.config().command(), 0); + assert_eq!(isp.config().bar_base(1), 0); + assert_eq!(isp.bar_read(1, reg::BUS_CTRL, 2), 0); + assert_eq!(isp.bar_read(1, reg::MAILBOX_BASE, 2), 0); + assert_eq!(isp.bar_read(1, reg::MAILBOX_BASE + 2, 2), 0x4953); +} + +/// The PROM reads the station address from the L1 and IRIX from the +/// Ethernet; both must report the configured one. +#[test] +fn the_configured_station_address_reaches_the_l1_and_the_ethernet() { + let mac = [0x08, 0x00, 0x69, 0x21, 0x43, 0x65]; + let board = crate::ip34_machine::FuelBoard::with_config(crate::ip34_machine::FuelBoardConfig { + payload: &[], clock: Arc::new(|| 0), flash_path: None, + station_mac: mac, + timekeeper: Arc::new(crate::dev::timekeeper::Timekeeper::fuel().unwrap()), + ps2_input: true, usb_input: true, + l1: Default::default(), + graphics: crate::config::GraphicsBoard::None, + }).unwrap(); + assert_eq!(board.l1.station_mac(), mac); + assert_eq!(board.eth.mac(), mac); +} + +/// The L1's reported revision and brick serial keep their defaults unless +/// the board is built with replacements. +#[test] +fn the_l1_identity_overrides_reach_the_controller() { + use crate::ip34_machine::{FuelBoard, FuelBoardConfig, L1Identity}; + let build = |l1| FuelBoard::with_config(FuelBoardConfig { + payload: &[], clock: Arc::new(|| 0), flash_path: None, + station_mac: [8, 0, 0x69, 0x12, 0x34, 0x56], + timekeeper: Arc::new(crate::dev::timekeeper::Timekeeper::fuel().unwrap()), + ps2_input: true, usb_input: false, l1, + graphics: crate::config::GraphicsBoard::None, + }).unwrap().l1.config(); + let stock = build(L1Identity::default()); + assert_eq!(stock.revision, Some([1, 48, 1])); + assert_eq!(stock.brick_serial.as_deref(), Some("EMUN00")); + let set = build(L1Identity { revision: Some([2, 3, 4]), brick_serial: Some("EMUN09".into()) }); + assert_eq!(set.revision, Some([2, 3, 4])); + assert_eq!(set.brick_serial.as_deref(), Some("EMUN09")); +} + +/// The Fuel's cache reports primary data tag stores to the memory controller: +/// the PROM's installs over its window start the private page, and its sweep +/// ends it. Other caches' tag stores and non-installs do nothing. +#[test] +fn the_prom_tag_stores_reach_the_memory_controller_through_the_cpu_cache() { + // An executor is large; a test thread's stack is not. + std::thread::Builder::new().stack_size(64 << 20).spawn(prom_tag_stores).unwrap().join().unwrap(); +} + +fn prom_tag_stores() { + use crate::cpu::mips_cache_v2::MipsCache; + use crate::cpu::mips_isa::{CACH_PD, CACH_PI, C_IST}; + let board = crate::ip34_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + let cpu = board.build_cpu(); + let window = |at: u64| board.md.physical().read64(at).status == BUS_OK; + assert!(!window(0x10_0000)); + cpu.cache.cache_op(C_IST | CACH_PD, 0xa800_0000_0010_0000, 0xffff_ffcf); + cpu.cache.cache_op(C_IST | CACH_PI, 0xa800_0000_0010_0000, 0x100c0); + assert!(!window(0x10_0000), "a pattern and an instruction cache tag start nothing"); + cpu.cache.cache_op(C_IST | CACH_PD, 0xa800_0000_0010_0000, 0x100c0); + assert!(window(0x10_0000)); + cpu.cache.cache_op(C_IST | CACH_PD, 0xa800_0000_0010_0000, 0); + assert!(!window(0x10_0000)); +} + +fn fuel_dma_board() -> crate::ip34_machine::FuelBoard { + let board = crate::ip34_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + assert_eq!(board.md.md_bus().write64(8, 0x33), BUS_OK); + board.xbridge.head(15).write32(0x24, 15); + board.xbridge.set_wide_slot_mask(15, 0); + board +} + +fn fuel_dma_address(address: u64) -> u64 { + (10 << 60) | (1 << 55) | address +} + +/// Storage DMA buffers can begin unaligned and cross a physical page boundary. +#[test] +fn dma_preserves_byte_order_and_crosses_ragged_pages() { + use crate::pci::PciDmaStatus; + let board = fuel_dma_board(); + let dma = board.xbridge.dma(15, 0); + let memory = board.md.memory_view(); + for (address, length) in [(0x1030, 8), (0x1103, 27), (0x1ffb, 19)] { + let data: Vec = (0..length).map(|i| 0x31 + i as u8).collect(); + assert_eq!(dma.dma_write(fuel_dma_address(address), &data), PciDmaStatus::Ok); + for (i, &byte) in data.iter().enumerate() { + assert_eq!(memory.read8(address + i as u64).data, byte); + } + let mut back = vec![0xff; length]; + assert_eq!(dma.dma_read(fuel_dma_address(address), &mut back), PciDmaStatus::Ok); + assert_eq!(back, data); + } + // Without byte swapping enabled, the bridge reverses each 32-bit word. + assert_eq!(dma.dma_write((10 << 60) | 0x3000, &[0, 1, 2, 3, 4, 5, 6, 7]), PciDmaStatus::Ok); + assert_eq!(memory.read64(0x3000).data, 0x0302010007060504); +} + +/// A failed DMA read must clear only refused chunks and preserve completed data. +#[test] +fn dma_refusals_clear_failed_reads_without_losing_completed_chunks() { + use crate::pci::PciDmaStatus; + let board = fuel_dma_board(); + let dma = board.xbridge.dma(15, 0); + let memory = board.md.memory_view(); + // Bank zero has 256 MiB fitted. The second qword is outside its backing. + let last = 0x0fff_fff8; + assert_eq!(memory.write64(last, 0x0102030405060708), BUS_OK); + let mut bytes = [0xff; 16]; + assert_eq!(dma.dma_read(fuel_dma_address(last), &mut bytes), PciDmaStatus::MasterAbort); + assert_eq!(bytes, [1, 2, 3, 4, 5, 6, 7, 8, 0, 0, 0, 0, 0, 0, 0, 0]); + assert_eq!(dma.dma_write(fuel_dma_address(last), &[0x42; 16]), PciDmaStatus::MasterAbort); + assert_eq!(memory.read64(last).data, 0x4242424242424242); + board.io.disable_device(15, 0); + bytes.fill(0xff); + assert_eq!(dma.dma_read(fuel_dma_address(last), &mut bytes), PciDmaStatus::MasterAbort); + assert_eq!(bytes, [0; 16]); + assert_eq!(dma.dma_write(fuel_dma_address(last), &[0x99; 8]), PciDmaStatus::MasterAbort); + assert_eq!(memory.read64(last).data, 0x4242424242424242); +} + +/// A wide DMA leg must report an uncorrectable word once without publishing bad bytes. +#[test] +fn fuel_dma_checks_an_aligned_uncorrectable_word_once() { + use crate::dev::bedrock_md::MemoryError; + use crate::pci::PciDmaStatus; + use std::sync::atomic::{AtomicU64, Ordering}; + let board = fuel_dma_board(); + assert_eq!(board.md.memory_view().write64(0, 0), BUS_OK); + assert_eq!(board.md.ecc_bus().write8(7, 3), BUS_OK); + let count = Arc::new(AtomicU64::new(0)); + let seen = count.clone(); + let md = Arc::downgrade(&board.md); + board.md.on_error(Arc::new(move |kind| { + assert_eq!(kind, MemoryError::Uncorrectable); + assert_eq!(md.upgrade().unwrap().current_cell(), u64::MAX); + seen.fetch_add(1, Ordering::Relaxed); + })); + // The former callback read the same persistent UCE once for each byte. + for address in 0..8 { + assert_eq!(board.io.inbound_read(15, Some(0), address, 1).status, BUS_OK); + } + assert_eq!(count.swap(0, Ordering::Relaxed), 8); + assert_ne!(board.md.md_bus().read64(0x60018).data & (1 << 61), 0); + let mut bytes = [0xff; 8]; + assert_eq!(board.xbridge.dma(15, 0).dma_read(fuel_dma_address(0), &mut bytes), PciDmaStatus::Ok); + assert_eq!(bytes, [0; 8]); + assert_eq!(count.load(Ordering::Relaxed), 1); + let record = board.md.md_bus().read64(0x60010).data; + assert_ne!(record & (1 << 60), 0); + assert_eq!(record & (1 << 61), 0); + assert_eq!(board.md.uncached_view().read64(0).status, BUS_ERR); +} + +/// Inbound Hub writes must preserve access width and the requesting device identity. +#[test] +fn dma_hub_access_preserves_width_and_source_permissions() { + use crate::pci::PciDmaStatus; + let board = fuel_dma_board(); + let dma = board.xbridge.dma(15, 0); + let hub = (1 << 47) | 0x1800000; + for (interface, offset) in [(0, 0), (1, 0x200000)] { + let address = fuel_dma_address(hub + offset + 0x90); + assert_eq!(dma.dma_write(address, &0x10cu64.to_be_bytes()), PciDmaStatus::Ok); + assert_ne!(board.pi[interface].read64(0x98).data & (1 << 12), 0); + for width in [1, 2, 4] { + assert_eq!(dma.dma_write(address, &[0; 8][..width]), PciDmaStatus::MasterAbort); + assert_ne!(board.pi[interface].read64(0x98).data & (1 << 12), 0); + } + assert_eq!(dma.dma_write(address, &12u64.to_be_bytes()), PciDmaStatus::Ok); + assert_eq!(board.pi[interface].read64(0x98).data & (1 << 12), 0); + } + let address = fuel_dma_address(hub + 0x90); + // I/O modify must retain PI region protection and requesting-slot identity. + assert_eq!(board.pi[0].write64(0x10, 0), BUS_OK); + assert_eq!(dma.dma_write(address, &0x10cu64.to_be_bytes()), PciDmaStatus::MasterAbort); + assert_eq!(board.pi[0].read64(0x98).data & (1 << 12), 0); + assert_eq!(board.pi[0].write64(0x10, 1), BUS_OK); + board.io.disable_device(15, 0); + assert_eq!(dma.dma_write(address, &0x10cu64.to_be_bytes()), PciDmaStatus::MasterAbort); + assert_eq!(board.pi[0].read64(0x98).data & (1 << 12), 0); + assert_eq!(board.xbridge.dma(15, 1).dma_write(address, &0x10cu64.to_be_bytes()), PciDmaStatus::Ok); + assert_ne!(board.pi[0].read64(0x98).data & (1 << 12), 0); + let mut bytes = [0xff; 8]; + assert_eq!(board.xbridge.dma(15, 1).dma_read( + fuel_dma_address(hub + 0x40), &mut bytes, + ), PciDmaStatus::MasterAbort); + assert_eq!(bytes, [0; 8]); + assert_eq!(board.bus.read64((1 << 57) | 0x1800040).status, BUS_OK); +} + +/// Rounded MMIO pages must not expose padding or bypass cell permissions. +#[test] +fn exact_windows_reject_padding_and_revoked_permissions() { + use std::sync::atomic::{AtomicBool, Ordering}; + struct Offset; + impl BusDevice for Offset { + fn read64(&self, offset: u64) -> BusRead64 { BusRead64::ok(offset) } + fn write64(&self, _offset: u64, _value: u64) -> u32 { BUS_OK } + } + let permitted = Arc::new(AtomicBool::new(true)); + let permission = permitted.clone(); + let window = Window::new(0x1000, 0x88, Arc::new(Offset)) + .permitted(Arc::new(move || permission.load(Ordering::Acquire))); + assert_eq!(window.read64(0x1080).data, 0x80); + assert_eq!(window.read64(0x1088).status, BUS_ERR); + assert_eq!(window.read64(0x1084).status, BUS_ERR); + assert_eq!(window.read64(0x0ff8).status, BUS_ERR); + assert_eq!(window.read64(u64::MAX).status, BUS_ERR); + permitted.store(false, Ordering::Release); + assert_eq!(window.read64(0x1000).status, BUS_ERR); + assert_eq!(window.write64(0x1000, 1), BUS_ERR); + permitted.store(true, Ordering::Release); + assert_eq!(window.write64(0x1000, 1), BUS_OK); +} + +/// Board default setup must preserve the direct RAM path and its shared DMA contents. +#[test] +fn direct_cached_banks_remain_visible_after_board_defaults_are_installed() { + let board = crate::ip34_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + assert_eq!(board.md.md_bus().write64(8, 0x33), BUS_OK); + let cached = board.cached_bus(); + assert!(cached.mem_ptr(0x2000).is_some()); + assert_eq!(cached.write64(0x2000, 0x123456789abcdef0), BUS_OK); + assert_eq!(board.md.dma_view().read64(0x2000).data, 0x123456789abcdef0); + assert!(cached.mem_ptr(3 << 57).is_none(), "UC selectors cannot alias cached RAM"); +} + +/// A device overlay replaces its pages, including padding, while its neighbors +/// keep their original address translation. +#[test] +fn sparse_overlays_keep_neighbors_and_refuse_the_new_windows_padding() { + let mut map = SparseWindowMap::default(); + map.map(0x1000, 0x4000, Arc::new(Word(1))); + map.map(0x2040, 0x80, Arc::new(Word(2))); + assert_eq!(map.read64(0x1000).data, 1); + assert_eq!(map.read64(0x2040).data, 2); + assert_eq!(map.read64(0x20b8).data, 2); + assert_eq!(map.read64(0x20bc).status, BUS_ERR); + assert_eq!(map.read64(0x2000).status, BUS_ERR); + assert_eq!(map.read64(0x20c0).status, BUS_ERR); + assert_eq!(map.read64(0x3000).data, 1); + assert_eq!(map.read64(0x5000).status, BUS_ERR); + map.map(0x1000, 0x4000, Arc::new(Word(3))); + assert_eq!(map.ranges.len(), 1); + assert_eq!(map.read64(0x2040).data, 3); +} + +/// Sparse MMIO holes remain bus errors and a revoked cell permission does not +/// fall through to an older mapping beneath the protected window. +#[test] +fn sparse_holes_and_revoked_permissions_never_expose_older_devices() { + use std::sync::atomic::{AtomicBool, Ordering}; + let permission = Arc::new(AtomicBool::new(true)); + let allowed = permission.clone(); + let mut map = SparseWindowMap::default(); + map.map(0, 0x3000, Arc::new(Word(1))); + map.map(0x1000, 8, Arc::new(Window::new(0, 8, Arc::new(Word(2))) + .permitted(Arc::new(move || allowed.load(Ordering::Acquire))))); + assert_eq!(map.read64(0x1000).data, 2); + permission.store(false, Ordering::Release); + assert_eq!(map.read64(0x1000).status, BUS_ERR); + assert_eq!(map.read64(0x1008).status, BUS_ERR); + assert_eq!(map.read64(0x2000).data, 1); + assert_eq!(map.read64(0x3000).status, BUS_ERR); +} + +/// Code-page lookup receives plain physical addresses, while tagged uncached +/// requests retain their view. Cached stores must also update the MD view. +#[cfg(feature = "jitv2")] +#[test] +fn code_generations_use_the_cached_plane_and_stable_ram_endpoint() { + struct Generations { + ram_only: bool, + generation: AtomicU64, + writes: std::sync::Mutex>, + } + impl Generations { + fn new(ram_only: bool) -> Arc { + Arc::new(Self { ram_only, generation: AtomicU64::new(0), writes: Default::default() }) + } + } + impl BusDevice for Generations { + fn gen_ptr(&self, address: u64) -> *const AtomicU64 { + if !self.ram_only || address < 0x1000 { &self.generation } else { std::ptr::null() } + } + fn write64(&self, address: u64, value: u64) -> u32 { + self.writes.lock().unwrap().push((address, value)); + BUS_OK + } + fn write64_masked(&self, address: u64, value: u64, mask: u64) -> u32 { + self.write64(address, value & mask) + } + fn write_block(&self, address: u64, values: &[u64]) -> u32 { + for (index, value) in values.iter().enumerate() { + self.write64(address + index as u64 * 8, *value); + } + BUS_OK + } + } + let physical = Physical::new(Vec::new()); + let mut bus = Ip35Bus::new(physical.clone()); + let raw_ram = Generations::new(false); + let stable_ram = Generations::new(true); + let rom = Generations::new(false); + map_device(&physical, 0, 0x1000, raw_ram.clone(), "raw RAM generation"); + map_device(&physical, 0x1fc00000, 0x1000, rom.clone(), "ROM generation"); + bus.set_ram_generations(stable_ram.clone()); + bus.map(3, 0, 0x1000, stable_ram.clone()); + bus.map(3, 0x3000, 8, stable_ram.clone()); + let cached = bus.cached_view(stable_ram.clone(), crate::dev::bedrock_mem::RamExtent::new(&[256, 256])); + assert_eq!(bus.gen_ptr(0x40), &stable_ram.generation as *const AtomicU64); + assert_eq!(cached.gen_ptr(0x40), &stable_ram.generation as *const AtomicU64); + assert_eq!(bus.gen_ptr(0x1fc00000), &rom.generation as *const AtomicU64); + assert_eq!(cached.gen_ptr(0x1fc00000), &rom.generation as *const AtomicU64); + assert_eq!(bus.gen_ptr((3 << 57) | 0x40), &stable_ram.generation as *const AtomicU64); + assert_eq!(bus.gen_ptr((3 << 57) | 0x3000), &stable_ram.generation as *const AtomicU64); + assert!(bus.gen_ptr((3 << 57) | 0x3008).is_null()); + assert!(bus.gen_ptr((1 << 57) | 0x1fc00000).is_null()); + assert!(bus.gen_ptr(1 << 50).is_null()); + assert_eq!(cached.write64(0x40, 0x1234), BUS_OK); + assert_eq!(cached.write_block(0x48, &[0x5678]), BUS_OK); + assert_eq!(cached.write64_masked(0x50, 0xaabb, 0xff), BUS_OK); + assert_eq!(*stable_ram.writes.lock().unwrap(), [(0x40, 0x1234), (0x48, 0x5678), (0x50, 0xbb)]); + assert_eq!(cached.write64(0x1fc00000, 0x4321), BUS_OK); + assert_eq!(*rom.writes.lock().unwrap(), [(0, 0x4321)], "the ROM window translates to a device-local offset"); + assert!(raw_ram.writes.lock().unwrap().is_empty()); +} + +/// Unmodeled Hub heads retain the PROM compatibility readback without +/// absorbing the adjacent, independently implemented L1 UART window. +#[test] +fn compatibility_hub_heads_do_not_hide_the_l1_uart_window() { + let board = crate::ip34_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + assert_ne!(board.md.current_cell() & 1, 0); + for base in [0x1000_0000, 0x3000_0000] { + for offset in [0, 8, 0x40, 0x78] { + let address = base + offset; + assert_eq!(board.bus.write8(address, 0xa5), BUS_OK); + assert_eq!(board.bus.write16(address, 0xa5a5), BUS_OK); + assert_eq!(board.bus.write32(address, 0xa5a5a5a5), BUS_OK); + assert_eq!(board.bus.write64(address, 0x2000), BUS_OK); + let value = board.bus.read8(address); + assert_eq!(value.status, BUS_OK); + assert_eq!(value.data, 0); + let value = board.bus.read16(address); + assert_eq!(value.status, BUS_OK); + assert_eq!(value.data, 0); + let value = board.bus.read32(address); + assert_eq!(value.status, BUS_OK); + assert_eq!(value.data, 0); + let value = board.bus.read64(address); + assert_eq!(value.status, BUS_OK); + assert_eq!(value.data, 0); + } + assert_eq!(board.bus.read64(base + 0x80).status, BUS_OK); + } +} + +/// A busy host port or unusable transcript path must not prevent a serial +/// machine from starting, including when any of its three ports serves CI. +#[test] +fn unavailable_serial_ports_and_logs_do_not_abort_the_machine() { + use crate::config::{FuelConsole, MachineConfig}; + let listeners: Vec<_> = (0..3).map(|_| std::net::TcpListener::bind(("127.0.0.1", 0)).unwrap()).collect(); + let ports: Vec<_> = listeners.iter().map(|socket| socket.local_addr().unwrap().port()).collect(); + let board = crate::ip34_machine::FuelBoard::new(&[], Arc::new(|| 0)).unwrap(); + let mut config = MachineConfig::default(); + config.fuel.l1_port = Some(ports[0]); + config.serial_port_a = Some(ports[1]); + config.serial_port_b = Some(ports[2]); + config.ci = true; + let missing = std::env::temp_dir().join(format!("iris-fuel-missing-log-{}", std::process::id())).join("absent").join("serial.txt"); + assert!(!missing.parent().unwrap().exists(), "the log parent must be absent"); + config.serial_log = Some(missing.to_string_lossy().into_owned()); + config.fuel.l1_log = config.serial_log.clone(); + for (console, selection) in [FuelConsole::L1, FuelConsole::Ioc3A, FuelConsole::Ioc3B].into_iter().enumerate() { + config.fuel.console = selection; + crate::ip34_machine::FuelMachine::connect_serial_for_test(&config, &board, console).unwrap(); + } + config.ci = false; + crate::ip34_machine::FuelMachine::connect_serial_for_test(&config, &board, 0).unwrap(); +} + +/// The Fuel's OHCI root ports each carry a low-speed device, and host +/// input reaches them through the IOC3's PS/2 controller object; the USB +/// pair is fitted only when asked for. +#[test] +fn the_fuel_plugs_a_usb_keyboard_and_mouse_only_when_configured_on() { + use crate::pci::PciBusDevice; + let ports = |usb_input| { + let board = crate::ip34_machine::FuelBoard::with_config(crate::ip34_machine::FuelBoardConfig { + payload: &[], clock: Arc::new(|| 0), flash_path: None, + station_mac: [8, 0, 0x69, 0x12, 0x34, 0x56], + timekeeper: Arc::new(crate::dev::timekeeper::Timekeeper::fuel().unwrap()), + ps2_input: true, usb_input, + l1: Default::default(), + graphics: crate::config::GraphicsBoard::None, + }).unwrap(); + // Power each port: one with a device shows it connected. + board.usb.bar_write(0, 0x54, 0x100, 4); + board.usb.bar_write(0, 0x58, 0x100, 4); + let status = [0x54, 0x58].map(|at| board.usb.bar_read(0, at, 4)); + (board, status) + }; + let (board, status) = ports(true); + for s in status { + assert_eq!(s & 1, 1, "device connected"); + assert_eq!(s & 0x200, 0x200, "low speed"); + } + // The mouse the USB side fits makes the shared controller report a mouse + // only once the guest has configured it. + assert!(!board.ioc3.ps2().mouse_enabled()); + let (_, status) = ports(false); + assert_eq!(status.map(|s| s & 1), [0, 0], "nothing plugged in"); +} + +/// Where the cached and uncached views of a frame differ, the JIT keys and +/// compiles the page by the view the CPU fetched it through. The reset PROM +/// fetches uncached at the same plain address that cached RAM answers once +/// memory covers its window; the compile pool used to read the cached view and +/// compile the RAM's bytes in place of the PROM's. +#[cfg(feature = "jitv2")] +#[test] +fn uncached_fetch_compiles_the_uncached_view_of_the_page() { + // The executor and the code generator are large; the default test stack is not. + std::thread::Builder::new() + .stack_size(64 << 20) + .spawn(uncached_fetch_compiles_the_uncached_view_body) + .unwrap() + .join() + .expect("thread panicked"); +} + +#[cfg(feature = "jitv2")] +fn uncached_fetch_compiles_the_uncached_view_body() { + use crate::cpu::jitv2::jitv2::{CompileRequest, UNCACHED_VIEW_PFN}; + use crate::cpu::mips_isa::{JIT_REGION_BOUNDARY_SENTINEL, OP_ADDIU}; + + /// Every word the same; a region-boundary word compiles to nothing. + struct Words { + word: u32, + generation: AtomicU64, + } + impl Words { + fn new(word: u32) -> Arc { + Arc::new(Self { word, generation: AtomicU64::new(0) }) + } + } + impl BusDevice for Words { + fn read32(&self, _: u64) -> BusRead32 { + BusRead32::ok(self.word) + } + fn read64(&self, _: u64) -> BusRead64 { + BusRead64::ok((self.word as u64) << 32 | self.word as u64) + } + fn gen_ptr(&self, _: u64) -> *const AtomicU64 { + &self.generation + } + } + + let reset_vector = 0x1fc0_0000u64; + let addiu_r1 = (OP_ADDIU << 26) | (1 << 16) | 1; + let rom = Words::new(addiu_r1); + let ram = Words::new(JIT_REGION_BOUNDARY_SENTINEL); + let physical = Physical::new(Vec::new()); + let mut bus = Ip35Bus::new(physical.clone()); + map_device(&physical, reset_vector, 0x1000, ram.clone(), "RAM over the reset window"); + bus.set_ram_generations(ram.clone()); + bus.map(0, reset_vector, 0x1000, rom.clone()); + let bus = Arc::new(bus); + let code_bus = bus.code_view(bus.cached_view(ram.clone(), crate::dev::bedrock_mem::RamExtent::new(&[512]))); + // The two views really disagree here. + assert_eq!(bus.read32(reset_vector).data, addiu_r1); + assert_eq!(bus.cached_view(ram.clone(), crate::dev::bedrock_mem::RamExtent::new(&[512])).read32(reset_vector).data, JIT_REGION_BOUNDARY_SENTINEL); + + let mut cpu = crate::ip34_machine::build_cpu(bus.clone()); + cpu.enable_jitv2_uncached_views(code_bus.clone()); + // The reset vector, KSEG1: uncached, plain physical address 0x1fc00000. + cpu.core.pc = 0xffff_ffff_bfc0_0000; + let _ = cpu.step_int(); + let page = cpu.jitv2_current_page(); + assert!(!page.is_null(), "the uncached fetch must land on a code page"); + let page = unsafe { &mut *page }; + assert_eq!(page.pfn, 0x1fc00 | UNCACHED_VIEW_PFN, "an uncached fetch keys the uncached view"); + assert!(page.is_uncached_view()); + assert_eq!(page.phys_base(), 0x1fc0_0000); + + page.mark_requested(0); + let req = CompileRequest { page: page as *mut _, compiled_for_fr1: true }; + let mut analyzer = crate::cpu::jitv2::analyzer::Analyzer::new(); + let mut codegen = crate::cpu::jitv2::codegen::Codegen::new(); + #[cfg(feature = "developer")] + crate::cpu::jitv2::comp::handle_request(&req, &cpu.jitv2_compile_bus(), &mut analyzer, &mut codegen, &crate::cpu::jitv2::JitStats::default()); + #[cfg(not(feature = "developer"))] + crate::cpu::jitv2::comp::handle_request(&req, &cpu.jitv2_compile_bus(), &mut analyzer, &mut codegen); + assert!(page.is_runnable(0), "the page must compile the PROM's ADDIU, not the RAM's boundary word"); + + // The cached view of the same frame is another page, with RAM's counter and bytes. + let jitv2 = cpu.jitv2.clone(); + let mut jit = jitv2.lock(); + let slot = jit.page_for(0x1fc00, 0x1fc0_0000, bus.as_ref(), false).unwrap(); + let cached = unsafe { &mut *jit.page_ptr(slot) }; + assert!(!cached.is_uncached_view()); + assert_ne!(cached as *mut _, page as *mut _); + cached.mark_requested(0); + let req = CompileRequest { page: cached as *mut _, compiled_for_fr1: true }; + #[cfg(feature = "developer")] + crate::cpu::jitv2::comp::handle_request(&req, &code_bus, &mut analyzer, &mut codegen, &crate::cpu::jitv2::JitStats::default()); + #[cfg(not(feature = "developer"))] + crate::cpu::jitv2::comp::handle_request(&req, &code_bus, &mut analyzer, &mut codegen); + assert!(!cached.is_runnable(0), "the cached page compiles RAM's boundary word to nothing"); + + // A write through one view invalidates only the pages that view's bytes feed. + ram.generation.fetch_add(1, std::sync::atomic::Ordering::Release); + assert!(page.is_runnable(0), "a cached-view write does not change the PROM's bytes"); + rom.generation.fetch_add(1, std::sync::atomic::Ordering::Release); + assert!(!page.is_runnable(0), "a write to the PROM's bytes invalidates its code"); +} + +#[cfg(feature = "jitv2")] +mod code_views { + use super::*; + use crate::cpu::jitv2::jitv2::{CompileRequest, UNCACHED_VIEW_PFN}; + use crate::cpu::mips_isa::{JIT_REGION_BOUNDARY_SENTINEL, OP_ADDIU}; + + struct Words(u32, AtomicU64); + impl BusDevice for Words { + fn read32(&self, _: u64) -> BusRead32 { + BusRead32::ok(self.0) + } + fn read64(&self, _: u64) -> BusRead64 { + BusRead64::ok((self.0 as u64) << 32 | self.0 as u64) + } + fn gen_ptr(&self, _: u64) -> *const AtomicU64 { + &self.1 + } + } + + const RESET: u64 = 0x1fc0_0000; + const ADDIU_R1: u32 = (OP_ADDIU << 26) | (1 << 16) | 1; + + /// A bus whose uncached attribute-0 view of the reset frame holds an ADDIU + /// and whose cached view holds a region boundary, as RAM over the PROM does. + fn bus_and_cpu(opted_in: bool) -> (Arc, Box) { + let rom = Arc::new(Words(ADDIU_R1, AtomicU64::new(0))); + let ram = Arc::new(Words(JIT_REGION_BOUNDARY_SENTINEL, AtomicU64::new(0))); + let physical = Physical::new(Vec::new()); + let mut bus = Ip35Bus::new(physical.clone()); + map_device(&physical, RESET, 0x1000, ram.clone(), "RAM over the reset window"); + bus.set_ram_generations(ram.clone()); + bus.map(0, RESET, 0x1000, rom.clone()); + // Code that is fetchable in the other views and above 4 GiB, so only + // the JIT's own range check keeps it out of a code page. + for attribute in 1..4 { + bus.map(attribute, RESET, 0x1000, rom.clone()); + } + bus.map(0, 0x1_0000_0000, 0x1000, rom); + let bus = Arc::new(bus); + let mut cpu = crate::ip34_machine::build_cpu(bus.clone()); + if opted_in { + let cached = bus.cached_view(ram, crate::dev::bedrock_mem::RamExtent::new(&[512])); + cpu.enable_jitv2_uncached_views(bus.code_view(cached)); + } + (bus, cpu) + } + + /// The executor and the code generator are large; the default test stack is not. + fn big_stack(body: fn()) { + std::thread::Builder::new().stack_size(64 << 20).spawn(body).unwrap().join().expect("thread panicked"); + } + + fn wait_runnable(page: &crate::cpu::jitv2::PhysicalCodePage) -> bool { + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(20); + while std::time::Instant::now() < deadline { + if page.is_runnable(0) { return true; } + std::thread::sleep(std::time::Duration::from_millis(5)); + } + false + } + + /// After a page-pool flush the restarted compile pool must still read + /// uncached pages through the compile bus. Restarted on the CPU's own bus it + /// rejects the view address and the page never compiles. + #[test] + fn a_pool_restart_after_a_flush_keeps_the_compile_bus() { + big_stack(|| { + let (_bus, mut cpu) = bus_and_cpu(true); + { + let jit = cpu.jitv2.clone(); + let mut g = jit.lock(); + let stats = g.stats.clone(); + g.compile_queue.set_thread_count(1); + g.compile_queue.start(cpu.jitv2_compile_bus(), stats); + } + cpu.jitv2_flush_from_here(false); + cpu.core.pc = 0xffff_ffff_bfc0_0000; + let _ = cpu.step_int(); + let page = unsafe { &mut *cpu.jitv2_current_page() }; + assert!(page.is_uncached_view()); + page.mark_requested(0); + let jit = cpu.jitv2.clone(); + let g = jit.lock(); + let stats = g.stats.clone(); + assert!(g.compile_queue.send(CompileRequest { page: page as *mut _, compiled_for_fr1: true }, &stats)); + drop(g); + assert!(wait_runnable(page), "the restarted pool must compile the uncached view"); + jit.lock().compile_queue.stop(); + }); + } + + /// A machine that has not opted in keeps one page per frame and the + /// original-size frame map, whatever the cache attribute of the fetch. + #[test] + fn an_executor_that_has_not_opted_in_keeps_the_plain_key() { + big_stack(|| { + let (_bus, mut cpu) = bus_and_cpu(false); + assert!(!cpu.jitv2.lock().uncached_views_enabled()); + cpu.core.pc = 0xffff_ffff_bfc0_0000; + let _ = cpu.step_int(); + let page = unsafe { &*cpu.jitv2_current_page() }; + assert_eq!(page.pfn, 0x1fc00); + assert_eq!(page.pfn & UNCACHED_VIEW_PFN, 0); + assert!(!page.is_uncached_view()); + }); + } + + /// Uncached attributes 1 to 3 and anything over 4 GiB stay in the interpreter. + #[test] + fn attributes_one_to_three_and_high_addresses_get_no_code_page() { + big_stack(|| { + for pc in [ + 0x9000_0000_0000_0000u64 | RESET, // attribute 0: a page + 0x9200_0000_0000_0000u64 | RESET, // attribute 1 + 0x9400_0000_0000_0000u64 | RESET, // attribute 2 + 0x9600_0000_0000_0000u64 | RESET, // attribute 3 + 0x9000_0001_0000_0000u64, // 4 GiB, attribute 0 + ] { + // A fresh executor each time: a tracked page would survive a + // fetch that wrongly kept the same frame. + let (_bus, mut cpu) = bus_and_cpu(true); + // KX, so XKPHYS addresses translate. + let status = cpu.core.cp0_status | 0x80; + cpu.set_cp0_status(status); + cpu.core.pc = pc; + let _ = cpu.step_int(); + let page = cpu.jitv2_current_page(); + if pc == 0x9000_0000_0000_0000u64 | RESET { + assert!(!page.is_null(), "attribute 0 below 4 GiB is compiled"); + } else { + assert!(page.is_null(), "{pc:#x} must not get a code page"); + } + } + }); + } +} + +fn fuel_with_graphics(graphics: crate::config::GraphicsBoard) -> crate::ip34_machine::FuelBoard { + crate::ip34_machine::FuelBoard::with_config(crate::ip34_machine::FuelBoardConfig { + payload: &[], clock: Arc::new(|| 0), flash_path: None, + station_mac: [0x08, 0x00, 0x69, 0x21, 0x43, 0x65], + timekeeper: Arc::new(crate::dev::timekeeper::Timekeeper::fuel().unwrap()), + ps2_input: true, usb_input: true, + l1: Default::default(), + graphics, + }).unwrap() +} + +/// The VPro's link status in the crossbar, in its slot's data half. +const VPRO_LINK_STATUS: u64 = 0x100 + 5 * 0x40 + 0x10 + 4; +/// A processor access to widget 13 through Bedrock's small window. +const VPRO_PIO: u64 = 13 << 24; + +// The VPro cases follow tests/qtest/sn1-fuel-vpro-test.c in +// jamesbraid/qemu-sgi-origami: widget-identity, profile-identity-and-routing +// and headless-fuel-has-no-board. + +// VPro host-interface registers the tests below touch, as byte offsets in +// the board's widget space. The board's own constants are private to it. +const ODSY_IDENTITY: u64 = 0x00; +const ODSY_CONTROL: u64 = 0x20; +const ODSY_CONTROL_WIDGET_MASK: u32 = 0xf; +const ODSY_INT_DEST_LOWER: u64 = 0x38; +const ODSY_INTR_CLEAR: u64 = 0x101c; +const ODSY_INTR_ENABLE_SET: u64 = 0x1034; +const ODSY_INTR_FIFO_LOW_WATER: u32 = 0x200; +const ODSY_ROUTE_DEST_UPPER: u64 = 0x1044; +const ODSY_ROUTE_DEST_LOWER: u64 = 0x104c; +const ODSY_ROUTE_VECTORS: u64 = 0x1054; +const ODSY_ROUTE_DEST_WIDGET_SHIFT: u32 = 25; + +/// A Fuel IRIX inventory lists its VPro under xtalk/13: the processor must +/// reach the board there, learn its widget number from it, and find the +/// link alive in the crossbar. +#[test] +fn the_vpro_answers_at_widget_13_with_its_link_alive() { + for (graphics, revision) in [(crate::config::GraphicsBoard::V12, 0xa), (crate::config::GraphicsBoard::V10, 0xa)] { + let board = fuel_with_graphics(graphics); + let identity = board.io.pio_read(VPRO_PIO | (ODSY_IDENTITY + 4), 4); + assert_eq!(identity.status, BUS_OK); + assert_eq!(identity.data as u32, (revision << 28) | 0x0c01_3047, "{graphics:?}"); + let control = board.io.pio_read(VPRO_PIO | (ODSY_CONTROL + 4), 4).data as u32; + assert_eq!(control & ODSY_CONTROL_WIDGET_MASK, 13); + assert_ne!(board.xbridge.crossbar().read32(VPRO_LINK_STATUS).data & (1 << 31), 0); + } +} + +/// The board's interrupt packets cross the XBridge to Bedrock as the PCI +/// heads' do: a raised source sets its vector's pending bit in the +/// processor interface the destination names, and a cleared one clears it. +#[test] +fn vpro_interrupt_packets_set_and_clear_a_processor_interface_pending_bit() { + let board = fuel_with_graphics(crate::config::GraphicsBoard::V12); + // Register space (bit 47), the node's hub window and PI0's pending + // modify register, at the Bedrock's crossbar port 0xa. + let packet = (1u64 << 47) | 0x180_0000 | 0x90; + let vector = 5; + let write = |offset: u64, value: u64| { + assert_eq!(board.io.pio_write(VPRO_PIO | offset, 4, value), BUS_OK, "{offset:#x}"); + }; + write(ODSY_ROUTE_DEST_UPPER, (0xa << ODSY_ROUTE_DEST_WIDGET_SHIFT) | (packet >> 32)); + write(ODSY_ROUTE_DEST_LOWER, packet & 0xffff_ffff); + write(ODSY_ROUTE_VECTORS, vector); + write(ODSY_INTR_ENABLE_SET, ODSY_INTR_FIFO_LOW_WATER as u64); + let pending = || board.pi[0].read64(0x98).data & (1 << vector); + assert_ne!(pending(), 0); + write(ODSY_INTR_CLEAR, ODSY_INTR_FIFO_LOW_WATER as u64); + assert_eq!(pending(), 0); +} + +/// A link reset of the graphics port resets the board behind it, as the +/// PCI port's does for the functions behind its head. +#[test] +fn a_graphics_link_reset_resets_the_vpro() { + let board = fuel_with_graphics(crate::config::GraphicsBoard::V12); + let destination = VPRO_PIO | (ODSY_INT_DEST_LOWER + 4); + assert_eq!(board.io.pio_write(destination, 4, 0x55), BUS_OK); + assert_eq!(board.io.pio_read(destination, 4).data, 0x55); + let link_reset = 0x100 + 5 * 0x40 + 0x30 + 4; + assert_eq!(board.xbridge.crossbar().write32(link_reset, 1), BUS_OK); + assert_eq!(board.io.pio_read(destination, 4).data, 0); +} + +/// Without graphics nothing answers at widget 13 and its link is down, as +/// on a headless Fuel. +#[test] +fn a_fuel_without_graphics_has_no_widget_13() { + let board = fuel_with_graphics(crate::config::GraphicsBoard::None); + assert!(board.vpro.is_none()); + assert_ne!(board.io.pio_read(VPRO_PIO | 4, 4).status, BUS_OK); + assert_eq!(board.xbridge.crossbar().read32(VPRO_LINK_STATUS).data & (1 << 31), 0); +} + +/// IRIX's VPro driver writes each processor interface's graphics credit +/// counter at attach and tears the board down if it reads back zero; the +/// registers must keep what the guest writes, per interface. +#[test] +fn processor_interface_graphics_registers_keep_their_writes() { + let board = fuel_with_graphics(crate::config::GraphicsBoard::V12); + // Slot A's credit counter and bias; PI1 keeps its own block. + assert_eq!(board.pi[0].write64(0x308, 100), BUS_OK); + assert_eq!(board.pi[0].write64(0x310, 0x1000 | 68), BUS_OK); + assert_eq!(board.pi[0].read64(0x308).data, 100); + assert_eq!(board.pi[0].read64(0x310).data, 68); + assert_eq!(board.pi[1].read64(0x308).data, 0); + board.pi[0].power_on(); + assert_eq!(board.pi[0].read64(0x308).data, 0); +} + +/// The memory map lists each slot that can hold RAM and the reset PROM. The +/// Fuel's 256 MiB slots stop below the PROM; the Tezro's 1 GiB slot 0 reaches +/// past it, and its RAM is listed on either side of the PROM's range. +#[test] +fn the_memory_map_lists_the_ram_slots_around_the_reset_prom() { + use crate::dev::bedrock_mem::RamExtent; + use crate::memory_map::MemKind; + let fuel = memory_map(&RamExtent::new(&[256, 256])); + let ram: Vec<_> = fuel.ranges().iter().filter(|r| r.kind == MemKind::Ram).map(|r| (r.base, r.end())).collect(); + assert_eq!(ram, (0..4u64).map(|slot| (slot << 30, (slot << 30) + (256 << 20))).collect::>()); + assert_eq!(fuel.find(0x1fc0_0000).map(|r| r.kind), Some(MemKind::Rom)); + + let tezro = memory_map(&RamExtent::new(&[1024, 1024, 512, 512, 512, 512])); + assert_eq!(tezro.find(0x1fbf_fff8).map(|r| r.kind), Some(MemKind::Ram)); + assert_eq!(tezro.find(0x1fc0_0000).map(|r| r.kind), Some(MemKind::Rom)); + assert_eq!(tezro.find(0x2000_0000).map(|r| r.kind), Some(MemKind::Ram)); + assert_eq!(tezro.find(0x3fff_fff8).map(|r| r.kind), Some(MemKind::Ram)); + assert!(tezro.is_ram(5 << 30), "slot 5, above 4 GiB"); + assert_eq!(tezro.end(), (7 << 30) + (512 << 20)); +} diff --git a/src/ip53_machine.rs b/src/ip53_machine.rs new file mode 100644 index 000000000..43f26b911 --- /dev/null +++ b/src/ip53_machine.rs @@ -0,0 +1,1279 @@ +//! The Tezro (IP53 board, IP35 software platform): R16000s on one Bedrock +//! node, a PIC host bridge and the IO9 board's IOC4, ISP12160 and BCM5701, +//! with Iris's CPU thread, monitor and serial console. +//! +//! Selected with `[machine] profile = "tezro_ip53"`. Shared keys select the PROM, +//! NVRAM, memory and SCSI devices; `[tezro]` selects the processor +//! population, the serial console and writable flash. The node itself is +//! the one the Fuel uses ([`Node`]); this board adds its widgets and its +//! inputs. With `[graphics] board = "v12"` a VPro V12 sits at widget 12 +//! behind the PXBOW; otherwise there is no graphics. +//! +//! The board follows the QEMU Tezro compositions: +//! +//! - The one-processor board has one PIC at XIO widget 15; the two- and +//! four-processor board adds a second at widget 11, with nothing on its +//! buses, and a V12 slot at widget 12 that is left empty. The PIC at +//! widget 15 is the IO9's: its PXBOW answers as widget 0 and forwards +//! the processor's accesses to widget 15. On the larger board it carries +//! widget 11 too, and shows that link present and alive (unlike QEMU's +//! composition, below). Its head 0 carries the IO9 PCI bus: the IOC4 at +//! device 1 (input 0), the ISP12160 at device 3 (input 2) and the +//! BCM5701 at device 4 (input 3), with REQ64# wired on the BCM5701's +//! slot. The PIC's interrupts and DMA go to Bedrock IO, the host widget +//! 10. +//! - The IOC4 carries the DS1744-compatible clock and NVRAM in its BAR and +//! the PS/2 keyboard and mouse; its four ports are serial streams beside +//! the L1's, and the L1 is the console by default. +//! - Bedrock board word 0x2001, the Tezro firmware record (the part's clock, +//! an authored configuration block), and an L1 that reports +//! brick type 0x39, revision 1.48.1, the first port-speed integer 1 and +//! the router `brick` reply. +//! +//! # What is approximated here +//! +//! - Every identity above (board word, L1 type and revision, PIC revision 3 +//! profile, PCI devices, revisions and interrupt routes, REQ64# on the +//! BCM5701's slot) is a software input the QEMU profiles choose, not a +//! value measured on a Tezro. +//! - The one-processor board has one PIC. The tower guide gives that board +//! two PCI-X buses and one XIO connector; that it is one PIC at widget 15 +//! is inferred from them. +//! - Every fitted processor runs from reset, each on its own thread, as in +//! QEMU's IP35 node; CPU enable is stored and starts or stops nothing. The +//! PROM's boot rendezvous holds a processor that arrives early +//! ([`Rendezvous`]), an approximation with no hardware counterpart. +//! - The second PIC is a port of the IO9 PIC's PXBOW, beside the host's +//! and the IO9 PIC's links: the PROM and IRIX learn which widgets exist +//! only from widget 0's link status and auxiliary status, so one +//! crossbar per node carries both PICs. That is a modelling consequence +//! of the discovery path, not a known route: whether the second PIC has +//! a crossbar of its own in front of it, and its part (0xd102 or 0xd112) +//! and link reset values beyond alive and present, are unknown. QEMU's +//! composition still gives the second PIC its own PXBOW, whose widget 0 +//! nothing maps, and leaves link 11 absent on the IO9 PIC's. The second +//! PIC's interrupts and DMA reach Bedrock IO directly, as the IO9 PIC's +//! do, rather than through the crossbar. +//! - PIC DMA aimed at a widget other than Bedrock IO is refused and logged, +//! as the Fuel's XBridge path refuses it. +//! - No EEPROM image is fitted for the IO9, the node board or the BCM5701: +//! no published source gives their contents. A BCM5701 EEPROM +//! transfer does not complete. +//! +//! Board composition ported from hw/mips/sgi/sn-machine.c and +//! hw/mips/sgi/sn1-node.c in jamesbraid/qemu-sgi-origami. + +use crate::config::{scsi_label, CpuModel, GraphicsBoard, MachineConfig, TezroConsole}; +use crate::cpu::mips_cache_shadow::{R16000ShadowCache, R16000_MHZ700_PRID}; +use crate::cpu::mips_cache_v2::{CpuModel as CacheCpuModel, MipsCache}; +use crate::cpu::mips_exec::{CpuDevice, MipsCpu, MipsCpuConfig, MipsExecutor}; +use crate::cpu::cpu_monitor::CpuMonitor; +use crate::cpu::mips_tlb::MipsTlb; +use crate::cpu::smp::World; +use crate::dev::bcm5701::{Bcm5701, Bcm5701Config}; +use crate::dev::bedrock_flash::{two_processor_environment, FirmwareRecord}; +use crate::dev::bedrock_io::{gfx_route, BedrockIo}; +use crate::dev::bedrock_mem::MemoryPopulation; +use crate::dev::bedrock_pi::{BedrockPi, NodeHooks, Population}; +use crate::dev::ds1744::{Ds1744, YearBase}; +use crate::dev::heart::{host_clock, Clock}; +use crate::dev::ioc4::{Ioc4, Ioc4Config}; +use crate::dev::isp12160::Isp12160; +use crate::dev::l1::L1Config; +use crate::dev::odyssey::{ + Odyssey, PacketSender, PiGfx, PiGfxLink, PI_GFX_BIAS, PI_GFX_CREDIT, PI_GFX_CREDIT_MASK, PI_GFX_INT_COUNT, + PI_GFX_REGS, PI_GFX_SLOTS, PI_GFX_SLOT_REGS, +}; +use crate::dev::pic::{Function, Pic, PicPeer, Refused}; +use crate::dev::pxbow::Pxbow; +use crate::dev::prom_image; +use crate::dev::ps2::Ps2Controller; +use crate::dev::serial_host::CiSerialBackend; +use crate::devlog::{init_devlog, LogModule}; +use crate::gdb_stub::CpuDebug; +use crate::gfx_display::GfxDisplay; +use crate::hptimer::TimerManager; +use crate::ip35::CPU_PHYS_MASK; +use crate::ip35::node::{connect_cpus, connect_serial_ports, Node, NodeConfig, SerialPort, Store, StoreCounter, WidgetDma}; +use crate::ip35::rendezvous::Rendezvous; +use crate::machine_state::GuestState; +use crate::monitor::{Monitor, DEFAULT_PORT as DEFAULT_MONITOR_PORT}; +use crate::net::{GatewayConfig, Link, NatControl}; +use crate::perf_monitor::PerfMonitor; +use crate::scsi::{attach_from_config, ScsiBus}; +use crate::traits::{BusDevice, Device, MachineEvent, Resettable, BUS_OK}; +use crate::workstation::{register_system_controller, DiscChanger, MachineRef, MachineSlot, Workstation}; +use parking_lot::Mutex; +use std::net::{Ipv4Addr, SocketAddr}; +use std::path::{Path, PathBuf}; +use std::sync::mpsc::Receiver; +use std::sync::{atomic::{AtomicBool, AtomicU64, Ordering}, Arc, Weak}; + +/// The Bedrock board word the QEMU Tezro profiles select. +const BOARD_ID: u64 = 0x2001; +/// The IO9's PIC, the second PIC of the two- and four-processor board, and +/// the widget Bedrock IO answers as. Widget 12 on that board is the V12's +/// XIO slot, which is left empty. +const IO9_WIDGET: u8 = 15; +const SECOND_PIC_WIDGET: u8 = 11; +const HOST_WIDGET: u8 = 10; +/// IO9's PCI functions: configuration device on head 0. +const IOC4_DEVICE: u8 = 1; +const SCSI_DEVICE: u8 = 3; +const ETH_DEVICE: u8 = 4; +/// The R16000 part a Tezro runs, from `[machine] cpu`. Both are the CPU +/// models' R16000 with a 4 MB secondary cache; they differ in PRId, the +/// Config reset word and the clock the firmware record gives. +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum TezroPart { + /// Revision 2.1 at 700 MHz (`r16000_700`), the default. + Mhz700, + /// Revision 2.2 at 800 MHz (`r16000`). + Mhz800, +} + +impl TezroPart { + /// The part `cpu` names. `r10000` names the family, as on the Fuel, and + /// takes the default part. + pub(crate) fn of(cpu: CpuModel) -> Result { + match cpu { + CpuModel::R16000Mhz700 | CpuModel::R10000 => Ok(Self::Mhz700), + CpuModel::R16000 => Ok(Self::Mhz800), + other => Err(format!("a Tezro has R16000s, not the {}", other.label())), + } + } + + /// The flash record the PROM checks this part against. + pub(crate) fn firmware(self) -> FirmwareRecord { + match self { + Self::Mhz700 => FirmwareRecord::TEZRO, + Self::Mhz800 => FirmwareRecord::TEZRO_800, + } + } + + /// The part's PRId, from the CPU models. + pub(crate) fn prid(self) -> u32 { + match self { + Self::Mhz700 => R16000_MHZ700_PRID, + Self::Mhz800 => ::PRID, + } + } + + /// The processor clock in MHz. + pub(crate) fn mhz(self) -> u32 { + self.firmware().cpu_clock_hz / 1_000_000 + } + + /// The count register runs at half the processor clock. + pub(crate) fn count_hz(self) -> u64 { + u64::from(self.firmware().cpu_clock_hz) / 2 + } +} +/// How often the poll returns lifted checked pages to the direct window, in +/// board nanoseconds. +const RETIRE_INTERVAL_NS: u64 = 100_000_000; +/// The serial connections: the L1, then IOC4 ports 0 to 3. +const SERIAL_PORTS: usize = 5; + +pub struct TezroMachine { + board: Arc, + /// The processors, CPU 0 first, in the board's slot order. + cpus: Vec>, + timers: Arc, + monitor: Arc, + monitor_port: u16, + ci_serial: Option>, + console: usize, + ci: bool, + scratch: Option, + nvram: String, + frames: Option>, + nat: Option<(Arc, (Ipv4Addr, Ipv4Addr, Ipv4Addr))>, + events: Option>, + slot: Option, + lifecycle: GuestState, +} + +impl TezroMachine { + pub fn new(cfg: &MachineConfig) -> Result { + cfg.validate()?; + let prom = prom_image::load(Path::new(&cfg.prom), 8 * 1024 * 1024).map_err(|e| format!("Tezro {e}"))?; + let clock = Arc::new(Ds1744::new(YearBase::From(1968))?); + clock.set_nvram_file(Path::new(&cfg.nvram))?; + let memory = cfg.tezro_memory()?; + #[cfg(feature = "jitv2")] + check_jitv2_memory(&memory)?; + let part = TezroPart::of(cfg.machine.cpu)?; + let board = Arc::new(TezroBoard::with_config(TezroBoardConfig { + payload: &prom, + clock: host_clock(), + flash_path: cfg.tezro.flash.as_deref().map(Path::new), + station_mac: cfg.network().mac, + timekeeper: clock, + memory: &memory, + cpus: usize::from(cfg.machine.cpus()), + part, + v12: cfg.graphics_board() == GraphicsBoard::V12, + })?); + let timers = Arc::new(TimerManager::new()); + board.ioc4.set_timer_manager(timers.clone()); + let count_hz = cfg.clock.fixed_mhz.map_or(part.count_hz(), |mhz| (mhz * 1_000_000.0) as u64).max(1); + let cpus = build_cpus(&board, &timers, count_hz); + let cpu = cpus[0].clone(); + board.status.set_cpu_cycles(cpu.cycles_ptr()); + board.status.set_count_hz(cpu.count_hz_atomic()); + for c in cpus.iter().skip(1) { + board.status.add_cpu_stats(c.cycles_ptr(), c.fasttick_count()); + } + let node = &board.node; + // A rebuild of the whole memory map runs with the other processors + // parked. + let world = Arc::new(World::new()); + world.set_slots(cpus.iter().map(|c| c.park_slot()).collect()); + let remap_world = world.clone(); + node.md.set_remap_guard(Arc::new(move |work: &mut dyn FnMut()| remap_world.stop_others(work))); + // One page pool and one body of compiled code, reaching each + // processor's state through the core it is called with. + #[cfg(feature = "jitv2")] + if cpus.len() > 1 { + if cfg!(feature = "jitv2_lockstep") { + return Err("[machine] cpus above 1 cannot run with jitv2_lockstep".into()); + } + let jit = cpu.jitv2(); + for c in &cpus { + c.share_jitv2(jit.clone(), world.clone()); + } + } + board.rendezvous.set_cpus(&cpus); + board.rendezvous.set_timer_manager(timers.clone()); + let rx = connect_cpus("Tezro", node, board.processors, &cpus); + // After connect_cpu set the windows: each compile worker takes the + // cache's geometry and the CPU model's policies once, as it starts. + // Workers read plain physical code addresses through the view the CPU + // fetched each page by: the cached plane for cached pages, the + // uncached attribute-0 view for the reset PROM's uncached fetches. + #[cfg(feature = "jitv2")] + { + let cpu_device: Arc = cpu.clone().as_device(); + let jit = cpu.jitv2(); + let mut g = jit.lock(); + g.compile_queue.set_thread_count(cfg.jitv2.threads.max(1)); + g.compile_queue.set_cpu(Arc::downgrade(&cpu_device)); + g.compile_queue.set_owner(Arc::downgrade(&jit)); + if cpus.len() > 1 { + g.compile_queue.set_world(world.clone()); + } + #[cfg(not(feature = "jitv2_lockstep"))] + { + let stats = g.stats.clone(); + g.compile_queue.start(board.code_bus(), stats); + } + // The compile pool keeps only a weak handle on the CPU. + std::mem::forget(cpu_device); + } + let scratch = Self::attach_scsi(cfg, &board)?; + let console = cfg.tezro.console.index(); + let ci_serial = cfg.ci.then(|| board.serial[console].clone()); + Self::connect_serial(cfg, &board, console)?; + let monitor = Monitor::new(); + monitor.register_device(init_devlog()); + monitor.register_device(Arc::new(CpuMonitor::new(cpus.clone()))); + monitor.register_device(PerfMonitor::new( + cpu.running_flag(), cpu.cycles_ptr(), cpu.fasttick_count(), None, None, + )); + monitor.register_device(node.md.physical().clone()); + monitor.register_device(node.l1.clone()); + monitor.register_device(board.scsi.clone()); + monitor.register_device(board.ioc4.ps2()); + for c in &cpus { + c.register_locks(); + } + board.scsi.register_locks(); + board.ioc4.register_locks(); + board.eth.register_locks(); + for pic in &board.pics { + pic.register_locks(); + } + node.flash.register_locks(); + node.md.register_locks(); + let (mut nat, mut frames) = (None, None); + let net = GatewayConfig::from_network(&cfg.network()); + if let Some(link) = Link::with_gateway(net.clone())? { + frames = Some(link.frames()); + nat = Some((link.nat(), (net.client_ip, net.gateway_ip, net.netmask))); + board.eth.attach(link); + } + eprintln!( + "iris: Tezro IP53, {} R16000 {} MHz, {} MiB, {}, serial console {:?}", + cpus.len(), + part.mhz(), + memory.total_mib(), + if board.v12.is_some() { cfg.graphics_board().label() } else { "no graphics" }, + cfg.tezro.console + ); + Ok(Self { + board, + cpus, + timers, + monitor: Arc::new(monitor), + monitor_port: cfg.monitor_port.unwrap_or(DEFAULT_MONITOR_PORT), + ci_serial, + console, + ci: cfg.ci, + scratch, + nvram: cfg.nvram.clone(), + nat, + frames, + events: Some(rx), + slot: None, + lifecycle: GuestState::Idle, + }) + } + + fn attach_scsi(cfg: &MachineConfig, board: &TezroBoard) -> Result, String> { + let attached = attach_from_config(cfg, Some("tezro"), &|channel| { + board.scsi.bus_for_channel(channel) + .ok_or_else(|| format!("controller {channel} (a Tezro has channels 0 and 1)")) + })?; + if let Some(device) = attached.daynaports.first() { + return Err(format!("{}: kind = \"daynaport\" is not available on the Tezro", + scsi_label(device.controller, device.id))); + } + Ok(attached.scratch) + } + + fn connect_serial(cfg: &MachineConfig, board: &TezroBoard, console: usize) -> Result<(), String> { + let [a, b] = cfg.serial_ports(); + let ioc4 = |port: usize| -> Box) + '_> { + Box::new(move |backend| board.ioc4.set_serial_backend(port, backend)) + }; + let ports = [ + SerialPort { + label: "L1", + port: cfg.tezro.l1_port.unwrap_or(8882), + setting: "tezro.l1_port", + ci: &board.serial[0], + log: cfg.tezro.l1_log.clone(), + connect: Box::new(|backend| board.node.l1.set_serial_backend(backend)), + }, + SerialPort { label: "IOC4 0", port: a, setting: "serial_port_a", ci: &board.serial[1], log: None, connect: ioc4(0) }, + SerialPort { label: "IOC4 1", port: b, setting: "serial_port_b", ci: &board.serial[2], log: None, connect: ioc4(1) }, + SerialPort { + label: "IOC4 2", + port: cfg.tezro.serial_port_c.unwrap_or(8883), + setting: "tezro.serial_port_c", + ci: &board.serial[3], + log: None, + connect: ioc4(2), + }, + SerialPort { + label: "IOC4 3", + port: cfg.tezro.serial_port_d.unwrap_or(8884), + setting: "tezro.serial_port_d", + ci: &board.serial[4], + log: None, + connect: ioc4(3), + }, + ]; + connect_serial_ports(cfg, "Tezro", &ports, console) + } +} + +impl Workstation for TezroMachine { + fn start(&mut self) { + self.lifecycle = GuestState::Starting; + self.monitor.clone().start_server(format!("127.0.0.1:{}", self.monitor_port)); + self.restart_peripherals(); + if !self.ci { + self.cpu_start(); + } + } + fn stop(&mut self) { + for c in &self.cpus { + c.clone().as_device().stop(); + } + self.board.stop(); + self.board.timekeeper.flush(); + if self.lifecycle != GuestState::Idle { + self.lifecycle = GuestState::Halted; + } + } + fn reset(&mut self) { + self.stop(); + for c in &self.cpus { + c.power_on(); + } + self.board.power_on(); + self.restart_peripherals(); + } + fn restart_peripherals(&mut self) { + self.board.start(); + } + /// Every processor starts at once, as from the node's reset. + fn cpu_start(&self) { + for c in &self.cpus { + c.clone().as_device().start(); + } + } + fn cpu_is_running(&self) -> bool { + self.cpus.iter().any(|c| c.clone().as_device().is_running()) + } + fn machine_slot(&mut self) -> MachineSlot { + let ptr = self as *mut Self as *mut dyn Workstation; + self.slot.get_or_insert_with(|| Arc::new(Mutex::new(MachineRef(ptr)))).clone() + } + fn register_system_controller(&mut self) { + let slot = self.machine_slot(); + let events = self.events.take().expect("Tezro system controller registered twice"); + register_system_controller(&self.monitor, slot, Some(events)); + } + fn monitor_addr(&self) -> Option { + self.monitor.bound_addr() + } + fn lifecycle(&self) -> GuestState { + self.lifecycle + } + fn set_lifecycle(&mut self, state: GuestState) { + self.lifecycle = state; + } + fn get_display(&self) -> Option> { + self.board.v12.as_ref().map(|v12| v12.display()) + } + fn register_monitor_device(&self, device: Arc) { + self.monitor.register_device(device); + } + fn get_ps2(&self) -> Arc { + self.board.ioc4.ps2() + } + fn get_timer_manager(&self) -> Arc { + self.timers.clone() + } + fn get_cpu_debug(&self) -> Arc { + self.cpu().clone().debug_adapter() + } + fn fasttick_count(&self) -> Arc { + self.cpu().fasttick_count() + } + fn cpu_counters(&self) -> Vec<(crate::cpu::mips_core::CyclesPtr, Arc)> { + self.cpus.iter().map(|c| (c.cycles_ptr(), c.fasttick_count())).collect() + } + fn get_ci_serial(&self) -> Option> { + self.ci_serial.clone() + } + fn inject_serial_console(&self, bytes: &[u8]) { + self.board.serial[self.console].push_host(bytes); + } + fn as_any_mut(&mut self) -> &mut dyn std::any::Any { + self + } + fn save_nvram(&self, path: Option<&str>) -> Result { + if path.is_some_and(|p| p != self.nvram) { + return Err("Tezro NVRAM save uses the configured path".into()); + } + self.board.timekeeper.flush(); + Ok(self.nvram.clone()) + } + fn scratch_path(&self) -> Option<&Path> { + self.scratch.as_deref() + } + fn disc_changer(&self) -> Option> { + Some(self.board.scsi.clone() as Arc) + } + fn pending_chd_sync_count(&self) -> usize { + self.scsi_buses().iter().map(|bus| bus.pending_chd_sync_count()).sum() + } + fn flush_disks(&self) -> std::io::Result<()> { + self.board.flush_disks() + } + fn sync_chd_disks( + &self, + only: Option, + progress: &mut dyn FnMut(usize, usize, f32), + cancel: &dyn Fn() -> bool, + ) -> std::io::Result { + ScsiBus::sync_chd_disks(&self.scsi_buses(), only, progress, cancel) + } + fn net_frames(&self) -> Option> { + self.frames.clone() + } + fn nat_control(&self) -> Option> { + self.nat.as_ref().map(|(n, _)| n.clone()) + } + fn nat_expected(&self) -> Option<(Ipv4Addr, Ipv4Addr, Ipv4Addr)> { + self.nat.as_ref().map(|(_, a)| *a) + } +} + +impl TezroMachine { + /// CPU 0, the one the front ends drive. + fn cpu(&self) -> &Arc { + &self.cpus[0] + } + + fn scsi_buses(&self) -> Vec<&ScsiBus> { + (0..2).filter_map(|channel| self.board.scsi.bus_for_channel(channel)).collect() + } +} + +impl Drop for TezroMachine { + fn drop(&mut self) { + if let Some(slot) = self.slot.take() { + slot.lock().0 = MachineRef::null(); + } + self.stop(); + self.monitor.shutdown(); + } +} + +/// The processors the machine runs, of the board's part: one for each +/// fitted slot, CPU 0 first, in the board's slot order. +fn build_cpus(board: &Arc, timers: &Arc, count_hz: u64) -> Vec> { + let count = board.processors.count(); + (0..count).map(|id| start_cpu::(board, timers, count_hz, id as u8, count)).collect() +} + +/// Processor `id` of `count`, of type `C`, with its timers; CPU 0 also +/// runs the board's poll. +fn start_cpu(board: &Arc, timers: &Arc, count_hz: u64, id: u8, count: usize) -> Arc +where + C: CacheCpuModel + From> + 'static, + MipsCpu: CpuDevice, +{ + let mut exec = build_cpu::(board); + // CPU 0's Compare matches blink the graphics status bar's LED. + exec.core.fasttick_count = if id == 0 { board.fasttick.clone() } else { Arc::new(AtomicU64::new(0)) }; + exec.core.set_timer_manager(timers.clone()); + exec.core.set_count_hz(count_hz); + exec.core.count_write_samples = true; + exec.core.cpu_id = id; + exec.core.smp = count > 1; + if id == 0 { + let poll_board = board.clone(); + exec.batch_hook = Some(Box::new(move |_| poll_board.poll())); + } + Arc::new(MipsCpu::new(*exec)) +} + +/// A jitv2 build keys the PROM's uncached code pages at frame 1 << 20 and +/// up, above every frame below 4 GiB (`Jitv2::enable_uncached_views`), so it +/// takes only memory that ends at or below 4 GiB: banks 0 to 3. The default +/// layout puts banks 4 and 5 above it. Refused here rather than left to the +/// JIT's assertion. +#[cfg(feature = "jitv2")] +fn check_jitv2_memory(memory: &MemoryPopulation) -> Result<(), String> { + if memory.fits_direct_window() { + return Ok(()); + } + Err(format!( + "a jitv2 build runs a Tezro only with memory below 4 GiB (at most four banks), since it \ + compiles the PROM's uncached code under page keys above it; banks = {:?} reach past 4 GiB: \ + set banks = [1024, 1024, 1024, 1024] or fewer, or use a build without jitv2", + memory.banks_mib() + )) +} + +/// A processor of type `C` with the board part's PRId and reset-mode +/// inputs, on the board's bus. +fn build_cpu>>(board: &TezroBoard) -> Box> { + let cfg = MipsCpuConfig::for_model::(Arc::new(crate::ip35::memory_map(&board.node.md.ram_extent()))); + let mut cpu = Box::new(MipsExecutor::new( + board.node.bus.clone() as Arc, + MipsTlb::new(cfg.tlb_entries), + &cfg, + )); + cpu.phys_mask = CPU_PHYS_MASK; + // The flash record's mode word and the processor's reset mode must + // agree, or the PROM enters its error routine before GP is set. + cpu.core.cp0_config = board.part.firmware().reset_config0(); + cpu.core.reset_prid = board.part.prid(); + cpu.core.cp0_prid = board.part.prid(); + cpu.cache = C::from(board.cached_bus()); + // The PROM fetches its own code uncached; with memory over its window + // the cached and uncached views of it differ. + #[cfg(feature = "jitv2")] + cpu.enable_jitv2_uncached_views(board.code_bus()); + // The PROM keeps its stack in cache lines it tags for a window with no + // memory behind it yet; the controller answers that window, each + // processor's own, while they stand. The hook runs on the processor's + // thread, which names the processor. + let memory = board.node.md.clone(); + cpu.cache.set_pd_tag_store_hook(Box::new(move |tag| memory.startup_tag_store(tag))); + cpu +} + +pub(crate) struct TezroBoardConfig<'a> { + pub payload: &'a [u8], + pub clock: Clock, + pub flash_path: Option<&'a Path>, + pub station_mac: [u8; 6], + /// The DS1744-compatible clock and NVRAM in the IOC4's BAR. + pub timekeeper: Arc, + pub memory: &'a MemoryPopulation, + /// Processors fitted. + pub cpus: usize, + /// The R16000 part fitted. + pub part: TezroPart, + /// A VPro V12 at widget 12. + pub v12: bool, +} + +/// The Tezro's devices around its Bedrock node. +pub(crate) struct TezroBoard { + pub node: Node, + /// The slots whose processors run: every fitted one. + pub processors: Population, + /// The PROM's boot rendezvous across them. + pub rendezvous: Arc, + /// The PICs in QEMU's host order, which numbers their PI level sources: + /// widget 11's first when the board has it, then the IO9's at widget 15. + pub pics: Vec>, + pub ioc4: Arc, + pub scsi: Arc, + pub eth: Arc, + pub timekeeper: Arc, + pub serial: [Arc; SERIAL_PORTS], + /// The R16000 part fitted, whose record the flash holds. + pub part: TezroPart, + /// The VPro V12, when the board has one. + pub v12: Option>, + /// Each PI's graphics flow-control registers. + pub gfx: [Arc>; 2], + /// The status bar's activity bits and the CPU's Compare matches, which + /// the graphics board shows. + pub heartbeat: Arc, + pub fasttick: Arc, + /// What the graphics board's status bar shows: the activity bits, the + /// Compare matches and the CPU's cycle count and Count rate. + pub status: Arc, + clock: Clock, + last_clock: AtomicU64, + /// When the poll last returned lifted checked pages to the direct window. + last_retire: AtomicU64, + running: AtomicBool, +} + +impl TezroBoard { + /// Flush the disks on both channels of the storage controller. + pub fn flush_disks(&self) -> std::io::Result<()> { + (0..2).try_for_each(|channel| self.scsi.bus_for_channel(channel).unwrap().flush()) + } + + #[cfg(test)] + pub fn new(cpus: usize) -> Result { + Self::with_config(TezroBoardConfig { + payload: &[], + clock: Arc::new(|| 0), + flash_path: None, + station_mac: L1Config::default().mac, + timekeeper: Arc::new(Ds1744::new(YearBase::From(1968))?), + memory: &MemoryPopulation::default(), + cpus, + part: TezroPart::Mhz700, + v12: false, + }) + } + + pub fn with_config(config: TezroBoardConfig<'_>) -> Result { + let TezroBoardConfig { payload, clock, flash_path, station_mac, timekeeper, memory, cpus, part, v12 } = config; + if ![1, 2, 4].contains(&cpus) { + return Err(format!("{cpus} processors; a Tezro board fits 1, 2 or 4")); + } + memory.check_for_cpus(cpus)?; + let running = Population::fitted_cpus(cpus)?; + let node = Node::build(NodeConfig { + name: "Tezro", + payload, + clock: clock.clone(), + flash_path, + banks_mib: &memory.banks_mib(), + population: running, + running, + firmware: part.firmware(), + // The two-processor board's empty B and D positions start + // disabled, as the QEMU boot tests seed them; one and four + // processors leave no empty position the PROM recovers. + flash_records: if cpus == 2 { two_processor_environment() } else { Vec::new() }, + board_id: BOARD_ID, + l1: L1Config { + type_code: 0x39, + revision: Some([1, 48, 1]), + mac: station_mac, + port_speed_first: 1, + router: true, + ..Default::default() + }, + })?; + let serial: [Arc; SERIAL_PORTS] = std::array::from_fn(|_| Arc::new(CiSerialBackend::new())); + node.l1.set_serial_backend(serial[0].clone()); + + let io9 = Pic::new(IO9_WIDGET); + let pxbow = io9.enable_pxbow(HOST_WIDGET, clock.clone()).map_err(|e| format!("Tezro PXBOW: {e:?}"))?; + node.io.connect_widget(0, pxbow.widget_zero()); + node.io.connect_widget(IO9_WIDGET, pxbow.route(IO9_WIDGET)); + let gfx: [Arc>; 2] = std::array::from_fn(|_| Arc::new(Mutex::new(BedrockGfx::new(clock.clone())))); + let rendezvous = Rendezvous::new(running, clock.clone(), &node.lb); + node.lb.connect_scratch_hooks(rendezvous.clone()); + let publication = Arc::downgrade(&rendezvous); + node.io.on_scratch0_write(Box::new(move || { + if let Some(r) = publication.upgrade() { + r.presence_published(); + } + })); + for (pi, gfx) in node.pi.iter().zip(&gfx) { + pi.set_node_hooks(Arc::new(PiHooks { gfx: gfx.clone(), rendezvous: Arc::downgrade(&rendezvous) })); + } + let heartbeat = Arc::new(AtomicU64::new(0)); + let fasttick = Arc::new(AtomicU64::new(0)); + let status = crate::disp::StatusSource::new(heartbeat.clone(), fasttick.clone()); + let v12 = v12.then(|| fit_v12(&node, running, &pxbow, &gfx, &status)).transpose()?; + let pics = if cpus == 1 { + vec![io9.clone()] + } else { + // The PROM finds widget 11 only as a live link of widget 0's + // crossbar, so the second PIC is a port of the same switch. + let second = Pic::new(SECOND_PIC_WIDGET); + pxbow + .attach_endpoint(SECOND_PIC_WIDGET, second.widget()) + .map_err(|e| format!("Tezro: second PIC on the PXBOW: {e:?}"))?; + node.io.connect_widget(SECOND_PIC_WIDGET, pxbow.route(SECOND_PIC_WIDGET)); + vec![second, io9.clone()] + }; + for (index, pic) in pics.iter().enumerate() { + pic.connect_peer(Arc::new(BedrockPeer(Arc::downgrade(&node.io)))) + .map_err(|_| "Tezro: PIC peer connected twice".to_string())?; + // Each PIC owns its own 16 PI level sources, as QEMU numbers them. + for (interface, pi) in node.pi.iter().enumerate() { + pic.connect_pi(interface, pi.clone(), index * 2 * crate::dev::pic::PINS); + } + } + + let ioc4 = Arc::new(Ioc4::new(Ioc4Config::tezro())); + ioc4.attach_clock(timekeeper.clone()); + for (port, backend) in serial[1..].iter().enumerate() { + ioc4.set_serial_backend(port, backend.clone()); + } + let scsi = Arc::new(Isp12160::new(Isp12160::new_bus(), Isp12160::new_bus())); + scsi.set_heartbeat(heartbeat.clone()); + let eth = Bcm5701::new(Bcm5701Config::new(node.l1.station_mac()))?; + io9.attach(0, IOC4_DEVICE, ioc4.clone(), Function { self_ordered_pio: true, req64: false }); + io9.attach(0, SCSI_DEVICE, scsi.clone(), Function::default()); + io9.attach(0, ETH_DEVICE, eth.clone(), Function { self_ordered_pio: false, req64: true }); + for device in [IOC4_DEVICE, SCSI_DEVICE, ETH_DEVICE] { + io9.wire_inta(0, device); + } + Ok(Self { + node, + processors: running, + rendezvous, + pics, + ioc4, + scsi, + eth, + timekeeper, + serial, + part, + v12, + gfx, + heartbeat, + fasttick, + status, + last_clock: AtomicU64::new(clock()), + last_retire: AtomicU64::new(clock()), + clock, + running: AtomicBool::new(false), + }) + } + + pub fn cached_bus(&self) -> Arc { + self.node.bus.cached_view(self.node.md.dma_view(), self.node.md.ram_extent()) + } + + /// The bus the JIT compiles code through; see [`crate::ip35::Ip35Bus::code_view`]. + #[cfg(feature = "jitv2")] + pub fn code_bus(&self) -> Arc { + self.node.bus.code_view(self.cached_bus()) + } + + /// Everything back to its power-on state. The functions reset before + /// their PIC, so each samples its slot's REQ64# as the bridge's reset + /// ends, as on the bus. + pub fn power_on(&self) { + let node = &self.node; + self.rendezvous.reset(); + for pi in &node.pi { + pi.power_on(); + } + node.md.power_on(); + node.lb.power_on(); + node.io.power_on(); + node.ni.power_on(); + node.xb.power_on(); + node.l1.power_on(); + node.flash.power_on(); + self.ioc4.power_on(); + self.scsi.power_on(); + self.eth.power_on(); + for pic in &self.pics { + pic.power_on(); + } + for gfx in &self.gfx { + gfx.lock().reset(); + } + if let Some(v12) = &self.v12 { + v12.power_on(); + } + } + + pub fn start(&self) { + if let Some(v12) = &self.v12 { + v12.start(); + } + self.last_clock.store((self.clock)(), Ordering::Release); + self.node.md.start_clock(); + self.running.store(true, Ordering::Release); + for pi in &self.node.pi { + pi.start(); + } + self.node.l1.start(); + self.ioc4.start(); + self.eth.start(); + self.scsi.start(); + } + + pub fn poll(&self) { + if self.running.load(Ordering::Acquire) { + let now = (self.clock)(); + let previous = self.last_clock.swap(now, Ordering::AcqRel); + self.node.flash.advance_ns(now.saturating_sub(previous)); + for pi in &self.node.pi { + pi.poll(); + } + // Pages whose checked marks were lifted below MD's batch limit + // keep their region off the direct window until returned. + let retired = self.last_retire.load(Ordering::Acquire); + if now.saturating_sub(retired) >= RETIRE_INTERVAL_NS { + self.last_retire.store(now, Ordering::Release); + self.node.md.retire_lifted_marks(); + } + } + } + + pub fn stop(&self) { + self.poll(); + if let Some(v12) = &self.v12 { + v12.stop(); + } + self.running.store(false, Ordering::Release); + self.node.md.stop_clock(); + self.scsi.stop(); + self.eth.stop(); + self.ioc4.stop(); + self.node.l1.stop(); + for pi in &self.node.pi { + pi.stop(); + } + } +} + +impl Drop for TezroBoard { + fn drop(&mut self) { + self.stop(); + } +} + +/// Where the V12 sits: widget 12 of the crossbar behind the PIC that owns +/// widget 0. On the two- and four-processor board this is QEMU's graphics +/// slot. On the one-processor board the tower guide gives one XIO connector +/// for a VPro board but names neither its widget nor its route; widget 12 +/// behind the same PXBOW is inferred, not established. +const V12_WIDGET: u8 = 12; + +/// The longest a stopped slot holds a store before letting it go. +const STALL_LIMIT_NS: u64 = 1_000_000_000; + +/// One Bedrock PI's graphics flow control: the register block, and the +/// Bedrock's stop below the bias. A counted store that leaves the counter +/// below the bias raises the slot's graphics interrupt and stops the slot: +/// its next counted store waits until the processor has taken the +/// interrupt, software has written the counter, the bias or the interrupt +/// count, or returned credits have lifted the counter above the bias (the +/// SN1 header's credit-counter comment, as QEMU's Bedrock models it). A +/// waiting store is answered busy, so the processor makes it again; it is +/// counted once, when it goes. +/// +/// Approximations: whether the edge is below the bias or at it is unknown; +/// below, QEMU's reading. How long the chip holds a store is unknown: after +/// STALL_LIMIT_NS of host time the hold ends and is logged, QEMU's bound. A +/// held processor here retries the store and can take an interrupt between +/// attempts, where the chip holds the store itself. +pub(crate) struct BedrockGfx { + regs: PiGfx, + /// When each slot's waiting store was first held. + held_since: [Option; PI_GFX_SLOTS], + clock: Clock, +} + +impl BedrockGfx { + fn new(clock: Clock) -> Self { + Self { regs: PiGfx::default(), held_since: [None; PI_GFX_SLOTS], clock } + } + + fn reset(&mut self) { + self.regs = PiGfx::default(); + self.held_since = [None; PI_GFX_SLOTS]; + } + + fn read(&self, n: u32) -> u64 { + self.regs.read(n) + } + + fn write(&mut self, n: u32, value: u64) { + self.regs.write(n, value); + let slot = (n / PI_GFX_SLOT_REGS) as usize; + match n % PI_GFX_SLOT_REGS { + PI_GFX_CREDIT | PI_GFX_BIAS => self.release_if_above(slot), + // IRIX's graphics interrupt handler zeroes it. + PI_GFX_INT_COUNT => self.release(slot), + _ => {} + } + } + + /// A store `slot`'s processor makes at I/O offset `address`. + fn store(&mut self, slot: usize, address: u64) -> Store { + if !self.regs.covers(slot, address) { + return Store::Uncounted; + } + if self.regs.stopped[slot] { + let now = (self.clock)(); + let since = *self.held_since[slot].get_or_insert(now); + if now.saturating_sub(since) < STALL_LIMIT_NS { + return Store::Held; + } + dlog!( + LogModule::Bedrock, + "PI graphics slot {} held for {} ms with the counter at {:#x} and the interrupt not taken; letting the store go", + slot_name(slot), + STALL_LIMIT_NS / 1_000_000, + self.regs.credit[slot] + ); + self.release(slot); + } + self.regs.count_store(slot, address, "Tezro PI"); + let below_bias = self.regs.credit[slot] < self.regs.bias[slot]; + if below_bias { + self.regs.stopped[slot] = true; + } + Store::Counted { below_bias } + } + + /// The processor took the slot's graphics interrupt. + fn taken(&mut self, slot: usize) { + self.release(slot); + } + + /// Put back exactly the credit a count took. Unlike a returned credit + /// this never wraps the counter up, since it only undoes a count. + fn refund(&mut self, slot: usize) { + self.regs.credit[slot] = (self.regs.credit[slot] + 1) & PI_GFX_CREDIT_MASK; + self.release_if_above(slot); + } + + fn add_credits(&mut self, slot: usize, credits: u32) { + self.regs.add_credits(slot, credits, "Tezro PI"); + self.release_if_above(slot); + } + + fn release(&mut self, slot: usize) { + self.regs.stopped[slot] = false; + self.held_since[slot] = None; + } + + fn release_if_above(&mut self, slot: usize) { + if self.regs.credit[slot] > self.regs.bias[slot] { + self.release(slot); + } + } +} + +fn slot_name(slot: usize) -> char { + (b'A' + slot as u8) as char +} + +/// A PI's node hooks: its graphics flow-control registers, served from its +/// block, and its real-time registers' part in the boot rendezvous. +struct PiHooks { + gfx: Arc>, + rendezvous: Weak, +} +impl NodeHooks for PiHooks { + fn rt_compare_written(&self, interface: u8, slot: usize) { + if let Some(r) = self.rendezvous.upgrade() { + r.rt_compare_written(interface, slot); + } + } + fn rt_counter_read(&self, _interface: u8) { + if let Some(r) = self.rendezvous.upgrade() { + r.rt_counter_read(); + } + } + fn gfx_read(&self, index: usize) -> Option { + (index < PI_GFX_REGS as usize).then(|| self.gfx.lock().read(index as u32)) + } + fn gfx_write(&self, index: usize, value: u64) -> bool { + if index >= PI_GFX_REGS as usize { + return false; + } + self.gfx.lock().write(index as u32, value); + true + } + fn gfx_reset(&self) { + self.gfx.lock().reset(); + } + fn gfx_interrupt_taken(&self, slot: usize) { + self.gfx.lock().taken(slot); + } +} + +/// The node's end of the V12's credit return: a widget's credits go to the +/// interface and slot the node's graphics map names. Only node 0's own +/// interfaces are reachable. +/// +/// A head reports a counted store it cannot take into its FIFO yet as owed, +/// and the node keeps a ledger of what each widget owes. A write to a map +/// entry first settles the debt of the widget the entry names, under the +/// entry as it stood, and drops the head's later repayments of it. IRIX's +/// detach zeroes the processor's entry and then saves the counter for the +/// next attach, so a credit arriving after the detach would be lost to every +/// later attach of that flow. This is the QEMU model's choice for a board +/// slower than the real one, not a known property of the I/O interface. +struct NodeCredits { + io: Weak, + gfx: [Arc>; 2], + owed: Mutex, +} +impl NodeCredits { + /// Credits from `widget`, to the slot `map` names. + fn route(&self, map: [u64; 2], widget: u8, n: u32) { + match gfx_route(map, widget) { + Some(r) if r.node == 0 => self.gfx[r.interface].lock().add_credits(r.slot, n), + other => dlog!(LogModule::Bedrock, "Tezro: {n} V12 credits for widget {widget} have no local route ({other:?})"), + } + } + + fn route_now(&self, widget: u8, n: u32) { + if let Some(io) = self.io.upgrade() { + self.route([io.gfx_map(0), io.gfx_map(1)], widget, n); + } + } + + /// A store replacing graphics map entry `entry`'s value `old` with `new`. + fn map_written(&self, entry: usize, old: u64, new: u64) { + let Some(io) = self.io.upgrade() else { return }; + if let Some((widget, credits)) = self.owed.lock().settle(old, new) { + let mut map = [io.gfx_map(0), io.gfx_map(1)]; + map[entry] = old; + self.route(map, widget, credits); + } + } +} +impl PiGfxLink for NodeCredits { + fn return_credits(&self, widget: u32, n: u32) { + self.route_now(widget as u8, n); + } + fn owe_credits(&self, widget: u32, n: i32) { + self.owed.lock().owe(widget as u8, n); + } + fn repay_credits(&self, widget: u32, n: u32) { + let due = self.owed.lock().repay(widget as u8, n); + if due != 0 { + self.route_now(widget as u8, due); + } + } +} + +/// The credits each of a node's sixteen widgets owes. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub(crate) struct OwedCredits([u32; 16]); + +impl OwedCredits { + /// `n` more credits owed by `widget`; a negative count cancels owed + /// credits a head reset threw away, never below zero. + pub fn owe(&mut self, widget: u8, n: i32) { + let owed = &mut self.0[usize::from(widget & 0xf)]; + *owed = if n >= 0 { owed.saturating_add(n as u32) } else { owed.saturating_sub(n.unsigned_abs()) }; + } + + /// `n` owed credits repaid: what is still owed of them, to route now. A + /// repayment of credits already settled routes nothing. + pub fn repay(&mut self, widget: u8, n: u32) -> u32 { + let owed = &mut self.0[usize::from(widget & 0xf)]; + let repaid = n.min(*owed); + *owed -= repaid; + repaid + } + + /// A store replacing graphics map entry `old` with `new`: the widget + /// `old` names and the credits it owed, to route under `old` before the + /// new value takes effect. Rewriting the same value settles nothing. + pub fn settle(&mut self, old: u64, new: u64) -> Option<(u8, u32)> { + let widget = (old & 0xf) as u8; + if old == new || self.0[usize::from(widget)] == 0 { + return None; + } + Some((widget, std::mem::take(&mut self.0[usize::from(widget)]))) + } +} + +/// Fit the V12 behind the PXBOW: each processor's stores into its slot's +/// graphics page are counted, the board's returned credits go where the +/// node's graphics map sends them, its interrupt packets and DMA reach +/// Bedrock IO as the PIC's do. +fn fit_v12( + node: &Node, + running: Population, + pxbow: &Arc, + gfx: &[Arc>; 2], + status: &Arc, +) -> Result, String> { + let io = Arc::downgrade(&node.io); + let packets: PacketSender = Arc::new(move |widget, address, vector, set| { + if widget != HOST_WIDGET { + dlog!(LogModule::Bedrock, "Tezro: V12 interrupt to widget {widget} refused; only Bedrock IO is reachable"); + return false; + } + let Some(io) = io.upgrade() else { return false }; + let payload = u64::from(vector) | if set { 1 << 8 } else { 0 }; + io.inbound_write(V12_WIDGET, Some(INBOUND_DEVICE), address, 8, payload) == BUS_OK + }); + let dma = Arc::new(WidgetDma { io: Arc::downgrade(&node.io), widget: V12_WIDGET }); + let v12 = Odyssey::new_widget_relative( + V12_WIDGET, + GraphicsBoard::V12.as_str(), + packets, + dma, + status.clone(), + )?; + let credits = Arc::new(NodeCredits { io: Arc::downgrade(&node.io), gfx: gfx.clone(), owed: Mutex::default() }); + v12.set_pi_link(credits.clone()); + let settle = Arc::downgrade(&credits); + node.io.on_gfx_map_write(Box::new(move |entry, old, new| { + if let Some(credits) = settle.upgrade() { + credits.map_written(entry, old, new); + } + })); + pxbow + .attach_endpoint(V12_WIDGET, v12.clone() as Arc) + .map_err(|e| format!("Tezro V12: {e:?}"))?; + node.io.connect_widget(V12_WIDGET, pxbow.route(V12_WIDGET)); + node.count_io_stores(Box::new(SlotStores { pi: node.pi.clone(), gfx: gfx.clone(), running })); + Ok(v12) +} + +/// Each processor's uncached I/O stores, counted against its own slot, as +/// each store enters the processor's own interface. A thread that runs no +/// processor counts as CPU 0, in PI0 slot A. +struct SlotStores { + pi: [Arc; 2], + gfx: [Arc>; 2], + running: Population, +} +impl SlotStores { + fn slot(&self) -> (usize, usize) { + let (interface, slot) = self.running.slot_of(crate::cpu::smp::current_cpu()).unwrap_or((0, 0)); + (usize::from(interface), slot) + } +} +impl StoreCounter for SlotStores { + fn count(&self, address: u64) -> Store { + let (interface, slot) = self.slot(); + let store = self.gfx[interface].lock().store(slot, address); + if store == (Store::Counted { below_bias: true }) { + // The slot's graphics interrupt: bit 1 + slot of bank 0. + self.pi[interface].set_pending(0, 1 + slot as u8); + } + store + } + fn refund(&self, _address: u64) { + let (interface, slot) = self.slot(); + self.gfx[interface].lock().refund(slot); + } +} + +/// The far end of a PIC's link: Bedrock IO. Interrupt packets and DMA arrive +/// there as inbound packets from the PIC's widget, device 0, as QEMU's +/// Bedrock IO takes them by default. DMA aimed at another widget is refused: +/// peer-to-peer XIO is not modelled. +struct BedrockPeer(Weak); + +const INBOUND_DEVICE: u8 = 0; + +impl BedrockPeer { + fn io(&self) -> Result, Refused> { + self.0.upgrade().ok_or(Refused) + } + + fn host(widget: u8) -> Result<(), Refused> { + if widget == HOST_WIDGET { + Ok(()) + } else { + dlog!(LogModule::Bedrock, "Tezro: PIC DMA to widget {widget} refused; only Bedrock IO is reachable"); + Err(Refused) + } + } + + /// The legs of a transfer: each the largest naturally aligned size of + /// eight bytes or fewer that fits, so word accesses keep their width. + fn legs(address: u64, len: usize) -> impl Iterator { + let mut offset = 0; + std::iter::from_fn(move || { + if offset == len { + return None; + } + let at = address + offset as u64; + let size = [8usize, 4, 2, 1] + .into_iter() + .find(|&n| at.is_multiple_of(n as u64) && n <= len - offset) + .unwrap_or(1); + let leg = (offset, size); + offset += size; + Some(leg) + }) + } +} + +impl PicPeer for BedrockPeer { + fn interrupt(&self, source_widget: u8, address: u64, payload: u64) -> Result<(), Refused> { + let io = self.io()?; + (io.inbound_write(source_widget, Some(INBOUND_DEVICE), address, 8, payload) == BUS_OK) + .then_some(()) + .ok_or(Refused) + } + + fn dma_read(&self, source_widget: u8, widget: u8, address: u64, data: &mut [u8]) -> Result<(), Refused> { + Self::host(widget)?; + let io = self.io()?; + for (offset, size) in Self::legs(address, data.len()) { + let value = io.inbound_read(source_widget, Some(INBOUND_DEVICE), address + offset as u64, size as u8); + if value.status != BUS_OK { + return Err(Refused); + } + data[offset..offset + size].copy_from_slice(&value.data.to_be_bytes()[8 - size..]); + } + Ok(()) + } + + fn dma_write(&self, source_widget: u8, widget: u8, address: u64, data: &[u8]) -> Result<(), Refused> { + Self::host(widget)?; + let io = self.io()?; + for (offset, size) in Self::legs(address, data.len()) { + let value = data[offset..offset + size].iter().fold(0u64, |v, b| (v << 8) | u64::from(*b)); + if io.inbound_write(source_widget, Some(INBOUND_DEVICE), address + offset as u64, size as u8, value) != BUS_OK { + return Err(Refused); + } + } + Ok(()) + } +} + +#[cfg(test)] +mod tests; diff --git a/src/ip53_machine/tests.rs b/src/ip53_machine/tests.rs new file mode 100644 index 000000000..a743b82ff --- /dev/null +++ b/src/ip53_machine/tests.rs @@ -0,0 +1,836 @@ +use super::*; +use crate::traits::{BUS_ERR, BUS_OK}; +use std::sync::atomic::AtomicU64; + +/// A PIO offset in Bedrock IO's small window: widget in bits 27:24. +fn widget(widget: u64, offset: u64) -> u64 { + (widget << 24) | offset +} + +/// Configuration register 0 of `device` on the IO9 PIC's head 0. +fn config_id(device: u64) -> u64 { + widget(15, 0x2_0000 + (device << 12)) +} + +/// The processor finds the PXBOW at widget 0 and the IO9's PIC at widget +/// 15, as the QEMU profile maps them; the one-processor board has no widget +/// 11 and no other widget answers. +#[test] +fn widget_0_is_the_pxbow_and_widget_15_the_io9_pic() { + let board = TezroBoard::new(1).unwrap(); + let io = &board.node.io; + let pxbow = io.pio_read(widget(0, 4), 4); + assert_eq!(pxbow.status, BUS_OK); + assert_eq!((pxbow.data >> 12) & 0xffff, u64::from(crate::dev::pxbow::PART)); + let pic = io.pio_read(widget(15, 0), 8); + assert_eq!(pic.status, BUS_OK); + assert_eq!(pic.data, u64::from(crate::dev::pic::identity(0))); + for absent in [8, 9, 11, 12, 13, 14] { + assert_eq!(io.pio_read(widget(absent, 0), 8).status, BUS_ERR, "widget {absent}"); + } + assert_eq!(board.pics.len(), 1); +} + +/// IO9's functions sit at PCI devices 1, 3 and 4 of head 0 with the +/// revisions the QEMU profile selects, and device 2 is empty. +#[test] +fn the_io9_bus_holds_the_ioc4_the_isp12160_and_the_bcm5701() { + let board = TezroBoard::new(1).unwrap(); + let io = &board.node.io; + for (device, id, revision) in [(1, 0x100a_10a9, 0x53), (3, 0x1216_1077, 0x06), (4, 0x1645_14e4, 0x00)] { + assert_eq!(io.pio_read(config_id(device), 4).data, id, "device {device}"); + assert_eq!(io.pio_read(config_id(device) + 8, 4).data & 0xff, revision, "device {device}"); + } + assert_eq!(io.pio_read(config_id(2), 4).status, BUS_ERR, "device 2 is empty"); +} + +/// The BCM5701's slot drives REQ64#, so the chip reports a 64-bit bus, +/// and still does after the board is reset. +#[test] +fn the_bcm5701_sees_req64_before_and_after_a_reset() { + use crate::pci::PciBusDevice; + const PCISTATE: usize = 0x70 / 4; + const BUS_32BIT: u32 = 1 << 4; + let board = TezroBoard::new(1).unwrap(); + assert_eq!(board.eth.config().read(PCISTATE) & BUS_32BIT, 0); + board.power_on(); + assert_eq!(board.eth.config().read(PCISTATE) & BUS_32BIT, 0); +} + +/// The board identity the PROM reads: Bedrock board word 0x2001 and the +/// L1's Tezro station address. +#[test] +fn the_board_reports_the_tezro_board_word() { + let board = TezroBoard::new(1).unwrap(); + let id = board.node.bus.read64(0x1000_0000 + 0x400_0000); + assert_eq!((id.status, id.data), (BUS_OK, BOARD_ID)); + assert_eq!(board.node.l1.station_mac(), L1Config::default().mac); +} + +/// PIC DMA reaches memory through Bedrock IO in naturally aligned legs, and +/// DMA aimed at any widget but Bedrock IO is refused. +#[test] +fn pic_dma_reaches_memory_through_bedrock_io_only() { + let board = TezroBoard::new(1).unwrap(); + // Enable bank pair 0, as the Fuel's DMA tests do. + assert_eq!(board.node.md.md_bus().write64(8, 0x33), BUS_OK); + let peer = BedrockPeer(Arc::downgrade(&board.node.io)); + let data: Vec = (1..=19).collect(); + assert_eq!(peer.dma_write(15, HOST_WIDGET, 0x1003, &data), Ok(())); + let mut back = vec![0; data.len()]; + assert_eq!(peer.dma_read(15, HOST_WIDGET, 0x1003, &mut back), Ok(())); + assert_eq!(back, data); + let memory = board.node.md.dma_view(); + assert_eq!(memory.read64(0x1000).data, 0x0000_0001_0203_0405); + assert_eq!(peer.dma_write(15, 12, 0x1003, &data), Err(Refused)); + assert_eq!(peer.dma_read(15, 0, 0x1003, &mut back), Err(Refused)); +} + +#[test] +fn transfer_legs_keep_aligned_widths() { + let legs: Vec<_> = BedrockPeer::legs(0x1003, 19).collect(); + assert_eq!(legs, [(0, 1), (1, 4), (5, 8), (13, 4), (17, 2)]); + assert_eq!(BedrockPeer::legs(0x2000, 16).collect::>(), [(0, 8), (8, 8)]); + assert_eq!(BedrockPeer::legs(0x2000, 0).count(), 0); +} + +/// The two- and four-processor board adds a second PIC at widget 11 with +/// nothing on its buses, keeps the IO9 on widget 15 behind widget 0, and +/// leaves the V12's widget 12 empty. +#[test] +fn two_or_four_processors_bring_the_second_pic_at_widget_11() { + for cpus in [2, 4] { + let board = TezroBoard::new(cpus).unwrap(); + let io = &board.node.io; + assert_eq!(board.pics.iter().map(|p| p.widget_number()).collect::>(), [11, 15]); + assert_eq!(io.pio_read(widget(11, 0), 8).data, u64::from(crate::dev::pic::identity(0))); + assert_eq!(io.pio_read(widget(15, 0), 8).data, u64::from(crate::dev::pic::identity(0))); + assert_eq!(io.pio_read(widget(12, 0), 8).status, BUS_ERR, "no V12"); + assert_eq!((io.pio_read(widget(0, 4), 4).data >> 12) & 0xffff, u64::from(crate::dev::pxbow::PART)); + for device in 1..=4 { + assert_eq!(io.pio_read(widget(11, 0x2_0000 + (device << 12)), 4).status, BUS_ERR, "widget 11 device {device}"); + } + assert_eq!(io.pio_read(config_id(1), 4).data, 0x100a_10a9); + } +} + +// Widget 0's crossbar as PROM 6.210 and the IP35 kernel walk it, from how the PROM finds +// widgets 0xb and 0xf. +// QEMU's sn/builder/tezro-two-four-processor-board in +// tests/qtest/sn-builder-test.c still expects link 11 absent; this board +// follows the PROM's walk instead. + +/// Link `n`'s register block in widget 0, as the 32-bit word address the +/// PROM issues: the low word of each doubleword. +fn link_word(n: u64, register: u64) -> u64 { + widget(0, 0x100 + (n - 8) * 0x40 + register + 4) +} +const LINK_CONTROL: u64 = 0x08; +const LINK_STATUS: u64 = 0x10; +const LINK_ARB_UPPER: u64 = 0x18; +const LINK_ARB_LOWER: u64 = 0x20; +const LINK_STATUS_CLEAR: u64 = 0x28; +const LINK_RESET: u64 = 0x30; +const LINK_AUX: u64 = 0x38; +const ALIVE: u64 = 1 << 31; +const PRESENT: u64 = 1 << 5; +const FAILURE_MODE: u64 = 1 << 6; + +fn read_word(board: &TezroBoard, address: u64) -> u64 { + let r = board.node.io.pio_read(address, 4); + assert_eq!(r.status, BUS_OK, "{address:#x}"); + r.data +} + +/// Widget 0's part and manufacturer, from its identity word. +fn widget_zero_part(board: &TezroBoard) -> (u64, u64) { + let id = read_word(board, widget(0, 4)); + ((id >> 12) & 0xffff, (id >> 1) & 0x7ff) +} + +fn alive_and_present(board: &TezroBoard, n: u64) -> (bool, bool) { + (read_word(board, link_word(n, LINK_STATUS)) & ALIVE != 0, read_word(board, link_word(n, LINK_AUX)) & PRESENT != 0) +} + +/// The PROM's per-link test: alive and present, except that link 15 is +/// taken regardless when widget 0 is part 0xd100. +fn prom_uses(board: &TezroBoard, n: u64) -> bool { + let (alive, present) = alive_and_present(board, n); + (alive && present) || (n == 15 && widget_zero_part(board).0 == 0xd100) +} + +/// The kernel's: alive, and present unless widget 0 is part 0xd100 with +/// manufacturer 0 and the link is 15. +fn kernel_uses(board: &TezroBoard, n: u64) -> bool { + let (alive, present) = alive_and_present(board, n); + alive && (present || (n == 15 && widget_zero_part(board) == (0xd100, 0))) +} + +/// Links 8 to 15 that pass both tests on their own bits, without either +/// guest's link-15 exception, as a widget mask. +fn live_links(board: &TezroBoard) -> u16 { + (8..16).filter(|&n| alive_and_present(board, n) == (true, true)).fold(0, |m, n| m | 1 << n) +} + +/// A board whose clock the test advances. +fn board_with_clock(cpus: usize) -> (TezroBoard, Arc) { + let now = Arc::new(AtomicU64::new(0)); + let clock = now.clone(); + let board = TezroBoard::with_config(TezroBoardConfig { + payload: &[], + clock: Arc::new(move || clock.load(Ordering::SeqCst)), + flash_path: None, + station_mac: L1Config::default().mac, + timekeeper: Arc::new(Ds1744::new(YearBase::From(1968)).unwrap()), + memory: &MemoryPopulation::default(), + cpus, + part: TezroPart::Mhz700, + v12: false, + }) + .unwrap(); + (board, now) +} + +/// On the two- and four-processor board widget 0's crossbar shows the +/// hub's link 10 and both PICs' links 11 and 15 alive and present, with no +/// failure mode and arbitration bits 0 and 2 clear, so the PROM and the +/// kernel both take 11 and 15 without leaning on the link-15 exception. +/// Every other link reads empty. A power-on keeps the same picture. +#[test] +fn widget_0_shows_both_pic_links_live_on_the_larger_board() { + for cpus in [2, 4] { + let board = TezroBoard::new(cpus).unwrap(); + for pass in ["built", "after power-on"] { + assert_eq!(widget_zero_part(&board), (0xd100, 0), "{cpus} CPUs, {pass}"); + assert_eq!(read_word(&board, widget(0, 4)) & 1, 1); + assert_eq!(live_links(&board), 1 << 10 | 1 << 11 | 1 << 15, "{cpus} CPUs, {pass}"); + for n in [11, 15] { + assert!(prom_uses(&board, n) && kernel_uses(&board, n), "link {n}, {cpus} CPUs, {pass}"); + assert_eq!(read_word(&board, link_word(n, LINK_AUX)) & FAILURE_MODE, 0, "link {n}"); + assert_eq!(read_word(&board, link_word(n, LINK_ARB_UPPER)) & 0b101, 0, "link {n}: unclaimed"); + } + for n in [8, 9, 12, 13, 14] { + assert_eq!(alive_and_present(&board, n), (false, false), "link {n}, {cpus} CPUs, {pass}"); + assert!(!prom_uses(&board, n) && !kernel_uses(&board, n)); + } + board.power_on(); + } + } +} + +/// The one-processor board has no second PIC: link 11 reads empty and +/// widget 11 does not answer, while link 15 stays usable. +#[test] +fn the_one_processor_board_shows_no_link_11() { + let board = TezroBoard::new(1).unwrap(); + assert_eq!(live_links(&board), 1 << 10 | 1 << 15); + assert_eq!(alive_and_present(&board, 11), (false, false)); + assert!(!prom_uses(&board, 11) && !kernel_uses(&board, 11)); + assert!(prom_uses(&board, 15) && kernel_uses(&board, 15)); + assert_eq!(board.node.io.pio_read(widget(11, 4), 4).status, BUS_ERR); +} + +/// Widget 11 answers as a bridge-class PIC with both heads, the identity +/// the PROM's "Found PIC" path needs (manufacturer 0x24, bit 0 set, part +/// 0xd102 on head 0), and takes the PROM's control-word writes on both +/// heads. Widget 15 keeps its own. +#[test] +fn widget_11_answers_as_a_pic_with_both_heads() { + use crate::dev::pic::HEAD_OFFSET; + let board = TezroBoard::new(4).unwrap(); + let io = &board.node.io; + for (head, part) in [(0, 0xd102), (HEAD_OFFSET, 0xd112)] { + let id = read_word(&board, widget(11, head + 4)); + assert_eq!((id & 1, (id >> 1) & 0x7ff, (id >> 12) & 0xffff), (1, 0x24, part), "head at {head:#x}"); + // Widget control, as the PROM's table-driven field writes it. + let control = widget(11, head + 0x24); + let before = read_word(&board, control); + assert_eq!(io.pio_write(control, 4, (before & !0xf000) | 0x4000), BUS_OK); + assert_eq!(read_word(&board, control) & 0xf000, 0x4000, "head at {head:#x}"); + } + assert_eq!(read_word(&board, widget(15, 4)) >> 12 & 0xffff, 0xd102); +} + +/// The PROM's bring-up of link 11 (read the status clear, write credit 2, +/// clear both arbitration registers, then the 0xd102 credit 3) leaves the +/// link alive and present and the credit field reading back. +#[test] +fn the_proms_link_bring_up_keeps_link_11_alive() { + let board = TezroBoard::new(4).unwrap(); + let io = &board.node.io; + read_word(&board, link_word(11, LINK_STATUS_CLEAR)); + assert_eq!(io.pio_write(link_word(11, LINK_CONTROL), 4, 0x0008_0000), BUS_OK); + assert_eq!(io.pio_write(link_word(11, LINK_ARB_UPPER), 4, 0), BUS_OK); + assert_eq!(io.pio_write(link_word(11, LINK_ARB_LOWER), 4, 0), BUS_OK); + assert_eq!(alive_and_present(&board, 11), (true, true)); + let control = read_word(&board, link_word(11, LINK_CONTROL)); + assert_eq!((control >> 18) & 0x1f, 2); + assert_eq!(io.pio_write(link_word(11, LINK_CONTROL), 4, (control & !(0x1f << 18)) | 3 << 18), BUS_OK); + assert_eq!((read_word(&board, link_word(11, LINK_CONTROL)) >> 18) & 0x1f, 3); + assert_eq!(alive_and_present(&board, 11), (true, true)); + assert_eq!(io.pio_read(widget(11, 4), 4).status, BUS_OK); +} + +/// The processor reaches widget 11 through the crossbar, not around it: a +/// reset of link 11 or of the hub's link 10 takes widget 11 out of reach +/// for the pulse, and it answers again, registers kept, once it ends. A +/// reset of link 15 leaves widget 11 alone. +#[test] +fn widget_11_is_routed_through_widget_0s_crossbar() { + let (board, now) = board_with_clock(4); + let io = &board.node.io; + let route = widget(11, 0x130); + assert_eq!(io.pio_write(route, 8, 0x1122_3344_5566_7788), BUS_OK); + for link in [11, 10] { + now.fetch_add(100, Ordering::SeqCst); + assert_eq!(io.pio_write(link_word(link, LINK_RESET), 4, 0), BUS_OK); + assert_eq!(io.pio_read(route, 8).status, BUS_ERR, "down during link {link}'s pulse"); + now.fetch_add(20, Ordering::SeqCst); + let r = io.pio_read(route, 8); + assert_eq!((r.status, r.data), (BUS_OK, 0x1122_3344_5566_7788), "after link {link}'s pulse"); + assert_eq!(alive_and_present(&board, 11), (true, true)); + } + now.fetch_add(100, Ordering::SeqCst); + assert_eq!(io.pio_write(link_word(15, LINK_RESET), 4, 0), BUS_OK); + assert_eq!(io.pio_read(route, 8).status, BUS_OK, "link 15's pulse"); + assert_eq!(io.pio_read(widget(15, 4), 4).status, BUS_ERR); + assert_eq!(board.pics[1].pxbow().unwrap().endpoint_widgets(), 1 << 11 | 1 << 15); + assert!(board.pics[0].pxbow().is_none(), "the second PIC has no crossbar of its own"); +} + +/// Every fitted processor slot reads present, in the board's order (CPU 1 +/// in PI1/A, CPUs 2 and 3 in the B slots), and every fitted processor runs +/// from reset with its enable set, as QEMU's node grants them. +#[test] +fn fitted_processors_read_present_and_start_enabled() { + use crate::dev::bedrock_pi::reg::{CPU_A_ENABLE, CPU_B_ENABLE, INTERFACE_SLOT_A, INTERFACE_SLOT_B}; + for (cpus, present) in [(1, [[1, 0], [0, 0]]), (2, [[1, 0], [1, 0]]), (4, [[1, 1], [1, 1]])] { + let board = TezroBoard::new(cpus).unwrap(); + assert_eq!(board.processors.count(), cpus); + for (interface, pi) in board.node.pi.iter().enumerate() { + let slots = [pi.read64(INTERFACE_SLOT_A).data, pi.read64(INTERFACE_SLOT_B).data]; + assert_eq!(slots, present[interface], "{cpus} CPUs, PI{interface}"); + let enabled = [pi.read64(CPU_A_ENABLE).data & 1, pi.read64(CPU_B_ENABLE).data & 1]; + assert_eq!(enabled, present[interface], "{cpus} CPUs, PI{interface}"); + } + } +} + +/// A store a processor makes into its graphics page is counted against its +/// own slot, and its interrupt raised on its own interface: CPU 1 sits in +/// PI1 slot A, CPU 2 in PI0 slot B. +#[test] +fn each_processors_graphics_stores_count_against_its_own_slot() { + let board = Arc::new(board_with_v12(4)); + let stores = SlotStores { pi: board.node.pi.clone(), gfx: board.gfx.clone(), running: board.processors }; + let stores = Arc::new(stores); + for (cpu, interface, slot) in [(1u8, 1usize, 0usize), (2, 0, 1)] { + let (stores, board) = (stores.clone(), board.clone()); + std::thread::spawn(move || { + crate::cpu::smp::set_current_cpu(cpu); + assert_eq!(stores.slot(), (interface, slot), "CPU {cpu}"); + // No page enabled: uncounted, and no slot's credit moves. + assert_eq!(stores.count(0x1000), Store::Uncounted); + let _ = &board; + }) + .join() + .unwrap(); + } + assert_eq!(stores.slot(), (0, 0), "a thread that runs no processor counts as CPU 0"); +} + +/// Only the counts a board fits are accepted. +#[test] +fn the_board_refuses_a_population_it_cannot_fit() { + assert!(TezroBoard::new(3).is_err()); + assert!(TezroBoard::new(0).is_err()); + assert!(TezroBoard::new(5).is_err()); +} + +/// Each R16000 part resets with the PRId and Config word the CPU models give +/// it, and the board's flash and count rate follow its clock. `r10000` takes +/// the default 700 MHz part; other parts are refused. +#[test] +fn each_r16000_part_resets_with_its_own_identity_and_clock() { + // Building a processor takes more stack than a debug build's test thread has. + std::thread::Builder::new().stack_size(64 << 20).spawn(|| { + use crate::config::CpuModel; + use crate::cpu::mips_cache_shadow::{CONFIG0_RESET_MASK, R16000_800_CONFIG0, R16000_MHZ700_CONFIG0}; + assert_eq!(TezroPart::of(CpuModel::R16000Mhz700), Ok(TezroPart::Mhz700)); + assert_eq!(TezroPart::of(CpuModel::R10000), Ok(TezroPart::Mhz700)); + assert_eq!(TezroPart::of(CpuModel::R16000), Ok(TezroPart::Mhz800)); + for cpu in [CpuModel::R4400, CpuModel::R5000, CpuModel::R12000, CpuModel::R14000] { + assert!(TezroPart::of(cpu).is_err(), "{cpu:?}"); + } + for (part, prid, config0, mhz) in [ + (TezroPart::Mhz700, R16000_MHZ700_PRID, R16000_MHZ700_CONFIG0, 700), + (TezroPart::Mhz800, 0x0f42, R16000_800_CONFIG0, 800), + ] { + // One GiB keeps the memory below 4 GiB, which a jitv2 build's + // uncached code views need. + let board = TezroBoard::with_config(TezroBoardConfig { + payload: &[], + clock: Arc::new(|| 0), + flash_path: None, + station_mac: L1Config::default().mac, + timekeeper: Arc::new(Ds1744::new(YearBase::From(1968)).unwrap()), + memory: &MemoryPopulation::one_gib(), + cpus: 1, + part, + v12: false, + }) + .unwrap(); + let cpu = build_cpu::(&board); + assert_eq!((cpu.core.cp0_prid, cpu.core.reset_prid), (prid, prid), "{part:?}"); + assert_eq!(cpu.core.cp0_config & CONFIG0_RESET_MASK, config0, "{part:?}"); + assert_eq!(cpu.core.cp0_config >> 26, 0b011_011, "{part:?}: 32 KB primary caches"); + assert_eq!(part.mhz(), mhz); + assert_eq!(part.count_hz(), u64::from(mhz) * 500_000, "{part:?}: Count at half the clock"); + assert_eq!(board.part.firmware().cpu_clock_hz, mhz * 1_000_000); + } + }).unwrap().join().unwrap(); +} + +/// The poll returns lifted checked pages to the direct window once every +/// interval of board time while the board runs, not on every batch. +#[test] +fn the_poll_retires_lifted_pages_once_an_interval() { + let now = Arc::new(AtomicU64::new(0)); + let clock = now.clone(); + let board = TezroBoard::with_config(TezroBoardConfig { + payload: &[], + clock: Arc::new(move || clock.load(Ordering::SeqCst)), + flash_path: None, + station_mac: L1Config::default().mac, + timekeeper: Arc::new(Ds1744::new(YearBase::From(1968)).unwrap()), + memory: &MemoryPopulation::default(), + cpus: 1, + part: TezroPart::Mhz700, + v12: false, + }) + .unwrap(); + board.running.store(true, Ordering::SeqCst); + now.store(RETIRE_INTERVAL_NS - 1, Ordering::SeqCst); + board.poll(); + assert_eq!(board.last_retire.load(Ordering::SeqCst), 0); + now.store(RETIRE_INTERVAL_NS, Ordering::SeqCst); + board.poll(); + assert_eq!(board.last_retire.load(Ordering::SeqCst), RETIRE_INTERVAL_NS); + board.running.store(false, Ordering::SeqCst); +} + +/// IOC4 reads its 32-bit registers in the processor's byte order through +/// the IO9 PIC's self-ordered slot: what the processor writes is the +/// register's value, as on the QEMU Tezro, where PROM 6.210 writes the +/// serial DMA control 0x100201ff and the TX producer 0x3f8. +#[test] +fn the_processor_reads_and_writes_ioc4_registers_in_its_own_byte_order() { + let board = TezroBoard::new(1).unwrap(); + let io = &board.node.io; + // IOC4's BAR at PCI memory 0x200000, decoding memory, reached through + // head 0's first short window. + assert_eq!(io.pio_write(config_id(1) + 0x10, 4, 0x20_0000), BUS_OK); + assert_eq!(io.pio_write(config_id(1) + 4, 4, 2), BUS_OK); + assert_eq!(io.pio_write(widget(15, 0x200), 8, (1 << 12) | 2), BUS_OK); + let reg = |offset: u64| widget(15, 0x20_0000 + offset); + assert_eq!(io.pio_read(reg(0x24), 4).data, 1, "MISC: the 66 MHz clock"); + assert_eq!(io.pio_write(reg(0x314), 4, 0x3f8), BUS_OK); + assert_eq!(io.pio_read(reg(0x314), 4).data, 0x3f8, "port 0 TX producer"); +} + +// VPro host-interface registers the tests below touch, as byte offsets in +// the board's widget space. The board's own constants are private to it. +const ODSY_IDENTITY: u64 = 0x00; +const ODSY_CMD_FIFO: u64 = 0x11_0000; + +fn board_with_v12(cpus: usize) -> TezroBoard { + TezroBoard::with_config(TezroBoardConfig { + payload: &[], + clock: Arc::new(|| 0), + flash_path: None, + station_mac: L1Config::default().mac, + timekeeper: Arc::new(Ds1744::new(YearBase::From(1968)).unwrap()), + memory: &MemoryPopulation::default(), + cpus, + part: TezroPart::Mhz700, + v12: true, + }) + .unwrap() +} + +/// The V12 answers as widget 12 behind the PXBOW on either board, and +/// widget 12 stays empty without one. (On the one-processor board the +/// placement is inferred; see V12_WIDGET.) +#[test] +fn the_v12_answers_at_widget_12_on_both_boards() { + for cpus in [1, 4] { + let board = board_with_v12(cpus); + let id = board.node.io.pio_read(widget(12, ODSY_IDENTITY), 4); + let direct = board.v12.as_ref().unwrap().read32(ODSY_IDENTITY); + assert_eq!((id.status, id.data), (BUS_OK, u64::from(direct.data)), "{cpus} CPUs"); + assert_eq!(board.pics[board.pics.len() - 1].pxbow().unwrap().endpoint_widgets() & (1 << 12), 1 << 12); + } + assert_eq!(TezroBoard::new(1).unwrap().node.io.pio_read(widget(12, ODSY_IDENTITY), 4).status, BUS_ERR); +} + +/// A CPU 0 store into its enabled graphics page is counted against PI0's +/// slot A credits, and the credit comes back along the node's graphics map +/// as the V12 takes the store. A store outside the page costs nothing. +/// Follows sn1/pi-gfx/ip53-odyssey-credit-return and +/// uncovered-stores-not-counted in tests/qtest/sn1-pi-gfx-test.c. +#[test] +fn a_counted_store_to_the_v12_returns_its_credit() { + use crate::dev::bedrock_io::reg::GFX_MAP_BASE; + use crate::dev::odyssey::{PI_GFX_PAGE_ADDRESS, PI_GFX_PAGE_ENABLE}; + let board = board_with_v12(4); + let (pi, io, bus) = (&board.node.pi[0], &board.node.io, &board.node.bus); + let credit = || pi.read64(0x308).data; + // The page holding the V12's command FIFO, enabled, with 1000 credits. + let page = widget(12, ODSY_CMD_FIFO) & PI_GFX_PAGE_ADDRESS; + assert_eq!(pi.write64(0x300, PI_GFX_PAGE_ENABLE | page), BUS_OK); + assert_eq!(pi.write64(0x308, 1000), BUS_OK); + // Widget 12's credits go to node 0, PI0, slot A. + assert_eq!(io.write64(GFX_MAP_BASE, 12), BUS_OK); + let io_view = 1u64 << 57; + assert_eq!(bus.write64(io_view | widget(12, ODSY_CMD_FIFO), 0x0001_0046), BUS_OK); + assert_eq!(credit(), 1000, "one store, one credit, back"); + // Outside the page: neither counted nor returned. + let _ = bus.write64(io_view | widget(11, 0x100), 0); + assert_eq!(credit(), 1000); +} + +/// A counted store the bus answers busy is made again by the processor and +/// counted again, so the refused attempt must cost nothing: the node puts +/// the credit back. A refund also undoes a count across the wrap. +#[test] +fn a_counted_store_refused_busy_is_refunded() { + use crate::dev::odyssey::{PI_GFX_PAGE_ADDRESS, PI_GFX_PAGE_ENABLE}; + struct Busy; + impl BusDevice for Busy { + fn write64(&self, _addr: u64, _val: u64) -> u32 { + crate::traits::BUS_BUSY + } + } + let board = board_with_v12(1); + let (pi, bus) = (&board.node.pi[0], &board.node.bus); + board.node.io.connect_widget(9, Arc::new(Busy)); + let page = widget(9, 0) & PI_GFX_PAGE_ADDRESS; + assert_eq!(pi.write64(0x300, PI_GFX_PAGE_ENABLE | page), BUS_OK); + assert_eq!(pi.write64(0x308, 5), BUS_OK); + let io_view = 1u64 << 57; + assert_eq!(bus.write64(io_view | widget(9, 0), 0), crate::traits::BUS_BUSY); + assert_eq!(pi.read64(0x308).data, 5, "refunded"); + assert_eq!(pi.write64(0x308, 0), BUS_OK); + assert_eq!(bus.write64(io_view | widget(9, 0), 0), crate::traits::BUS_BUSY); + assert_eq!(pi.read64(0x308).data, 0, "back across the wrap"); +} + +/// Follows sn1/pi-gfx/map-rewrite-settles-owed-credits: owed credits are +/// delivered when the entry naming the widget is rewritten, a write of the +/// value already there settles nothing, a repayment of settled credits is +/// dropped, one still owed is routed, and a reset's abandonment cancels +/// what it threw away. +#[test] +fn a_map_rewrite_settles_owed_credits() { + let mut l = OwedCredits::default(); + let entry = 0xc; + l.owe(0xc, 5); + assert_eq!(l.settle(entry, entry), None, "same value"); + assert_eq!(l.settle(entry, 0), Some((0xc, 5))); + assert_eq!(l.repay(0xc, 5), 0, "already settled"); + assert_eq!(l.settle(0, entry), None, "widget 0 owes nothing"); + l.owe(0xc, 3); + assert_eq!(l.repay(0xc, 2), 2); + l.owe(0xc, -1); + assert_eq!(l.settle(entry, 0), None, "nothing left owed"); + l.owe(0xb, -1); + assert_eq!(l, OwedCredits::default(), "never below zero"); +} + +/// On the board, credits the V12 owes reach PI0's slot A when the map +/// entry naming widget 12 is rewritten, under the entry as it stood, and +/// the V12's later repayment of them is dropped. +#[test] +fn the_board_settles_the_v12s_owed_credits_on_a_map_write() { + use crate::dev::bedrock_io::reg::GFX_MAP_BASE; + let board = board_with_v12(4); + let (pi, io) = (&board.node.pi[0], &board.node.io); + let link = board.v12.as_ref().unwrap().pi_link().unwrap(); + assert_eq!(pi.write64(0x308, 100), BUS_OK); + assert_eq!(io.write64(GFX_MAP_BASE, 12), BUS_OK); + link.owe_credits(12, 8); + assert_eq!(pi.read64(0x308).data, 100, "owed, not yet returned"); + assert_eq!(io.write64(GFX_MAP_BASE, 0), BUS_OK, "the detach"); + assert_eq!(pi.read64(0x308).data, 108); + assert_eq!(io.write64(GFX_MAP_BASE, 12), BUS_OK); + link.repay_credits(12, 8); + assert_eq!(pi.read64(0x308).data, 108, "settled credits are not paid twice"); + link.owe_credits(12, 2); + link.repay_credits(12, 2); + assert_eq!(pi.read64(0x308).data, 110); +} + +fn stalled_gfx(credits: u64, bias: u64) -> (BedrockGfx, Arc) { + use crate::dev::odyssey::{PI_GFX_BIAS, PI_GFX_CREDIT, PI_GFX_PAGE, PI_GFX_PAGE_ENABLE}; + let now = Arc::new(std::sync::atomic::AtomicU64::new(0)); + let clock = now.clone(); + let mut g = BedrockGfx::new(Arc::new(move || clock.load(Ordering::SeqCst))); + g.write(PI_GFX_PAGE, PI_GFX_PAGE_ENABLE | GFX_PAGE); + g.write(PI_GFX_CREDIT, credits); + g.write(PI_GFX_BIAS, bias); + (g, now) +} + +const GFX_PAGE: u64 = 0x0140_0000; + +/// Follows sn1/pi-gfx/stall-below-bias: below the bias the slot stops and +/// its next counted store waits, uncounted, until credits lift the counter +/// above the bias or the processor takes the interrupt; each counted store +/// below the bias asks for the interrupt again. Stores outside the page +/// and the other slot go on. +#[test] +fn a_slot_below_its_bias_holds_its_next_store() { + use crate::dev::odyssey::{PI_GFX_BIAS, PI_GFX_CREDIT, PI_GFX_INT_COUNT}; + let (mut g, _) = stalled_gfx(6, 5); + assert_eq!(g.store(0, GFX_PAGE), Store::Counted { below_bias: false }, "at the bias"); + assert_eq!(g.store(0, GFX_PAGE), Store::Counted { below_bias: true }); + assert_eq!(g.store(0, GFX_PAGE), Store::Held); + assert_eq!(g.read(PI_GFX_CREDIT), 4, "a held store is not counted"); + assert_eq!(g.store(0, 0x0142_0158), Store::Uncounted); + assert_eq!(g.store(1, GFX_PAGE), Store::Uncounted); + g.add_credits(0, 1); + assert_eq!(g.store(0, GFX_PAGE), Store::Held, "5 is not above the bias"); + g.add_credits(0, 1); + assert_eq!(g.store(0, GFX_PAGE), Store::Counted { below_bias: false }); + assert_eq!(g.store(0, GFX_PAGE), Store::Counted { below_bias: true }); + g.taken(0); + assert_eq!(g.store(0, GFX_PAGE), Store::Counted { below_bias: true }, "taken: goes at once"); + g.write(PI_GFX_INT_COUNT, 0); + assert_eq!(g.store(0, GFX_PAGE), Store::Counted { below_bias: true }, "the handler's counter write"); + g.write(PI_GFX_BIAS, 0); + assert_eq!(g.store(0, GFX_PAGE), Store::Counted { below_bias: false }, "a lower bias"); +} + +/// The hold is bounded, as QEMU's is, so a slot nobody releases cannot +/// stop its processor for good. +#[test] +fn a_held_store_goes_after_the_limit() { + use crate::dev::odyssey::PI_GFX_CREDIT; + let (mut g, now) = stalled_gfx(3, 5); + assert_eq!(g.store(0, GFX_PAGE), Store::Counted { below_bias: true }); + now.store(10, Ordering::SeqCst); + assert_eq!(g.store(0, GFX_PAGE), Store::Held); + now.store(10 + STALL_LIMIT_NS - 1, Ordering::SeqCst); + assert_eq!(g.store(0, GFX_PAGE), Store::Held); + now.store(10 + STALL_LIMIT_NS, Ordering::SeqCst); + assert_eq!(g.store(0, GFX_PAGE), Store::Counted { below_bias: true }); + assert_eq!(g.read(PI_GFX_CREDIT), 1); + now.store(20 + STALL_LIMIT_NS, Ordering::SeqCst); + assert_eq!(g.store(0, GFX_PAGE), Store::Held, "and stops again"); +} + +/// On the board a held store is answered busy without reaching the V12, +/// and the PI's pending-modify clear of the slot's graphics bit, which the +/// processor's interrupt handler makes, lets the next one go. +#[test] +fn the_board_holds_pi0s_store_below_the_bias_until_the_interrupt_is_taken() { + use crate::dev::odyssey::{PI_GFX_PAGE_ADDRESS, PI_GFX_PAGE_ENABLE}; + let board = board_with_v12(4); + let (pi, bus) = (&board.node.pi[0], &board.node.bus); + let page = widget(12, ODSY_CMD_FIFO) & PI_GFX_PAGE_ADDRESS; + assert_eq!(pi.write64(0x300, PI_GFX_PAGE_ENABLE | page), BUS_OK); + assert_eq!(pi.write64(0x308, 1000), BUS_OK); + assert_eq!(pi.write64(0x310, 1000), BUS_OK, "slot A's bias"); + let io_view = 1u64 << 57; + let store = || bus.write64(io_view | widget(12, ODSY_CMD_FIFO), 0x0001_0046); + assert_eq!(store(), BUS_OK, "999 is below the bias: the slot stops"); + assert_eq!(store(), crate::traits::BUS_BUSY); + assert_eq!(pi.read64(0x308).data, 999, "held, not counted; no map entry returns credits"); + assert_eq!(pi.write64(0x90, 1), BUS_OK, "pending-modify: clear bit 1"); + assert_eq!(store(), BUS_OK); +} + +/// A hub register as the processor reaches it through the two register +/// windows of uncached space 1: the local form at 0x0100_0000 and the +/// remote form at 0x0180_0000, both onto this node's registers. +fn hub_registers(board: &TezroBoard, window: u64, offset: u64) -> crate::traits::BusRead64 { + board.node.bus.read64((1 << 57) | window | offset) +} +const HUB_WINDOWS: [u64; 2] = [0x0100_0000, 0x0180_0000]; + +/// The PROM writes the hub's widget control (0x400020) with destination +/// credits 4 in bits 15:13 for the integrated crossbar, and keeps the +/// widget number in bits 3:0. +#[test] +fn the_hub_keeps_the_proms_destination_credits_of_four() { + for window in HUB_WINDOWS { + let board = TezroBoard::new(1).unwrap(); + let control = hub_registers(&board, window, 0x40_0020).data; + assert_eq!(control & 0xf, 10, "the hub's own widget number"); + let value = (control & !(7 << 13)) | (4 << 13); + assert_eq!(board.node.bus.write64((1 << 57) | window | 0x40_0020, value), BUS_OK); + let back = hub_registers(&board, window, 0x40_0020).data; + assert_eq!((back >> 13) & 7, 4, "window {window:#x}"); + assert_eq!(back & 0xf, 10); + } +} + +/// A one-node boot reads the network interface as an uncabled port: status +/// 1, reset enable 0, error status 0 and the error-clear register 0 however +/// often it is read, and neither network error bit (39, 40) is up in the +/// processor interface's second pending register. Through both windows. +#[test] +fn a_one_node_boot_reads_the_network_interface_quiet() { + for cpus in [1, 4] { + let board = TezroBoard::new(cpus).unwrap(); + for window in HUB_WINDOWS { + for (offset, expected) in [(0x68_0000, 1), (0x68_0010, 0), (0x68_0100, 0), (0x68_0120, 0)] { + for count in 0..4 { + let r = hub_registers(&board, window, offset); + assert_eq!((r.status, r.data), (BUS_OK, expected), "{cpus} CPUs, {window:#x} + {offset:#x}, read {count}"); + } + } + } + for (interface, pi) in board.node.pi.iter().enumerate() { + let pending = pi.read64(crate::dev::bedrock_pi::reg::PENDING_STATUS_B).data; + assert_eq!(pending & (1 << 39 | 1 << 40), 0, "{cpus} CPUs, PI{interface}"); + } + } +} + +/// PROM 6.210's crossbar check on the Tezro: it writes the read-only lower +/// error address, wants the register-access error with the hub's widget in +/// the status source, then enables that condition, writes the upper error +/// address, and wants the error again and vector 0x10 at the PI. The stores +/// complete, and the arbitration lock of link 15 (bit 1) holds its value. +#[test] +fn the_proms_crossbar_check_passes_on_the_tezro_board() { + let board = TezroBoard::new(1).unwrap(); + let io = &board.node.io; + let status_clear = widget(0, 0x54); + let (err_upper, err_lower, control) = (widget(0, 0x14), widget(0, 0x1c), widget(0, 0x24)); + let pending_0 = || board.node.pi[0].read64(crate::dev::bedrock_pi::reg::PENDING_STATUS_A).data; + // Vector 0x10 to the hub, at PI 0x90 of its register space (address + // bit 47 selects the registers). + assert_eq!(io.pio_write(widget(0, 0x34), 4, (0x10 << 24) | (10 << 16) | 0x8000), BUS_OK); + assert_eq!(io.pio_write(widget(0, 0x3c), 4, 0x0180_0090), BUS_OK); + + assert_eq!(io.pio_write(err_lower, 4, 0xdead_beef), BUS_OK, "no bus error"); + let status = read_word(&board, status_clear); + assert_ne!(status & (1 << 5), 0, "register-access error"); + assert_eq!((status >> 6) & 0xf, 10, "the hub's widget is the source"); + assert_eq!(read_word(&board, status_clear) & (1 << 5), 0, "the alias cleared it"); + assert_eq!(pending_0() & (1 << 16), 0, "nothing was enabled yet"); + + assert_eq!(io.pio_write(control, 4, 0x20), BUS_OK); + assert_eq!(io.pio_write(err_upper, 4, 0), BUS_OK, "no bus error"); + let status = read_word(&board, widget(0, 0x0c)); + assert_ne!(status & (1 << 5), 0, "register-access error again"); + assert_eq!((status >> 6) & 0xf, 10); + assert_ne!(pending_0() & (1 << 16), 0, "vector 0x10 reached the PI"); + // Acknowledging the error lowers the vector again. + assert_ne!(read_word(&board, status_clear) & (1 << 5), 0); + assert_eq!(pending_0() & (1 << 16), 0); + + // The PROM's lock: bit 1 of link 15's arbitration-upper register. + let lock = link_word(15, LINK_ARB_UPPER); + assert_eq!(io.pio_write(lock, 4, 2), BUS_OK); + assert_eq!(read_word(&board, lock) & 2, 2); +} + +/// A jitv2 build refuses memory that reaches past 4 GiB, as the default +/// layout does, with a message naming a layout it takes, and builds the +/// processor for four 1024 MiB banks, which end at 4 GiB. +#[cfg(feature = "jitv2")] +#[test] +fn a_jitv2_build_takes_memory_only_below_4_gib() { + let refused = check_jitv2_memory(&MemoryPopulation::default()).unwrap_err(); + assert!(refused.contains("banks = [1024, 1024, 1024, 1024]"), "{refused}"); + let four = MemoryPopulation::from_mib(&[1024; 4]).unwrap(); + check_jitv2_memory(&four).unwrap(); + // Building a processor takes more stack than a debug build's test thread has. + std::thread::Builder::new().stack_size(64 << 20).spawn(move || { + let board = TezroBoard::with_config(TezroBoardConfig { + payload: &[], + clock: Arc::new(|| 0), + flash_path: None, + station_mac: L1Config::default().mac, + timekeeper: Arc::new(Ds1744::new(YearBase::From(1968)).unwrap()), + memory: &four, + cpus: 1, + part: TezroPart::Mhz700, + v12: false, + }) + .unwrap(); + drop(build_cpu::(&board)); + }).unwrap().join().unwrap(); +} + +/// A guest power-off exits the process without dropping the board, so the +/// flush is what carries an overlay's writes to the next run, on either +/// ISP12160 channel. A second board reads the disk while the first is still +/// alive, so nothing the first board's drop does can be what it sees; both +/// are then dropped, so no board is leaked. +#[test] +fn flushed_overlay_writes_survive_a_power_off_that_skips_drop() { + use crate::scsi::{scsi_cmd, ScsiDataLength, ScsiRequest}; + for channel in 0..2 { + let path = std::env::temp_dir().join(format!("iris-tezro-flush-{}-{channel}", std::process::id())); + std::fs::write(&path, vec![0u8; 4 * 512]).unwrap(); + let attach = |board: &TezroBoard| { + let bus = board.scsi.bus_for_channel(channel).unwrap(); + bus.attach_device(1, path.to_str().unwrap(), false, Vec::new(), true, None, true).unwrap(); + }; + let board = TezroBoard::new(1).unwrap(); + attach(&board); + let write = ScsiRequest { + cdb: vec![scsi_cmd::WRITE_10, 0, 0, 0, 0, 2, 0, 0, 1, 0], + data_len: ScsiDataLength::Unlimited, + data_in: Some(vec![0x5A; 512]), + }; + board.scsi.bus_for_channel(channel).unwrap().request(1, &write).unwrap().unwrap(); + board.flush_disks().unwrap(); + let writer = board; + + let board = TezroBoard::new(1).unwrap(); + attach(&board); + let read = ScsiRequest { + cdb: vec![scsi_cmd::READ_10, 0, 0, 0, 0, 2, 0, 0, 1, 0], + data_len: ScsiDataLength::Unlimited, + data_in: None, + }; + let got = board.scsi.bus_for_channel(channel).unwrap().request(1, &read).unwrap().unwrap(); + assert_eq!(got.data, vec![0x5A; 512], "channel {channel}"); + drop(board); + drop(writer); + let _ = std::fs::remove_file(format!("{}.overlay", path.display())); + let _ = std::fs::remove_file(format!("{}.overlay.dirty", path.display())); + let _ = std::fs::remove_file(&path); + } +} + +/// The cached view the processors' caches sit on swaps RAM above 4 GiB, so +/// SC stays atomic where the direct window does not reach. +#[test] +fn the_cached_view_swaps_ram_above_4_gib() { + let board = TezroBoard::new(4).unwrap(); + // Banks 0 to 5: two 1 GiB banks and four of 512 MiB, as MD's tests set them. + assert_eq!(board.node.md.md_bus().write64(8, 0x53 | (0x43 << 8) | (0x43 << 16)), BUS_OK); + let cached = board.cached_bus(); + let at = (5u64 << 30) + 0x80; + assert_eq!(cached.write64(at, 1), BUS_OK); + assert_eq!(cached.cas64(at, 1, 2), Some(true)); + assert_eq!(cached.cas64(at, 1, 3), Some(false)); + assert_eq!(cached.read64(at).data, 2); +} + +/// Only the two-processor board's fresh flash marks its empty B and D +/// positions disabled. +#[test] +fn only_the_two_processor_flash_starts_with_b_and_d_disabled() { + for (cpus, disabled) in [(1, false), (2, true), (4, false)] { + let bytes = TezroBoard::new(cpus).unwrap().node.flash.contents(); + assert_eq!(&bytes[0x9e0101..0x9e0109] == b"DisableB", disabled, "{cpus} CPUs"); + assert_eq!(&bytes[0x9e0141..0x9e0149] == b"DisableD", disabled, "{cpus} CPUs"); + } +} diff --git a/src/lib.rs b/src/lib.rs index c4b68b397..0009811a1 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -181,6 +181,10 @@ pub mod ip32_machine; /// The SGI Octane (IP30): its bus and machine -- see `docs/Octane/ip30-address-map.md`. pub mod ip30; pub mod ip30_machine; +/// Fuel has an IP34 board and uses the IP35 software platform and processor views. +pub mod ip35; +pub mod ip34_machine; +pub mod ip53_machine; pub mod machine_state; pub mod platform; pub mod physical; diff --git a/src/machine.rs b/src/machine.rs index 2bc5ac217..bb98104e5 100644 --- a/src/machine.rs +++ b/src/machine.rs @@ -27,7 +27,10 @@ use crate::cpu::mips_tlb::MipsTlb; use crate::cpu::mips_exec::{MipsExecutor, MipsCpu, MipsCpuConfig, MipsCpuDebugAdapter}; use crate::gdb_stub::CpuDebug; use crate::cpu::mips_cache_v2::{MipsCache, R4400Cache, R5000Cache}; -use crate::cpu::mips_cache_shadow::{R10000ShadowCache, R4400ShadowCache, R5000ShadowCache}; +use crate::cpu::mips_cache_shadow::{ + CONFIG0_RESET_MASK, R10000ShadowCache, R12000ShadowCache, R14000_PRID, R16000ShadowCache, R16000_MHZ700_CONFIG0, + R16000_MHZ700_PRID, R4400ShadowCache, R5000ShadowCache, +}; use crate::dev::hpc3::Hpc3; use crate::dev::ioc::{Ioc, GioSlot, GIO_SLOT_MAP, profile_idx}; use crate::monitor::Monitor; @@ -50,7 +53,7 @@ pub fn set_emulator_profile(profile: MachineProfile) { fn emulator_profile() -> Option { let raw = EMULATOR_PROFILE.load(std::sync::atomic::Ordering::Relaxed); - [MachineProfile::IndyIp24, MachineProfile::Indigo2Ip22, MachineProfile::Indigo2Ip28, MachineProfile::O2Ip32, MachineProfile::OctaneIp30] + [MachineProfile::IndyIp24, MachineProfile::Indigo2Ip22, MachineProfile::Indigo2Ip28, MachineProfile::O2Ip32, MachineProfile::OctaneIp30, MachineProfile::FuelIp34, MachineProfile::TezroIp53] .into_iter().find(|p| *p as u8 == raw) } @@ -65,6 +68,8 @@ fn emulator_name_thirds(profile: Option) -> &'static [&'static s Some(MachineProfile::Indigo2Ip28) => &["IRIX", "Iris", "IPsomething", "Indigo2", "IP28"], Some(MachineProfile::O2Ip32) => &["IRIX", "Iris", "IPsomething", "IP32"], Some(MachineProfile::OctaneIp30) => &["IRIX", "Iris", "IPsomething", "IP30"], + Some(MachineProfile::FuelIp34) => &["IRIX", "Iris", "IPsomething", "IP34"], + Some(MachineProfile::TezroIp53) => &["IRIX", "Iris", "IP53"], None => &["IRIX", "Iris", "IPsomething"], } } @@ -354,6 +359,11 @@ impl Machine { crate::config::CpuModel::R4400 => ::L2_SIZE > 0, crate::config::CpuModel::R5000 => ::L2_SIZE > 0, crate::config::CpuModel::R10000 => ::L2_SIZE > 0, + crate::config::CpuModel::R12000 => ::L2_SIZE > 0, + crate::config::CpuModel::R14000 => ::L2_SIZE > 0, + crate::config::CpuModel::R16000 | crate::config::CpuModel::R16000Mhz700 => { + ::L2_SIZE > 0 + } }; if !model_has_l2 { eeprom_mc.lock().set_cachsz(0); @@ -741,12 +751,22 @@ impl Machine { // arm below monomorphises its own CPU — no per-model branch on the hot path. let sysad: Arc = phys.clone(); let memory = Arc::new(crate::ip22_bus::memory_map(cfg.machine.profile.ip28())); - macro_rules! build_cpu { ($cache:ty) => {{ + macro_rules! build_cpu { ($cache:ty $(, $prid:expr $(, $config0:expr)?)?) => {{ let cfg = MipsCpuConfig::for_model::<$cache>(memory.clone()); let tlb = MipsTlb::new(cfg.tlb_entries); let mut executor: MipsExecutor = MipsExecutor::new(sysad.clone(), tlb, &cfg); // The MC decodes 32 physical address bits; see `MipsExecutor::phys_mask`. executor.phys_mask = u32::MAX as u64; + // A part that shares another's model differs in PRId and, where its + // Config reset word differs, in that word's low 25 bits. + $( + executor.core.reset_prid = $prid; + executor.core.cp0_prid = $prid; + $( + executor.core.cp0_config = + (executor.core.cp0_config & !CONFIG0_RESET_MASK) | $config0; + )? + )? // Load default symbol maps if they exist { @@ -800,6 +820,12 @@ impl Machine { // tag and data arrays that exist only to answer CACHE ops and the // PROM's diagnostics. See mips_cache_shadow.rs. crate::config::CpuModel::R10000 => build_cpu!(R10000ShadowCache), + crate::config::CpuModel::R12000 => build_cpu!(R12000ShadowCache), + crate::config::CpuModel::R14000 => build_cpu!(R12000ShadowCache, R14000_PRID), + crate::config::CpuModel::R16000 => build_cpu!(R16000ShadowCache), + crate::config::CpuModel::R16000Mhz700 => { + build_cpu!(R16000ShadowCache, R16000_MHZ700_PRID, R16000_MHZ700_CONFIG0) + } }; // The developer status bar shows the CPU's Count rate. @@ -2177,11 +2203,13 @@ mod emulator_name_tests { Some(MachineProfile::Indigo2Ip28) => &["Indy", "IP24", "IP22", "IP32", "O2"], Some(MachineProfile::O2Ip32) => &["Indy", "Indigo2", "IP24", "IP22", "IP28", "IP30", "O2"], Some(MachineProfile::OctaneIp30) => &["Indy", "Indigo2", "IP24", "IP22", "IP28", "IP32", "O2"], + Some(MachineProfile::FuelIp34) => &["Indy", "Indigo2", "IP24", "IP22", "IP28", "IP30", "IP32", "O2"], + Some(MachineProfile::TezroIp53) => &["Indy", "Indigo2", "IP24", "IP22", "IP28", "IP30", "IP32", "IP34", "O2"], None => &["Indy", "Indigo2", "IP24", "IP22", "IP28", "IP30", "IP32", "O2"], } }; for p in [None, Some(MachineProfile::IndyIp24), Some(MachineProfile::Indigo2Ip22), - Some(MachineProfile::Indigo2Ip28), Some(MachineProfile::O2Ip32), Some(MachineProfile::OctaneIp30)] { + Some(MachineProfile::Indigo2Ip28), Some(MachineProfile::O2Ip32), Some(MachineProfile::OctaneIp30), Some(MachineProfile::FuelIp34), Some(MachineProfile::TezroIp53)] { let thirds = emulator_name_thirds(p); assert!(!thirds.is_empty()); for w in thirds { @@ -2193,7 +2221,7 @@ mod emulator_name_tests { #[test] fn profile_round_trips_through_the_global() { - for p in [MachineProfile::IndyIp24, MachineProfile::Indigo2Ip22, MachineProfile::Indigo2Ip28, MachineProfile::O2Ip32, MachineProfile::OctaneIp30] { + for p in [MachineProfile::IndyIp24, MachineProfile::Indigo2Ip22, MachineProfile::Indigo2Ip28, MachineProfile::O2Ip32, MachineProfile::OctaneIp30, MachineProfile::FuelIp34, MachineProfile::TezroIp53] { set_emulator_profile(p); assert_eq!(emulator_profile(), Some(p)); } diff --git a/src/pci.rs b/src/pci.rs index f81627099..f0841b1c6 100644 --- a/src/pci.rs +++ b/src/pci.rs @@ -46,6 +46,69 @@ pub enum Space { Mem, } +/// Initial value and writable bits for a function-specific configuration dword. +#[derive(Clone, Copy, Debug)] +pub struct PciRegisterPolicy { + /// Dword index. Header, BAR, subsystem identity, and interrupt dwords are managed separately. + pub reg_dword: usize, + /// Value restored by [`PciConfig::reset`]. + pub initial: u32, + /// Bits that configuration writes may change. + pub writable_mask: u32, +} + +/// Function-specific write behavior for a type 0 PCI configuration header. +/// +/// Register policies are copied into the header during construction; the +/// borrowed slice is not retained. +#[derive(Clone, Copy, Debug)] +pub struct PciConfigPolicy<'a> { + /// Fixed Status bits that read as one. + pub status: u16, + /// Device-latched Status bits that writes acknowledge with one. + pub status_w1c_mask: u16, + /// Writable Command bits. + pub command_mask: u16, + /// Writable bits in the latency timer byte. + pub latency_mask: u8, + /// Writable bits in the cache-line-size byte. + pub cache_line_mask: u8, + /// Initial values and write masks for function-specific dwords. + pub registers: &'a [PciRegisterPolicy], +} + +impl<'a> Default for PciConfigPolicy<'a> { + fn default() -> Self { + Self { + status: 0x0200, + status_w1c_mask: 0, + command_mask: 0xffff, + latency_mask: 0xff, + cache_line_mask: 0xff, + registers: &[], + } + } +} + +/// A function that computes some of its own configuration registers: the +/// indirect register and memory windows of a NIC, or a status bit the chip +/// drives. The header reaches it from every configuration cycle, whichever +/// bridge generated it. +/// +/// Calls arrive with no header lock held, so a hook may use the header. +pub trait PciConfigHook: Send + Sync { + /// The value dword `reg` reads as, when the function computes it. + fn read(&self, reg: usize) -> Option; + + /// A write of `val` under `mask` to dword `reg`. `true` means the + /// function took it and the header stores nothing. Only a full-dword + /// write is offered: narrower ones take the header's own path. + fn write(&self, reg: usize, val: u32, mask: u32) -> bool; + + /// Dword `reg` was written through the header's own path. + fn written(&self, _reg: usize) {} +} + /// One function's type 0 configuration header. /// /// Enough of it to be discovered and configured: identity, class, command @@ -61,6 +124,13 @@ pub struct PciConfig { /// Dwords `16..caps_end` hold a capability list, which firmware cannot /// overwrite. 0 when the function has none. caps_end: usize, + status: u16, + status_w1c_mask: u16, + command_mask: u16, + latency_mask: u8, + cache_line_mask: u8, + writable_masks: [u32; 64], + hook: std::sync::OnceLock>, } impl PciConfig { @@ -68,6 +138,27 @@ impl PciConfig { /// three bytes of register 0x08. `pin` is 0 for none, or [`PCI_INTA`] /// and on; the line register is left for firmware to assign. pub fn new(vendor: u16, device: u16, class: u32, revision: u8, bars: [Bar; 6], pin: u8) -> Self { + Self::new_with_policy( + vendor, + device, + class, + revision, + bars, + pin, + PciConfigPolicy::default(), + ) + } + + /// Construct a type 0 header with function-specific register behavior. + pub fn new_with_policy( + vendor: u16, + device: u16, + class: u32, + revision: u8, + bars: [Bar; 6], + pin: u8, + policy: PciConfigPolicy<'_>, + ) -> Self { let mut r = [0u32; 64]; r[0] = ((device as u32) << 16) | vendor as u32; r[2] = (class << 8) | revision as u32; @@ -77,7 +168,27 @@ impl PciConfig { r[4 + i] = 1; } } - Self { regs: Mutex::new(r), bars, power_on: r, caps_end: 0 } + let mut writable_masks = [u32::MAX; 64]; + for register in policy.registers { + assert!( + matches!(register.reg_dword, 10 | 12..=14 | 16..=63), + "pci: register policy requires a function-specific dword in 0..64"); + r[register.reg_dword] = register.initial; + writable_masks[register.reg_dword] = register.writable_mask; + } + Self { + regs: Mutex::new(r), + bars, + power_on: r, + status: policy.status, + status_w1c_mask: policy.status_w1c_mask, + command_mask: policy.command_mask, + latency_mask: policy.latency_mask, + cache_line_mask: policy.cache_line_mask, + writable_masks, + caps_end: 0, + hook: std::sync::OnceLock::new(), + } } /// Give the function a capability list: `caps` are the raw dwords of the @@ -97,6 +208,17 @@ impl PciConfig { self } + /// Let the function compute some registers. Set once, after construction. + pub fn set_hook(&self, hook: Arc) { + let _ = self.hook.set(hook); + } + + /// Change dword `reg` as the chip does, outside the guest's write mask. + pub fn update(&self, reg: usize, f: impl FnOnce(u32) -> u32) { + let mut r = self.regs.lock().unwrap(); + r[reg] = f(r[reg]); + } + /// RST#: back to the power-up header. The command register clears, so /// the function stops decoding, and the BARs and line forget what /// firmware assigned. @@ -111,26 +233,70 @@ impl PciConfig { /// Register `reg` (a dword index, 0..64). pub fn read(&self, reg: usize) -> u32 { + if let Some(value) = self.hook.get().and_then(|h| h.read(reg)) { + return value; + } + self.read_header(reg) + } + + /// A configuration read of `width` bytes (1, 2 or 4) at byte `offset`, + /// as a bridge's cycle makes it. Only a full dword reaches the function's + /// hook; a narrower read takes the header's own bytes, as a narrower + /// write does. + pub fn read_sized(&self, offset: usize, width: usize) -> u32 { + let dword = if width == 4 { self.read(offset / 4) } else { self.read_header(offset / 4) }; + (dword >> ((offset & 3) * 8)) & (u32::MAX >> (32 - width * 8)) + } + + fn read_header(&self, reg: usize) -> u32 { let r = self.regs.lock().unwrap(); match reg { - // Status: a medium-speed device with no errors to report. // A function with a capability list also sets status bit 4. - 1 => (r[1] & 0xffff) | 0x0200_0000 | if r[13] != 0 { 0x0010_0000 } else { 0 }, + 1 => (r[1] & (0xffff | ((self.status_w1c_mask as u32) << 16))) + | ((self.status as u32) << 16) + | if r[13] != 0 { 0x0010_0000 } else { 0 }, _ => r[reg], } } pub fn write(&self, reg: usize, val: u32) { + self.write_masked(reg, val, u32::MAX); + } + + /// Write selected bits of dword `reg`. Both `val` and `mask` use the + /// register's bit positions. Unselected Status events are not acknowledged. + pub fn write_masked(&self, reg: usize, val: u32, mask: u32) { + if mask == 0 { + return; + } + if let Some(hook) = self.hook.get() { + if mask == u32::MAX && hook.write(reg, val, mask) { + return; + } + self.write_stored(reg, val, mask); + hook.written(reg); + return; + } + self.write_stored(reg, val, mask); + } + + fn write_stored(&self, reg: usize, val: u32, mask: u32) { let mut r = self.regs.lock().unwrap(); match reg { - // Identity and class are read-only. + // Identity, class and header type are read-only. 0 | 2 => {} - // Cache line size and latency timer are stored; the header type - // and BIST bytes above them are not. - 3 => r[3] = (r[3] & 0xffff_0000) | (val & 0xffff), - // Command; status bits are write-one-to-clear and none are set. - 1 => r[1] = val & 0xffff, + 3 => { + let mask = mask & (((self.latency_mask as u32) << 8) | self.cache_line_mask as u32); + r[3] = (r[3] & !mask) | (val & mask); + } + // Command writes replace selected bits; Status events clear on ones. + 1 => { + let clear = val & mask & ((self.status_w1c_mask as u32) << 16); + r[1] = (r[1] & !(mask & 0xffff) & !clear) + | (val & mask & self.command_mask as u32); + } 4..=9 => { + let val = (r[reg] & !mask) | (val & mask); let bar = self.bars[reg - 4]; r[reg] = match bar { Bar::Unused => 0, @@ -144,11 +310,27 @@ impl PciConfig { // The capability list is read-only. 16.. if reg < self.caps_end => {} // Interrupt pin is read-only; the line is firmware's scratch. - 15 => r[15] = (r[15] & 0xff00) | (val & !0xff00), - _ => r[reg] = val, + 15 => { + let mask = mask & !0xff00; + r[15] = (r[15] & !mask) | (val & mask); + } + _ => { + let mask = mask & self.writable_masks[reg]; + r[reg] = (r[reg] & !mask) | (val & mask); + } } } + /// Latch selected device Status events without replacing earlier events. + pub fn set_status_bits(&self, bits: u16) { + self.regs.lock().unwrap()[1] |= ((bits & self.status_w1c_mask) as u32) << 16; + } + + /// Set the board-assigned Subsystem ID, which survives function reset. + pub fn set_subsystem_identity(&self, value: u32) { + self.regs.lock().unwrap()[11] = value; + } + /// The base address firmware programmed into BAR `i`, type bits masked. pub fn bar_base(&self, i: usize) -> u32 { let v = self.regs.lock().unwrap()[4 + i]; @@ -264,6 +446,11 @@ pub trait PciBusDevice: Send + Sync { /// The bridge gives a function that masters the bus its way to memory. fn connect_dma(&self, _dma: std::sync::Arc) {} + + /// The bridge's REQ64# as reset ends: whether the function's slot drives + /// it. A 64-bit agent samples it to learn its bus width; a 32-bit one + /// ignores it. + fn req64_sampled(&self, _asserted: bool) {} } /// How a bus master's access ended. @@ -436,6 +623,14 @@ mod tests { [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], PCI_INTA) } + fn scsi_like_with_policy(policy: PciConfigPolicy<'_>) -> PciConfig { + PciConfig::new_with_policy( + 0x9004, 0x8078, 0x0100_00, 1, + [Bar::Io(0x100), Bar::Mem(0x1000), Bar::Unused, Bar::Unused, Bar::Unused, Bar::Unused], PCI_INTA, + policy, + ) + } + /// Firmware learns how much space a device wants by writing all ones to /// a BAR and reading back which address bits stuck. #[test] @@ -502,7 +697,7 @@ mod tests { #[test] fn reset_keeps_the_subsystem_identity() { let c = scsi_like(); - c.write(11, 0xc304_10a9); + c.set_subsystem_identity(0xc304_10a9); c.write(4, 0x0000_1000); c.reset(); assert_eq!(c.read(11), 0xc304_10a9); @@ -529,6 +724,121 @@ mod tests { assert_eq!(c.interrupt_pin(), PCI_INTA); } + #[test] + fn header_policies_mask_writes_and_restore_reset_values() { + for (status, command, latency, cache) in [(0x0200, 0xffff, 0, 0), (0x02b0, 0x0156, 0xf0, 0x0f)] { + let c = scsi_like_with_policy(PciConfigPolicy { + status, + status_w1c_mask: 0, + command_mask: command, + latency_mask: latency, + cache_line_mask: cache, + registers: &[PciRegisterPolicy { + reg_dword: 18, + initial: 0xaabb_0020, + writable_mask: 0xff00, + }], + }); + c.write(1, u32::MAX); + c.write(3, u32::MAX); + c.write(18, u32::MAX); + assert_eq!(c.read(1), (u32::from(status) << 16) | u32::from(command)); + assert_eq!(c.read(3), (u32::from(latency) << 8) | u32::from(cache)); + assert_eq!(c.read(18), 0xaabb_ff20); + c.reset(); + assert_eq!(c.read(1), u32::from(status) << 16); + assert_eq!(c.read(3), 0); + assert_eq!(c.read(18), 0xaabb_0020); + } + } + + #[test] + fn masked_writes_preserve_unselected_fields_and_status_events() { + let c = scsi_like_with_policy(PciConfigPolicy { + status: 0x0200, + status_w1c_mask: 0x8040, + command_mask: 0x0157, + latency_mask: 0xff, + cache_line_mask: 0xff, + registers: &[PciRegisterPolicy { + reg_dword: 18, + initial: 0xaabb_0000, + writable_mask: 0xffff, + }], + }); + c.set_status_bits(0x8040); + c.write(1, 0x0157); + c.write(3, 0x1234); + c.write(5, 0x1234_5000); + c.write(4, 0x1201); + for (reg, value, mask, expected) in [ + (1, 0xffff_0002, 0xff, 0x8240_0102), + (1, 0xffff_0000, 0x8000_0000, 0x0240_0102), + (1, 0x0040_0000, 0x00ff_0000, 0x0200_0102), + (1, u32::MAX, 0, 0x0200_0102), + (3, 0xffff_00ab, 0xff, 0x12ab), + (5, 0xabcd_ffff, 0xffff_0000, 0xabcd_5000), + (4, 0x3400, 0xff00, 0x3401), + (15, u32::MAX, 0xff00, 0x0100), + (18, u32::MAX, 0xff, 0xaabb_00ff), + ] { + c.write_masked(reg, value, mask); + assert_eq!(c.read(reg), expected, "register {reg}"); + } + c.set_status_bits(u16::MAX); + c.reset(); + assert_eq!(c.read(1), 0x0200_0000); + let default = scsi_like(); + default.set_status_bits(u16::MAX); + assert_eq!(default.read(1), 0x0200_0000); + } + + #[test] + fn snapshots_restore_registers_with_destination_write_policies() { + use crate::traits::Saveable; + let src = scsi_like_with_policy(PciConfigPolicy { + status: 0x02b0, + status_w1c_mask: 0xc000, + command_mask: 0x0157, + latency_mask: 0xff, + cache_line_mask: 0xff, + registers: &[PciRegisterPolicy { + reg_dword: 18, + initial: 0, + writable_mask: 0x100, + }], + }); + src.set_status_bits(0x8000); + src.write(3, 0x1234); + src.write(18, 0x100); + let saved = src.save_state(); + for (events, command, cache) in [(0, 7, 0x0f), (0xc000, 0x0157, 0xff)] { + let dst = scsi_like_with_policy(PciConfigPolicy { + status: 0x0200, + status_w1c_mask: events, + command_mask: command, + latency_mask: 0, + cache_line_mask: cache, + registers: &[PciRegisterPolicy { + reg_dword: 18, + initial: 0x20, + writable_mask: 0x200, + }], + }); + dst.load_state(&saved).unwrap(); + assert_eq!(dst.save_state(), saved); + assert_eq!(dst.read(1), 0x0200_0000 | ((0x8000 & u32::from(events)) << 16)); + dst.write_masked(1, u32::MAX, u32::MAX); + dst.write(3, u32::MAX); + dst.write(18, u32::MAX); + assert_eq!(dst.read(1), 0x0200_0000 | u32::from(command)); + assert_eq!(dst.read(3), (0x1234 & !u32::from(cache)) | u32::from(cache)); + assert_eq!(dst.read(18), 0x300); + dst.reset(); + assert_eq!(dst.read(18), 0x20); + } + } + struct Nothing; impl BusDevice for Nothing {} diff --git a/src/ppmem/ppmem.rs b/src/ppmem/ppmem.rs index d2e350a07..45217dcfe 100644 --- a/src/ppmem/ppmem.rs +++ b/src/ppmem/ppmem.rs @@ -700,6 +700,9 @@ struct SpaceState { /// The window's size in bytes. Never changes. window: u64, mappings: Vec, + /// Ranges still mapped on the host but off the direct path; see + /// [`PpMemSpace::withdraw_range`]. + withdrawn: Vec, /// Where the bitmap is published. Points at `own_bitmap` until a consumer /// hands over its own inline field, so it is never null and publication is /// an unconditional store. @@ -765,6 +768,7 @@ impl PpMemSpace { state: UnsafeCell::new(SpaceState { window, mappings: Vec::new(), + withdrawn: Vec::new(), sink, sink2, more_sinks: Vec::new(), @@ -801,6 +805,12 @@ impl PpMemSpace { self.banks.len() } + /// How many ranges are mapped now; a remap scrubs each one. + #[cfg(test)] + pub(crate) fn mapping_count(&self) -> usize { + unsafe { &*self.state.get() }.mappings.len() + } + /// Map `size` bytes of `bank`, from object offset `src`, at window /// `offset`, without repeating, with the generation counters in lockstep. /// The O2 maps its memory this way: one 32 MB slot of the window at a @@ -829,6 +839,72 @@ impl PpMemSpace { Ok(()) } + /// Return `at..at + len` of the window, and its counters, to unmapped. + fn scrub_range(&self, at: u64, len: u64) { + // Best-effort: a failure here leaves the range mapped, which is + // still safe — it just isn't reverted. + // `scrub`, not `unmap`: see `AddrSpace::scrub`. + let _ = unsafe { self.space.scrub(at as usize, len as usize) }; + #[cfg(feature = "jitv2")] + { + let gen_off = at / GEN_RATIO; + let gen_len = len / GEN_RATIO; + let gran = super::map::granularity() as u64; + if gen_len >= gran && gen_off % gran == 0 && gen_len % gran == 0 { + let _ = unsafe { self.gen_space.scrub(gen_off as usize, gen_len as usize) }; + } + } + } + + /// Take `offset..offset + size` off the direct path without unmapping + /// it. The bitmap is republished to every sink with the regions this + /// leaves partly covered clear; the host mapping and its counters stay, + /// so a processor that read the old bitmap an instant ago still reaches + /// the bank's bytes rather than a scrubbed page, and needs no other + /// processor stopped. A mapping that only partly overlaps keeps the rest + /// on the path, but a bit is set only for a region still wholly mapped, + /// so withdrawing part of a region takes the whole region off. + /// [`Self::restore_range`] puts it back. CPU thread only, as the rest. + pub fn withdraw_range(&self, offset: u64, size: u64) { + assert!(offset + size <= self.window_size(), "ppmem: [{offset:#x}, +{size:#x}) escapes the window"); + let st = unsafe { &mut *self.state.get() }; + let end = offset + size; + let mut kept = Vec::with_capacity(st.mappings.len() + 1); + for m in std::mem::take(&mut st.mappings) { + if m.at >= end || m.at + m.len <= offset { + kept.push(m); + continue; + } + let (from, to) = (m.at.max(offset), (m.at + m.len).min(end)); + st.withdrawn.push(Mapping { at: from, len: to - from }); + if m.at < offset { + kept.push(Mapping { at: m.at, len: offset - m.at }); + } + if m.at + m.len > end { + kept.push(Mapping { at: end, len: m.at + m.len - end }); + } + } + st.mappings = kept; + Self::recompute_bitmap(st); + } + + /// Put what [`Self::withdraw_range`] took off the direct path in + /// `offset..offset + size` back on it, republishing the bitmap. Returns + /// whether anything was withdrawn there; if not, nothing was mapped + /// and the caller maps the range with [`Self::map_range`]. + pub fn restore_range(&self, offset: u64, size: u64) -> bool { + let st = unsafe { &mut *self.state.get() }; + let end = offset + size; + let (back, still): (Vec<_>, Vec<_>) = std::mem::take(&mut st.withdrawn) + .into_iter() + .partition(|m| m.at >= offset && m.at + m.len <= end); + st.withdrawn = still; + let any = !back.is_empty(); + st.mappings.extend(back); + Self::recompute_bitmap(st); + any + } + /// Recompute the bitmap from the live mapping list. /// /// A bit is set only if the whole region (64MB in a 4GB window) is covered, so a region that @@ -876,6 +952,9 @@ impl PpMemSpace { state .mappings .retain(|m| !(m.at >= at && m.at + m.len <= at + len)); + state + .withdrawn + .retain(|m| !(m.at >= at && m.at + m.len <= at + len)); state.mappings.push(Mapping { at, len }); Self::recompute_bitmap(state); } @@ -896,22 +975,8 @@ impl PpMemSpace { impl MappedMemory for PpMemSpace { fn clear_mappings(&self) { let st = unsafe { &mut *self.state.get() }; - for m in std::mem::take(&mut st.mappings) { - // Best-effort: a failure here leaves the range mapped, which is - // still safe — it just isn't reverted. - // `scrub`, not `unmap`: see `AddrSpace::scrub`. - let _ = unsafe { self.space.scrub(m.at as usize, m.len as usize) }; - #[cfg(feature = "jitv2")] - { - let gen_off = m.at / GEN_RATIO; - let gen_len = m.len / GEN_RATIO; - let gran = super::map::granularity() as u64; - if gen_len >= gran && gen_off % gran == 0 && gen_len % gran == 0 { - let _ = unsafe { - self.gen_space.scrub(gen_off as usize, gen_len as usize) - }; - } - } + for m in std::mem::take(&mut st.mappings).into_iter().chain(std::mem::take(&mut st.withdrawn)) { + self.scrub_range(m.at, m.len); } unsafe { *st.sink = 0; @@ -1491,6 +1556,41 @@ mod tests { assert_eq!(space.mapped_bitmap(), 0, "bitmap not cleared"); } + /// Withdrawing one region clears only its bit in every sink and leaves + /// the host mapping, so the bank's bytes stay reachable; restoring it sets + /// the bit again without mapping anything. Clearing the mappings scrubs + /// withdrawn ranges too. + #[test] + fn withdraw_range_clears_one_region_in_every_sink_and_keeps_the_mapping() { + let (space, banks) = PpMemSpace::with_bank_sizes(&[256]).unwrap(); + let (mut second, mut third) = (0u64, 0u64); + unsafe { + space.set_bitmap_sink(&mut second); + space.set_bitmap_sink2(&mut third); + } + let region = 64 * MB as u64; + for i in 0..3 { + space.map_range(0, i * region, i * region, region).unwrap(); + } + assert_eq!((space.mapped_bitmap(), third), (0b111, 0b111)); + space.withdraw_range(region, region); + assert_eq!((space.mapped_bitmap(), third), (0b101, 0b101)); + assert_eq!(space.mapping_count(), 2); + // The host mapping is still there: a store through the window lands in the bank. + unsafe { *(space.window_base().add(region as usize + 0x100) as *mut u32) = 0x0BAD_C0DE }; + assert_eq!(banks[0].read32(region + 0x100).data, 0x0BAD_C0DE); + assert!(space.restore_range(region, region)); + assert_eq!((space.mapped_bitmap(), third), (0b111, 0b111)); + assert!(!space.restore_range(region, region), "nothing left to restore"); + // Part of a region leaves the whole region off the direct path. + space.withdraw_range(region + 0x1000, 0x1000); + assert_eq!((space.mapped_bitmap(), third), (0b101, 0b101)); + space.clear_mappings(); + assert!(!space.restore_range(0, 3 * region), "a clear scrubs what was withdrawn"); + assert_eq!((space.mapped_bitmap(), third), (0, 0)); + let _ = second; + } + /// The low-512KB alias: bank 0's first 512KB visible at physical 0, the /// same pages as at LOMEM_BASE. Replaces `physical.rs`'s AliasBus. #[test] diff --git a/src/scsi.rs b/src/scsi.rs index bfb2ffda8..eb82e3347 100644 --- a/src/scsi.rs +++ b/src/scsi.rs @@ -162,6 +162,7 @@ pub struct ScsiDevice { kind: DeviceKind, buffer: Vec, // Internal buffer for WRITE_BUFFER/READ_BUFFER commands pending_sense: [u8; 18], // Stored sense data, served on REQUEST_SENSE + pending_attention: Option, /// Physical sector size: always 2048 for CD-ROM (Yellow Book), 512 for HDD. /// Used for TOC lead-out LBA (CD spec requires TOC in physical sectors). phys_block_size: u64, @@ -222,6 +223,7 @@ impl ScsiDevice { kind: if is_cdrom { DeviceKind::Cdrom } else { DeviceKind::Disk }, buffer: vec![0u8; SCSI_BUFFER_SIZE], pending_sense: [0u8; 18], + pending_attention: None, phys_block_size: if is_cdrom { CDROM_SECTOR } else { 512 }, // CD-ROM drives default to 2048-byte logical blocks (Sony CDU-76S behaviour). // HDD defaults to 512. dksc switches CD-ROM to 512 for EFS, back to 2048 for ISO. @@ -242,6 +244,7 @@ impl ScsiDevice { kind: DeviceKind::Cdrom, buffer: vec![0u8; SCSI_BUFFER_SIZE], pending_sense: [0u8; 18], + pending_attention: None, phys_block_size: CDROM_SECTOR, logical_block_size: 2048, identity, @@ -326,6 +329,10 @@ impl ScsiDevice { ScsiResponse { status: 0x02, data: vec![] } } + fn take_attention(&mut self) -> Option { + self.pending_attention.take().or_else(|| self.media.take_changed().then_some(0x28)) + } + fn exec_test_unit_ready(&mut self, _cdb: &[u8]) -> Result { if self.media.take_changed() { // Sense key 0x06 UNIT_ATTENTION, ASC 0x28 "Not Ready to Ready Transition / Medium Changed" @@ -1118,11 +1125,10 @@ impl ScsiTarget for ScsiDevice { } // Unit attention fires before any other command (except REQUEST_SENSE/INQUIRY). - if req.cdb[0] != scsi_cmd::REQUEST_SENSE - && req.cdb[0] != scsi_cmd::INQUIRY - && self.media.take_changed() - { - return Ok(self.check_condition(0x06, 0x28, 0x00)); + if req.cdb[0] != scsi_cmd::REQUEST_SENSE && req.cdb[0] != scsi_cmd::INQUIRY { + if let Some(asc) = self.take_attention() { + return Ok(self.check_condition(0x06, asc, 0x00)); + } } let mut response = match req.cdb[0] { @@ -1268,6 +1274,12 @@ impl ScsiTarget for ScsiDevice { fn block_size(&self) -> Option { Some(self.logical_block_size) } + fn bus_reset(&mut self) { + self.pending_sense = [0; 18]; + self.pending_sense[0] = 0x70; + self.pending_attention = Some(0x29); + } + fn reset_block_size(&mut self, cdrom_512: bool) { if cdrom_512 && self.is_cdrom() { self.logical_block_size = 512; @@ -1867,12 +1879,38 @@ mod tests { cd.changer().unwrap().load(&chd.display().to_string()).unwrap(); assert!(cd.has_media()); assert_eq!(cd.size(), 16 * 2048, "the CHD's size, not the file's"); - // The medium change is reported once, then the disc reads. + // A reset is reported before a media change that was already pending. + crate::scsi::ScsiTarget::bus_reset(&mut cd); + let reset = cd.request(&request(&[0, 0, 0, 0, 0, 0], None)).unwrap(); + assert_eq!(reset.status, 0x02, "unit attention after the reset"); + let reset_sense = cd.request(&request(&[0x03, 0, 0, 0, 18, 0], None)).unwrap(); + assert_eq!(reset_sense.data[12], 0x29, "reset attention comes first"); + let tur = cd.request(&request(&[0, 0, 0, 0, 0, 0], None)).unwrap(); - assert_eq!(tur.status, 0x02, "unit attention after the change"); + assert_eq!(tur.status, 0x02, "the pending media change follows"); + let sense = cd.request(&request(&[0x03, 0, 0, 0, 18, 0], None)).unwrap(); + assert_eq!(sense.data[12], 0x28, "media change keeps its ASC"); let read = cd.request(&request(&[0x28, 0, 0, 0, 0, 3, 0, 0, 1, 0], None)).unwrap(); assert_eq!(read.status, 0x00); assert_eq!(read.data, vec![0x42u8; 2048]); let _ = std::fs::remove_dir_all(dir); } + #[test] + fn bus_reset_reports_unit_attention_once_and_keeps_block_size() { + let mut dev = cdrom(); + dev.logical_block_size = 512; + dev.set_sense(0x05, 0x20, 0x00); + crate::scsi::ScsiTarget::bus_reset(&mut dev); + + let old_sense = run(&mut dev, &[0x03, 0, 0, 0, 18, 0]); + assert_eq!((old_sense.data[2], old_sense.data[12]), (0, 0), "reset clears previous sense"); + let response = run(&mut dev, &[0x00, 0, 0, 0, 0, 0]); + assert_eq!(response.status, 0x02, "a bus reset arms unit attention"); + let sense = run(&mut dev, &[0x03, 0, 0, 0, 18, 0]); + assert_eq!(sense.data[12], 0x29, "the reset attention has the reset ASC"); + + let response = run(&mut dev, &[0x00, 0, 0, 0, 0, 0]); + assert_eq!(response.status, 0x00, "the attention is reported once"); + assert_eq!(dev.block_sizes().1, 512, "bus reset leaves the selected block size alone"); + } } diff --git a/src/scsi/bus.rs b/src/scsi/bus.rs index 4283d763f..cb978d19a 100644 --- a/src/scsi/bus.rs +++ b/src/scsi/bus.rs @@ -51,8 +51,15 @@ pub trait ScsiTarget: Send { /// The logical block size, for a target that has one. fn block_size(&self) -> Option { None } - /// A controller's bus reset: block size back to what the controller - /// leaves, with a CD-ROM at 512 bytes if `cdrom_512`. + /// A SCSI bus reset reached this target, which owes the initiator a + /// unit attention on its next command. The ISP12160 calls this when the + /// guest resets either of its buses; the WD33C93A, AIC-7880 and ISP1040 + /// models never told their targets about a reset and still do not. + fn bus_reset(&mut self) {} + + /// The AIC-7880's bus reset: block size back to what that controller + /// leaves, with a CD-ROM at 512 bytes if `cdrom_512`. Only the AIC-7880 + /// calls this. fn reset_block_size(&mut self, _cdrom_512: bool) {} /// Put the mode parameters a guest can change back to what the target diff --git a/src/traits.rs b/src/traits.rs index 604cb65c2..a0f322836 100644 --- a/src/traits.rs +++ b/src/traits.rs @@ -223,6 +223,20 @@ pub trait BusDevice: Send + Sync { BUS_OK } + /// An uncached SB (`size` 1) or SH (`size` 2) to `addr`, with the whole + /// source register `gpr` for a bus that can see the lanes the store does + /// not select. The CPU calls this instead of `write8`/`write16` for an + /// uncached SB or SH. + /// + /// The architectural byte or halfword is the low bits of `gpr`, which is + /// all the default delivers. Which other lanes a real processor drives for + /// a narrow store is undefined, so only a device that models a path + /// observing them (Bedrock's memory ECC view) overrides this, and a bus + /// that forwards accesses must forward this call too. + fn write_narrow_from(&self, addr: u64, size: usize, gpr: u64) -> u32 { + if size == 1 { self.write8(addr, gpr as u8) } else { self.write16(addr, gpr as u16) } + } + /// JIT v2 (clanker/jitv2/jit-v2-design.md §2.4): pointer to the generation counter /// for the 4KB physical page containing `addr`, or null if this device doesn't /// back JIT-compilable memory (MMIO, etc). RAM devices return one counter per diff --git a/src/workstation.rs b/src/workstation.rs index 0d5537a80..036beac28 100644 --- a/src/workstation.rs +++ b/src/workstation.rs @@ -40,6 +40,8 @@ use crate::traits::{Device, MachineEvent}; pub fn build(cfg: MachineConfig) -> Result, String> { crate::machine::set_emulator_profile(cfg.machine.profile); match cfg.machine.profile { + crate::config::MachineProfile::FuelIp34 => Ok(Box::new(crate::ip34_machine::FuelMachine::new(&cfg)?)), + crate::config::MachineProfile::TezroIp53 => Ok(Box::new(crate::ip53_machine::TezroMachine::new(&cfg)?)), // The O2 is its own machine (CRIME, MACE, GBE), not an MC/HPC3 one. crate::config::MachineProfile::O2Ip32 => Ok(Box::new(crate::ip32_machine::O2Machine::new(&cfg)?)), // So is the Octane (HEART, Xbow, Bridge). diff --git a/tezro.toml.example b/tezro.toml.example new file mode 100644 index 000000000..31d09b95a --- /dev/null +++ b/tezro.toml.example @@ -0,0 +1,43 @@ +# SGI Tezro (IP53 board, IP35 software platform), serial-only setup. +# Copy to tezro.toml and run from the repository root: iris --config tezro.toml +# Create ignore/tezro/ and supply your decoded raw IP35 PROM 6.210 payload +# and an IRIX disk installed on a Tezro: a disk installed on another machine +# carries that machine's device numbering. +# No PROM or disk images are included. Convert QCOW2 disks to raw first. + +prom = "ignore/tezro/ip35prom.payload.bin" +nvram = "ignore/tezro/nvram.bin" +# banks left unset: 4 GiB as 1024, 1024 and four 512 MiB banks. +headless = true +no_audio = true +monitor_port = 18888 +serial_port_a = 18880 +serial_port_b = 18881 + +[machine] +profile = "tezro_ip53" +# Optional; left out, it is r16000_700, the 700 MHz R16000. "r16000" is the +# 800 MHz part; both have a 4 MiB secondary cache. "r10000" names the family +# and takes the 700 MHz part; other values are refused. +cpu = "r16000_700" +# Processors fitted: 1, 2 or 4 (the default). Each runs on its own host thread. +cpus = 4 + +[graphics] +board = "none" + +[tezro] +flash = "ignore/tezro/flash.bin" +console = "l1" +l1_port = 18882 + +[network] +# Disconnected cable. For NAT, use mode = "nat". +mode = "off" + +[scsi.1] +path = "ignore/tezro/irix.raw" +cdrom = false +# Keep the base disk unchanged. Guest writes go to irix.raw.overlay. +overlay = true +# write_cache = false # default true. false stores every write before it completes, like SGI's drives (slower); true can lose ~5 s of writes in a host crash. See STORAGE.md