Repository navigation
Fit PCI add-in cards on the O2 from the config - #422
Draft
jamesbraid wants to merge 12 commits into
Draft
jamesbraid wants to merge 12 commits into
jamesbraid wants to merge 12 commits into
Conversation
Fuel uses two independent SCSI buses behind one ISP12160 function. Reuse the ISP1040 command engine and let the part trait own the firmware, register and channel differences. A bus reset reaches every attached target as unit attention rather than a media change. Keep the reset attention on the shared target interface so a controller can notify each device through the same bus. The ISP12160 is its only caller: the WD33C93A, AIC-7880 and ISP1040 models do not call it, so their targets behave as before. A bus reset restores each disk's write cache bit per channel: a reset of one channel leaves the other's disks alone, and a chip or soft reset restores both. The engine changes reach the Octane's ISP1040: mailbox commands now run with dispatch held, and the target interface gains the bus reset. The Octane's IRIX 6.5.30 SCSI disk (Octane2, R10000, 256 MiB, V12, jitv2, release build, headless, NAT) boots to the login prompt and runs commands on this branch's tip and on main c6dea8c: login after 126 s on the branch and 109 s on main in one run each (host load makes the times indicative only), the same `hinv` listing of both integral controllers and the disk, and the same command output.
MacePci::attach and Bridge::attach each repeated the same four steps: hand the function its way to memory, read its interrupt pin, build the line for that pin if there is one, and connect it. They differ only in the DMA object and in what a pin is wired to. pci::plug takes those two as arguments and does the rest, so a bus that gains a way to place a function at a device number does it in one place. No behaviour change: the same objects are connected in the same order, and a function with no pin is still left unconnected.
MacePci kept each pin's level and ORed the pins routed to one CRIME input in a private router whose routing was a fn pointer baked in at construction. The Fuel and Tezro buses carry copies of the same logic. IntxTable is that logic as a type in pci.rs, with the routes held as data (IntxTable::route) instead of a function. A bus can then wire a function at a device number it learns about after construction without changing the bridge. MacePci fills the table from its existing routing function, so the O2 routes exactly what it did: slot 1 INTA to CRIME 8, slot 2 INTA to CRIME 9, everything else dropped. Levels, the snapshot "pins" array, the pci status line and the order in which a shared input is driven are unchanged. The table also drops the per-change scan that called the routing function for all 128 lines; the scan now reads the stored routes.
PCI functions need different writable and write-one-to-clear header fields. Describe those fields in one construction policy while keeping the current header defaults for existing functions. Forward narrow Bridge configuration writes with their byte enables so write-one-to-clear bits are handled correctly when a function supplies them. Existing Octane functions do not use that status policy, so their behavior stays unchanged.
Which device numbers on a machine's PCI bus take an add-in card is a fact about the chassis, not something a configuration should be able to invent. Describe it once, as a const table per bus: each device number is either an onboard function the machine wires itself, or an add-in slot with how well its number is known and the public manual or driver it comes from. Only an established or observed number can be filled; the rest can be listed with what would settle them. The O2 has one add-in slot, device 3, which Linux and OpenBSD agree on and the hardware guides describe as a single PCI board position. The Octane's Bridge has four onboard devices and no add-in slot: its PCI boards hang off an XIO port behind a Bridge of their own, and neither the port nor the device numbers are established, so that table says so instead of guessing. The O2's `pci slots` monitor command prints the table with what is fitted and where each add-in pin goes. Nothing routes slot 3 yet, so it reports INTA to INTD unrouted.
A fitted card needs more from a machine than a slot: the SCSI buses it brings, a worker to start and stop, state to save. Give every card the same small interface and list the kinds in one static table, each entry building a fresh card, so a machine can fit several of the same kind. Nothing here depends on a particular bus. The first card is a dual-channel PCI SCSI adapter built on the QLogic ISP12160. It reuses the ISP12160 model added just before it in this series and brings two wide SCSI buses. It takes no options: nothing about the part is configurable on the board.
Turn a card entry into a placement or a message that says what to fix. An entry names a bus (optional on a machine with one), a slot, a card and the card's own options. The slot is a PCI device number or the label printed on the chassis; labels are resolved here and nowhere else, so the buses, snapshots and the monitor only ever see device numbers. Checks run in the order a reader would fix them: the machine has PCI, the bus exists, the slot is an add-in slot and its number is known, the slot is free, the card exists, its options are ones it takes. Onboard devices are named in the message instead of reported as missing, because they are real and cannot be replaced. Several cards of one kind are allowed; their SCSI buses are numbered after the machine's own, in entry order. The command-line form is `[BUS:]SLOT=CARD[,KEY=VALUE]...`. An entry on the command line replaces a configuration entry for the same slot.
Let a configuration fit a card in an add-in slot: one `[[pci]]` table per card with `slot`, `card` and the card's own options inline, or `--pci [BUS:]SLOT=CARD[,KEY=VALUE]...` on the command line. Validation runs with the rest of the configuration, so a mistake stops the run before any machine is built. The card's SCSI buses become controllers after the machine's own, so `[scsi.N]` can put a disk on them: controllers 2 and 3 for the first ISP12160 on an O2. Their ids go to 15, since the card's buses are wide; the O2's own buses stay at 7. Without a card the controller limits and their message are as before.
Wire configured cards into the O2. The bridge routes the slot's INTA to CRIME input 10 and INTB to INTD to the shared inputs 13 to 15, as the MACE specification's PCI interrupt tables have them, and Linux, OpenBSD and NetBSD agree; a shared input is the OR of every pin routed to it. Whether the board's connector wires INTB to INTD is not known. This is the only change a machine without a card sees, and nothing answers in the slot then. The machine builds the cards before it attaches `[scsi.N]` disks, so a disk can sit on a card's bus (controllers 2 and 3 for an ISP12160), then hands them to the bus, which attaches them and owns their lifetime: they start and stop with the onboard controllers, their disks are flushed and synced with the others, their copy-on-write overlays are saved by controller number, and their state is part of the bus's saved state. Restoring state saved with a different number of cards is refused.
A snapshot taken with a card in the slot holds the card's registers and its ring state. Restored into a machine without the card, that state has nowhere to go; restored the other way round, the card would sit at power-on under a kernel that has been driving it. Neither is a state the machine was ever in, so list the fitted cards in the manifest, by bus, device number and card, and compare before touching anything. The O2 refuses with a line per difference. Device numbers, not chassis labels, are recorded, so relabelling a slot later cannot break an old save. The key is optional and left out when nothing is fitted, so snapshots without cards read and write as before and the schema version does not change; an older build restoring a save with cards ignores the key and fails later on the saved slot instead. The check does not honour IRIS_SNAPSHOT_SKIP_CHECK.
The O2's IRIX has no driver for the ISP12160: it finds the card on the PCI bus and lists it as an unclaimed adapter, but has nothing to bind, so no controller or disk shows in `hinv`. The Adaptec AIC-7880 is the chip the O2 already drives onboard, and it also came on PCI cards, so the same driver claims one in the add-in slot and the guest exercises the slot end to end: configuration, BARs, DMA and the interrupt on CRIME input 10. The card is the existing controller model with one bus, numbered as the next controller after the machine's, which is why a card is now built knowing its first controller number.
Describe `[[pci]]`, `--pci`, the O2's one slot and its interrupt wiring, the two cards and how a card's SCSI buses are numbered, the `pci slots` monitor command and the snapshot rule. Say which card IRIX 6.5.30 drives: the AIC-7880 shows in `hinv` with its disk, the ISP12160 is found on the bus but has no driver.
Owner
|
nice! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The O2 has one real PCI add-in slot and iris had no way to put a card in it. Now
[[pci]]in the config, or--pci 3=aic7880, fits one and the guest finds it at PCI device 3 with its interrupt on CRIME input 10.Only slots the hardware has can be named. Each bus has a table of device numbers, how well each is known, and the public driver or manual it comes from. Onboard devices are in the table too, so asking for the onboard SCSI controller says so. The slot's four pins are routed as the MACE specification's interrupt tables have them: INTA to CRIME 10, INTB to INTD to the shared inputs 13 to 15, each the OR of every pin routed to it. Linux, OpenBSD and NetBSD agree. What the board's connector wires INTB to INTD to is not known. Chassis labels are accepted in the config file as another spelling of the device number; the O2's slot has none, and
--pcitakes device numbers only.A card's SCSI buses are numbered after the machine's own, so
[scsi.N]can put a disk on them. Snapshots record the fitted cards and refuse a restore into a machine with others.Two cards. The ISP12160 was the card asked for, but IRIX 6.5.30 on the O2 has no driver for it: it shows up as "PCI Adapter ID ... PCI slot 3" in
hinv -vand no controller. The AIC-7880 is the chip the O2 already has onboard, it also came on PCI cards, and the same IRIX driver claims it, so it is the one that proves the slot end to end:hinvlists "PCI SCSI controller 2" and its disk, andhalt -ppowers off.Two commits are the Fuel branch's, unchanged: the PCI header write policy and the ISP12160 part. The second reworks
isp1040.rs, the Octane's onboard controller (mailbox commands run with dispatch held, bus reset reaches targets); the Octane IRIX 6.5.30 disk boots the same with it. They drop out of the Fuel rebase.The Octane's XIO PCI shoebox and shoehorn come in a follow-on PR. The Octane has a table of its four onboard devices and no add-in slot until then.
Not done: the card buses have no monitor
scsicommands (the ISP12160's names would collide with the onboard ones), so disks go in the config.Performance
No machine without cards runs different code on the hot paths. Against upstream main 28fa516, interleaved, 15 full-suite runs per build, R10000, ship profile: nothing is slower. transcend, linpack, stream_triad and whetstone are within noise. uncached (+4.4%), llsc (+1.5%) and dhrystone (+1.2%) moved up, with overlapping spreads, which is code placement rather than a real gain.
Testing
Tested: lib tests in release with jitv2 and in debug; each commit's pci, ip32, snapshot, config tests; each pin of a card in the slot reaches its CRIME input, and two pins on a shared input OR. IRIX 6.5.30 booted on an O2 with the AIC-7880 card and a disk on it. Not tested: the ISP12160 under a guest that drives it.