diff --git a/.claude/board/EPIPHANIES.md b/.claude/board/EPIPHANIES.md index b3a0080be..5a3dcefdc 100644 --- a/.claude/board/EPIPHANIES.md +++ b/.claude/board/EPIPHANIES.md @@ -1,3 +1,93 @@ +## 2026-09-18 — E-BYTES-ARE-STORED-INTEGERS-ARE-PROJECTED-1 — byte-agnosticism is the STORAGE superpower and little-endian is the COMPUTE superpower; the bug is always a stored projection + +**Status:** OPERATOR-RULED (the framing is the operator's: *"byte is storage +superpower, LE is compute superpower"*) + SHIPPED at the one site that +violated it (`NodeRow::edges` is now the byte-backed `EdgeFacet([u8; 16])`). +**Confidence:** HIGH on the mechanics — measured census, 1356 contract tests, +every consumer unchanged, the new falsifier disable-verified red-then-green. +HIGH on the doctrine as a *reading of this tree*: two of the three sites +already obeyed it before it was written down. + +### The line + +**Bytes are stored. Integers are projected.** + +- **Storage is byte-agnostic, and that is a superpower:** a byte array has no + endianness to get wrong, every bit pattern is valid, no niche, and the + in-memory image equals the wire image on every target for free. +- **Little-endian is a superpower too, but a COMPUTE one:** memory order and + arithmetic significance agree, so a prefix compare is `vpxor` + `tzcnt` with + nothing materialized. That is why the facet's byte-chain LCP measures + **1.72 ns** (`examples/facet_axis_lcp_probe.rs`, #1245) — the reinterpret IS + the speed. +- **They must not be mixed, and the failure has exactly one shape: storing a + projection.** The moment a typed, native-endian value becomes the stored + image, the storage lane inherits the compute lane's endianness — and the + only defence left is a target guard. + +### The tree already agreed, at two sites out of three (measured) + +| site | form | verdict | +|---|---|---| +| `NodeGuid([u8; 16])` | stores bytes, projects via `.facet()` | obeys — the exemplar | +| `AttentionFocusFacet { facet: FacetCascade, .. }` | holds the TYPED facet, is not `repr(C)`, no `SoaEnvelope`, and reaches bytes only through the explicit `to_bytes()` encode | obeys — compute, contained, costs nothing | +| `NodeRow::edges` | stored a `FacetCascade` inside a `repr(C, align(64))` row whose `as_le_bytes()` reinterprets `&[NodeRow] → &[u8]` for Lance | **violated** — a stored projection | + +The third is why #1246 had to add +`const _: () = assert!(cfg!(target_endian = "little"))` to `facet::`. That +guard was a stopgap holding a seam shut, not a fix. + +### What shipped + +`EdgeFacet([u8; 16])` — `#[repr(C, align(16))]`, the exact mirror of +`NodeGuid` — with `as_bytes` / `as_bytes_mut` / `from_bytes` / `to_bytes` / +`facet()`, and `pub type EdgeBlock = EdgeFacet` so every call site compiles +unchanged. Consequence: **all three `NodeRow` fields are now byte arrays** +(`[u8;16] | [u8;16] | [u8;480]`), so the 512-byte row contains no +native-endian integer at all, and `as_le_bytes` is byte-identical across +targets by construction rather than by assertion. Both stale SAFETY comments +are corrected in place — one of them had said, in its own last sentence, that +`EdgeBlock` was "the one field that is not" a byte array. + +### The correction this entry carries + +#1246's arc entry and PR body both said byte-backing `edges` "would retire the +`target_endian` guard entirely." **That was wrong, and the code said so.** +`FacetCascade::as_bytes` / `ref_from_bytes` are still reinterprets, and they +are *supposed* to be — that reinterpret is the 1.72 ns hot path. So the guard +stays; what changed is its blast radius. It no longer protects a row at rest +(that dependency is gone); it protects the compute lens's own +`reinterpret == encode` identity, where a violation can mis-read a value in +flight and nothing more. The guard's comment is rewritten to say exactly that. + +### Why the change was nearly free (the census, not a guess) + +Every `EdgeBlock` site in the tree is `default()`, `as_bytes()` / +`as_bytes_mut()`, `from_bytes()`, equality, or a `Copy` — and every +struct-literal construction and every `.facet_classid` / `.tiers` read is on a +*projected* facet (`FacetCascade::from_bytes(&bytes).tiers[0]`, +`… .facet_classid == CLASSID`). **Not one is a field access on +`NodeRow::edges`.** #1246's migration had already moved them all to bytes. The +two real consumers (`symbiont::key_render`, `soa_graph`) read +`eb.as_bytes()[..12]` / `[12..]` and are untouched. So this change did not +impose the doctrine — it ratified what the code was already doing, and let the +type system say it. + +### The reusable pattern, named + +`byte-backed newtype + .facet() projection`. Storage type owns the bytes and +offers no integer; the lens type owns the integers and is obtained by an +explicit decode. If a type is `repr(C)` AND reachable from a stored image AND +contains a multi-byte integer, it is a stored projection — fix it by moving the +integer behind a projection, not by adding a target guard. + +### Residue (unchanged by this entry) + +The readers that still split those 16 bytes at 12 — the V1 `12 + 4` carving — +remain named in `ISS-EDGE-BLOCK-WAS-A-SECOND-TYPE-FOR-THE-SAME-FACET`. Byte +backing neither fixes nor worsens that; it is a *reading* of the bytes, and the +ClassView is still what should decide it. + ## 2026-09-17 (18) — E-THE-SECOND-FACET-IS-NOT-AN-EDGE-BLOCK-1 — bytes 16..32 are just another content-blind facet cascade; giving them their own type was how the V1 `12 + 4` carving survived its own retirement **Status:** OPERATOR-RULED (verbatim below) + SHIPPED (`pub type EdgeBlock = diff --git a/.claude/board/LATEST_STATE.md b/.claude/board/LATEST_STATE.md index aa83165f3..b0a4ce9bd 100644 --- a/.claude/board/LATEST_STATE.md +++ b/.claude/board/LATEST_STATE.md @@ -1,3 +1,40 @@ +## 2026-09-18 (2) — CONTRACT INVENTORY DELTA: `EdgeFacet([u8; 16])` — `NodeRow::edges` is byte-backed, so no field of the 512-byte row is a native-endian integer any more + +- **Added:** `lance_graph_contract::canonical_node::EdgeFacet` — + `#[repr(C, align(16))]` over `[u8; 16]`, the exact mirror of `NodeGuid`, with + `as_bytes` / `as_bytes_mut` / `from_bytes` / `to_bytes` / `facet()` and + `From`/`Into` against `FacetCascade`. **`pub type EdgeBlock = EdgeFacet`** — + the name is unchanged, the TYPE it points at changed. +- **Retyped:** `NodeRow::edges` was `FacetCascade` (since the 2026-09-17 alias); + it is now `EdgeFacet`. Byte positions, `NODE_ROW_STRIDE`, + `ENVELOPE_LAYOUT_VERSION` and `node_rows_from_le_bytes` are all unchanged. +- **State consumers should know:** nothing breaks. Every call site is + `default()`, `as_bytes()`, `as_bytes_mut()`, `from_bytes()`, equality or a + `Copy`, and all of those are preserved; the typed cascade is now reached by + **projection** — `row.edges.facet()` — the same way `NodeGuid::facet()` has + always worked. `weather-poc` (40 tests) and `lance-graph-planner` build and + pass unchanged. +- **Why:** `NodeRow` is `#[repr(C, align(64))]` and `as_le_bytes()` reinterprets + `&[NodeRow] → &[u8]` for Lance, so a typed `edges` made the compute type's + native-endian `u32` part of the **stored** image. Doctrine and full census: + `EPIPHANIES.md` `E-BYTES-ARE-STORED-INTEGERS-ARE-PROJECTED-1` — *bytes are + stored, integers are projected*. +- **The `target_endian` guard STAYS, narrowed.** #1246's arc entry said + byte-backing `edges` would retire it entirely; that was wrong. + `FacetCascade::as_bytes` is still a reinterpret **by design** — it is the + 1.72 ns hot path — so the `reinterpret == encode` identity is still assumed. + What changed is the blast radius: a violation can now mis-read a value in + flight, never corrupt a row at rest. The guard's comment says so. +- **Corrected in place:** both `canonical_node.rs` SAFETY comments. One of them + ended by asserting `EdgeBlock` was "the one field that is not" a byte array — + true when written, false now, and struck rather than silently reworded. +- **Falsifier:** `edges_store_bytes_verbatim_and_project_the_integer_little_endian` + asserts both superpowers at once — the stored bytes are verbatim (endian-free) + AND `facet().facet_classid` decodes `0xDEAD_BEEF` little-endian. Disable-run: + byte-swapping the first four bytes in `from_bytes` fails it on "stored bytes + are verbatim"; restoring passes. 1356 contract tests green, clippy clean under + `-D warnings`. + ## 2026-09-18 — PR #1246 merged (`568965e9`): `EdgeBlock` is a `FacetCascade` on `main`, `Pred::Range` is in the mask-risc IR, and no manifest in this repo carries a `.0.0` pin The two entries below dated 2026-09-17 (3) and (2) describe what is now on diff --git a/crates/lance-graph-contract/src/canonical_node.rs b/crates/lance-graph-contract/src/canonical_node.rs index 5039b5be7..fab12147d 100644 --- a/crates/lance-graph-contract/src/canonical_node.rs +++ b/crates/lance-graph-contract/src/canonical_node.rs @@ -685,7 +685,105 @@ impl core::fmt::Display for NodeGuid { /// `ISS-EDGE-BLOCK-WAS-A-SECOND-TYPE-FOR-THE-SAME-FACET`. New code says /// `FacetCascade` and reads through `as_bytes()` / the ClassView; migration /// pointer per I-LEGACY-API-FEATURE-GATED. -pub type EdgeBlock = crate::facet::FacetCascade; +#[derive(Clone, Copy, PartialEq, Eq, Hash, Default)] +#[repr(C, align(16))] +pub struct EdgeFacet([u8; 16]); + +impl EdgeFacet { + /// The 16 bytes, verbatim. This is a byte read, not a reinterpret of a + /// typed field, so it is identical on every target. + #[inline] + #[must_use] + pub const fn as_bytes(&self) -> &[u8; 16] { + &self.0 + } + + /// The 16 bytes, mutably — the only write path (`EdgeCodecFlavor` decides + /// what a byte MEANS; this type never does). + #[inline] + pub fn as_bytes_mut(&mut self) -> &mut [u8; 16] { + &mut self.0 + } + + /// Adopt 16 stored bytes. No decode: the bytes ARE the value. + #[inline] + #[must_use] + pub const fn from_bytes(b: &[u8; 16]) -> Self { + Self(*b) + } + + /// The 16 bytes by value. + #[inline] + #[must_use] + pub const fn to_bytes(self) -> [u8; 16] { + self.0 + } + + /// **The projection.** Read these bytes through the typed cascade lens — + /// `facet_classid(4) | 6×(8:8)` — exactly as [`NodeGuid::facet`] does for + /// the key. The integer fields are DECODED here (`u32::from_le_bytes`), + /// they are not stored here. + #[inline] + #[must_use] + pub const fn facet(&self) -> crate::facet::FacetCascade { + crate::facet::FacetCascade::from_bytes(&self.0) + } +} + +impl core::fmt::Debug for EdgeFacet { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + write!(f, "EdgeFacet({:02x?})", self.0) + } +} + +impl From for EdgeFacet { + /// Encode a computed facet down into stored bytes — explicitly, via + /// [`FacetCascade::to_bytes`](crate::facet::FacetCascade::to_bytes). + #[inline] + fn from(f: crate::facet::FacetCascade) -> Self { + Self(f.to_bytes()) + } +} + +impl From for crate::facet::FacetCascade { + #[inline] + fn from(e: EdgeFacet) -> Self { + e.facet() + } +} + +/// The historical name for bytes 16..32. Kept as an alias so every existing +/// call site (`EdgeBlock::default()`, `as_bytes()`, `from_bytes()`, equality) +/// compiles unchanged; what changed is the TYPE it names. +/// +/// # Migration — this alias is source-BREAKING for field access +/// +/// `EdgeBlock` named [`FacetCascade`](crate::facet::FacetCascade) before this +/// change, so its `facet_classid` / `tiers` fields and every inherent +/// `FacetCascade` method were reachable straight through the alias. They are +/// not any more: [`EdgeFacet`] deliberately exposes **bytes only**, because a +/// typed integer inside [`NodeRow`] is a *stored projection* — the one shape +/// this failure ever takes. +/// +/// The fix is to name the projection you were implicitly taking: +/// +/// ``` +/// # use lance_graph_contract::canonical_node::EdgeBlock; +/// # let edges = EdgeBlock::default(); +/// // before: edges.facet_classid +/// let classid = edges.facet().facet_classid; +/// # let _ = classid; +/// ``` +/// +/// This breaks LOUDLY — it is a compile error at the call site, never a silent +/// reinterpretation of stored bytes — which is why no feature gate or +/// read-mode alias is warranted under `I-LEGACY-API-FEATURE-GATED` (that rule +/// governs a name whose *semantics* change silently, not one that stops +/// compiling). No consumer is known to be affected: every in-tree site is +/// `default()` / byte access / equality / `Copy`, and the one named +/// out-of-repo consumer of this module, `lance-graph-java`'s `lgj-abi`, +/// imports `canonical_node::EdgeCodecFlavor` and never the block type. +pub type EdgeBlock = EdgeFacet; /// Which edge-codec flavor a class uses to *read* its node's edge block. /// @@ -1678,16 +1776,18 @@ impl<'a> SoaEnvelope for NodeRowPacket<'a> { // every byte position is valid for reads (no padding past size_of, // alignment of NodeRow (64) ⊇ alignment of u8 (1)). // - // The NodeGuid and EdgeBlock fields hold their bytes in canon-LE - // order, so the resulting byte slice IS the envelope's LE packet — no - // translation needed at the boundary. NodeGuid earns that by being - // `[u8; 16]` outright. EdgeBlock does NOT: since the 2026-09-17 alias - // it is a `FacetCascade`, whose `facet_classid` is a native-endian - // `u32`, so this cast reproduces the LE packet only on a little-endian - // target. That is enforced, not assumed — `facet::` carries a - // crate-level `const _: () = assert!(cfg!(target_endian = "little"))` - // for exactly this path (codex P2 on #1246). Do not restate EdgeBlock - // as a byte array here; it is the one field that is not. + // The resulting byte slice IS the envelope's LE packet, with no + // translation at the boundary and no target dependency, because EVERY + // field of NodeRow is byte-backed: `NodeGuid([u8; 16])`, + // `EdgeFacet([u8; 16])`, `value: [u8; 480]`. There is no native-endian + // integer anywhere in the 512 bytes to reorder. + // + // ⊘ This comment previously said the opposite of its last sentence — + // that EdgeBlock was "the one field that is not" a byte array, being a + // `FacetCascade` with a native-endian `u32`, so the cast reproduced the + // LE packet on little-endian targets only. True from the 2026-09-17 + // alias until `edges` was byte-backed; the typed cascade is now reached + // by PROJECTION (`row.edges.facet()`), never stored. unsafe { core::slice::from_raw_parts( self.rows.as_ptr().cast::(), @@ -1739,12 +1839,12 @@ pub fn node_rows_from_le_bytes(bytes: &[u8]) -> Option<&[NodeRow]> { // (const-asserted above). We checked (1) bytes.len() is an exact multiple of // the stride, so n rows span the whole slice with no trailing bytes, and (2) // the pointer is aligned to align_of::() (64). Every bit pattern in - // the 512 bytes is a valid NodeRow (NodeGuid is `[u8; 16]`; EdgeBlock is a - // `FacetCascade`, 16 B `repr(C, align(16))` whose fields are all plain - // integers, so it too has no niche; value is `[u8; 480]`) — nothing to - // invalidate, so the reinterpretation is sound. Soundness does not imply - // byte-identity across targets: see `as_le_bytes` above and the - // little-endian assert in `facet::` that this path also relies on. The returned slice borrows `bytes` for its lifetime (no copy). + // the 512 bytes is a valid NodeRow — all three fields are byte arrays + // (`NodeGuid([u8; 16])`, `EdgeFacet([u8; 16])`, `value: [u8; 480]`), so none + // has a niche — nothing to invalidate, and the reinterpretation is sound. + // Since no field is a native-endian integer, byte-identity across targets + // holds too (it did not while `edges` was a typed `FacetCascade`; see + // `as_le_bytes` above). The returned slice borrows `bytes` for its lifetime (no copy). Some(unsafe { core::slice::from_raw_parts(bytes.as_ptr().cast::(), n) }) } @@ -2159,8 +2259,8 @@ mod tests { core::mem::align_of::(), core::mem::align_of::() ); - let f: crate::facet::FacetCascade = e; // an alias, not a conversion - assert_eq!(f.as_bytes(), &[0u8; 16]); + let f: crate::facet::FacetCascade = e.facet(); // a PROJECTION, not an alias + assert_eq!(f.to_bytes(), [0u8; 16]); let mut g = EdgeBlock::from_bytes(&[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]); g.as_bytes_mut()[12] = 0; assert_eq!(g.as_bytes()[11], 12); @@ -2168,6 +2268,41 @@ mod tests { assert_eq!(g.as_bytes()[13], 14); } + /// **The two superpowers, in one test.** What is STORED is bytes, verbatim, + /// on any target; the integer is PROJECTED out of them, little-endian, by an + /// explicit decode. + /// + /// This is the test the old `edges: FacetCascade` could not pass on a + /// big-endian target: `as_bytes()` was a reinterpret of a native-endian + /// `u32`, so the first assert would have seen a byte-swapped class id and + /// the stored row image would silently disagree with `to_bytes()`. Now the + /// first assert is a byte identity (nothing to swap) and the second is a + /// decode that names its own endianness — so BOTH hold everywhere, which is + /// why bytes 16..32 no longer depend on the target at all. + #[test] + fn edges_store_bytes_verbatim_and_project_the_integer_little_endian() { + let b: [u8; 16] = [ + 0xEF, 0xBE, 0xAD, 0xDE, // classid, LE on the wire + 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, + ]; + let e = EdgeFacet::from_bytes(&b); + + // STORAGE: byte-agnostic. The stored image is the source bytes, and a + // round trip through the row field cannot reorder them. + assert_eq!(e.as_bytes(), &b, "stored bytes are verbatim"); + assert_eq!(e.to_bytes(), b); + + // COMPUTE: little-endian, and it says so. The integer exists only here. + assert_eq!( + e.facet().facet_classid, + 0xDEAD_BEEF, + "the class id is DECODED from the stored bytes, never stored as a u32" + ); + + // And the projection is lossless back to the same stored bytes. + assert_eq!(EdgeFacet::from(e.facet()).as_bytes(), &b); + } + #[test] fn edge_codec_flavor_default_is_coarse_only() { // Zero-fallback default: the all-zero reading is the canon bootstrap. diff --git a/crates/lance-graph-contract/src/facet.rs b/crates/lance-graph-contract/src/facet.rs index 6a0af6179..14f5de45e 100644 --- a/crates/lance-graph-contract/src/facet.rs +++ b/crates/lance-graph-contract/src/facet.rs @@ -99,19 +99,28 @@ pub struct FacetCascade { pub tiers: [FacetTier; 6], } -// The facet is a STORED row field (`NodeRow::edges`), and `as_bytes` / -// `ref_from_bytes` are pure pointer reinterprets — so the struct's in-memory -// image IS the canonical LE row image. `facet_classid` is a native-endian -// `u32`, which makes that identity hold on little-endian targets ONLY: on a -// big-endian target `from_bytes` (explicitly `u32::from_le_bytes`) and -// `as_bytes` (a reinterpret) would disagree on bytes `[0..4)`, silently -// byte-swapping a non-zero class id through serialization. The predecessor -// type at this row offset (`EdgeBlock { in_family: [u8; 12], out_family: -// [u8; 4] }`) was byte-backed and so endian-independent; aliasing it to this -// typed facet is what introduced the dependency. Fail LOUD at compile time -// rather than corrupt a row image at runtime — the round trip is pinned by -// `le_byte_image_round_trips_with_a_non_zero_classid` below. -// (codex P2 on PR #1246; `ISS-EDGE-BLOCK-WAS-A-SECOND-TYPE-FOR-THE-SAME-FACET`.) +// ⊘ NARROWED. This guard was introduced (codex P2 on #1246) because +// `NodeRow::edges` WAS a `FacetCascade`, so this struct's in-memory image was +// the canonical stored row image — and `facet_classid` is a native-endian +// `u32`, so on a big-endian target `as_bytes` (a reinterpret) and `from_bytes` +// (an explicit `u32::from_le_bytes`) would disagree on `[0..4)` and silently +// byte-swap a class id through serialization. **That storage dependency is +// GONE:** `edges` is now the byte-backed `EdgeFacet([u8; 16])`, so the whole +// 512-byte row is `[u8;16] | [u8;16] | [u8;480]` with no native-endian integer +// anywhere in it, and nothing typed is stored. +// +// What the guard still protects is this type AS A COMPUTE LENS: `as_bytes` / +// `ref_from_bytes` remain pointer reinterprets, and that is deliberate — the +// reinterpret IS the fast path (`examples/facet_axis_lcp_probe.rs` measures the +// byte-chain LCP at 1.72 ns precisely because nothing is materialized). So the +// `reinterpret == encode` identity is still assumed, and is still pinned by +// `le_byte_image_round_trips_with_a_non_zero_classid` below — but a violation +// can now only mis-read a value in flight, never corrupt a row at rest. +// +// The doctrine, in one line: BYTES ARE STORED, INTEGERS ARE PROJECTED. Storage +// is byte-agnostic (any target, any bit pattern); little-endian is a COMPUTE +// superpower and lives on this side of the projection only. +// (`ISS-EDGE-BLOCK-WAS-A-SECOND-TYPE-FOR-THE-SAME-FACET`.) const _: () = assert!( cfg!(target_endian = "little"), "FacetCascade's reinterpret-based LE byte image assumes a little-endian target"