Skip to content

Streaming: tighter pages hold 41% more clusters in the same pool (file version 5) - #87

Merged
SamG-Coder merged 1 commit into
mainfrom
claude/tighter-pages
Nov 13, 2026
Merged

SamG-Coder merged 1 commit into
mainfrom
claude/tighter-pages

Conversation

@SamG-Coder

@SamG-Coder SamG-Coder commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

What and why

A page pool of fixed slots holds as many clusters as fit in a slot, and a streamed view is as detailed as its pool lets it be. This makes a cluster take less of a slot and the slots fuller, as scene file version 5. On a 31-million-triangle garden scene the same pool holds 41% more clusters, pages are 36% less to download, and frame times do not move.

  • A vertex is 12 bytes, not 16. Two bytes were padding, and the normal had 32 bits. It is now octahedral in 16, a signed byte for each coordinate (Nanite's default), with the four codes around the exact point tried for the nearest: 0.31° out on average, under 1° at worst. Positions and UVs keep every bit, so nothing moves and no seam can open.
  • A triangle is 3 bytes, not 4, end to end within its cluster. A mesh being built still has a word for each; pages are packed when they are made. This is the "compact three-byte triangle stream" docs/performance-audit.md left for a format change.
  • Offsets are words. A cluster's vertex_offset counts words of the pool, as triangle_offset did, so a slot need not be a whole number of vertices and a banded pool needs no special arithmetic. Shaders read both through a new common/pool.wgsl.
  • Skins lie end to end behind their cluster's vertices: a skinned vertex is 20 bytes where it was 24.
  • A page is filtered before it is deflated (file::filter_page): each field of a vertex as its difference from the vertex before, one byte of it to a run. Hosts that read a file hand pages over with the new Renderer::deliver_fetched, which undoes it; deliver still takes a page as the GPU reads it.
  • Room left in a page is filled from the next twelve sets of the same mesh and round, where any fit.
  • page_stats, a new example in solstice-asset, measures any bake: cargo run --release -p solstice-asset --example page_stats -- FILE.
Version 4 Version 5
Vertex, triangle 16 and 4 bytes 12 and 3
Pages, unpacked 1,294.6 MiB 970.8 MiB
Pages, as stored 787.3 MiB 499.9 MiB
The scene file 1,062.7 MB 761.2 MB
Pages 11,309 8,033
Slots full to 91.6% 96.7%
Clusters to a slot 47.9 67.4

docs/streaming.md has the rest under "Tighter pages": what each of six views wants before and after, and two packing ideas that were measured and left out because they made views want more pages.

Before merging

  • Version 4 files no longer open. The reader says "unsupported file version 4". Merging deploys the new viewer to Pages, so the published scenes (solstice-scene-*) must be baked again and published with python scripts/publish-scenes.py --bake at the same time, or the live site cannot open them in between. Zorah and any split .web file need baking again too. I have not done any of that.
  • The picture changes a little. A version 5 render of the test scene differs from its version 4 render by 35 to 37 dB, where two renders of one file differ by 44 to 47 dB. The difference is soft shifts in bounced light, not edges: normals a fraction of a degree apart now weld into one vertex, so meshes simplify slightly differently. Side by side they look the same.
  • The scene the numbers come from is not in this repository. It is a castle in its gardens, mostly leaf-by-leaf planting, which is why vertices were 82% of its pages. The cove and the forest were not baked for sizes.

Verified

  • Local checks: bash scripts/check-changed.sh passes on this head (d2efb37), in a clean checkout holding nothing but this commit. It found 13 of the 17 packages affected, so that is fmt, clippy with -D warnings, 878 tests in release including the 93 GPU tests, the wasm32 builds of the viewer, the editor and bake-web with clippy, and node apps/viewer/web/editor/test.mjs.
  • New tests: packed triangles and skins read back as they went in; a packed normal is within a degree everywhere and 0.35 on average; filtering a page loses nothing and deflates to under half; room left in a page is filled from the set that follows. The paging tests now read pages back by the word.
  • Every cluster read back from a real bake: page_stats unpacks every 16th page of the test scene and checks each triangle of each cluster names a vertex its cluster has (34,009 clusters).
  • Measured before and after on the test scene, both formats baked from the same sources and run back to back at 1600x900:
    • With the default 128 MiB pool, pages short of room after 400 frames: 1,153 to 643 from the gate, 204 to 0 from the air, 658 to 272 in the maze.
    • With a pool that holds everything, pages installed after 600 frames: 2,759 to 2,080 from the gate (345 to 260 MiB), and 20 to 25% fewer in five other views.
    • Frame time with the whole scene loaded, 400 frames: 5.56 ms to 5.53 from the gate (raster 2.03 to 2.00, resolve 0.30 to 0.30), 3.52 to 3.46 in the maze, 2.94 to 2.92 indoors.
  • In the browser: the version 5 scene streamed through the rebuilt wasm viewer with no GPU errors, to 2,560 pages in a 320 MiB pool, in a pane 800x600. Walking and collision still work natively (--play).
  • One thing to know about the GPU tests: in a debug build they crashed with an access violation twice while a browser tab held a large scene on the same GPU, and passed one at a time and again in parallel once the tab was closed. In release, as the script runs them, they passed first time.

Not measured: a browser on a real network, and the sizes of a scene with skinned meshes or regions (their tests pass).

Authorship

Written by: Claude Fable 5.2 through Claude Code

Reviewed by: not yet reviewed.

  • All code, text and comments in this pull request were written by a frontier AI model, with no human-written code, and will be reviewed by a frontier AI model before merging, as the Code of Conduct requires.

🤖 Generated with Claude Code

…fuller pages

A page pool of fixed slots holds as many clusters as fit in a slot. This makes a
cluster smaller and the slots fuller, as scene file version 5:

- A vertex is 12 bytes, not 16: the padding goes, and the normal is octahedral
  in sixteen bits, the nearest of the four codes around the exact point.
  Positions and UVs keep every bit.
- A triangle is 3 bytes, end to end within its cluster. A mesh being built keeps
  a word for each; pages are packed when they are made.
- A cluster's vertex offset counts words of the pool, as its triangle offset
  did. Shaders read both through common/pool.wgsl.
- Skins lie end to end behind their cluster's vertices.
- A page's vertices are filtered before it is deflated, each field as its
  difference from the vertex before and each byte in a run of its own. Hosts
  that read a file hand pages over with Renderer::deliver_fetched, which undoes
  it.
- Room left in a page is filled from the next few sets of the same mesh and
  round.

On a 31-million-triangle garden scene: pages 1,295 MiB to 971 unpacked and 787
to 500 stored, 47.9 clusters to a slot to 67.4, the same frame times.
docs/streaming.md has the measurements, and what was tried and left out.
`cargo run -p solstice-asset --example page_stats -- FILE` measures a bake.

A version 4 file no longer opens: scenes must be baked again.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@SamG-Coder
SamG-Coder merged commit f214c0e into main Nov 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant