perf: address routing snapshots by position

[djwhitt]

Review scope

This stacked PR changes only target addressing. Its base, #3943, owns compact targets, batch reuse, missing-generation repair, byte-aware capacity, and lifecycle telemetry.

Boundary	Revision
Current main used for the cumulative memory comparison	fcb4c938
ID-keyed base (#3943)	6820c6ee
This positional head	4754419c

What changes

The tree and ordered target tuple are built from the same sorted rule list. The tree then emits zero-based tuple positions instead of database rule IDs.

ID-keyed #3943	Positional #3946
Tree emits rule ID	Tree emits tuple position
Binary-search ID index	Direct tuple access
Sparse lookup searches then reads ETS	Sparse lookup reads the ETS element directly
Dense lookup rebuilds an ID-keyed map	Dense lookup reads the target tuple directly
Snapshot stores IDs and targets	Snapshot stores targets only

No cache lifecycle, batch API, recovery, telemetry, or capacity-policy changes remain in this PR.

Correctness constraints

• The tree and target tuple use one query and one deterministic ordering.
• Every emitted position addresses the target from that same immutable generation.
• Replacement, eviction, restart fallback, and conditional repair retain perf: preserve routing snapshots without copying unmatched rules #3943's consistency behavior.
• Missing, out-of-range, and nil targets remain rejected.

Incremental effect of this PR

Same corrected fixtures and Linux / 6-core / Elixir 1.19.5 / OTP 27.3.4.6 environment as #3943. These are local microbenchmarks, not production-throughput claims.

One adjacent run per representation:

Rules / matches	ID-keyed #3943	Positional child	Approximate change
100 / 8	7.02 us	5.92 us	1.19x faster
100 / 1	11.65 us	10.60 us	1.10x faster
100 / all	24.78 us	19.11 us	1.30x faster
1,000 / 8	35.97 us	33.92 us	1.06x faster
1,000 / 1	129.17 us	115.80 us	1.12x faster
1,000 / all	347.55 us	264.88 us	1.31x faster

Sparse results have substantial scheduler/GC variance. An earlier matching-only probe found positional and ID keys within 1%; the dense gains from avoiding map reconstruction are more persuasive than the small sparse differences.

Additional positional effects:

• Prepare-once allocation for 1,000 rules / 8 matches fell from 56.20 KB to 25.80 KB for 10 events and from 0.54 MB to 0.24 MB for 100 events: 2.18-2.25x lower.
• Removing the index and entry IDs reduced the already-compact modeled snapshot by another 29-39%.
• Wall-clock batch results varied between runs, so no additional positional batch-speed claim is made.

Incremental commands and details: source_routing_followup_results.md.

MIX_ENV=test ../bin/x mix run test/profiling/source_routing_bench.exs
MIX_ENV=test ../bin/x mix run test/profiling/source_routing_batch_bench.exs

Cumulative final stack vs current main

A branch-neutral external harness used the same database-backed 100/1,000-rule fixtures and ran twice per revision in the same environment. Caches were warm before Benchee measurement; retained deltas and allocation results reproduced across both runs.

Retained routing-cache memory

The ETS delta includes Rules.Cache and, on the final stack, the routing snapshot data/source/expiry tables:

1,000-rule cache state	Matches	Main	Final	Change
One representative event	1	119.9 KB	158.8 KB	32.5% higher
One representative event	8	99.7 KB	128.9 KB	29.2% higher
One representative event	1,000	1.80 MB	121.2 KB	93.2% lower / 14.8x
All rules warmed	1	1.90 MB	158.8 KB	91.7% lower / 12.0x
All rules warmed	8	1.55 MB	128.9 KB	91.7% lower / 12.1x
All rules warmed	1,000	1.84 MB	121.2 KB	93.4% lower / 15.2x

Interpretation:

• Main lazily caches only %Rule{} records that have matched; final stores all compact targets immediately. That gives final a 29-33% higher first-event floor in the 1,000-rule sparse fixtures.
• Main retained memory grows as distinct rules are encountered, while final remains nearly flat.
• Once all main per-rule entries are warm, final uses 91-93% less ETS memory. The same comparison at 100 rules was 91.1-92.8% lower.
• An externalized-term payload model, excluding ETS table overhead, was 96.1-97.2% lower across fully warmed 100/1,000-rule fixtures.

Per-batch allocation

Benchee allocation for 1,000 rules:

Matches / events	Main	Final	Change
1 / 10	3.21 MB	3.22 MB	comparable
1 / 100	38.42 MB	27.55 MB	28.3% lower / 1.39x
8 / 10	875.42 KB	112.48 KB	87.2% lower / 7.78x
8 / 100	7.46 MB	338.19 KB	95.6% lower / 22.6x
1,000 / 10	28.03 MB	5.98 MB	78.7% lower / 4.69x
1,000 / 100	284.52 MB	67.58 MB	76.2% lower / 4.21x

These measurements are retained ETS deltas, externalized term sizes, and per-operation allocations—not total VM RSS.

Validation

• 90 focused snapshot/cache/tree/router tests, 0 failures, 1 existing excluded benchmark test.
• 25 additional rules/context-cache/gossip/cache-buster/BigQuery consumer tests, 0 failures.
• Formatter and MIX_ENV=test mix compile through project wrappers.
• MIX_ENV=test mix lint.all; only 32 existing design suggestions.
• MIX_ENV=test mix test.typings; 158 configured errors skipped, 10 existing unnecessary skips, task passed.
• Two same-environment cumulative memory runs against current main.
• Hosted Elixir CI and Playwright pass at 4754419c.
• Production-shaped concurrent throughput was not run locally.