Skip to content

Add zero-copy views of constant maps over a ByteBuffer - #3

Merged
lemire merged 1 commit into
mainfrom
constmap-views
Nov 4, 2026
Merged

lemire merged 1 commit into
mainfrom
constmap-views

Conversation

@lemire

@lemire lemire commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #2. Adds wrap(ByteBuffer) to ConstMap, VerifiedConstMap and PairedVerifiedConstMap. It queries a serialized map in place, without copying, from a heap, direct or memory-mapped buffer, as BinaryFuseFilter.wrap does for filters.

try (FileChannel ch = FileChannel.open(Path.of("built-by-go.vmap"), StandardOpenOption.READ)) {
  VerifiedConstMap view = VerifiedConstMap.wrap(ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size()));
  view.get("banana");
}

Compatibility with constmap, rsconstmap and fastconstmap

Views read exactly the files the other implementations write. There is no new format.

  • CMAP0001 (Go, Rust), CMAP0003 (fastconstmap), and VMAP0001 / PMAP0001 with or without fastconstmap's key count in the pad word.
  • No alignment requirement. Go/Rust CMAP0001 has a 28-byte header that leaves the array 4 bytes off an 8-byte boundary. fastconstmap's from_buffer refuses these files; here they wrap at any offset.
  • A view serializes to the bytes it wraps, unchanged, so the result stays readable by the other implementations. On-heap maps still write exactly what Go and Rust write.
  • Legacy fastconstmap 0.9 files and files of the wrong kind are refused with the same messages as load.

Tests (ConstMapViewTest):

  • Every file in the interop set (rust.*, fastconstmap.*) is wrapped at 16 offsets, in heap and direct buffers of both byte orders, and at an odd offset inside a memory-mapped file.
  • Each test checks all 200 interop keys, 1000 absent keys, that the buffer state is untouched, and that the view writes back the very bytes it wraps.
  • Bytes written by a view load back with load. A view of fastconstmap's VMAP converts with paired() to exactly Rust's PMAP.
  • Views agree with on-heap maps for sizes 0 to 30,000 on get, get(ByteBuffer), getMany, paired and serialization.
  • I checked that the suite catches a wrong read offset in a view: 9 failures.

Design

  • The three types become sealed interfaces, as BinaryFuseFilter is, with package-private on-heap (Heap*) and in-place (Buffer*) implementations. They are not in a release yet, and source call sites (build, load, get, ...) are unchanged.
  • wrap runs in constant time. It validates the magic, checks that the segment parameters describe the array length, and checks that the buffer is long enough. It does not verify the FNV-1a checksum, which would read the whole map; load still does. This matches BinaryFuseFilter.wrap.
  • Since the parameters are validated, a view never reads outside the map: on corrupt bytes it returns wrong values, never exceptions (tested).
  • ConstMaps.read is split into readHeader and verifyChecksum.

Performance

Measured with ConstMapBenchmark on an Intel Xeon Gold 6548N: JDK 25, one core pinned on NUMA node 1, 2 forks × 5 iterations. Numbers are ns per lookup of a present string key, in random order.

keys heap direct mapped
ConstMap.get 1M 38.9 41.3 44.5
10M 44.7 46.6 54.8
VerifiedConstMap.get 1M 43.4 47.5 54.9
10M 49.7 51.6 66.2
PairedVerifiedConstMap.get 1M 42.9 44.6 51.3
10M 47.8 48.2 59.1
ConstMap.getMany 1M 29.0 30.1 31.1
10M 30.3 31.0 35.9
VerifiedConstMap.getMany 1M 34.9 35.2 38.8
10M 35.8 37.1 49.4
PairedVerifiedConstMap.getMany 1M 31.8 31.8 33.9
10M 32.9 32.9 37.8
  • A direct buffer costs 0–4 ns per lookup over the heap.
  • A mapped file costs more, up to 10–16 ns at 10M keys for the verified map, which reads two arrays. getMany and the paired layout recover most of it.
  • Absolute times on this Xeon are about 2× those measured on an Apple M4 Max, and grow little from 1M to 10M keys, so hashing and reading the query strings dominate on this machine.

./mvnw package passes: 84 tests, javadoc doclint clean.

ConstMap, VerifiedConstMap and PairedVerifiedConstMap gain a static
wrap(ByteBuffer) that queries a serialized map in place, from a heap, direct
or memory-mapped buffer, as BinaryFuseFilter.wrap does for filters. The three
types become sealed interfaces with an on-heap and a buffer implementation;
call sites (build, load, get, ...) are unchanged.

Views read exactly the formats shared with constmap (Go), rsconstmap (Rust)
and fastconstmap (C/Python): CMAP0001, fastconstmap's CMAP0003, VMAP0001 and
PMAP0001 (with or without fastconstmap's key count). There is no alignment
requirement, so Go's 28-byte CMAP0001 header can be wrapped at any offset.
A view serializes to the bytes it wraps, unchanged.

wrap validates the header in constant time (magic, segment parameters
consistent with the array length, buffer length) but not the checksum; load
still verifies it. Views cannot read outside the map on corrupt input.

Tests wrap every file written by the other implementations at 16 offsets, in
heap and direct buffers of both byte orders and in memory-mapped files, and
compare views with on-heap maps for get, getMany, paired and serialization.
The JMH benchmark gains heap/direct/mapped storage.
@lemire
lemire merged commit 0a12806 into main Nov 4, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant