Repository navigation
Add zero-copy views of constant maps over a ByteBuffer - #3
Merged
Merged
Conversation
ConstMap, VerifiedConstMap and PairedVerifiedConstMap gain a static wrap(ByteBuffer) that queries a serialized map in place, from a heap, direct or memory-mapped buffer, as BinaryFuseFilter.wrap does for filters. The three types become sealed interfaces with an on-heap and a buffer implementation; call sites (build, load, get, ...) are unchanged. Views read exactly the formats shared with constmap (Go), rsconstmap (Rust) and fastconstmap (C/Python): CMAP0001, fastconstmap's CMAP0003, VMAP0001 and PMAP0001 (with or without fastconstmap's key count). There is no alignment requirement, so Go's 28-byte CMAP0001 header can be wrapped at any offset. A view serializes to the bytes it wraps, unchanged. wrap validates the header in constant time (magic, segment parameters consistent with the array length, buffer length) but not the checksum; load still verifies it. Views cannot read outside the map on corrupt input. Tests wrap every file written by the other implementations at 16 offsets, in heap and direct buffers of both byte orders and in memory-mapped files, and compare views with on-heap maps for get, getMany, paired and serialization. The JMH benchmark gains heap/direct/mapped storage.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2. Adds
wrap(ByteBuffer)toConstMap,VerifiedConstMapandPairedVerifiedConstMap. It queries a serialized map in place, without copying, from a heap, direct or memory-mapped buffer, asBinaryFuseFilter.wrapdoes for filters.Compatibility with constmap, rsconstmap and fastconstmap
Views read exactly the files the other implementations write. There is no new format.
CMAP0001(Go, Rust),CMAP0003(fastconstmap), andVMAP0001/PMAP0001with or without fastconstmap's key count in the pad word.CMAP0001has a 28-byte header that leaves the array 4 bytes off an 8-byte boundary. fastconstmap'sfrom_bufferrefuses these files; here they wrap at any offset.load.Tests (
ConstMapViewTest):rust.*,fastconstmap.*) is wrapped at 16 offsets, in heap and direct buffers of both byte orders, and at an odd offset inside a memory-mapped file.load. A view of fastconstmap's VMAP converts withpaired()to exactly Rust's PMAP.get,get(ByteBuffer),getMany,pairedand serialization.Design
BinaryFuseFilteris, with package-private on-heap (Heap*) and in-place (Buffer*) implementations. They are not in a release yet, and source call sites (build,load,get, ...) are unchanged.wrapruns in constant time. It validates the magic, checks that the segment parameters describe the array length, and checks that the buffer is long enough. It does not verify the FNV-1a checksum, which would read the whole map;loadstill does. This matchesBinaryFuseFilter.wrap.ConstMaps.readis split intoreadHeaderandverifyChecksum.Performance
Measured with
ConstMapBenchmarkon an Intel Xeon Gold 6548N: JDK 25, one core pinned on NUMA node 1, 2 forks × 5 iterations. Numbers are ns per lookup of a present string key, in random order.ConstMap.getVerifiedConstMap.getPairedVerifiedConstMap.getConstMap.getManyVerifiedConstMap.getManyPairedVerifiedConstMap.getManygetManyand the paired layout recover most of it../mvnw packagepasses: 84 tests, javadoc doclint clean.