From 5dbd59d252819132a8994a6bd01fab27a1b08ee0 Mon Sep 17 00:00:00 2001 From: Daniel Lemire Date: Tue, 3 Nov 2026 23:16:39 -0500 Subject: [PATCH] Add zero-copy views of constant maps over a ByteBuffer ConstMap, VerifiedConstMap and PairedVerifiedConstMap gain a static wrap(ByteBuffer) that queries a serialized map in place, from a heap, direct or memory-mapped buffer, as BinaryFuseFilter.wrap does for filters. The three types become sealed interfaces with an on-heap and a buffer implementation; call sites (build, load, get, ...) are unchanged. Views read exactly the formats shared with constmap (Go), rsconstmap (Rust) and fastconstmap (C/Python): CMAP0001, fastconstmap's CMAP0003, VMAP0001 and PMAP0001 (with or without fastconstmap's key count). There is no alignment requirement, so Go's 28-byte CMAP0001 header can be wrapped at any offset. A view serializes to the bytes it wraps, unchanged. wrap validates the header in constant time (magic, segment parameters consistent with the array length, buffer length) but not the checksum; load still verifies it. Views cannot read outside the map on corrupt input. Tests wrap every file written by the other implementations at 16 offsets, in heap and direct buffers of both byte orders and in memory-mapped files, and compare views with on-heap maps for get, getMany, paired and serialization. The JMH benchmark gains heap/direct/mapped storage. --- README.md | 37 ++- .../jbinfuse/bench/ConstMapBenchmark.java | 39 ++- .../fastfilter/jbinfuse/BufferConstMap.java | 90 +++++++ .../BufferPairedVerifiedConstMap.java | 110 ++++++++ .../jbinfuse/BufferVerifiedConstMap.java | 123 +++++++++ .../github/fastfilter/jbinfuse/ConstMap.java | 212 ++++++--------- .../github/fastfilter/jbinfuse/ConstMaps.java | 57 +++- .../fastfilter/jbinfuse/HeapConstMap.java | 98 +++++++ .../jbinfuse/HeapPairedVerifiedConstMap.java | 108 ++++++++ .../jbinfuse/HeapVerifiedConstMap.java | 123 +++++++++ .../jbinfuse/PairedVerifiedConstMap.java | 202 +++++--------- .../fastfilter/jbinfuse/VerifiedConstMap.java | 223 ++++++--------- .../fastfilter/jbinfuse/ConstMapViewTest.java | 255 ++++++++++++++++++ 13 files changed, 1264 insertions(+), 413 deletions(-) create mode 100644 src/main/java/io/github/fastfilter/jbinfuse/BufferConstMap.java create mode 100644 src/main/java/io/github/fastfilter/jbinfuse/BufferPairedVerifiedConstMap.java create mode 100644 src/main/java/io/github/fastfilter/jbinfuse/BufferVerifiedConstMap.java create mode 100644 src/main/java/io/github/fastfilter/jbinfuse/HeapConstMap.java create mode 100644 src/main/java/io/github/fastfilter/jbinfuse/HeapPairedVerifiedConstMap.java create mode 100644 src/main/java/io/github/fastfilter/jbinfuse/HeapVerifiedConstMap.java create mode 100644 src/test/java/io/github/fastfilter/jbinfuse/ConstMapViewTest.java diff --git a/README.md b/README.md index ee91593..4f7c6a6 100644 --- a/README.md +++ b/README.md @@ -208,11 +208,38 @@ VerifiedConstMap fromPython = VerifiedConstMap.load(Files.readAllBytes(Path.of(" - `load` validates the magic bytes, the segment parameters and the FNV-1a checksum, and throws `InvalidFilterException` otherwise, including for a file of another kind of map. -The constant maps are always loaded onto the heap; there is no in-place view as for filters. +### Querying a constant map in place + +As with filters, `wrap(ByteBuffer)` returns a view that queries a serialized map in place, +without copying it, from a heap, direct or memory-mapped buffer. It reads every format above, +written by any of the four implementations: + +```java +try (FileChannel ch = FileChannel.open(Path.of("built-by-go.vmap"), StandardOpenOption.READ)) { + MappedByteBuffer mapped = ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size()); + VerifiedConstMap view = VerifiedConstMap.wrap(mapped); // constant time, no copy + view.get("banana"); +} +``` + +- Views give the same answers as on-heap maps, including `getMany` and `paired()` (which + returns a new on-heap map). This is tested on the files written by rsconstmap and + fastconstmap, at every alignment, in heap, direct and memory-mapped buffers. +- There is no alignment requirement. A Go or Rust `ConstMap` file (`CMAP0001`), whose 28-byte + header leaves the array 4 bytes off an 8-byte boundary, can be wrapped too, at any offset; + fastconstmap's `from_buffer` refuses those files. +- `wrap` validates the header in constant time (magic bytes, segment parameters consistent + with the array length, and a buffer long enough for the whole map) but does not verify the + checksum, which would read the whole map; `load` does. On corrupt bytes, a view gives wrong + answers but never reads outside the map. +- A view serializes to the bytes it wraps, unchanged (so a `CMAP0003` file from fastconstmap + stays one, and keeps its key count). On-heap maps write the formats of Go and Rust. +- The view is valid as long as the buffer's bytes are not modified. The buffer's position, + limit and byte order are not changed. ### Constant map performance -JMH (`ConstMapBenchmark`) on an Apple M4 Max with JDK 27, ns per lookup of a present key, in +JMH (`ConstMapBenchmark`, on-heap maps) on an Apple M4 Max with JDK 27, ns per lookup of a present key, in random order, with keys like `key-123456-1e3779b9`: | | 1 million keys | 10 million keys | @@ -234,9 +261,9 @@ straight from their chars; other strings are first encoded to UTF-8, which costs | --- | --- | | `BinaryFuse8`, `BinaryFuse16` | `build(long[])`, `build(long[], int)`, `buildFromBytes(Collection)` | | `BinaryFuseFilter` (interface) | `contains(long)`, `contains(byte[])`, `contains(byte[], int, int)`, `contains(ByteBuffer)`, `fingerprintBits()`, `keyCount()`, `serializedSize()`, `serialize(ByteBuffer)`, `toByteArray()`; static `wrap(ByteBuffer)`, `load(ByteBuffer)`, `load(byte[])` | -| `ConstMap` | static `build(List, long[])`, `buildFromBytes(List, long[])`, `buildFromHashes(long[], long[])`, `load(ByteBuffer)`, `load(byte[])`; `get(String)`, `get(byte[])`, `get(byte[], int, int)`, `get(ByteBuffer)`, `getByHash(long)`, `getMany(List)`, `getMany(List, long[])`, `arrayLength()`, `serializedSize()`, `serialize(ByteBuffer)`, `toByteArray()` | -| `VerifiedConstMap` | as `ConstMap`, plus `containsKey(String)`, `paired()` and the constant `NOT_FOUND` | -| `PairedVerifiedConstMap` | as `ConstMap`, plus `containsKey(String)` | +| `ConstMap` (interface) | static `build(List, long[])`, `buildFromBytes(List, long[])`, `buildFromHashes(long[], long[])`, `load(ByteBuffer)`, `load(byte[])`, `wrap(ByteBuffer)`; `get(String)`, `get(byte[])`, `get(byte[], int, int)`, `get(ByteBuffer)`, `getByHash(long)`, `getMany(List)`, `getMany(List, long[])`, `arrayLength()`, `serializedSize()`, `serialize(ByteBuffer)`, `toByteArray()` | +| `VerifiedConstMap` (interface) | as `ConstMap`, plus `containsKey(String)`, `paired()` and the constant `NOT_FOUND` | +| `PairedVerifiedConstMap` (interface) | as `ConstMap`, plus `containsKey(String)` | | `XxHash64` | `hash(String)`, `hash(byte[])`, `hash(byte[], int, int)`, `hash(byte[], int, int, long)`, `hash(ByteBuffer)`, `hash(ByteBuffer, int, int, long)` | | `WyHash` | `hash(byte[])`, `hash(byte[], int, int)`, `hash(byte[], int, int, long)`, `hash(ByteBuffer)`, `hash(ByteBuffer, int, int, long)`, `hash64(long, long)` | | `InvalidFilterException` | thrown for corrupt or invalid serialized input | diff --git a/benchmarks/src/main/java/io/github/fastfilter/jbinfuse/bench/ConstMapBenchmark.java b/benchmarks/src/main/java/io/github/fastfilter/jbinfuse/bench/ConstMapBenchmark.java index 833e49f..2e184f4 100644 --- a/benchmarks/src/main/java/io/github/fastfilter/jbinfuse/bench/ConstMapBenchmark.java +++ b/benchmarks/src/main/java/io/github/fastfilter/jbinfuse/bench/ConstMapBenchmark.java @@ -3,6 +3,12 @@ import io.github.fastfilter.jbinfuse.ConstMap; import io.github.fastfilter.jbinfuse.PairedVerifiedConstMap; import io.github.fastfilter.jbinfuse.VerifiedConstMap; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.channels.FileChannel; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; import java.util.ArrayList; import java.util.List; import java.util.SplittableRandom; @@ -41,6 +47,13 @@ public class ConstMapBenchmark { @Param({"const", "verified", "paired"}) String kind; + /** + * Where the map lives: on the heap, or a view ({@code wrap}) of its serialized bytes in a + * direct buffer or a memory-mapped temporary file. + */ + @Param({"heap", "direct", "mapped"}) + String storage; + ConstMap cm; VerifiedConstMap vm; PairedVerifiedConstMap pm; @@ -48,7 +61,7 @@ public class ConstMapBenchmark { long[] out; @Setup - public void setup() { + public void setup() throws IOException { List keys = new ArrayList<>(size); long[] values = new long[size]; for (int i = 0; i < size; i++) { @@ -60,6 +73,30 @@ public void setup() { case "verified" -> vm = VerifiedConstMap.build(keys, values); default -> pm = PairedVerifiedConstMap.build(keys, values); } + if (!storage.equals("heap")) { + byte[] bytes = switch (kind) { + case "const" -> cm.toByteArray(); + case "verified" -> vm.toByteArray(); + default -> pm.toByteArray(); + }; + ByteBuffer buf; + if (storage.equals("direct")) { + buf = ByteBuffer.allocateDirect(bytes.length); + buf.put(bytes).flip(); + } else { + Path file = Files.createTempFile("constmap", ".bin"); + file.toFile().deleteOnExit(); + Files.write(file, bytes); + try (FileChannel ch = FileChannel.open(file, StandardOpenOption.READ)) { + buf = ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size()); + } + } + switch (kind) { + case "const" -> cm = ConstMap.wrap(buf); + case "verified" -> vm = VerifiedConstMap.wrap(buf); + default -> pm = PairedVerifiedConstMap.wrap(buf); + } + } SplittableRandom r = new SplittableRandom(size); queries = new ArrayList<>(QUERIES); for (int i = 0; i < QUERIES; i++) { diff --git a/src/main/java/io/github/fastfilter/jbinfuse/BufferConstMap.java b/src/main/java/io/github/fastfilter/jbinfuse/BufferConstMap.java new file mode 100644 index 0000000..0b014ee --- /dev/null +++ b/src/main/java/io/github/fastfilter/jbinfuse/BufferConstMap.java @@ -0,0 +1,90 @@ +package io.github.fastfilter.jbinfuse; + +import java.nio.ByteBuffer; +import java.util.Arrays; +import java.util.List; +import java.util.Objects; + +/** A view of a serialized {@link ConstMap}, queried in place. Created by {@link ConstMap#wrap}. */ +final class BufferConstMap implements ConstMap { + private final ByteBuffer serialized; + private final ByteBuffer data; + private final long seed; + private final int segmentLength; + private final int segmentLengthMask; + private final long segmentCountLength; + private final int arrayLength; + + private BufferConstMap(ByteBuffer buffer, ConstMaps.Header h) { + ConstMaps.Params p = h.params(); + this.serialized = ConstMaps.slice(buffer, h.start(), h.totalBytes()); + this.data = ConstMaps.slice(buffer, h.arrayStart(), 8 * p.arrayLength()); + this.seed = p.seed(); + this.segmentLength = p.segmentLength(); + this.segmentLengthMask = p.segmentLengthMask(); + this.segmentCountLength = p.segmentCountLength(); + this.arrayLength = p.arrayLength(); + } + + static BufferConstMap wrap(ByteBuffer buffer) { + return new BufferConstMap(buffer, ConstMaps.readHeader(buffer, ConstMaps.Kind.CONST, 1)); + } + + @Override + public long getByHash(long keyHash) { + if (arrayLength == 0) { + return 0; + } + long hash = ConstMaps.mixsplit(keyHash, seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + int h1 = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); + int h2 = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); + final ByteBuffer d = data; + return d.getLong(8 * h0) ^ d.getLong(8 * h1) ^ d.getLong(8 * h2); + } + + @Override + public void getMany(List keys, long[] dst) { + int n = keys.size(); + Objects.checkFromIndexSize(0, n, dst.length); + if (arrayLength == 0) { + Arrays.fill(dst, 0, n, 0); + return; + } + final ByteBuffer d = data; + final int[] h = new int[3 * ConstMaps.BLOCK]; + for (int start = 0; start < n; start += ConstMaps.BLOCK) { + int m = Math.min(ConstMaps.BLOCK, n - start); + for (int j = 0; j < m; j++) { + long hash = ConstMaps.mixsplit(XxHash64.hash(keys.get(start + j)), seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + h[3 * j] = 8 * h0; + h[3 * j + 1] = 8 * ((h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask)); + h[3 * j + 2] = 8 * ((h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask)); + } + for (int j = 0; j < m; j++) { + dst[start + j] = d.getLong(h[3 * j]) ^ d.getLong(h[3 * j + 1]) ^ d.getLong(h[3 * j + 2]); + } + } + } + + @Override + public int arrayLength() { + return arrayLength; + } + + @Override + public int serializedSize() { + return serialized.capacity(); + } + + @Override + public void serialize(ByteBuffer out) { + ConstMaps.copy(serialized, out); + } + + @Override + public String toString() { + return "ConstMapView{words=" + arrayLength + ", bytes=" + serializedSize() + "}"; + } +} diff --git a/src/main/java/io/github/fastfilter/jbinfuse/BufferPairedVerifiedConstMap.java b/src/main/java/io/github/fastfilter/jbinfuse/BufferPairedVerifiedConstMap.java new file mode 100644 index 0000000..6065f10 --- /dev/null +++ b/src/main/java/io/github/fastfilter/jbinfuse/BufferPairedVerifiedConstMap.java @@ -0,0 +1,110 @@ +package io.github.fastfilter.jbinfuse; + +import java.nio.ByteBuffer; +import java.util.Arrays; +import java.util.List; +import java.util.Objects; + +/** + * A view of a serialized {@link PairedVerifiedConstMap}, queried in place. Created by + * {@link PairedVerifiedConstMap#wrap}. + */ +final class BufferPairedVerifiedConstMap implements PairedVerifiedConstMap { + private final ByteBuffer serialized; + /** The value for position i at byte 16i, its check word at byte 16i + 8. */ + private final ByteBuffer slots; + private final long seed; + private final int segmentLength; + private final int segmentLengthMask; + private final long segmentCountLength; + private final int arrayLength; + + private BufferPairedVerifiedConstMap(ByteBuffer buffer, ConstMaps.Header h) { + ConstMaps.Params p = h.params(); + this.serialized = ConstMaps.slice(buffer, h.start(), h.totalBytes()); + this.slots = ConstMaps.slice(buffer, h.arrayStart(), 16 * p.arrayLength()); + this.seed = p.seed(); + this.segmentLength = p.segmentLength(); + this.segmentLengthMask = p.segmentLengthMask(); + this.segmentCountLength = p.segmentCountLength(); + this.arrayLength = p.arrayLength(); + } + + static BufferPairedVerifiedConstMap wrap(ByteBuffer buffer) { + return new BufferPairedVerifiedConstMap(buffer, + ConstMaps.readHeader(buffer, ConstMaps.Kind.PAIRED, 2)); + } + + @Override + public long getByHash(long keyHash) { + if (arrayLength == 0) { + return ConstMaps.NOT_FOUND; + } + long hash = ConstMaps.mixsplit(keyHash, seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + int h1 = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); + int h2 = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); + final ByteBuffer s = slots; + int a = 16 * h0; + int b = 16 * h1; + int c = 16 * h2; + if ((s.getLong(a + 8) ^ s.getLong(b + 8) ^ s.getLong(c + 8)) + != ConstMaps.fingerprint(hash)) { + return ConstMaps.NOT_FOUND; + } + return s.getLong(a) ^ s.getLong(b) ^ s.getLong(c); + } + + @Override + public void getMany(List keys, long[] dst) { + int n = keys.size(); + Objects.checkFromIndexSize(0, n, dst.length); + if (arrayLength == 0) { + Arrays.fill(dst, 0, n, ConstMaps.NOT_FOUND); + return; + } + final ByteBuffer s = slots; + final int[] h = new int[3 * ConstMaps.BLOCK]; + final long[] fp = new long[ConstMaps.BLOCK]; + for (int start = 0; start < n; start += ConstMaps.BLOCK) { + int m = Math.min(ConstMaps.BLOCK, n - start); + for (int j = 0; j < m; j++) { + long hash = ConstMaps.mixsplit(XxHash64.hash(keys.get(start + j)), seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + h[3 * j] = 16 * h0; + h[3 * j + 1] = 16 * ((h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask)); + h[3 * j + 2] = 16 * ((h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask)); + fp[j] = ConstMaps.fingerprint(hash); + } + for (int j = 0; j < m; j++) { + int a = h[3 * j]; + int b = h[3 * j + 1]; + int c = h[3 * j + 2]; + long v = s.getLong(a) ^ s.getLong(b) ^ s.getLong(c); + dst[start + j] = (s.getLong(a + 8) ^ s.getLong(b + 8) ^ s.getLong(c + 8)) == fp[j] + ? v : ConstMaps.NOT_FOUND; + } + } + } + + @Override + public int arrayLength() { + return arrayLength; + } + + @Override + public int serializedSize() { + return serialized.capacity(); + } + + @Override + public void serialize(ByteBuffer out) { + ConstMaps.copy(serialized, out); + } + + @Override + public String toString() { + return "PairedVerifiedConstMapView{positions=" + arrayLength + ", bytes=" + serializedSize() + + "}"; + } +} diff --git a/src/main/java/io/github/fastfilter/jbinfuse/BufferVerifiedConstMap.java b/src/main/java/io/github/fastfilter/jbinfuse/BufferVerifiedConstMap.java new file mode 100644 index 0000000..beb6dcd --- /dev/null +++ b/src/main/java/io/github/fastfilter/jbinfuse/BufferVerifiedConstMap.java @@ -0,0 +1,123 @@ +package io.github.fastfilter.jbinfuse; + +import java.nio.ByteBuffer; +import java.util.Arrays; +import java.util.List; +import java.util.Objects; + +/** + * A view of a serialized {@link VerifiedConstMap}, queried in place. Created by + * {@link VerifiedConstMap#wrap}. + */ +final class BufferVerifiedConstMap implements VerifiedConstMap { + private final ConstMaps.Params params; + private final ByteBuffer serialized; + private final ByteBuffer data; + private final ByteBuffer checks; + private final long seed; + private final int segmentLength; + private final int segmentLengthMask; + private final long segmentCountLength; + private final int arrayLength; + + private BufferVerifiedConstMap(ByteBuffer buffer, ConstMaps.Header h) { + ConstMaps.Params p = h.params(); + int n = p.arrayLength(); + this.params = p; + this.serialized = ConstMaps.slice(buffer, h.start(), h.totalBytes()); + this.data = ConstMaps.slice(buffer, h.arrayStart(), 8 * n); + this.checks = ConstMaps.slice(buffer, h.arrayStart() + 8 * n, 8 * n); + this.seed = p.seed(); + this.segmentLength = p.segmentLength(); + this.segmentLengthMask = p.segmentLengthMask(); + this.segmentCountLength = p.segmentCountLength(); + this.arrayLength = n; + } + + static BufferVerifiedConstMap wrap(ByteBuffer buffer) { + return new BufferVerifiedConstMap(buffer, + ConstMaps.readHeader(buffer, ConstMaps.Kind.VERIFIED, 2)); + } + + @Override + public long getByHash(long keyHash) { + if (arrayLength == 0) { + return NOT_FOUND; + } + long hash = ConstMaps.mixsplit(keyHash, seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + int h1 = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); + int h2 = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); + final ByteBuffer c = checks; + int a = 8 * h0; + int b = 8 * h1; + int e = 8 * h2; + if ((c.getLong(a) ^ c.getLong(b) ^ c.getLong(e)) != ConstMaps.fingerprint(hash)) { + return NOT_FOUND; + } + final ByteBuffer d = data; + return d.getLong(a) ^ d.getLong(b) ^ d.getLong(e); + } + + @Override + public void getMany(List keys, long[] dst) { + int n = keys.size(); + Objects.checkFromIndexSize(0, n, dst.length); + if (arrayLength == 0) { + Arrays.fill(dst, 0, n, NOT_FOUND); + return; + } + final ByteBuffer d = data; + final ByteBuffer c = checks; + final int[] h = new int[3 * ConstMaps.BLOCK]; + final long[] fp = new long[ConstMaps.BLOCK]; + for (int start = 0; start < n; start += ConstMaps.BLOCK) { + int m = Math.min(ConstMaps.BLOCK, n - start); + for (int j = 0; j < m; j++) { + long hash = ConstMaps.mixsplit(XxHash64.hash(keys.get(start + j)), seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + h[3 * j] = 8 * h0; + h[3 * j + 1] = 8 * ((h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask)); + h[3 * j + 2] = 8 * ((h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask)); + fp[j] = ConstMaps.fingerprint(hash); + } + for (int j = 0; j < m; j++) { + int a = h[3 * j]; + int b = h[3 * j + 1]; + int e = h[3 * j + 2]; + long v = d.getLong(a) ^ d.getLong(b) ^ d.getLong(e); + dst[start + j] = (c.getLong(a) ^ c.getLong(b) ^ c.getLong(e)) == fp[j] ? v : NOT_FOUND; + } + } + } + + @Override + public PairedVerifiedConstMap paired() { + long[] slots = new long[2 * arrayLength]; + for (int i = 0; i < arrayLength; i++) { + slots[2 * i] = data.getLong(8 * i); + slots[2 * i + 1] = checks.getLong(8 * i); + } + return HeapPairedVerifiedConstMap.of(params, slots); + } + + @Override + public int arrayLength() { + return arrayLength; + } + + @Override + public int serializedSize() { + return serialized.capacity(); + } + + @Override + public void serialize(ByteBuffer out) { + ConstMaps.copy(serialized, out); + } + + @Override + public String toString() { + return "VerifiedConstMapView{positions=" + arrayLength + ", bytes=" + serializedSize() + "}"; + } +} diff --git a/src/main/java/io/github/fastfilter/jbinfuse/ConstMap.java b/src/main/java/io/github/fastfilter/jbinfuse/ConstMap.java index cf0f733..1d1337d 100644 --- a/src/main/java/io/github/fastfilter/jbinfuse/ConstMap.java +++ b/src/main/java/io/github/fastfilter/jbinfuse/ConstMap.java @@ -2,7 +2,6 @@ import java.nio.ByteBuffer; import java.util.List; -import java.util.Objects; /** * An immutable map from keys (strings or byte sequences) to 64-bit values, in about 9 bytes @@ -17,31 +16,22 @@ * (Go). Its serialized form is shared with constmap, with * rsconstmap (Rust) and with * fastconstmap (C and Python): a map - * built by any of them loads in the others. For the same keys and values, this class writes - * exactly the bytes that the Go and Rust implementations write. + * built by any of them loads in the others. For the same keys and values, the maps built here + * serialize to exactly the bytes that the Go and Rust implementations write. + * + *

There are two implementations with identical answers: on-heap maps, from the + * {@code build} methods and {@link #load(ByteBuffer)}, which are the fastest; and views from + * {@link #wrap(ByteBuffer)}, which query a serialized map in place in a heap, direct or + * memory-mapped buffer, without copying it. * *

String keys are hashed through their UTF-8 encoding, so {@code get("abc")} and * {@code get("abc".getBytes(UTF_8))} are the same lookup. Values are 64-bit words; read them as * unsigned if you need values above {@link Long#MAX_VALUE}. * - *

Instances are immutable and thread-safe. + *

Instances are immutable and thread-safe (a view, as long as its buffer's bytes are not + * modified). */ -public final class ConstMap { - private final ConstMaps.Params params; - private final long seed; - private final int segmentLength; - private final int segmentLengthMask; - private final long segmentCountLength; - private final long[] data; - - private ConstMap(ConstMaps.Params params, long[] data) { - this.params = params; - this.seed = params.seed(); - this.segmentLength = params.segmentLength(); - this.segmentLengthMask = params.segmentLengthMask(); - this.segmentCountLength = params.segmentCountLength(); - this.data = data; - } +public sealed interface ConstMap permits HeapConstMap, BufferConstMap { /** * Builds a map from string keys and their values: {@code keys.get(i)} maps to @@ -49,13 +39,13 @@ private ConstMap(ConstMaps.Params params, long[] data) { * * @param keys the keys, distinct * @param values the values, one per key - * @return the map + * @return the on-heap map * @throws IllegalArgumentException if the lengths differ, if two keys are equal (detected * with high probability), or if there are too many keys for one map */ - public static ConstMap build(List keys, long[] values) { + static ConstMap build(List keys, long[] values) { ConstMaps.checkLengths(keys.size(), values.length); - return buildFromHashes(ConstMaps.hashStrings(keys), values); + return HeapConstMap.build(ConstMaps.hashStrings(keys), values); } /** @@ -64,13 +54,13 @@ public static ConstMap build(List keys, long[] values) { * * @param keys the keys, distinct * @param values the values, one per key - * @return the map + * @return the on-heap map * @throws IllegalArgumentException if the lengths differ, if two keys are equal (detected * with high probability), or if there are too many keys for one map */ - public static ConstMap buildFromBytes(List keys, long[] values) { + static ConstMap buildFromBytes(List keys, long[] values) { ConstMaps.checkLengths(keys.size(), values.length); - return buildFromHashes(ConstMaps.hashBytes(keys), values); + return HeapConstMap.build(ConstMaps.hashBytes(keys), values); } /** @@ -79,19 +69,62 @@ public static ConstMap buildFromBytes(List keys, long[] values) { * * @param keyHashes the XXH64 hashes (seed 0) of the keys, distinct * @param values the values, one per key - * @return the map + * @return the on-heap map * @throws IllegalArgumentException if the lengths differ, if two hashes are equal (detected * with high probability), or if there are too many keys for one map */ - public static ConstMap buildFromHashes(long[] keyHashes, long[] values) { + static ConstMap buildFromHashes(long[] keyHashes, long[] values) { ConstMaps.checkLengths(keyHashes.length, values.length); - if (keyHashes.length == 0) { - return new ConstMap(ConstMaps.Params.EMPTY, new long[0]); - } - ConstMaps.Peeled p = ConstMaps.peel(keyHashes); - long[] data = new long[p.params().arrayLength()]; - ConstMaps.assign(p, values, data, null); - return new ConstMap(p.params(), data); + return HeapConstMap.build(keyHashes, values); + } + + /** + * Reads a serialized map starting at the buffer's position into a new on-heap map, after + * validating its header and checksum. Accepts both the {@code CMAP0001} format (Go, Rust, + * this library) and fastconstmap's {@code CMAP0003}, which adds a key count that is + * ignored. The buffer's position is not changed. + * + * @param buffer a buffer holding a serialized map at its position + * @return the on-heap map + * @throws InvalidFilterException if the input is not a valid {@code ConstMap} (including a + * {@code VerifiedConstMap} or {@code PairedVerifiedConstMap}, or a file from + * fastconstmap 0.9 or earlier, which used another hash), or is corrupt + */ + static ConstMap load(ByteBuffer buffer) { + return HeapConstMap.read(buffer); + } + + /** + * Reads a serialized map, as {@link #load(ByteBuffer)} does. + * + * @param bytes the serialized map + * @return the on-heap map + * @throws InvalidFilterException if the input is invalid or corrupt + */ + static ConstMap load(byte[] bytes) { + return load(ByteBuffer.wrap(bytes)); + } + + /** + * Returns a map that queries the serialized map starting at the buffer's position directly, + * without copying it. The buffer may be on the heap, direct or a + * {@link java.nio.MappedByteBuffer}; its position, limit and byte order are not changed. It + * reads the same formats as {@link #load(ByteBuffer)} ({@code CMAP0001} and + * {@code CMAP0003}), at any offset: there is no alignment requirement. + * + *

This method runs in constant time: it validates the header (magic bytes, segment + * parameters consistent with the array length, and a buffer long enough for the whole map) + * but does not verify the checksum, which would read the whole map; use + * {@link #load(ByteBuffer)} for that. A view answers like the on-heap map as long as the + * buffer's bytes are not modified; on corrupt bytes it returns wrong values but never reads + * outside the map. + * + * @param buffer a buffer holding a serialized map at its position + * @return a view backed by the buffer + * @throws InvalidFilterException if the header is invalid or the buffer is too short + */ + static ConstMap wrap(ByteBuffer buffer) { + return BufferConstMap.wrap(buffer); } /** @@ -101,16 +134,7 @@ public static ConstMap buildFromHashes(long[] keyHashes, long[] values) { * @param keyHash the XXH64 hash of the key * @return the value if the key is in the map, an arbitrary value otherwise */ - public long getByHash(long keyHash) { - if (data.length == 0) { - return 0; - } - long hash = ConstMaps.mixsplit(keyHash, seed); - int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); - int h1 = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); - int h2 = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); - return data[h0] ^ data[h1] ^ data[h2]; - } + long getByHash(long keyHash); /** * Returns the value of a string key (hashed through its UTF-8 encoding). @@ -118,7 +142,7 @@ public long getByHash(long keyHash) { * @param key the key * @return the value if the key is in the map, an arbitrary value otherwise */ - public long get(String key) { + default long get(String key) { return getByHash(XxHash64.hash(key)); } @@ -128,7 +152,7 @@ public long get(String key) { * @param key the key * @return the value if the key is in the map, an arbitrary value otherwise */ - public long get(byte[] key) { + default long get(byte[] key) { return getByHash(XxHash64.hash(key)); } @@ -141,7 +165,7 @@ public long get(byte[] key) { * @return the value if the key is in the map, an arbitrary value otherwise * @throws IndexOutOfBoundsException if the range is not within the array */ - public long get(byte[] key, int off, int len) { + default long get(byte[] key, int off, int len) { return getByHash(XxHash64.hash(key, off, len)); } @@ -152,7 +176,7 @@ public long get(byte[] key, int off, int len) { * @param key the key * @return the value if the key is in the map, an arbitrary value otherwise */ - public long get(ByteBuffer key) { + default long get(ByteBuffer key) { return getByHash(XxHash64.hash(key)); } @@ -166,32 +190,7 @@ public long get(ByteBuffer key) { * @param dst where the values go; only the first {@code keys.size()} entries are written * @throws IndexOutOfBoundsException if {@code dst} is shorter than {@code keys} */ - public void getMany(List keys, long[] dst) { - int n = keys.size(); - Objects.checkFromIndexSize(0, n, dst.length); - if (data.length == 0) { - java.util.Arrays.fill(dst, 0, n, 0); - return; - } - final long[] d = data; - final int[] h = new int[3 * BLOCK]; - for (int start = 0; start < n; start += BLOCK) { - int m = Math.min(BLOCK, n - start); - for (int j = 0; j < m; j++) { - long hash = ConstMaps.mixsplit(XxHash64.hash(keys.get(start + j)), seed); - int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); - h[3 * j] = h0; - h[3 * j + 1] = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); - h[3 * j + 2] = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); - } - for (int j = 0; j < m; j++) { - dst[start + j] = d[h[3 * j]] ^ d[h[3 * j + 1]] ^ d[h[3 * j + 2]]; - } - } - } - - /** Keys per block in {@link #getMany}. */ - static final int BLOCK = 8; + void getMany(List keys, long[] dst); /** * Looks up many string keys at once, as {@link #getMany(List, long[])} does, into a new @@ -200,7 +199,7 @@ public void getMany(List keys, long[] dst) { * @param keys the keys * @return the values, one per key */ - public long[] getMany(List keys) { + default long[] getMany(List keys) { long[] dst = new long[keys.size()]; getMany(keys, dst); return dst; @@ -212,76 +211,39 @@ public long[] getMany(List keys) { * * @return the array length */ - public int arrayLength() { - return data.length; - } + int arrayLength(); /** - * Returns the number of bytes written by {@link #serialize(ByteBuffer)}: a 28-byte header, - * the array, and an 8-byte checksum. + * Returns the number of bytes written by {@link #serialize(ByteBuffer)}: for an on-heap map, + * a 28-byte header, the array and an 8-byte checksum; for a view, the size of the + * serialized map it wraps. * * @return the serialized size in bytes */ - public int serializedSize() { - return ConstMaps.CMAP_HEADER + 8 * data.length + 8; - } + int serializedSize(); /** - * Writes the map at the buffer's current position, in the {@code CMAP0001} format shared - * with the Go, Rust and C implementations, and advances the position by - * {@link #serializedSize()}. The buffer's byte order is ignored and left unchanged. + * Writes the map at the buffer's current position and advances the position by + * {@link #serializedSize()}. An on-heap map writes the {@code CMAP0001} format shared with + * the Go, Rust and C implementations; a view writes the bytes it wraps, unchanged (so a + * {@code CMAP0003} file from fastconstmap stays one). The buffer's byte order is ignored + * and left unchanged. * * @param out the destination * @throws java.nio.BufferOverflowException if fewer than {@link #serializedSize()} bytes * remain * @throws java.nio.ReadOnlyBufferException if the buffer is read-only */ - public void serialize(ByteBuffer out) { - ConstMaps.write(out, ConstMaps.MAGIC_CMAP, params, data.length, false, data); - } + void serialize(ByteBuffer out); /** * Returns the serialized form of the map, as written by {@link #serialize(ByteBuffer)}. * * @return a new array of {@link #serializedSize()} bytes */ - public byte[] toByteArray() { + default byte[] toByteArray() { byte[] bytes = new byte[serializedSize()]; serialize(ByteBuffer.wrap(bytes)); return bytes; } - - /** - * Reads a serialized map starting at the buffer's position into a new on-heap map, after - * validating its header and checksum. Accepts both the {@code CMAP0001} format (Go, Rust, - * this class) and fastconstmap's {@code CMAP0003}, which adds a key count that is ignored. - * The buffer's position is not changed. - * - * @param buffer a buffer holding a serialized map at its position - * @return the map - * @throws InvalidFilterException if the input is not a valid {@code ConstMap} (including a - * {@code VerifiedConstMap} or {@code PairedVerifiedConstMap}, or a file from - * fastconstmap 0.9 or earlier, which used another hash), or is corrupt - */ - public static ConstMap load(ByteBuffer buffer) { - ConstMaps.Header h = ConstMaps.read(buffer, ConstMaps.Kind.CONST, 1); - int n = h.params().arrayLength(); - return new ConstMap(h.params(), ConstMaps.readWords(buffer, h.start(), n)); - } - - /** - * Reads a serialized map, as {@link #load(ByteBuffer)} does. - * - * @param bytes the serialized map - * @return the map - * @throws InvalidFilterException if the input is invalid or corrupt - */ - public static ConstMap load(byte[] bytes) { - return load(ByteBuffer.wrap(bytes)); - } - - @Override - public String toString() { - return "ConstMap{words=" + data.length + ", bytes=" + serializedSize() + "}"; - } } diff --git a/src/main/java/io/github/fastfilter/jbinfuse/ConstMaps.java b/src/main/java/io/github/fastfilter/jbinfuse/ConstMaps.java index 096ab33..7be5e7a 100644 --- a/src/main/java/io/github/fastfilter/jbinfuse/ConstMaps.java +++ b/src/main/java/io/github/fastfilter/jbinfuse/ConstMaps.java @@ -28,6 +28,9 @@ private ConstMaps() {} static final int MAX_SEGMENT_LENGTH = 262144; static final int MAX_ITERATIONS = 100; + /** Keys per block in the batched lookups ({@code getMany}). */ + static final int BLOCK = 8; + /** * Largest array length we build or load: a paired map holds two words per position in one * Java array. @@ -460,14 +463,33 @@ static void write(ByteBuffer out, long magic, Params p, int count, boolean pad, out.position(w.position()); } - /** A parsed header: the parameters, the header size and where the arrays start. */ - record Header(Params params, int headerSize, int start) {} + /** + * A parsed header: the parameters, the absolute index of the first byte of the serialized + * map, the header size (so the arrays start at {@code start + headerSize}), and the total + * size including the checksum. + */ + record Header(Params params, int start, int headerSize, int totalBytes) { + int arrayStart() { + return start + headerSize; + } + } /** * Reads and validates the header of a map of the given kind at the buffer's position, and * verifies the trailing checksum over {@code wordsPerPosition} words per array position. */ static Header read(ByteBuffer buffer, Kind kind, int wordsPerPosition) { + Header h = readHeader(buffer, kind, wordsPerPosition); + verifyChecksum(buffer, h); + return h; + } + + /** + * Reads and validates the header of a map of the given kind at the buffer's position, in + * constant time: magic bytes, segment parameters consistent with the array length, and a + * buffer long enough for the whole map. Does not verify the checksum. + */ + static Header readHeader(ByteBuffer buffer, Kind kind, int wordsPerPosition) { ByteBuffer b = buffer.duplicate().order(ByteOrder.LITTLE_ENDIAN); final int start = b.position(); if (b.remaining() < 8) { @@ -494,7 +516,8 @@ static Header read(ByteBuffer buffer, Kind kind, int wordsPerPosition) { } if (found != kind) { throw new InvalidFilterException("this is a " + found.className + ", not a " - + kind.className + "; load it with " + found.className + ".load"); + + kind.className + "; use " + found.className + ".load or " + found.className + + ".wrap"); } final int headerSize = (kind == Kind.CONST && !counted) ? CMAP_HEADER : VMAP_HEADER; if (b.remaining() < headerSize) { @@ -531,9 +554,15 @@ static Header read(ByteBuffer buffer, Kind kind, int wordsPerPosition) { } params = new Params(seed, segmentLength, segmentCount, (int) count); } + return new Header(params, start, headerSize, (int) total); + } + + /** Verifies the trailing FNV-1a checksum of a map whose header was read by readHeader. */ + static void verifyChecksum(ByteBuffer buffer, Header header) { + ByteBuffer b = buffer.duplicate().order(ByteOrder.LITTLE_ENDIAN); long h = FNV_OFFSET; - int end = (int) (start + total - 8); - int i = start; + int end = header.start() + header.totalBytes() - 8; + int i = header.start(); for (; i + 8 <= end; i += 8) { h = fnv(h, b.getLong(i)); } @@ -545,7 +574,23 @@ static Header read(ByteBuffer buffer, Kind kind, int wordsPerPosition) { throw new InvalidFilterException(String.format( "checksum mismatch (stored %016x, computed %016x)", stored, h)); } - return new Header(params, headerSize, start + headerSize); + } + + /** + * Writes the serialized bytes a view wraps, unchanged, at {@code out}'s position, and + * advances the position. + */ + static void copy(ByteBuffer serialized, ByteBuffer out) { + writer(out, serialized.capacity()); + out.put(serialized.duplicate().clear()); + } + + /** + * A little-endian slice of {@code bytes} bytes of the buffer from absolute index + * {@code from}, independent of the buffer's position, limit and byte order. + */ + static ByteBuffer slice(ByteBuffer buffer, int from, int bytes) { + return buffer.slice(from, bytes).order(ByteOrder.LITTLE_ENDIAN); } /** Reads {@code n} little-endian words starting at absolute index {@code from}. */ diff --git a/src/main/java/io/github/fastfilter/jbinfuse/HeapConstMap.java b/src/main/java/io/github/fastfilter/jbinfuse/HeapConstMap.java new file mode 100644 index 0000000..0e92617 --- /dev/null +++ b/src/main/java/io/github/fastfilter/jbinfuse/HeapConstMap.java @@ -0,0 +1,98 @@ +package io.github.fastfilter.jbinfuse; + +import java.nio.ByteBuffer; +import java.util.Arrays; +import java.util.List; +import java.util.Objects; + +/** An on-heap {@link ConstMap}. */ +final class HeapConstMap implements ConstMap { + private final ConstMaps.Params params; + private final long seed; + private final int segmentLength; + private final int segmentLengthMask; + private final long segmentCountLength; + private final long[] data; + + private HeapConstMap(ConstMaps.Params params, long[] data) { + this.params = params; + this.seed = params.seed(); + this.segmentLength = params.segmentLength(); + this.segmentLengthMask = params.segmentLengthMask(); + this.segmentCountLength = params.segmentCountLength(); + this.data = data; + } + + static HeapConstMap build(long[] keyHashes, long[] values) { + if (keyHashes.length == 0) { + return new HeapConstMap(ConstMaps.Params.EMPTY, new long[0]); + } + ConstMaps.Peeled p = ConstMaps.peel(keyHashes); + long[] data = new long[p.params().arrayLength()]; + ConstMaps.assign(p, values, data, null); + return new HeapConstMap(p.params(), data); + } + + static HeapConstMap read(ByteBuffer buffer) { + ConstMaps.Header h = ConstMaps.read(buffer, ConstMaps.Kind.CONST, 1); + int n = h.params().arrayLength(); + return new HeapConstMap(h.params(), ConstMaps.readWords(buffer, h.arrayStart(), n)); + } + + @Override + public long getByHash(long keyHash) { + if (data.length == 0) { + return 0; + } + long hash = ConstMaps.mixsplit(keyHash, seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + int h1 = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); + int h2 = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); + return data[h0] ^ data[h1] ^ data[h2]; + } + + @Override + public void getMany(List keys, long[] dst) { + int n = keys.size(); + Objects.checkFromIndexSize(0, n, dst.length); + if (data.length == 0) { + Arrays.fill(dst, 0, n, 0); + return; + } + final long[] d = data; + final int[] h = new int[3 * ConstMaps.BLOCK]; + for (int start = 0; start < n; start += ConstMaps.BLOCK) { + int m = Math.min(ConstMaps.BLOCK, n - start); + for (int j = 0; j < m; j++) { + long hash = ConstMaps.mixsplit(XxHash64.hash(keys.get(start + j)), seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + h[3 * j] = h0; + h[3 * j + 1] = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); + h[3 * j + 2] = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); + } + for (int j = 0; j < m; j++) { + dst[start + j] = d[h[3 * j]] ^ d[h[3 * j + 1]] ^ d[h[3 * j + 2]]; + } + } + } + + @Override + public int arrayLength() { + return data.length; + } + + @Override + public int serializedSize() { + return ConstMaps.CMAP_HEADER + 8 * data.length + 8; + } + + @Override + public void serialize(ByteBuffer out) { + ConstMaps.write(out, ConstMaps.MAGIC_CMAP, params, data.length, false, data); + } + + @Override + public String toString() { + return "ConstMap{words=" + data.length + ", bytes=" + serializedSize() + "}"; + } +} diff --git a/src/main/java/io/github/fastfilter/jbinfuse/HeapPairedVerifiedConstMap.java b/src/main/java/io/github/fastfilter/jbinfuse/HeapPairedVerifiedConstMap.java new file mode 100644 index 0000000..41f00aa --- /dev/null +++ b/src/main/java/io/github/fastfilter/jbinfuse/HeapPairedVerifiedConstMap.java @@ -0,0 +1,108 @@ +package io.github.fastfilter.jbinfuse; + +import java.nio.ByteBuffer; +import java.util.Arrays; +import java.util.List; +import java.util.Objects; + +/** An on-heap {@link PairedVerifiedConstMap}. */ +final class HeapPairedVerifiedConstMap implements PairedVerifiedConstMap { + private final ConstMaps.Params params; + private final long seed; + private final int segmentLength; + private final int segmentLengthMask; + private final long segmentCountLength; + /** The value for position i at slots[2i], its check word at slots[2i+1]. */ + private final long[] slots; + + private HeapPairedVerifiedConstMap(ConstMaps.Params params, long[] slots) { + this.params = params; + this.seed = params.seed(); + this.segmentLength = params.segmentLength(); + this.segmentLengthMask = params.segmentLengthMask(); + this.segmentCountLength = params.segmentCountLength(); + this.slots = slots; + } + + static HeapPairedVerifiedConstMap of(ConstMaps.Params params, long[] slots) { + return new HeapPairedVerifiedConstMap(params, slots); + } + + static HeapPairedVerifiedConstMap read(ByteBuffer buffer) { + ConstMaps.Header h = ConstMaps.read(buffer, ConstMaps.Kind.PAIRED, 2); + int n = h.params().arrayLength(); + return new HeapPairedVerifiedConstMap(h.params(), + ConstMaps.readWords(buffer, h.arrayStart(), 2 * n)); + } + + @Override + public long getByHash(long keyHash) { + if (slots.length == 0) { + return ConstMaps.NOT_FOUND; + } + long hash = ConstMaps.mixsplit(keyHash, seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + int h1 = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); + int h2 = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); + final long[] s = slots; + int i0 = 2 * h0; + int i1 = 2 * h1; + int i2 = 2 * h2; + if ((s[i0 + 1] ^ s[i1 + 1] ^ s[i2 + 1]) != ConstMaps.fingerprint(hash)) { + return ConstMaps.NOT_FOUND; + } + return s[i0] ^ s[i1] ^ s[i2]; + } + + @Override + public void getMany(List keys, long[] dst) { + int n = keys.size(); + Objects.checkFromIndexSize(0, n, dst.length); + if (slots.length == 0) { + Arrays.fill(dst, 0, n, ConstMaps.NOT_FOUND); + return; + } + final long[] s = slots; + final int[] h = new int[3 * ConstMaps.BLOCK]; + final long[] fp = new long[ConstMaps.BLOCK]; + for (int start = 0; start < n; start += ConstMaps.BLOCK) { + int m = Math.min(ConstMaps.BLOCK, n - start); + for (int j = 0; j < m; j++) { + long hash = ConstMaps.mixsplit(XxHash64.hash(keys.get(start + j)), seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + h[3 * j] = 2 * h0; + h[3 * j + 1] = 2 * ((h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask)); + h[3 * j + 2] = 2 * ((h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask)); + fp[j] = ConstMaps.fingerprint(hash); + } + for (int j = 0; j < m; j++) { + int a = h[3 * j]; + int b = h[3 * j + 1]; + int c = h[3 * j + 2]; + long v = s[a] ^ s[b] ^ s[c]; + dst[start + j] = (s[a + 1] ^ s[b + 1] ^ s[c + 1]) == fp[j] ? v : ConstMaps.NOT_FOUND; + } + } + } + + @Override + public int arrayLength() { + return slots.length / 2; + } + + @Override + public int serializedSize() { + return ConstMaps.VMAP_HEADER + 8 * slots.length + 8; + } + + @Override + public void serialize(ByteBuffer out) { + ConstMaps.write(out, ConstMaps.MAGIC_PMAP, params, slots.length / 2, true, slots); + } + + @Override + public String toString() { + return "PairedVerifiedConstMap{positions=" + arrayLength() + ", bytes=" + serializedSize() + + "}"; + } +} diff --git a/src/main/java/io/github/fastfilter/jbinfuse/HeapVerifiedConstMap.java b/src/main/java/io/github/fastfilter/jbinfuse/HeapVerifiedConstMap.java new file mode 100644 index 0000000..d516f48 --- /dev/null +++ b/src/main/java/io/github/fastfilter/jbinfuse/HeapVerifiedConstMap.java @@ -0,0 +1,123 @@ +package io.github.fastfilter.jbinfuse; + +import java.nio.ByteBuffer; +import java.util.Arrays; +import java.util.List; +import java.util.Objects; + +/** An on-heap {@link VerifiedConstMap}. */ +final class HeapVerifiedConstMap implements VerifiedConstMap { + private final ConstMaps.Params params; + private final long seed; + private final int segmentLength; + private final int segmentLengthMask; + private final long segmentCountLength; + private final long[] data; + private final long[] checks; + + private HeapVerifiedConstMap(ConstMaps.Params params, long[] data, long[] checks) { + this.params = params; + this.seed = params.seed(); + this.segmentLength = params.segmentLength(); + this.segmentLengthMask = params.segmentLengthMask(); + this.segmentCountLength = params.segmentCountLength(); + this.data = data; + this.checks = checks; + } + + static HeapVerifiedConstMap build(long[] keyHashes, long[] values) { + if (keyHashes.length == 0) { + return new HeapVerifiedConstMap(ConstMaps.Params.EMPTY, new long[0], new long[0]); + } + ConstMaps.Peeled p = ConstMaps.peel(keyHashes); + int n = p.params().arrayLength(); + long[] data = new long[n]; + long[] checks = new long[n]; + ConstMaps.assign(p, values, data, checks); + return new HeapVerifiedConstMap(p.params(), data, checks); + } + + static HeapVerifiedConstMap read(ByteBuffer buffer) { + ConstMaps.Header h = ConstMaps.read(buffer, ConstMaps.Kind.VERIFIED, 2); + int n = h.params().arrayLength(); + return new HeapVerifiedConstMap(h.params(), ConstMaps.readWords(buffer, h.arrayStart(), n), + ConstMaps.readWords(buffer, h.arrayStart() + 8 * n, n)); + } + + @Override + public long getByHash(long keyHash) { + if (data.length == 0) { + return NOT_FOUND; + } + long hash = ConstMaps.mixsplit(keyHash, seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + int h1 = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); + int h2 = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); + if ((checks[h0] ^ checks[h1] ^ checks[h2]) != ConstMaps.fingerprint(hash)) { + return NOT_FOUND; + } + return data[h0] ^ data[h1] ^ data[h2]; + } + + @Override + public void getMany(List keys, long[] dst) { + int n = keys.size(); + Objects.checkFromIndexSize(0, n, dst.length); + if (data.length == 0) { + Arrays.fill(dst, 0, n, NOT_FOUND); + return; + } + final long[] d = data; + final long[] c = checks; + final int[] h = new int[3 * ConstMaps.BLOCK]; + final long[] fp = new long[ConstMaps.BLOCK]; + for (int start = 0; start < n; start += ConstMaps.BLOCK) { + int m = Math.min(ConstMaps.BLOCK, n - start); + for (int j = 0; j < m; j++) { + long hash = ConstMaps.mixsplit(XxHash64.hash(keys.get(start + j)), seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + h[3 * j] = h0; + h[3 * j + 1] = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); + h[3 * j + 2] = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); + fp[j] = ConstMaps.fingerprint(hash); + } + for (int j = 0; j < m; j++) { + int a = h[3 * j]; + int b = h[3 * j + 1]; + int e = h[3 * j + 2]; + long v = d[a] ^ d[b] ^ d[e]; + dst[start + j] = (c[a] ^ c[b] ^ c[e]) == fp[j] ? v : NOT_FOUND; + } + } + } + + @Override + public PairedVerifiedConstMap paired() { + long[] slots = new long[2 * data.length]; + for (int i = 0; i < data.length; i++) { + slots[2 * i] = data[i]; + slots[2 * i + 1] = checks[i]; + } + return HeapPairedVerifiedConstMap.of(params, slots); + } + + @Override + public int arrayLength() { + return data.length; + } + + @Override + public int serializedSize() { + return ConstMaps.VMAP_HEADER + 16 * data.length + 8; + } + + @Override + public void serialize(ByteBuffer out) { + ConstMaps.write(out, ConstMaps.MAGIC_VMAP, params, data.length, true, data, checks); + } + + @Override + public String toString() { + return "VerifiedConstMap{positions=" + data.length + ", bytes=" + serializedSize() + "}"; + } +} diff --git a/src/main/java/io/github/fastfilter/jbinfuse/PairedVerifiedConstMap.java b/src/main/java/io/github/fastfilter/jbinfuse/PairedVerifiedConstMap.java index 0cf170a..5664f23 100644 --- a/src/main/java/io/github/fastfilter/jbinfuse/PairedVerifiedConstMap.java +++ b/src/main/java/io/github/fastfilter/jbinfuse/PairedVerifiedConstMap.java @@ -1,9 +1,7 @@ package io.github.fastfilter.jbinfuse; import java.nio.ByteBuffer; -import java.util.Arrays; import java.util.List; -import java.util.Objects; /** * A {@link VerifiedConstMap} with each value stored next to its check word, so that a lookup @@ -14,32 +12,16 @@ *

This is a Java implementation of the {@code PairedVerifiedConstMap} of * constmap (Go). Its serialized form * ({@code PMAP0001}) is shared with constmap, rsconstmap (Rust) and fastconstmap (C and - * Python), and for the same keys and values this class writes exactly the bytes that the Go - * and Rust implementations write. See {@link ConstMap} for how keys are hashed. + * Python), and for the same keys and values the maps built here serialize to exactly the + * bytes that the Go and Rust implementations write. See {@link ConstMap} for how keys are + * hashed, and for the two implementations: on-heap maps and in-place views + * ({@link #wrap(ByteBuffer)}). * - *

Instances are immutable and thread-safe. + *

Instances are immutable and thread-safe (a view, as long as its buffer's bytes are not + * modified). */ -public final class PairedVerifiedConstMap { - private final ConstMaps.Params params; - private final long seed; - private final int segmentLength; - private final int segmentLengthMask; - private final long segmentCountLength; - /** The value for position i at slots[2i], its check word at slots[2i+1]. */ - private final long[] slots; - - private PairedVerifiedConstMap(ConstMaps.Params params, long[] slots) { - this.params = params; - this.seed = params.seed(); - this.segmentLength = params.segmentLength(); - this.segmentLengthMask = params.segmentLengthMask(); - this.segmentCountLength = params.segmentCountLength(); - this.slots = slots; - } - - static PairedVerifiedConstMap of(ConstMaps.Params params, long[] slots) { - return new PairedVerifiedConstMap(params, slots); - } +public sealed interface PairedVerifiedConstMap + permits HeapPairedVerifiedConstMap, BufferPairedVerifiedConstMap { /** * Builds a map from string keys and their values: {@code keys.get(i)} maps to @@ -47,11 +29,11 @@ static PairedVerifiedConstMap of(ConstMaps.Params params, long[] slots) { * * @param keys the keys, distinct * @param values the values, one per key - * @return the map + * @return the on-heap map * @throws IllegalArgumentException if the lengths differ, if two keys are equal (detected * with high probability), or if there are too many keys for one map */ - public static PairedVerifiedConstMap build(List keys, long[] values) { + static PairedVerifiedConstMap build(List keys, long[] values) { return VerifiedConstMap.build(keys, values).paired(); } @@ -61,11 +43,11 @@ public static PairedVerifiedConstMap build(List keys, long[] values) { * * @param keys the keys, distinct * @param values the values, one per key - * @return the map + * @return the on-heap map * @throws IllegalArgumentException if the lengths differ, if two keys are equal (detected * with high probability), or if there are too many keys for one map */ - public static PairedVerifiedConstMap buildFromBytes(List keys, long[] values) { + static PairedVerifiedConstMap buildFromBytes(List keys, long[] values) { return VerifiedConstMap.buildFromBytes(keys, values).paired(); } @@ -75,14 +57,56 @@ public static PairedVerifiedConstMap buildFromBytes(List keys, long[] va * * @param keyHashes the XXH64 hashes (seed 0) of the keys, distinct * @param values the values, one per key - * @return the map + * @return the on-heap map * @throws IllegalArgumentException if the lengths differ, if two hashes are equal (detected * with high probability), or if there are too many keys for one map */ - public static PairedVerifiedConstMap buildFromHashes(long[] keyHashes, long[] values) { + static PairedVerifiedConstMap buildFromHashes(long[] keyHashes, long[] values) { return VerifiedConstMap.buildFromHashes(keyHashes, values).paired(); } + /** + * Reads a serialized map ({@code PMAP0001}) starting at the buffer's position into a new + * on-heap map, after validating its header and checksum. The buffer's position is not + * changed. + * + * @param buffer a buffer holding a serialized map at its position + * @return the on-heap map + * @throws InvalidFilterException if the input is not a valid {@code PairedVerifiedConstMap} + * or is corrupt + */ + static PairedVerifiedConstMap load(ByteBuffer buffer) { + return HeapPairedVerifiedConstMap.read(buffer); + } + + /** + * Reads a serialized map, as {@link #load(ByteBuffer)} does. + * + * @param bytes the serialized map + * @return the on-heap map + * @throws InvalidFilterException if the input is invalid or corrupt + */ + static PairedVerifiedConstMap load(byte[] bytes) { + return load(ByteBuffer.wrap(bytes)); + } + + /** + * Returns a map that queries the serialized map ({@code PMAP0001}) starting at the buffer's + * position directly, without copying it. The buffer may be on the heap, direct or a + * {@link java.nio.MappedByteBuffer}; its position, limit and byte order are not changed. + * There is no alignment requirement. + * + *

This method runs in constant time: it validates the header but does not verify the + * checksum (use {@link #load(ByteBuffer)} for that). See {@link ConstMap#wrap(ByteBuffer)}. + * + * @param buffer a buffer holding a serialized map at its position + * @return a view backed by the buffer + * @throws InvalidFilterException if the header is invalid or the buffer is too short + */ + static PairedVerifiedConstMap wrap(ByteBuffer buffer) { + return BufferPairedVerifiedConstMap.wrap(buffer); + } + /** * Returns the value of a key whose XXH64 hash (seed 0) is given. Equivalent to the other * {@code get} methods on the key itself. @@ -90,23 +114,7 @@ public static PairedVerifiedConstMap buildFromHashes(long[] keyHashes, long[] va * @param keyHash the XXH64 hash of the key * @return the value, or {@link VerifiedConstMap#NOT_FOUND} if the key is not in the map */ - public long getByHash(long keyHash) { - if (slots.length == 0) { - return ConstMaps.NOT_FOUND; - } - long hash = ConstMaps.mixsplit(keyHash, seed); - int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); - int h1 = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); - int h2 = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); - final long[] s = slots; - int i0 = 2 * h0; - int i1 = 2 * h1; - int i2 = 2 * h2; - if ((s[i0 + 1] ^ s[i1 + 1] ^ s[i2 + 1]) != ConstMaps.fingerprint(hash)) { - return ConstMaps.NOT_FOUND; - } - return s[i0] ^ s[i1] ^ s[i2]; - } + long getByHash(long keyHash); /** * Returns the value of a string key (hashed through its UTF-8 encoding). @@ -114,7 +122,7 @@ public long getByHash(long keyHash) { * @param key the key * @return the value, or {@link VerifiedConstMap#NOT_FOUND} if the key is not in the map */ - public long get(String key) { + default long get(String key) { return getByHash(XxHash64.hash(key)); } @@ -124,7 +132,7 @@ public long get(String key) { * @param key the key * @return the value, or {@link VerifiedConstMap#NOT_FOUND} if the key is not in the map */ - public long get(byte[] key) { + default long get(byte[] key) { return getByHash(XxHash64.hash(key)); } @@ -137,7 +145,7 @@ public long get(byte[] key) { * @return the value, or {@link VerifiedConstMap#NOT_FOUND} if the key is not in the map * @throws IndexOutOfBoundsException if the range is not within the array */ - public long get(byte[] key, int off, int len) { + default long get(byte[] key, int off, int len) { return getByHash(XxHash64.hash(key, off, len)); } @@ -148,7 +156,7 @@ public long get(byte[] key, int off, int len) { * @param key the key * @return the value, or {@link VerifiedConstMap#NOT_FOUND} if the key is not in the map */ - public long get(ByteBuffer key) { + default long get(ByteBuffer key) { return getByHash(XxHash64.hash(key)); } @@ -161,7 +169,7 @@ public long get(ByteBuffer key) { * false positive), false if it is not or if its value is * {@link VerifiedConstMap#NOT_FOUND} */ - public boolean containsKey(String key) { + default boolean containsKey(String key) { return get(key) != ConstMaps.NOT_FOUND; } @@ -175,35 +183,7 @@ public boolean containsKey(String key) { * @param dst where the values go; only the first {@code keys.size()} entries are written * @throws IndexOutOfBoundsException if {@code dst} is shorter than {@code keys} */ - public void getMany(List keys, long[] dst) { - int n = keys.size(); - Objects.checkFromIndexSize(0, n, dst.length); - if (slots.length == 0) { - Arrays.fill(dst, 0, n, ConstMaps.NOT_FOUND); - return; - } - final long[] s = slots; - final int[] h = new int[3 * ConstMap.BLOCK]; - final long[] fp = new long[ConstMap.BLOCK]; - for (int start = 0; start < n; start += ConstMap.BLOCK) { - int m = Math.min(ConstMap.BLOCK, n - start); - for (int j = 0; j < m; j++) { - long hash = ConstMaps.mixsplit(XxHash64.hash(keys.get(start + j)), seed); - int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); - h[3 * j] = 2 * h0; - h[3 * j + 1] = 2 * ((h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask)); - h[3 * j + 2] = 2 * ((h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask)); - fp[j] = ConstMaps.fingerprint(hash); - } - for (int j = 0; j < m; j++) { - int a = h[3 * j]; - int b = h[3 * j + 1]; - int c = h[3 * j + 2]; - long v = s[a] ^ s[b] ^ s[c]; - dst[start + j] = (s[a + 1] ^ s[b + 1] ^ s[c + 1]) == fp[j] ? v : ConstMaps.NOT_FOUND; - } - } - } + void getMany(List keys, long[] dst); /** * Looks up many string keys at once, as {@link #getMany(List, long[])} does, into a new @@ -212,7 +192,7 @@ public void getMany(List keys, long[] dst) { * @param keys the keys * @return the values, one per key, {@link VerifiedConstMap#NOT_FOUND} for missing keys */ - public long[] getMany(List keys) { + default long[] getMany(List keys) { long[] dst = new long[keys.size()]; getMany(keys, dst); return dst; @@ -224,9 +204,7 @@ public long[] getMany(List keys) { * * @return the number of positions */ - public int arrayLength() { - return slots.length / 2; - } + int arrayLength(); /** * Returns the number of bytes written by {@link #serialize(ByteBuffer)}: a 32-byte header, @@ -234,65 +212,29 @@ public int arrayLength() { * * @return the serialized size in bytes */ - public int serializedSize() { - return ConstMaps.VMAP_HEADER + 8 * slots.length + 8; - } + int serializedSize(); /** - * Writes the map at the buffer's current position, in the {@code PMAP0001} format shared - * with the Go, Rust and C implementations, and advances the position by - * {@link #serializedSize()}. The buffer's byte order is ignored and left unchanged. + * Writes the map at the buffer's current position and advances the position by + * {@link #serializedSize()}. An on-heap map writes the {@code PMAP0001} format shared with + * the Go, Rust and C implementations; a view writes the bytes it wraps, unchanged. The + * buffer's byte order is ignored and left unchanged. * * @param out the destination * @throws java.nio.BufferOverflowException if fewer than {@link #serializedSize()} bytes * remain * @throws java.nio.ReadOnlyBufferException if the buffer is read-only */ - public void serialize(ByteBuffer out) { - ConstMaps.write(out, ConstMaps.MAGIC_PMAP, params, slots.length / 2, true, slots); - } + void serialize(ByteBuffer out); /** * Returns the serialized form of the map, as written by {@link #serialize(ByteBuffer)}. * * @return a new array of {@link #serializedSize()} bytes */ - public byte[] toByteArray() { + default byte[] toByteArray() { byte[] bytes = new byte[serializedSize()]; serialize(ByteBuffer.wrap(bytes)); return bytes; } - - /** - * Reads a serialized map ({@code PMAP0001}) starting at the buffer's position into a new - * on-heap map, after validating its header and checksum. The buffer's position is not - * changed. - * - * @param buffer a buffer holding a serialized map at its position - * @return the map - * @throws InvalidFilterException if the input is not a valid {@code PairedVerifiedConstMap} - * or is corrupt - */ - public static PairedVerifiedConstMap load(ByteBuffer buffer) { - ConstMaps.Header h = ConstMaps.read(buffer, ConstMaps.Kind.PAIRED, 2); - int n = h.params().arrayLength(); - return new PairedVerifiedConstMap(h.params(), ConstMaps.readWords(buffer, h.start(), 2 * n)); - } - - /** - * Reads a serialized map, as {@link #load(ByteBuffer)} does. - * - * @param bytes the serialized map - * @return the map - * @throws InvalidFilterException if the input is invalid or corrupt - */ - public static PairedVerifiedConstMap load(byte[] bytes) { - return load(ByteBuffer.wrap(bytes)); - } - - @Override - public String toString() { - return "PairedVerifiedConstMap{positions=" + arrayLength() + ", bytes=" + serializedSize() - + "}"; - } } diff --git a/src/main/java/io/github/fastfilter/jbinfuse/VerifiedConstMap.java b/src/main/java/io/github/fastfilter/jbinfuse/VerifiedConstMap.java index 87f6a25..6593563 100644 --- a/src/main/java/io/github/fastfilter/jbinfuse/VerifiedConstMap.java +++ b/src/main/java/io/github/fastfilter/jbinfuse/VerifiedConstMap.java @@ -1,9 +1,7 @@ package io.github.fastfilter.jbinfuse; import java.nio.ByteBuffer; -import java.util.Arrays; import java.util.List; -import java.util.Objects; /** * An immutable map from keys (strings or byte sequences) to 64-bit values that detects @@ -20,36 +18,21 @@ *

This is a Java implementation of the {@code VerifiedConstMap} of * constmap (Go). Its serialized form * ({@code VMAP0001}) is shared with constmap, rsconstmap (Rust) and fastconstmap (C and - * Python), and for the same keys and values this class writes exactly the bytes that the Go - * and Rust implementations write. See {@link ConstMap} for how keys are hashed. + * Python), and for the same keys and values the maps built here serialize to exactly the + * bytes that the Go and Rust implementations write. See {@link ConstMap} for how keys are + * hashed, and for the two implementations: on-heap maps and in-place views + * ({@link #wrap(ByteBuffer)}). * - *

Instances are immutable and thread-safe. + *

Instances are immutable and thread-safe (a view, as long as its buffer's bytes are not + * modified). */ -public final class VerifiedConstMap { +public sealed interface VerifiedConstMap permits HeapVerifiedConstMap, BufferVerifiedConstMap { /** * The value returned for a key that is not in the map: all 64 bits set ({@code -1}, or * 264-1 read as unsigned). A key mapped to this value cannot be told apart from a * missing key. */ - public static final long NOT_FOUND = ConstMaps.NOT_FOUND; - - private final ConstMaps.Params params; - private final long seed; - private final int segmentLength; - private final int segmentLengthMask; - private final long segmentCountLength; - private final long[] data; - private final long[] checks; - - private VerifiedConstMap(ConstMaps.Params params, long[] data, long[] checks) { - this.params = params; - this.seed = params.seed(); - this.segmentLength = params.segmentLength(); - this.segmentLengthMask = params.segmentLengthMask(); - this.segmentCountLength = params.segmentCountLength(); - this.data = data; - this.checks = checks; - } + long NOT_FOUND = ConstMaps.NOT_FOUND; /** * Builds a map from string keys and their values: {@code keys.get(i)} maps to @@ -57,13 +40,13 @@ private VerifiedConstMap(ConstMaps.Params params, long[] data, long[] checks) { * * @param keys the keys, distinct * @param values the values, one per key - * @return the map + * @return the on-heap map * @throws IllegalArgumentException if the lengths differ, if two keys are equal (detected * with high probability), or if there are too many keys for one map */ - public static VerifiedConstMap build(List keys, long[] values) { + static VerifiedConstMap build(List keys, long[] values) { ConstMaps.checkLengths(keys.size(), values.length); - return buildFromHashes(ConstMaps.hashStrings(keys), values); + return HeapVerifiedConstMap.build(ConstMaps.hashStrings(keys), values); } /** @@ -72,13 +55,13 @@ public static VerifiedConstMap build(List keys, long[] values) { * * @param keys the keys, distinct * @param values the values, one per key - * @return the map + * @return the on-heap map * @throws IllegalArgumentException if the lengths differ, if two keys are equal (detected * with high probability), or if there are too many keys for one map */ - public static VerifiedConstMap buildFromBytes(List keys, long[] values) { + static VerifiedConstMap buildFromBytes(List keys, long[] values) { ConstMaps.checkLengths(keys.size(), values.length); - return buildFromHashes(ConstMaps.hashBytes(keys), values); + return HeapVerifiedConstMap.build(ConstMaps.hashBytes(keys), values); } /** @@ -87,21 +70,55 @@ public static VerifiedConstMap buildFromBytes(List keys, long[] values) * * @param keyHashes the XXH64 hashes (seed 0) of the keys, distinct * @param values the values, one per key - * @return the map + * @return the on-heap map * @throws IllegalArgumentException if the lengths differ, if two hashes are equal (detected * with high probability), or if there are too many keys for one map */ - public static VerifiedConstMap buildFromHashes(long[] keyHashes, long[] values) { + static VerifiedConstMap buildFromHashes(long[] keyHashes, long[] values) { ConstMaps.checkLengths(keyHashes.length, values.length); - if (keyHashes.length == 0) { - return new VerifiedConstMap(ConstMaps.Params.EMPTY, new long[0], new long[0]); - } - ConstMaps.Peeled p = ConstMaps.peel(keyHashes); - int n = p.params().arrayLength(); - long[] data = new long[n]; - long[] checks = new long[n]; - ConstMaps.assign(p, values, data, checks); - return new VerifiedConstMap(p.params(), data, checks); + return HeapVerifiedConstMap.build(keyHashes, values); + } + + /** + * Reads a serialized map ({@code VMAP0001}) starting at the buffer's position into a new + * on-heap map, after validating its header and checksum. The buffer's position is not + * changed. + * + * @param buffer a buffer holding a serialized map at its position + * @return the on-heap map + * @throws InvalidFilterException if the input is not a valid {@code VerifiedConstMap} or is + * corrupt + */ + static VerifiedConstMap load(ByteBuffer buffer) { + return HeapVerifiedConstMap.read(buffer); + } + + /** + * Reads a serialized map, as {@link #load(ByteBuffer)} does. + * + * @param bytes the serialized map + * @return the on-heap map + * @throws InvalidFilterException if the input is invalid or corrupt + */ + static VerifiedConstMap load(byte[] bytes) { + return load(ByteBuffer.wrap(bytes)); + } + + /** + * Returns a map that queries the serialized map ({@code VMAP0001}) starting at the buffer's + * position directly, without copying it. The buffer may be on the heap, direct or a + * {@link java.nio.MappedByteBuffer}; its position, limit and byte order are not changed. + * There is no alignment requirement. + * + *

This method runs in constant time: it validates the header but does not verify the + * checksum (use {@link #load(ByteBuffer)} for that). See {@link ConstMap#wrap(ByteBuffer)}. + * + * @param buffer a buffer holding a serialized map at its position + * @return a view backed by the buffer + * @throws InvalidFilterException if the header is invalid or the buffer is too short + */ + static VerifiedConstMap wrap(ByteBuffer buffer) { + return BufferVerifiedConstMap.wrap(buffer); } /** @@ -111,19 +128,7 @@ public static VerifiedConstMap buildFromHashes(long[] keyHashes, long[] values) * @param keyHash the XXH64 hash of the key * @return the value, or {@link #NOT_FOUND} if the key is not in the map */ - public long getByHash(long keyHash) { - if (data.length == 0) { - return NOT_FOUND; - } - long hash = ConstMaps.mixsplit(keyHash, seed); - int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); - int h1 = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); - int h2 = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); - if ((checks[h0] ^ checks[h1] ^ checks[h2]) != ConstMaps.fingerprint(hash)) { - return NOT_FOUND; - } - return data[h0] ^ data[h1] ^ data[h2]; - } + long getByHash(long keyHash); /** * Returns the value of a string key (hashed through its UTF-8 encoding). @@ -131,7 +136,7 @@ public long getByHash(long keyHash) { * @param key the key * @return the value, or {@link #NOT_FOUND} if the key is not in the map */ - public long get(String key) { + default long get(String key) { return getByHash(XxHash64.hash(key)); } @@ -141,7 +146,7 @@ public long get(String key) { * @param key the key * @return the value, or {@link #NOT_FOUND} if the key is not in the map */ - public long get(byte[] key) { + default long get(byte[] key) { return getByHash(XxHash64.hash(key)); } @@ -154,7 +159,7 @@ public long get(byte[] key) { * @return the value, or {@link #NOT_FOUND} if the key is not in the map * @throws IndexOutOfBoundsException if the range is not within the array */ - public long get(byte[] key, int off, int len) { + default long get(byte[] key, int off, int len) { return getByHash(XxHash64.hash(key, off, len)); } @@ -165,7 +170,7 @@ public long get(byte[] key, int off, int len) { * @param key the key * @return the value, or {@link #NOT_FOUND} if the key is not in the map */ - public long get(ByteBuffer key) { + default long get(ByteBuffer key) { return getByHash(XxHash64.hash(key)); } @@ -177,7 +182,7 @@ public long get(ByteBuffer key) { * @return true if the key is in the map (or, with probability about 2-64, a * false positive), false if it is not or if its value is {@link #NOT_FOUND} */ - public boolean containsKey(String key) { + default boolean containsKey(String key) { return get(key) != NOT_FOUND; } @@ -191,36 +196,7 @@ public boolean containsKey(String key) { * @param dst where the values go; only the first {@code keys.size()} entries are written * @throws IndexOutOfBoundsException if {@code dst} is shorter than {@code keys} */ - public void getMany(List keys, long[] dst) { - int n = keys.size(); - Objects.checkFromIndexSize(0, n, dst.length); - if (data.length == 0) { - Arrays.fill(dst, 0, n, NOT_FOUND); - return; - } - final long[] d = data; - final long[] c = checks; - final int[] h = new int[3 * ConstMap.BLOCK]; - final long[] fp = new long[ConstMap.BLOCK]; - for (int start = 0; start < n; start += ConstMap.BLOCK) { - int m = Math.min(ConstMap.BLOCK, n - start); - for (int j = 0; j < m; j++) { - long hash = ConstMaps.mixsplit(XxHash64.hash(keys.get(start + j)), seed); - int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); - h[3 * j] = h0; - h[3 * j + 1] = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); - h[3 * j + 2] = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); - fp[j] = ConstMaps.fingerprint(hash); - } - for (int j = 0; j < m; j++) { - int a = h[3 * j]; - int b = h[3 * j + 1]; - int e = h[3 * j + 2]; - long v = d[a] ^ d[b] ^ d[e]; - dst[start + j] = (c[a] ^ c[b] ^ c[e]) == fp[j] ? v : NOT_FOUND; - } - } - } + void getMany(List keys, long[] dst); /** * Looks up many string keys at once, as {@link #getMany(List, long[])} does, into a new @@ -229,25 +205,18 @@ public void getMany(List keys, long[] dst) { * @param keys the keys * @return the values, one per key, {@link #NOT_FOUND} for missing keys */ - public long[] getMany(List keys) { + default long[] getMany(List keys) { long[] dst = new long[keys.size()]; getMany(keys, dst); return dst; } /** - * Returns the same map in the paired layout, as a copy. + * Returns the same map in the paired layout, as a new on-heap map (also for a view). * * @return a {@link PairedVerifiedConstMap} giving the same answers */ - public PairedVerifiedConstMap paired() { - long[] slots = new long[2 * data.length]; - for (int i = 0; i < data.length; i++) { - slots[2 * i] = data[i]; - slots[2 * i + 1] = checks[i]; - } - return PairedVerifiedConstMap.of(params, slots); - } + PairedVerifiedConstMap paired(); /** * Returns the number of positions in each of the map's two arrays: about 1.125 per key for @@ -255,9 +224,7 @@ public PairedVerifiedConstMap paired() { * * @return the array length */ - public int arrayLength() { - return data.length; - } + int arrayLength(); /** * Returns the number of bytes written by {@link #serialize(ByteBuffer)}: a 32-byte header, @@ -265,65 +232,29 @@ public int arrayLength() { * * @return the serialized size in bytes */ - public int serializedSize() { - return ConstMaps.VMAP_HEADER + 16 * data.length + 8; - } + int serializedSize(); /** - * Writes the map at the buffer's current position, in the {@code VMAP0001} format shared - * with the Go, Rust and C implementations, and advances the position by - * {@link #serializedSize()}. The buffer's byte order is ignored and left unchanged. + * Writes the map at the buffer's current position and advances the position by + * {@link #serializedSize()}. An on-heap map writes the {@code VMAP0001} format shared with + * the Go, Rust and C implementations; a view writes the bytes it wraps, unchanged. The + * buffer's byte order is ignored and left unchanged. * * @param out the destination * @throws java.nio.BufferOverflowException if fewer than {@link #serializedSize()} bytes * remain * @throws java.nio.ReadOnlyBufferException if the buffer is read-only */ - public void serialize(ByteBuffer out) { - ConstMaps.write(out, ConstMaps.MAGIC_VMAP, params, data.length, true, data, checks); - } + void serialize(ByteBuffer out); /** * Returns the serialized form of the map, as written by {@link #serialize(ByteBuffer)}. * * @return a new array of {@link #serializedSize()} bytes */ - public byte[] toByteArray() { + default byte[] toByteArray() { byte[] bytes = new byte[serializedSize()]; serialize(ByteBuffer.wrap(bytes)); return bytes; } - - /** - * Reads a serialized map ({@code VMAP0001}) starting at the buffer's position into a new - * on-heap map, after validating its header and checksum. The buffer's position is not - * changed. - * - * @param buffer a buffer holding a serialized map at its position - * @return the map - * @throws InvalidFilterException if the input is not a valid {@code VerifiedConstMap} or is - * corrupt - */ - public static VerifiedConstMap load(ByteBuffer buffer) { - ConstMaps.Header h = ConstMaps.read(buffer, ConstMaps.Kind.VERIFIED, 2); - int n = h.params().arrayLength(); - return new VerifiedConstMap(h.params(), ConstMaps.readWords(buffer, h.start(), n), - ConstMaps.readWords(buffer, h.start() + 8 * n, n)); - } - - /** - * Reads a serialized map, as {@link #load(ByteBuffer)} does. - * - * @param bytes the serialized map - * @return the map - * @throws InvalidFilterException if the input is invalid or corrupt - */ - public static VerifiedConstMap load(byte[] bytes) { - return load(ByteBuffer.wrap(bytes)); - } - - @Override - public String toString() { - return "VerifiedConstMap{positions=" + data.length + ", bytes=" + serializedSize() + "}"; - } } diff --git a/src/test/java/io/github/fastfilter/jbinfuse/ConstMapViewTest.java b/src/test/java/io/github/fastfilter/jbinfuse/ConstMapViewTest.java new file mode 100644 index 0000000..0f40560 --- /dev/null +++ b/src/test/java/io/github/fastfilter/jbinfuse/ConstMapViewTest.java @@ -0,0 +1,255 @@ +package io.github.fastfilter.jbinfuse; + +import static io.github.fastfilter.jbinfuse.ConstMapTest.NOT_FOUND; +import static io.github.fastfilter.jbinfuse.ConstMapTest.checkInterop; +import static io.github.fastfilter.jbinfuse.ConstMapTest.keys; +import static io.github.fastfilter.jbinfuse.ConstMapTest.resource; +import static io.github.fastfilter.jbinfuse.ConstMapTest.values; +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.IOException; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.MappedByteBuffer; +import java.nio.channels.FileChannel; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.util.ArrayList; +import java.util.List; +import java.util.SplittableRandom; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +/** Zero-copy views: {@code wrap(ByteBuffer)} on the three kinds of constant map. */ +class ConstMapViewTest { + + /** The bytes at {@code offset} in a buffer of the given kind, with garbage around them. */ + static ByteBuffer place(byte[] bytes, int offset, boolean direct, ByteOrder order) { + ByteBuffer b = direct ? ByteBuffer.allocateDirect(offset + bytes.length + 11) + : ByteBuffer.allocate(offset + bytes.length + 11); + for (int i = 0; i < b.capacity(); i++) { + b.put(i, (byte) (0xA5 ^ i)); + } + b.put(offset, bytes); + b.position(offset); + b.order(order); + return b; + } + + static Object wrap(String ext, ByteBuffer b) { + return switch (ext) { + case "cmap" -> ConstMap.wrap(b); + case "vmap" -> VerifiedConstMap.wrap(b); + default -> PairedVerifiedConstMap.wrap(b); + }; + } + + static ConstMapTest.Lookup lookup(Object view) { + if (view instanceof ConstMap c) { + return c::get; + } else if (view instanceof VerifiedConstMap v) { + return v::get; + } + return ((PairedVerifiedConstMap) view)::get; + } + + static byte[] bytes(Object view) { + if (view instanceof ConstMap c) { + return c.toByteArray(); + } else if (view instanceof VerifiedConstMap v) { + return v.toByteArray(); + } + return ((PairedVerifiedConstMap) view).toByteArray(); + } + + /** + * Files written by rsconstmap (Rust) and fastconstmap (C/Python), which include Go's + * 28-byte CMAP0001 header, fastconstmap's CMAP0003 and its key count in the pad word of + * VMAP0001 and PMAP0001, wrapped at every alignment, from heap and direct buffers in either + * byte order: the answers are those of the original maps, the buffer is untouched, and the + * view serializes to the very bytes it wraps. + */ + @ParameterizedTest + @ValueSource(strings = {"rust.cmap", "rust.vmap", "rust.pmap", "fastconstmap.cmap", + "fastconstmap.vmap", "fastconstmap.pmap"}) + void wrapsFilesOfTheOtherImplementations(String name) throws IOException { + byte[] file = resource(name); + String ext = name.substring(name.length() - 4); + boolean verified = !ext.equals("cmap"); + for (int offset = 0; offset < 16; offset++) { + for (boolean direct : new boolean[] {false, true}) { + for (ByteOrder order : new ByteOrder[] {ByteOrder.BIG_ENDIAN, ByteOrder.LITTLE_ENDIAN}) { + ByteBuffer b = place(file, offset, direct, order); + int limit = b.limit(); + Object view = wrap(ext, b); + String id = name + " offset " + offset + (direct ? " direct " : " heap ") + order; + checkInterop(id, lookup(view), verified); + assertEquals(offset, b.position(), id); + assertEquals(limit, b.limit(), id); + assertEquals(order, b.order(), id); + assertArrayEquals(file, bytes(view), id); + } + } + } + } + + /** The bytes a view writes are a valid file for the on-heap loaders, and the same map. */ + @Test + void viewBytesLoadAndViewsAgreeWithLoadedMaps() throws IOException { + for (String src : new String[] {"rust", "fastconstmap"}) { + byte[] c = resource(src + ".cmap"); + byte[] v = resource(src + ".vmap"); + byte[] p = resource(src + ".pmap"); + ConstMap cv = ConstMap.wrap(ByteBuffer.wrap(c)); + VerifiedConstMap vv = VerifiedConstMap.wrap(ByteBuffer.wrap(v)); + PairedVerifiedConstMap pv = PairedVerifiedConstMap.wrap(ByteBuffer.wrap(p)); + checkInterop(src, ConstMap.load(cv.toByteArray())::get, false); + checkInterop(src, VerifiedConstMap.load(vv.toByteArray())::get, true); + checkInterop(src, PairedVerifiedConstMap.load(pv.toByteArray())::get, true); + assertEquals(ConstMap.load(c).arrayLength(), cv.arrayLength()); + assertEquals(VerifiedConstMap.load(v).arrayLength(), vv.arrayLength()); + assertEquals(PairedVerifiedConstMap.load(p).arrayLength(), pv.arrayLength()); + // Converting a view of a VMAP to the paired layout gives the PMAP the others wrote, + // apart from the pad word that fastconstmap fills with its key count. + byte[] paired = vv.paired().toByteArray(); + byte[] expected = resource("rust.pmap"); + assertArrayEquals(expected, paired, src); + } + } + + @Test + void wrapsMemoryMappedFiles(@TempDir Path dir) throws IOException { + for (String name : new String[] {"rust.cmap", "fastconstmap.cmap", "rust.vmap", + "fastconstmap.vmap", "rust.pmap", "fastconstmap.pmap"}) { + // At an odd offset inside a larger file, as when maps are embedded in a container. + byte[] file = resource(name); + byte[] container = new byte[file.length + 100]; + System.arraycopy(file, 0, container, 37, file.length); + Path path = dir.resolve(name); + Files.write(path, container); + try (FileChannel ch = FileChannel.open(path, StandardOpenOption.READ)) { + MappedByteBuffer mapped = ch.map(FileChannel.MapMode.READ_ONLY, 0, ch.size()); + mapped.position(37); + String ext = name.substring(name.length() - 4); + Object view = wrap(ext, mapped); + checkInterop(name, lookup(view), !ext.equals("cmap")); + assertArrayEquals(file, bytes(view), name); + } + } + } + + @ParameterizedTest + @ValueSource(ints = {0, 1, 2, 5, 100, 1001, 30000}) + void viewsAnswerLikeOnHeapMaps(int n) { + List k = keys("v-", n); + long[] v = values(n); + ConstMap cm = ConstMap.build(k, v); + VerifiedConstMap vm = VerifiedConstMap.build(k, v); + PairedVerifiedConstMap pm = vm.paired(); + ConstMap cv = ConstMap.wrap(place(cm.toByteArray(), 3, true, ByteOrder.BIG_ENDIAN)); + VerifiedConstMap vv = VerifiedConstMap.wrap(place(vm.toByteArray(), 5, true, + ByteOrder.BIG_ENDIAN)); + PairedVerifiedConstMap pv = PairedVerifiedConstMap.wrap(place(pm.toByteArray(), 7, false, + ByteOrder.LITTLE_ENDIAN)); + List queries = new ArrayList<>(k); + SplittableRandom r = new SplittableRandom(n); + for (int i = 0; i < 2000; i++) { + queries.add("absent-" + r.nextLong()); + } + for (String q : queries) { + assertEquals(cm.get(q), cv.get(q)); + assertEquals(vm.get(q), vv.get(q)); + assertEquals(pm.get(q), pv.get(q)); + byte[] qb = q.getBytes(java.nio.charset.StandardCharsets.UTF_8); + assertEquals(vm.get(qb), vv.get(ByteBuffer.wrap(qb))); + } + assertArrayEquals(cm.getMany(queries), cv.getMany(queries)); + assertArrayEquals(vm.getMany(queries), vv.getMany(queries)); + assertArrayEquals(pm.getMany(queries), pv.getMany(queries)); + assertArrayEquals(cm.toByteArray(), cv.toByteArray()); + assertArrayEquals(vm.toByteArray(), vv.toByteArray()); + assertArrayEquals(pm.toByteArray(), pv.toByteArray()); + assertArrayEquals(pm.toByteArray(), vv.paired().toByteArray()); + assertEquals(cm.serializedSize(), cv.serializedSize()); + assertEquals(vm.arrayLength(), vv.arrayLength()); + assertEquals(pm.arrayLength(), pv.arrayLength()); + if (n == 0) { + assertEquals(NOT_FOUND, vv.get("x")); + assertEquals(NOT_FOUND, pv.get("x")); + } + } + + @Test + void viewSerializesAtThePositionAndAdvancesIt() { + VerifiedConstMap vm = VerifiedConstMap.build(keys("s", 50), values(50)); + VerifiedConstMap view = VerifiedConstMap.wrap(ByteBuffer.wrap(vm.toByteArray())); + ByteBuffer out = ByteBuffer.allocate(view.serializedSize() + 9); + out.position(9); + view.serialize(out); + assertEquals(out.capacity(), out.position()); + out.position(9); + assertArrayEquals(vm.toByteArray(), VerifiedConstMap.load(out).toByteArray()); + assertThrows(java.nio.BufferOverflowException.class, + () -> view.serialize(ByteBuffer.allocate(view.serializedSize() - 1))); + assertThrows(java.nio.ReadOnlyBufferException.class, + () -> view.serialize(ByteBuffer.allocate(view.serializedSize()).asReadOnlyBuffer())); + } + + @Test + void wrapRefusesWhatLoadRefuses() throws IOException { + for (String name : new String[] {"fastconstmap-0.9.cmap", "fastconstmap-0.9.vmap"}) { + ByteBuffer b = ByteBuffer.wrap(resource(name)); + for (String ext : new String[] {"cmap", "vmap", "pmap"}) { + InvalidFilterException e = assertThrows(InvalidFilterException.class, + () -> wrap(ext, b)); + assertTrue(e.getMessage().contains("fastconstmap 0.9"), e.getMessage()); + } + } + assertThrows(InvalidFilterException.class, + () -> ConstMap.wrap(ByteBuffer.wrap(resource("rust.vmap")))); + assertThrows(InvalidFilterException.class, + () -> VerifiedConstMap.wrap(ByteBuffer.wrap(resource("rust.pmap")))); + assertThrows(InvalidFilterException.class, + () -> PairedVerifiedConstMap.wrap(ByteBuffer.wrap(resource("rust.cmap")))); + for (String name : new String[] {"rust.cmap", "fastconstmap.cmap", "rust.vmap", + "rust.pmap"}) { + byte[] good = resource(name); + String ext = name.substring(name.length() - 4); + for (int len = 0; len < good.length; len += 7) { + ByteBuffer cut = ByteBuffer.wrap(good, 0, len); + assertThrows(InvalidFilterException.class, () -> wrap(ext, cut), name + " " + len); + } + // Header fields that do not describe the array, under a valid checksum. + for (int[] e : new int[][] {{16, 3}, {16, 0}, {20, 0}, {20, -1}, {24, 1}}) { + byte[] bad = good.clone(); + ByteBuffer.wrap(bad).order(ByteOrder.LITTLE_ENDIAN).putInt(e[0], e[1]); + assertThrows(InvalidFilterException.class, () -> wrap(ext, ByteBuffer.wrap(bad)), + name + " offset " + e[0]); + } + } + } + + /** + * wrap runs in constant time and so does not verify the checksum; load does. On corrupt + * array bytes a view gives wrong answers but never reads outside the map. + */ + @Test + void corruptArraysAreCaughtByLoadButNeverReadOutOfBoundsByViews() throws IOException { + byte[] bad = resource("rust.pmap"); + for (int i = 40; i < bad.length - 8; i += 3) { + bad[i] ^= (byte) (i * 31); + } + assertThrows(InvalidFilterException.class, () -> PairedVerifiedConstMap.load(bad)); + PairedVerifiedConstMap view = PairedVerifiedConstMap.wrap(ByteBuffer.wrap(bad)); + SplittableRandom r = new SplittableRandom(1); + for (int i = 0; i < 100000; i++) { + view.getByHash(r.nextLong()); + } + } +}