diff --git a/README.md b/README.md index 3f43463..ee91593 100644 --- a/README.md +++ b/README.md @@ -20,6 +20,9 @@ filters for the same false-positive rate, and a query reads exactly three memory `BinaryFuse16`), the fastest; and **in-place views** of a serialized filter in a `ByteBuffer` (heap, direct, or a memory-mapped file), with near-zero heap use and instant open. See [Choosing a query mode](#choosing-a-query-mode). +- **Constant maps** from string or byte-array keys to 64-bit values, built on the same + construction (about 9 or 18 bytes per key), whose files are interchangeable with the Go, + Rust and C/Python implementations. See [Constant maps](#constant-maps). ## Maven @@ -139,16 +142,106 @@ misses on 4 KiB page-cache pages, whereas a large Java heap typically uses 2 MiB huge pages.) Choose `wrap` when heap memory or open time matters more than those nanoseconds, for example for many large filters. +## Constant maps + +The binary fuse construction also gives an immutable map from keys to 64-bit values that does +not store the keys: a lookup is one hash, three array reads and two XORs. + +```java +import io.github.fastfilter.jbinfuse.ConstMap; +import io.github.fastfilter.jbinfuse.VerifiedConstMap; + +List keys = List.of("apple", "banana", "cherry"); +long[] values = {100, 200, 300}; + +ConstMap cm = ConstMap.build(keys, values); // about 9 bytes per key +cm.get("banana"); // 200 +cm.get("grape"); // an arbitrary value! + +VerifiedConstMap vm = VerifiedConstMap.build(keys, values); // about 18 bytes per key +vm.get("banana"); // 200 +vm.get("grape"); // VerifiedConstMap.NOT_FOUND (-1) +vm.containsKey("grape"); // false + +long[] many = vm.getMany(List.of("apple", "grape")); // {100, NOT_FOUND} +``` + +| | `ConstMap` | `VerifiedConstMap` | `PairedVerifiedConstMap` | +| --- | --- | --- | --- | +| bytes per key | about 9 | about 18 | about 18 | +| missing key | arbitrary value | `NOT_FOUND` | `NOT_FOUND` | +| cache lines per lookup | 3 | 6 (3 for most misses) | 3 | +| best when | you only look up keys in the map | most lookups miss | most lookups hit | + +Keys must be distinct. String keys are hashed through their UTF-8 encoding with XXH64 +(`XxHash64`), so `get("abc")` and `get("abc".getBytes(UTF_8))` are the same lookup; there are +also `buildFromBytes(List, long[])`, `get(byte[], int, int)`, `get(ByteBuffer)`, and +`buildFromHashes`/`getByHash` taking XXH64 hashes you computed yourself. A verified map reports +a missing key as present with probability about 2-64; a key whose value is +`NOT_FOUND` cannot be told apart from a missing one. `vm.paired()` converts a +`VerifiedConstMap` to a `PairedVerifiedConstMap`. + +`getMany` hashes a block of keys before reading the array, so that the processor can overlap +the memory accesses of several lookups; it returns exactly what a loop over `get` would. + +### Interoperability with constmap, rsconstmap and fastconstmap + +These classes are a Java implementation of [constmap](https://github.com/lemire/constmap) (Go), +also implemented by [rsconstmap](https://github.com/lemire/rsconstmap) (Rust) and +[fastconstmap](https://github.com/lemire/fastconstmap) (C, with Python bindings). All four +share the key hash, the mixing, the seed sequence, the layout and the file formats, so a map +saved by any of them loads in the others: + +```java +Files.write(Path.of("map.vmap"), vm.toByteArray()); // Go, Rust, Python read it +VerifiedConstMap fromPython = VerifiedConstMap.load(Files.readAllBytes(Path.of("built-by-python.vmap"))); +``` + +- For the same keys and values, the Java classes write exactly the bytes that Go and Rust + write: `CMAP0001`, `VMAP0001` and `PMAP0001`. This is tested against the files of the other + implementations and against the Go package for sizes from 0 to 1,234,567 keys. +- `ConstMap.load` also reads fastconstmap's `CMAP0003` (the same format with a key count, + which is ignored). fastconstmap's key count in the padding word of the other two formats is + ignored too. +- Files from fastconstmap 0.9 and earlier (`CMAP0002`, `VCMP0002`) used another hash (XXH3) + and are refused with a message saying so; rebuild them with fastconstmap 0.10 or later. +- `load` validates the magic bytes, the segment parameters and the FNV-1a checksum, and throws + `InvalidFilterException` otherwise, including for a file of another kind of map. + +The constant maps are always loaded onto the heap; there is no in-place view as for filters. + +### Constant map performance + +JMH (`ConstMapBenchmark`) on an Apple M4 Max with JDK 27, ns per lookup of a present key, in +random order, with keys like `key-123456-1e3779b9`: + +| | 1 million keys | 10 million keys | +| --- | ---: | ---: | +| `ConstMap.get(String)` | 16.7 | 33.0 | +| `ConstMap.getMany` | 16.2 | 24.8 | +| `VerifiedConstMap.get(String)` | 25.8 | 65.4 | +| `VerifiedConstMap.getMany` | 20.1 | 38.1 | +| `PairedVerifiedConstMap.get(String)` | 23.1 | 44.9 | +| `PairedVerifiedConstMap.getMany` | 20.7 | 28.9 | + +About 12 ns of a lookup at one million keys is hashing the string. ASCII strings are hashed +straight from their chars; other strings are first encoded to UTF-8, which costs more. +`get(byte[])` with a ready byte array takes about 11.8 ns at one million keys. + ## API summary | Type | Members | | --- | --- | | `BinaryFuse8`, `BinaryFuse16` | `build(long[])`, `build(long[], int)`, `buildFromBytes(Collection)` | | `BinaryFuseFilter` (interface) | `contains(long)`, `contains(byte[])`, `contains(byte[], int, int)`, `contains(ByteBuffer)`, `fingerprintBits()`, `keyCount()`, `serializedSize()`, `serialize(ByteBuffer)`, `toByteArray()`; static `wrap(ByteBuffer)`, `load(ByteBuffer)`, `load(byte[])` | +| `ConstMap` | static `build(List, long[])`, `buildFromBytes(List, long[])`, `buildFromHashes(long[], long[])`, `load(ByteBuffer)`, `load(byte[])`; `get(String)`, `get(byte[])`, `get(byte[], int, int)`, `get(ByteBuffer)`, `getByHash(long)`, `getMany(List)`, `getMany(List, long[])`, `arrayLength()`, `serializedSize()`, `serialize(ByteBuffer)`, `toByteArray()` | +| `VerifiedConstMap` | as `ConstMap`, plus `containsKey(String)`, `paired()` and the constant `NOT_FOUND` | +| `PairedVerifiedConstMap` | as `ConstMap`, plus `containsKey(String)` | +| `XxHash64` | `hash(String)`, `hash(byte[])`, `hash(byte[], int, int)`, `hash(byte[], int, int, long)`, `hash(ByteBuffer)`, `hash(ByteBuffer, int, int, long)` | | `WyHash` | `hash(byte[])`, `hash(byte[], int, int)`, `hash(byte[], int, int, long)`, `hash(ByteBuffer)`, `hash(ByteBuffer, int, int, long)`, `hash64(long, long)` | | `InvalidFilterException` | thrown for corrupt or invalid serialized input | -All filters are immutable and safe for concurrent queries. +All filters and maps are immutable and safe for concurrent queries. ## Size and false-positive rate @@ -247,7 +340,12 @@ of the library will keep reading version 1. ``` The wyhash test vectors in `src/test/resources/wyhash_vectors.txt` were generated from the -reference C implementation by `tools/wyhash_vectors.c` (see the comment at the top of that file). +reference C implementation by `tools/wyhash_vectors.c` (see the comment at the top of that file), +and the XXH64 vectors in `src/test/resources/xxh64_vectors.txt` by `tools/xxh64_vectors.c`. +`src/test/resources/constmap_vectors.txt` holds the size and checksum of maps built by the Go +constmap package, written by `tools/constmap_vectors` (`go run .` in that directory), and +`src/test/resources/constmap/` holds the interoperability files of the constmap repository, +written by rsconstmap and fastconstmap. ## Release procedure @@ -270,7 +368,7 @@ The workflow uses the repository secrets `GPG_PRIVATE_KEY`, `GPG_PASSPHRASE`, `M ## License Apache License 2.0; see [LICENSE](LICENSE). The wyhash algorithm is by Wang Yi and is in the -public domain. +public domain. The XXH64 algorithm is by Yann Collet. ## Reference diff --git a/benchmarks/src/main/java/io/github/fastfilter/jbinfuse/bench/ConstMapBenchmark.java b/benchmarks/src/main/java/io/github/fastfilter/jbinfuse/bench/ConstMapBenchmark.java new file mode 100644 index 0000000..833e49f --- /dev/null +++ b/benchmarks/src/main/java/io/github/fastfilter/jbinfuse/bench/ConstMapBenchmark.java @@ -0,0 +1,106 @@ +package io.github.fastfilter.jbinfuse.bench; + +import io.github.fastfilter.jbinfuse.ConstMap; +import io.github.fastfilter.jbinfuse.PairedVerifiedConstMap; +import io.github.fastfilter.jbinfuse.VerifiedConstMap; +import java.util.ArrayList; +import java.util.List; +import java.util.SplittableRandom; +import java.util.concurrent.TimeUnit; +import org.openjdk.jmh.annotations.Benchmark; +import org.openjdk.jmh.annotations.BenchmarkMode; +import org.openjdk.jmh.annotations.Fork; +import org.openjdk.jmh.annotations.Measurement; +import org.openjdk.jmh.annotations.Mode; +import org.openjdk.jmh.annotations.OperationsPerInvocation; +import org.openjdk.jmh.annotations.OutputTimeUnit; +import org.openjdk.jmh.annotations.Param; +import org.openjdk.jmh.annotations.Scope; +import org.openjdk.jmh.annotations.Setup; +import org.openjdk.jmh.annotations.State; +import org.openjdk.jmh.annotations.Warmup; + +/** + * Average time of one lookup of a present string key, in random order, one at a time with + * {@code get(String)} or in batches with {@code getMany}. The query strings are allocated in + * query order, so that reading them does not add scattered loads that belong to the caller + * rather than to the map. + */ +@BenchmarkMode(Mode.AverageTime) +@OutputTimeUnit(TimeUnit.NANOSECONDS) +@Warmup(iterations = 3, time = 1) +@Measurement(iterations = 5, time = 1) +@Fork(1) +@State(Scope.Benchmark) +public class ConstMapBenchmark { + static final int QUERIES = 1 << 16; + + @Param({"1000000", "10000000"}) + int size; + + @Param({"const", "verified", "paired"}) + String kind; + + ConstMap cm; + VerifiedConstMap vm; + PairedVerifiedConstMap pm; + List queries; + long[] out; + + @Setup + public void setup() { + List keys = new ArrayList<>(size); + long[] values = new long[size]; + for (int i = 0; i < size; i++) { + keys.add("key-" + i + "-" + Integer.toHexString(i * 0x9E3779B9)); + values[i] = i; + } + switch (kind) { + case "const" -> cm = ConstMap.build(keys, values); + case "verified" -> vm = VerifiedConstMap.build(keys, values); + default -> pm = PairedVerifiedConstMap.build(keys, values); + } + SplittableRandom r = new SplittableRandom(size); + queries = new ArrayList<>(QUERIES); + for (int i = 0; i < QUERIES; i++) { + // A copy of the chars, not new String(key), which would share the key's array. + queries.add(new String(keys.get(r.nextInt(size)).toCharArray())); + } + out = new long[QUERIES]; + } + + @Benchmark + @OperationsPerInvocation(QUERIES) + public long get() { + long sum = 0; + switch (kind) { + case "const" -> { + for (String q : queries) { + sum += cm.get(q); + } + } + case "verified" -> { + for (String q : queries) { + sum += vm.get(q); + } + } + default -> { + for (String q : queries) { + sum += pm.get(q); + } + } + } + return sum; + } + + @Benchmark + @OperationsPerInvocation(QUERIES) + public long[] getMany() { + switch (kind) { + case "const" -> cm.getMany(queries, out); + case "verified" -> vm.getMany(queries, out); + default -> pm.getMany(queries, out); + } + return out; + } +} diff --git a/pom.xml b/pom.xml index c64525c..5d7fe37 100644 --- a/pom.xml +++ b/pom.xml @@ -7,7 +7,7 @@ jar jbinfuse - Small, fast binary fuse filters (8-bit and 16-bit) for Java, queryable off-heap from serialized bytes. + Small, fast binary fuse filters (8-bit and 16-bit) for Java, queryable off-heap from serialized bytes, and constant maps from keys to 64-bit values compatible with constmap (Go), rsconstmap (Rust) and fastconstmap (C/Python). https://github.com/FastFilter/jbinfuse diff --git a/src/main/java/io/github/fastfilter/jbinfuse/ConstMap.java b/src/main/java/io/github/fastfilter/jbinfuse/ConstMap.java new file mode 100644 index 0000000..cf0f733 --- /dev/null +++ b/src/main/java/io/github/fastfilter/jbinfuse/ConstMap.java @@ -0,0 +1,287 @@ +package io.github.fastfilter.jbinfuse; + +import java.nio.ByteBuffer; +import java.util.List; +import java.util.Objects; + +/** + * An immutable map from keys (strings or byte sequences) to 64-bit values, in about 9 bytes + * per key, that does not store the keys. A lookup is one XXH64 hash, three array reads and two + * XORs. + * + *

Looking up a key that was not in the map returns an arbitrary value: use + * {@link VerifiedConstMap} or {@link PairedVerifiedConstMap} if you need to detect missing + * keys. + * + *

This is a Java implementation of constmap + * (Go). Its serialized form is shared with constmap, with + * rsconstmap (Rust) and with + * fastconstmap (C and Python): a map + * built by any of them loads in the others. For the same keys and values, this class writes + * exactly the bytes that the Go and Rust implementations write. + * + *

String keys are hashed through their UTF-8 encoding, so {@code get("abc")} and + * {@code get("abc".getBytes(UTF_8))} are the same lookup. Values are 64-bit words; read them as + * unsigned if you need values above {@link Long#MAX_VALUE}. + * + *

Instances are immutable and thread-safe. + */ +public final class ConstMap { + private final ConstMaps.Params params; + private final long seed; + private final int segmentLength; + private final int segmentLengthMask; + private final long segmentCountLength; + private final long[] data; + + private ConstMap(ConstMaps.Params params, long[] data) { + this.params = params; + this.seed = params.seed(); + this.segmentLength = params.segmentLength(); + this.segmentLengthMask = params.segmentLengthMask(); + this.segmentCountLength = params.segmentCountLength(); + this.data = data; + } + + /** + * Builds a map from string keys and their values: {@code keys.get(i)} maps to + * {@code values[i]}. Keys must be distinct. Construction is deterministic. + * + * @param keys the keys, distinct + * @param values the values, one per key + * @return the map + * @throws IllegalArgumentException if the lengths differ, if two keys are equal (detected + * with high probability), or if there are too many keys for one map + */ + public static ConstMap build(List keys, long[] values) { + ConstMaps.checkLengths(keys.size(), values.length); + return buildFromHashes(ConstMaps.hashStrings(keys), values); + } + + /** + * Builds a map from byte-array keys and their values: {@code keys.get(i)} maps to + * {@code values[i]}. Keys must be distinct. Construction is deterministic. + * + * @param keys the keys, distinct + * @param values the values, one per key + * @return the map + * @throws IllegalArgumentException if the lengths differ, if two keys are equal (detected + * with high probability), or if there are too many keys for one map + */ + public static ConstMap buildFromBytes(List keys, long[] values) { + ConstMaps.checkLengths(keys.size(), values.length); + return buildFromHashes(ConstMaps.hashBytes(keys), values); + } + + /** + * Builds a map from the XXH64 hashes of the keys ({@link XxHash64#hash(byte[])}) and their + * values, for callers that hash keys as they stream them. Neither array is modified. + * + * @param keyHashes the XXH64 hashes (seed 0) of the keys, distinct + * @param values the values, one per key + * @return the map + * @throws IllegalArgumentException if the lengths differ, if two hashes are equal (detected + * with high probability), or if there are too many keys for one map + */ + public static ConstMap buildFromHashes(long[] keyHashes, long[] values) { + ConstMaps.checkLengths(keyHashes.length, values.length); + if (keyHashes.length == 0) { + return new ConstMap(ConstMaps.Params.EMPTY, new long[0]); + } + ConstMaps.Peeled p = ConstMaps.peel(keyHashes); + long[] data = new long[p.params().arrayLength()]; + ConstMaps.assign(p, values, data, null); + return new ConstMap(p.params(), data); + } + + /** + * Returns the value of a key whose XXH64 hash (seed 0) is given. Equivalent to the other + * {@code get} methods on the key itself. + * + * @param keyHash the XXH64 hash of the key + * @return the value if the key is in the map, an arbitrary value otherwise + */ + public long getByHash(long keyHash) { + if (data.length == 0) { + return 0; + } + long hash = ConstMaps.mixsplit(keyHash, seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + int h1 = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); + int h2 = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); + return data[h0] ^ data[h1] ^ data[h2]; + } + + /** + * Returns the value of a string key (hashed through its UTF-8 encoding). + * + * @param key the key + * @return the value if the key is in the map, an arbitrary value otherwise + */ + public long get(String key) { + return getByHash(XxHash64.hash(key)); + } + + /** + * Returns the value of a byte-array key. + * + * @param key the key + * @return the value if the key is in the map, an arbitrary value otherwise + */ + public long get(byte[] key) { + return getByHash(XxHash64.hash(key)); + } + + /** + * Returns the value of the key made of {@code len} bytes of {@code key} from {@code off}. + * + * @param key the array holding the key + * @param off the index of its first byte + * @param len its length + * @return the value if the key is in the map, an arbitrary value otherwise + * @throws IndexOutOfBoundsException if the range is not within the array + */ + public long get(byte[] key, int off, int len) { + return getByHash(XxHash64.hash(key, off, len)); + } + + /** + * Returns the value of the key made of the buffer's bytes from its position to its limit. + * The buffer is not modified. + * + * @param key the key + * @return the value if the key is in the map, an arbitrary value otherwise + */ + public long get(ByteBuffer key) { + return getByHash(XxHash64.hash(key)); + } + + /** + * Looks up many string keys at once: {@code dst[i]} receives the value of + * {@code keys.get(i)}, exactly as {@link #get(String)} would return it. Hashing every key + * of a block before reading the array lets the processor overlap the memory accesses of + * several lookups, which helps when the map is larger than the cache. + * + * @param keys the keys + * @param dst where the values go; only the first {@code keys.size()} entries are written + * @throws IndexOutOfBoundsException if {@code dst} is shorter than {@code keys} + */ + public void getMany(List keys, long[] dst) { + int n = keys.size(); + Objects.checkFromIndexSize(0, n, dst.length); + if (data.length == 0) { + java.util.Arrays.fill(dst, 0, n, 0); + return; + } + final long[] d = data; + final int[] h = new int[3 * BLOCK]; + for (int start = 0; start < n; start += BLOCK) { + int m = Math.min(BLOCK, n - start); + for (int j = 0; j < m; j++) { + long hash = ConstMaps.mixsplit(XxHash64.hash(keys.get(start + j)), seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + h[3 * j] = h0; + h[3 * j + 1] = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); + h[3 * j + 2] = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); + } + for (int j = 0; j < m; j++) { + dst[start + j] = d[h[3 * j]] ^ d[h[3 * j + 1]] ^ d[h[3 * j + 2]]; + } + } + } + + /** Keys per block in {@link #getMany}. */ + static final int BLOCK = 8; + + /** + * Looks up many string keys at once, as {@link #getMany(List, long[])} does, into a new + * array. + * + * @param keys the keys + * @return the values, one per key + */ + public long[] getMany(List keys) { + long[] dst = new long[keys.size()]; + getMany(keys, dst); + return dst; + } + + /** + * Returns the number of 64-bit words in the map's array: about 1.125 per key for large + * maps, more for small ones. + * + * @return the array length + */ + public int arrayLength() { + return data.length; + } + + /** + * Returns the number of bytes written by {@link #serialize(ByteBuffer)}: a 28-byte header, + * the array, and an 8-byte checksum. + * + * @return the serialized size in bytes + */ + public int serializedSize() { + return ConstMaps.CMAP_HEADER + 8 * data.length + 8; + } + + /** + * Writes the map at the buffer's current position, in the {@code CMAP0001} format shared + * with the Go, Rust and C implementations, and advances the position by + * {@link #serializedSize()}. The buffer's byte order is ignored and left unchanged. + * + * @param out the destination + * @throws java.nio.BufferOverflowException if fewer than {@link #serializedSize()} bytes + * remain + * @throws java.nio.ReadOnlyBufferException if the buffer is read-only + */ + public void serialize(ByteBuffer out) { + ConstMaps.write(out, ConstMaps.MAGIC_CMAP, params, data.length, false, data); + } + + /** + * Returns the serialized form of the map, as written by {@link #serialize(ByteBuffer)}. + * + * @return a new array of {@link #serializedSize()} bytes + */ + public byte[] toByteArray() { + byte[] bytes = new byte[serializedSize()]; + serialize(ByteBuffer.wrap(bytes)); + return bytes; + } + + /** + * Reads a serialized map starting at the buffer's position into a new on-heap map, after + * validating its header and checksum. Accepts both the {@code CMAP0001} format (Go, Rust, + * this class) and fastconstmap's {@code CMAP0003}, which adds a key count that is ignored. + * The buffer's position is not changed. + * + * @param buffer a buffer holding a serialized map at its position + * @return the map + * @throws InvalidFilterException if the input is not a valid {@code ConstMap} (including a + * {@code VerifiedConstMap} or {@code PairedVerifiedConstMap}, or a file from + * fastconstmap 0.9 or earlier, which used another hash), or is corrupt + */ + public static ConstMap load(ByteBuffer buffer) { + ConstMaps.Header h = ConstMaps.read(buffer, ConstMaps.Kind.CONST, 1); + int n = h.params().arrayLength(); + return new ConstMap(h.params(), ConstMaps.readWords(buffer, h.start(), n)); + } + + /** + * Reads a serialized map, as {@link #load(ByteBuffer)} does. + * + * @param bytes the serialized map + * @return the map + * @throws InvalidFilterException if the input is invalid or corrupt + */ + public static ConstMap load(byte[] bytes) { + return load(ByteBuffer.wrap(bytes)); + } + + @Override + public String toString() { + return "ConstMap{words=" + data.length + ", bytes=" + serializedSize() + "}"; + } +} diff --git a/src/main/java/io/github/fastfilter/jbinfuse/ConstMaps.java b/src/main/java/io/github/fastfilter/jbinfuse/ConstMaps.java new file mode 100644 index 0000000..096ab33 --- /dev/null +++ b/src/main/java/io/github/fastfilter/jbinfuse/ConstMaps.java @@ -0,0 +1,557 @@ +package io.github.fastfilter.jbinfuse; + +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.charset.StandardCharsets; +import java.util.Arrays; +import java.util.List; +import java.util.Objects; + +/** + * Construction and serialization shared by {@link ConstMap}, {@link VerifiedConstMap} and + * {@link PairedVerifiedConstMap}. + * + *

Everything here follows constmap (Go) + * exactly: the key hash (XXH64, seed 0), the mixing ({@code murmur64(hash + seed)}), the seed + * sequence (splitmix64 from 1), the segment parameters, the retry schedule and the peeling + * order. For the same keys and values the three maps therefore hold the same words as the Go, + * Rust (rsconstmap) and C/Python + * (fastconstmap) implementations, and + * serialize to the same bytes as Go and Rust. + */ +final class ConstMaps { + private ConstMaps() {} + + /** Returned by verified lookups for keys that are not in the map. */ + static final long NOT_FOUND = -1L; + + static final int MAX_SEGMENT_LENGTH = 262144; + static final int MAX_ITERATIONS = 100; + + /** + * Largest array length we build or load: a paired map holds two words per position in one + * Java array. + */ + static final int MAX_ARRAY_LENGTH = (Integer.MAX_VALUE - 16) / 2; + + // --------------------------------------------------------------------------------------- + // Hashing. + + static long murmur64(long h) { + h ^= h >>> 33; + h *= 0xff51afd7ed558ccdL; + h ^= h >>> 33; + h *= 0xc4ceb9fe1a85ec53L; + h ^= h >>> 33; + return h; + } + + static long mixsplit(long key, long seed) { + return murmur64(key + seed); + } + + /** The verification word derived from a mixed hash. */ + static long fingerprint(long hash) { + return hash ^ (hash >>> 32); + } + + static long[] hashStrings(List keys) { + long[] h = new long[keys.size()]; + int i = 0; + for (String k : keys) { + h[i++] = XxHash64.hash(k); + } + return h; + } + + static long[] hashBytes(List keys) { + long[] h = new long[keys.size()]; + int i = 0; + for (byte[] k : keys) { + h[i++] = XxHash64.hash(k); + } + return h; + } + + // --------------------------------------------------------------------------------------- + // Layout. + + /** + * The parameters of a map. Positions are {@code h0 = mulhi(hash, segmentCount * + * segmentLength)}, {@code h1 = (h0 + segmentLength) ^ ((hash >>> 18) & mask)} and {@code h2 = + * (h0 + 2 * segmentLength) ^ (hash & mask)}, all below {@code arrayLength = (segmentCount + + * 2) * segmentLength}. + */ + record Params(long seed, int segmentLength, int segmentCount, int arrayLength) { + static final Params EMPTY = new Params(0, 0, 0, 0); + + int segmentLengthMask() { + return segmentLength - 1; + } + + long segmentCountLength() { + return (long) segmentCount * segmentLength; + } + } + + static int calculateSegmentLength(int size) { + if (size == 0) { + return 4; + } + return 1 << (int) Math.floor(Math.log(size) / Math.log(3.33) + 2.25); + } + + static double calculateSizeFactor(int size) { + return Math.max(1.125, 0.875 + 0.25 * Math.log(1000000) / Math.log(size)); + } + + // --------------------------------------------------------------------------------------- + // Construction. + + /** + * The outcome of peeling: the seed and parameters that worked, and for each stack entry + * (to be assigned in reverse) the mixed hash, which of its three positions it owns, and the + * index of the key in the caller's arrays. + */ + record Peeled(Params params, long[] order, byte[] which, int[] keyIndex) {} + + static void checkLengths(int keys, int values) { + if (keys != values) { + throw new IllegalArgumentException( + "keys and values must have equal length (" + keys + " vs " + values + ")"); + } + } + + /** + * Finds a seed and an assignment order for keys with the given XXH64 hashes. A port of the + * construction loop of constmap's {@code New}; where the Go code recovers each key's value + * through a map from mixed hash to value, this keeps the XOR of the key indices in each + * cell next to the XOR of their hashes, so a singleton cell names its key directly. The + * peeling, and so the result, is the same. + * + * @param hashes the XXH64 key hashes; must not be empty + */ + static Peeled peel(long[] hashes) { + final int size = hashes.length; + int segmentLength = Math.min(calculateSegmentLength(size), MAX_SEGMENT_LENGTH); + long cap0 = 0; + if (size > 1) { + cap0 = Math.round(size * calculateSizeFactor(size)); + } + long totalSegmentCount = Math.max(3, (cap0 + segmentLength - 1) / segmentLength); + long arrayLengthLong = totalSegmentCount * segmentLength; + if (arrayLengthLong > MAX_ARRAY_LENGTH) { + throw new IllegalArgumentException("too many keys for one map: " + size); + } + final int capacity = (int) arrayLengthLong; + int segmentCount = (int) totalSegmentCount - 2; + + long rngcounter = 1; + rngcounter += 0x9E3779B97F4A7C15L; + long seed = splitmix64Value(rngcounter); + + final int[] alone = new int[capacity]; + final byte[] t2count = new byte[capacity]; + final long[] t2hash = new long[capacity]; + final int[] t2index = new int[capacity]; + final byte[] reverseH = new byte[size]; + final long[] reverseOrder = new long[size + 1]; + final int[] reverseIndex = new int[size]; + final int[] bucketIndex = new int[size]; + reverseOrder[size] = 1; + + for (int iterations = 0; ; iterations++) { + if (iterations > MAX_ITERATIONS) { + throw new IllegalArgumentException("failed to construct map, possible duplicate keys"); + } + if (size > 4 && size < 1_000_000) { + switch (iterations % 4) { + case 2 -> { + segmentLength /= 2; + segmentCount = segmentCount * 2 + 2; + } + case 3 -> { + segmentLength *= 2; + segmentCount = segmentCount / 2 - 1; + } + default -> { } + } + } + final int segLen = segmentLength; + final int mask = segLen - 1; + final long scl = (long) segmentCount * segLen; + + int blockBits = 1; + while ((1L << blockBits) < segmentCount) { + blockBits++; + } + final int[] startPos = new int[1 << blockBits]; + for (int i = 0; i < startPos.length; i++) { + startPos[i] = (int) (((long) i * size) >>> blockBits); + } + final int blockMask = (1 << blockBits) - 1; + for (int k = 0; k < size; k++) { + long hash = mixsplit(hashes[k], seed); + int segmentIndex = (int) (hash >>> (64 - blockBits)); + while (reverseOrder[startPos[segmentIndex]] != 0) { + segmentIndex = (segmentIndex + 1) & blockMask; + } + reverseOrder[startPos[segmentIndex]] = hash; + bucketIndex[startPos[segmentIndex]] = k; + startPos[segmentIndex]++; + } + + boolean hasError = false; + for (int i = 0; i < size; i++) { + long hash = reverseOrder[i]; + int key = bucketIndex[i]; + int i1 = (int) Math.unsignedMultiplyHigh(hash, scl); + int i2 = (i1 + segLen) ^ ((int) (hash >>> 18) & mask); + int i3 = (i1 + 2 * segLen) ^ ((int) hash & mask); + t2count[i1] += 4; + t2hash[i1] ^= hash; + t2index[i1] ^= key; + t2count[i2] = (byte) ((t2count[i2] + 4) ^ 1); + t2hash[i2] ^= hash; + t2index[i2] ^= key; + t2count[i3] = (byte) ((t2count[i3] + 4) ^ 2); + t2hash[i3] ^= hash; + t2index[i3] ^= key; + if ((t2hash[i1] & t2hash[i2] & t2hash[i3]) == 0) { + if ((t2hash[i1] == 0 && t2count[i1] == 8) + || (t2hash[i2] == 0 && t2count[i2] == 8) + || (t2hash[i3] == 0 && t2count[i3] == 8)) { + throw new IllegalArgumentException("duplicate key hash detected"); + } + } + if ((t2count[i1] & 0xFF) < 4 || (t2count[i2] & 0xFF) < 4 + || (t2count[i3] & 0xFF) < 4) { + hasError = true; + } + } + + int stacksize = 0; + if (!hasError) { + int qsize = 0; + for (int i = 0; i < capacity; i++) { + alone[qsize] = i; + if (((t2count[i] & 0xFF) >>> 2) == 1) { + qsize++; + } + } + final int segLenToMinusSegLenX2 = segLen ^ (-(2 * segLen)); + while (qsize > 0) { + qsize--; + int index = alone[qsize]; + if (((t2count[index] & 0xFF) >>> 2) == 1) { + long hash = t2hash[index]; + int found = t2count[index] & 3; + reverseH[stacksize] = (byte) found; + reverseOrder[stacksize] = hash; + int key = t2index[index]; + reverseIndex[stacksize] = key; + stacksize++; + + int h01 = (int) (hash >>> 18) & mask; + int h02 = (int) hash & mask; + int is0 = -(((found - 1) & 0xFF) >>> 7); + int is1 = -(found & 1); + int is2 = -(found >>> 1); + int other1 = index + (segLen ^ (segLenToMinusSegLenX2 & is2)); + int other2 = index - (segLen ^ (segLenToMinusSegLenX2 & is0)); + other1 ^= (h01 & ~is2) ^ (h02 & ~is0); + other2 ^= (h01 & ~is0) ^ (h02 & ~is1); + int f1 = (is0 & 1) | (is1 & 2); + int f2 = (is0 & 2) | (is2 & 1); + + alone[qsize] = other1; + if (((t2count[other1] & 0xFF) >>> 2) == 2) { + qsize++; + } + t2count[other1] = (byte) ((t2count[other1] - 4) ^ f1); + t2hash[other1] ^= hash; + t2index[other1] ^= key; + + alone[qsize] = other2; + if (((t2count[other2] & 0xFF) >>> 2) == 2) { + qsize++; + } + t2count[other2] = (byte) ((t2count[other2] - 4) ^ f2); + t2hash[other2] ^= hash; + t2index[other2] ^= key; + } + } + } + if (stacksize == size) { + Params params = new Params(seed, segLen, segmentCount, capacity); + return new Peeled(params, Arrays.copyOf(reverseOrder, size), reverseH, reverseIndex); + } + Arrays.fill(reverseOrder, 0, size, 0); + Arrays.fill(t2count, (byte) 0); + Arrays.fill(t2hash, 0); + Arrays.fill(t2index, 0); + rngcounter += 0x9E3779B97F4A7C15L; + seed = splitmix64Value(rngcounter); + } + } + + /** The output of splitmix64 for an already-advanced state. */ + private static long splitmix64Value(long z) { + z = (z ^ (z >>> 30)) * 0xBF58476D1CE4E5B9L; + z = (z ^ (z >>> 27)) * 0x94D049BB133111EBL; + return z ^ (z >>> 31); + } + + /** + * Fills {@code data} (and, if not null, {@code checks}) so that the three positions of each + * key XOR to its value (and to its fingerprint). + */ + static void assign(Peeled p, long[] values, long[] data, long[] checks) { + final Params params = p.params(); + final int segLen = params.segmentLength(); + final int mask = params.segmentLengthMask(); + final long scl = params.segmentCountLength(); + final long[] order = p.order(); + final byte[] which = p.which(); + final int[] keyIndex = p.keyIndex(); + for (int i = order.length - 1; i >= 0; i--) { + long hash = order[i]; + int h0 = (int) Math.unsignedMultiplyHigh(hash, scl); + int h1 = (h0 + segLen) ^ ((int) (hash >>> 18) & mask); + int h2 = (h0 + 2 * segLen) ^ ((int) hash & mask); + int a; + int b; + int c; + switch (which[i]) { + case 0 -> { + a = h0; + b = h1; + c = h2; + } + case 1 -> { + a = h1; + b = h2; + c = h0; + } + default -> { + a = h2; + b = h0; + c = h1; + } + } + data[a] = values[keyIndex[i]] ^ data[b] ^ data[c]; + if (checks != null) { + checks[a] = fingerprint(hash) ^ checks[b] ^ checks[c]; + } + } + } + + // --------------------------------------------------------------------------------------- + // Serialization. + // + // All three formats are little-endian and end with the 64-bit FNV-1a hash of every byte + // before it: + // + // ConstMap "CMAP0001" seed:8 segmentLength:4 segmentCount:4 dataLength:4 + // data:8*dataLength fnv:8 (28-byte header) + // fastconstmap writes "CMAP0003": the same with a 4-byte key count + // after dataLength (32-byte header). Both are read. + // VerifiedConstMap "VMAP0001" seed:8 segmentLength:4 segmentCount:4 dataLength:4 + // pad:4 data:8*dataLength checks:8*dataLength fnv:8 + // PairedVerified... "PMAP0001" seed:8 segmentLength:4 segmentCount:4 slotCount:4 + // pad:4 (value:8 check:8)*slotCount fnv:8 + // + // fastconstmap stores its key count in the pad word; every reader ignores it. + + static final long FNV_OFFSET = 0xcbf29ce484222325L; + static final long FNV_PRIME = 0x100000001b3L; + + static final long MAGIC_CMAP = magic("CMAP0001"); + static final long MAGIC_CMAP_COUNTED = magic("CMAP0003"); + static final long MAGIC_VMAP = magic("VMAP0001"); + static final long MAGIC_PMAP = magic("PMAP0001"); + static final long MAGIC_LEGACY_CMAP = magic("CMAP0002"); + static final long MAGIC_LEGACY_VMAP = magic("VCMP0002"); + + static final int CMAP_HEADER = 28; + static final int VMAP_HEADER = 32; + + /** The kinds of map, for naming a file handed to the wrong reader. */ + enum Kind { + CONST("ConstMap"), + VERIFIED("VerifiedConstMap"), + PAIRED("PairedVerifiedConstMap"); + + final String className; + + Kind(String className) { + this.className = className; + } + } + + private static long magic(String s) { + byte[] b = s.getBytes(StandardCharsets.US_ASCII); + return ByteBuffer.wrap(b).order(ByteOrder.LITTLE_ENDIAN).getLong(); + } + + static long fnv(long h, long word) { + for (int i = 0; i < 8; i++) { + h ^= word & 0xFF; + h *= FNV_PRIME; + word >>>= 8; + } + return h; + } + + static long fnv32(long h, int word) { + for (int i = 0; i < 4; i++) { + h ^= word & 0xFF; + h *= FNV_PRIME; + word >>>= 8; + } + return h; + } + + /** A little-endian view of {@code out} at its position, after checking the space. */ + static ByteBuffer writer(ByteBuffer out, long size) { + Objects.requireNonNull(out); + if (out.isReadOnly()) { + throw new java.nio.ReadOnlyBufferException(); + } + if (out.remaining() < size) { + throw new java.nio.BufferOverflowException(); + } + return out.duplicate().order(ByteOrder.LITTLE_ENDIAN); + } + + /** + * Writes a header and the given arrays (each one after the other) and the checksum, and + * advances {@code out}'s position. + */ + static void write(ByteBuffer out, long magic, Params p, int count, boolean pad, + long[]... arrays) { + long size = (pad ? VMAP_HEADER : CMAP_HEADER) + 8L; + for (long[] a : arrays) { + size += 8L * a.length; + } + ByteBuffer w = writer(out, size); + long h = FNV_OFFSET; + w.putLong(magic); + h = fnv(h, magic); + w.putLong(p.seed()); + h = fnv(h, p.seed()); + w.putInt(p.segmentLength()); + h = fnv32(h, p.segmentLength()); + w.putInt(p.segmentCount()); + h = fnv32(h, p.segmentCount()); + w.putInt(count); + h = fnv32(h, count); + if (pad) { + w.putInt(0); + h = fnv32(h, 0); + } + for (long[] a : arrays) { + for (long v : a) { + w.putLong(v); + h = fnv(h, v); + } + } + w.putLong(h); + out.position(w.position()); + } + + /** A parsed header: the parameters, the header size and where the arrays start. */ + record Header(Params params, int headerSize, int start) {} + + /** + * Reads and validates the header of a map of the given kind at the buffer's position, and + * verifies the trailing checksum over {@code wordsPerPosition} words per array position. + */ + static Header read(ByteBuffer buffer, Kind kind, int wordsPerPosition) { + ByteBuffer b = buffer.duplicate().order(ByteOrder.LITTLE_ENDIAN); + final int start = b.position(); + if (b.remaining() < 8) { + throw new InvalidFilterException("truncated input: no room for the magic bytes"); + } + long magic = b.getLong(start); + boolean counted = false; + Kind found; + if (magic == MAGIC_CMAP) { + found = Kind.CONST; + } else if (magic == MAGIC_CMAP_COUNTED) { + found = Kind.CONST; + counted = true; + } else if (magic == MAGIC_VMAP) { + found = Kind.VERIFIED; + } else if (magic == MAGIC_PMAP) { + found = Kind.PAIRED; + } else if (magic == MAGIC_LEGACY_CMAP || magic == MAGIC_LEGACY_VMAP) { + throw new InvalidFilterException("this file was written by fastconstmap 0.9 or earlier" + + " with a different key hash (XXH3); rebuild it from its keys with fastconstmap" + + " 0.10 or later, which writes the shared format"); + } else { + throw new InvalidFilterException("invalid magic bytes: not a constant map"); + } + if (found != kind) { + throw new InvalidFilterException("this is a " + found.className + ", not a " + + kind.className + "; load it with " + found.className + ".load"); + } + final int headerSize = (kind == Kind.CONST && !counted) ? CMAP_HEADER : VMAP_HEADER; + if (b.remaining() < headerSize) { + throw new InvalidFilterException("truncated input: " + b.remaining() + + " bytes, the header alone is " + headerSize); + } + long seed = b.getLong(start + 8); + int segmentLength = b.getInt(start + 16); + int segmentCount = b.getInt(start + 20); + long count = b.getInt(start + 24) & 0xFFFFFFFFL; + if (count > MAX_ARRAY_LENGTH) { + throw new InvalidFilterException("map too large for Java arrays: " + count + " positions"); + } + long total = headerSize + 8L * wordsPerPosition * count + 8; + if (b.remaining() < total) { + throw new InvalidFilterException("truncated input: " + b.remaining() + + " bytes, expected " + total); + } + Params params; + if (count == 0) { + params = Params.EMPTY; + } else { + if (segmentLength <= 0 || (segmentLength & (segmentLength - 1)) != 0) { + throw new InvalidFilterException( + "segment length " + Integer.toUnsignedString(segmentLength) + " is not a power of two"); + } + if (segmentCount <= 0) { + throw new InvalidFilterException("invalid segment count " + + Integer.toUnsignedString(segmentCount)); + } + if (((long) segmentCount + 2) * segmentLength != count) { + throw new InvalidFilterException(count + " positions but the segment parameters describe " + + ((long) segmentCount + 2) * segmentLength); + } + params = new Params(seed, segmentLength, segmentCount, (int) count); + } + long h = FNV_OFFSET; + int end = (int) (start + total - 8); + int i = start; + for (; i + 8 <= end; i += 8) { + h = fnv(h, b.getLong(i)); + } + if (i < end) { + h = fnv32(h, b.getInt(i)); + } + long stored = b.getLong(end); + if (stored != h) { + throw new InvalidFilterException(String.format( + "checksum mismatch (stored %016x, computed %016x)", stored, h)); + } + return new Header(params, headerSize, start + headerSize); + } + + /** Reads {@code n} little-endian words starting at absolute index {@code from}. */ + static long[] readWords(ByteBuffer buffer, int from, int n) { + long[] words = new long[n]; + buffer.duplicate().order(ByteOrder.LITTLE_ENDIAN).position(from).asLongBuffer().get(words); + return words; + } +} diff --git a/src/main/java/io/github/fastfilter/jbinfuse/InvalidFilterException.java b/src/main/java/io/github/fastfilter/jbinfuse/InvalidFilterException.java index 1feb02e..cd03483 100644 --- a/src/main/java/io/github/fastfilter/jbinfuse/InvalidFilterException.java +++ b/src/main/java/io/github/fastfilter/jbinfuse/InvalidFilterException.java @@ -1,9 +1,9 @@ package io.github.fastfilter.jbinfuse; /** - * Thrown when serialized bytes are not a valid binary fuse filter: wrong magic bytes, an - * unsupported format version or variant, inconsistent parameters, truncated input or a - * checksum mismatch. + * Thrown when serialized bytes are not a valid binary fuse filter or constant map: wrong magic + * bytes, an unsupported format version or variant, inconsistent parameters, truncated input or + * a checksum mismatch. */ public final class InvalidFilterException extends IllegalArgumentException { private static final long serialVersionUID = 1L; diff --git a/src/main/java/io/github/fastfilter/jbinfuse/PairedVerifiedConstMap.java b/src/main/java/io/github/fastfilter/jbinfuse/PairedVerifiedConstMap.java new file mode 100644 index 0000000..0cf170a --- /dev/null +++ b/src/main/java/io/github/fastfilter/jbinfuse/PairedVerifiedConstMap.java @@ -0,0 +1,298 @@ +package io.github.fastfilter.jbinfuse; + +import java.nio.ByteBuffer; +import java.util.Arrays; +import java.util.List; +import java.util.Objects; + +/** + * A {@link VerifiedConstMap} with each value stored next to its check word, so that a lookup + * touches three cache lines instead of six. Same answers, same size (about 18 bytes per key); + * faster when most keys looked up are present, slower when most are missing (the verified + * layout then reads only its check array, half the size of the map). + * + *

This is a Java implementation of the {@code PairedVerifiedConstMap} of + * constmap (Go). Its serialized form + * ({@code PMAP0001}) is shared with constmap, rsconstmap (Rust) and fastconstmap (C and + * Python), and for the same keys and values this class writes exactly the bytes that the Go + * and Rust implementations write. See {@link ConstMap} for how keys are hashed. + * + *

Instances are immutable and thread-safe. + */ +public final class PairedVerifiedConstMap { + private final ConstMaps.Params params; + private final long seed; + private final int segmentLength; + private final int segmentLengthMask; + private final long segmentCountLength; + /** The value for position i at slots[2i], its check word at slots[2i+1]. */ + private final long[] slots; + + private PairedVerifiedConstMap(ConstMaps.Params params, long[] slots) { + this.params = params; + this.seed = params.seed(); + this.segmentLength = params.segmentLength(); + this.segmentLengthMask = params.segmentLengthMask(); + this.segmentCountLength = params.segmentCountLength(); + this.slots = slots; + } + + static PairedVerifiedConstMap of(ConstMaps.Params params, long[] slots) { + return new PairedVerifiedConstMap(params, slots); + } + + /** + * Builds a map from string keys and their values: {@code keys.get(i)} maps to + * {@code values[i]}. Keys must be distinct. Construction is deterministic. + * + * @param keys the keys, distinct + * @param values the values, one per key + * @return the map + * @throws IllegalArgumentException if the lengths differ, if two keys are equal (detected + * with high probability), or if there are too many keys for one map + */ + public static PairedVerifiedConstMap build(List keys, long[] values) { + return VerifiedConstMap.build(keys, values).paired(); + } + + /** + * Builds a map from byte-array keys and their values: {@code keys.get(i)} maps to + * {@code values[i]}. Keys must be distinct. Construction is deterministic. + * + * @param keys the keys, distinct + * @param values the values, one per key + * @return the map + * @throws IllegalArgumentException if the lengths differ, if two keys are equal (detected + * with high probability), or if there are too many keys for one map + */ + public static PairedVerifiedConstMap buildFromBytes(List keys, long[] values) { + return VerifiedConstMap.buildFromBytes(keys, values).paired(); + } + + /** + * Builds a map from the XXH64 hashes of the keys ({@link XxHash64#hash(byte[])}) and their + * values. Neither array is modified. + * + * @param keyHashes the XXH64 hashes (seed 0) of the keys, distinct + * @param values the values, one per key + * @return the map + * @throws IllegalArgumentException if the lengths differ, if two hashes are equal (detected + * with high probability), or if there are too many keys for one map + */ + public static PairedVerifiedConstMap buildFromHashes(long[] keyHashes, long[] values) { + return VerifiedConstMap.buildFromHashes(keyHashes, values).paired(); + } + + /** + * Returns the value of a key whose XXH64 hash (seed 0) is given. Equivalent to the other + * {@code get} methods on the key itself. + * + * @param keyHash the XXH64 hash of the key + * @return the value, or {@link VerifiedConstMap#NOT_FOUND} if the key is not in the map + */ + public long getByHash(long keyHash) { + if (slots.length == 0) { + return ConstMaps.NOT_FOUND; + } + long hash = ConstMaps.mixsplit(keyHash, seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + int h1 = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); + int h2 = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); + final long[] s = slots; + int i0 = 2 * h0; + int i1 = 2 * h1; + int i2 = 2 * h2; + if ((s[i0 + 1] ^ s[i1 + 1] ^ s[i2 + 1]) != ConstMaps.fingerprint(hash)) { + return ConstMaps.NOT_FOUND; + } + return s[i0] ^ s[i1] ^ s[i2]; + } + + /** + * Returns the value of a string key (hashed through its UTF-8 encoding). + * + * @param key the key + * @return the value, or {@link VerifiedConstMap#NOT_FOUND} if the key is not in the map + */ + public long get(String key) { + return getByHash(XxHash64.hash(key)); + } + + /** + * Returns the value of a byte-array key. + * + * @param key the key + * @return the value, or {@link VerifiedConstMap#NOT_FOUND} if the key is not in the map + */ + public long get(byte[] key) { + return getByHash(XxHash64.hash(key)); + } + + /** + * Returns the value of the key made of {@code len} bytes of {@code key} from {@code off}. + * + * @param key the array holding the key + * @param off the index of its first byte + * @param len its length + * @return the value, or {@link VerifiedConstMap#NOT_FOUND} if the key is not in the map + * @throws IndexOutOfBoundsException if the range is not within the array + */ + public long get(byte[] key, int off, int len) { + return getByHash(XxHash64.hash(key, off, len)); + } + + /** + * Returns the value of the key made of the buffer's bytes from its position to its limit. + * The buffer is not modified. + * + * @param key the key + * @return the value, or {@link VerifiedConstMap#NOT_FOUND} if the key is not in the map + */ + public long get(ByteBuffer key) { + return getByHash(XxHash64.hash(key)); + } + + /** + * Returns whether a string key is in the map, that is, whether {@link #get(String)} does not + * return {@link VerifiedConstMap#NOT_FOUND}. + * + * @param key the key + * @return true if the key is in the map (or, with probability about 2-64, a + * false positive), false if it is not or if its value is + * {@link VerifiedConstMap#NOT_FOUND} + */ + public boolean containsKey(String key) { + return get(key) != ConstMaps.NOT_FOUND; + } + + /** + * Looks up many string keys at once: {@code dst[i]} receives the value of + * {@code keys.get(i)}, exactly as {@link #get(String)} would return it. Hashing every key + * of a block before reading the array lets the processor overlap the memory accesses of + * several lookups. + * + * @param keys the keys + * @param dst where the values go; only the first {@code keys.size()} entries are written + * @throws IndexOutOfBoundsException if {@code dst} is shorter than {@code keys} + */ + public void getMany(List keys, long[] dst) { + int n = keys.size(); + Objects.checkFromIndexSize(0, n, dst.length); + if (slots.length == 0) { + Arrays.fill(dst, 0, n, ConstMaps.NOT_FOUND); + return; + } + final long[] s = slots; + final int[] h = new int[3 * ConstMap.BLOCK]; + final long[] fp = new long[ConstMap.BLOCK]; + for (int start = 0; start < n; start += ConstMap.BLOCK) { + int m = Math.min(ConstMap.BLOCK, n - start); + for (int j = 0; j < m; j++) { + long hash = ConstMaps.mixsplit(XxHash64.hash(keys.get(start + j)), seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + h[3 * j] = 2 * h0; + h[3 * j + 1] = 2 * ((h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask)); + h[3 * j + 2] = 2 * ((h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask)); + fp[j] = ConstMaps.fingerprint(hash); + } + for (int j = 0; j < m; j++) { + int a = h[3 * j]; + int b = h[3 * j + 1]; + int c = h[3 * j + 2]; + long v = s[a] ^ s[b] ^ s[c]; + dst[start + j] = (s[a + 1] ^ s[b + 1] ^ s[c + 1]) == fp[j] ? v : ConstMaps.NOT_FOUND; + } + } + } + + /** + * Looks up many string keys at once, as {@link #getMany(List, long[])} does, into a new + * array. + * + * @param keys the keys + * @return the values, one per key, {@link VerifiedConstMap#NOT_FOUND} for missing keys + */ + public long[] getMany(List keys) { + long[] dst = new long[keys.size()]; + getMany(keys, dst); + return dst; + } + + /** + * Returns the number of positions (value and check word pairs) in the map: about 1.125 per + * key for large maps, more for small ones. + * + * @return the number of positions + */ + public int arrayLength() { + return slots.length / 2; + } + + /** + * Returns the number of bytes written by {@link #serialize(ByteBuffer)}: a 32-byte header, + * the slots, and an 8-byte checksum. + * + * @return the serialized size in bytes + */ + public int serializedSize() { + return ConstMaps.VMAP_HEADER + 8 * slots.length + 8; + } + + /** + * Writes the map at the buffer's current position, in the {@code PMAP0001} format shared + * with the Go, Rust and C implementations, and advances the position by + * {@link #serializedSize()}. The buffer's byte order is ignored and left unchanged. + * + * @param out the destination + * @throws java.nio.BufferOverflowException if fewer than {@link #serializedSize()} bytes + * remain + * @throws java.nio.ReadOnlyBufferException if the buffer is read-only + */ + public void serialize(ByteBuffer out) { + ConstMaps.write(out, ConstMaps.MAGIC_PMAP, params, slots.length / 2, true, slots); + } + + /** + * Returns the serialized form of the map, as written by {@link #serialize(ByteBuffer)}. + * + * @return a new array of {@link #serializedSize()} bytes + */ + public byte[] toByteArray() { + byte[] bytes = new byte[serializedSize()]; + serialize(ByteBuffer.wrap(bytes)); + return bytes; + } + + /** + * Reads a serialized map ({@code PMAP0001}) starting at the buffer's position into a new + * on-heap map, after validating its header and checksum. The buffer's position is not + * changed. + * + * @param buffer a buffer holding a serialized map at its position + * @return the map + * @throws InvalidFilterException if the input is not a valid {@code PairedVerifiedConstMap} + * or is corrupt + */ + public static PairedVerifiedConstMap load(ByteBuffer buffer) { + ConstMaps.Header h = ConstMaps.read(buffer, ConstMaps.Kind.PAIRED, 2); + int n = h.params().arrayLength(); + return new PairedVerifiedConstMap(h.params(), ConstMaps.readWords(buffer, h.start(), 2 * n)); + } + + /** + * Reads a serialized map, as {@link #load(ByteBuffer)} does. + * + * @param bytes the serialized map + * @return the map + * @throws InvalidFilterException if the input is invalid or corrupt + */ + public static PairedVerifiedConstMap load(byte[] bytes) { + return load(ByteBuffer.wrap(bytes)); + } + + @Override + public String toString() { + return "PairedVerifiedConstMap{positions=" + arrayLength() + ", bytes=" + serializedSize() + + "}"; + } +} diff --git a/src/main/java/io/github/fastfilter/jbinfuse/VerifiedConstMap.java b/src/main/java/io/github/fastfilter/jbinfuse/VerifiedConstMap.java new file mode 100644 index 0000000..87f6a25 --- /dev/null +++ b/src/main/java/io/github/fastfilter/jbinfuse/VerifiedConstMap.java @@ -0,0 +1,329 @@ +package io.github.fastfilter.jbinfuse; + +import java.nio.ByteBuffer; +import java.util.Arrays; +import java.util.List; +import java.util.Objects; + +/** + * An immutable map from keys (strings or byte sequences) to 64-bit values that detects + * missing keys, in about 18 bytes per key, without storing the keys. Next to the array of + * values of a {@link ConstMap} it keeps an array of 64-bit check words: a lookup returns + * {@link #NOT_FOUND} for a key that was not in the map, except with probability about + * 2-64. + * + *

A lookup of a present key reads three check words and three values, in two arrays. A + * lookup of a missing key usually stops after the check words, so this layout is the better + * one when most keys looked up are missing; {@link PairedVerifiedConstMap} stores each value + * next to its check word and is faster when most are present. {@link #paired()} converts. + * + *

This is a Java implementation of the {@code VerifiedConstMap} of + * constmap (Go). Its serialized form + * ({@code VMAP0001}) is shared with constmap, rsconstmap (Rust) and fastconstmap (C and + * Python), and for the same keys and values this class writes exactly the bytes that the Go + * and Rust implementations write. See {@link ConstMap} for how keys are hashed. + * + *

Instances are immutable and thread-safe. + */ +public final class VerifiedConstMap { + /** + * The value returned for a key that is not in the map: all 64 bits set ({@code -1}, or + * 264-1 read as unsigned). A key mapped to this value cannot be told apart from a + * missing key. + */ + public static final long NOT_FOUND = ConstMaps.NOT_FOUND; + + private final ConstMaps.Params params; + private final long seed; + private final int segmentLength; + private final int segmentLengthMask; + private final long segmentCountLength; + private final long[] data; + private final long[] checks; + + private VerifiedConstMap(ConstMaps.Params params, long[] data, long[] checks) { + this.params = params; + this.seed = params.seed(); + this.segmentLength = params.segmentLength(); + this.segmentLengthMask = params.segmentLengthMask(); + this.segmentCountLength = params.segmentCountLength(); + this.data = data; + this.checks = checks; + } + + /** + * Builds a map from string keys and their values: {@code keys.get(i)} maps to + * {@code values[i]}. Keys must be distinct. Construction is deterministic. + * + * @param keys the keys, distinct + * @param values the values, one per key + * @return the map + * @throws IllegalArgumentException if the lengths differ, if two keys are equal (detected + * with high probability), or if there are too many keys for one map + */ + public static VerifiedConstMap build(List keys, long[] values) { + ConstMaps.checkLengths(keys.size(), values.length); + return buildFromHashes(ConstMaps.hashStrings(keys), values); + } + + /** + * Builds a map from byte-array keys and their values: {@code keys.get(i)} maps to + * {@code values[i]}. Keys must be distinct. Construction is deterministic. + * + * @param keys the keys, distinct + * @param values the values, one per key + * @return the map + * @throws IllegalArgumentException if the lengths differ, if two keys are equal (detected + * with high probability), or if there are too many keys for one map + */ + public static VerifiedConstMap buildFromBytes(List keys, long[] values) { + ConstMaps.checkLengths(keys.size(), values.length); + return buildFromHashes(ConstMaps.hashBytes(keys), values); + } + + /** + * Builds a map from the XXH64 hashes of the keys ({@link XxHash64#hash(byte[])}) and their + * values. Neither array is modified. + * + * @param keyHashes the XXH64 hashes (seed 0) of the keys, distinct + * @param values the values, one per key + * @return the map + * @throws IllegalArgumentException if the lengths differ, if two hashes are equal (detected + * with high probability), or if there are too many keys for one map + */ + public static VerifiedConstMap buildFromHashes(long[] keyHashes, long[] values) { + ConstMaps.checkLengths(keyHashes.length, values.length); + if (keyHashes.length == 0) { + return new VerifiedConstMap(ConstMaps.Params.EMPTY, new long[0], new long[0]); + } + ConstMaps.Peeled p = ConstMaps.peel(keyHashes); + int n = p.params().arrayLength(); + long[] data = new long[n]; + long[] checks = new long[n]; + ConstMaps.assign(p, values, data, checks); + return new VerifiedConstMap(p.params(), data, checks); + } + + /** + * Returns the value of a key whose XXH64 hash (seed 0) is given. Equivalent to the other + * {@code get} methods on the key itself. + * + * @param keyHash the XXH64 hash of the key + * @return the value, or {@link #NOT_FOUND} if the key is not in the map + */ + public long getByHash(long keyHash) { + if (data.length == 0) { + return NOT_FOUND; + } + long hash = ConstMaps.mixsplit(keyHash, seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + int h1 = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); + int h2 = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); + if ((checks[h0] ^ checks[h1] ^ checks[h2]) != ConstMaps.fingerprint(hash)) { + return NOT_FOUND; + } + return data[h0] ^ data[h1] ^ data[h2]; + } + + /** + * Returns the value of a string key (hashed through its UTF-8 encoding). + * + * @param key the key + * @return the value, or {@link #NOT_FOUND} if the key is not in the map + */ + public long get(String key) { + return getByHash(XxHash64.hash(key)); + } + + /** + * Returns the value of a byte-array key. + * + * @param key the key + * @return the value, or {@link #NOT_FOUND} if the key is not in the map + */ + public long get(byte[] key) { + return getByHash(XxHash64.hash(key)); + } + + /** + * Returns the value of the key made of {@code len} bytes of {@code key} from {@code off}. + * + * @param key the array holding the key + * @param off the index of its first byte + * @param len its length + * @return the value, or {@link #NOT_FOUND} if the key is not in the map + * @throws IndexOutOfBoundsException if the range is not within the array + */ + public long get(byte[] key, int off, int len) { + return getByHash(XxHash64.hash(key, off, len)); + } + + /** + * Returns the value of the key made of the buffer's bytes from its position to its limit. + * The buffer is not modified. + * + * @param key the key + * @return the value, or {@link #NOT_FOUND} if the key is not in the map + */ + public long get(ByteBuffer key) { + return getByHash(XxHash64.hash(key)); + } + + /** + * Returns whether a string key is in the map, that is, whether {@link #get(String)} does not + * return {@link #NOT_FOUND}. + * + * @param key the key + * @return true if the key is in the map (or, with probability about 2-64, a + * false positive), false if it is not or if its value is {@link #NOT_FOUND} + */ + public boolean containsKey(String key) { + return get(key) != NOT_FOUND; + } + + /** + * Looks up many string keys at once: {@code dst[i]} receives the value of + * {@code keys.get(i)}, exactly as {@link #get(String)} would return it. Hashing every key + * of a block before reading the arrays lets the processor overlap the memory accesses of + * several lookups. + * + * @param keys the keys + * @param dst where the values go; only the first {@code keys.size()} entries are written + * @throws IndexOutOfBoundsException if {@code dst} is shorter than {@code keys} + */ + public void getMany(List keys, long[] dst) { + int n = keys.size(); + Objects.checkFromIndexSize(0, n, dst.length); + if (data.length == 0) { + Arrays.fill(dst, 0, n, NOT_FOUND); + return; + } + final long[] d = data; + final long[] c = checks; + final int[] h = new int[3 * ConstMap.BLOCK]; + final long[] fp = new long[ConstMap.BLOCK]; + for (int start = 0; start < n; start += ConstMap.BLOCK) { + int m = Math.min(ConstMap.BLOCK, n - start); + for (int j = 0; j < m; j++) { + long hash = ConstMaps.mixsplit(XxHash64.hash(keys.get(start + j)), seed); + int h0 = (int) Math.unsignedMultiplyHigh(hash, segmentCountLength); + h[3 * j] = h0; + h[3 * j + 1] = (h0 + segmentLength) ^ ((int) (hash >>> 18) & segmentLengthMask); + h[3 * j + 2] = (h0 + 2 * segmentLength) ^ ((int) hash & segmentLengthMask); + fp[j] = ConstMaps.fingerprint(hash); + } + for (int j = 0; j < m; j++) { + int a = h[3 * j]; + int b = h[3 * j + 1]; + int e = h[3 * j + 2]; + long v = d[a] ^ d[b] ^ d[e]; + dst[start + j] = (c[a] ^ c[b] ^ c[e]) == fp[j] ? v : NOT_FOUND; + } + } + } + + /** + * Looks up many string keys at once, as {@link #getMany(List, long[])} does, into a new + * array. + * + * @param keys the keys + * @return the values, one per key, {@link #NOT_FOUND} for missing keys + */ + public long[] getMany(List keys) { + long[] dst = new long[keys.size()]; + getMany(keys, dst); + return dst; + } + + /** + * Returns the same map in the paired layout, as a copy. + * + * @return a {@link PairedVerifiedConstMap} giving the same answers + */ + public PairedVerifiedConstMap paired() { + long[] slots = new long[2 * data.length]; + for (int i = 0; i < data.length; i++) { + slots[2 * i] = data[i]; + slots[2 * i + 1] = checks[i]; + } + return PairedVerifiedConstMap.of(params, slots); + } + + /** + * Returns the number of positions in each of the map's two arrays: about 1.125 per key for + * large maps, more for small ones. + * + * @return the array length + */ + public int arrayLength() { + return data.length; + } + + /** + * Returns the number of bytes written by {@link #serialize(ByteBuffer)}: a 32-byte header, + * the two arrays, and an 8-byte checksum. + * + * @return the serialized size in bytes + */ + public int serializedSize() { + return ConstMaps.VMAP_HEADER + 16 * data.length + 8; + } + + /** + * Writes the map at the buffer's current position, in the {@code VMAP0001} format shared + * with the Go, Rust and C implementations, and advances the position by + * {@link #serializedSize()}. The buffer's byte order is ignored and left unchanged. + * + * @param out the destination + * @throws java.nio.BufferOverflowException if fewer than {@link #serializedSize()} bytes + * remain + * @throws java.nio.ReadOnlyBufferException if the buffer is read-only + */ + public void serialize(ByteBuffer out) { + ConstMaps.write(out, ConstMaps.MAGIC_VMAP, params, data.length, true, data, checks); + } + + /** + * Returns the serialized form of the map, as written by {@link #serialize(ByteBuffer)}. + * + * @return a new array of {@link #serializedSize()} bytes + */ + public byte[] toByteArray() { + byte[] bytes = new byte[serializedSize()]; + serialize(ByteBuffer.wrap(bytes)); + return bytes; + } + + /** + * Reads a serialized map ({@code VMAP0001}) starting at the buffer's position into a new + * on-heap map, after validating its header and checksum. The buffer's position is not + * changed. + * + * @param buffer a buffer holding a serialized map at its position + * @return the map + * @throws InvalidFilterException if the input is not a valid {@code VerifiedConstMap} or is + * corrupt + */ + public static VerifiedConstMap load(ByteBuffer buffer) { + ConstMaps.Header h = ConstMaps.read(buffer, ConstMaps.Kind.VERIFIED, 2); + int n = h.params().arrayLength(); + return new VerifiedConstMap(h.params(), ConstMaps.readWords(buffer, h.start(), n), + ConstMaps.readWords(buffer, h.start() + 8 * n, n)); + } + + /** + * Reads a serialized map, as {@link #load(ByteBuffer)} does. + * + * @param bytes the serialized map + * @return the map + * @throws InvalidFilterException if the input is invalid or corrupt + */ + public static VerifiedConstMap load(byte[] bytes) { + return load(ByteBuffer.wrap(bytes)); + } + + @Override + public String toString() { + return "VerifiedConstMap{positions=" + data.length + ", bytes=" + serializedSize() + "}"; + } +} diff --git a/src/main/java/io/github/fastfilter/jbinfuse/XxHash64.java b/src/main/java/io/github/fastfilter/jbinfuse/XxHash64.java new file mode 100644 index 0000000..24ec26a --- /dev/null +++ b/src/main/java/io/github/fastfilter/jbinfuse/XxHash64.java @@ -0,0 +1,321 @@ +package io.github.fastfilter.jbinfuse; + +import java.lang.invoke.MethodHandles; +import java.lang.invoke.VarHandle; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.charset.StandardCharsets; +import java.util.Objects; + +/** + * A Java implementation of XXH64, the 64-bit hash of xxHash by Yann Collet + * (github.com/Cyan4973/xxHash). + * + *

The results are bit-for-bit identical to the reference C function + * {@code XXH64(input, length, seed)}, and so to Go's {@code xxhash.Sum64} (seed 0). This is + * the key hash of the constant maps ({@link ConstMap}, {@link VerifiedConstMap}, + * {@link PairedVerifiedConstMap}), which must agree with the Go, Rust and C implementations + * for their files to be interchangeable. Bytes are always read as little-endian, regardless + * of the platform or of the byte order of a {@link ByteBuffer}. + * + *

This class is stateless and thread-safe. + */ +public final class XxHash64 { + private static final long P1 = 0x9E3779B185EBCA87L; + private static final long P2 = 0xC2B2AE3D27D4EB4FL; + private static final long P3 = 0x165667B19E3779F9L; + private static final long P4 = 0x85EBCA77C2B2AE63L; + private static final long P5 = 0x27D4EB2F165667C5L; + + private static final VarHandle LONG_LE = + MethodHandles.byteArrayViewVarHandle(long[].class, ByteOrder.LITTLE_ENDIAN); + private static final VarHandle INT_LE = + MethodHandles.byteArrayViewVarHandle(int[].class, ByteOrder.LITTLE_ENDIAN); + + private XxHash64() {} + + /** + * Hashes the UTF-8 encoding of a string, with seed 0. This is how the constant maps hash + * {@code String} keys, and equals Go's {@code xxhash.Sum64String(s)}. + * + * @param s the string to hash + * @return the 64-bit hash + */ + public static long hash(String s) { + final int len = s.length(); + // Fast path for ASCII strings, whose UTF-8 encoding is one byte per char: hash the chars + // directly rather than allocate the encoding. Any other char sends us to the encoder. + int p = 0; + long h; + if (len >= 32) { + long v1 = P1 + P2; + long v2 = P2; + long v3 = 0; + long v4 = -P1; + final int limit = len - 32; + do { + long a = ascii8(s, p); + long b = ascii8(s, p + 8); + long c = ascii8(s, p + 16); + long d = ascii8(s, p + 24); + if (((a | b | c | d) & 0x8080808080808080L) != 0) { + return hashEncoded(s); + } + v1 = round(v1, a); + v2 = round(v2, b); + v3 = round(v3, c); + v4 = round(v4, d); + p += 32; + } while (p <= limit); + h = Long.rotateLeft(v1, 1) + Long.rotateLeft(v2, 7) + Long.rotateLeft(v3, 12) + + Long.rotateLeft(v4, 18); + h = mergeRound(h, v1); + h = mergeRound(h, v2); + h = mergeRound(h, v3); + h = mergeRound(h, v4); + } else { + h = P5; + } + h += len; + while (p + 8 <= len) { + long a = ascii8(s, p); + if ((a & 0x8080808080808080L) != 0) { + return hashEncoded(s); + } + h ^= round(0, a); + h = Long.rotateLeft(h, 27) * P1 + P4; + p += 8; + } + if (p + 4 <= len) { + long a = asciiWord(s, p, 4); + if ((a & 0x80808080L) != 0) { + return hashEncoded(s); + } + h ^= a * P1; + h = Long.rotateLeft(h, 23) * P2 + P3; + p += 4; + } + while (p < len) { + char c = s.charAt(p); + if (c >= 0x80) { + return hashEncoded(s); + } + h ^= c * P5; + h = Long.rotateLeft(h, 11) * P1; + p++; + } + return avalanche(h); + } + + /** + * The little-endian word of {@code n} chars from {@code p}, each char that is not ASCII + * replaced by 0x80 so that it shows in the caller's {@code 0x80} mask. + */ + private static long asciiWord(String s, int p, int n) { + long w = 0; + for (int i = n - 1; i >= 0; i--) { + char c = s.charAt(p + i); + w = (w << 8) | (c < 0x80 ? c : 0x80); + } + return w; + } + + private static long ascii8(String s, int p) { + return asciiWord(s, p, 8); + } + + private static long hashEncoded(String s) { + byte[] b = s.getBytes(StandardCharsets.UTF_8); + return hash(b, 0, b.length, 0); + } + + /** + * Hashes a whole byte array with seed 0. + * + * @param data the bytes to hash + * @return the 64-bit hash + */ + public static long hash(byte[] data) { + return hash(data, 0, data.length, 0); + } + + /** + * Hashes {@code len} bytes of {@code data} starting at {@code off}, with seed 0. + * + * @param data the array holding the bytes + * @param off the index of the first byte to hash + * @param len the number of bytes to hash + * @return the 64-bit hash + * @throws IndexOutOfBoundsException if the range is not within {@code data} + */ + public static long hash(byte[] data, int off, int len) { + return hash(data, off, len, 0); + } + + private static long round(long acc, long input) { + acc += input * P2; + acc = Long.rotateLeft(acc, 31); + return acc * P1; + } + + private static long mergeRound(long acc, long val) { + acc ^= round(0, val); + return acc * P1 + P4; + } + + private static long avalanche(long h) { + h ^= h >>> 33; + h *= P2; + h ^= h >>> 29; + h *= P3; + h ^= h >>> 32; + return h; + } + + private static long r8(byte[] p, int i) { + return (long) LONG_LE.get(p, i); + } + + private static long r4(byte[] p, int i) { + return ((int) INT_LE.get(p, i)) & 0xFFFFFFFFL; + } + + /** + * Hashes {@code len} bytes of {@code data} starting at {@code off}, with the given seed. + * + * @param data the array holding the bytes + * @param off the index of the first byte to hash + * @param len the number of bytes to hash + * @param seed the seed + * @return the 64-bit hash, equal to the C reference {@code XXH64(data + off, len, seed)} + * @throws IndexOutOfBoundsException if the range is not within {@code data} + */ + public static long hash(byte[] data, int off, int len, long seed) { + Objects.checkFromIndexSize(off, len, data.length); + int p = off; + final int end = off + len; + long h; + if (len >= 32) { + long v1 = seed + P1 + P2; + long v2 = seed + P2; + long v3 = seed; + long v4 = seed - P1; + final int limit = end - 32; + do { + v1 = round(v1, r8(data, p)); + v2 = round(v2, r8(data, p + 8)); + v3 = round(v3, r8(data, p + 16)); + v4 = round(v4, r8(data, p + 24)); + p += 32; + } while (p <= limit); + h = Long.rotateLeft(v1, 1) + Long.rotateLeft(v2, 7) + Long.rotateLeft(v3, 12) + + Long.rotateLeft(v4, 18); + h = mergeRound(h, v1); + h = mergeRound(h, v2); + h = mergeRound(h, v3); + h = mergeRound(h, v4); + } else { + h = seed + P5; + } + h += len; + while (p + 8 <= end) { + h ^= round(0, r8(data, p)); + h = Long.rotateLeft(h, 27) * P1 + P4; + p += 8; + } + if (p + 4 <= end) { + h ^= r4(data, p) * P1; + h = Long.rotateLeft(h, 23) * P2 + P3; + p += 4; + } + while (p < end) { + h ^= (data[p] & 0xFFL) * P5; + h = Long.rotateLeft(h, 11) * P1; + p++; + } + return avalanche(h); + } + + /** + * Hashes the bytes of a buffer from its position to its limit, with seed 0. The buffer's + * position, limit and byte order are not changed. + * + * @param buf the buffer + * @return the 64-bit hash + */ + public static long hash(ByteBuffer buf) { + return hash(buf, buf.position(), buf.remaining(), 0); + } + + private static long r8(ByteBuffer p, int i, boolean swap) { + long v = p.getLong(i); + return swap ? Long.reverseBytes(v) : v; + } + + private static long r4(ByteBuffer p, int i, boolean swap) { + int v = p.getInt(i); + return (swap ? Integer.reverseBytes(v) : v) & 0xFFFFFFFFL; + } + + /** + * Hashes {@code len} bytes of a buffer starting at absolute index {@code off}, with the + * given seed. The buffer's position, limit and byte order are not changed. + * + * @param buf the buffer + * @param off the absolute index of the first byte to hash + * @param len the number of bytes to hash + * @param seed the seed + * @return the 64-bit hash, equal to {@link #hash(byte[], int, int, long)} on the same bytes + * @throws IndexOutOfBoundsException if the range is not within the buffer's limit + */ + public static long hash(ByteBuffer buf, int off, int len, long seed) { + if (buf.hasArray()) { + return hash(buf.array(), buf.arrayOffset() + Objects.checkFromIndexSize(off, len, + buf.limit()), len, seed); + } + Objects.checkFromIndexSize(off, len, buf.limit()); + final boolean swap = buf.order() != ByteOrder.LITTLE_ENDIAN; + int p = off; + final int end = off + len; + long h; + if (len >= 32) { + long v1 = seed + P1 + P2; + long v2 = seed + P2; + long v3 = seed; + long v4 = seed - P1; + final int limit = end - 32; + do { + v1 = round(v1, r8(buf, p, swap)); + v2 = round(v2, r8(buf, p + 8, swap)); + v3 = round(v3, r8(buf, p + 16, swap)); + v4 = round(v4, r8(buf, p + 24, swap)); + p += 32; + } while (p <= limit); + h = Long.rotateLeft(v1, 1) + Long.rotateLeft(v2, 7) + Long.rotateLeft(v3, 12) + + Long.rotateLeft(v4, 18); + h = mergeRound(h, v1); + h = mergeRound(h, v2); + h = mergeRound(h, v3); + h = mergeRound(h, v4); + } else { + h = seed + P5; + } + h += len; + while (p + 8 <= end) { + h ^= round(0, r8(buf, p, swap)); + h = Long.rotateLeft(h, 27) * P1 + P4; + p += 8; + } + if (p + 4 <= end) { + h ^= r4(buf, p, swap) * P1; + h = Long.rotateLeft(h, 23) * P2 + P3; + p += 4; + } + while (p < end) { + h ^= (buf.get(p) & 0xFFL) * P5; + h = Long.rotateLeft(h, 11) * P1; + p++; + } + return avalanche(h); + } +} diff --git a/src/main/java/io/github/fastfilter/jbinfuse/package-info.java b/src/main/java/io/github/fastfilter/jbinfuse/package-info.java index 3c66485..ef932ee 100644 --- a/src/main/java/io/github/fastfilter/jbinfuse/package-info.java +++ b/src/main/java/io/github/fastfilter/jbinfuse/package-info.java @@ -8,5 +8,12 @@ * {@link io.github.fastfilter.jbinfuse.BinaryFuseFilter#serialize(java.nio.ByteBuffer)}, and * query the serialized bytes in place with * {@link io.github.fastfilter.jbinfuse.BinaryFuseFilter#wrap(java.nio.ByteBuffer)}. + * + *

The same construction also gives compact immutable maps from keys to 64-bit values: + * {@link io.github.fastfilter.jbinfuse.ConstMap} (about 9 bytes per key, arbitrary answers for + * missing keys), and {@link io.github.fastfilter.jbinfuse.VerifiedConstMap} and + * {@link io.github.fastfilter.jbinfuse.PairedVerifiedConstMap} (about 18 bytes per key, missing + * keys detected). Their files are interchangeable with those of the Go, Rust and C/Python + * implementations (constmap, rsconstmap, fastconstmap). */ package io.github.fastfilter.jbinfuse; diff --git a/src/test/java/io/github/fastfilter/jbinfuse/ConstMapTest.java b/src/test/java/io/github/fastfilter/jbinfuse/ConstMapTest.java new file mode 100644 index 0000000..83bfda0 --- /dev/null +++ b/src/test/java/io/github/fastfilter/jbinfuse/ConstMapTest.java @@ -0,0 +1,430 @@ +package io.github.fastfilter.jbinfuse; + +import static org.junit.jupiter.api.Assertions.assertArrayEquals; +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertFalse; +import static org.junit.jupiter.api.Assertions.assertNotEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.SplittableRandom; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +class ConstMapTest { + + static final long NOT_FOUND = VerifiedConstMap.NOT_FOUND; + + static byte[] resource(String name) throws IOException { + try (InputStream in = ConstMapTest.class.getResourceAsStream("/constmap/" + name)) { + if (in == null) { + throw new IOException("missing resource " + name); + } + return in.readAllBytes(); + } + } + + static List keys(String prefix, int n) { + List keys = new ArrayList<>(n); + for (int i = 0; i < n; i++) { + keys.add(prefix + i); + } + return keys; + } + + static long[] values(int n) { + long[] v = new long[n]; + for (int i = 0; i < n; i++) { + v[i] = i * 0x9E3779B97F4A7C15L; + } + return v; + } + + // --------------------------------------------------------------------------------------- + // Interoperability with the Go, Rust and C implementations. + // + // src/test/resources/constmap/ holds the interop files of github.com/lemire/constmap + // (testdata/interop at commit 6f9f4a9): rust.* from rsconstmap, fastconstmap.* from + // fastconstmap 0.10+, and fastconstmap-0.9.* from the legacy XXH3 format. All were built + // from keys "key-i" mapped to 7 * i, for i in [0, 200). + + static final List INTEROP_KEYS = keys("key-", 200); + + static long[] interopValues() { + long[] v = new long[200]; + for (int i = 0; i < 200; i++) { + v[i] = 7L * i; + } + return v; + } + + interface Lookup { + long get(String key); + } + + static void checkInterop(String name, Lookup map, boolean verified) { + long[] v = interopValues(); + for (int i = 0; i < 200; i++) { + assertEquals(v[i], map.get(INTEROP_KEYS.get(i)), name + " key-" + i); + } + if (verified) { + for (int i = 0; i < 1000; i++) { + assertEquals(NOT_FOUND, map.get("absent-" + i), name + " absent-" + i); + } + } + } + + @ParameterizedTest + @ValueSource(strings = {"rust", "fastconstmap"}) + void readsOtherImplementations(String src) throws IOException { + checkInterop(src + ".cmap", ConstMap.load(resource(src + ".cmap"))::get, false); + checkInterop(src + ".vmap", VerifiedConstMap.load(resource(src + ".vmap"))::get, true); + checkInterop(src + ".pmap", PairedVerifiedConstMap.load(resource(src + ".pmap"))::get, + true); + } + + @Test + void writesTheSameBytesAsGoAndRust() throws IOException { + long[] v = interopValues(); + VerifiedConstMap vm = VerifiedConstMap.build(INTEROP_KEYS, v); + assertArrayEquals(resource("rust.cmap"), ConstMap.build(INTEROP_KEYS, v).toByteArray()); + assertArrayEquals(resource("rust.vmap"), vm.toByteArray()); + assertArrayEquals(resource("rust.pmap"), vm.paired().toByteArray()); + assertArrayEquals(resource("rust.pmap"), + PairedVerifiedConstMap.build(INTEROP_KEYS, v).toByteArray()); + } + + @Test + void writesWhatFastconstmapWritesApartFromTheKeyCount() throws IOException { + long[] v = interopValues(); + byte[] ours = ConstMap.build(INTEROP_KEYS, v).toByteArray(); + byte[] theirs = resource("fastconstmap.cmap"); + assertEquals("CMAP0003", new String(theirs, 0, 8, StandardCharsets.US_ASCII)); + assertArrayEquals(Arrays.copyOfRange(ours, 8, 28), Arrays.copyOfRange(theirs, 8, 28)); + assertEquals(200, ByteBuffer.wrap(theirs).order(ByteOrder.LITTLE_ENDIAN).getInt(28)); + assertArrayEquals(Arrays.copyOfRange(ours, 28, ours.length - 8), + Arrays.copyOfRange(theirs, 32, theirs.length - 8)); + VerifiedConstMap vm = VerifiedConstMap.build(INTEROP_KEYS, v); + for (String ext : new String[] {"vmap", "pmap"}) { + byte[] mine = ext.equals("vmap") ? vm.toByteArray() : vm.paired().toByteArray(); + byte[] c = resource("fastconstmap." + ext); + assertArrayEquals(Arrays.copyOfRange(mine, 0, 28), Arrays.copyOfRange(c, 0, 28), ext); + assertArrayEquals(Arrays.copyOfRange(mine, 32, mine.length - 8), + Arrays.copyOfRange(c, 32, c.length - 8), ext); + } + } + + @Test + void legacyFastconstmapFilesAreNamed() throws IOException { + for (String name : new String[] {"fastconstmap-0.9.cmap", "fastconstmap-0.9.vmap"}) { + byte[] b = resource(name); + for (Runnable load : List.of(() -> ConstMap.load(b), + () -> VerifiedConstMap.load(b), () -> PairedVerifiedConstMap.load(b))) { + InvalidFilterException e = assertThrows(InvalidFilterException.class, load::run); + assertTrue(e.getMessage().contains("fastconstmap 0.9"), e.getMessage()); + } + } + } + + @Test + void wrongKindIsNamed() throws IOException { + InvalidFilterException e = assertThrows(InvalidFilterException.class, + () -> ConstMap.load(resource("rust.vmap"))); + assertTrue(e.getMessage().contains("VerifiedConstMap"), e.getMessage()); + e = assertThrows(InvalidFilterException.class, + () -> VerifiedConstMap.load(resource("rust.pmap"))); + assertTrue(e.getMessage().contains("PairedVerifiedConstMap"), e.getMessage()); + e = assertThrows(InvalidFilterException.class, + () -> PairedVerifiedConstMap.load(resource("rust.cmap"))); + assertTrue(e.getMessage().contains("is a ConstMap"), e.getMessage()); + } + + /** + * Checks the bytes written for many sizes and two key sets (one non-ASCII) against + * github.com/lemire/constmap itself (tools/constmap_vectors): same size and same trailing + * FNV-1a checksum, which covers every other byte. This exercises the small-size + * parameters, the retries with halved and doubled segments below a million keys, and the + * path without them above. + */ + @Test + void matchesGoReferenceForManySizes() throws IOException { + int checked = 0; + try (BufferedReader r = new BufferedReader(new InputStreamReader( + ConstMapTest.class.getResourceAsStream("/constmap_vectors.txt"), + StandardCharsets.UTF_8))) { + String last = null; + ConstMap cm = null; + VerifiedConstMap vm = null; + for (String line; (line = r.readLine()) != null; ) { + String[] f = line.split(" "); + String prefix = f[1]; + int n = Integer.parseInt(f[2]); + String id = prefix + n; + if (!id.equals(last)) { + List k = keys(prefix, n); + long[] v = values(n); + cm = ConstMap.build(k, v); + vm = VerifiedConstMap.build(k, v); + last = id; + if (n <= 100000) { + for (int i = 0; i < n; i++) { + assertEquals(v[i], cm.get(k.get(i))); + assertEquals(v[i], vm.get(k.get(i))); + } + } + } + byte[] bytes = switch (f[0]) { + case "C" -> cm.toByteArray(); + case "V" -> vm.toByteArray(); + default -> vm.paired().toByteArray(); + }; + assertEquals(Integer.parseInt(f[3]), bytes.length, line); + long sum = ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN) + .getLong(bytes.length - 8); + assertEquals(Long.parseUnsignedLong(f[4], 16), sum, line); + checked++; + } + } + assertTrue(checked > 150); + } + + // --------------------------------------------------------------------------------------- + // Behavior. + + @ParameterizedTest + @ValueSource(ints = {0, 1, 2, 3, 5, 17, 1000, 50000}) + void allKindsAgree(int n) { + List k = keys("k", n); + long[] v = values(n); + ConstMap cm = ConstMap.build(k, v); + VerifiedConstMap vm = VerifiedConstMap.build(k, v); + PairedVerifiedConstMap pm = PairedVerifiedConstMap.build(k, v); + for (int i = 0; i < n; i++) { + String key = k.get(i); + byte[] b = key.getBytes(StandardCharsets.UTF_8); + assertEquals(v[i], cm.get(key)); + assertEquals(v[i], cm.get(b)); + assertEquals(v[i], vm.get(key)); + assertEquals(v[i], vm.get(ByteBuffer.wrap(b))); + assertEquals(v[i], pm.get(key)); + assertEquals(v[i], pm.getByHash(XxHash64.hash(b))); + assertTrue(vm.containsKey(key)); + assertTrue(pm.containsKey(key)); + } + SplittableRandom rnd = new SplittableRandom(n); + for (int i = 0; i < 10000; i++) { + String absent = "absent-" + rnd.nextLong(); + assertEquals(NOT_FOUND, vm.get(absent)); + assertEquals(NOT_FOUND, pm.get(absent)); + assertFalse(vm.containsKey(absent)); + } + assertEquals(vm.arrayLength(), cm.arrayLength()); + assertEquals(vm.arrayLength(), pm.arrayLength()); + assertEquals(vm.serializedSize(), pm.serializedSize()); + } + + @Test + void byteKeysMatchStringKeys() { + List k = keys("é-", 300); + List b = new ArrayList<>(); + for (String s : k) { + b.add(s.getBytes(StandardCharsets.UTF_8)); + } + long[] v = values(300); + assertArrayEquals(ConstMap.build(k, v).toByteArray(), + ConstMap.buildFromBytes(b, v).toByteArray()); + assertArrayEquals(VerifiedConstMap.build(k, v).toByteArray(), + VerifiedConstMap.buildFromBytes(b, v).toByteArray()); + assertArrayEquals(PairedVerifiedConstMap.build(k, v).toByteArray(), + PairedVerifiedConstMap.buildFromBytes(b, v).toByteArray()); + ConstMap cm = ConstMap.buildFromBytes(b, v); + byte[] padded = new byte[64]; + byte[] key = b.get(42); + System.arraycopy(key, 0, padded, 9, key.length); + assertEquals(v[42], cm.get(padded, 9, key.length)); + ByteBuffer direct = ByteBuffer.allocateDirect(64); + direct.position(3); + direct.put(key); + direct.flip().position(3); + assertEquals(v[42], cm.get(direct)); + assertEquals(3, direct.position()); + } + + @ParameterizedTest + @ValueSource(ints = {0, 1, 7, 8, 9, 100, 1001}) + void getManyMatchesGet(int n) { + List k = keys("m", n); + long[] v = values(n); + List queries = new ArrayList<>(k); + for (int i = 0; i < n / 2 + 3; i++) { + queries.add("missing-" + i); + } + ConstMap cm = ConstMap.build(k, v); + VerifiedConstMap vm = VerifiedConstMap.build(k, v); + PairedVerifiedConstMap pm = vm.paired(); + long[] c = cm.getMany(queries); + long[] w = vm.getMany(queries); + long[] p = new long[queries.size() + 2]; + p[queries.size()] = 12345; + pm.getMany(queries, p); + for (int i = 0; i < queries.size(); i++) { + String q = queries.get(i); + assertEquals(cm.get(q), c[i]); + assertEquals(vm.get(q), w[i]); + assertEquals(pm.get(q), p[i]); + } + assertEquals(12345, p[queries.size()]); + assertThrows(IndexOutOfBoundsException.class, + () -> vm.getMany(queries, new long[queries.size() - 1])); + } + + @Test + void emptyMaps() { + ConstMap cm = ConstMap.build(List.of(), new long[0]); + VerifiedConstMap vm = VerifiedConstMap.build(List.of(), new long[0]); + PairedVerifiedConstMap pm = vm.paired(); + assertEquals(NOT_FOUND, vm.get("x")); + assertEquals(NOT_FOUND, pm.get("x")); + cm.get("x"); // arbitrary value, but no exception + assertEquals(0, ConstMap.load(cm.toByteArray()).arrayLength()); + assertEquals(NOT_FOUND, VerifiedConstMap.load(vm.toByteArray()).get("x")); + assertEquals(NOT_FOUND, PairedVerifiedConstMap.load(pm.toByteArray()).get("x")); + } + + @Test + void roundTripsAtAnOffsetWithoutTouchingTheBuffer() { + List k = keys("r", 5000); + long[] v = values(5000); + VerifiedConstMap vm = VerifiedConstMap.build(k, v); + ByteBuffer buf = ByteBuffer.allocateDirect(vm.serializedSize() + 20) + .order(ByteOrder.BIG_ENDIAN); + buf.position(13); + vm.serialize(buf); + assertEquals(13 + vm.serializedSize(), buf.position()); + assertEquals(ByteOrder.BIG_ENDIAN, buf.order()); + buf.position(13); + VerifiedConstMap back = VerifiedConstMap.load(buf); + assertEquals(13, buf.position()); + assertArrayEquals(vm.toByteArray(), back.toByteArray()); + for (int i = 0; i < 5000; i += 7) { + assertEquals(v[i], back.get(k.get(i))); + } + + ConstMap cm = ConstMap.build(k, v); + assertArrayEquals(cm.toByteArray(), ConstMap.load(cm.toByteArray()).toByteArray()); + PairedVerifiedConstMap pm = vm.paired(); + assertArrayEquals(pm.toByteArray(), + PairedVerifiedConstMap.load(pm.toByteArray()).toByteArray()); + + assertThrows(java.nio.BufferOverflowException.class, + () -> cm.serialize(ByteBuffer.allocate(cm.serializedSize() - 1))); + assertThrows(java.nio.ReadOnlyBufferException.class, + () -> cm.serialize(ByteBuffer.allocate(cm.serializedSize()).asReadOnlyBuffer())); + } + + @Test + void detectsCorruptionAndTruncation() throws IOException { + for (String name : new String[] {"rust.cmap", "fastconstmap.cmap", "rust.vmap", + "rust.pmap"}) { + byte[] good = resource(name); + for (int i = 0; i < good.length; i++) { + byte[] bad = good.clone(); + bad[i] ^= 0x10; + assertThrows(InvalidFilterException.class, () -> load(name, bad), name + " byte " + i); + } + for (int len = 0; len < good.length; len += 13) { + byte[] cut = Arrays.copyOf(good, len); + assertThrows(InvalidFilterException.class, () -> load(name, cut), name + " len " + len); + } + } + } + + private static void load(String name, byte[] b) { + if (name.endsWith("cmap")) { + ConstMap.load(b); + } else if (name.endsWith("vmap")) { + VerifiedConstMap.load(b); + } else { + PairedVerifiedConstMap.load(b); + } + } + + /** Rewrites the FNV-1a checksum of a serialized map after editing it. */ + private static void reseal(byte[] b) { + long h = ConstMaps.FNV_OFFSET; + for (int i = 0; i < b.length - 8; i++) { + h ^= b[i] & 0xFF; + h *= ConstMaps.FNV_PRIME; + } + ByteBuffer.wrap(b).order(ByteOrder.LITTLE_ENDIAN).putLong(b.length - 8, h); + } + + @Test + void rejectsInconsistentParametersEvenWithAValidChecksum() throws IOException { + byte[] good = resource("rust.vmap"); + ByteBuffer g = ByteBuffer.wrap(good).order(ByteOrder.LITTLE_ENDIAN); + int segmentLength = g.getInt(16); + int segmentCount = g.getInt(20); + int[][] edits = { + {16, segmentLength + 1}, + {16, 0}, + {16, segmentLength * 2}, + {20, 0}, + {20, segmentCount + 1}, + {20, -1}, + }; + for (int[] e : edits) { + byte[] bad = good.clone(); + ByteBuffer.wrap(bad).order(ByteOrder.LITTLE_ENDIAN).putInt(e[0], e[1]); + reseal(bad); + assertThrows(InvalidFilterException.class, () -> VerifiedConstMap.load(bad), + "offset " + e[0] + " value " + e[1]); + } + // The pad word (fastconstmap's key count) is ignored. + byte[] counted = good.clone(); + ByteBuffer.wrap(counted).order(ByteOrder.LITTLE_ENDIAN).putInt(28, 12345); + reseal(counted); + checkInterop("counted", VerifiedConstMap.load(counted)::get, true); + } + + @Test + void rejectsBadInput() { + assertThrows(IllegalArgumentException.class, + () -> ConstMap.build(List.of("a", "b"), new long[1])); + assertThrows(IllegalArgumentException.class, + () -> VerifiedConstMap.build(List.of("a", "b", "a"), new long[3])); + assertThrows(IllegalArgumentException.class, + () -> ConstMap.buildFromHashes(new long[] {5, 5}, new long[2])); + List many = keys("d", 1000); + many.add("d500"); + assertThrows(IllegalArgumentException.class, + () -> ConstMap.build(many, new long[many.size()])); + assertThrows(InvalidFilterException.class, () -> ConstMap.load(new byte[40])); + } + + @Test + void valuesUseAll64Bits() { + List k = List.of("min", "max", "minus-one", "zero"); + long[] v = {Long.MIN_VALUE, Long.MAX_VALUE, -1L, 0}; + ConstMap cm = ConstMap.build(k, v); + PairedVerifiedConstMap pm = PairedVerifiedConstMap.build(k, v); + for (int i = 0; i < 4; i++) { + assertEquals(v[i], cm.get(k.get(i))); + assertEquals(v[i], pm.get(k.get(i))); + } + // A value equal to NOT_FOUND cannot be told from a missing key. + assertFalse(pm.containsKey("minus-one")); + assertNotEquals(NOT_FOUND, pm.get("zero")); + } +} diff --git a/src/test/java/io/github/fastfilter/jbinfuse/XxHash64Test.java b/src/test/java/io/github/fastfilter/jbinfuse/XxHash64Test.java new file mode 100644 index 0000000..15d7ca2 --- /dev/null +++ b/src/test/java/io/github/fastfilter/jbinfuse/XxHash64Test.java @@ -0,0 +1,115 @@ +package io.github.fastfilter.jbinfuse; + +import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertThrows; +import static org.junit.jupiter.api.Assertions.assertTrue; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStreamReader; +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.nio.charset.StandardCharsets; +import org.junit.jupiter.api.Test; + +/** + * Checks XxHash64 against vectors produced by the reference C implementation + * (tools/xxh64_vectors.c). + */ +class XxHash64Test { + + private static long hex(String s) { + return Long.parseUnsignedLong(s, 16); + } + + @Test + void matchesReferenceVectors() throws IOException { + byte[] data = WyHashTest.referenceData(); + int bytesChecked = 0; + int stringsChecked = 0; + try (BufferedReader r = new BufferedReader(new InputStreamReader( + XxHash64Test.class.getResourceAsStream("/xxh64_vectors.txt"), StandardCharsets.UTF_8))) { + for (String line; (line = r.readLine()) != null; ) { + if (line.charAt(0) == 'B') { + String[] f = line.split(" "); + int len = Integer.parseInt(f[1]); + long seed = hex(f[2]); + long want = hex(f[3]); + assertEquals(want, XxHash64.hash(data, 0, len, seed), line); + // Unaligned offset inside a larger array. + byte[] shifted = new byte[len + 3]; + System.arraycopy(data, 0, shifted, 3, len); + assertEquals(want, XxHash64.hash(shifted, 3, len, seed), line); + // Direct buffers, in both byte orders. + ByteBuffer direct = ByteBuffer.allocateDirect(len + 5).order(ByteOrder.BIG_ENDIAN); + direct.position(5); + direct.put(data, 0, len); + assertEquals(want, XxHash64.hash(direct, 5, len, seed), line); + direct.order(ByteOrder.LITTLE_ENDIAN); + assertEquals(want, XxHash64.hash(direct, 5, len, seed), line); + if (seed == 0) { + direct.position(5); + assertEquals(want, XxHash64.hash(direct), line); + assertEquals(5, direct.position()); + } + bytesChecked++; + } else { + int sp = line.lastIndexOf(' '); + String s = line.substring(2, sp); + long want = hex(line.substring(sp + 1)); + assertEquals(want, XxHash64.hash(s), line); + assertEquals(want, XxHash64.hash(s.getBytes(StandardCharsets.UTF_8)), line); + stringsChecked++; + } + } + } + assertTrue(bytesChecked > 600); + assertEquals(8, stringsChecked); + } + + /** + * hash(String) hashes ASCII strings without encoding them; it must agree with hashing the + * UTF-8 encoding for every string, whatever the position of the first non-ASCII char. + */ + @Test + void stringHashEqualsHashOfUtf8() { + char[] specials = {0x7F, 0x80, 0xA0, 0xFF, 0x100, 0x180, 0x7FF, 0x800, 0xFFFF, 0xD83D, + 0xDE00}; + java.util.SplittableRandom r = new java.util.SplittableRandom(42); + for (int len = 0; len <= 80; len++) { + char[] c = new char[len]; + for (int i = 0; i < len; i++) { + c[i] = (char) (0x20 + r.nextInt(0x5F)); + } + String ascii = new String(c); + assertEquals(XxHash64.hash(ascii.getBytes(StandardCharsets.UTF_8)), XxHash64.hash(ascii)); + for (int pos = 0; pos < len; pos++) { + for (char sp : specials) { + char[] d = c.clone(); + d[pos] = sp; + String t = new String(d); + assertEquals(XxHash64.hash(t.getBytes(StandardCharsets.UTF_8)), XxHash64.hash(t), + "len " + len + " pos " + pos + " char " + (int) sp); + } + } + } + String pair = "abc\uD83D\uDE00defghijklmnopqrstuvwxyz0123456789"; + assertEquals(XxHash64.hash(pair.getBytes(StandardCharsets.UTF_8)), XxHash64.hash(pair)); + } + + @Test + void heapBufferWithOffset() { + byte[] data = WyHashTest.referenceData(); + ByteBuffer slice = ByteBuffer.wrap(data, 7, 100).slice(); + assertEquals(XxHash64.hash(data, 7, 100), XxHash64.hash(slice)); + } + + @Test + void rejectsBadRanges() { + byte[] b = new byte[10]; + assertThrows(IndexOutOfBoundsException.class, () -> XxHash64.hash(b, 5, 6)); + assertThrows(IndexOutOfBoundsException.class, () -> XxHash64.hash(b, -1, 2)); + ByteBuffer d = ByteBuffer.allocateDirect(10); + assertThrows(IndexOutOfBoundsException.class, () -> XxHash64.hash(d, 5, 6, 0)); + } +} diff --git a/src/test/resources/constmap/fastconstmap-0.9.cmap b/src/test/resources/constmap/fastconstmap-0.9.cmap new file mode 100644 index 0000000..4b6959d Binary files /dev/null and b/src/test/resources/constmap/fastconstmap-0.9.cmap differ diff --git a/src/test/resources/constmap/fastconstmap-0.9.vmap b/src/test/resources/constmap/fastconstmap-0.9.vmap new file mode 100644 index 0000000..92f52c4 Binary files /dev/null and b/src/test/resources/constmap/fastconstmap-0.9.vmap differ diff --git a/src/test/resources/constmap/fastconstmap.cmap b/src/test/resources/constmap/fastconstmap.cmap new file mode 100644 index 0000000..a4eae7e Binary files /dev/null and b/src/test/resources/constmap/fastconstmap.cmap differ diff --git a/src/test/resources/constmap/fastconstmap.pmap b/src/test/resources/constmap/fastconstmap.pmap new file mode 100644 index 0000000..6fe3f43 Binary files /dev/null and b/src/test/resources/constmap/fastconstmap.pmap differ diff --git a/src/test/resources/constmap/fastconstmap.vmap b/src/test/resources/constmap/fastconstmap.vmap new file mode 100644 index 0000000..b5da579 Binary files /dev/null and b/src/test/resources/constmap/fastconstmap.vmap differ diff --git a/src/test/resources/constmap/rust.cmap b/src/test/resources/constmap/rust.cmap new file mode 100644 index 0000000..fab75ae Binary files /dev/null and b/src/test/resources/constmap/rust.cmap differ diff --git a/src/test/resources/constmap/rust.pmap b/src/test/resources/constmap/rust.pmap new file mode 100644 index 0000000..2a82a19 Binary files /dev/null and b/src/test/resources/constmap/rust.pmap differ diff --git a/src/test/resources/constmap/rust.vmap b/src/test/resources/constmap/rust.vmap new file mode 100644 index 0000000..84aaa66 Binary files /dev/null and b/src/test/resources/constmap/rust.vmap differ diff --git a/src/test/resources/constmap_vectors.txt b/src/test/resources/constmap_vectors.txt new file mode 100644 index 0000000..3209937 --- /dev/null +++ b/src/test/resources/constmap_vectors.txt @@ -0,0 +1,163 @@ +C key- 0 36 b08f909cf4ee9c9f +V key- 0 40 d80c24182d4760a2 +C key- 1 132 8c3798089db9c0dc +V key- 1 232 948604c5f9400c32 +P key- 1 232 7a91fa0305a8f9f4 +C key- 2 132 a9a82966df86f2dc +V key- 2 232 5da376ac9ef6c10e +P key- 2 232 51db9da5c57b039c +C key- 3 228 775cd0377157f4a9 +V key- 3 424 7e2b16acb8ea6589 +P key- 3 424 2a98a28315629437 +C key- 4 228 8eafa0ba32939c5d +V key- 4 424 8c86b589b1a4e2c1 +P key- 4 424 4464a5d804d54d8b +C key- 5 228 87a2383b6f30cf1c +V key- 5 424 a327e29e6cadbe35 +P key- 5 424 61613f2728f4c1fb +C key- 6 228 a4d538f1194684b7 +V key- 6 424 d2f4d28b7e60c54b +P key- 6 424 affcd6e06e1c2499 +C key- 7 228 4c811df3439d6513 +V key- 7 424 f6cb01b6ab538d9b +P key- 7 424 cdb06eebf637e825 +C key- 8 228 7bd49c111e88db9b +V key- 8 424 622342ca6d3a72b5 +P key- 8 424 0b2afef0320b2007 +C key- 9 420 d43e39c20ce11aa8 +V key- 9 808 86fafebf7a814c69 +P key- 9 808 2f6907c7b0a67a57 +C key- 10 420 d5fff092a6b93a83 +V key- 10 808 36c373f08059ea53 +P key- 10 808 ca6fe6222b761805 +C key- 11 420 2874ef7efaa31df0 +V key- 11 808 b29240e45fbd2f94 +P key- 11 808 a7f2bc342ce381a6 +C key- 16 420 3605643c2d38aa21 +V key- 16 808 7a368e9a4fc2bfc6 +P key- 16 808 8348835d56cf6bb0 +C key- 31 804 1db6d7513a0a00ae +V key- 31 1576 995f26a10c091e0e +P key- 31 1576 10d2b13acfe8bf0c +C key- 32 804 167721670e86afa2 +V key- 32 1576 3f4d3a0da3e8cabe +P key- 32 1576 b2142c492b091498 +C key- 33 804 1adc37856e5f6e46 +V key- 33 1576 66b2fc4f5277603b +P key- 33 1576 e2bb2811bf0fd3fd +C key- 100 1572 0cff8cfdea8643b6 +V key- 100 3112 1d03e20851149fcb +P key- 100 3112 debf4dfa082c4201 +C key- 200 2596 cab46b964b0ee1b9 +V key- 200 5160 6f86d1d7898a38df +P key- 200 5160 5be21627f742d42d +C key- 255 3108 ce9069b77f4f1dae +V key- 255 6184 5eb9830ccd763f96 +P key- 255 6184 c0ee208ef045b710 +C key- 256 3108 af2d5490e915062d +V key- 256 6184 43a3c8ded8a24fc5 +P key- 256 6184 0bf1e2072e22c63f +C key- 1000 11300 c81997f974ebb649 +V key- 1000 22568 7f31fbf8cd495d21 +P key- 1000 22568 2787b5b036033f2f +C key- 1023 12324 4c3d8da9971dbc20 +V key- 1023 24616 e09e33a13da7da6d +P key- 1023 24616 b429f07aaff64f0f +C key- 4096 45092 3d9d0a17ac9213a7 +V key- 4096 90152 d6961946e245b820 +P key- 4096 90152 04dbbc2bdc1555be +C key- 10000 102436 10d44bbff1033aa9 +V key- 10000 204840 f6d4b26ccf083b82 +P key- 10000 204840 95a2e4fec7b21af8 +C key- 65537 622628 f8369c4f73930a91 +V key- 65537 1245224 41ca5e6701f02c66 +P key- 65537 1245224 8164c123056e7390 +C key- 100000 950308 080eddf3829cb700 +V key- 100000 1900584 13340c358780bdbf +P key- 100000 1900584 7f2f13cd8d49a439 +C key- 999999 9044004 5245b80c27d2be5a +V key- 999999 18087976 82fdef3c00638035 +P key- 999999 18087976 182acce0bd5c5153 +C key- 1000000 9044004 66969969b940f866 +V key- 1000000 18087976 136e792b11d81b2d +P key- 1000000 18087976 442ab4c0b14d99eb +C key- 1234567 11141156 a694bc31794f7788 +V key- 1234567 22282280 c8107af018ec984a +P key- 1234567 22282280 9ce9c7b7edf9a388 +C clé-日本- 0 36 b08f909cf4ee9c9f +V clé-日本- 0 40 d80c24182d4760a2 +C clé-日本- 1 132 8c3798089db9c0dc +V clé-日本- 1 232 5e46e0a686bf8287 +P clé-日本- 1 232 f0837c54bbdb0fe5 +C clé-日本- 2 132 409027ed7b489b4d +V clé-日本- 2 232 b5cf782d66169f90 +P clé-日本- 2 232 dddb5afbfd05b09e +C clé-日本- 3 228 469cbea803ed5884 +V clé-日本- 3 424 1a0770d94f26cb4b +P clé-日本- 3 424 8d56fe9154c96c5d +C clé-日本- 4 228 578ad9a713268a49 +V clé-日本- 4 424 5e2e63482dcff5e9 +P clé-日本- 4 424 b037a6083c60b67b +C clé-日本- 5 228 3d17f5496dd17641 +V clé-日本- 5 424 edf676dc89cc9a5c +P clé-日本- 5 424 46eb3256c7918cfe +C clé-日本- 6 228 7232b203a627714a +V clé-日本- 6 424 1de7f2aecf3c34e5 +P clé-日本- 6 424 6c28e88cc139c89b +C clé-日本- 7 228 ad4c1b3da9a296cd +V clé-日本- 7 424 db309d0cb742f421 +P clé-日本- 7 424 ebf78316111e98e7 +C clé-日本- 8 228 b331ecc18545ccca +V clé-日本- 8 424 1d085cd32c9763e8 +P clé-日本- 8 424 841b5afa22a31166 +C clé-日本- 9 420 844c5a766a2b2c4f +V clé-日本- 9 808 846c3fbd553ed7f9 +P clé-日本- 9 808 f0f460911e4defdf +C clé-日本- 10 420 71a7c378dc1b9ad3 +V clé-日本- 10 808 6c21b30e296fa57d +P clé-日本- 10 808 4652867623314f47 +C clé-日本- 11 420 d4f1592a647fb9a3 +V clé-日本- 11 808 3d39f22b1ad35d55 +P clé-日本- 11 808 fed4e4405e3efccb +C clé-日本- 16 420 c6e42f66e9ca85a8 +V clé-日本- 16 808 01466565483f0ad0 +P clé-日本- 16 808 76b88004a93a028e +C clé-日本- 31 804 9c068a3af246e36b +V clé-日本- 31 1576 534fe357866effdc +P clé-日本- 31 1576 a0c37ca7b552dd2e +C clé-日本- 32 804 26ea68feab2bacef +V clé-日本- 32 1576 64489e19ee1a5245 +P clé-日本- 32 1576 e21d06d49edb7c2f +C clé-日本- 33 804 c4b88065fdd92bab +V clé-日本- 33 1576 bf7f8d11d60a6410 +P clé-日本- 33 1576 411c87aee87fc52e +C clé-日本- 100 1572 3c8c9ca2b64b6950 +V clé-日本- 100 3112 b7f9512ec4adce4f +P clé-日本- 100 3112 683f2c8b79c7f819 +C clé-日本- 200 2596 7004d4e4fdd8dd9c +V clé-日本- 200 5160 5c0ef05c53475893 +P clé-日本- 200 5160 85422940d6548ad5 +C clé-日本- 255 3108 045e92f6d18aacf4 +V clé-日本- 255 6184 30d7bc0ea15ffa88 +P clé-日本- 255 6184 f917e0319cea8ef2 +C clé-日本- 256 3108 5ff51f6fd82d284c +V clé-日本- 256 6184 364a21e597f7f578 +P clé-日本- 256 6184 5e2428a176c1650a +C clé-日本- 1000 11300 4d6989ad981b8076 +V clé-日本- 1000 22568 abebd88ebb8c1196 +P clé-日本- 1000 22568 fac041771233a8a0 +C clé-日本- 1023 12324 ca4805ef437caf7c +V clé-日本- 1023 24616 a461f8f879a17cb5 +P clé-日本- 1023 24616 ee02c2aac29d46c3 +C clé-日本- 4096 45092 61b8cf3c3becbd17 +V clé-日本- 4096 90152 9af573f18b014e5a +P clé-日本- 4096 90152 9310ac97bf8a5e94 +C clé-日本- 10000 102436 5a4b3d358c1ade7c +V clé-日本- 10000 204840 071dc26b24dcef14 +P clé-日本- 10000 204840 09c95c6fff972e4e +C clé-日本- 65537 622628 85533a8ebd70b59c +V clé-日本- 65537 1245224 c0166ac9bd2e007a +P clé-日本- 65537 1245224 56cce3cd882e29a0 +C clé-日本- 100000 950308 326a636368694d96 +V clé-日本- 100000 1900584 a6e2c119ffb84ecf +P clé-日本- 100000 1900584 a7c53a14713f3255 diff --git a/src/test/resources/xxh64_vectors.txt b/src/test/resources/xxh64_vectors.txt new file mode 100644 index 0000000..b49f8a3 --- /dev/null +++ b/src/test/resources/xxh64_vectors.txt @@ -0,0 +1,748 @@ +B 0 0000000000000000 ef46db3751d8e999 +B 1 0000000000000000 c029548aed30b115 +B 2 0000000000000000 65c17f01e8b0432f +B 3 0000000000000000 de283464585fb486 +B 4 0000000000000000 9c60c088adf1b064 +B 5 0000000000000000 75c553f921a41a68 +B 6 0000000000000000 303446fa960c3dc8 +B 7 0000000000000000 a7856e4b4ae0713b +B 8 0000000000000000 4ea76a7ae29a0229 +B 9 0000000000000000 9df500a55cadd86e +B 10 0000000000000000 4752a9e540acbaee +B 11 0000000000000000 f4e086436eccf65e +B 12 0000000000000000 cdd5d798ee2e3c8f +B 13 0000000000000000 4e687215e58e3657 +B 14 0000000000000000 9b74917db82d971f +B 15 0000000000000000 0a7510a375dbd3b4 +B 16 0000000000000000 04d7c550726b4d69 +B 17 0000000000000000 3128abd2637ec3cb +B 18 0000000000000000 f18f8cdaf8fe9f84 +B 19 0000000000000000 892b3bb531770ae2 +B 20 0000000000000000 30ea2dd1002ad5c8 +B 21 0000000000000000 430c373be21a0352 +B 22 0000000000000000 1a327abe677d35fd +B 23 0000000000000000 3d6f6e31bc3a9030 +B 24 0000000000000000 f01684fe357a96d7 +B 25 0000000000000000 369add96c84bb7d5 +B 26 0000000000000000 a68c43e90f137539 +B 27 0000000000000000 c3d8f4b6a5fe19a2 +B 28 0000000000000000 cc4ac225ae457112 +B 29 0000000000000000 4b4c3361bdcc05bb +B 30 0000000000000000 d91c2632a16a4ce5 +B 31 0000000000000000 5764401a583a0fbd +B 32 0000000000000000 9371a00dffbeea90 +B 33 0000000000000000 1e9080dc8761a290 +B 34 0000000000000000 5c4d12b27bb797e0 +B 35 0000000000000000 03369b721bf2eff8 +B 36 0000000000000000 8aa59f98348f546c +B 37 0000000000000000 f8a288d09e4efcf6 +B 38 0000000000000000 4108d698458dfdc7 +B 39 0000000000000000 e15b1fb0ef0f1e69 +B 40 0000000000000000 c4ffa56820df25a3 +B 41 0000000000000000 ffda1ef91205ce53 +B 42 0000000000000000 106619acd0837528 +B 43 0000000000000000 ec9f0a1da8b1248d +B 44 0000000000000000 b71a2e06f4bd3927 +B 45 0000000000000000 7ea6b859a53bc959 +B 46 0000000000000000 aeab2773e885b028 +B 47 0000000000000000 a86704b64a69954b +B 48 0000000000000000 d7ada67221fac7c8 +B 49 0000000000000000 6634943a4cea31cb +B 50 0000000000000000 0531a9e403aa4b38 +B 51 0000000000000000 8d011ffe1489e5f8 +B 52 0000000000000000 df997718898c814e +B 53 0000000000000000 8ce4e1811489a5c0 +B 54 0000000000000000 fbd8ec737351ea0d +B 55 0000000000000000 9f25f754940df757 +B 56 0000000000000000 93332f7a4a021eea +B 57 0000000000000000 b46c434d1119735b +B 58 0000000000000000 cfe9ec2da87f945e +B 59 0000000000000000 8d51564321a4988f +B 60 0000000000000000 25db764c9fa0ce52 +B 61 0000000000000000 9c4e985750037fe1 +B 62 0000000000000000 5aa3b49e3fa827ea +B 63 0000000000000000 7abe23b3e8b3bc35 +B 64 0000000000000000 267945da91dbcbdc +B 65 0000000000000000 488616254d6738f3 +B 66 0000000000000000 35a50561c99d4f11 +B 67 0000000000000000 8ef055a407871f59 +B 68 0000000000000000 1ddee93a0915d9ab +B 69 0000000000000000 3e84a75790432150 +B 70 0000000000000000 25a997889dc0943d +B 71 0000000000000000 a87c4b31cb80ef28 +B 72 0000000000000000 5b328aacfe96a926 +B 73 0000000000000000 762054cd5ee266ea +B 74 0000000000000000 575bf71be623e95d +B 75 0000000000000000 42778743fba06a50 +B 76 0000000000000000 045f5ead58888cb4 +B 77 0000000000000000 ab4ac145772f28e9 +B 78 0000000000000000 6e82d239f70b17a1 +B 79 0000000000000000 6be2f7c95309ffbd +B 80 0000000000000000 ba339f3df57ac7ca +B 81 0000000000000000 f65fa6d874218eb7 +B 82 0000000000000000 90e8e27cf882373b +B 83 0000000000000000 e11c5c541944f7e0 +B 84 0000000000000000 196aa731c434ccad +B 85 0000000000000000 06e1fefbdf0e9234 +B 86 0000000000000000 f301b7fb4692db87 +B 87 0000000000000000 2a44dae05ce5bcb4 +B 88 0000000000000000 dd31eb7405ca01da +B 89 0000000000000000 1fc193343fbb20cd +B 90 0000000000000000 7f398240023bcd7f +B 91 0000000000000000 d57a4cce3b3d818a +B 92 0000000000000000 63291c1789de3fa8 +B 93 0000000000000000 65004273170e541f +B 94 0000000000000000 158b47ca3144d2a5 +B 95 0000000000000000 845142419affa22b +B 96 0000000000000000 a9e53805b8998533 +B 97 0000000000000000 36ba459b3f7536fd +B 98 0000000000000000 02a5f40b884e694d +B 99 0000000000000000 1021b6eacdf067c8 +B 100 0000000000000000 24b207ec4e0065e5 +B 101 0000000000000000 b755e1cea598a006 +B 102 0000000000000000 ff6c4398f88946a2 +B 103 0000000000000000 2ef449307831fb9f +B 104 0000000000000000 18e7844107a9539d +B 105 0000000000000000 edb97c2c275d662a +B 106 0000000000000000 d7c62010aae49bac +B 107 0000000000000000 4e09d06f321abb53 +B 108 0000000000000000 2ab0a274de0a9d12 +B 109 0000000000000000 718c9c63769403a8 +B 110 0000000000000000 43b0be9d788f90ed +B 111 0000000000000000 b959d9f605f54db0 +B 112 0000000000000000 a8666681fd9c0557 +B 113 0000000000000000 4f90d8dda815a738 +B 114 0000000000000000 1c9077af3ee02436 +B 115 0000000000000000 651c51e6fdb22fbf +B 116 0000000000000000 76c3df53f223a4de +B 117 0000000000000000 b112e977af355aff +B 118 0000000000000000 61c88837e405066a +B 119 0000000000000000 cb3737d244623ea2 +B 120 0000000000000000 103de63aff34ccfd +B 121 0000000000000000 10a684f4a627da95 +B 122 0000000000000000 d879e617d0d7c4eb +B 123 0000000000000000 8df771f453b25227 +B 124 0000000000000000 e023b63ec4db0c06 +B 125 0000000000000000 ffe11a29b9ae255e +B 126 0000000000000000 82f7791eaa7a4219 +B 127 0000000000000000 448c86ee488ed464 +B 128 0000000000000000 46707f3055642118 +B 129 0000000000000000 d9da3a9d49cfae16 +B 130 0000000000000000 6a369c415df4ea1c +B 131 0000000000000000 c6652f43889a052c +B 132 0000000000000000 8a6d947119c39fde +B 133 0000000000000000 5c089c7b005887bd +B 134 0000000000000000 cfcd524138e9abe6 +B 135 0000000000000000 54e602f6bb872b71 +B 136 0000000000000000 bc5881370cb5572b +B 137 0000000000000000 45bec2be9e987972 +B 138 0000000000000000 ff0b1a7bff98c312 +B 139 0000000000000000 7bbd8eb6004c8abf +B 140 0000000000000000 006bbbc9ddac5565 +B 141 0000000000000000 282e10aaa597a54a +B 142 0000000000000000 f275f93b29395225 +B 143 0000000000000000 9a01d86ac6b4c174 +B 144 0000000000000000 0b4670c7b44c6671 +B 145 0000000000000000 9dc68c88174c388f +B 146 0000000000000000 805931f4cc5140fa +B 147 0000000000000000 edd56fceeb646777 +B 148 0000000000000000 f831f3e706db5920 +B 149 0000000000000000 3e4235d6ca3472b2 +B 150 0000000000000000 cb3a433d89184be1 +B 151 0000000000000000 f91bae50551e9436 +B 152 0000000000000000 6ab5d56d72322b4d +B 153 0000000000000000 241b69a9f0ae3ac9 +B 154 0000000000000000 154d631ed6a8808d +B 155 0000000000000000 b4abe16e4cf22813 +B 156 0000000000000000 a6525a82a65edd5b +B 157 0000000000000000 c4692545f92e2b14 +B 158 0000000000000000 64eee28e8548e204 +B 159 0000000000000000 48aeae8e7a495d60 +B 160 0000000000000000 f8f2acab4de584f1 +B 161 0000000000000000 9115118bcfe0fc30 +B 198 0000000000000000 53080c2f4cf92e19 +B 235 0000000000000000 caea91768e9265dc +B 272 0000000000000000 074b34c90cab89a7 +B 309 0000000000000000 826c7e8a5c1a8ec6 +B 346 0000000000000000 ac2c7674b37c2de1 +B 383 0000000000000000 a2bfaa2a8356ba1d +B 420 0000000000000000 6292444a90c3b247 +B 457 0000000000000000 db51bfe48efc27f7 +B 494 0000000000000000 0de914515bdf9e69 +B 531 0000000000000000 ed39ed0d01c7da39 +B 568 0000000000000000 9278345c93658eed +B 605 0000000000000000 6a708f46d1930025 +B 642 0000000000000000 d59ede684c8fc51f +B 679 0000000000000000 e31a1a63f83ba7f5 +B 716 0000000000000000 4173f7bdffe19723 +B 753 0000000000000000 fe038fcd653df624 +B 790 0000000000000000 1d02afc7cf2cb69a +B 827 0000000000000000 e5f217679e870bdc +B 864 0000000000000000 327d7eee95768b45 +B 901 0000000000000000 5479e60dbfa67cc6 +B 938 0000000000000000 5afb929ad9ac7423 +B 975 0000000000000000 da987b5ab56eda66 +B 1012 0000000000000000 2aebe03d5c48c032 +B 0 0000000000000001 d5afba1336a3be4b +B 1 0000000000000001 aff5604b9c7132a4 +B 2 0000000000000001 e07e5dec08a6252b +B 3 0000000000000001 6ca00da364a3c531 +B 4 0000000000000001 06ee33b71e890641 +B 5 0000000000000001 b77433092cbb5bd6 +B 6 0000000000000001 860b4fdbb02c2ffe +B 7 0000000000000001 37cf85bbf024a682 +B 8 0000000000000001 519ebed74ba89704 +B 9 0000000000000001 aec3c9024edfce1b +B 10 0000000000000001 8f3575eb448d58f3 +B 11 0000000000000001 37361f24cdd5e7db +B 12 0000000000000001 5601583bc56e4942 +B 13 0000000000000001 61db5fe4e6938b96 +B 14 0000000000000001 eead26e0783f0a29 +B 15 0000000000000001 7e8077e03aa3a69d +B 16 0000000000000001 5ba4493f5813ba6d +B 17 0000000000000001 6b01aa71e50fcaef +B 18 0000000000000001 4b7fcea75bde7aed +B 19 0000000000000001 725608cd683ef099 +B 20 0000000000000001 5f59dbb828cfa094 +B 21 0000000000000001 f7364cbf1a7566f8 +B 22 0000000000000001 e6e6d170fabf95f9 +B 23 0000000000000001 5c10b02e3e46e149 +B 24 0000000000000001 f015a2e9d9e9b467 +B 25 0000000000000001 59c65feaddc78215 +B 26 0000000000000001 24c26839f6fbc8ac +B 27 0000000000000001 ac05d4a44d22dd34 +B 28 0000000000000001 3a4f9902a3972813 +B 29 0000000000000001 aca39456259be6b8 +B 30 0000000000000001 173cefdf6a35a789 +B 31 0000000000000001 6175977f06de39d4 +B 32 0000000000000001 40495f4ee7d5c99c +B 33 0000000000000001 1a1ea77437814e70 +B 34 0000000000000001 f29928cb522d5e6c +B 35 0000000000000001 f085df742eb8224a +B 36 0000000000000001 72a5f4ef6579326a +B 37 0000000000000001 46a5e5e8d3c9a4b4 +B 38 0000000000000001 9d9b966c76b282d1 +B 39 0000000000000001 18e6fe26ff10821a +B 40 0000000000000001 e3efd5bbd9a7f841 +B 41 0000000000000001 1e9f7bcae342db7f +B 42 0000000000000001 4ff46dee6a05dab3 +B 43 0000000000000001 07ca85ed612d921f +B 44 0000000000000001 3c92d982f0f23247 +B 45 0000000000000001 7c589c9247ee2980 +B 46 0000000000000001 e352a0361b3933db +B 47 0000000000000001 de0dbe74173ebb38 +B 48 0000000000000001 bb40dd7e2a119e16 +B 49 0000000000000001 ab0e0cd1fc57ca20 +B 50 0000000000000001 0de8b66600653042 +B 51 0000000000000001 7d65aa91bb76b524 +B 52 0000000000000001 2e187b8a83468828 +B 53 0000000000000001 23d65ee216c590a6 +B 54 0000000000000001 ddb24b699f9fbe37 +B 55 0000000000000001 3d3a1b6cbd201a7e +B 56 0000000000000001 df9ae3ee484e92c3 +B 57 0000000000000001 f6cc7200af670f72 +B 58 0000000000000001 ff8d1e4bc4ba477e +B 59 0000000000000001 66ad457cc31e8b38 +B 60 0000000000000001 6ec3e3e2dc45b378 +B 61 0000000000000001 fb5d2a73d54205ff +B 62 0000000000000001 b5fc28b5aad7a210 +B 63 0000000000000001 ea8ab6d0a8370941 +B 64 0000000000000001 c1c458384c56cd5d +B 65 0000000000000001 64ea51905c1f557d +B 66 0000000000000001 23e72f1b25504a3b +B 67 0000000000000001 9cb6c0a529121298 +B 68 0000000000000001 9f61b978abe60598 +B 69 0000000000000001 35a0889ef96e2301 +B 70 0000000000000001 2c35ecb16be359ad +B 71 0000000000000001 92e60096b66c82f5 +B 72 0000000000000001 bf75aa9575f85189 +B 73 0000000000000001 34a08bb80917fe17 +B 74 0000000000000001 021811c53a7f5005 +B 75 0000000000000001 cf5ff2b36fec4b1b +B 76 0000000000000001 6b1e5b8a51a76cfe +B 77 0000000000000001 38782ed7523d2f85 +B 78 0000000000000001 5e8e1daf4d1f81f6 +B 79 0000000000000001 76ffa89a1e11991f +B 80 0000000000000001 992516f77203ebac +B 81 0000000000000001 70796b1663acc34c +B 82 0000000000000001 66c9f21a37e1134e +B 83 0000000000000001 31022e3bd9eb528d +B 84 0000000000000001 70c051754d93a568 +B 85 0000000000000001 f0f0fe9e9ca3a585 +B 86 0000000000000001 b3e6a79a5ad3eb1d +B 87 0000000000000001 7a90f6dd1dbb32ff +B 88 0000000000000001 3c68960e81056515 +B 89 0000000000000001 d022784a9ed5f58a +B 90 0000000000000001 eaf7227e9adca951 +B 91 0000000000000001 0c2d7aadc37d818a +B 92 0000000000000001 59d8c75fc79bd331 +B 93 0000000000000001 1774b34661d527a4 +B 94 0000000000000001 6e0a09c69de14fb8 +B 95 0000000000000001 82596b1ae2eeea63 +B 96 0000000000000001 5d7d3d30628426d2 +B 97 0000000000000001 3bd2071f695e5661 +B 98 0000000000000001 9e7fe10af848af81 +B 99 0000000000000001 2c9d1bfcfee7d555 +B 100 0000000000000001 8bd943dc0af3e2d7 +B 101 0000000000000001 1ecd87c74155bcd9 +B 102 0000000000000001 6efd98d4f35b0701 +B 103 0000000000000001 a93ad4ffd1f85ea3 +B 104 0000000000000001 2066c65b9ae1aa1c +B 105 0000000000000001 5a730089d99ddd9b +B 106 0000000000000001 399704b702827cd9 +B 107 0000000000000001 5cbb5bcbe7da2c5f +B 108 0000000000000001 51cf8054ef0cb568 +B 109 0000000000000001 c5b075af092c23dc +B 110 0000000000000001 14d71561bb3ffd4d +B 111 0000000000000001 b5f6e2c0b2a2a629 +B 112 0000000000000001 0e4e0de2b9ae7c58 +B 113 0000000000000001 2349479a5841cfc7 +B 114 0000000000000001 64a2dd79e57aa830 +B 115 0000000000000001 cc36c7684b8064a8 +B 116 0000000000000001 9c7bf5f22a2215ed +B 117 0000000000000001 bcb8444543287d15 +B 118 0000000000000001 4d0a25fbffae3b9e +B 119 0000000000000001 bb15bf363ffe3eba +B 120 0000000000000001 aa940a03e90fcf1f +B 121 0000000000000001 f070ff0e4b5c3eae +B 122 0000000000000001 e692f0891f799dc7 +B 123 0000000000000001 e201a0fa05617f9a +B 124 0000000000000001 49374e63a3d796af +B 125 0000000000000001 cd611d28507b971b +B 126 0000000000000001 3e1846f2e0ed9d1b +B 127 0000000000000001 04bd9b6c37d5a451 +B 128 0000000000000001 1dffd4b556b3ef65 +B 129 0000000000000001 3f38a465ada3d939 +B 130 0000000000000001 29a9e3ae7bcd445d +B 131 0000000000000001 67b92f51e25c3731 +B 132 0000000000000001 411dabf59854a40e +B 133 0000000000000001 8ed01bceb0896fc3 +B 134 0000000000000001 99746bfe2176b7b6 +B 135 0000000000000001 076c7bccb28e738f +B 136 0000000000000001 3364664b63396be1 +B 137 0000000000000001 a64a23fa8cbb817b +B 138 0000000000000001 351a77387e1dfb08 +B 139 0000000000000001 4e0dadbe3398c7d2 +B 140 0000000000000001 54c1725968ce35de +B 141 0000000000000001 63726a6daebea79e +B 142 0000000000000001 a0ff086d9c8c4436 +B 143 0000000000000001 5742ba94315972fe +B 144 0000000000000001 38dc9d0914ce5cc8 +B 145 0000000000000001 707e6289c9b8d06a +B 146 0000000000000001 b2c3cae23bfe7d68 +B 147 0000000000000001 d00ecd2caa5aba36 +B 148 0000000000000001 3e32523364ecd189 +B 149 0000000000000001 6d7fa5c6b3cbd39f +B 150 0000000000000001 4517514e44242984 +B 151 0000000000000001 d2c3dd14898c244c +B 152 0000000000000001 5444ea04ee6e63bd +B 153 0000000000000001 0efe3c209d2bc7f3 +B 154 0000000000000001 df2cc6d899d71a55 +B 155 0000000000000001 be1aec1738908a3d +B 156 0000000000000001 453b7bac674b6043 +B 157 0000000000000001 c91d3f02448fd324 +B 158 0000000000000001 32e2efe8b60e4bdb +B 159 0000000000000001 664cea5899301a3c +B 160 0000000000000001 33250c8b7b0f29b0 +B 161 0000000000000001 1ec38d0a7f933ac2 +B 198 0000000000000001 2b6f958615a266a4 +B 235 0000000000000001 278b32d09c38351c +B 272 0000000000000001 bd1789009e3c5367 +B 309 0000000000000001 7f8d84cc0ba7d0c0 +B 346 0000000000000001 b1cc69d7247a94c1 +B 383 0000000000000001 1345dbea5bf9c171 +B 420 0000000000000001 4eddfe91069d501d +B 457 0000000000000001 91b22305a6ea968a +B 494 0000000000000001 0b1cc28730f99367 +B 531 0000000000000001 98a7a52fdda929b0 +B 568 0000000000000001 c549310520e12837 +B 605 0000000000000001 35c41f9c13669635 +B 642 0000000000000001 91feefa4ad1d3427 +B 679 0000000000000001 c58b086d533339fa +B 716 0000000000000001 ab2658417e1878a7 +B 753 0000000000000001 3449e19ad0055fad +B 790 0000000000000001 ed64ce9186b25c6c +B 827 0000000000000001 502e9a3834fef188 +B 864 0000000000000001 80cc72cd96dd0ad2 +B 901 0000000000000001 ebd73d8ab90b7174 +B 938 0000000000000001 bb457518fe602bc2 +B 975 0000000000000001 68550ed8331b00bc +B 1012 0000000000000001 3263205f3e069dd1 +B 0 9e3779b97f4a7c15 c4349fc93c010000 +B 1 9e3779b97f4a7c15 119fc9b9102da345 +B 2 9e3779b97f4a7c15 65fa64f350192019 +B 3 9e3779b97f4a7c15 bc9bf4de4b9e5c41 +B 4 9e3779b97f4a7c15 46dd3fda955c2a4d +B 5 9e3779b97f4a7c15 39053e8996b999da +B 6 9e3779b97f4a7c15 469854f993a54adf +B 7 9e3779b97f4a7c15 0cf083448b87b10b +B 8 9e3779b97f4a7c15 3922b4e9e5359503 +B 9 9e3779b97f4a7c15 f6b14af2c9dde880 +B 10 9e3779b97f4a7c15 17335c5705a47d57 +B 11 9e3779b97f4a7c15 fc6306aa93037942 +B 12 9e3779b97f4a7c15 28796069b272a3fb +B 13 9e3779b97f4a7c15 d69f61b2d6a2d91d +B 14 9e3779b97f4a7c15 9a8cfc2bcf1f42da +B 15 9e3779b97f4a7c15 76a5a668eb04f598 +B 16 9e3779b97f4a7c15 fbd2fb5bc71dd643 +B 17 9e3779b97f4a7c15 90b634a9dd5dafd3 +B 18 9e3779b97f4a7c15 ed9ebf557c0b2b96 +B 19 9e3779b97f4a7c15 5e572e881f53adc5 +B 20 9e3779b97f4a7c15 dd158cab1a1eecbe +B 21 9e3779b97f4a7c15 9981b3edeb614373 +B 22 9e3779b97f4a7c15 59835ba392b41533 +B 23 9e3779b97f4a7c15 8faa9c965d511bc6 +B 24 9e3779b97f4a7c15 2644e887af6ba328 +B 25 9e3779b97f4a7c15 d15e504c0eeb6b61 +B 26 9e3779b97f4a7c15 c16f054bbee426ba +B 27 9e3779b97f4a7c15 43f4edf90d3139f8 +B 28 9e3779b97f4a7c15 18e531c78c1da657 +B 29 9e3779b97f4a7c15 9c9ce3b3d1342b94 +B 30 9e3779b97f4a7c15 26a74ff87b8a12a1 +B 31 9e3779b97f4a7c15 62d41077d620697a +B 32 9e3779b97f4a7c15 37ebc0ebf19b3a29 +B 33 9e3779b97f4a7c15 48d1ff1f02f7ab42 +B 34 9e3779b97f4a7c15 8e7d3c487bad22de +B 35 9e3779b97f4a7c15 63c380a5d15f3b96 +B 36 9e3779b97f4a7c15 ac123da9faf3b4bc +B 37 9e3779b97f4a7c15 09687df66038a961 +B 38 9e3779b97f4a7c15 7c44c3b772e00a74 +B 39 9e3779b97f4a7c15 797a76a31ca5d257 +B 40 9e3779b97f4a7c15 3eaef72b51468e0a +B 41 9e3779b97f4a7c15 75549339724ce000 +B 42 9e3779b97f4a7c15 88fa7b8e5e5c54cc +B 43 9e3779b97f4a7c15 189bea32379c1949 +B 44 9e3779b97f4a7c15 7966b8df4ee36074 +B 45 9e3779b97f4a7c15 553406141b5f6563 +B 46 9e3779b97f4a7c15 7812070597c4b994 +B 47 9e3779b97f4a7c15 0f1c7a6819f08ebb +B 48 9e3779b97f4a7c15 7952646aa3db1366 +B 49 9e3779b97f4a7c15 2d3636176f989252 +B 50 9e3779b97f4a7c15 7a066ed5c18d780e +B 51 9e3779b97f4a7c15 ec9b0192338b5c8e +B 52 9e3779b97f4a7c15 3a3a6e2b9966529e +B 53 9e3779b97f4a7c15 cef8b6c8d6b908f5 +B 54 9e3779b97f4a7c15 1956485b0d34c45c +B 55 9e3779b97f4a7c15 0df494bab999bcd5 +B 56 9e3779b97f4a7c15 eb4337eaef680dd0 +B 57 9e3779b97f4a7c15 bebab5f5931b9ef4 +B 58 9e3779b97f4a7c15 24cd6bfe7609392d +B 59 9e3779b97f4a7c15 5ef7fe668cbef563 +B 60 9e3779b97f4a7c15 0430b6d4f97b763a +B 61 9e3779b97f4a7c15 40935b6832eb8e63 +B 62 9e3779b97f4a7c15 4b2b091b4a9bb7f8 +B 63 9e3779b97f4a7c15 e32a655ba549e4fe +B 64 9e3779b97f4a7c15 a290eeb97e292490 +B 65 9e3779b97f4a7c15 c8dbe0665b032ec0 +B 66 9e3779b97f4a7c15 1d787d1dcccd78d0 +B 67 9e3779b97f4a7c15 8179c209a9f0c2b4 +B 68 9e3779b97f4a7c15 23a7e9bc59a85b85 +B 69 9e3779b97f4a7c15 27f1ec7e51547c95 +B 70 9e3779b97f4a7c15 72ca1ede9b55f0f2 +B 71 9e3779b97f4a7c15 8f57d2bd703a4d78 +B 72 9e3779b97f4a7c15 1cadcfa2ed04845e +B 73 9e3779b97f4a7c15 46a30c989192211e +B 74 9e3779b97f4a7c15 d8ffc6e262918442 +B 75 9e3779b97f4a7c15 1184cd325648bdcf +B 76 9e3779b97f4a7c15 bbb8695d4859bfbd +B 77 9e3779b97f4a7c15 1c050e82844462f5 +B 78 9e3779b97f4a7c15 cd059050c29faf68 +B 79 9e3779b97f4a7c15 6378114ba345d39c +B 80 9e3779b97f4a7c15 460449343d136e5b +B 81 9e3779b97f4a7c15 df86df0bb1106787 +B 82 9e3779b97f4a7c15 f1e465da9bf40e61 +B 83 9e3779b97f4a7c15 fd90eefacc7d6ba3 +B 84 9e3779b97f4a7c15 bcb120ab8fc4abbc +B 85 9e3779b97f4a7c15 c6ba4602cbc635a5 +B 86 9e3779b97f4a7c15 982003145983494b +B 87 9e3779b97f4a7c15 2895be3805e78c07 +B 88 9e3779b97f4a7c15 f4bc3aca08f9eb52 +B 89 9e3779b97f4a7c15 49af158f62b09e25 +B 90 9e3779b97f4a7c15 a962a79710c6c5ee +B 91 9e3779b97f4a7c15 0fabcd1d4046b309 +B 92 9e3779b97f4a7c15 299bf23e5337c481 +B 93 9e3779b97f4a7c15 d514df8c1d68be1a +B 94 9e3779b97f4a7c15 a5cbe18d5fbe0836 +B 95 9e3779b97f4a7c15 00e128dca7a94751 +B 96 9e3779b97f4a7c15 671734859dec8a53 +B 97 9e3779b97f4a7c15 7721baf18a56c50e +B 98 9e3779b97f4a7c15 3b14246e2ae17297 +B 99 9e3779b97f4a7c15 6a925a0747d2cb5d +B 100 9e3779b97f4a7c15 fc8b2311063af0ef +B 101 9e3779b97f4a7c15 3b74d2307713648a +B 102 9e3779b97f4a7c15 549ab97cdffc541e +B 103 9e3779b97f4a7c15 e84ba6f3d2214416 +B 104 9e3779b97f4a7c15 1c57d4391ccbc654 +B 105 9e3779b97f4a7c15 44c844f2ea704883 +B 106 9e3779b97f4a7c15 c75b94ef30ee86d5 +B 107 9e3779b97f4a7c15 f70e7a742069fa0c +B 108 9e3779b97f4a7c15 7710f4f8357a8200 +B 109 9e3779b97f4a7c15 54f2043393542cb6 +B 110 9e3779b97f4a7c15 4f94a42dba352e36 +B 111 9e3779b97f4a7c15 0425cd924afdff99 +B 112 9e3779b97f4a7c15 e4732974b539df64 +B 113 9e3779b97f4a7c15 990365274ba92823 +B 114 9e3779b97f4a7c15 9ad91947d5dd8813 +B 115 9e3779b97f4a7c15 8dbbd5ef434e3049 +B 116 9e3779b97f4a7c15 b1590030c48460e0 +B 117 9e3779b97f4a7c15 ee6ecb42eba18c1d +B 118 9e3779b97f4a7c15 c4c56c99989ac150 +B 119 9e3779b97f4a7c15 dea578e0c6683ad8 +B 120 9e3779b97f4a7c15 c192d7a550caf3b8 +B 121 9e3779b97f4a7c15 2e17430869da9111 +B 122 9e3779b97f4a7c15 0d26dfe697e99c23 +B 123 9e3779b97f4a7c15 000db93523dc01e9 +B 124 9e3779b97f4a7c15 37466f77f8246625 +B 125 9e3779b97f4a7c15 7da8151048efc7d1 +B 126 9e3779b97f4a7c15 fa5506d8bb8c8c8e +B 127 9e3779b97f4a7c15 606d6229a987b805 +B 128 9e3779b97f4a7c15 cb5f2fdacf4e2836 +B 129 9e3779b97f4a7c15 c53bb2071f025d31 +B 130 9e3779b97f4a7c15 88fad7a2017c6ba0 +B 131 9e3779b97f4a7c15 22bff21786802f1f +B 132 9e3779b97f4a7c15 ae4ba944c08e3617 +B 133 9e3779b97f4a7c15 cd201529fff43886 +B 134 9e3779b97f4a7c15 8acf19d8676c5935 +B 135 9e3779b97f4a7c15 c15834256b2b022a +B 136 9e3779b97f4a7c15 3e181658b86eda86 +B 137 9e3779b97f4a7c15 2aabc2e504bcc327 +B 138 9e3779b97f4a7c15 872ed7c7b23d38c0 +B 139 9e3779b97f4a7c15 04b245a84bdf6f8c +B 140 9e3779b97f4a7c15 12da34b7b49a6d7f +B 141 9e3779b97f4a7c15 dcc142949b95764b +B 142 9e3779b97f4a7c15 fda71f7fb531317c +B 143 9e3779b97f4a7c15 0aa6947d66d6921d +B 144 9e3779b97f4a7c15 6b7eaad968e838d8 +B 145 9e3779b97f4a7c15 0b70e72b6cb6217d +B 146 9e3779b97f4a7c15 9d7b3091d44df6e9 +B 147 9e3779b97f4a7c15 42bf62387bb41c9c +B 148 9e3779b97f4a7c15 706e3018116717c8 +B 149 9e3779b97f4a7c15 5b82039dc2eae175 +B 150 9e3779b97f4a7c15 f20e93ee7ece9b30 +B 151 9e3779b97f4a7c15 302a102205c9f08c +B 152 9e3779b97f4a7c15 4cbe6da85452040a +B 153 9e3779b97f4a7c15 39afc930b624de72 +B 154 9e3779b97f4a7c15 a8ee09bf62d1e3c1 +B 155 9e3779b97f4a7c15 3c54c3e630919cbc +B 156 9e3779b97f4a7c15 42469dffbd0f23d5 +B 157 9e3779b97f4a7c15 e4af87b20bd98fe1 +B 158 9e3779b97f4a7c15 3b7d7c6dd75065e0 +B 159 9e3779b97f4a7c15 ad76dd6f427b88ea +B 160 9e3779b97f4a7c15 701a13951d5170fb +B 161 9e3779b97f4a7c15 1a63571003cc72cd +B 198 9e3779b97f4a7c15 fe02072d419952ba +B 235 9e3779b97f4a7c15 5407c4fdf0ec2a5b +B 272 9e3779b97f4a7c15 1451b15a74e4f484 +B 309 9e3779b97f4a7c15 9896e83346ad1064 +B 346 9e3779b97f4a7c15 1158ff7d80691c97 +B 383 9e3779b97f4a7c15 104a1dc36532961b +B 420 9e3779b97f4a7c15 3dfc9a8ffd6236d4 +B 457 9e3779b97f4a7c15 85aa9a58fa5e73ad +B 494 9e3779b97f4a7c15 a17cab2ce9f2ccbb +B 531 9e3779b97f4a7c15 fd6b215ea0406fb2 +B 568 9e3779b97f4a7c15 3efe6a7a99a73b90 +B 605 9e3779b97f4a7c15 c6cf1d0a7201d879 +B 642 9e3779b97f4a7c15 5278c0abd960dd04 +B 679 9e3779b97f4a7c15 dea7e1483a822d9a +B 716 9e3779b97f4a7c15 77db47760b1f4e26 +B 753 9e3779b97f4a7c15 24533637619e9221 +B 790 9e3779b97f4a7c15 661d6e5870a24407 +B 827 9e3779b97f4a7c15 58e5f4e6f5ded475 +B 864 9e3779b97f4a7c15 ba023421c368f457 +B 901 9e3779b97f4a7c15 c53c1eec8bb6219c +B 938 9e3779b97f4a7c15 e2b319e652a62162 +B 975 9e3779b97f4a7c15 babf5fd50569c103 +B 1012 9e3779b97f4a7c15 a7a07a1d38288877 +B 0 ffffffffffffffff 298f4c84b24f5380 +B 1 ffffffffffffffff 2d980947d3576243 +B 2 ffffffffffffffff 9c5dd4e8d1f0ca29 +B 3 ffffffffffffffff 404443532b1e368c +B 4 ffffffffffffffff 49acf47e1c50f62c +B 5 ffffffffffffffff 8afc5bb89767393a +B 6 ffffffffffffffff 499b51a0e95911a1 +B 7 ffffffffffffffff 5789a59371910208 +B 8 ffffffffffffffff 3ea55f6e8a2173a9 +B 9 ffffffffffffffff ee4a6c5f54646d0f +B 10 ffffffffffffffff 0f4fe68de41294c7 +B 11 ffffffffffffffff 6ac348863b91c923 +B 12 ffffffffffffffff bae2e80c74ce6d3b +B 13 ffffffffffffffff 75a12f16c4aad469 +B 14 ffffffffffffffff 81f2bcd4c961bbf2 +B 15 ffffffffffffffff 68a40398f384bc7c +B 16 ffffffffffffffff 18066aeceefd92d0 +B 17 ffffffffffffffff 2b6e16d533f98868 +B 18 ffffffffffffffff bac08252efe7d0cd +B 19 ffffffffffffffff 486bc59285428277 +B 20 ffffffffffffffff 6064e3f9ec73ccea +B 21 ffffffffffffffff c611433a3d73ad1f +B 22 ffffffffffffffff 4a297171fed5cf99 +B 23 ffffffffffffffff fb3146f0778b032b +B 24 ffffffffffffffff f5edf07adeef0876 +B 25 ffffffffffffffff 2b4ed28b4ad759b6 +B 26 ffffffffffffffff 25d522ddb346af68 +B 27 ffffffffffffffff 10a0cac02fe51cc1 +B 28 ffffffffffffffff e9fe2e08a820e32d +B 29 ffffffffffffffff 1e264c723dc7fae8 +B 30 ffffffffffffffff 4619cd20a6c9b439 +B 31 ffffffffffffffff 2defb84d18feeb81 +B 32 ffffffffffffffff e9bd2f9240517550 +B 33 ffffffffffffffff cca55ec2b43e63e0 +B 34 ffffffffffffffff da1c018fb5f8d63f +B 35 ffffffffffffffff a1cb17c794c9d2ec +B 36 ffffffffffffffff 8104fcbb9aedb5b0 +B 37 ffffffffffffffff e2cbd9fcdd68579f +B 38 ffffffffffffffff 8b004595d8e744f4 +B 39 ffffffffffffffff e1fe4bfd480013e2 +B 40 ffffffffffffffff 352d301f92f2f74e +B 41 ffffffffffffffff 324432de005811de +B 42 ffffffffffffffff f1ae5f5ebbb1a2a4 +B 43 ffffffffffffffff f484effb270fe63c +B 44 ffffffffffffffff 7df7b1747a584f0a +B 45 ffffffffffffffff a319cf217121a987 +B 46 ffffffffffffffff ddac343ee175e3d2 +B 47 ffffffffffffffff 73d3556cb93df6a7 +B 48 ffffffffffffffff df807a2855829d31 +B 49 ffffffffffffffff 4bde26432a3c3f88 +B 50 ffffffffffffffff b5dda7dc6a0e49cd +B 51 ffffffffffffffff b02ca28cc063d49d +B 52 ffffffffffffffff 625e23213b3ed3f4 +B 53 ffffffffffffffff 980b942b5c402bf6 +B 54 ffffffffffffffff f28ee2c6c5308d49 +B 55 ffffffffffffffff 65de7c1ad55217e9 +B 56 ffffffffffffffff b0e6bd034baca628 +B 57 ffffffffffffffff 7a3ad677ef7e92aa +B 58 ffffffffffffffff 675b79fbf43618e5 +B 59 ffffffffffffffff 5d07dcfeb8412ba5 +B 60 ffffffffffffffff 0f208ec0160560ce +B 61 ffffffffffffffff 2f883872a02e0ad0 +B 62 ffffffffffffffff 70594ea0bc4c7e85 +B 63 ffffffffffffffff e3bb1f8684ed65d6 +B 64 ffffffffffffffff 04d6c2ccc30a5b70 +B 65 ffffffffffffffff 7fdeaa092a40427c +B 66 ffffffffffffffff 7ef267b65c9695c3 +B 67 ffffffffffffffff b64554824d71da00 +B 68 ffffffffffffffff a881ee1b38dc8930 +B 69 ffffffffffffffff f970b5d1ee9323c1 +B 70 ffffffffffffffff 10816a23dd8dd4b3 +B 71 ffffffffffffffff a4011456a59d0867 +B 72 ffffffffffffffff fa4cbcf6f688dfbc +B 73 ffffffffffffffff ca3aa7665b48ed0e +B 74 ffffffffffffffff f10a7bfaabcf0f07 +B 75 ffffffffffffffff a6f4804d39bb201d +B 76 ffffffffffffffff 0809ff8fa3a61d60 +B 77 ffffffffffffffff 6bc840a840729f83 +B 78 ffffffffffffffff ecb7c45e752c75e0 +B 79 ffffffffffffffff e9171be24c177d32 +B 80 ffffffffffffffff 4888a8325a24ba06 +B 81 ffffffffffffffff 0887747f38bc97a1 +B 82 ffffffffffffffff b5115fe9fdd53c12 +B 83 ffffffffffffffff 44bcff3ab13fc71c +B 84 ffffffffffffffff 6f423d9307e08e10 +B 85 ffffffffffffffff 10f3c8587e2910dd +B 86 ffffffffffffffff 1ed3a1044293c245 +B 87 ffffffffffffffff 0cbf0ba343d8343a +B 88 ffffffffffffffff 1fa1393556c7281f +B 89 ffffffffffffffff ad2efeb2c420788f +B 90 ffffffffffffffff 7f1db95e18df3187 +B 91 ffffffffffffffff 85645d61dff4d3a4 +B 92 ffffffffffffffff 69b11f5b644581b5 +B 93 ffffffffffffffff 82bab3e292efc894 +B 94 ffffffffffffffff f5345d370f19597a +B 95 ffffffffffffffff e5203b67d488d1e3 +B 96 ffffffffffffffff 1ed5e77a31f3ebd6 +B 97 ffffffffffffffff f3b14354662be35e +B 98 ffffffffffffffff efde95988824315d +B 99 ffffffffffffffff d40070174bb3b43b +B 100 ffffffffffffffff 434e3f8a1dd420c2 +B 101 ffffffffffffffff de28efe35b883091 +B 102 ffffffffffffffff 007d4941ccf62464 +B 103 ffffffffffffffff 184d0dc18aec4c3c +B 104 ffffffffffffffff bd1f9b56bb36cec6 +B 105 ffffffffffffffff cfd3977fd98d4d4f +B 106 ffffffffffffffff b1423f302459fb95 +B 107 ffffffffffffffff 542e071dfb4781ed +B 108 ffffffffffffffff 9c1c3795794bb6ae +B 109 ffffffffffffffff f641ff87487b5814 +B 110 ffffffffffffffff 0719c5b5a2e37408 +B 111 ffffffffffffffff dfe88b2d035d1f60 +B 112 ffffffffffffffff 96132b19d532b7b1 +B 113 ffffffffffffffff 6cdf5ab00b7199fa +B 114 ffffffffffffffff 44835b15ce7e4e9d +B 115 ffffffffffffffff 60071533f0a10ec4 +B 116 ffffffffffffffff 1d45d4977208308f +B 117 ffffffffffffffff 5d63901246d71ba2 +B 118 ffffffffffffffff 9304c422a31cee48 +B 119 ffffffffffffffff 70a591413dc933d8 +B 120 ffffffffffffffff e15063a91b5a7daf +B 121 ffffffffffffffff bb1614b96f33854d +B 122 ffffffffffffffff 793b90f0ff63ff10 +B 123 ffffffffffffffff a2c69a91d9599fac +B 124 ffffffffffffffff 6fa5007d9c1c3623 +B 125 ffffffffffffffff 02284db7dd0c03cb +B 126 ffffffffffffffff aadc920e4f8e4994 +B 127 ffffffffffffffff caafb0c7cb12e3b7 +B 128 ffffffffffffffff 1351f0729e1a166b +B 129 ffffffffffffffff 252bc5ace0e45950 +B 130 ffffffffffffffff d57ff7dbaef3c077 +B 131 ffffffffffffffff 216196de8b14dc6a +B 132 ffffffffffffffff 3f925ad73028d635 +B 133 ffffffffffffffff c2d090bce0ec48e5 +B 134 ffffffffffffffff a913bee07b3eded7 +B 135 ffffffffffffffff 197721ef9d090a9c +B 136 ffffffffffffffff b1506da720cc327d +B 137 ffffffffffffffff 0468d0ec36d3710e +B 138 ffffffffffffffff 15dd52569246c497 +B 139 ffffffffffffffff 745fec721982d9ad +B 140 ffffffffffffffff dedddb1302e78e21 +B 141 ffffffffffffffff 3fe6dd9b54e46ce2 +B 142 ffffffffffffffff 32f6bc2ebeb890da +B 143 ffffffffffffffff 97f343a62c9316f5 +B 144 ffffffffffffffff 0236c5846f24ea68 +B 145 ffffffffffffffff b165815529b52e63 +B 146 ffffffffffffffff 23d79b36366bb17e +B 147 ffffffffffffffff 38d4b433d236c5dc +B 148 ffffffffffffffff f308054a76329e75 +B 149 ffffffffffffffff d57b32b00b197adb +B 150 ffffffffffffffff 5521e4a2a913a404 +B 151 ffffffffffffffff d5d8f871273b17b0 +B 152 ffffffffffffffff 980a810c0e2ee568 +B 153 ffffffffffffffff dc20a51041474fd8 +B 154 ffffffffffffffff 3337c1a72c4ad400 +B 155 ffffffffffffffff f85e435ec34c694a +B 156 ffffffffffffffff 3ab9d5a1bfe1ce4f +B 157 ffffffffffffffff fb444193cbe56ab2 +B 158 ffffffffffffffff 2b1ca46c98e25f73 +B 159 ffffffffffffffff 2ba9830cd7ec23e7 +B 160 ffffffffffffffff ec5b3054ceb0f97a +B 161 ffffffffffffffff c65f42c9ee11988b +B 198 ffffffffffffffff af3faec66d9c893a +B 235 ffffffffffffffff 63a0227746375aef +B 272 ffffffffffffffff 0393dd36aaa122b1 +B 309 ffffffffffffffff 19de4ff95f3488a7 +B 346 ffffffffffffffff 3904a21a7f215f41 +B 383 ffffffffffffffff 10ec94724e64f363 +B 420 ffffffffffffffff f6de5f884a9a7f53 +B 457 ffffffffffffffff a301b81c429dc91e +B 494 ffffffffffffffff 4854c9309690d4ae +B 531 ffffffffffffffff 0cb11d0159bc7f4c +B 568 ffffffffffffffff b39387ef9b16f1fb +B 605 ffffffffffffffff ceed8367bb4bdad3 +B 642 ffffffffffffffff d71dc1fe714f618a +B 679 ffffffffffffffff fcc965ae4e82f73e +B 716 ffffffffffffffff d55dbfbd587e375e +B 753 ffffffffffffffff 3f14e5188140ea17 +B 790 ffffffffffffffff 710664044cd82161 +B 827 ffffffffffffffff a792d1352cdfadb5 +B 864 ffffffffffffffff 292474b9ecb7c600 +B 901 ffffffffffffffff e96f097e8e2e877a +B 938 ffffffffffffffff af4af4040cd9f54b +B 975 ffffffffffffffff 25656e9a6a65fcc4 +B 1012 ffffffffffffffff b9b9b1d3d2593d46 +S ef46db3751d8e999 +S a d24ec4f1a98c6e5b +S abc 44bc2cf5ad770999 +S apple 5889a1c15c94729f +S banana cef162e1813c8ce2 +S key-0 12daf06715ffa373 +S key-199 4b25b6f26e7dd9de +S The quick brown fox jumps over the lazy dog 0b242d361fda71bc diff --git a/tools/constmap_vectors/go.mod b/tools/constmap_vectors/go.mod new file mode 100644 index 0000000..0eae922 --- /dev/null +++ b/tools/constmap_vectors/go.mod @@ -0,0 +1,7 @@ +module github.com/fastfilter/jbinfuse/tools/constmap_vectors + +go 1.24 + +require github.com/lemire/constmap v1.2.0 + +require github.com/cespare/xxhash/v2 v2.3.0 // indirect diff --git a/tools/constmap_vectors/go.sum b/tools/constmap_vectors/go.sum new file mode 100644 index 0000000..3e4b5e7 --- /dev/null +++ b/tools/constmap_vectors/go.sum @@ -0,0 +1,4 @@ +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/lemire/constmap v1.2.0 h1:s01Nysj+TlehXDxb9phBh3e2LUVqyxh4M/lHFGpQzro= +github.com/lemire/constmap v1.2.0/go.mod h1:ypGfd1GuresgQI6Olw7Rgs8kK1CSSeNVtPuSUWg0gnE= diff --git a/tools/constmap_vectors/main.go b/tools/constmap_vectors/main.go new file mode 100644 index 0000000..1b7618b --- /dev/null +++ b/tools/constmap_vectors/main.go @@ -0,0 +1,68 @@ +// Command constmap_vectors prints the serialized size and trailing FNV-1a +// checksum of maps built by github.com/lemire/constmap (Go), the reference +// implementation that the Java ConstMap, VerifiedConstMap and +// PairedVerifiedConstMap must match byte for byte. The checksum covers every +// byte before it, so equal checksums mean equal files. +// +// Run (from this directory): +// go run . > ../../src/test/resources/constmap_vectors.txt +// +// For each size n and key set, the keys are fmt.Sprintf(prefix+"%d", i) for i +// in [0, n), and the value of key i is i * 0x9E3779B97F4A7C15 (mod 2^64). +// Output lines: +// +// where kind is C (ConstMap), V (VerifiedConstMap) or P (PairedVerifiedConstMap). +package main + +import ( + "bytes" + "encoding/binary" + "fmt" + "io" + "os" + + "github.com/lemire/constmap" +) + +func emit(kind, prefix string, n int, w io.WriterTo) { + var b bytes.Buffer + if _, err := w.WriteTo(&b); err != nil { + panic(err) + } + raw := b.Bytes() + sum := binary.LittleEndian.Uint64(raw[len(raw)-8:]) + fmt.Printf("%s %s %d %d %016x\n", kind, prefix, n, len(raw), sum) +} + +func main() { + sizes := []int{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 16, 31, 32, 33, 100, 200, 255, + 256, 1000, 1023, 4096, 10000, 65537, 100000, 999999, 1000000, 1234567} + for _, prefix := range []string{"key-", "clé-日本-"} { + for _, n := range sizes { + if prefix != "key-" && n > 100000 { + continue + } + keys := make([]string, n) + values := make([]uint64, n) + for i := range keys { + keys[i] = fmt.Sprintf(prefix+"%d", i) + values[i] = uint64(i) * 0x9E3779B97F4A7C15 + } + cm, err := constmap.New(keys, values) + if err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + vm, err := constmap.NewVerified(keys, values) + if err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + emit("C", prefix, n, cm) + emit("V", prefix, n, vm) + if n > 0 { + emit("P", prefix, n, vm.Paired()) + } + } + } +} diff --git a/tools/xxh64_vectors.c b/tools/xxh64_vectors.c new file mode 100644 index 0000000..d34ff5d --- /dev/null +++ b/tools/xxh64_vectors.c @@ -0,0 +1,45 @@ +// Generates XXH64 test vectors from the reference C implementation of xxHash +// (https://github.com/Cyan4973/xxHash, xxhash.h; any release since 0.8 gives +// the same XXH64 results), for the Java XxHash64 class. +// +// Build and run (from the repository root): +// cc -O2 -I path/to/xxhash tools/xxh64_vectors.c -o /tmp/xxh64_vectors +// /tmp/xxh64_vectors > src/test/resources/xxh64_vectors.txt +// +// The input bytes are produced by the same 64-bit LCG as tools/wyhash_vectors.c, +// which the Java test reproduces: +// state = state * 6364136223846793005 + 1442695040888963407; byte = state >> 56 +// Output lines: +// B XXH64(data[0..len), len, seed) +// S XXH64 of an ASCII string, seed 0 +#define XXH_INLINE_ALL +#include +#include +#include "xxhash.h" + +int main(void) { + static uint8_t data[1024]; + uint64_t state = 0x9E3779B97F4A7C15ULL; + for (int i = 0; i < 1024; i++) { + state = state * 6364136223846793005ULL + 1442695040888963407ULL; + data[i] = (uint8_t)(state >> 56); + } + const uint64_t seeds[] = {0, 1, 0x9E3779B97F4A7C15ULL, 0xFFFFFFFFFFFFFFFFULL}; + for (size_t s = 0; s < sizeof(seeds) / sizeof(seeds[0]); s++) { + for (int len = 0; len <= 160; len++) { + printf("B %d %016llx %016llx\n", len, (unsigned long long)seeds[s], + (unsigned long long)XXH64(data, (size_t)len, seeds[s])); + } + for (int len = 161; len <= 1024; len += 37) { + printf("B %d %016llx %016llx\n", len, (unsigned long long)seeds[s], + (unsigned long long)XXH64(data, (size_t)len, seeds[s])); + } + } + const char *strings[] = {"", "a", "abc", "apple", "banana", "key-0", "key-199", + "The quick brown fox jumps over the lazy dog"}; + for (size_t i = 0; i < sizeof(strings) / sizeof(strings[0]); i++) { + printf("S %s %016llx\n", strings[i], + (unsigned long long)XXH64(strings[i], strlen(strings[i]), 0)); + } + return 0; +}