Skip to content

Add constant maps compatible with constmap, rsconstmap and fastconstmap - #2

Merged
lemire merged 1 commit into
mainfrom
constmap
Nov 4, 2026
Merged

lemire merged 1 commit into
mainfrom
constmap

Conversation

@lemire

@lemire lemire commented Nov 4, 2026

Copy link
Copy Markdown
Member

Adds ConstMap, VerifiedConstMap and PairedVerifiedConstMap: immutable maps from string or byte-array keys to 64-bit values, built with the binary fuse construction. They are a Java port of constmap (Go), and their files are interchangeable with rsconstmap (Rust) and fastconstmap (C/Python).

VerifiedConstMap vm = VerifiedConstMap.build(List.of("apple", "banana"), new long[] {100, 200});
vm.get("banana");   // 200
vm.get("grape");    // VerifiedConstMap.NOT_FOUND
Files.write(Path.of("map.vmap"), vm.toByteArray());   // readable by Go, Rust, Python

Compatibility

  • Uses the same key hash (XXH64 of the UTF-8 bytes), mixing, splitmix64 seed sequence, segment parameters, retry schedule and peeling order as the Go code.
  • Byte-identical output with Go and Rust for CMAP0001, VMAP0001 and PMAP0001, checked two ways:
    • against rust.* from constmap's testdata/interop (copied to src/test/resources/constmap/);
    • against the Go package itself for 29 sizes from 0 to 1,234,567 keys, with ASCII and non-ASCII keys. tools/constmap_vectors (Go) records each file's size and trailing FNV-1a checksum, which covers every byte, so no large fixtures are committed. This covers the small-size parameters, the retries with halved and doubled segments below 1M keys, and the path without retries above.
  • Reads the files of fastconstmap 0.10+, including CMAP0003 with its key count, which is ignored, like the key count fastconstmap puts in the padding word.
  • Refuses fastconstmap 0.9 files (CMAP0002/VCMP0002, XXH3) with a message that names them. A file of the wrong map kind is also named.
  • load checks the magic, the segment parameters (power-of-two segment length, (count+2)*length == array length) and the checksum, and throws InvalidFilterException on failure.

Implementation notes

  • XxHash64: a dependency-free XXH64, checked against 740 vectors from the reference C xxhash (tools/xxh64_vectors.c). hash(String) hashes ASCII strings straight from their chars, with no allocation (11.7 vs 17.6 ns), and encodes other strings to UTF-8 first. lz4-java and zero-allocation-hashing also provide XXH64; I kept the library dependency-free.
  • Construction follows Go, with one change. Go recovers values through a map[hash]value. Here each cell keeps the XOR of its key indices next to the XOR of the hashes, so a singleton cell names its key directly. Peeling order and output are unchanged.
  • getMany hashes a block of 8 keys before reading the arrays: 25–40% faster at 10M keys.
  • Maps are on-heap only. Zero-copy views (as in BinaryFuseFilter.wrap, or fastconstmap's from_buffer) are left for a follow-up.

Performance (Apple M4 Max, JDK 27, present keys in random order)

1M keys 10M keys
ConstMap.get(String) 16.7 ns 33.0 ns
ConstMap.getMany 16.2 ns 24.8 ns
VerifiedConstMap.get(String) 25.8 ns 65.4 ns
PairedVerifiedConstMap.get(String) 23.1 ns 44.9 ns

./mvnw package passes: 66 tests, javadoc doclint clean.

ConstMap, VerifiedConstMap and PairedVerifiedConstMap are immutable maps
from string or byte-array keys to 64-bit values built with the binary fuse
construction, ported from github.com/lemire/constmap (Go). They share the
key hash (XXH64), mixing, seed sequence, layout and file formats with
constmap, rsconstmap (Rust) and fastconstmap (C/Python):

- for the same input they write the same bytes as Go and Rust (CMAP0001,
  VMAP0001, PMAP0001), checked against the interop files of the constmap
  repository and against the Go package for sizes from 0 to 1,234,567 keys
  (tools/constmap_vectors);
- they read fastconstmap's CMAP0003 and ignore its key counts, and name the
  legacy fastconstmap 0.9 formats (XXH3) instead of misreading them;
- load validates magic, segment parameters and the FNV-1a checksum.

Also adds XxHash64, a dependency-free XXH64 checked against vectors from the
reference C implementation (tools/xxh64_vectors.c), with an allocation-free
path for ASCII strings, plus batched getMany lookups and a JMH benchmark.
@lemire
lemire merged commit 1e894eb into main Nov 4, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant