Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,10 +104,10 @@ cmake -S . -B build -DRABITQ_BUILD_TESTS=ON \
-DRABITQ_BUILD_SAMPLES=OFF -DCMAKE_BUILD_TYPE=Release \
-DRABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF
cmake --build build --parallel 2
ctest --test-dir build --output-on-failure
ctest --no-tests=error --test-dir build --output-on-failure
```

The first configuration needs Git and network access to download GoogleTest.
The first configuration needs network access to download GoogleTest.
Increase `--parallel 2` if your machine has enough memory for more compiler jobs.
Native CPU tuning is disabled here to keep generic code portable. On Windows
or Apple Silicon, use the matching commands in [the test guide](../tests/README.md#quick-start).
Expand Down
1 change: 0 additions & 1 deletion .github/scripts/ci_changes.py
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,6 @@ def selects_core(path):
"build_indexes",
"download_datasets",
"install_baselines",
"plot_homepage",
"plot_results",
"query_index",
"reproduce",
Expand Down
5 changes: 2 additions & 3 deletions .github/scripts/test_ci_changes.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ def test_known_support_paths_do_not_select_unrelated_builds(self):
".github/ISSUE_TEMPLATE/bug_report.yml": set(),
".github/CODEOWNERS": set(),
".github/release.yml": {"python_quality"},
"benchmarks/plot_homepage.py": {"python_quality"},
"benchmarks/plot_results.py": {"python_quality"},
"sample/run_cpp.sh": {"shell"},
"sample/run_python.sh": {"shell"},
"example.sh": {"shell"}, # Deleted paths still occur in a push diff.
Expand All @@ -87,7 +87,7 @@ def test_known_support_paths_do_not_select_unrelated_builds(self):
self.assertEqual(classify([path]), expected)

def test_support_paths_do_not_mask_source_or_unknown_changes(self):
support = [".github/ISSUE_TEMPLATE/config.yml", "benchmarks/plot_homepage.py"]
support = [".github/ISSUE_TEMPLATE/config.yml", "benchmarks/plot_results.py"]
self.assertEqual(
classify(support + ["src/core/api.cpp"]),
{
Expand Down Expand Up @@ -364,7 +364,6 @@ def test_tagged_main_routing_changes_preserve_lightweight_companion_checks(self)
for companions, expected in (
(["scripts/check-python.sh"], {"python_quality", "shell"}),
(["sample/run_cpp.sh"], {"python_quality", "shell"}),
(["benchmarks/plot_homepage.py"], {"python_quality"}),
(["benchmarks/build_indexes.py"], {"python_quality"}),
([".github/ISSUE_TEMPLATE/config.yml"], {"python_quality"}),
(["src/core/api.cpp"], set(GROUPS)),
Expand Down
1 change: 1 addition & 0 deletions .github/scripts/test_ci_workflow_contracts.py
Original file line number Diff line number Diff line change
Expand Up @@ -869,6 +869,7 @@ def test_documentation_summary_is_available_for_non_doc_prs(self):
self.assertEqual(needs(build), ["changes"])
self.assertIn("needs.changes.outputs.docs == 'true'", build["if"])
self.assertIn("needs.changes.result != 'success'", build["if"])
self.assertIn("github.event_name == 'push'", build["if"])
self.assertEqual(classify(["docs/docs/quick_start.md"]), {"docs"})


Expand Down
4 changes: 3 additions & 1 deletion .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,9 @@ jobs:

build:
needs: changes
if: ${{ !cancelled() && (needs.changes.result != 'success' || needs.changes.outputs.docs == 'true') }}
# Main pushes always deploy: a newer push can cancel a docs-changing run, and
# its own diff would otherwise leave the site at the older commit.
if: ${{ !cancelled() && (github.event_name == 'push' || needs.changes.result != 'success' || needs.changes.outputs.docs == 'true') }}
name: Build documentation
runs-on: ubuntu-latest
timeout-minutes: 10
Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ Use `ctest --no-tests=error`, including filtered and installed-consumer runs.

C++17 quantization/ANN library with Python bindings (`rabitqlib`). Components depend in one
direction: `index` → `fastscan` → `core`; core and FastScan support OpenMP OFF, ANN/Python require ON.
Stable float32 v1 API: `include/rabitqlib/core/api.hpp`. Implementations: `src/{core,simd,fastscan,index}/`.
Stable float32 v1 API: `include/rabitqlib/core/api.hpp`. Implementations: `src/{core,simd,utils,fastscan,index}/`.
Bindings: `python_bindings/`; tests: `tests/{unit,python}/`; site sources: `docs/docs/`.
See the [full code map](docs/development.md#project-and-code-map) when locating unfamiliar code.

Expand Down
56 changes: 28 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
<div align="center">

<p><img src="docs/docs/assets/logo.svg" alt="RaBitQ Library logo" width="800"></p>
<p><img src="https://vectordb-ntu.github.io/RaBitQ-Library/assets/logo.svg" alt="RaBitQ Library logo" width="800"></p>

<p>
<a href="https://pypi.org/project/rabitqlib/"><img alt="PyPI" src="https://img.shields.io/pypi/v/rabitqlib.svg?cacheSeconds=300"></a>
Expand All @@ -12,17 +12,17 @@
</div>

RaBitQ-Library is a research-backed library for vector quantization and
approximate nearest-neighbor search. See the [research papers](docs/docs/papers.md).
approximate nearest-neighbor search. See the [research papers](https://vectordb-ntu.github.io/RaBitQ-Library/papers/).

## News

- **v0.7.2:** Python type hints, expanded API documentation, path-like inputs,
clearer input errors, and concise index representations. Internal quantization
refactoring and streamlined regression checks preserve existing APIs and index formats.
See the [release notes](docs/docs/releases/0.7.2.md).
See the [release notes](https://vectordb-ntu.github.io/RaBitQ-Library/releases/0.7.2/).
- **v0.7.0:** Modular C++ components, stable core API v1, optional
OpenMP for numerical components, and unified index counts. See the
[upgrade notes](docs/docs/releases/0.7.0.md).
[upgrade notes](https://vectordb-ntu.github.io/RaBitQ-Library/releases/0.7.0/).

## Vector indexes

Expand All @@ -33,14 +33,14 @@ QPS–recall tradeoffs.

| Index | Search approach | Storage and updates |
| --- | --- | --- |
| [IVF-RaBitQ](docs/docs/index/ivf.md) | Partitioned search with batch scoring. | 1–9-bit codes or raw-vector reranking; add/remove. |
| [HNSW-RaBitQ](docs/docs/index/hnsw.md) | Graph traversal over quantized vectors. | 1–9-bit codes plus graph edges; add/remove and resize. |
| [SymphonyQG](docs/docs/index/qg.md) | Graph search with batched distance estimation. | Raw or 4/8-bit storage; updates require rebuilding. |
| [IVF-RaBitQ](https://vectordb-ntu.github.io/RaBitQ-Library/index/ivf/) | Partitioned search with batch scoring. | 1–9-bit codes or raw-vector reranking; add/remove. |
| [HNSW-RaBitQ](https://vectordb-ntu.github.io/RaBitQ-Library/index/hnsw/) | Graph traversal over quantized vectors. | 1–9-bit codes plus graph edges; add/remove and resize. |
| [SymphonyQG](https://vectordb-ntu.github.io/RaBitQ-Library/index/qg/) | Graph search with batched distance estimation. | Raw or 4/8-bit storage; updates require rebuilding. |

[Native clustering methods](docs/docs/clustering.md) prepare centroids and
[Native clustering methods](https://vectordb-ntu.github.io/RaBitQ-Library/clustering/) prepare centroids and
assignments for IVF-RaBitQ and HNSW-RaBitQ.

See the [benchmarks](docs/docs/benchmarks/bench.md) for measured results,
See the [benchmarks](https://vectordb-ntu.github.io/RaBitQ-Library/benchmarks/bench/) for measured results,
settings, and limitations.

## Quick start
Expand All @@ -53,7 +53,7 @@ python -m pip install --upgrade "rabitqlib>=0.7.2"

Wheels support CPython 3.11–3.14 on Linux x86-64/ARM64, Windows x86-64,
and macOS 14+ ARM64. x86-64 requires AVX2/FMA and optionally uses
AVX-512; ARM64 uses NEON. See [platform requirements and source installation](docs/docs/installation.md#requirements).
AVX-512; ARM64 uses NEON. See [platform requirements and source installation](https://vectordb-ntu.github.io/RaBitQ-Library/installation/#requirements).

Build and search a small IVF-RaBitQ index with synthetic data; no dataset download is
required. This example demonstrates the API, not performance.
Expand All @@ -78,17 +78,17 @@ print(ids.shape, distances.shape) # (5, 10) (5, 10)

IDs identify input rows; distances here estimate squared L2 distance, with smaller
values first. Quantization remains approximate even when all clusters are probed.
The [complete quick start](docs/docs/quick_start.md) explains result handling,
The [complete quick start](https://vectordb-ntu.github.io/RaBitQ-Library/quick_start/) explains result handling,
measures recall while varying `nprobe`, and shows save/load.

For other indexes, see the [HNSW-RaBitQ](docs/docs/index/hnsw.md#minimal-workflow) and
[SymphonyQG](docs/docs/index/qg.md#minimal-workflow) examples.
For other indexes, see the [HNSW-RaBitQ](https://vectordb-ntu.github.io/RaBitQ-Library/index/hnsw/#minimal-workflow) and
[SymphonyQG](https://vectordb-ntu.github.io/RaBitQ-Library/index/qg/#minimal-workflow) examples.

### C++

Use the [C++ integration guide](docs/docs/cpp_integration.md) for source builds,
Use the [C++ integration guide](https://vectordb-ntu.github.io/RaBitQ-Library/cpp_integration/) for source builds,
component linking, and installation. Start with the
[C++ index examples](docs/docs/cpp_integration.md#c-examples), or use the
[C++ index examples](https://vectordb-ntu.github.io/RaBitQ-Library/cpp_integration/#c-examples), or use the
quantization components below in your own engine.

## Quantization components
Expand All @@ -98,12 +98,12 @@ engine, independently of the included indexes.

| Component | What it provides | Start here |
| --- | --- | --- |
| `rabitqlib::core` | Rotation, per-vector encoding, query preparation, and distance estimation. | [Core example](sample/cpp/core/) · [Stable v1 API](docs/docs/rabitq/core_api.md) |
| `rabitqlib::fastscan` | Batch encoding and lookup-table scoring over 32-vector layouts, built on core. | [FastScan example](sample/cpp/fastscan/) |
| `rabitqlib::core` | Rotation, per-vector encoding, query preparation, and distance estimation. | [Core example](https://github.com/VectorDB-NTU/RaBitQ-Library/tree/main/sample/cpp/core) · [Stable v1 API](https://vectordb-ntu.github.io/RaBitQ-Library/rabitq/core_api/) |
| `rabitqlib::fastscan` | Batch encoding and lookup-table scoring over 32-vector layouts, built on core. | [FastScan example](https://github.com/VectorDB-NTU/RaBitQ-Library/tree/main/sample/cpp/fastscan) |

See [component integration](docs/docs/cpp_integration.md#core-only-c-integration)
for build options and compatibility, the [quantizer guide](docs/docs/rabitq/quantizer.md)
for encoding choices, and the [core benchmark](benchmarks/core/README.md) for
See [component integration](https://vectordb-ntu.github.io/RaBitQ-Library/cpp_integration/#core-only-c-integration)
for build options and compatibility, the [quantizer guide](https://vectordb-ntu.github.io/RaBitQ-Library/rabitq/quantizer/)
for encoding choices, and the [core benchmark](https://github.com/VectorDB-NTU/RaBitQ-Library/blob/main/benchmarks/core/README.md) for
component throughput and estimation error.

## Adoption and resources
Expand All @@ -114,8 +114,8 @@ Two examples of direct library integration:

| Project | Integration |
| --- | --- |
| [txtai](docs/docs/integrations/txtai.md) | Uses the Python IVF-RaBitQ and HNSW-RaBitQ indexes as an embedding-search backend. |
| [zvec](docs/docs/integrations/zvec.md) | Uses core quantization in its own IVF-RaBitQ and HNSW-RaBitQ indexes, with FastScan batch scoring in IVF-RaBitQ. |
| [txtai](https://vectordb-ntu.github.io/RaBitQ-Library/integrations/txtai/) | Uses the Python IVF-RaBitQ and HNSW-RaBitQ indexes as an embedding-search backend. |
| [zvec](https://vectordb-ntu.github.io/RaBitQ-Library/integrations/zvec/) | Uses core quantization in its own IVF-RaBitQ and HNSW-RaBitQ indexes, with FastScan batch scoring in IVF-RaBitQ. |

The linked walkthroughs describe integration boundaries and provide versioned
source references.
Expand Down Expand Up @@ -191,15 +191,15 @@ listed above.

### Further reading

- [Documentation](https://vectordb-ntu.github.io/RaBitQ-Library/) and [algorithm overview](docs/docs/rabitq/rabitq.md).
- [Papers and citations](docs/docs/papers.md) for the research behind the library.
- [Release notes](https://github.com/VectorDB-NTU/RaBitQ-Library/releases) and [0.7 migration guide](docs/docs/modular_migration.md).
- [Documentation](https://vectordb-ntu.github.io/RaBitQ-Library/) and [algorithm overview](https://vectordb-ntu.github.io/RaBitQ-Library/rabitq/rabitq/).
- [Papers and citations](https://vectordb-ntu.github.io/RaBitQ-Library/papers/) for the research behind the library.
- [Release notes](https://github.com/VectorDB-NTU/RaBitQ-Library/releases) and [0.7 migration guide](https://vectordb-ntu.github.io/RaBitQ-Library/modular_migration/).
- [GPU implementation](https://github.com/Stardust-SJF/cuvs_rabitq/tree/cuvs_ivf_rabitq) in cuVS.

## Contributing

Start with the [contribution guide](.github/CONTRIBUTING.md#your-first-contribution)
or [starter tasks](.github/CONTRIBUTING.md#starter-tasks). Report bugs and request
Start with the [contribution guide](https://github.com/VectorDB-NTU/RaBitQ-Library/blob/main/.github/CONTRIBUTING.md#your-first-contribution)
or [starter tasks](https://github.com/VectorDB-NTU/RaBitQ-Library/blob/main/.github/CONTRIBUTING.md#starter-tasks). Report bugs and request
features through [GitHub Issues](https://github.com/VectorDB-NTU/RaBitQ-Library/issues/new/choose).

## Authors and acknowledgements
Expand All @@ -213,4 +213,4 @@ Jiabao Jin, Xiaoyao Zhong, and Jinjing Zhou for their valuable feedback.

## License

RaBitQ Library is available under the [Apache License 2.0](LICENSE).
RaBitQ Library is available under the [Apache License 2.0](https://github.com/VectorDB-NTU/RaBitQ-Library/blob/main/LICENSE).
2 changes: 1 addition & 1 deletion benchmarks/DBPEDIA_MANIFEST.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"dataset": "openai-1536-angular",
"source": "SuperKMeans benchmark data_openai.bin and data_openai_test.bin",
"source": "openai-1536-angular.hdf5 train/test splits (Google Drive file 1cwIY76n_HEbbZAANJVWiVkjBsIy3b9jB), converted to fvecs",
"metric": "inner_product",
"normalization": "unit-normalized source values preserved exactly",
"dimension": 1536,
Expand Down
4 changes: 2 additions & 2 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,8 @@
For embedding the quantizer in another system, use the independent
[core C++ benchmark](core/README.md). It needs only core and reports per-repeat timings,
distance-estimate error and quantized full-scan recall. The workflows below benchmark
complete ANN/clustering systems. For dispatch, encoding, FastScan, sidecar loading and
Python ID-conversion regression measurements, use the
complete ANN/clustering systems. For dispatch, encoding, FastScan, sidecar loading, IVF add
and Python ID-conversion regression measurements, use the
[component hot-path benchmarks](hotpaths/README.md).

See the [benchmark overview](../docs/docs/benchmarks/bench.md) for datasets,
Expand Down
8 changes: 4 additions & 4 deletions benchmarks/core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ cmake -S . -B build-core-bench -DCMAKE_BUILD_TYPE=Release \
-DRABITQ_BUILD_SAMPLES=OFF -DRABITQ_USE_OPENMP=OFF \
-DRABITQ_BUILD_BENCHMARKS=ON -DRABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF
cmake --build build-core-bench --parallel
ctest --test-dir build-core-bench --output-on-failure
ctest --no-tests=error --test-dir build-core-bench --output-on-failure
./build-core-bench/benchmarks/core/rabitq_core_benchmark > core-results.csv
```

Expand All @@ -22,7 +22,7 @@ For an installed package, configure this directory directly:
cmake -S benchmarks/core -B build-core-bench-installed \
-DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH="<core-install-prefix>"
cmake --build build-core-bench-installed --parallel
ctest --test-dir build-core-bench-installed --output-on-failure
ctest --no-tests=error --test-dir build-core-bench-installed --output-on-failure
./build-core-bench-installed/rabitq_core_benchmark 128 4 4096 64 7 > core-results.csv
```

Expand Down Expand Up @@ -107,8 +107,8 @@ valid code/factor prototypes are encoded, then replicated at distinct memory add
This models access and call costs, not dataset-dependent ANN quality. Inputs are already
in the numerical working domain; no rotation or top-k retrieval is timed or evaluated.
The metric is L2 with a zero centroid. Prototype scores must match all three references
exactly before timing. Raw timings and double-precision checksums are printed for every
path and repetition; path order rotates and each timed pass has its own warmup.
within a computed float32 rounding bound before timing. Raw timings and
double-precision checksums are printed for every path and repetition; path order rotates and each timed pass has its own warmup.

Compare baseline/candidate binaries with identical harness source, compiler flags and
parameters. Alternate process order, retain all samples, and verify checksums alongside
Expand Down
9 changes: 7 additions & 2 deletions benchmarks/hotpaths/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ Enable `RABITQ_BUILD_BENCHMARKS=ON` in a Release build. Targets follow enabled c
| `rabitq_core_hotpaths` | core only | One-bit compact encoding or raw L2 calls |
| `rabitq_fastscan_hotpaths` | FastScan | Cached standard/HACC batch scoring |
| `rabitq_load_hotpaths` | index | HNSW centroid-sidecar load and validation |
| `rabitq_ivf_add_hotpaths` | index | One IVF `add` call on a freshly loaded index |
| `rabitq_ids_benchmark` module + `ids.py` | Python bindings | Production cluster-ID conversion helper |

No additional production dependencies are introduced. Core/FastScan benchmarks support
Expand All @@ -27,7 +28,7 @@ cmake -S . -B build-hotpaths -DCMAKE_BUILD_TYPE=Release \
-DRABITQ_BUILD_BENCHMARKS=ON -DRABITQ_BUILD_SAMPLES=OFF \
-DRABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF
cmake --build build-hotpaths --parallel
ctest --test-dir build-hotpaths -R 'Hotpath' --output-on-failure
ctest --no-tests=error --test-dir build-hotpaths -R 'Hotpath' --output-on-failure
```

Set both `RABITQ_BUILD_INDEX=OFF` and `RABITQ_BUILD_FASTSCAN=OFF` for pure core.
Expand All @@ -50,6 +51,9 @@ Examples (from the repository root):
./build-hotpaths/benchmarks/hotpaths/rabitq_load_hotpaths prepare "$TMPDIR/centroids" 128 16384 1
./build-hotpaths/benchmarks/hotpaths/rabitq_load_hotpaths load "$TMPDIR/centroids" 128 16384 7

# path dim clusters base_n add_n repetitions single|routed; path receives the base index
./build-hotpaths/benchmarks/hotpaths/rabitq_ivf_add_hotpaths "$TMPDIR/ivf-add" 64 8 256 16 3 routed

python benchmarks/hotpaths/ids.py --module-dir build-hotpaths/python_bindings
```

Expand All @@ -61,7 +65,8 @@ the deterministic input. Original production indexes are not benchmark fixtures.

- IVF add verifies the expected point counts and samples new points' self-neighbors
outside the timed call, in addition to comparing output hashes across repetitions.
- Inputs use an LCG with seed 42 (query/centroid seed 91 where applicable), metric L2.
- Inputs use an LCG with seed 42 (query/centroid seed 91 where applicable; IVF add uses
seed 91 for added points and seed 7 for centroids), metric L2.
Dimensions for encoding/scoring are multiples of 32. Use 64, 128 and 960 dimensions;
encoding rows 32/4096 and threads 1/4 retain the cases from the original investigation.
- Core encoding times the existing serial template at one thread and OpenMP template
Expand Down
Loading
Loading