feat(python): train, set_centroids and add for IvfIndex and HnswIndex

README.md

@@ -60,26 +60,26 @@ required. This example demonstrates the API, not performance.
 
 ```python
 import numpy as np
-from rabitqlib import FinalAssignmentMode, IvfIndex, RaBitQKMeans
+from rabitqlib import IvfIndex
 
 rng = np.random.default_rng(42)
 data = rng.standard_normal((500, 64)).astype(np.float32)
 queries = rng.standard_normal((5, 64)).astype(np.float32)
 
-clustering = RaBitQKMeans(
-    64, 5, num_threads=2, final_assignment=FinalAssignmentMode.Exact
-)
-clustering.train(data)
-index = IvfIndex(dim=64, max_elements=len(data), num_clusters=5, nbits=4, metric="l2")
-index.build(data, clustering.centroids, clustering.assignments)
+index = IvfIndex(64, 5, nbits=4, metric="l2")
+index.train(data)
+index.add(data)
 ids, distances = index.search(queries, k=10, nprobe=5)
 print(ids.shape, distances.shape)  # (5, 10) (5, 10)
 ```
 
-IDs identify input rows; distances here estimate squared L2 distance, with smaller
-values first. Quantization remains approximate even when all clusters are probed.
+`train` learns the centroids, and `add` quantizes the vectors into their
+clusters. IDs identify added rows; distances here estimate squared L2 distance,
+with smaller values first. Quantization remains approximate even when all clusters are probed.
 The [complete quick start](https://vectordb-ntu.github.io/RaBitQ-Library/quick_start/) explains result handling,
-measures recall while varying `nprobe`, and shows save/load.
+measures recall while varying `nprobe`, and shows save/load. Code written for
+0.7 still runs in 0.8 with deprecation warnings; see the
+[migration guide](https://vectordb-ntu.github.io/RaBitQ-Library/python_migration/).
 
 For other indexes, see the [HNSW-RaBitQ](https://vectordb-ntu.github.io/RaBitQ-Library/index/hnsw/#minimal-workflow) and
 [SymphonyQG](https://vectordb-ntu.github.io/RaBitQ-Library/index/qg/#minimal-workflow) examples.

docs/docs/clustering.md

@@ -7,6 +7,14 @@ algorithms based on the cluster count.
 
 For installation, see [Quick start](quick_start.md#install).
 
+You do not need these classes to build an index: `IvfIndex.train` and
+`HnswIndex.train` run them for you (see
+[IVF-RaBitQ training](index/ivf.md#training-centroids-in-python)). Use them
+directly to inspect assignments, reuse one clustering across several indexes,
+or pass the result to `set_centroids` and `add(..., cluster_ids=...)`. The
+indexes train with their own defaults, including sampling at most 256 vectors
+per centroid; the classes below default to training on every vector.
+
 ## Choose a method
 
 | Class | Assignment | Recommended use | Cluster count |
@@ -118,8 +126,11 @@ and `n > k * max_points_per_centroid`, training draws
 does; Faiss enables this by default with 256. Centroid initialization and all
 Lloyd iterations use only this sample. The final centroids then assign all `n`
 vectors with the selected `final_assignment` mode, as [`assign`](#assign-new-vectors)
-does, so `assignments` and `distances` keep shape `(n,)` and can be passed
-directly to index builders.
+does, so `assignments` and `distances` keep shape `(n,)` and `assignments` can be
+passed directly to an index's `add(..., cluster_ids=...)`. C++ callers that need
+only the centroids can call `train_centroids(n, x)` instead of `train`, which
+skips this final assignment and leaves `assignments`, `distances` and
+`final_obj` empty or zero.
 
 `seed` selects the sample through a random stream separate from centroid
 initialization. Training that does not sample, because the parameter is `0` or

docs/docs/index/hnsw.md

@@ -13,20 +13,25 @@ degree, and index parameters.
 ### Python
 
 Run the [Quick start](../quick_start.md#build-and-search-an-ivf-index) to prepare
-`data`, `queries` and `clustering`, then use those same arrays with HNSW-RaBitQ:
+`data` and `queries`, then use those same arrays with HNSW-RaBitQ:
 
 ```python
 from rabitqlib import HnswIndex
 
-index = HnswIndex(
-    dim=data.shape[1], max_elements=len(data), M=16, ef_construction=100, nbits=4,
-)
-index.build(data, clustering.centroids, clustering.assignments)
+index = HnswIndex(data.shape[1], nbits=4, M=16, ef_construction=100)
+index.train(data)  # 16 RaBitQKMeans centroids by default
+index.add(data, num_threads=1)  # one thread gives a reproducible graph
 ids, distances = index.search(queries, k=10, ef=100)
 index.save("hnsw.index")
 loaded = HnswIndex.load("hnsw.index")
 ```
 
+`train` and `set_centroids` work as for [IVF-RaBitQ](ivf.md#training-centroids-in-python),
+with `num_clusters` centroids (default 16) and always RaBitQKMeans; `index.cp`
+starts from the same defaults with `seed`'s low 32 bits. The first `add` into the
+empty index links vectors with their exact coordinates, like the 0.7 `build`;
+later adds link through the stored codes (see [Recall after `add`](#recall-after-add)).
+
 ### C++
 
 Link `rabitqlib::index`. Here `data`, `centroids` and `queries` are row-major
@@ -58,16 +63,19 @@ see [C++ integration](../cpp_integration.md).
 | Parameter | Contract |
 | --- | --- |
 | `dim` | Original input dimension; Python supports 64–65,536 and pads internally. |
-| `max_elements` | Capacity; build accepts 1 through this many rows. Reserve extra capacity or call `resize` before adding beyond it. |
-| `nbits` / C++ `total_bits` | 1–9 total bits including the sign bit; Python defaults to 8. Raw-vector storage is not supported. |
-| `M`, `ef_construction` | Graph connectivity and construction search budget; Python defaults are 16 and 200. `M` must be at least 2 and is capped at 10,000; `ef_construction` is raised to at least `M`. |
-| `metric`, `random_seed` | Python defaults are `"l2"` and 100; `"ip"` is also supported. |
-| `k`, `ef` | Positive result count and search window. Python requires `k <= max_elements`; omitted/zero `ef` uses `max(k, 10)`, positive `ef` is raised to at least `k`, and `ef` above the stored point count is clamped to it. |
-
-`random_seed` controls graph levels and the FHT/Kac rotation (using its low 32 bits).
+| `nbits` / C++ `total_bits` | Required and, in Python, keyword-only: 1–9 total bits including the sign bit. Raw-vector storage is not supported. |
+| `num_clusters` | Python keyword, default 16: centroids `train` learns or `set_centroids` supplies. |
+| `M`, `ef_construction` | Graph connectivity and construction search budget; Python defaults are 16 and 200. `M` must be at least 2 and is capped at 10,000; `ef_construction` is raised to at least `M`. The base layer keeps up to `2 * M` edges per node. |
+| `metric`, `seed` | Python defaults are `"l2"` and 100; `"ip"` is also supported. `seed` was called `random_seed` before 0.8. |
+| `max_elements` | Python reports the current capacity, which `add` grows geometrically (at least doubling) and `resize` can preallocate. C++ `add` never grows it. |
+| `k`, `ef` | Positive result count and search window. Python requires `k <= num_stored_points`, so a trained index without points rejects every `k`; omitted/zero `ef` uses `max(k, 10)`, positive `ef` is raised to at least `k`, and `ef` above the stored point count is clamped to it. |
+
+`seed` controls graph levels and the FHT/Kac rotation (using its low 32 bits).
 With identical inputs and parameters, single-threaded construction is reproducible
-within the same build environment. Parallel insertion order and different toolchains
-or SIMD backends can still change results. Loading restores the saved rotation.
+within the same build environment. Python `train` and `add` default to every
+available CPU, so **default builds are not reproducible**: pass `num_threads=1` to
+`add` when you need the same graph again. Different toolchains or SIMD backends can
+still change results. Loading restores the saved rotation.
 
 Results use squared L2 or `1 - dot` estimates, ordered from smallest to largest.
 Python fills missing results with ID `2**32 - 1` and distance `+inf`; C++ returns
@@ -196,11 +204,9 @@ void HierarchicalNSW::resize(size_t new_max_elements);
   In C++, initialize the index through its parameterized constructor or `load`
   before calling `resize`.
 
-In Python:
+In Python, `add` grows the capacity when needed:
 
 ```python
-# Grow before adding; preserve any capacity already available.
-index.resize(max(index.max_elements, index.num_stored_points + len(vectors)))
 new_ids = index.add(vectors)                       # route to the nearest centroids
 # Alternatively, choose the clusters instead of the call above:
 # new_ids = index.add(vectors, cluster_ids=labels)
@@ -221,9 +227,10 @@ retrying, or rebuild to restore full reachability.
 See the shared [index count contract](usage.md#index-counts). Removed
 records still occupy capacity.
 
-`add` never grows the index: it rejects a whole batch whose size exceeds
+C++ `add` never grows the index: it rejects a whole batch whose size exceeds
 `max_elements() - num_stored_points()`, and `resize` is the only way to raise
-that. Reserving capacity avoids reallocating and copying the whole base-layer record storage during `add`.
+that. Python `add` calls `resize` first when the batch does not fit, to at least
+double the capacity. Reserving capacity avoids reallocating and copying the whole base-layer record storage during `add`.
 It does not make insertion allocation-free: batch scratch, upper-layer link lists,
 and label-map entries still require allocations, and the label map may rehash.
 

docs/docs/index/ivf.md

@@ -14,8 +14,9 @@ width, number of clusters, and `nprobe`.
 ### Python
 
 The [complete Quick Start](../quick_start.md) is the runnable IVF-RaBitQ example:
-train centroids, call `IvfIndex.build`, search, then save and reload the index.
-The update examples below reuse its built `index`.
+create an index, `train` its centroids, `add` the vectors, search, then save and
+reload it. [Training](#training-centroids-in-python) describes the clustering
+options, and the update examples below reuse the Quick Start's `index`.
 
 ### C++
 
@@ -46,12 +47,12 @@ See [C++ integration](../cpp_integration.md) for build setup and
 | Parameter | Contract |
 | --- | --- |
 | `dim` | Original input dimension; Python supports 64–65,536 and pads internally. |
-| `max_elements` | Python IVF-RaBitQ build requires exactly this many rows; `add` grows storage automatically. C++ calls this constructor argument `num_points`. |
-| `num_clusters` | Must match the centroid count; every cluster ID must be in `[0, num_clusters)`. |
-| `nbits` / C++ `total_bits` | 1–9 total bits including the sign bit, or 32 for owned raw-vector reranking. |
+| `num_clusters` | Number of centroids `train` learns or `set_centroids` supplies; every cluster ID must be in `[0, num_clusters)`. |
+| `nbits` / C++ `total_bits` | Required and, in Python, keyword-only: 1–9 total bits including the sign bit, or 32 for owned raw-vector reranking. |
+| `seed` | Rotation seed in `[0, 2**32 - 1]` and the initial `cp.seed`. Python's default `None` draws one at random; `index.seed` reports it, and passing it back reproduces the index. |
 | `metric` | Python `"l2"` or `"ip"`; defaults to `"l2"`. |
 | `initializer` | Defaults to `"auto"`; see [centroid routing](#centroid-routing). |
-| `k`, `nprobe` | Positive result count and cluster probe count; C++ and Python both require `k` not to exceed `max_elements`, which `add` increases. |
+| `k`, `nprobe` | Positive result count and cluster probe count; C++ and Python both require `k` not to exceed the stored point count (`max_elements`), which `add` increases. |
 | `high_accuracy` | Python defaults to `None`: HACC for 4–9 bits, standard FastScan otherwise. |
 
 Results contain IDs and squared L2 or `1 - dot` distances, ordered from smallest
@@ -125,8 +126,7 @@ The optional final C++ constructor argument selects `InitializerType::Auto`,
 `initializer="auto"`, `"flat"`, `"flat_rabitq"`, or `"hnsw"`:
 
 ```python
-index = IvfIndex(dim, max_elements, num_clusters, nbits=4,
-                 initializer="flat_rabitq")
+index = IvfIndex(dim, num_clusters, nbits=4, initializer="flat_rabitq")
 print(index.initializer)  # resolved routing type
 ```
 
@@ -199,6 +199,39 @@ For example:
 ivf.construct(data.data(), centroids.data(), cluster_ids.data(), true);
 ```
 
+### Training centroids in Python
+
+`train(x)` learns `num_clusters` centroids from the rows of `x`, which must have
+at least `num_clusters` rows, and stores them in the index; `add` then assigns
+and quantizes vectors. Training uses the settings in `index.cp`, a
+[`RaBitQKMeansParameters`](../clustering.md#rabitqkmeans) whose defaults differ
+from the k-means classes: 25 iterations, at most 256 sampled vectors per
+centroid (`max_points_per_centroid`), spherical k-means for `metric="ip"`, and
+`seed` equal to the index seed. Change fields before training:
+
+```python
+index = IvfIndex(dim, 1024, nbits=4, metric="ip")
+index.cp.niter = 10
+index.train(sample)  # num_threads=None uses index.cp.num_threads (0: all CPUs)
+index.add(vectors)
+```
+
+- Up to 2,000 clusters, training uses RaBitQKMeans; above that it uses
+  [QGKMeans](../clustering.md#qgkmeans), copying the shared fields of `cp` and
+  keeping its default graph settings. The threshold is provisional and has not
+  been measured.
+- `set_centroids(centroids)` uses centroids of shape `(num_clusters, dim)`
+  computed elsewhere instead of training.
+- Training or setting centroids again replaces them while the index stores no
+  vectors; once it stores any (including removed ones), both raise
+  `RuntimeError`. A failed call leaves the index unchanged.
+- Before training, `add`, `search`, `remove`, `reserve` and `save` raise
+  `RuntimeError`. A trained index without vectors cannot be saved, and its
+  `search` rejects every `k` with `ValueError`; add vectors first. Loaded
+  indexes are trained, and their `seed` is `None`.
+- `train` and `add` default to every available CPU; `search` defaults to one
+  thread.
+
 ### Training centroids in C++
 
 Instead of passing centroids to `construct`, create an index without a point
@@ -216,7 +249,11 @@ index.add(data.data(), num_points);
 ```
 
 - The constructor takes `dim`, `num_clusters`, `total_bits`, the metric and the
-  rotation seed, followed by the optional rotator and initializer types. The
+  rotation seed, followed by the optional rotator and initializer types. Pass
+  the seed: without it, four arguments such as `IVF(dim, num_clusters, 4,
+  METRIC_IP)` select the point-count constructor instead, reading them as
+  `(num_points, dim, num_clusters, total_bits)` with the metric converted to a
+  bit count, which may compile and run without an error. The
   index starts with no point storage. `construct` rebuilds an index from
   `max_elements()` rows, so it fails on this index until `add` has stored points.
 - An index created with a point count is built by `construct`: `train` and
@@ -358,10 +395,11 @@ Python `vectors` has shape `(n, index.dim)` and is converted to contiguous float
 one-dimensional integer array or sequence with one valid cluster ID per vector;
 `remove()` likewise accepts a one-dimensional integer array or sequence of point
 IDs and returns the number newly removed. Duplicate IDs count once. Python
-`num_threads` defaults to 1; `fast_quantization` defaults to `False`.
+`num_threads` defaults to 0 (every available CPU) and `fast_quantization` to
+`False`; pass both by keyword.
 
-After appending, rebuilding with `construct` or `build` requires
-`num_stored_points` rows, including removed records. Updates require exclusive
+After appending, rebuilding with C++ `construct` requires `num_stored_points`
+rows, including removed records. Updates require exclusive
 access; see the
 [concurrency contract](usage.md#threading-and-file-paths).
 

docs/docs/index/usage.md

@@ -17,7 +17,8 @@ storage before native processing; inputs remain unchanged.
 Data, centroids and queries must be finite. Coordinates are also limited to the
 magnitude the clustering APIs accept, `sqrt(sqrt(FLT_MAX) / (64 * dim))` (about
 `5e7` for `dim = 128`), so that squared distances and their products stay inside
-float32. `build`, `add` and `search` raise `ValueError` (C++:
+float32. `train`, `set_centroids`, `add` and `search` (and SymphonyQG `build`)
+raise `ValueError` (C++:
 `std::invalid_argument`) for NaN, infinite, or larger coordinates before changing
 the index. The C++ `quantize_*` encoders in `core/encoder.hpp` assume their
 callers have validated the input.
@@ -42,22 +43,22 @@ IVF-RaBitQ, HNSW-RaBitQ, and SymphonyQG expose these read-only Python properties
 | `num_points` | Nonremoved points |
 | `num_stored_points` | Stored records, including removed ones |
 
-`0 <= num_points <= num_stored_points <= max_elements`. Before building, both
-point counts are zero; configured capacity may be nonzero.
-
-At build time, IVF-RaBitQ requires exactly `max_elements` input rows. HNSW-RaBitQ accepts
-1 through `max_elements` rows, so it can reserve capacity for later additions.
+`0 <= num_points <= num_stored_points <= max_elements`. Before vectors are
+added, both point counts are zero; configured capacity may be nonzero. For
+IVF-RaBitQ and HNSW-RaBitQ, `len(index)` equals `num_points`, and `is_trained`
+reports whether centroids are set (by `train`, `set_centroids`, or `load`).
 
 Removal decreases only `num_points`; slots and IDs remain occupied. Append IDs
 start at the previous `num_stored_points`. Use this stored count for ID bounds,
 and `num_points` for a live count. Counts survive save/load.
 
-IVF-RaBitQ grows automatically and has no spare logical capacity after build or append.
-HNSW-RaBitQ's `resize` changes only capacity. SymphonyQG has no incremental updates;
-all three counts are equal after build.
+IVF-RaBitQ grows automatically and has no spare logical capacity after an append.
+HNSW-RaBitQ's Python `add` grows its capacity geometrically, and `resize` changes
+only capacity, for example to preallocate it. SymphonyQG has no incremental
+updates; all three counts are equal after build.
 
-`repr(index)` shows dimension, metric, quantization bits, live point count and build
-state. IVF/HNSW also show stored count, capacity and centroid count; IVF includes
+`repr(index)` shows dimension, metric, quantization bits and live point count,
+with the training state for IVF/HNSW and the build state for SymphonyQG. IVF/HNSW also show stored count, capacity and centroid count; IVF includes
 the resolved initializer, and SymphonyQG includes its degree bound. The summary
 reads only metadata and follows the same concurrency rules as property reads.
 
@@ -118,7 +119,7 @@ construction, updates, and file I/O. The following contract applies to
 | Operation on the same index | Concurrent access |
 | --- | --- |
 | `search`, `search_batch`, `save`, `repr`, and property reads | May run together |
-| `build`, `add`, `remove`, `reserve`, and `resize`, where available | Require exclusive access |
+| `train`, `set_centroids`, `build`, `add`, `remove`, `reserve`, `resize`, and setting `cp`, where available | Require exclusive access |
 
 A conflicting call immediately raises
 `RuntimeError("Index is busy: conflicting operation in progress")`.
@@ -138,7 +139,10 @@ index updates themselves and use separate search output buffers.
 
 `num_threads=0` selects the detected available logical CPU count. Positive values
 set an upper limit, capped at that count; small workloads may use fewer workers.
-Python index methods default to one thread; clustering defaults to `0`.
+IVF-RaBitQ and HNSW-RaBitQ `train` and `add` default to `0`, as does clustering;
+`search`, the deprecated `build`, and SymphonyQG methods default to one thread.
+Multithreaded HNSW-RaBitQ `add` links points in a nondeterministic order, so pass
+`num_threads=1` when a build must be reproducible.
 On Linux, CPU detection accounts for affinity and OpenMP binding. Apply binding
 before starting the program.
 

docs/docs/modular_migration.md

@@ -175,7 +175,7 @@ ID bounds, append offsets, and minimum resize capacity. Use `num_points` for a
 live count, without subtracting a separate deletion counter.
 
 The IVF-RaBitQ constructor keyword `max_elements` still specifies the required initial
-build size. SymphonyQG's C++ `num_vertices()` still returns the configured graph
+build size; 0.8 deprecates it (see [Migrating to 0.8](python_migration.md)). SymphonyQG's C++ `num_vertices()` still returns the configured graph
 size for builders. Existing index files need no conversion: loading recovers
 counts from their removal markers.