[gouyt13]

Summary

First of three PRs that move the Python indexes to a Faiss-style train → add → search workflow, so callers no longer run k-means themselves and pass centroids in. This PR adds the C++ side: a "trained but empty" state for IVF-RaBitQ and HNSW-RaBitQ, centroid training on the index, and add into an empty trained index. The Python API (PR 2) and SymphonyQG (PR 3) follow separately.

IVF

• New constructor without a point count: IVF(dim, num_clusters, total_bits, metric, seed, rotator_type, initializer_type). It seeds the rotation explicitly and is not meant for construct.
• train(n, x, cp) trains num_clusters() centroids and sets them as set_centroids does. It uses RaBitQKMeans up to kMaxRaBitQKMeansClusters (2000) clusters and QGKMeans above it. The QGKMeans path copies the fields the two parameter types share and keeps the default graph settings. The 2000 threshold is provisional and has not been measured.
• set_centroids(centroids, num_threads) builds the centroid routing and empty clusters, so add works on the index.
• is_trained().

HNSW

• train(num_clusters, n, x, cp) (RaBitQKMeans), set_centroids(num_clusters, centroids) and is_trained().
• The first add into an empty graph links points with their exact vectors, using the same code as construct (split out as link_exact). With one thread it produces the same index as construct with the same centroids and cluster IDs. If it fails, the graph is left empty and the centroids are kept. Later adds keep linking through the stored codes, as before.

Shared

• include/rabitqlib/index/training.hpp: default_training_parameters(metric, seed) gives 25 iterations, at most 256 sampled points per centroid (the k-means classes keep their own default of 0), and spherical k-means for inner product.
• LloydKMeans::train_centroids(n, x) trains the same centroids as train but skips the final assignment of all n points, which the indexes do not need; add assigns points anyway. It leaves assignments/distances empty and final_obj zero; assign() still works. The cost saving is an argument from complexity (one extra full-data pass), not a measurement.

State rules

• train and set_centroids replace the centroids only while the index stores no points (removed points count); otherwise they throw std::logic_error. A failed call leaves the index unchanged.
• Before training, add throws std::logic_error on both indexes, and IVF search throws as before. HNSW search on an empty index, trained or not, still returns no results (existing C++ contract, relied on by HnswConstructionAllocationTest and ModulesStaticInitialization). The Python layer in PR 2 will raise there.

Compatibility

• Additions only, except that saving an index with no stored points now throws std::logic_error ("... has no points to save; add points first"). For HNSW this includes an untrained, empty index, which earlier releases saved as an empty file. Such files still load. This keeps every newly saved index trained when loaded.
• The IVF and HNSW add errors before training (and IVF search's) now read "must be trained, constructed or loaded". The Python wrappers check state first, so Python messages are unchanged.
• Existing constructors and construct keep their signatures and behavior. The old IVF constructor now shares validation with the new one; when several arguments are invalid at once, which std::invalid_argument is reported first can differ.
• Public index headers only forward-declare RaBitQKMeansParameters; they include no clustering, encoding or rotator headers.
• No saved-index format changes, no Python API changes and no new dependencies.

Testing

All checks ran on this commit (based on 6ba97a8), on Linux x86-64, GCC 14.3, RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF:

• CTest (--no-tests=error): 483/483 passed (468 on main plus 15 new). One test (ThreadCount.BoundParallelForUsesRuntimePlacementAndPropagatesErrors) skipped because OMP_PROC_BIND was not set, as on main.
• Static analysis and formatting: full ./scripts/check-tidy.sh (library and Python bindings, clang-tidy 23) passed. ./scripts/check-format.sh and git diff --check passed. No new compiler warnings compared with main.
• ISA symbols: ./scripts/check-isa-symbols.sh passed (43 translation units); hnsw.hpp is included by the ISA-flagged HNSW search sources.
• Installed consumer: install plus tests/consumer build and test, 3/3 passed.
• Python (after reinstalling into the project environment and confirming the imported extension contains the new code): pytest 888 passed, 2 skipped (macOS/ARM wheel-only tests). Editor/mypy/Pyright checks were not run because the Python API is unchanged.
• Docs: mkdocs build --strict passed. index/ivf.md and index/hnsw.md gain a short C++ "Training centroids" section; the Python docs change in PR 2.
• New tests:
  • IvfTrainTest (7) and HnswTrainTest (5):
    • add/search/save before training, and the index is unchanged and usable afterwards;
    • IVF set_centroids + add produces byte-identical saved indexes to construct with the same rotation, for L2/IP and 1/4/9/32 bits, with explicit and routed cluster IDs;
    • HNSW set_centroids plus a single-threaded first add produces byte-identical saved indexes to construct, for L2/IP, 1/4/9 bits and both quantization modes;
    • retraining while empty, and rejection with points, including loaded indexes;
    • invalid centroids, cluster counts and training data, and that failures leave the centroids unchanged;
    • search and save/load round trip after add, and the empty-save error;
    • training with 2000 clusters reproduces RaBitQKMeans centroids exactly, and 2001 reproduces QGKMeans. Changing <= to < in the threshold makes this test fail;
    • default parameters, including unit-norm centroids under inner product.
  • KMeansTrainCentroidsTest (2): train_centroids matches train centroids and objectives for both classes, with and without sampling and in both final modes; it leaves the assignments empty, and assign still matches.
  • HnswFirstAddAllocationTest (1): every allocation failure in the first add keeps the centroids and an empty graph, and a retry succeeds.
• No performance claims; the 2000 threshold and niter choice will be measured separately.