Add HNSW insertion after construction

docs/docs/index/hnsw.md

@@ -98,3 +98,81 @@ Repeat until `candidate_set` is empty:
    - Insert the neighbor into `candidate_set` with its (possibly refined) estimated distance.  
 
 The search terminates when `candidate_set` is empty.
+
+## Updating an Index
+
+A constructed or loaded index can gain points without the original data. The
+index file format does not change.
+
+```c++
+std::vector<PID> HierarchicalNSW::add(
+    const float* data,
+    size_t n,
+    const PID* cluster_ids = nullptr,
+    bool faster = false,
+    size_t num_threads = 1
+);
+
+void HierarchicalNSW::resize(size_t new_max_elements);
+```
+
+- **data**, **n**: `n` new vectors in the original coordinates. Vector `i` receives
+  the label `num_points() + i`, so labels stay dense, and `add` returns them.
+- **cluster_ids**: The cluster of each new vector, in `[0, num_clusters)`. When it
+  is `nullptr`, each vector goes to its nearest centroid, chosen the same way a
+  query is routed.
+- **faster**, **num_threads**: Same as in `construct`. A point keeps the label
+  `num_points() + i` whatever order the threads finish in, because the slots are
+  reserved as one block before the inserts start.
+- **new_max_elements**: The new capacity. It must be at least `num_points()`.
+  In C++, initialize the index through its parameterized constructor or `load`
+  before calling `resize`.
+
+In Python:
+
+```python
+new_ids = index.add(vectors)                       # route to the nearest centroids
+new_ids = index.add(vectors, cluster_ids=labels)   # or choose the clusters
+index.resize(index.max_elements + 100_000)
+```
+
+It is not safe to call `add` or `resize` while another thread searches the same
+index.
+
+`add` prepares the result IDs, quantized vectors, and link-list allocations before
+changing the graph or point count. A failure during preparation leaves those
+unchanged. A failure during graph linking can leave the batch counted in
+`num_points()` with only some points reachable through the graph. The index
+remains safe to search, save, load, and destroy; inspect `num_points()` before
+retrying, or rebuild to restore full reachability.
+
+### Capacity
+
+`add` never grows the index: it throws once `num_points()` reaches
+`max_elements()`, and `resize` is the only way to raise that. An index built with
+spare capacity absorbs points with no reallocation at all, so sizing
+`max_elements` ahead of time is the cheap path.
+
+`resize` rebuilds the base-layer storage and copies it, so it needs memory for
+both copies while it runs and costs time proportional to the whole index. It
+invalidates every pointer into the index. The per-node upper-layer link lists are
+not copied, only their pointer table.
+
+### How added points are linked
+
+`construct` borrows the caller's vectors and links points using exact distances
+between them. That pointer is not retained, so `add` cannot use it, and instead
+scores candidates with the estimator the search already uses: the vector being
+compared against is rebuilt from its stored RaBitQ code, sign bit and extra bits
+together, and the other side is scored straight from its codes. This is the same
+approach quantized SymphonyQG construction takes.
+
+Graph quality therefore depends on how a point arrived.
+
+### Recall after `add`
+
+The centroids and the rotation are fixed when the index is constructed, so `add`
+never retrains them. If the added vectors come from a different distribution, or
+the index grows many times beyond its original size, recall can drop. Rebuild
+from the original data with new centroids when recall matters more than the cost
+of a rebuild.

include/rabitqlib/index/hnsw/hnsw.hpp

@@ -61,11 +61,50 @@ class HierarchicalNSW {
     [[nodiscard]] size_t ef_construction() const { return ef_construction_; }
     [[nodiscard]] MetricType metric_type() const { return metric_type_; }
     [[nodiscard]] size_t max_elements() const { return max_elements_; }
+    [[nodiscard]] size_t num_points() const { return cur_element_count_; }
+    [[nodiscard]] PID cluster_id_of(PID internal_id) const {
+        return get_clusterid_by_internalid(internal_id);
+    }
 
     void save(const char*) const;
     void load(const char*);
 
     void construct(size_t, const float*, size_t, const float*, PID*, size_t, bool);
+
+    /**
+     * @brief Insert points into a constructed or loaded index.
+     *
+     * Linked with the stored quantized codes, not the exact vectors construct uses,
+     * which are not kept. Centroids and rotation are never retrained. Point `i` gets
+     * the label `num_points() + i` whatever `num_threads` is. Throws rather than
+     * growing past `max_elements()`. Not safe to call while another thread searches.
+     * Preparation failures leave the graph and point count unchanged. A failure
+     * during linking may leave a partially linked batch counted in the index,
+     * which remains safe to search, save, load, and destroy.
+     *
+     * @param data New points, `n * dim` floats in the original coordinates
+     * @param n Number of new points
+     * @param cluster_ids Cluster of each point in `[0, num_clusters)`, or nullptr to
+     * route each one to its nearest centroid
+     * @param faster Same meaning as in `construct`
+     * @param num_threads Threads used to route and insert
+     * @return The label given to each new point
+     */
+    std::vector<PID> add(
+        const float*, size_t, const PID* = nullptr, bool = false, size_t = 1
+    );
+
+    /**
+     * @brief Grow the number of points the index can hold.
+     *
+     * Reallocates and copies the base layer, so it needs room for both copies and
+     * costs time proportional to the whole index. Unchanged if it throws, and
+     * invalidates every pointer into the index.
+     * Requires initialization through the parameterized constructor or `load`.
+     *
+     * @param new_max_elements The new capacity, at least `num_points()`
+     */
+    void resize(size_t);
     std::vector<std::vector<std::pair<float, PID>>> search(
         const float*, size_t, size_t, size_t, size_t
     );
@@ -124,6 +163,7 @@ class HierarchicalNSW {
     };
 
    private:
+    friend struct HnswPruningTestAccess;
     friend maxheap<std::pair<float, PID>> detail::search_knn_neon(
         HierarchicalNSW&, const float*, size_t
     );
@@ -360,6 +400,14 @@ class HierarchicalNSW {
     PID mutually_connect_new_element(PID, maxheap<std::pair<float, PID>>&, int);
 
     void get_neighbors_by_heuristic2(maxheap<std::pair<float, PID>>&, size_t);
+
+    // The same steps once the original data is gone, defined in hnsw_quant.hpp.
+    void reconstruct_rotated(PID, float*) const;
+    float get_quant_dist(PID target, PID query) const;
+    void add_point_quant(PID, int, std::unique_ptr<char, void (*)(void*)>&);
+    maxheap<std::pair<float, PID>> search_base_layer_quant(PID, PID, int);
+    PID mutually_connect_quant(PID, maxheap<std::pair<float, PID>>&, int);
+    void get_neighbors_by_heuristic2_quant(maxheap<std::pair<float, PID>>&, size_t);
 };
 
 inline HierarchicalNSW::HierarchicalNSW(
@@ -1520,3 +1568,6 @@ inline void HierarchicalNSW::searchBaseLayerST_AdaptiveRerankOptDirect(
 }
 
 }  // namespace rabitqlib::hnsw
+
+// Included last: these definitions need the complete class.
+#include "rabitqlib/index/hnsw/hnsw_quant.hpp"  // IWYU pragma: export