Relax dimension padding requirements from 64 to 32

AGENTS.md

@@ -147,7 +147,8 @@ Recommended:
   pack/unpack, and estimation coverage. Test both `METRIC_L2` and `METRIC_IP` where supported.
 - IVF/HNSW quantized total bits are one sign bit plus `ex_bits`, with totals 1 through 9.
   IVF also accepts `bits == 32`: one-bit filtering plus owned original float32 vectors
-  in place of extra-bit codes; its raw-mode persistence has a magic/version header.
+  in place of extra-bit codes. New IVF files have a magic/version header and an explicit
+  padded dimension; legacy quantized files and raw v1 files infer 64-dimension padding.
   IVF search defaults to HACC for 4–9 bits and standard FastScan for 1–3 bits or raw storage.
   SymphonyQG supports
   raw storage (`quantization_bits == 0`) and quantized storage at 4 or 8 bits.
@@ -170,13 +171,20 @@ do not use it where callers expect in-place mutation or pointer identity.
 
 ### Rotation and padded dimensions
 
-Indexes quantize and search in the rotated, padded domain. FHT/Kac rotation pads to a multiple of
-64; callers provide vectors in the original `dim`, while internal code and quantization buffers
+Indexes quantize and search in the rotated, padded domain. FHT/Kac, RaBitQKMeans, QGKMeans,
+SymphonyQG, IVF (all bit widths), and HNSW pad to multiples of 32. Packed extra-bit
+and sign codes retain the historical full 64-coordinate blocks and compact final
+32-coordinate tails. Binary query planes round their scratch storage up to
+whole uint64_t words; 32-coordinate padding keeps HNSW record strides PID-aligned. Callers
+provide vectors in the original `dim`, while internal code and quantization buffers
 generally use `padded_dim`. A buffer allocated for `dim` must never receive `padded_dim` values.
 Centroids, data, and queries compared by the same estimator must be in the same domain.
 
 Rotator state is part of persisted index state. Loading an index must restore the exact rotation
 used at construction; generating a new random rotator produces silently incorrect distances.
+Honor stored padded dimensions instead of recomputing them from current defaults. Old IVF formats
+omit this field and must retain their historical 64-dimension padding when loaded.
+Unreleased pad16 files whose stored dimension is not divisible by 32 are rejected.
 
 ### Distance conventions
 

docs/docs/index/hnsw.md

@@ -242,3 +242,8 @@ A file that has removals cannot be read by earlier releases. Releases 0.3.7
 through 0.5.1 check every cluster ID on `load` and reject the file as invalid
 instead of returning removed points. Releases before 0.3.7 do not check cluster
 IDs, so do not open such a file with them.
+
+New HNSW indexes pad dimensions to multiples of 32 for all supported bit widths.
+Loading preserves the saved padded dimension and rotation, so older indexes
+using multiples of 64 remain readable. Older library versions cannot load new
+files whose padded dimension is not a multiple of 64.

docs/docs/index/ivf.md

@@ -87,14 +87,24 @@ Construction reuses a rotation buffer of at most 32 vectors per worker. This
 temporary storage uses at most `workers * 32 * padded_dim * sizeof(float)` bytes,
 independently of cluster sizes.
 
+New indexes pad the rotated dimension to a multiple of 32 for every supported
+bit width, including raw mode (32). This applies to both rotator types.
+Extra-bit layouts retain full 64-coordinate blocks and store any final
+32-coordinate tail compactly.
+
 After construction, you can directly save the index file to disk:
 ```c++
 ivf.save(output_index_file);
 ```
-Raw-mode files use a magic/version header and include both the original vectors
-and the rotation state. Loading restores the mode automatically, so querying
-does not require an external dataset. Existing quantized files retain their
-format and remain compatible; versions before 0.3.3 cannot load raw-mode files.
+New files use the `RABQIDX1` magic, version 1, storage flags, and an explicit
+padded dimension before the index metadata. Loading restores the storage mode,
+rotation state, and padded dimension; raw files also contain the original vectors,
+so querying does not require an external dataset.
+
+Legacy unversioned quantized files and raw v1 files remain readable and retain
+their original 64-dimension padding. Saving a loaded legacy index writes the new
+header without changing its padded dimension or encoded data. Older library
+versions cannot read the new format.
 
 ### Data Layout
 The main data layout for our IVF is organized as follows:

docs/docs/index/qg.md

@@ -7,6 +7,10 @@ FastScan estimates neighbor distances; visited vertices are scored using stored
 raw vectors or quantized codes.
 
 Set `quantization_bits=0` for raw vectors (default), or `4`/`8` for quantized SymphonyQG.
+All three modes pad the rotated domain to a multiple of 32. Loading an older
+index preserves its saved padded dimension, including historical multiples of 64.
+Older library versions that enforce 64-dimension padding cannot load new files
+whose padded dimension is not a multiple of 64.
 Quantized vectors share a rotated global centroid. Refinement temporarily
 reconstructs source vectors and estimates distances to stored target codes;
 reverse edges are rescored because these estimates are directional. Raw refinement

docs/docs/rabitq/rotator.md

@@ -6,10 +6,20 @@ RaBitQLib provides two types of random rotation. All implementations sample and
 By default, the library uses the `FFHT + Kac’s Walk` method. 
 
 The default rotator accepts input dimensions from 64 through 65,536 inclusive,
-including 16,384. It pads to a multiple of 64 and uses AVX or NEON kernels
+including 16,384. It pads to a multiple of 32 and uses AVX or NEON kernels
 for power-of-two blocks through `2^16`. Allocate the
 output using `rotator->size()`, which can exceed the input dimension.
 
+RaBitQKMeans, QGKMeans, SymphonyQG, IVF, and HNSW also use multiples of 32
+for all their supported bit widths.
+Index loading preserves the padded domain used when the index was built.
+Reducing the padded dimension changes the sampled rotation and quantization even
+with the same seed. Newly built indexes can therefore have different recall and
+latency; smaller padding does not guarantee faster search or unchanged accuracy.
+When restoring a standalone rotator saved with earlier 64-dimension padding,
+pass that original padded dimension to `choose_rotator` before loading its state.
+The saved sign bytes do not contain dimension metadata.
+
 The implementation can be found in `rotator.hpp`.
 
 ```css
@@ -30,12 +40,12 @@ The implementation can be found in `rotator.hpp`.
 #include "rabitqlib/utils/rotator.hpp"
 
 int main() {
-    const size_t dim = 769;  // deliberately not a multiple of 64
+    const size_t dim = 769;  // deliberately not a multiple of 32
     std::unique_ptr<rabitqlib::Rotator<float>> rotator(
         rabitqlib::choose_rotator<float>(dim, rabitqlib::RotatorType::FhtKacRotator)
     );
     std::vector<float> x(dim, 1.0F);
-    std::vector<float> x_prime(rotator->size());  // 832 elements
+    std::vector<float> x_prime(rotator->size());  // 800 elements
     rotator->rotate(x.data(), x_prime.data());
 }
 ```
@@ -46,7 +56,7 @@ to quantization and estimation routines that operate on rotated vectors.
 
 For a random orthogonal transformation, select `RotatorType::MatrixRotator`.
 It keeps the input dimension by default; pass a padded dimension explicitly if
-subsequent packed-code operations require a multiple of 64. Its storage and
+subsequent packed-code operations require a multiple of 32. Its storage and
 computation cost are quadratic in the dimension.
 
 ## FFHT + Kac’s Walk

include/rabitqlib/index/hnsw/hnsw.hpp

@@ -464,19 +464,19 @@ inline HierarchicalNSW::HierarchicalNSW(
     max_elements_ = max_elements;
     dim_ = dim;
     rotator_.reset(choose_rotator<float>(
-        dim, RotatorType::FhtKacRotator, round_up_to_multiple(dim_, 64)
+        dim, RotatorType::FhtKacRotator, round_up_to_multiple(dim_, 32)
     ));
     padded_dim_ = rotator_->size();
     /* check size */
-    assert(padded_dim_ % 64 == 0);
+    assert(padded_dim_ % 32 == 0);
     assert(padded_dim_ >= dim_);
     ex_bits_ = total_bits - 1;
 
     if (total_bits < 1 || total_bits > 9) {
         throw std::invalid_argument("HNSW quantization bits must be in [1, 9]");
     };
 
-    assert(padded_dim_ % 64 == 0);
+    assert(padded_dim_ % 32 == 0);
 
     ip_func_ = select_excode_ipfunc(ex_bits_);
 
@@ -500,9 +500,7 @@ inline HierarchicalNSW::HierarchicalNSW(
                      sizeof(PID);  // (# of edges + edges) + (cluster_id) + (external label)
     offsetExData_ = offsetBinData_ + size_bin_data_;  // (# of edges + edges) + (cluster_id)
                                                       // + (external label) + (BinData)
-    size_data_per_element_ =
-        offsetExData_ + size_ex_data_;  // (# of edges + edges) + (cluster_id) + (external
-                                        // label) + (BinData) + (ExData)
+    size_data_per_element_ = offsetExData_ + size_ex_data_;
     data_level0_memory_ =
         memory::huge_page_allocate<char>(max_elements_ * size_data_per_element_);
     if (data_level0_memory_ == nullptr) {
@@ -652,8 +650,8 @@ inline void HierarchicalNSW::load(const char* filename) {
         loaded.num_cluster_ > buffer::kSearchBufferMaxPointCount ||
         (element_count != 0 && loaded.num_cluster_ == 0) || loaded.dim_ < 64 ||
         loaded.dim_ > rotator_impl::FhtKacRotator::kMaxDim ||
-        loaded.padded_dim_ != round_up_to_multiple(loaded.dim_, 64) ||
-        loaded.ex_bits_ > 8 ||
+        loaded.padded_dim_ < loaded.dim_ || loaded.padded_dim_ % 32 != 0 ||
+        loaded.padded_dim_ > rotator_impl::FhtKacRotator::kMaxDim || loaded.ex_bits_ > 8 ||
         (loaded.metric_type_ != METRIC_L2 && loaded.metric_type_ != METRIC_IP) ||
         loaded.M_ < 2 || loaded.M_ > 10000 || loaded.maxM_ != loaded.M_ ||
         loaded.maxM0_ != 2 * loaded.M_ || loaded.ef_construction_ < loaded.M_ ||

include/rabitqlib/index/hnsw/hnsw_quant.hpp

@@ -60,11 +60,11 @@ inline void HierarchicalNSW::reconstruct_rotated(PID id, float* out) const {
     const uint8_t* sign_code = bin.bin_code();
     for (size_t word = 0; word * 64 < padded_dim_; ++word) {
         uint64_t bits = 0;
-        std::memcpy(&bits, sign_code + (word * sizeof(uint64_t)), sizeof(bits));
         const size_t base = word * 64;
         const size_t span = std::min<size_t>(64, padded_dim_ - base);
+        std::memcpy(&bits, sign_code + (word * sizeof(uint64_t)), span / 8);
         for (size_t j = 0; j < span; ++j) {
-            combined[base + j] = static_cast<uint16_t>((bits >> (63 - j)) & 1ULL);
+            combined[base + j] = static_cast<uint16_t>((bits >> (span - 1 - j)) & 1ULL);
         }
     }
 

include/rabitqlib/index/ivf/ivf.hpp

@@ -67,6 +67,8 @@ class IVF {
 
     static constexpr uint64_t kRawFormatMagic = 0x3157415251464252ULL;
     static constexpr uint32_t kRawFormatVersion = 1;
+    static constexpr uint64_t kFormatMagic = 0x3158444951424152ULL;  // "RABQIDX1"
+    static constexpr uint32_t kFormatVersion = 1;
 
     void
     quantize_cluster(Cluster&, const std::vector<PID>&, const float*, const float*, float*, float*, const quant::RabitqConfig&);
@@ -297,7 +299,7 @@ inline IVF::IVF(
     if (dim == 0 || dim > (std::numeric_limits<size_t>::max() / 32) - 64) {
         throw std::invalid_argument("IVF dimension is invalid or too large");
     }
-    padded_dim_ = round_up_to_multiple(dim_, 64);
+    padded_dim_ = round_up_to_multiple(dim_, 32);
     const size_t max_size = std::numeric_limits<size_t>::max();
     if (exceeds_storage(n) || cluster_num > max_size / sizeof(float) / padded_dim_ ||
         (type == RotatorType::MatrixRotator && dim > max_size / sizeof(float) / padded_dim_
@@ -306,7 +308,7 @@ inline IVF::IVF(
     }
     rotator_.reset(choose_rotator<float>(dim, type, padded_dim_));
     /* check size */
-    assert(padded_dim_ % 64 == 0);
+    assert(padded_dim_ % 32 == 0);
     assert(padded_dim_ >= dim_);
 }
 
@@ -780,14 +782,14 @@ inline void IVF::save(const char* filename) const {
 
     std::ofstream output(rabitqlib::io_impl::filesystem_path(filename), std::ios::binary);
     output.exceptions(std::ios::failbit | std::ios::badbit);
-    if (raw_reranking_) {
-        output.write(
-            reinterpret_cast<const char*>(&kRawFormatMagic), sizeof(kRawFormatMagic)
-        );
-        output.write(
-            reinterpret_cast<const char*>(&kRawFormatVersion), sizeof(kRawFormatVersion)
-        );
-    }
+    const uint32_t flags = raw_reranking_ ? 1 : 0;
+    const uint64_t stored_padded_dim = padded_dim_;
+    output.write(reinterpret_cast<const char*>(&kFormatMagic), sizeof(kFormatMagic));
+    output.write(reinterpret_cast<const char*>(&kFormatVersion), sizeof(kFormatVersion));
+    output.write(reinterpret_cast<const char*>(&flags), sizeof(flags));
+    output.write(
+        reinterpret_cast<const char*>(&stored_padded_dim), sizeof(stored_padded_dim)
+    );
 
     /* Save meta data */
     output.write(reinterpret_cast<const char*>(&num_), sizeof(size_t));
@@ -842,8 +844,23 @@ inline void IVF::load(const char* filename) {
     input.seekg(0);
     uint64_t magic = 0;
     input.read(reinterpret_cast<char*>(&magic), sizeof(magic));
-    loaded.raw_reranking_ = magic == kRawFormatMagic;
-    if (loaded.raw_reranking_) {
+    const bool explicit_padding = magic == kFormatMagic;
+    if (explicit_padding) {
+        uint32_t version = 0, flags = 0;
+        uint64_t stored_padded_dim = 0;
+        input.read(reinterpret_cast<char*>(&version), sizeof(version));
+        input.read(reinterpret_cast<char*>(&flags), sizeof(flags));
+        input.read(reinterpret_cast<char*>(&stored_padded_dim), sizeof(stored_padded_dim));
+        if (version != kFormatVersion) {
+            throw std::runtime_error("Unsupported IVF index version");
+        }
+        if (flags > 1 || stored_padded_dim > std::numeric_limits<size_t>::max()) {
+            throw std::runtime_error("Invalid IVF index metadata");
+        }
+        loaded.raw_reranking_ = flags == 1;
+        loaded.padded_dim_ = static_cast<size_t>(stored_padded_dim);
+    } else if (magic == kRawFormatMagic) {
+        loaded.raw_reranking_ = true;
         uint32_t version = 0;
         input.read(reinterpret_cast<char*>(&version), sizeof(version));
         if (version != kRawFormatVersion) {
@@ -868,7 +885,14 @@ inline void IVF::load(const char* filename) {
         loaded.dim_ > (std::numeric_limits<size_t>::max() / 32) - 64) {
         throw std::runtime_error("Invalid IVF index metadata");
     }
-    loaded.padded_dim_ = round_up_to_multiple(loaded.dim_, 64);
+    if (!explicit_padding) {
+        // Both historical formats omit padded_dim and always used 64-coordinate blocks.
+        loaded.padded_dim_ = round_up_to_multiple(loaded.dim_, 64);
+    }
+    if (loaded.padded_dim_ < loaded.dim_ || loaded.padded_dim_ % 32 != 0 ||
+        loaded.padded_dim_ > (std::numeric_limits<size_t>::max() / 32) - 64) {
+        throw std::runtime_error("Invalid padded dimension in IVF index file");
+    }
     // Bound every payload by the actual file before allocating or multiplying sizes.
     size_t remaining = file_bytes - static_cast<size_t>(input.tellg());
     const auto consume = [&remaining](size_t count, size_t bytes) {

include/rabitqlib/index/query.hpp

@@ -161,7 +161,8 @@ class SplitSingleQuery {
         size_t metric_type = METRIC_L2,
         size_t base_bits = 1
     )
-        : rotated_query_(rotated_query), QueryBin_(padded_dim * kNumBits / 64, 0) {
+        : rotated_query_(rotated_query)
+        , QueryBin_(div_round_up(padded_dim, 64) * kNumBits, 0) {
         if (base_bits < 1 || base_bits > 8 || ex_bits > 8 || base_bits + ex_bits > 9) {
             throw std::invalid_argument(
                 "SplitSingleQuery requires base_bits in [1, 8], ex_bits in [0, 8], "

include/rabitqlib/quantization/pack_excode.hpp

@@ -58,8 +58,7 @@ inline void packing_8bit_excode(const uint8_t* o_raw, uint8_t* o_compact, size_t
  *
  * @param o_raw unpacked code, code for each dim is represented by uint8
  * @param o_compact compact format of code
- * @param dim   dimension of code, NOTICE: different num of bits requried different
- *               dimension padding, dim should obey the corresponding requirement
+ * @param dim dimension of code, a multiple of 32 for every supported bit width
  * @param ex_bits number of bits used for code
  */
 inline void packing_rabitqplus_code(

include/rabitqlib/quantization/rabitq_impl.hpp

@@ -229,7 +229,7 @@ inline void one_bit_compact_code(
     );
 }
 
-// Requires a positive padded_dim divisible by sizeof(TC) * 8.
+// Requires a positive padded_dim divisible by 8.
 template <
     typename T,
     typename TC,
@@ -248,19 +248,17 @@ inline void one_bit_compact_codes(
     FError f_error,
     MetricType metric_type = METRIC_L2
 ) {
-    constexpr size_t kTypeBits = sizeof(TC) * 8;
-
 #pragma omp parallel for if (Parallel)
     for (std::ptrdiff_t index = 0; index < static_cast<std::ptrdiff_t>(num); ++index) {
         const size_t i = static_cast<size_t>(index);
         T add;
         T rescale;
         T error;
-        one_bit_compact_code(
+        one_bit_compact_code_to_bytes<T, TC>(
             data + (padded_dim * i),
             centroid,
             padded_dim,
-            compact_code + (padded_dim / kTypeBits * i),
+            reinterpret_cast<uint8_t*>(compact_code) + (padded_dim / 8 * i),
             add,
             rescale,
             error,
@@ -272,8 +270,9 @@ inline void one_bit_compact_codes(
     }
 }
 
-// Encoding requires a positive padded_dim divisible by 8; index pipelines use 64.
-// packed_code needs ceil(num / 32) * 32 * (padded_dim / 8) bytes, including tail padding.
+// Encoding requires a positive padded_dim divisible by 8; index pipelines use multiples
+// of 32. packed_code needs ceil(num / 32) * 32 * (padded_dim / 8) bytes, including tail
+// padding.
 template <
     typename T,
     bool Parallel = false,

include/rabitqlib/utils/bitops.hpp

@@ -1,13 +1,32 @@
 #pragma once
 
+#include <cstddef>
 #include <cstdint>
+#include <cstring>
 
 #if defined(_MSC_VER)
 #include <intrin.h>
 #endif
 
 namespace rabitqlib::bitops {
 
+// Load 1..8 bytes of a compact word, zero-extending a final partial word.
+inline uint64_t load_word(const void* data, size_t bytes) {
+    if (bytes == 8) {
+        uint64_t word;
+        std::memcpy(&word, data, sizeof(word));
+        return word;
+    }
+    if (bytes == 4) {
+        uint32_t word;
+        std::memcpy(&word, data, sizeof(word));
+        return word;
+    }
+    uint64_t word = 0;
+    std::memcpy(&word, data, bytes);
+    return word;
+}
+
 inline unsigned popcount32(uint32_t value) {
 #if defined(_MSC_VER)
     value -= (value >> 1) & 0x55555555U;
@@ -18,8 +37,11 @@ inline unsigned popcount32(uint32_t value) {
 #endif
 }
 
-inline unsigned popcount64(uint64_t value) {
-#if defined(_MSC_VER)
+static inline unsigned popcount64(uint64_t value) {
+#if defined(_MSC_VER) || \
+    ((defined(__x86_64__) || defined(__i386__)) && !defined(__POPCNT__))
+    // Inline the software fallback on generic x86 instead of calling libgcc
+    // once per word. ISA-specific builds still use their native instruction.
     value -= (value >> 1) & 0x5555555555555555ULL;
     value = (value & 0x3333333333333333ULL) + ((value >> 2) & 0x3333333333333333ULL);
     return static_cast<unsigned>(