[gouyt13]

Summary

SymphonyQG currently reserves up to 300 completion candidates per vertex and retains discarded edges across both pruning stages. Record candidates only during final reverse-edge pruning and complete each row immediately with reusable worker-local buffers, removing the global per-vertex candidate array and its reservations.

Reverse candidate collection still finishes before rows are modified, and graph encoding remains deferred until all rows are complete. Keep the existing candidate cap/order, thresholds and per-row fallback seeds. Add regression coverage for final/nonfinal completion, delayed graph writes, candidate recording and scratch reuse; simplify the two implementation-coupled test fixtures identified in review. Public API signatures and saved-index formats are unchanged.

Candidate provenance intentionally changes: search-pruned edges no longer participate in completion. This is a quality tradeoff relative to main, not a lossless refactor. Fusion equivalence checks compare against an already reverse-only baseline.

Testing

• Portable Clang 23 Release build (RABITQ_ENABLE_NATIVE_OPTIMIZATION=OFF), followed by ctest --output-on-failure: 408 passed, 1 conditional skip; the separate OpenMPBinding test passed.
• Fresh wheel built from this patch and installed into a task-local target using the existing Python environment; imported extension path/hash verified. python -m pytest tests/python/test_symqg.py tests/python/test_qgkmeans.py -q: 153 passed.
• ./scripts/check-format.sh, ./scripts/check-python.sh, git diff --check: passed.
• Focused ./scripts/check-tidy.sh using a compilation-database subset for src/index/qg_builder.cpp: passed. The initial declaration/definition parameter-name diagnostic was fixed.
• Independent delayed-versus-fused completion probe: 24 configurations / 192 pass comparisons, covering raw/4/8-bit, L2/IP, cache on/off, 1/4 pass threads, repeated final passes and reset. Ordered neighbor IDs and distance bits matched with identical setup. A separate full GIST1M frozen-pool replay matched all 32 million ordered edges. These checks do not establish deterministic independent parallel builds or unchanged Recall.

Measurements and quality limits

Two separate local comparisons isolated the two parts of the change. Both used full SIFT1M/GIST1M, raw L2 storage, degree 32, construction ef 200, graph/builder seed 42, PiPNN seed 555, three alternating build pairs, a Xeon Gold 6418H, Clang 23.1.4 Release/native tuning OFF, and 48 logical build threads bound to NUMA node 0. The host was shared.

With reverse-only provenance held fixed, moving to worker-local completion produced:

Dataset	Peak RSS, median GiB before → after	Build time, median paired change
SIFT1M	4.300 → 2.200	−7.82%
GIST1M	11.685 → 11.663	−2.50%

Build time includes initialization/refinement/cleanup and excludes input loading and index saving. RSS is the whole-child high-water mark, including loading/saving; reducing refinement storage does not necessarily lower the overall peak. No phase-resolved peak-memory claim is made.

Separately, changing both-stage recording to reverse-only recording, while retaining global storage on both sides, measured the following at search ef 64 (Recall@10, supplied GT, all 10,000 SIFT / 1,000 GIST queries, one query thread on CPU/memory node 0):

Dataset	Recall change, median paired percentage points	QPS change, median paired
SIFT1M	+0.251	−1.26%
GIST1M	−0.130	−2.19%

Across the fixed search-ef grid 16–1024, median paired raw Recall changes ranged from +0.006 to +0.730 percentage points on SIFT and −0.650 to +0.160 on GIST. These observations preserve the quality regressions and are not an end-to-end paired measurement of the combined PR. Independent parallel graph builds can differ; no universal speedup or Recall-preservation claim is made.

Raw RSS/build observations for the fusion-only comparison

[
  {
    "dataset": "sift",
    "build_seconds_before": [
      10.383282199,
      10.358223093,
      10.333064319
    ],
    "build_seconds_after": [
      9.542925938,
      9.558814312,
      9.524627338
    ],
    "max_rss_kib_before": [
      4507912,
      4510328,
      4508800
    ],
    "max_rss_kib_after": [
      2307744,
      2306700,
      2298432
    ]
  },
  {
    "dataset": "gist",
    "build_seconds_before": [
      33.642027709,
      33.685617752,
      34.055308525
    ],
    "build_seconds_after": [
      32.907826994,
      32.842064494,
      32.868561347
    ],
    "max_rss_kib_before": [
      12252208,
      12243804,
      12257892
    ],
    "max_rss_kib_after": [
      12231808,
      12227168,
      12229648
    ]
  }
]

Each entry follows pair order; relative changes are medians of paired ratios. Generated indexes were hashed and removed after measurement.