Refactor disk index build structure

[wuw92]

Why

This is a follow-up to #1254. Removing checkpoint and continuation support eliminated the workflow state that previously justified much of the disk-build indirection, allowing the remaining build path to be simplified and its module ownership clarified.

Module changes

1. Rename core.rs to merged_index.rs; the renamed module retains the merged-build implementation and RAM estimation.
2. Move shared builder fixtures and end-to-end disk-build tests from the renamed module to tests.rs.
3. Move IndexBuildStrategy and determine_build_strategy from the renamed module to build.rs, where the one-shot versus merged decision is made.
4. Move the single in-memory graph construction pipeline from build.rs to inmem_builder.rs.

build.rs
├── one-shot ────────────────> inmem_builder.rs
└── merged ──> merged_index.rs ──> inmem_builder.rs

Compatibility

This change makes builder::quantizer and builder::tokio private, including the previously public BuildQuantizer, BuildQuantizer::train, and create_runtime APIs. These are implementation details of diskann-disk rather than supported downstream extension points. Downstream crates should use the public disk-index build APIs and manage their own Tokio runtime when needed.

Tests

• test_build_from_iter_one_shot_with_metric verifies one-shot builds for L2, inner-product, and cosine distance.
• test_build_multi_sector_per_node exercises the disk layout and search path when one node spans multiple 512-byte sectors.
• test_build_from_iter_one_shot_with_associated_data verifies that vectors, neighbors, and associated data are persisted correctly without intermediate in-memory index files.
• test_build_from_iter_merged_index forces the low-memory merged-build path and verifies search results and the configured maximum degree.
• test_build_quantization_type_failure_cases verifies that unsupported build quantization settings are rejected with the expected error.
• test_disk_index_builder covers one-shot and merged builds with full precision, 1-bit scalar quantization, and product quantization, then validates persisted PQ data and search results.
• test_disk_minmax_index_builder covers one-shot and merged builds for MinMax vector data and validates the resulting search behavior.

[Copilot]

Pull request overview

This PR refactors the diskann-disk disk index build path to clarify module ownership: merged-index building logic is separated from top-level orchestration, and the in-memory single-graph build pipeline is consolidated behind an internal entry point, while keeping behavior unchanged.

Changes:

• Extract merged-index build implementation (partition/shard build/merge/cleanup) into a dedicated merged_index module.
• Move the in-memory build pipeline into inmem_builder and call it from the top-level disk builder.
• Centralize builder test fixtures under build::builder::tests and update test imports accordingly.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
diskann-disk/src/search/provider/disk_provider.rs	Updates test fixture import path to the new builder test module location.
diskann-disk/src/build/builder/tests.rs	Introduces centralized builder test fixtures and helpers under build::builder::tests.
diskann-disk/src/build/builder/quantizer.rs	Updates documentation wording for build quantizer types.
diskann-disk/src/build/builder/mod.rs	Adjusts module layout (adds merged_index, exposes tests as pub(crate) under cfg(test)).
diskann-disk/src/build/builder/merged_index.rs	New module containing merged-index build implementation and RAM estimation tests.
diskann-disk/src/build/builder/inmem_builder.rs	Consolidates the one-shot in-memory build pipeline behind an internal entry point.
diskann-disk/src/build/builder/build.rs	Keeps top-level orchestration and policy; wires in merged_index + inmem_builder entry points.
diskann-disk/src/build/builder/core.rs	Deletes the previous combined “core” module after refactor.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

[harsha-simhadri]

could you please briefly list the tests in this PR and their intent in the PR description? thanks

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 94.28571% with 26 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.45%. Comparing base (bacc7ef) to head (bfce216).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
diskann-disk/src/build/builder/inmem_builder.rs	92.91%	9 Missing ⚠️
diskann-disk/src/build/builder/merged_index.rs	97.00%	9 Missing ⚠️
diskann-disk/src/build/builder/build.rs	68.00%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1271      +/-   ##
==========================================
- Coverage   91.06%   90.45%   -0.62%     
==========================================
  Files         500      498       -2     
  Lines       95938    95138     -800     
==========================================
- Hits        87369    86055    -1314     
- Misses       8569     9083     +514     

Flag	Coverage Δ
miri	90.45% <94.28%> (-0.62%)	⬇️
unittests	90.11% <94.28%> (-0.92%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
diskann-disk/src/build/builder/quantizer.rs	92.85% <100.00%> (ø)
diskann-disk/src/build/builder/tokio.rs	91.66% <100.00%> (ø)
diskann-disk/src/search/provider/disk_provider.rs	93.92% <ø> (ø)
diskann-disk/src/build/builder/build.rs	87.76% <68.00%> (-4.67%)	⬇️
diskann-disk/src/build/builder/inmem_builder.rs	89.14% <92.91%> (+3.43%)	⬆️
diskann-disk/src/build/builder/merged_index.rs	97.00% <97.00%> (ø)

... and 44 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[wuw92]

could you please briefly list the tests in this PR and their intent in the PR description? thanks

I’ve updated the PR description with a brief list of the covered build scenarios and the intent of each test.

[partychen]

Before this change, builder::quantizer and builder::tokio were public modules, and downstream crates could use at least:

• diskann_disk::build::builder::quantizer::BuildQuantizer
• BuildQuantizer::train
• diskann_disk::build::builder::tokio::create_runtime

Making both modules private, together with reducing those items to pub(super), causes existing downstream code using those paths to stop compiling. This is more than an internal module reorganization, although the PR is otherwise presented as a behavior-preserving structural refactor.

If these APIs are intentionally being removed, could the PR description call out the breaking API change? Otherwise, should the old public paths be preserved or deprecated for a compatibility window?

[wuw92]

Before this change, builder::quantizer and builder::tokio were public modules, and downstream crates could use at least:

• diskann_disk::build::builder::quantizer::BuildQuantizer
• BuildQuantizer::train
• diskann_disk::build::builder::tokio::create_runtime

Making both modules private, together with reducing those items to pub(super), causes existing downstream code using those paths to stop compiling. This is more than an internal module reorganization, although the PR is otherwise presented as a behavior-preserving structural refactor.

If these APIs are intentionally being removed, could the PR description call out the breaking API change? Otherwise, should the old public paths be preserved or deprecated for a compatibility window?

Thanks for calling this out. The visibility reduction is intentional. BuildQuantizer, BuildQuantizer::train, and the builder-specific create_runtime helper are implementation details used to orchestrate diskann-disk builds; they are not intended as downstream extension points.
I checked the current internal and M365 sources and found no remaining uses of these APIs. I will update the PR description to explicitly call out the breaking visibility changes.