From cb045f7d2716e93e659e31c2a40660002733c306 Mon Sep 17 00:00:00 2001 From: gouyt13clear Date: Wed, 21 Oct 2026 20:39:24 +0800 Subject: [PATCH] docs: reduce root entries and relocate clustering helpers --- CONTRIBUTING.md => .github/CONTRIBUTING.md | 34 +++++++++---------- .github/ISSUE_TEMPLATE/config.yml | 2 +- .github/pull_request_template.md | 2 +- .github/scripts/ci_changes.py | 2 +- .github/scripts/test_ci_changes.py | 11 ++++++ README.md | 6 ++-- docs/development.md | 10 +++--- docs/maintaining.md | 4 +-- sample/python/README.md | 17 +++++++++- .../python/clustering_cli}/ivf.py | 0 .../python/clustering_cli}/utils/io.py | 0 scripts/check-python.sh | 4 +-- tests/README.md | 2 +- tests/python/test_examples.py | 2 +- 14 files changed, 61 insertions(+), 35 deletions(-) rename CONTRIBUTING.md => .github/CONTRIBUTING.md (79%) rename {python => sample/python/clustering_cli}/ivf.py (100%) rename {python => sample/python/clustering_cli}/utils/io.py (100%) diff --git a/CONTRIBUTING.md b/.github/CONTRIBUTING.md similarity index 79% rename from CONTRIBUTING.md rename to .github/CONTRIBUTING.md index e9a883ae..3b6b6079 100644 --- a/CONTRIBUTING.md +++ b/.github/CONTRIBUTING.md @@ -42,11 +42,11 @@ Source builds support Linux and Windows x86-64 with AVX2/FMA (AVX-512 optional), and Linux and macOS ARM64 with NEON. They need CMake 3.20 or newer, a C++17 compiler, and OpenMP for ANN/Python. Core/FastScan can disable OpenMP with `RABITQ_USE_OPENMP=OFF`; pure core also has scalar kernels for CPUs without -the FastScan SIMD baseline. See [component builds](docs/docs/cpp_integration.md#core-only-c-integration). +the FastScan SIMD baseline. See [component builds](../docs/docs/cpp_integration.md#core-only-c-integration). Windows x86-64 uses Visual Studio 2026. See the -[platform-specific build instructions](tests/README.md#quick-start) for -[Linux ARM64](tests/README.md#linux-arm64) and -[Apple Silicon](tests/README.md#macos-arm64). +[platform-specific build instructions](../tests/README.md#quick-start) for +[Linux ARM64](../tests/README.md#linux-arm64) and +[Apple Silicon](../tests/README.md#macos-arm64). Documentation-only contributions do not need this hardware or compiler setup. On Ubuntu or Debian: @@ -110,7 +110,7 @@ ctest --test-dir build --output-on-failure The first configuration needs Git and network access to download GoogleTest. Increase `--parallel 2` if your machine has enough memory for more compiler jobs. Native CPU tuning is disabled here to keep generic code portable. On Windows -or Apple Silicon, use the matching commands in [the test guide](tests/README.md#quick-start). +or Apple Silicon, use the matching commands in [the test guide](../tests/README.md#quick-start). ## Before opening a pull request @@ -122,17 +122,17 @@ test during development, then apply every matching row below before submitting. | --- | --- | | Repository Markdown | Check accuracy and local links; `git diff --check` | | Documentation site | Above, plus the strict MkDocs build shown above | -| Python sources | [Python lint and formatting](docs/development.md#python-formatting-and-linting), plus affected tests with `python -m pytest` | -| First-party C++ | [C++ formatting](docs/development.md#c-formatting), affected build/tests, and [static analysis](docs/development.md#static-analysis); focused analysis is sufficient during iteration | +| Python sources | [Python lint and formatting](../docs/development.md#python-formatting-and-linting), plus affected tests with `python -m pytest` | +| First-party C++ | [C++ formatting](../docs/development.md#c-formatting), affected build/tests, and [static analysis](../docs/development.md#static-analysis); focused analysis is sufficient during iteration | | Bindings or Python-visible C++ behavior | Both Python and C++ checks; reinstall before testing and verify package/extension paths as described below | | Examples | Run the example and its language checks | -| Shell scripts | [ShellCheck](docs/development.md#shell-scripts) on affected scripts | +| Shell scripts | [ShellCheck](../docs/development.md#shell-scripts) on affected scripts | For shared code or changes affecting multiple indexes, run the full relevant C++ or Python suite. Changes to SIMD, metrics, packing, persistence, hot paths, or build configuration also need the applicable -[specialized checks](AGENTS.md#verification-by-change-type). Consult the -[implementation recipes](docs/development.md#implementation-recipes) before changing +[specialized checks](../AGENTS.md#verification-by-change-type). Consult the +[implementation recipes](../docs/development.md#implementation-recipes) before changing these contracts; preserve public APIs and stored index formats unless a breaking change has been agreed on. @@ -144,21 +144,21 @@ In your PR, describe: **CI and merge requirements:** CI selects jobs from the changed files and may run broader checks than your focused local tests. Core library and binding changes -require the full [static analysis check](docs/development.md#static-analysis) before merging. +require the full [static analysis check](../docs/development.md#static-analysis) before merging. Tests and examples are intentionally excluded from CI clang-tidy; use focused local analysis for substantial logic changes. If a tool or supported CPU is unavailable, explain that in the PR so the -maintainer can help arrange validation. See [CI routing](docs/maintaining.md#checks-selected-by-changed-files) +maintainer can help arrange validation. See [CI routing](../docs/maintaining.md#checks-selected-by-changed-files) for details. ## If setup fails | Problem | What to check | | --- | --- | -| CMake cannot find OpenMP or a compiler | Check the [platform-specific prerequisites](tests/README.md#prerequisites). On Apple Silicon, install `libomp` and pass its prefix as `OpenMP_ROOT`. Include the compiler version and CMake error when asking for help. | +| CMake cannot find OpenMP or a compiler | Check the [platform-specific prerequisites](../tests/README.md#prerequisites). On Apple Silicon, install `libomp` and pass its prefix as `OpenMP_ROOT`. Include the compiler version and CMake error when asking for help. | | Unsupported CPU or architecture | Use Linux or Windows x86-64 with AVX2/FMA, or Linux or macOS ARM64 with NEON. Disabling native tuning does not add an unsupported architecture. Report your CPU model in the issue or PR. | | Tests do not reflect your Python or C++ edits | Re-run the platform-specific install command above in the active environment, then check the import paths below. Re-run the regression test that exercises your change. | -| Formatting or analysis reports the wrong tool version | Use the pinned versions and executable overrides in the [development reference](docs/development.md). | +| Formatting or analysis reports the wrong tool version | Use the pinned versions and executable overrides in the [development reference](../docs/development.md). | Check which installed package and extension Python imports: @@ -178,7 +178,7 @@ another. Each task can be a separate PR. | Task | Where to start | Done when | | --- | --- | --- | -| Automate the documentation quick-start check | [Quick start](docs/docs/quick_start.md), [documentation workflow](.github/workflows/docs.yml), and [Python tests](tests/python/) | A check executes the documented Python example blocks in order against the current package, verifies the save/load assertions, and cleans up temporary files without keeping a second copy of the example. | +| Automate the documentation quick-start check | [Quick start](../docs/docs/quick_start.md), [documentation workflow](../.github/workflows/docs.yml), and [Python tests](../tests/python/) | A check executes the documented Python example blocks in order against the current package, verifies the save/load assertions, and cleans up temporary files without keeping a second copy of the example. | ## Feedback and further reading @@ -186,9 +186,9 @@ another. Each task can be a separate PR. reviews. Review timing depends on availability; keep questions in the relevant issue or PR so others can learn from the answers. -- [Development reference](docs/development.md): formatters, analysis, +- [Development reference](../docs/development.md): formatters, analysis, performance requirements, and implementation recipes. -- [Maintainer reference](docs/maintaining.md): CI routing, releases, and PR labels. +- [Maintainer reference](../docs/maintaining.md): CI routing, releases, and PR labels. Use [GitHub Issues](https://github.com/VectorDB-NTU/RaBitQ-Library/issues/new/choose) for bug reports, feature requests, and usage or contribution questions. Check diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml index abf8f7d4..f14b9996 100644 --- a/.github/ISSUE_TEMPLATE/config.yml +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -4,5 +4,5 @@ contact_links: url: https://vectordb-ntu.github.io/RaBitQ-Library/ about: Find installation instructions, examples, and index guides. - name: Maintenance and feedback - url: https://github.com/VectorDB-NTU/RaBitQ-Library/blob/main/CONTRIBUTING.md#feedback-and-further-reading + url: https://github.com/VectorDB-NTU/RaBitQ-Library/blob/main/.github/CONTRIBUTING.md#feedback-and-further-reading about: Find the maintainer, contribution guide, and published releases. diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index e9d4d956..1b332b1a 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -6,4 +6,4 @@ Mention any API, saved-index format, or dependency changes if applicable. --> ## Testing +See .github/CONTRIBUTING.md for required checks; include benchmark results for performance claims. --> diff --git a/.github/scripts/ci_changes.py b/.github/scripts/ci_changes.py index 408502d9..aad7260b 100644 --- a/.github/scripts/ci_changes.py +++ b/.github/scripts/ci_changes.py @@ -82,7 +82,7 @@ def classify(paths): selected.add("python_quality") elif path.startswith("sample/python/"): selected.update(("python_quality", "python_build", "wheels")) - elif path.startswith(("python/", ".github/scripts/")): + elif path.startswith(".github/scripts/"): selected.add("python_quality") elif path == ".github/workflows/test.yaml": selected.update(("cpp", "cpp_format", "cpp_tidy")) diff --git a/.github/scripts/test_ci_changes.py b/.github/scripts/test_ci_changes.py index 6afb658d..8dfa2341 100644 --- a/.github/scripts/test_ci_changes.py +++ b/.github/scripts/test_ci_changes.py @@ -13,6 +13,7 @@ def test_docs_only_skip_builds(self): classify( [ "README.md", + ".github/CONTRIBUTING.md", "docs/docs/index/ivf.md", "docs/mkdocs.yml", "docs/requirements.txt", @@ -61,6 +62,16 @@ def test_language_specific_changes(self): "python_build", "wheels", }, + "sample/python/clustering_cli/ivf.py": { + "python_quality", + "python_build", + "wheels", + }, + "sample/python/clustering_cli/utils/io.py": { + "python_quality", + "python_build", + "wheels", + }, ".github/scripts/prepare_release.py": {"python_quality"}, "scripts/check-python.sh": {"shell", "python_quality"}, "scripts/build-macos-openmp.sh": {"shell", "wheels"}, diff --git a/README.md b/README.md index b3fc20d5..edc99475 100644 --- a/README.md +++ b/README.md @@ -220,10 +220,10 @@ listed above. ## Contributing -Start with the [contribution guide](CONTRIBUTING.md#your-first-contribution) -or [starter tasks](CONTRIBUTING.md#starter-tasks). Report bugs and request +Start with the [contribution guide](.github/CONTRIBUTING.md#your-first-contribution) +or [starter tasks](.github/CONTRIBUTING.md#starter-tasks). Report bugs and request features through [GitHub Issues](https://github.com/VectorDB-NTU/RaBitQ-Library/issues/new/choose). -See [maintenance and feedback](CONTRIBUTING.md#feedback-and-further-reading) for maintainer information. +See [maintenance and feedback](.github/CONTRIBUTING.md#feedback-and-further-reading) for maintainer information. ## Acknowledgements diff --git a/docs/development.md b/docs/development.md index cb5ccb63..1a7f78b2 100644 --- a/docs/development.md +++ b/docs/development.md @@ -1,10 +1,10 @@ # Development reference -Start with [Contributing to RaBitQ](../CONTRIBUTING.md) for setup and your first PR. +Start with [Contributing to RaBitQ](../.github/CONTRIBUTING.md) for setup and your first PR. This reference covers tool configuration and changes to library internals. Run commands from the repository root in your activated -[project environment](../CONTRIBUTING.md#python-environment), after installing -the [source-build prerequisites](../CONTRIBUTING.md#setup-for-code-changes). +[project environment](../.github/CONTRIBUTING.md#python-environment), after installing +the [source-build prerequisites](../.github/CONTRIBUTING.md#setup-for-code-changes). Refresh APT package lists with `sudo apt-get update` before installing tools. ## Compiler and tool versions @@ -234,8 +234,8 @@ To apply Python formatting and safe automatic lint fixes before running the check: ```bash -ruff check --fix python python_bindings sample/python tests/python -ruff format python python_bindings sample/python tests/python +ruff check --fix python_bindings sample/python tests/python +ruff format python_bindings sample/python tests/python ``` ## Shell scripts diff --git a/docs/maintaining.md b/docs/maintaining.md index 051cc968..be6fcb62 100644 --- a/docs/maintaining.md +++ b/docs/maintaining.md @@ -1,6 +1,6 @@ # Maintainer reference -For your first contribution, see [Contributing to RaBitQ](../CONTRIBUTING.md). +For your first contribution, see [Contributing to RaBitQ](../.github/CONTRIBUTING.md). This reference describes CI routing and release maintenance. ## Checks selected by changed files @@ -18,7 +18,7 @@ only the affected job groups: | `tests/python/test_cpp_clustering_examples.py` | Python lint and C++ checks; examples run in the Ubuntu job | | `tests/python/conftest.py` | Python lint, C++ checks, and Python builds/wheels | | `sample/python/` | Python lint, source-install smoke tests, comparison tests, and wheels | -| Scripts under `python/` | Python lint | +| Python automation under `.github/scripts/` | Python lint (CI routing scripts trigger broader checks) | | CMake configuration | C++ tests and clang-tidy, plus Python builds/wheels | | `pyproject.toml` | Python lint, builds, and wheels; no C++ checks | | Shell scripts | ShellCheck and any check driven by the changed script | diff --git a/sample/python/README.md b/sample/python/README.md index c481dd3a..ebd910e5 100644 --- a/sample/python/README.md +++ b/sample/python/README.md @@ -2,7 +2,7 @@ Run these commands from the repository root in your project Python environment with `rabitqlib>=0.7.0` and NumPy installed. When working from source, -[install the library from the same checkout](../../CONTRIBUTING.md#python-changes) +[install the library from the same checkout](../../.github/CONTRIBUTING.md#python-changes) so the examples and native extension use matching APIs. Clustering and indexing both use RaBitQ Library; FAISS is needed only for `compare_with_faiss.py`. @@ -55,3 +55,18 @@ python sample/python/hnsw_rabitq_indexing.py --total-bits 5 --degree 16 \ Query with `hnsw_rabitq_querying.py`. SymphonyQG builds directly from vectors; use `symqg_indexing.py` and `symqg_querying.py` without a clustering stage. + +## Separate centroid and assignment files + +The helper formerly at `python/ivf.py` now lives at +`sample/python/clustering_cli/ivf.py`. It writes centroids as `.fvecs` and +assignments as `.ivecs` for workflows using separate files: + +```sh +python sample/python/clustering_cli/ivf.py data/gist/gist_base.fvecs 4096 \ + data/gist/centroids.fvecs data/gist/assignments.ivecs --method qg +``` + +Use `--method rabitq` for flat assignment or the optional positional `ip` metric +with normalized input. For reusable Python index inputs in `.npz`, use +`kmeans_clustering.py` above. diff --git a/python/ivf.py b/sample/python/clustering_cli/ivf.py similarity index 100% rename from python/ivf.py rename to sample/python/clustering_cli/ivf.py diff --git a/python/utils/io.py b/sample/python/clustering_cli/utils/io.py similarity index 100% rename from python/utils/io.py rename to sample/python/clustering_cli/utils/io.py diff --git a/scripts/check-python.sh b/scripts/check-python.sh index d42b2e31..75ae6fd6 100755 --- a/scripts/check-python.sh +++ b/scripts/check-python.sh @@ -18,5 +18,5 @@ if [[ "$($ruff --version)" != "ruff 0.16.1" ]]; then fi cd "$repo_root" -"$ruff" format --check python python_bindings sample/python tests/python -"$ruff" check python python_bindings sample/python tests/python +"$ruff" format --check python_bindings sample/python tests/python +"$ruff" check python_bindings sample/python tests/python diff --git a/tests/README.md b/tests/README.md index 969e4d30..fb9dd877 100644 --- a/tests/README.md +++ b/tests/README.md @@ -139,7 +139,7 @@ clustering dimension/storage matrix. ## Python tests -Use an activated [project environment](../CONTRIBUTING.md#python-environment). +Use an activated [project environment](../.github/CONTRIBUTING.md#python-environment). Rebuild and install the extension before testing C++ changes. These commands work in PowerShell and Bash: diff --git a/tests/python/test_examples.py b/tests/python/test_examples.py index 0286d556..e62c8237 100644 --- a/tests/python/test_examples.py +++ b/tests/python/test_examples.py @@ -300,7 +300,7 @@ def test_legacy_clustering_formats_without_faiss(tmp_path, example_data, argumen centroids_path = tmp_path / "centroids.fvecs" labels_path = tmp_path / "labels.ivecs" run_example( - EXAMPLES.parents[1] / "python" / "ivf.py", + EXAMPLES / "clustering_cli" / "ivf.py", data_path, 16, centroids_path,