Tracking: ongoing refactors — case-config finalization and hardening

[jamesgao-jpg]

Summary

Tracking issue for the ongoing refactor work on case configuration handling in VectorDBBench. The goal is to make case-config construction and validation deterministic, mutation-free, and consistent across all task entry points (CLI and web UI), then harden the config model so classes can be made immutable.

Background / Why

db_case_config was touched in four layers — CLI construction, run() FTS routing, assembler in-place mutation, and task-runner/client reads — and the CLI-only routing was skipped entirely on the web UI path. This made config-handling changes easy to break, and the two entry points could diverge. A follow-up audit of the config model also found self-assigning cached-field methods in several backends that prevent immutable config classes.

Scope (work items)

1. Finalize db_case_config at a single choke point — PR #872 (WIP)

• New vectordb_bench/backend/db_case_config.py with finalize_db_case_config(db, case_type, base_config, *, parameters, dataset).
• Routes both entry points (cli.run() and web UI generate_tasks()) through one resolver; the assembler no longer mutates the config; select_cli_db_case_config remains as a thin backward-compatible wrapper.
• Cross-PR note: the refactor moves the FTS-compatible field whitelist (including force_merge_target_size_mb) into the resolver, subsuming the one-line whitelist fix in PR feat: add CLI control for Milvus force-merge size and toggle #871 (cli/cli.py). When both merge, drop feat: add CLI control for Milvus force-merge size and toggle #871's cli/cli.py hunk.

2. Enforce frozen=True across all case-config classes (approved, deferred — separate PR)

• ~55 root classes across 44 backend config files.
• Rework the self-assigning cached-field methods in:
  • aws_opensearch / oss_opensearch parse_metric()
  • hologres
  • pgvector
• Cross-backend regression sweep after the change.

Definition of done

• Both task entry points produce identical, mutation-free case configs.
• No in-place mutation of db_case_config remains outside the resolver.
• All case-config classes are frozen=True with no self-assigning cached-field methods.
• Full backend regression sweep passes (unit tests + ruff/black clean).

References

• PR [WIP] refactor: finalize db_case_config at a single choke point #872 — refactor: finalize db_case_config at a single choke point
• PR feat: add CLI control for Milvus force-merge size and toggle #871 — feat: add CLI control for Milvus force-merge size and toggle (whitelist overlap)
• Issue [Question/Enhancement] Unconditional force-merge compaction in Milvus client _optimize() may be too costly and non-reproducible at 100M scale (LAION-100M) #825 — force-merge compaction concerns that motivated the force_merge_target_size_mb whitelist entry

[jamesgao-jpg]

Work item: unify the search-level (level) field across FTS and AutoIndex configs

Follow-up from reviewing PR #876 (feat: add search level to Milvus AutoIndex): that PR adds level to the Milvus AutoIndexConfig (dense). The FTS path already carries its own level, in a separate definition:

• ZillizCloudFtsConfig(MilvusFtsConfig) declares level: int = 1 and sends it in search_param() (zilliz_cloud/config.py:53, 71); MilvusFtsConfig itself has no level field.
• MilvusAutoIndexConfig gets level: int | None = None with ge=1, le=10 validation (PR feat: add search level to Milvus AutoIndex #876).

Refactor need: MilvusFtsConfig (via its Zilliz subclass) and the AutoIndex configs currently define level twice, with different defaults (None vs 1) and different semantics (dense AutoIndex search level vs Zilliz Cloud FTS level). We should eventually make the FTS config share the level field with MilvusAutoIndexConfig — a single shared definition (a small level field on a common base or a mixin used by both the AutoIndex chain and the FTS configs) so validation bounds, defaults, and CLI exposure are single-sourced and the frozen=True hardening applies once instead of per-class.

Constraints to record while scoping it:

• FTS configs must not inherit the dense AutoIndexConfig — that would strip the BM25/sparse/analyzer machinery. Sharing has to be via a common field/mixin, not class inheritance from the dense config.
• Semantics differ per provider: open-source Milvus FTS (BM25) sends no level today; only Zilliz Cloud FTS does. The shared field would be inert on MilvusFtsConfig unless a provider opt-in is defined — needs an explicit decision on whether Milvus FTS should ever send a level.
• Default divergence: AutoIndex omits when unset (None → server default); Zilliz FTS always sends 1 today. Unification should preserve behavior or gate any change behind a deliberate compatibility decision (existing zillizautoindex result files record level: 1).

Suggested shape: define the field once (e.g., SearchLevelField mixin or a shared base) with int | None, ge=1, le=10; AutoIndex configs use omit-when-absent; ZillizCloudFtsConfig keeps its 1 default via an override. This slots into scope item 2 (frozen=True hardening) since the FTS configs are among the root classes being hardened.