feat(turbopuffer): update turbopuffer backend with best practices

[nvb]

Summary

This PR updates the turbopuffer backend with best practices from https://turbopuffer.com/docs/.

Usage

To run the Performance768D100M workload on turbopuffer:

1. follow the quickstart guide to create an API key
2. Install and run VectorDBBench

export TURBOPUFFER_API_KEY="<TURBOPUFFER_API_KEY>"
export TURBOPUFFER_REGION="<TURBOPUFFER_REGION>"

vectordbbench turbopuffer \
  --api-key "$TURBOPUFFER_API_KEY" \
  --region "$TURBOPUFFER_REGION" \
  --namespace vdbbench_laion100m \
  --db-label turbopuffer-search-laion100m \
  --task-label turbopuffer-search-laion100m \
  --case-type Performance768D100M \
  --metric-type L2 \
  --k 100 \
  --insert-batch-size 10000 \
  --load-concurrency 4 \
  --disable-backpressure \
  --num-concurrency 1,5,10,20,30,40,60,80,100,120,160 \
  --probes=200 \
  --concurrency-duration 120 \
  --concurrency-timeout 3600 \
  --drop-old \
  --load \
  --search-concurrent \
  --search-serial \
  --pin-namespace \
  --pin-replicas=2

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: nvb
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[nvb]

/assign @XuanYang-cn

[yhmo]

vectordb_bench/backend/clients/turbopuffer/turbopuffer.py line:341
High ---- This completion check can never detect an in-progress warm: the SDK's hint_cache_warm() returns NamespaceHintCacheWarmResponse(status='ACCEPTED', message=...), and status is always the literal "ACCEPTED", so str(response).lower() always contains "accepted" and the loop returns after the first 5s poll regardless of whether the cache is still warming. That defeats the stated intent of this PR (poll until the cache is warm instead of sleeping): on large namespaces the benchmark proceeds to search while the cache is still cold, producing misleading latency numbers. The state signal lives in response.message ("cache is already warming" while in progress); suggest checking that field instead, e.g. return only when "already warming" is not present.

[yhmo]

vectordb_bench/backend/clients/turbopuffer/turbopuffer.py line:319
Medium ---- Both new wait loops (_wait_for_index here and _warm_cache below) are unbounded while True polls with no deadline. If index.status never reports "up-to-date" (e.g. the namespace keeps receiving writes, or the server reports an unexpected status value), optimize() hangs the entire benchmark run with no abort path. The existing wait_for_namespace_pinning helper already applies a deadline (timeout param); consider doing the same here so a stuck namespace surfaces as a TimeoutError rather than an indefinite hang.

[yhmo]

vectordb_bench/backend/clients/turbopuffer/turbopuffer.py line:546
Medium ---- Question: the query vector is always base64-encoded via _encode_vector, but vector_encoding="base64" is only sent when payload_profile == VECTOR. The public API docs describe vector_encoding as the encoding of vectors in the RESPONSE (default "float"), and do not say the server auto-detects a base64 query vector inside rank_by. With the default IDS_ONLY and SCALAR_LABEL payload profiles this code now sends a base64 query string with no encoding flag. Can you confirm the server interprets the base64 request vector correctly in those paths, or should the flag be passed unconditionally?

[yhmo]

tests/test_turbopuffer_cli.py line:152
Medium ---- This fake cannot catch the premature return in _warm_cache: the real SDK returns NamespaceHintCacheWarmResponse(status='ACCEPTED', message=...), so str(response).lower() always contains "accepted" and the poll exits on the first call regardless of warm state. A faithful double that returns a model with status always "ACCEPTED" and a different message while warming is in progress would expose the bug in turbopuffer.py line 341. There is also no test coverage for the new base64 vector encoding, f16 schema generation, consistency param, or probes wiring in the insert/search paths.

[yhmo]

vectordb_bench/backend/clients/turbopuffer/turbopuffer.py line:15
Low ---- numpy and orjson are imported at module level but neither is declared as a direct dependency in pyproject.toml: orjson is only present transitively via the turbopuffer SDK extra, and numpy via pymilvus/scikit-learn. A fresh vectordb-bench[turbopuffer] install works only because those transitive packages are guaranteed today. Declaring both in the turbopuffer extra would make the module's requirements explicit and robust against dependency reshuffles.

[yhmo]

vectordb_bench/backend/clients/turbopuffer/turbopuffer.py line:559
Low ---- The --probes feature relies on the undocumented __debug_settings {probes_min/probes_max} hook passed through extra_body; it is not part of the public API docs I could find. Because published benchmark results depend on this setting, a server-side rename or removal would silently change search behavior. Consider documenting the dependency on this debug setting or adding a check so a run fails loudly if it stops taking effect.