From 0b6168d2b1422656ee8c221af318c607e7c9267a Mon Sep 17 00:00:00 2001 From: "hotdata-automation[bot]" <267177015+hotdata-automation[bot]@users.noreply.github.com> Date: Mon, 12 Oct 2026 13:49:29 +0000 Subject: [PATCH] chore(indexes): clarify vector index constraints and algorithms --- CHANGELOG.md | 1 + docs/CreateIndexRequest.md | 4 ++-- docs/IndexesApi.md | 2 +- src/apis/indexes_api.rs | 3 ++- src/models/create_index_request.rs | 10 +++++----- 5 files changed, 11 insertions(+), 9 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 11c1379..2d07b33 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- chore(indexes): clarify vector index constraints and algorithms - chore(indexes): clarify vector precision documentation - chore: clarify sorted table and index creation constraints - feat(indexes): add nlist field to index responses diff --git a/docs/CreateIndexRequest.md b/docs/CreateIndexRequest.md index 1c565ca..62fc420 100644 --- a/docs/CreateIndexRequest.md +++ b/docs/CreateIndexRequest.md @@ -4,13 +4,13 @@ Name | Type | Description | Notes ------------ | ------------- | ------------- | ------------- -**algorithm** | Option<**Algorithm**> | How a vector index organises the vectors it searches. Omit this field for `hnsw`, which is the default. `hnsw` — builds a graph of the vectors and keeps it in memory. Searches are very fast, and the memory a search needs grows with the whole table, so a large enough table cannot be served at all. `ivf` — groups the vectors into clusters and reads only the clusters nearest the search. Searches are considerably slower than `hnsw`, and the memory a search needs follows how much of the index it reads rather than the size of the table, so a table far too large for `hnsw` can still be searched. It keeps a copy of the table's rows beside the vectors so a search is answered without reading the table; that copy is extra storage, and how much depends on `vector_precision`, which decides how compactly the copied vectors are held. Available for columns that already hold vectors, with the `l2` and `cosine` metrics. (enum: hnsw, ivf) | [optional] +**algorithm** | Option<**Algorithm**> | How a vector index organises the vectors it searches. Omit this field for `hnsw`, which is the default. `hnsw` — builds a graph of the vectors and keeps it in memory. Searches are very fast, and the memory a search needs grows with the whole table, so a large enough table cannot be served at all. `ivf` — groups the vectors into clusters and reads only the clusters nearest the search. Searches are considerably slower than `hnsw`, and the memory a search needs follows how much of the index it reads rather than the size of the table, so a table far too large for `hnsw` can still be searched. It keeps a copy of the table's rows beside the vectors so a search is answered without reading the table; that copy is extra storage, and how much depends on `vector_precision`, which decides how compactly the copied vectors are held. Available with the `l2` and `cosine` metrics, for columns that already hold vectors and for text columns with an embedding provider, which generates the vectors. Search an `ivf` index with `ORDER BY vector_distance(column, '') LIMIT ` (or the matching distance function and a query vector), naming the columns to return rather than selecting `*`; `vector_search` works with `hnsw` indexes only. Generating embeddings for a very large table (tens of millions of rows) is not supported yet. (enum: hnsw, ivf) | [optional] **r#async** | Option<**bool**> | When true, create the index as a background job and return a job ID for polling. | [optional][default to false] **async_after_ms** | Option<**i32**> | If set (requires `async` = true), wait up to this many milliseconds for the index build to finish: if it completes in time the index is returned (201), otherwise a 202 with a job ID to poll. Must be between 1000 and the server maximum; a value out of that range, or set without `async` = true, is rejected with 400. | [optional] **columns** | **Vec** | Columns to index. Required for all index types. | **description** | Option<**String**> | User-facing description of the embedding (e.g., \"product descriptions\"). | [optional] **dimensions** | Option<**i32**> | Output vector dimensions. Some models support multiple dimension sizes (e.g., OpenAI text-embedding-3-small supports 512 or 1536). If omitted, the model's default dimensions are used | [optional] -**embedding_provider_id** | Option<**String**> | Embedding provider ID. When set for a vector index, the source column is treated as text and embeddings are generated automatically. The vector index is then built on the generated embedding column (`{column}_embedding` by default). | [optional] +**embedding_provider_id** | Option<**String**> | Embedding provider ID. When set for a vector index, the source column is treated as text and embeddings are generated automatically. The vector index is then built on the generated embedding column (`{column}_embedding` by default), with either algorithm. For an `ivf` index the metric must be `l2` or `cosine`: a provider whose metric is `dot` is refused unless `metric` names one of those. Every later load and text search embeds with the same provider, and is refused if the provider's model or endpoint has changed since the index was created; changing its API key is fine. Dropping the index keeps the generated column as an ordinary column, which loads may leave out (its rows get no vector) and which a new embedding index can generate into again. | [optional] **index_name** | **String** | | **index_type** | Option<**IndexType**> | Index type. `sorted` supports range queries, `bm25` full-text search, and `vector` similarity search. (enum: sorted, bm25, vector) | [optional][default to Sorted] **metric** | Option<**String**> | Distance metric for vector indexes: \"l2\", \"cosine\", or \"dot\". When omitted, defaults to \"l2\" for float array columns or the provider's preferred metric for text columns with auto-embedding. | [optional] diff --git a/docs/IndexesApi.md b/docs/IndexesApi.md index 0d9c0fc..87dd3c1 100644 --- a/docs/IndexesApi.md +++ b/docs/IndexesApi.md @@ -49,7 +49,7 @@ Name | Type | Description | Required | Notes > delete_index(connection_id, schema, table, index_name) Delete an index -Delete a specific index from a table. +Delete a specific index from a table. An index cannot be deleted while a load into its table is running; that request gets 409 and can be retried once the load finishes. Deleting a vector index that generated its own embeddings keeps the generated column as an ordinary column of the table. ### Parameters diff --git a/src/apis/indexes_api.rs b/src/apis/indexes_api.rs index 5d64cc4..99c668b 100644 --- a/src/apis/indexes_api.rs +++ b/src/apis/indexes_api.rs @@ -28,6 +28,7 @@ pub enum CreateIndexError { #[serde(untagged)] pub enum DeleteIndexError { Status404(models::ApiErrorResponse), + Status409(models::ApiErrorResponse), Status500(models::ApiErrorResponse), UnknownValue(serde_json::Value), } @@ -128,7 +129,7 @@ pub async fn create_index( } } -/// Delete a specific index from a table. +/// Delete a specific index from a table. An index cannot be deleted while a load into its table is running; that request gets 409 and can be retried once the load finishes. Deleting a vector index that generated its own embeddings keeps the generated column as an ordinary column of the table. pub async fn delete_index( configuration: &configuration::Configuration, connection_id: &str, diff --git a/src/models/create_index_request.rs b/src/models/create_index_request.rs index 0a80a81..bfcaa5a 100644 --- a/src/models/create_index_request.rs +++ b/src/models/create_index_request.rs @@ -11,10 +11,10 @@ use crate::models; use serde::{Deserialize, Serialize}; -/// CreateIndexRequest : Request body for POST .../indexes Two constraints span the whole table rather than this request alone. A table holds at most one vector index, whichever algorithm it uses. A second vector index is refused until the first is dropped. And a vector index that generates its own embeddings — that is, one created with `embedding_provider_id` — has to be the only index on its table. So a table that already carries any index (sorted, full-text, or vector) will not accept an embedding-backed vector index, and a table that already carries an embedding-backed vector index will not accept any further index of any type. To move between the two arrangements, drop what is there first. Plan for it when designing a table: combining full-text search with generated embeddings on one table is not possible, so use a separate table for the second index, or supply the embeddings yourself. A vector index over a column that already holds vectors — no `embedding_provider_id` — is not affected by that second rule and coexists with sorted and full-text indexes normally. Embedding generation also rewrites the table to add its generated column, and that rewrite cannot preserve a declared partition. An embedding-backed vector index is therefore refused on a partitioned table. No index of any type can be created on a table that declares a sort order (`sorted_by`). Such a request is refused with 400. +/// CreateIndexRequest : Request body for POST .../indexes Two constraints span the whole table rather than this request alone. A table holds at most one vector index, whichever algorithm it uses. A second vector index is refused until the first is dropped. And a vector index that generates its own embeddings — that is, one created with `embedding_provider_id` — has to be the only index on its table. So a table that already carries any index (sorted, full-text, or vector) will not accept an embedding-backed vector index, and a table that already carries an embedding-backed vector index will not accept any further index of any type. This applies to both vector algorithms. To move between the two arrangements, drop what is there first. Plan for it when designing a table: combining full-text search with generated embeddings on one table is not possible, so use a separate table for the second index, or supply the embeddings yourself. A vector index over a column that already holds vectors — no `embedding_provider_id` — is not affected by that second rule and coexists with sorted and full-text indexes normally. Embedding generation also rewrites the table to add its generated column, and that rewrite cannot preserve a declared partition. An embedding-backed vector index is therefore refused on a partitioned table. No index of any type can be created on a table that declares a sort order (`sorted_by`). Such a request is refused with 400. #[derive(Clone, Default, Debug, PartialEq, Serialize, Deserialize)] pub struct CreateIndexRequest { - /// How a vector index organises the vectors it searches. Omit this field for `hnsw`, which is the default. `hnsw` — builds a graph of the vectors and keeps it in memory. Searches are very fast, and the memory a search needs grows with the whole table, so a large enough table cannot be served at all. `ivf` — groups the vectors into clusters and reads only the clusters nearest the search. Searches are considerably slower than `hnsw`, and the memory a search needs follows how much of the index it reads rather than the size of the table, so a table far too large for `hnsw` can still be searched. It keeps a copy of the table's rows beside the vectors so a search is answered without reading the table; that copy is extra storage, and how much depends on `vector_precision`, which decides how compactly the copied vectors are held. Available for columns that already hold vectors, with the `l2` and `cosine` metrics. + /// How a vector index organises the vectors it searches. Omit this field for `hnsw`, which is the default. `hnsw` — builds a graph of the vectors and keeps it in memory. Searches are very fast, and the memory a search needs grows with the whole table, so a large enough table cannot be served at all. `ivf` — groups the vectors into clusters and reads only the clusters nearest the search. Searches are considerably slower than `hnsw`, and the memory a search needs follows how much of the index it reads rather than the size of the table, so a table far too large for `hnsw` can still be searched. It keeps a copy of the table's rows beside the vectors so a search is answered without reading the table; that copy is extra storage, and how much depends on `vector_precision`, which decides how compactly the copied vectors are held. Available with the `l2` and `cosine` metrics, for columns that already hold vectors and for text columns with an embedding provider, which generates the vectors. Search an `ivf` index with `ORDER BY vector_distance(column, '') LIMIT ` (or the matching distance function and a query vector), naming the columns to return rather than selecting `*`; `vector_search` works with `hnsw` indexes only. Generating embeddings for a very large table (tens of millions of rows) is not supported yet. #[serde(rename = "algorithm", skip_serializing_if = "Option::is_none")] pub algorithm: Option, /// When true, create the index as a background job and return a job ID for polling. @@ -47,7 +47,7 @@ pub struct CreateIndexRequest { skip_serializing_if = "Option::is_none" )] pub dimensions: Option>, - /// Embedding provider ID. When set for a vector index, the source column is treated as text and embeddings are generated automatically. The vector index is then built on the generated embedding column (`{column}_embedding` by default). + /// Embedding provider ID. When set for a vector index, the source column is treated as text and embeddings are generated automatically. The vector index is then built on the generated embedding column (`{column}_embedding` by default), with either algorithm. For an `ivf` index the metric must be `l2` or `cosine`: a provider whose metric is `dot` is refused unless `metric` names one of those. Every later load and text search embeds with the same provider, and is refused if the provider's model or endpoint has changed since the index was created; changing its API key is fine. Dropping the index keeps the generated column as an ordinary column, which loads may leave out (its rows get no vector) and which a new embedding index can generate into again. #[serde( rename = "embedding_provider_id", default, @@ -98,7 +98,7 @@ pub struct CreateIndexRequest { } impl CreateIndexRequest { - /// Request body for POST .../indexes Two constraints span the whole table rather than this request alone. A table holds at most one vector index, whichever algorithm it uses. A second vector index is refused until the first is dropped. And a vector index that generates its own embeddings — that is, one created with `embedding_provider_id` — has to be the only index on its table. So a table that already carries any index (sorted, full-text, or vector) will not accept an embedding-backed vector index, and a table that already carries an embedding-backed vector index will not accept any further index of any type. To move between the two arrangements, drop what is there first. Plan for it when designing a table: combining full-text search with generated embeddings on one table is not possible, so use a separate table for the second index, or supply the embeddings yourself. A vector index over a column that already holds vectors — no `embedding_provider_id` — is not affected by that second rule and coexists with sorted and full-text indexes normally. Embedding generation also rewrites the table to add its generated column, and that rewrite cannot preserve a declared partition. An embedding-backed vector index is therefore refused on a partitioned table. No index of any type can be created on a table that declares a sort order (`sorted_by`). Such a request is refused with 400. + /// Request body for POST .../indexes Two constraints span the whole table rather than this request alone. A table holds at most one vector index, whichever algorithm it uses. A second vector index is refused until the first is dropped. And a vector index that generates its own embeddings — that is, one created with `embedding_provider_id` — has to be the only index on its table. So a table that already carries any index (sorted, full-text, or vector) will not accept an embedding-backed vector index, and a table that already carries an embedding-backed vector index will not accept any further index of any type. This applies to both vector algorithms. To move between the two arrangements, drop what is there first. Plan for it when designing a table: combining full-text search with generated embeddings on one table is not possible, so use a separate table for the second index, or supply the embeddings yourself. A vector index over a column that already holds vectors — no `embedding_provider_id` — is not affected by that second rule and coexists with sorted and full-text indexes normally. Embedding generation also rewrites the table to add its generated column, and that rewrite cannot preserve a declared partition. An embedding-backed vector index is therefore refused on a partitioned table. No index of any type can be created on a table that declares a sort order (`sorted_by`). Such a request is refused with 400. pub fn new(columns: Vec, index_name: String) -> CreateIndexRequest { CreateIndexRequest { algorithm: None, @@ -118,7 +118,7 @@ impl CreateIndexRequest { } } } -/// How a vector index organises the vectors it searches. Omit this field for `hnsw`, which is the default. `hnsw` — builds a graph of the vectors and keeps it in memory. Searches are very fast, and the memory a search needs grows with the whole table, so a large enough table cannot be served at all. `ivf` — groups the vectors into clusters and reads only the clusters nearest the search. Searches are considerably slower than `hnsw`, and the memory a search needs follows how much of the index it reads rather than the size of the table, so a table far too large for `hnsw` can still be searched. It keeps a copy of the table's rows beside the vectors so a search is answered without reading the table; that copy is extra storage, and how much depends on `vector_precision`, which decides how compactly the copied vectors are held. Available for columns that already hold vectors, with the `l2` and `cosine` metrics. +/// How a vector index organises the vectors it searches. Omit this field for `hnsw`, which is the default. `hnsw` — builds a graph of the vectors and keeps it in memory. Searches are very fast, and the memory a search needs grows with the whole table, so a large enough table cannot be served at all. `ivf` — groups the vectors into clusters and reads only the clusters nearest the search. Searches are considerably slower than `hnsw`, and the memory a search needs follows how much of the index it reads rather than the size of the table, so a table far too large for `hnsw` can still be searched. It keeps a copy of the table's rows beside the vectors so a search is answered without reading the table; that copy is extra storage, and how much depends on `vector_precision`, which decides how compactly the copied vectors are held. Available with the `l2` and `cosine` metrics, for columns that already hold vectors and for text columns with an embedding provider, which generates the vectors. Search an `ivf` index with `ORDER BY vector_distance(column, '') LIMIT ` (or the matching distance function and a query vector), naming the columns to return rather than selecting `*`; `vector_search` works with `hnsw` indexes only. Generating embeddings for a very large table (tens of millions of rows) is not supported yet. #[derive(Clone, Copy, Debug, Eq, PartialEq, Ord, PartialOrd, Hash, Serialize, Deserialize)] pub enum Algorithm { #[serde(rename = "hnsw")]