embeddings: default check_embedding_ctx_length=False for OpenAI-compa… - #129
Merged
Conversation
…tible endpoints langchain OpenAIEmbeddings defaults to check_embedding_ctx_length=True, which tokenizes client-side with tiktoken and sends OpenAI token IDs. Non-OpenAI models behind an OpenAI-compatible endpoint (e.g. Qwen) can't interpret those IDs and return nonsensical embeddings. Default the flag to False so raw text is sent and the server tokenizes correctly. No-op for genuine OpenAI models; callers can opt back in with check_embedding_ctx_length=True. Scoped to the OpenAI path, so the Bedrock/Amazon branch is unaffected. Co-authored-by: Cursor <cursoragent@cursor.com>
kkampli-singlestore
requested review from
kesmit13,
mgiannakopoulos,
pmishchenko-ua and
volodymyr-memsql
as code owners
July 21, 2026 14:18
Only default check_embedding_ctx_length=False for self-hosted / non-OpenAI models
(e.g. Qwen on the 'Nova' platform). Genuine OpenAI/Azure models ('Azure'/'OpenAI')
keep langchain's default (True), preserving correct tiktoken tokenization and
client-side long-input chunking for them.
Co-authored-by: Cursor <cursoragent@cursor.com>
volodymyr-memsql
approved these changes
Jul 22, 2026
…models - Azure (OpenAI) models: keep check_embedding_ctx_length=True so langchain uses tiktoken (correct for these models) with the passed model name, incl. its long- input chunking. - Non-OpenAI models (e.g. Qwen on 'Nova'): send raw text (check_embedding_ctx_length =False) so the server's own tokenizer is used, and split long inputs into character-bounded chunks client-side, embedding each and length-weighted-averaging them into one vector per input -- irrespective of the flag -- so long texts never hit the server's hard context limit (which otherwise errors or silently truncates). Output stays 1:1 with input rows; short inputs pass through unchanged. Co-authored-by: Cursor <cursoragent@cursor.com>
…False otherwise Non-OpenAI models (e.g. Qwen on 'Nova') get wrong embeddings from langchain's default tiktoken tokenization (it sends OpenAI token IDs). Send raw text for them so the server tokenizes with the model's own tokenizer; keep tiktoken (True) for Azure/OpenAI models, where it's correct. Long-input chunking deferred. Co-authored-by: Cursor <cursoragent@cursor.com>
Re-add _ChunkedOpenAIEmbeddings: for non-Azure models (e.g. Qwen on 'Nova'), send raw text (check_embedding_ctx_length=False) and split long inputs into character-bounded chunks, embedding each and length-weighted-averaging into one vector per input, so long texts never hit the server's context limit. Azure keeps tiktoken (True). Co-authored-by: Cursor <cursoragent@cursor.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6def72b. Configure here.
kkampli-singlestore
had a problem deploying
to
publish
July 22, 2026 15:29 — with 
GitHub Actions
Failure
GitHub Actions
Failure
There was a problem hiding this comment.
Pull request overview
Adjusts the embeddings factory to avoid LangChain/OpenAI tokenization for non-Azure OpenAI-compatible endpoints (e.g., Qwen on Nova), and adds client-side character chunking + weighted reduction so long inputs don’t overflow server context limits.
Changes:
- Introduces
_ChunkedOpenAIEmbeddingsto chunk long texts and reduce chunk embeddings back into a single vector per input. - Updates
SingleStoreEmbeddingsFactoryto keep Azure behavior (LangChain tokenization/context handling) while defaulting non-Azure paths to raw-text mode (check_embedding_ctx_length=False) and using_ChunkedOpenAIEmbeddings. - Adds unit tests covering factory branching and chunk/reduction behavior (sync + async).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| singlestoredb/ai/embeddings.py | Adds a chunking OpenAIEmbeddings subclass and adjusts factory defaults based on hosting platform. |
| singlestoredb/tests/test_embeddings.py | Adds unit tests for the new factory branching and chunked embedding reduction logic. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+82
to
+106
| sys.modules.pop('singlestoredb.ai.embeddings', None) | ||
| sys.modules.pop('_test_embeddings_module', None) | ||
|
|
||
| httpx = types.ModuleType('httpx') | ||
| httpx.Client = MockClient | ||
| httpx.Timeout = MockTimeout | ||
| sys.modules['httpx'] = httpx | ||
|
|
||
| langchain_openai = types.ModuleType('langchain_openai') | ||
| langchain_openai.OpenAIEmbeddings = MockOpenAIEmbeddings | ||
| sys.modules['langchain_openai'] = langchain_openai | ||
|
|
||
| langchain_aws = types.ModuleType('langchain_aws') | ||
| langchain_aws.BedrockEmbeddings = MockBedrockEmbeddings | ||
| sys.modules['langchain_aws'] = langchain_aws | ||
|
|
||
| botocore = types.ModuleType('botocore') | ||
| botocore.UNSIGNED = 'unsigned' | ||
| sys.modules['botocore'] = botocore | ||
|
|
||
| botocore_config = types.ModuleType('botocore.config') | ||
| botocore_config.Config = MockConfig | ||
| sys.modules['botocore.config'] = botocore_config | ||
|
|
||
| sys.modules['boto3'] = MockBoto3('boto3') |
Comment on lines
+82
to
+101
| def _reduce( | ||
| self, | ||
| num_texts: int, | ||
| owner: List[int], | ||
| flat: List[str], | ||
| embeddings: List[List[float]], | ||
| ) -> List[List[float]]: | ||
| out: List[List[float]] = [] | ||
| for i in range(num_texts): | ||
| idxs = [j for j, o in enumerate(owner) if o == i] | ||
| if len(idxs) == 1: | ||
| out.append(embeddings[idxs[0]]) | ||
| else: | ||
| out.append( | ||
| self._average( | ||
| [embeddings[j] for j in idxs], | ||
| [max(1, len(flat[j])) for j in idxs], | ||
| ), | ||
| ) | ||
| return out |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Don’t use OpenAI’s tokenizer for Qwen — LangChain’s default (check_embedding_ctx_length=True) turns text into OpenAI token IDs. Qwen can’t read those, so embeddings are garbage. Send raw text instead (False) so the server tokenizes correctly.
Keep OpenAI/Azure on the default — Real OpenAI/Azure models still use True (tiktoken + LangChain’s built-in long-text handling). Only non-OpenAI/Nova models get the raw-text default.
Chunk long Qwen inputs ourselves — With raw text, LangChain won’t split long docs, so the server 400s or truncates. For those models, split into character chunks, embed each, then average into one vector so long text still works.
Test cases:
1.
Before:
label cosine_vs_anchor
anchor 1.0
unrelated2 0.3536
same_intent 0.3461
paraphrase 0.304
unrelated 0.2523
After
Chunking Long Text
Note
Medium Risk
Changes default embedding behavior for all non-Azure OpenAI-compatible inference paths, which can alter vector quality and similarity for existing Nova/Qwen workloads but fixes incorrect embeddings from wrong tokenization.
Overview
Fixes broken embeddings for OpenAI-compatible non-Azure models (e.g. Qwen on Nova) by stopping LangChain’s tiktoken path:
SingleStoreEmbeddingsFactorynow defaultscheck_embedding_ctx_length=Falseand returns_ChunkedOpenAIEmbeddingsso the server receives raw text and tokenizes with the model’s own tokenizer.Azure-hosted OpenAI models are unchanged — they still use stock
OpenAIEmbeddingswithcheck_embedding_ctx_length=Trueand LangChain’s built-in context handling.Long inputs on Nova-style models are split into ~24k-character chunks, embedded per chunk, then merged via length-weighted averaging and L2 normalization into one vector per document (sync and async
embed_documents). Unit tests cover factory branching, chunk splitting, and reduction behavior.Reviewed by Cursor Bugbot for commit 7cf344c. Bugbot is set up for automated code reviews on this repo. Configure here.