EmbeddingSimilarity reuses cached embeddings after switching the default client with init() #256

Bug description

The Python EmbeddingSimilarity scorer can reuse embeddings from the previous default client after init(client=...) selects a different client. This silently returns a score based on the old provider's vectors without contacting the newly configured provider.

This affects scorers constructed both before and after changing the default client. The same cache is also shared across asyncio tasks that have different default-client contexts.

Reproduction

Reproduced on main at 586cc167eb625d84b11581936b1e11619ec0d15a (autoevals 0.4.1), macOS ARM64, Python 3.12.15, OpenAI SDK 2.41.0, and HTTPX 0.28.1.

The example uses real OpenAI SDK clients and the public scorer APIs, with HTTPX MockTransport intercepting all HTTP requests. It requires no API keys or real network requests. The controlled embedding fixtures use the same model name at two different endpoints:

• Provider A returns identical vectors for output and reference, so the score is 1.
• Provider B returns orthogonal vectors, so the score is 0.

"""Offline reproduction using public scorers and two mock OpenAI endpoints.

The embedding vectors are controlled fixtures, not real model judgments.
No production code is replaced or monkeypatched. HTTPX blocks real networking.
"""

import json

import httpx
from openai import AsyncOpenAI, OpenAI

from autoevals import EmbeddingSimilarity, init


def mock_provider(name, calls):
    def handle(request):
        body = json.loads(request.content)
        text = body["input"]
        calls.append({"provider": name, "input": text, "model": body["model"]})
        vector = [0.0, 1.0] if name == "B" and text.startswith("reference") else [1.0, 0.0]
        return httpx.Response(
            200,
            json={
                "object": "list",
                "model": body["model"],
                "data": [{"object": "embedding", "index": 0, "embedding": vector}],
                "usage": {"prompt_tokens": 1, "total_tokens": 1},
            },
        )

    return handle


def sync_reproduction():
    calls = []
    with (
        OpenAI(
            api_key="offline-test-key",
            base_url="https://provider-a.example/v1",
            http_client=httpx.Client(transport=httpx.MockTransport(mock_provider("A", calls))),
        ) as client_a,
        OpenAI(
            api_key="offline-test-key",
            base_url="https://provider-b.example/v1",
            http_client=httpx.Client(transport=httpx.MockTransport(mock_provider("B", calls))),
        ) as client_b,
    ):
        init(client=client_a)
        first = EmbeddingSimilarity(expected_min=0).eval("output-sync", "reference-sync").score
        init(client=client_b)
        before = len(calls)
        cached = EmbeddingSimilarity(expected_min=0).eval("output-sync", "reference-sync").score
        requests_after_switch = len(calls) - before
        fresh = EmbeddingSimilarity(expected_min=0).eval("output-fresh-sync", "reference-fresh-sync").score
        explicit = EmbeddingSimilarity(client=client_b, expected_min=0).eval("output-sync", "reference-sync").score
    init()
    return {
        "mode": "sync",
        "provider_a_score": first,
        "provider_b_same_inputs_score": cached,
        "provider_b_requests_after_switch": requests_after_switch,
        "provider_b_fresh_inputs_score": fresh,
        "provider_b_explicit_client_score": explicit,
        "requests": calls,
    }


if __name__ == "__main__":
    print(json.dumps(sync_reproduction(), indent=2))

Actual behavior

On the unmodified revision, the relevant output is:

{
  "mode": "sync",
  "provider_a_score": 1.0,
  "provider_b_same_inputs_score": 1.0,
  "provider_b_requests_after_switch": 0,
  "provider_b_fresh_inputs_score": 0.0,
  "provider_b_explicit_client_score": 0.0
}

The same-input score after selecting B is still 1, while B receives no requests. Fresh inputs and an explicitly supplied B client both produce 0. I also reproduced the same behavior with AsyncOpenAI and eval_async.

Expected behavior

Once B is selected as the default client, evaluating the original input pair should use embeddings associated with B and return 0 in this fixture. A client context must not reuse vectors from another client context. Repeated requests under the same configuration should still benefit from caching.

Likely cause and related work

EmbeddingSimilarity._cache_key() uses id(self.client) when a client was supplied explicitly, otherwise None. However, request execution resolves a missing explicit client through the default-client ContextVar set by init(). The default client therefore changes the actual request but not the cache key.

This is a follow-up to #211 / #213, which addressed model/prefix isolation and included explicit-client isolation. The remaining case here is a client selected through init(), rather than another model/prefix cache fix.

Local validation

I have prepared a local fix that keys embeddings by the effective client, using an internal identity token for LLMClient instances so that replaced default clients cannot collide through reused integer object IDs.

• Before the fix, the new repository regression cases reported 3 failures; the 2 existing cache cases passed.
• After the fix, 40 related tests pass locally, including default-client switching, same-client cache reuse, explicit-client precedence, and asynchronous context isolation.
• I personally reran the related tests: 40 passed in 0.85s.
• The changed files pass the project's pre-commit hooks.
• The full repository suite, live API integration tests, and GitHub CI have not been run.

AI assistance: OpenAI Codex assisted with investigation, reproduction, implementation, and regression tests.