[Jacqueline122]

Summary

Fixes #256.

After init(client=...) switches the default client, Python's EmbeddingSimilarity can return a score computed from the previous client's cached vectors without contacting the new provider. In the offline reproduction, provider B should yield 0.0 but returns the cached 1.0.

Key embeddings by the effective explicit or context-default client. An internal identity token on LLMClient prevents cache collisions if a replaced client's integer object ID is reused. Repeated evaluations with the same client still reuse cached embeddings.

Regression tests cover sync/async client switching, cache reuse across scorer instances, explicit-client precedence, and isolation between asyncio tasks. This follows the model/prefix isolation work in #213; the remaining case is a default client configured through init().

Validation

• Before the fix: 3 new regression cases failed; the 2 existing cache cases passed.
• After the fix: I ran the related tests locally — 40 passed.
• Changed files pass the project's pre-commit hooks.
• Full repository tests and live API integration tests were not run.

To run the 40 targeted tests from the repository root:

uv run --extra dev --extra scipy pytest \
  py/autoevals/test_embeddings.py::test_embedding_cache_isolated_by_default_client \
  py/autoevals/test_embeddings.py::test_embedding_cache_isolated_by_async_client_context \
  py/autoevals/test_embeddings.py::test_embedding_cache_isolated_by_configuration \
  py/autoevals/test_oai.py \
  py/autoevals/test_init_models.py \
  py/autoevals/test_ragas.py::test_answer_correctness_uses_custom_embedding_model \
  py/autoevals/test_json.py \
  py/autoevals/test_values.py -q

AI assistance

OpenAI Codex assisted with investigation, reproduction, implementation, and regression tests.