Python: relevance-score gates (min_relevance_score / DEFAULT_RELEVANCE) have no polarity cases in tests: a memory that says the opposite of the query is recalled as highly relevant. #14295

Activity

1. DariusHQ commented on Aug 17, 2026

   

   Darius (DariusHQ)

   on Aug 17, 2026

   Last edited by DariusHQ

   More actions

   Hi Scott, I took a look at the current C# main implementation before attempting a PR, and noticed a couple of differences from the current issue description.
   In the current TextMemoryPlugin, DefaultRelevance is 0.0, not 0.75, while SemanticTextMemory.SearchAsync does pass minRelevanceScore through to IMemoryStore.

   I also couldn't find an equivalent relevance-score gate in the current VectorStoreTextSearch successor path, it appears to use VectorSearchOptions<TRecord> and vector-store search without the same minRelevanceScore parameter

   I may be looking at a newer/different path than the one used for the audit. Could you clarify which current C# implementation/commit the DEFAULT_RELEVANCE = 0.75 and collection.as_text_search references are targeting?

   I wanted to confirm the intended path before preparing a PR so that the tests match the current SK implementation.
2. poeticize commented on Aug 25, 2026

   

   Scott Frias (poeticize)

   on Aug 25, 2026

   Author

   More actions

   Darius,

   Thanks for actually reading the code before writing the PR, and sorry for the slow reply.

   You are looking at the right code. I pointed you at the wrong language: the audit ran against the
   Python implementation, and DEFAULT_RELEVANCE and collection.as_text_search are Python
   identifiers. The issue body never said so, which is my fault, and I have fixed it. Re-verified
   against main at b39d95a today:

   • python/semantic_kernel/core_plugins/text_memory_plugin.py#L22:
     DEFAULT_RELEVANCE: Final[float] = 0.75, the default for TextMemoryPlugin.recall(...) at L55.
     The class deprecation at L27 is the one pointing at collection.as_text_search.
   • python/semantic_kernel/memory/semantic_text_memory.py#L129:
     SemanticTextMemory.search(..., min_relevance_score: float = 0.0, ...), passed through to the
     store at L153, exactly as you described the C# behaving.

   Your C# readings are both correct.
   TextMemoryPlugin.cs#L47
   has private const double DefaultRelevance = 0.0;, so the .NET plugin ships an open gate where the
   Python one ships 0.75. I will add it that divergence in the issue.

   Your VectorStoreTextSearch observation is the more interesting one, and strengthens the
   case. There is no minRelevanceScore there: ExecuteVectorSearchCoreAsync builds a VectorSearchOptions<TRecord> carrying Filter only, and passes top. So the successor path does not carry the score gate forward, it drops it. A caller who migrates off TextMemoryPlugin moves from a threshold that can at least be raised to a top-k that always returns its k, and the "administer the study drug" record comes back as the top hit with no floor left to argue about. My issue said the migration ports the blind spot forward. That was too generous, and I would rather state it your way.

   On where the Python gap actually sits, my original wording was loose. recall has no behavioral unit
   test at all. Across python/tests, TextMemoryPlugin appears only in test_serialization.py, and
   the one test that exercises the gate is
   test_embedding_service_with_memory.py#L155-L169,
   an integration test needing live credentials, which calls memory.search(...) four times and passes
   min_relevance_score=0.0 at every call site. The threshold is never exercised at a non-zero value
   anywhere in the suite, and the fixtures are taxonomic facts (Sharks are fish, Penguins are birds), so
   no pair in the suite shares wording while disagreeing about what it decides.

   One caution if you do write the PR, because there is already an open one against this issue (#14319)
   that ran into it. A unit test cannot call an embedding model, so the tempting move is to hand-author
   the vectors, and a hand-authored vector asserts only the fixture author's arithmetic. The test then
   passes whatever the sentences say, which is the failure this issue is about, reproduced one level up.
   What I think does work:

   1. Record real vectors once from a named encoder for the four sentences and check them in, with
      provenance in the fixture (model id, dimension, date, generating script). The assertion then says
      something about the world: the opposite-meaning record clears the shipped default while the
      faithful paraphrase does not. It stays frozen and deterministic, and it fails if the claim stops
      being true.
   2. Prefer an encoder the ecosystem actually runs, such as text-embedding-3-small, over the one in
      my paper. The effect does not hinge on the choice; the mutation class scored 0.83 to 0.9997 across
      all nine encoders I tested.
   3. Keep any live-encoder assertion in the integration project behind its existing trait, not in the
      unit tests.

   I am happy to generate the recorded vectors and hand them over so you keep authorship of the PR, in
   Python or C# or both. Say which and I will produce them.
3. Scott Frias (poeticize)

   changed the title [-]Relevance-score gates (min_relevance_score / DEFAULT_RELEVANCE) have no polarity cases in tests: a memory that says the opposite of the query is recalled as highly relevant.[/-] [+]Python: relevance-score gates (min_relevance_score / DEFAULT_RELEVANCE) have no polarity cases in tests: a memory that says the opposite of the query is recalled as highly relevant.[/+] on Aug 25, 2026