search: exclude memories by status/metadata inside the vector query (avoid over-fetch + post-filter)

[markjbrown]

Problem

Callers that soft-prune or otherwise mark memories with a status (e.g. a retention policy that sets retention_status = "pruned", or any lifecycle/status flag) currently cannot exclude those memories inside the vector search. search / search_cosmos support include_superseded (a c.superseded_by predicate) but no general status/metadata exclusion, so callers must over-fetch (inflate top_k) and post-filter in Python.

That has three costs:

1. RU — the vector query scans/returns rows that are immediately discarded.
2. Relevance budget — top_k is spent on soon-dropped rows, so genuinely relevant memories can fall outside the window.
3. Context tokens — pruned/inactive memories are embedded into the recall path before being filtered out.

Current behavior (code refs)

• search_cosmos delegates straight to store.search:
  • async azure/cosmos/agent_memory/aio/cosmos_memory_client.py:695
  • sync azure/cosmos/agent_memory/cosmos_memory_client.py:657
• store.search builds the query via the QueryBuilder + build_search_sql:
  • async azure/cosmos/agent_memory/aio/store/memory_store.py:804
  • sync azure/cosmos/agent_memory/store/memory_store.py:840
• The only status-like predicate today is include_superseded, applied as qb.add_is_null_or_undefined("c.superseded_by") (e.g. aio/store/memory_store.py:335-336).

Proposed change

Add an optional status/metadata exclusion predicate to store.search and surface it on search_cosmos (async + sync), applied inside the SQL like include_superseded — so pruned/inactive memories are filtered by the engine, not in the caller.

Two shapes for maintainers to choose between (happy to implement whichever you prefer):

• A — top-level status list: exclude_statuses: list[str] | None → (NOT IS_DEFINED(c.status) OR c.status NOT IN (...)). Simple; assumes a conventional top-level c.status.
• B — metadata predicate (matches the SDK's c.metadata bag): e.g. exclude_metadata: dict[str, list[str]] | None → predicates on c.metadata.<key>. More general and idiomatic to the existing metadata extension point.

Note: the reference caller (below) currently writes a top-level c.retention_status, but I'm glad to align it to whichever convention you choose (e.g. move it under metadata).

Use case

The travel-multi-agent-workshop Module-08 memory-retention optimization soft-prunes superseded / low-salience memories (a reversible mark) so recall stays cheaper and cleaner. It wants recall to exclude pruned memories in the vector query rather than post-filter.

Offer

I have the implementation ready to follow the include_superseded pattern across async+sync search/search_cosmos (+ a unit test). Tell me which API shape you prefer (A or B, and the field/convention) and I'll open the PR. This issue will be cross-linked from the travel workshop PR.