Skip to content

FIX Correlate multi-turn attack error results - #2322

Open
romanlutz wants to merge 1 commit into
microsoft:mainfrom
romanlutz:romanlutz-daily-audit-2026-08-04
Open

FIX Correlate multi-turn attack error results#2322
romanlutz wants to merge 1 commit into
microsoft:mainfrom
romanlutz:romanlutz-daily-audit-2026-08-04

Conversation

@romanlutz

@romanlutz romanlutz commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Description

PR #1720 added generic persistence for uncaught attack exceptions. That handler looked for context.conversation_id, which is correct for SingleTurnAttackContext, but multi-turn attacks store the active ID at context.session.conversation_id. For every terminal multi-turn exception, the lookup therefore failed and the handler generated a new UUID.

If turns had already been stored under conversation A, the resulting AttackOutcome.ERROR row was persisted under unrelated conversation B:

  • The conversation messages remained intact under A.
  • The AttackResult was valid but incorrectly referenced B.
  • The database accepted the mismatch because conversation_id is a string rather than a foreign key, and a legitimate attack may fail before writing any messages.
  • Backend history and frontend views then queried statistics for B, commonly reporting zero messages and no preview while the partial conversation under A was disconnected from the result.

This is logical record corruption rather than malformed rows or lost message data. It affects only uncaught terminal exceptions in multi-turn attacks since #1720; successful attacks, ordinary non-exception outcomes, and single-turn errors are unaffected. Existing affected rows cannot be repaired safely in a general migration because a generated ID from this bug is indistinguishable from a legitimate pre-first-message failure without attribution, logs, or other execution evidence identifying the original conversation.

This change resolves the conversation ID from the direct context field first, then from the multi-turn conversation session, retaining UUID generation only for contexts that expose neither layout.

Tests and Documentation

  • Added a regression test using a real MultiTurnAttackContext and ConversationSession to verify timeout error results retain the active conversation ID.
  • Ran the attack strategy and Crescendo unit suites: 127 passed.
  • Ran the dataset, Many-shot, and scenario-technique invariant suites: 4,688 passed.
  • Ruff, formatting, ty, repository commit hooks, and git diff --check passed.
  • Documentation: N/A; this corrects internal error-result correlation without changing the public API.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 0d1f97bd-49a5-4a6d-b814-fecef1e2a8cb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants