Skip to content

BUG: Inferred objective is empty when tool calls come between the request and the answer #3515

Open

Description

Describe the bug

extract_objective_from_previous_turn_async (used when a scorer infers the objective from the request) only reads the turn at sequence - 1. When a target runs a tool loop, memory stores the user request, the function_call, the function_call_output and then the answer, so the turn before the answer is the tool output. It has no user text, and the scorer ends up with an empty objective.

#3410 fixed the same problem in MemoryInterface.get_request_from_response_async, and WildGuardScorer already looks back to the latest earlier user turn.

Steps to Reproduce

Store a conversation of user question (seq 2), assistant function_call (seq 3), tool function_call_output (seq 4) and assistant answer (seq 5), then call extract_objective_from_previous_turn_async on the answer.

Expected Results

The objective is the user question from seq 2.

Actual Results

An empty string.

Versions

Linux, Python 3.13.16, PyRIT 1.2.0.dev0 from main (96de757)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions