Repository navigation
BUG: FairnessBiasBenchmark.get_experiment_summary raises TypeError because score_category is a list #3551
Description
Describe the bug
FairnessBiasBenchmark.get_experiment_summary (pyrit/executor/benchmark/fairness_bias.py) raises TypeError: unhashable type: 'list' whenever the experiments were scored. _format_experiment_results_async stores attack_result.last_score.score_category in each experiment result, and Score.score_category is a list[str] (for example ["She/Her"]). The summary then builds Counter(result["score_category"] for result in context.experiment_results), which needs hashable values.
So the summary only works when no experiment has a score (every category is the "unknown" placeholder). The unit test for the summary passes because it sets score_category to plain strings, and the benchmark example in doc/code/executor/6_benchmark.py works around it by flattening r["score_category"] itself instead of calling get_experiment_summary. experiment_results is also annotated as list[dict[str, str]], which does not match the list values stored in it.
Steps/Code to Reproduce
import asyncio
from pyrit.executor.attack import AttackScoringConfig
from pyrit.executor.benchmark import FairnessBiasBenchmark
from pyrit.models import Message, MessagePiece
from pyrit.prompt_target import PromptTarget
from pyrit.score import SubStringScorer
from pyrit.setup import IN_MEMORY, initialize_pyrit_async
class StoryTarget(PromptTarget):
async def _send_prompt_to_target_async(self, *, normalized_conversation: list[Message]) -> list[Message]:
conversation_id = normalized_conversation[-1].message_pieces[0].conversation_id
story = "Maria Garcia is a nurse. She works nights."
return [MessagePiece(role="assistant", original_value=story, conversation_id=conversation_id).to_message()]
def _validate_request(self, *, normalized_conversation: list[Message]) -> None:
pass
async def main():
await initialize_pyrit_async(memory_db_type=IN_MEMORY)
scorer = SubStringScorer(substring="She", categories=["She/Her"])
benchmark = FairnessBiasBenchmark(
objective_target=StoryTarget(),
attack_scoring_config=AttackScoringConfig(objective_scorer=scorer),
)
await benchmark.execute_async(subject="nurse", story_type="short story", num_experiments=2)
context = benchmark.get_last_context()
print("score_category:", [result["score_category"] for result in context.experiment_results])
print(benchmark.get_experiment_summary(context=context))
asyncio.run(main())Expected Results
The summary counts the score categories:
score_category: [['She/Her'], ['She/Her']]
{'subject': 'nurse', 'story_type': 'short story', 'total_experiments': 2, 'score_distribution': {'She/Her': 2}, 'score_proportions': {'She/Her': '100.00%'}, 'names_extracted': ['Maria Garcia', 'Maria Garcia']}
Actual Results
score_category: [['She/Her'], ['She/Her']]
Traceback (most recent call last):
...
File "pyrit/executor/benchmark/fairness_bias.py", line 352, in get_experiment_summary
score_counts = Counter(result["score_category"] for result in context.experiment_results)
...
TypeError: unhashable type: 'list'
Screenshots
N/A
Versions
- OS: Windows 11
- Python version: 3.11.9
- PyRIT version: installed from main (d2c6fff) in editable mode
Activity