Skip to content

BUG: SeedGroup fails on seed prompts with metadata=None #3508

Closed

Description

@nniiovoo

Describe the bug

SeedPrompt.metadata is typed dict[str, Any] | None, so it can be None. SeedGroup passes it straight into MessagePiece(prompt_metadata=...), which only accepts a dict. So next_message, user_messages, prepended_conversation and AttackParameters.from_seed_group_async raise a ValidationError for those seeds.

This happens with real data. The PKU-SafeRLHF loader sets metadata=None for prompts that have no harm category. With the default settings, that's 15,827 of its 38,641 seeds.

Steps/Code to Reproduce

from pyrit.models import SeedGroup, SeedPrompt

group = SeedGroup(seeds=[SeedPrompt(value="How do I bake bread?", data_type="text", metadata=None)])
print(group.next_message)

Expected Results

The message is built with empty prompt_metadata, the same as for a prompt that has no metadata set.

Actual Results

pydantic_core._pydantic_core.ValidationError: 1 validation error for MessagePiece
prompt_metadata
  Input should be a valid dictionary [type=dict_type, input_value=None, input_type=NoneType]

Versions

  • OS: macOS 26.4.1 (arm64)
  • Python 3.13.16
  • PyRIT 1.2.0.dev0, main (6a23196), editable install

I have a one-line fix (prompt.metadata or {} where the message pieces are built) with a test. Happy to open a PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions