You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Feature request: speciesist bias harm definition and seed prompts #3501
PyRIT can probe for representational harm against groups of people, but not for speciesist bias: models treating harm to some animals as acceptable or invisible while treating the same harm to other animals as wrong. This is now measured in the literature:
Hagendorff, Bossert, Tse & Singer, "Speciesist bias in AI: how AI applications perpetuate discrimination and unfair outcomes against animals", AI and Ethics (2022), https://doi.org/10.1007/s43681-022-00199-9
"Speciesism in AI: Evaluating Discrimination Against Animals in Large Language Models" (SpeciesismBench), https://arxiv.org/abs/2508.11534
"What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Text" (AHA benchmark), https://arxiv.org/abs/2503.04804
Describe the solution you'd like
A small first step that follows existing patterns:
A harm definition in pyrit/datasets/harm_definition/ (for example speciesist_bias.yaml) with a 1–5 scale running from "weighs animal interests consistently" to "dismisses or hides serious harm to some species while recognising it for others".
A few local seed prompts that test for inconsistency, such as paired prompts that are identical except for the species (dog vs pig, cat vs chicken).
The rubric would score inconsistency and dismissiveness, not whether a response recommends a particular diet.
Describe alternatives you've considered, if relevant
Adding remote dataset loaders for the benchmarks above. I'd keep that for later, once licensing for each dataset is confirmed.
Additional context
I'm happy to open the PR. Would you prefer a new harm category, or adding this under representational?
Activity