Skip to content

BUG: AzureContentFilterScorer.get_scorer_metrics always returns None #3544

Open

Description

AzureContentFilterScorer only sets evaluation_file_mapping inside evaluate_async. A scorer you just built for one category has no mapping, so get_scorer_metrics() returns None even though the registry has its entry with a matching eval hash.

scorer = AzureContentFilterScorer(harm_categories=[TextCategory.HATE])
scorer.evaluation_file_mapping   # None
scorer.get_scorer_metrics()      # None, but hate_speech_metrics.jsonl has this eval hash

Same for violence, sexual and self harm. Setting the mapping in the constructor for single-category scorers fixes it, like the Likert scorers already do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions