Skip to content

fix: warn when a task resolves to zero datasets - #57

Merged
warreveys merged 1 commit into
techwolf-ai:mainfrom
federetyk:fix/warn-empty-dataset-ids
Jul 22, 2026
Merged

warreveys merged 1 commit into
techwolf-ai:mainfrom
federetyk:fix/warn-empty-dataset-ids

Conversation

@federetyk

Copy link
Copy Markdown
Contributor

Description

  • Task.__init__ silently produced an empty task in two cases: when the requested languages matched no dataset, and when every selected dataset was skipped with DatasetConfigNotSupported during loading. Both cases now log a warning that names the task and explains why it is empty.
  • For cross-lingual tasks the first case is easy to hit: a dataset is selected only when the requested languages cover both its query and target languages, so SearchQueryCandidateRanking(split="test", languages=["de"]) selects nothing. The warning states this and shows the required call shape (e.g. languages=["de", "en"] for a German-to-English dataset).
  • Motivated by a RecSys 2026 reviewer report: the call above returned empty results with no indication why, and the reviewer concluded the cross-lingual CRec evaluation was not reproducible from the documented API.
  • Behavior is otherwise unchanged. An empty task still evaluates to no results, which existing flows rely on (e.g. examples/run_paper_results.py expects MELSRanking to have zero datasets for languages=["en"]).
  • Tests: new TestEmptyDatasetWarnings in tests/test_task_loading.py covers both warnings plus a no-warning control case. The tests use an autouse fixture that temporarily enables propagation on the workrb logger, since setup_logger sets propagate = False, which hides records from pytest's caplog.

Checklist

  • Added new tests for new functionality
  • Tested locally with example tasks
  • Code follows project style guidelines
  • Documentation updated
  • No new warnings introduced
  • If the rankings artifact schema changed: bumped SCHEMA_VERSION in workrb/rankings.py and updated SUPPORTED_SCHEMA_VERSIONS

Task.__init__ silently produced an empty task in two cases: when the
requested languages matched no dataset (for cross-lingual tasks the
request must cover both query and target languages), and when every
selected dataset was skipped with DatasetConfigNotSupported during
loading. Both cases now log a warning that names the task and explains
why it is empty.

Reported by a RecSys 2026 reviewer: SearchQueryCandidateRanking with
languages=["de"] returned empty results with no indication why.

@warreveys warreveys left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution!

@warreveys
warreveys merged commit c417039 into techwolf-ai:main Jul 22, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants