[sylvesterkaczmarek]

Fixes #364.

Change

Keep unparseable event-verification responses in the score as abstentions instead of removing them. The current evaluator can report macro F1 of 1.0 when only one of three grounded samples has a valid answer. The correction counts an abstention as a false negative for its ground-truth class, without including a synthetic abstention class in the class average.

The labels used by fully parseable results retain their existing selection. Valid Predictions remains the parse-coverage count, Total Samples remains the input count, and overall accuracy is retained. An entirely invalid set of grounded responses scores zero. Empty or ungrounded input remains unscored. Ground-truth lookup and answer extraction are unchanged.

This affects the shared MetropolisEventVerification evaluator, including VANTAGE_EventVerification. The function docstring documents the abstention policy. The generated/vendored score reporter is unchanged; tests exercise it to verify the published headline follows the corrected macro-F1 result.

Verification

• All 14 regression cases pass on macOS arm64 and Linux arm64 with Python 3.12. Six fail and eight controls pass against the original evaluator.
• Cases cover one-correct/multiple-invalid results, both ground-truth classes, partially and entirely invalid outputs, missing responses, existing case-insensitive answer parsing, empty input and missing-ground-truth handling.
• Fully valid reports are checked against every original scikit-learn report field. The score fixtures use analytically known F1 values and verify scored support, parse counts and the headline extraction.
• Tests use actual pandas and scikit-learn, the actual evaluator, its base class and file helpers. They write prediction CSVs, read the generated result CSVs and verify the input files remain unchanged. No scoring, parsing or file-I/O operation is mocked.
• The test fixture bypasses only the package initializers that eagerly import unrelated model registries and change SSL defaults. Dataset construction is also bypassed to avoid downloading videos; synthetic annotations are supplied to the real evaluation method. These tests therefore validate scoring and reporting, not video loading or inference.
• Python correctness lint checks, syntax compilation and git diff --check pass. The Linux runtime has networking disabled.

From evaluation/cosmos3/reasoner/vlmevalkit, with the evaluator's CPU dependencies and pytest installed:

python -m pytest -q tests/test_event_verification.py

The complete requirements environment cannot be installed on macOS arm64 because eva-decord has no compatible wheel. Local validation used the real CPU dependencies needed by this evaluator; no requirements files were changed. Full benchmark runs, video loading, CUDA, model inference and remote datasets were not exercised. No existing test, reporting port or metric tolerance was weakened.

Scope and coordination

Based on main 77ef1929. Issue #364 was opened on 24 September 2026. I described this approach in the issue before implementing it. It was unassigned and no competing evaluator patch was found when rechecked before submission. The open broad evaluation PR #333 does not change this evaluator.

This PR contains only the evaluator correction and its regression file. It adds no runtime dependency, model, benchmark dataset or external-service call. The commit is signed off and its signature is verified by GitHub.

[sylvesterkaczmarek]

Following up on the event-verification scoring fix. Could a maintainer review the abstention policy and regression coverage, and approve the waiting workflow?

[yaoxu-crypto]

Thanks for this, @sylvesterkaczmarek. The Cosmos cookbooks and evaluation suites have moved to NVIDIA/cosmos-framework under .agents/skills/ and were removed from this repo in #384, so this PR can no longer merge here.

The abstention fix for #364 isn't in Cosmos Framework yet. Could you re-submit it against .agents/skills/evaluation/cosmos3/reasoner/vlmevalkit/vlmeval/dataset/metropolis_event_verification.py, with your new test?

The relative paths are unchanged, just under .agents/skills/ (for example cookbooks/cosmos3/... is now .agents/skills/cookbooks/cosmos3/...). Please note that Cosmos Framework requires DCO sign-off on each commit (git commit -s, see CONTRIBUTING.md). I'm closing this PR because it can't merge here, but the work is welcome there. Please link this PR from the new one so the earlier discussion and reviews carry over. Thanks for your patience, and sorry it sat here while the move happened.

[sylvesterkaczmarek]

Thanks for the guidance, @yaoxu-crypto. I have resubmitted the evaluator correction and its regression tests as NVIDIA/cosmos-framework#350. All 14 focused cases pass in the new repository, and the commit is DCO-signed. Appreciate the help with the move!