Skip to content

Scoring

Jump to bottom
mcmahj45 edited this page Jul 24, 2026 · 2 revisions

Scoring

DRES supports multiple scoring methods. The scoring method for a task is set at the task type level in the competition template. See Competition Management for how to configure task types.

KIS — Known Item Search

Used for tasks where participants must find a single specific item (e.g. a video clip or image).

How it works:

  • A team scores points only if they submit the correct item.
  • The score decreases linearly the longer it takes them to find it.
  • Each wrong submission before the correct one reduces the final score by a fixed penalty.

Formula:

score = max(0, maxPointsAtTaskEnd + (maxPointsPerTask - maxPointsAtTaskEnd) × timeFraction - wrongSubmissions × penalty)

Where timeFraction = 1 - (timeOfCorrectSubmission - taskStart) / taskDuration

Default parameters:

Parameter Default
maxPointsPerTask 1000 (score for an instant correct submission)
maxPointsAtTaskEnd 500 (minimum score for a correct submission at the very end)
penaltyPerWrongSubmission 100

These can be overridden per task type.

AVS — Ad-hoc Video Search

Used for tasks where participants must find as many relevant videos as possible within the task time.

How it works:

  • Teams are rewarded for each distinct correct video they find.
  • Wrong submissions for a given video reduce the score for that video.
  • Videos where the team submitted only wrong answers and never found the correct item also subtract a penalty — it is not limited to penalizing wrong submissions made before an eventual correct one.
  • The final score is normalised by the total number of distinct correct videos found across all teams.

Formula (per team):

score = max(0, Σ over correctly-found videos: (1 - wrongSubmissionsForVideo × penalty)
             + Σ over videos never found correctly: (wrongSubmissionsForVideo × -penalty))
        / totalCorrectVideos × maxPointsPerTask

Default parameters:

Parameter Default
maxPointsPerTask 1000
penalty 0.2 (per wrong submission per video)

LEGACY_AVS

The previous AVS scoring formula from earlier versions of DRES. Kept for backwards compatibility with existing evaluation setups. New evaluations should use AVS instead.

NOOP — No Scoring

No score is calculated. Useful for:

  • Practice tasks where results do not count
  • Tasks where scoring is handled by an external system

Score Display

During a run, scores are shown live in the scoreboard in the run viewer. There are two views:

  • Task group view — scores for the current task group only
  • Competition overview — stacked breakdown across all task groups

After a run, the full score history (how each team's score changed over time) is available at /evaluation/scores/{runId}.

Scores can also be exported as a CSV file — see Downloading Results.

Clone this wiki locally