Repository navigation
[Bug]: long_time_metrics reflect only last batch of data and not the average over all batches #78
Description
The long_time_metrics saved to time_logs inside the Trainer.validation_loop method are overriden for every batch. This means that the metrics use only the last batch
A fix would be something like this:
# FIX: Accumulate time_logs averages instead of overwriting with |=
if k in long_time_metrics or "spectral_error" in k:
for key, val in new_time_logs.items():
if key in time_logs:
time_logs[key] += val / denom
else:
time_logs[key] = val / denom
Activity
Adonyth commentedon Jul 12, 2026
We are running an independent quantified-impact study of this issue together with the related benchmark/metric issues (#49, #67, #74, #75, #85): how much do these documented defects move the published baseline numbers, and what general lessons follow for rollout-benchmark metric design. The full study design was frozen today (2026-07-12); tamper-evident commitment: sha256
87c9ea19a266dfe4168fc4e63bc904e124091a97a3f2d2f472848d2bf6434017(manifest: https://gist.github.com/Adonyth/d11062585e03d37331801a41fe7138b7). We will share quantified results — and upstream PRs where fixes are straightforward — in these threads as they land.Adonyth commentedon Jul 12, 2026
Quantified confirmation + fix (as promised above):
time_logs |= new_time_logsinTrainer.validation_loopoverwrites long-time-metric keys each batch, so the logged curves reflect only the final batch — a two-batch counterexample with per-batch values 100.0 and 1.0 reports 1.0 instead of the correct 50.5. Theloss_dictaccumulation just below it already does this correctly; PR #93 mirrors that accumulation (samedenomweighting) with a regression test that fails on current master and passes with the fix.Average long_time_metrics time curves over validation batches