Skip to content

Save LR scheduler and grad scaler state in checkpoints - #102

Open
barrosyan wants to merge 1 commit into
PolymathicAI:masterfrom
barrosyan:fix/checkpoint-scheduler-scaler-state
Open

barrosyan wants to merge 1 commit into
PolymathicAI:masterfrom
barrosyan:fix/checkpoint-scheduler-scaler-state

Conversation

@barrosyan

Copy link
Copy Markdown

Fixes #77.

Resuming from a checkpoint re-initialized the learning rate scheduler and the AMP grad scaler, so a resumed run diverged from an uninterrupted one.

Changes:

  • Trainer.save_model now stores lr_scheduler_state_dict (when a scheduler is set) and grad_scaler_state_dict.
  • Trainer.load_checkpoint restores both. If a checkpoint has no scheduler state (older checkpoints), it logs a warning and keeps the scheduler at its initial state.
  • Rename the misspelled optimizer_state_dit key to optimizer_state_dict. Loading still falls back to the old key, so existing checkpoints keep working.
  • Add tests/benchmark/test_trainer_checkpoint.py: one test resumes mid-schedule and checks the LR, scheduler step and grad scale; one loads a checkpoint written with the legacy key. The first test fails without this change.

Resuming from a checkpoint re-initialized the learning rate scheduler and
the AMP grad scaler, so a resumed run diverged from an uninterrupted one.
Store both state dicts and restore them in load_checkpoint.

Also rename the misspelled optimizer_state_dit key to optimizer_state_dict,
while still reading the old key so existing checkpoints keep loading.

Fixes PolymathicAI#77

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Checkpoint resume diverges from continuous training due to missing LR scheduler/grad scaler states

1 participant