Repository navigation
Conversation
Resuming from a checkpoint re-initialized the learning rate scheduler and the AMP grad scaler, so a resumed run diverged from an uninterrupted one. Store both state dicts and restore them in load_checkpoint. Also rename the misspelled optimizer_state_dit key to optimizer_state_dict, while still reading the old key so existing checkpoints keep loading. Fixes PolymathicAI#77
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #77.
Resuming from a checkpoint re-initialized the learning rate scheduler and the AMP grad scaler, so a resumed run diverged from an uninterrupted one.
Changes:
Trainer.save_modelnow storeslr_scheduler_state_dict(when a scheduler is set) andgrad_scaler_state_dict.Trainer.load_checkpointrestores both. If a checkpoint has no scheduler state (older checkpoints), it logs a warning and keeps the scheduler at its initial state.optimizer_state_ditkey tooptimizer_state_dict. Loading still falls back to the old key, so existing checkpoints keep working.tests/benchmark/test_trainer_checkpoint.py: one test resumes mid-schedule and checks the LR, scheduler step and grad scale; one loads a checkpoint written with the legacy key. The first test fails without this change.