[fix][fsdp] Resume bitsandbytes checkpoints strictly - #2007
Draft
bvolpato wants to merge 1 commit into
Draft
Conversation
bvolpato
force-pushed
the
bvolpato/fsdp-bnb-checkpoint-resume
branch
from
August 10, 2026 00:56
fcdd6d9 to
52951bc
Compare
bvolpato
force-pushed
the
bvolpato/fsdp-bnb-checkpoint-resume
branch
from
August 10, 2026 01:04
52951bc to
035c703
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
FSDP checkpoints for bitsandbytes 4-bit modules contain auxiliary quantization state next to each weight, including absmax, quantization maps, nested maps, and serialized FP4 or NF4 state. A freshly constructed module can consume its weight while reporting those auxiliary entries as unexpected. SkyRL currently requests strict loading, so resume stops before optimizer, scheduler, RNG, or dataloader state can be restored.
Change
Why this belongs in SkyRL
SkyRL writes and restores FSDP checkpoint state. Resume must understand metadata emitted by supported 4-bit training modules. Fix is checkpoint-format compatibility, independent of model architecture or rollout backend. It complements QLoRA support in #1968 without combining model-loading changes into this PR.
Compatibility and safety
Validation
CI note
Current full-tree Black failure is already present on main in tests/tinker/test_engine.py and is isolated in #2009.