Skip to content

fix(checkpointing): load_biencoder_checkpoint passes release twice to get_checkpoint_name - #6975

Open
Anai-Guo wants to merge 2 commits into
NVIDIA:mainfrom
Anai-Guo:fix/biencoder-checkpoint-name-release-arg
Open

fix(checkpointing): load_biencoder_checkpoint passes release twice to get_checkpoint_name#6975
Anai-Guo wants to merge 2 commits into
NVIDIA:mainfrom
Anai-Guo:fix/biencoder-checkpoint-name-release-arg

Conversation

@Anai-Guo

@Anai-Guo Anai-Guo commented Aug 29, 2026

Copy link
Copy Markdown

What does this PR do?

load_biencoder_checkpoint() cannot run. It calls:

checkpoint_name = get_checkpoint_name(
    load_path, iteration, args.use_distributed_optimizer, release=False
)

but the third parameter of get_checkpoint_name() is release:

def get_checkpoint_name(
    checkpoints_path,
    iteration,
    release=False,          # <-- third positional
    pipeline_parallel=None,
    ...
):

So release is supplied both positionally and by keyword, and the call raises:

TypeError: get_checkpoint_name() got multiple values for argument 'release'

There is no use_distributed_optimizer parameter for that value to land in, so the
extra argument is simply stale — it looks left over from an older signature.

The fix

Drop the stray positional. args is still used in the function (args.load), so
nothing else changes.

checkpoint_name = get_checkpoint_name(load_path, iteration, release=False)

Reformatted onto one line because it now fits inside the configured
line_length = 100.

Why this is the right call shape

Every other get_checkpoint_name() call site in checkpointing.py already passes
release as the third positional and nothing extra, e.g.

get_checkpoint_name(load_dir, iteration, release, return_base_dir=True)   # L293, L1932, L2070
get_checkpoint_name(load_dir, iteration, release)                         # L2997

Binding every call site in the file against the real signature: 15 consistent, 1
inconsistent
— the one this PR fixes.

Verification

No GPU needed for this one. I lifted the real get_checkpoint_name signature out of
the source with ast, rebuilt it as an inspect.Signature, and replayed both call
shapes:

call result
get_checkpoint_name(load_path, iteration, args.use_distributed_optimizer, release=False) (current) TypeError: multiple values for argument 'release'
get_checkpoint_name(load_path, iteration, release=False) (this PR) binds cleanly, release=False

pylint's E1124 (redundant-keyword-arg) covers exactly this shape, but the
repo's pre-commit pylint hook does not flag it today, which is presumably how it
survived.

🤖 Generated with Claude Code

load_biencoder_checkpoint() calls

    get_checkpoint_name(load_path, iteration, args.use_distributed_optimizer,
                        release=False)

but get_checkpoint_name()'s third parameter IS `release`, so the call passes
`release` both positionally and by keyword and raises
`TypeError: get_checkpoint_name() got multiple values for argument 'release'`.
There is no `use_distributed_optimizer` parameter to receive it.

Every other call site in the file passes `release` as the third positional and
nothing else; this one is the only inconsistent one.

Signed-off-by: Anai-Guo <antai12232931@outlook.com>
@Anai-Guo
Anai-Guo requested review from a team as code owners August 29, 2026 13:19
@copy-pr-bot

copy-pr-bot Bot commented Aug 29, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@svcnvidia-nemo-ci
svcnvidia-nemo-ci marked this pull request as draft August 29, 2026 13:19
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been automatically converted to draft because all PRs must start as drafts.

When you are ready for review, click Ready for Review to begin the review process. This will:

  1. Add the oncall reviewer (optional reviewer)
  2. Add required review teams based on your changes

See the contribution guide for more details.

@Anai-Guo
Anai-Guo marked this pull request as ready for review August 30, 2026 22:55
@svcnvidia-nemo-ci
svcnvidia-nemo-ci requested a review from a team August 30, 2026 22:55
checkpoint_name = get_checkpoint_name(
load_path, iteration, args.use_distributed_optimizer, release=False
)
checkpoint_name = get_checkpoint_name(load_path, iteration, release=False)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wish get_checkpoint_name had adopted https://realpython.com/python-asterisk-and-slash-special-parameters/

Are you interested in adding them?

@wujingyue

Copy link
Copy Markdown
Contributor

/claude fix

@wujingyue wujingyue self-assigned this Aug 31, 2026
Signed-off-by: svcnvidia-nemo-ci <svcnvidia-nemo-ci@nvidia.com>
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

🛠️ Claude fix commit 646030b49fa6 (attempt 1)

⚠️ This explanation is AI-generated and may be inaccurate; the exact commit is authoritative.

What changed
No edits made; none were needed. The pinned base merge is already textually clean: no unmerged paths, no conflict markers, and the staged tree (bc71185) is byte-identical to an independently recomputed git merge-tree auto-merge. I verified the PR's one-line change is correct (it drops a stale positional arg that landed in get_checkpoint_name's ’release’ slot) and audited all 19 call sites repo-wide as already matching the current signature. File parses, compiles, and is under the 100-char limit.

Files changed by Claude

  • No additional file edits; the pinned base was merged.

Why
No conflict or failure observed. The merge auto-resolved because the change sets are disjoint: the base touches 41 inference/ssm/fsdp files, the PR touches only megatron/training/checkpointing.py, which the base does not modify. The claude-fix-ci log dir was empty, so no terminal failure was in scope. Caveat: black, ruff, pylint, pytest and torch are unavailable here, so I ran no formatter or test.

DCO
@Anai-Guo, please fix any DCO failures on your commits before merge. DCO does not block this workflow.

View exact commit

Sanitized and posted by svcnvidia-nemo-ci.

@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

/ok to test 646030b

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Aug 31, 2026
@svcnvidia-nemo-ci

Copy link
Copy Markdown
Contributor

❌ Claude fix stopped because exact-SHA CI did not complete in time. View exact-SHA CI.

@svcnvidia-nemo-ci svcnvidia-nemo-ci removed the waiting-on-customer Waiting on the original author to respond label Aug 31, 2026

@maanug-nv maanug-nv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thanks

@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-customer Waiting on the original author to respond label Aug 31, 2026
@jaredcasper

Copy link
Copy Markdown
Contributor

The biencoder model was removed, the correct fix here is to remove the load_biencoder_checkpoint function all together. (and _add_biencoder_args() from arguments.py) while we're at it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request waiting-on-customer Waiting on the original author to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants