Skip to content

Fixes CI base image pulls by not inheriting runner registry credentials - #7330

Closed
myurasov-nv wants to merge 2 commits into
developfrom
ci/anonymous-base-image-pull
Closed

myurasov-nv wants to merge 2 commits into
developfrom
ci/anonymous-base-image-pull

Conversation

@myurasov-nv

@myurasov-nv myurasov-nv commented Aug 25, 2026 •

Copy link
Copy Markdown
Member

Description

CI docker builds intermittently fail to resolve nvcr.io/nvidia/isaac-sim:6.0.1 with denied: Access Denied. setup-docker-config copies the runner's ~/.docker/config.json into the job's temp DOCKER_CONFIG, and NGC denies an inherited org credential lacking a grant instead of falling back to anonymous. Now the temp config starts with empty auths, and compute-deps-hash surfaces resolve errors instead of silently dropping the base-image digest from the cache key. Verified by anonymous manifest fetch with a clean config.

Before After
On a runner with a persistent nvcr.io login, fork-PR builds that miss the deps cache fail at FROM with denied: Access Denied; digest-resolve failures are silently swallowed and drop the digest from the deps-cache key Public base images pull anonymously regardless of runner state; a failed digest resolve fails the step with the underlying error

Reproduction

  1. On a machine whose ~/.docker/config.json holds a valid inline nvcr.io credential for an org with no grant on nvidia/isaac-sim, resolve the base image the same way the build does:

    docker buildx imagetools inspect nvcr.io/nvidia/isaac-sim:6.0.1
    

    Observed in CI: ERROR: failed to authorize: failed to fetch oauth token: denied: Access Denied (https://github.com/isaac-sim/IsaacLab/actions/runs/32741844858; fork PR with a deps-cache miss, and the job logged NGC_API_KEY not set - skipping nvcr.io login, so the only credential came from the runner's home config). The denial requires a valid credential from a non-granted org: a missing or invalid credential falls back to anonymous access and succeeds, which is why only some runners fail.

  2. Repeat with a clean config, as the patched action now provides:

    DOCKER_CONFIG=$(mktemp -d) docker buildx imagetools inspect nvcr.io/nvidia/isaac-sim:6.0.1 --format '{{json .Manifest.Digest}}'
    

    Returns "sha256:783444c706538aa76cf5126e911ddc5e618779e6105305ad4af4260362a30aa9"

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Release backport

  • Backport this pull request to the active release branch after it merges into develop

Checklist

  • I have read and understood the contribution guidelines
  • I have run the pre-commit checks with ./isaaclab.sh --format
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • I have added a changelog fragment under source/<pkg>/changelog.d/ for every touched package (do not edit CHANGELOG.rst or bump extension.toml — CI handles that)
  • I have added my name to the CONTRIBUTORS.md or my name already exists there

@myurasov-nv myurasov-nv changed the title Fix CI base image pulls by not inheriting runner registry credentials Fixes CI base image pulls by not inheriting runner registry credentials Aug 25, 2026
@myurasov-nv

myurasov-nv commented Aug 25, 2026 •

Copy link
Copy Markdown
Member Author

Closing in favour of #7332, which targets the branch where this failure actually occurs.

This PR changed .github/actions/_lib/setup-docker-config on develop to stop inheriting the runner's registry credentials. Two things were wrong with that approach:

  1. The affected code path is not on develop. The reported failure is on a PR based on release/3.0.0-beta2, which pins the public nvcr.io/nvidia/isaac-sim:6.0.1. develop pins an image in a private NGC org that the CI credential is entitled to read, so base image pulls there are unaffected. .github/actions/_lib/ also does not exist on the release branch, where the equivalent logic is inline in ecr-build-push-pull, so this diff could not reach the failing code.
  2. Removing the inherited credential would have regressed fork PRs on develop. With no NGC_API_KEY, that credential is the only nvcr.io credential a fork PR has, and develop's base image is private. Any fork PR whose base build missed the exact-commit ECR cache would have started failing to read the base image manifest.

#7332 keeps the credential and instead verifies it can read the configured base image, retrying without it only when the registry refuses it. That is branch-agnostic, so it is correct for a public or a private base image without the regression.

The compute-deps-hash half of this PR (b7b3615, stop swallowing the digest resolve error) was independently correct and is carried over into #7332, with bounded retries added so a transient registry error costs a retry rather than a job.

@myurasov-nv
myurasov-nv deleted the ci/anonymous-base-image-pull branch September 11, 2026 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant