Skip to content

fix(sandbox): skip read-only mounts during recursive chown of /sandbox#2341

Open
varshaprasad96 wants to merge 1 commit into
NVIDIA:mainfrom
varshaprasad96:fix/sandbox-erofs-readonly-mounts
Open

fix(sandbox): skip read-only mounts during recursive chown of /sandbox#2341
varshaprasad96 wants to merge 1 commit into
NVIDIA:mainfrom
varshaprasad96:fix/sandbox-erofs-readonly-mounts

Conversation

@varshaprasad96

@varshaprasad96 varshaprasad96 commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Summary

Fix sandbox supervisor crash (EROFS: Read-only file system) when the recursive chown /sandbox encounters read-only submounts. This is common in gVisor-based Kubernetes deployments where read-only volume mounts enforce per-directory immutability (Landlock is unavailable under gVisor).

Related Issue

Closes #2294

Changes

  • Mount-boundary detection: Compare st_dev of each path against the root /sandbox device — paths on a different filesystem are skipped entirely, avoiding the chown call on nested read-only mounts
  • EROFS tolerance: If chown still returns EROFS (e.g. the root mount itself is read-only), the error is logged at debug level and startup continues
  • Refactored chown_sandbox_home: Split into entry-point (symlink guard + root_dev capture) and chown_recursive (inner walk with st_dev and EROFS guards)
  • Added unit tests: chown_recursive_skips_different_device and chown_sandbox_home_does_not_chown_across_device_boundary; strengthened existing symlink rejection test

Testing

  • mise run pre-commit passes (Rust lint/format/clippy clean; unrelated Python proto env issue)
  • Unit tests added/updated — 6 chown-related tests pass
  • Manually verified on a kind cluster with a read-only PVC at /sandbox/readonly-data: pod starts successfully, writable paths owned by sandbox user, read-only mount retains root ownership
  • E2E tests added/updated — Kubernetes e2e coverage is a separate follow-up

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

@copy-pr-bot

copy-pr-bot Bot commented Jul 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

The sandbox supervisor crashed with EROFS when the recursive chown of
/sandbox encountered read-only submounts. This is common in gVisor-based
Kubernetes deployments where read-only volume mounts are the only way to
enforce per-directory immutability (Landlock is unavailable under gVisor).

The fix adds two guards to the ownership walk:

1. Mount-boundary detection via st_dev comparison — paths on a different
   filesystem than /sandbox are skipped entirely, avoiding the chown
   call on nested read-only mounts.

2. EROFS tolerance — if chown still returns EROFS (e.g. the root mount
   itself is read-only), the error is logged at debug level and startup
   continues.

Symlink skipping (already present) is preserved and extracted into the
recursive walker for consistency.

Manually verified on a kind cluster with a read-only PVC mounted at
/sandbox/readonly-data: the pod starts successfully, writable paths are
owned by the sandbox user, and the read-only mount retains root
ownership. Kubernetes e2e test coverage is a separate follow-up.

Closes NVIDIA#2294

Signed-off-by: Varsha Prasad Narsing <varshaprasad96@gmail.com>
@varshaprasad96
varshaprasad96 force-pushed the fix/sandbox-erofs-readonly-mounts branch from 6dadfae to 90fae13 Compare July 17, 2026 06:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Kubernetes sandbox crashes EROFS — recursive chown /sandbox fails on read-only submounts (0.0.82)

1 participant