[YuanTingHsieh]

Fixes #5500.

Part of #5434.

Description

Add opt-in Process task GPU visibility through a static, administrator-owned gpu_devices list. A worker requesting N GPUs sees the first N configured IDs; CPU task workers and task CJs have empty CUDA visibility. This does not reserve GPU memory/compute, exclude unrelated processes, or schedule between CJs. Existing ResourceManager contracts and implementations are unchanged.

The implementation incorporates merged #5490 / main d05110ab4570e941a04355cfc68710faaba41c87. Portable resource_spec describes task compute, including @default and explicit site-zero overrides; independent cj_resource_spec describes coordinator admission. Consistent legacy Process GPU counts remain supported. Unenforceable Process CPU/memory/custom claims fail before launch.

The mode/placement revision corrects documentation and configuration guidance rather than changing worker placement. The shared runtime already executes ordinary Python Client API scripts directly inside the disposable worker. The site selects Process, Kubernetes or the separately developed Slurm backend; applications choose the same job/task lifetime model.

Client API configuration	Job lifetime	Task lifetime
Default ScriptRunner/recipe, in_process	Runs inside resident CJ.	Runs directly inside the disposable worker, not CJ. Supported script path.
external_process, launch_once=True	Existing persistent owned trainer.	Rejected by shared validation on every backend.
external_process, launch_once=False	Existing fresh trainer per task.	Rejected by the same shared validation.
attach	Existing externally owned trainer/session.	Unsupported without a separate ownership contract.

Job-lifetime external_process with launch_once=False already gives each task a fresh interpreter. Task lifetime adds durable staged output and confirmed settlement of the entire owned compute scope before CJ publication. Saved-result retransmission needs no live trainer; local completion, server receipt and algorithm acceptance remain separate. Uncertain settlement retains ownership and fences new work.

The Process guide now includes a minimal default recipe, the mode matrix, contention assumptions and startup/model/optimizer reload costs. Resident job mode is preferable when retaining an expensive model, environment or live session matters more than releasing compute between tasks. The validation error explains that in_process means inside the disposable worker and directs Python scripts to existing ScriptRunner/recipe defaults. Unsupported commands are neither rewritten into scripts nor silently moved to job lifetime.

Shared follow-up under #5434: arbitrary commands, separate environments and torchrun need one task-worker command path across Process/Kubernetes/Slurm. It must preserve argv and allocation/environment inheritance, own and cancel the complete process tree, stage durable results and checkpoints, and prove settlement before publication/retry. This is a parity gap, not a fundamental incompatibility. It is outside this GPU PR; removing guards or adding a Process-only command backend would not supply the contract. Attach stays externally owned/resident pending a separate explicit design.

Docker task PR #5514 is closed as deferred; existing Docker job support remains. This PR does not change Kubernetes/Slurm branches or duplicate shared fixes #5561 (published-handle/backend settlement), #5562 (HOCON admission), or #5570 (worker communication bootstrap). Those fixes need their own review and subsequent integration qualification.

The latest test audit moves executable qualification helpers into tests/integration_test/tools/process_gpu/, keeps only the submitted workload in data/process_gpu/, places harness unit tests under tests/unit_test/integration_test/, and moves Process job-launcher tests to the directory matching their production module. A shared probe helper removes the command tool's dependency on a pytest entry point. Atomic probe records plus bounded partial-JSON reads eliminate a demonstrated reader race; the hook unit test also releases its paused thread on assertion failure. The CUDA scope case now explicitly names its local test publisher instead of implying that it exercises CJ transport.

The qualification exporter now declares the built-in linear model through the supported recipe config rather than a live PyTorch object. This reduces exported jobs from about 43 MB of recursively bundled third-party sources to 7.7 KB while retaining the same regression target and convergence assertions. Instructions pin imports to the frozen candidate and invoke tools as modules.

Types of changes

• Non-breaking change (opt-in feature and clearer configuration guidance).
• New tests added to cover the changes.
• Quick tests passed locally by running the complete ./runtest.sh suite.
• In-line docstrings updated.
• Documentation updated.

Testing

Signed current head: 9702a255122120a05886b9fcd4ec6a572402a469; tree: f8558b59f14d2c9dcf5074ec2ef9cf2473b61a47.

Current-head checks:

• 716 passed, 1 platform skip across every unit-test module changed by this PR. This includes GPU visibility/policy, real CPU worker failures and recovery, supervisor settlement and saved-result retry, relocated Process job-launcher tests, harness activation/site isolation, bounded partial-record reads, and job/task export without vendoring PyTorch.
• 2 real two-site CPU simulator controls passed using the corrected qualification fixture, one per lifetime. Both produced final weights approximately [2, -3] and per-site final MSE 3.20e-13; exported jobs were 7,668 and 7,747 bytes.
• The six real-CUDA cases are discovered and skip without an explicit device assignment. This verifies the guard, not CUDA execution.
• The standalone CUDA/reuse helpers import without pytest or an integration-test entry module; the generated CUDA probe script compiles.
• Full project ./runtest.sh -s --skip-install and git diff --check: passed.
• Documentation files are unchanged by this audit. The preceding SKIP_API_DOCS=1 Sphinx build passed with 133 existing warnings and no added warning text; the mode matrix, minimal recipe and worker-placement explanation rendered in HTML.
• Local audit findings above are addressed. Current-head CI is pending.

Earlier evidence remains tied to its source:

• Preceding head 0874fedff9caefc45606d33faa8c14b78ed84d38: 525 passing focused unit cases and four real two-site CPU job/task controls in simulator and provisioned POC, plus full style and the documentation build. These are separate from the revised fixture runs above.

• Previous head b811786e7dd9b0411f7d2025983156a1d3c2a739: 3,230 passing unit cases, six CPU E2E cases, two resident-CJ recovery cases, full style and 23 successful CI checks; independent integration review had zero unresolved findings.

• On that exact earlier head, Colossus passed six real-CUDA scope cases, including real OOM followed by healthy work, exception/abrupt exit, cancellation/timeout descendant cleanup and device reuse before prior publication finished. The launcher, supervisor, worker execution and GPU-policy code are unchanged in this revision; those runs still qualify their recorded source, not a completed new-head product campaign.

• Earlier full-fixture CPU/job deployment expired before a CJ started. A thinner diagnostic job completed, but does not qualify the full product fixture. Setup failures remain separate from product results.

Still required: exact-head Colossus CPU/GPU job/task product controls, real CUDA through the deployed CJ/worker path, real OOM followed by healthy work, cancellation/descendant cleanup, actual CJ delayed-publication ordering and saved-result/lost-receipt retry. Qualification after integrating the separate shared fixes must identify that integrated tree explicitly. Use existing owned capacity without duplicate submissions or new leases.

Risk

Concurrent CJs with the same static assignment may contend with each other and unrelated GPU users. Visibility is not exclusivity, a memory quota or a GPU health guarantee. Process cannot enforce CPU/memory limits and does not reset GPUs or repair drivers.

Supported Process cleanup assumes trusted code in one owned POSIX process group with sole leader reaping. Detached descendants and external reapers are outside that contract. Once task-runtime evidence exists, automatic replacement after abrupt CJ death remains conservatively blocked: CJ group exit cannot prove settlement of separately grouped task workers. A living CJ can accept fresh tasks after confirmed worker cleanup, including settled OOM. Configure finite worker timeouts.

Canonical site configuration supplies policy, authorization and logging. Deployments relying on recursively discovered nested resources files must move intended policy to the canonical site configuration. Existing resident job behavior remains unchanged.

How to revert

Stop new work and confirm owned worker settlement, remove gpu_devices, restore ordinary resident-job resource configuration if needed, and revert the feature. Preserve failure/runtime evidence and historical unreleased-profile ledgers; do not remove evidence to bypass an unresolved cleanup fence.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 91.75258% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 72.23%. Comparing base (d05110a) to head (9702a25).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...vflare/app_common/job_launcher/process_launcher.py	80.00%	2 Missing ⚠️
...common/resource_consumers/gpu_resource_consumer.py	0.00%	2 Missing ⚠️
...app_common/job_launcher/client_process_launcher.py	95.83%	1 Missing ⚠️
nvflare/app_common/logging/constants.py	50.00%	1 Missing ⚠️
nvflare/fuel/utils/config_service.py	87.50%	1 Missing ⚠️
...vflare/private/fed/client/task_execution_policy.py	95.65%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5486      +/-   ##
==========================================
+ Coverage   72.14%   72.23%   +0.08%     
==========================================
  Files        1052     1052              
  Lines      113844   113917      +73     
==========================================
+ Hits        82131    82286     +155     
+ Misses      31713    31631      -82     

Flag	Coverage Δ
unit-tests	72.23% <91.75%> (+0.08%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

[greptile-apps]

Confidence Score: 4/5

[Medium impact] The changes affect Process task execution and canonical site resource settings.

The null-launcher crash still blocks merging; no new actionable issue was found in the qualification-tool changes.

Findings

1.  Null launcher blocks jobs ▶

Summary

This PR adds opt-in GPU visibility for Process task workers and keeps their CJ GPU-hidden. It also blocks automatic CJ replacement when retained task evidence cannot establish worker cleanup.

Changes since the previous review focus on qualification tools:

• Process task workers see the first configured GPU IDs their task requests.
• Default Python scripts run inside disposable task workers for task lifetime.
• Process client jobs stop restarting automatically after task workers may exist.
• Canonical site settings load from JSON, HOCON, and YAML files.

_{Reviews (2) · Last reviewed commit: "Correct Process GPU test layout and qual..." · Reviewed by Greptile}

[greptile-apps]

Null launcher blocks jobs

When site configuration sets task_execution.launcher to null, prepare_environment calls .get() on None and crashes. TaskExecutionPolicy.from_config accepts this setting and selects the default ProcessTaskLauncher. Since every Process CJ launch now calls this hook, ordinary resident jobs also fail before starting. Normalize the null launcher to an empty dictionary before checking gpu_devices.

Suggested change

	launcher = data.get("task_execution", {}).get("launcher", {})
	launcher = data.get("task_execution", {}).get("launcher") or {}

[YuanTingHsieh]

Confirmed and reproduced. An explicit null launcher is accepted by TaskExecutionPolicy as the default CPU Process launcher, but the new CJ environment hook dereferenced it as a dictionary.

The local fix normalizes the null value before testing for an opt-in gpu_devices assignment. Added 12 regression cases covering null/omitted launchers in JSON, YAML and HOCON for resident jobs and normalized CPU task jobs. All six explicit-null cases fail before the fix; the focused launch/policy suite passes 93 cases afterward, and the full project style check passes. Existing CJ environment values and resident GPU admission remain unchanged without the opt-in.

The fix is in signed commit 15d5a987052b302b6335ddd1eb4c70109584b40b, pending publication to this PR. Leaving this thread unresolved until that commit is available for review.