Add SRTSlurm workload integration

conf/experimental/srt_slurm/agentx_dse/README.md

@@ -0,0 +1,63 @@
+# AgentX DSE smoke test
+
+This example provisions pinned SRT Slurm and InferenceX repositories, downloads
+Qwen3-0.6B through CloudAI's model installer, then runs
+SGLang with Qwen3-0.6B on one serving GPU of a four-GPU GB200 node. Grid search
+tries context lengths 2048 and 4096 and maximizes `p90_intvty`.
+
+The client replays two public fixtures from InferenceX's pinned AIPerf submodule
+for 60 seconds. It uses `--unsafe-override`; this is a correctness smoke test,
+not a canonical SemiAnalysis submission or a performance comparison.
+
+## Configure
+
+All `/shared/...` paths and the `gpu` partition are placeholders:
+
+- Copy the common [Slurm system template](../../../common/system/example_slurm_cluster.toml)
+  to your own `system.toml`. Adapt its partition, node groups, GPU counts, and
+  environment variables to your cluster.
+- Put `scenario.toml`, `srtslurm.yaml`, `recipe.yaml`, and `agentx_client.sh` in
+  `/shared/cloudai/example`, or update the scenario paths and the corresponding
+  `/agentx-test` mount together.
+- Set `install_path = "/shared/cloudai/install"` and
+  `output_path = "/shared/cloudai/results"` in your system config. CloudAI exports
+  its absolute installation root as `CLOUDAI_INSTALL_FOLDER`; SRT expands that
+  variable in the InferenceX mount in `srtslurm.yaml`. SRT derives its own checkout
+  root, so changing `install_path` does not require editing either repo path.
+- Set `hf_home_path = "/shared/huggingface"` in your system config. The scenario
+  declares Qwen3-0.6B and its commit in `hf_models`; `cloudai install` downloads that revision
+  into that cache. In `srtslurm.yaml`, `model_paths.qwen` must point to the
+  snapshot under the same cache root (`hf_home_path/hub/models--Qwen--Qwen3-0.6B/snapshots/<revision>`).
+  The snapshot revision must match the scenario's model revision. Keep the
+  accompanying `blobs` directory mounted at `/blobs`: SRT mounts the snapshot at
+  `/model`, where its `../../blobs` symlinks need that target. Both YAML paths must
+  use the same `hf_home_path` cache root. CloudAI does not synchronize these paths;
+  the cache must be accessible on the submission and compute nodes.
+- Provide a compatible SGLang container at `/shared/images/sglang.sqsh`. SRT
+  mounts the model snapshot at `/model`; the client uses its tokenizer offline.
+- Choose your partition in both system and SRT cluster configs. CloudAI uses
+  its partition for the setup allocation; SRT generates and submits benchmark
+  jobs using its recipe and cluster config, and Slurm schedules them.
+- Configure the Slurm CLI environment and any site-required container settings.
+  Dependencies require network access. uv caches and managed Python downloads
+  use node-local `/tmp`; other application caches still need writable storage.
+
+The `srt-slurm` repository entry supplies srtctl, regardless of entry order.
+The InferenceX entry installs its pinned submodules. The `/infmax-workspace` mount
+connects that checkout to the custom benchmark client. If the InferenceX commit
+changes, update the checkout basename in `srtslurm.yaml` to match.
+
+## Run
+
+After adapting the paths, run from the directory containing the input files.
+Pass the path to your adapted copy of the common system template:
+
+```bash
+mkdir -p /shared/cloudai/install /shared/cloudai/results /shared/huggingface
+cloudai install --system-config /path/to/system.toml --test-scenario scenario.toml
+cloudai run --system-config /path/to/system.toml --test-scenario scenario.toml
+```
+
+Each trial writes its SRT job artifacts and `srt_slurm_results.csv`. The DSE
+iteration writes `trajectory.csv` and the selected `agentx.toml` configuration;
+scenario and DSE HTML reports summarize the results.

conf/experimental/srt_slurm/agentx_dse/agentx_client.sh

@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# AgentX/DSE smoke-test client: replay two pinned AIPerf fixtures for 60 seconds
+# using InferenceX benchmark helpers, avoiding the full dataset download.
+# The resulting AgentX artifacts validate CloudAI result extraction and DSE.
+set -euo pipefail
+export INFMAX_CONTAINER_WORKSPACE=/infmax-workspace
+source "$INFMAX_CONTAINER_WORKSPACE/benchmarks/benchmark_lib.sh"
+install_agentic_deps
+
+# Use the mounted model tokenizer without fetching it from the Hub.
+export HF_HUB_CACHE="$(mktemp -d)"
+export HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1
+tokenizer_cache="$HF_HUB_CACHE/models--Qwen--Qwen3-0.6B"
+revision=c1899de289a04d12100db370d81485cdf75e47ca
+mkdir -p "$tokenizer_cache/refs" "$tokenizer_cache/snapshots"
+ln -s /model "$tokenizer_cache/snapshots/$revision"
+printf '%s' "$revision" > "$tokenizer_cache/refs/main"
+
+# Replay the same two pinned public test fixtures in each trial.
+traces="$(mktemp -d)"
+cp "$AIPERF_DIR/tests/fixtures/weka_traces/simple.json" "$traces/"
+cp "$AIPERF_DIR/tests/fixtures/weka_traces/one_subagent.json" "$traces/"
+export MODEL=Qwen/Qwen3-0.6B MODEL_PREFIX=qwen3 FRAMEWORK=sglang PRECISION=bf16
+export DURATION=60 CONC=1 PORT=8000 RESULT_FILENAME=qwen_c1
+export IS_MULTINODE=false DISAGG=false KV_OFFLOADING=none KV_OFFLOAD_BACKEND=''
+export AIPERF_WARMUP_REQUESTS_PER_LANE=1 AIPERF_EXPERIMENTAL_FAST=0
+export AIPERF_EXTRA_INPUTS=ignore_eos:true
+export AGENTIC_OUTPUT_DIR=/logs/agentic
+result_dir="$AGENTIC_OUTPUT_DIR/conc_1"
+mkdir -p "$result_dir"
+TRACE_SOURCE_FLAG="--input-file $traces --custom-dataset-type weka_trace"
+build_replay_cmd "$result_dir"
+REPLAY_CMD="${REPLAY_CMD/--trajectory-start-min-ratio 0.25/--trajectory-start-min-ratio 0.0}"
+REPLAY_CMD="${REPLAY_CMD/--trajectory-start-max-ratio 0.75/--trajectory-start-max-ratio 0.0}"
+REPLAY_CMD+=' --tokenizer Qwen/Qwen3-0.6B'
+run_agentic_replay_and_write_outputs "$result_dir"

conf/experimental/srt_slurm/agentx_dse/recipe.yaml

@@ -0,0 +1,26 @@
+# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+schema: 2
+name: qwen-agentx
+model: {path: qwen, container: sglang, precision: bf16}
+resources: {gpu_type: gb200, gpus_per_node: 4}
+slurm: {time_limit: '00:20:00'}
+frontend: {type: sglang, enable_multiple_frontends: false}
+engine: sglang
+roles:
+  agg:
+    nodes: 1
+    workers: 1
+    gpus: 1
+    args:
+      tensor-parallel-size: 1
+      context-length: 2048
+      mem-fraction-static: 0.2
+      disable-cuda-graph: true
+      served-model-name: Qwen/Qwen3-0.6B
+health_check: {max_attempts: 60, interval_seconds: 5}
+benchmark:
+  type: custom
+  command: bash /agentx-test/agentx_client.sh

conf/experimental/srt_slurm/agentx_dse/scenario.toml

@@ -0,0 +1,53 @@
+# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name = "agentx-dse"
+
+[[Tests]]
+id = "agentx"
+name = "qwen_agentx"
+description = "Two-step AgentX DSE correctness smoke test"
+test_template_name = "SRTSlurm"
+hf_models = [
+  { model_name = "Qwen/Qwen3-0.6B", revision = "c1899de289a04d12100db370d81485cdf75e47ca" },
+]
+agent = "grid_search"
+agent_steps = 2
+agent_metrics = ["p90_intvty"]
+agent_reward_function = "identity"
+
+  [[Tests.git_repos]]
+  url = "https://github.com/NVIDIA/srt-slurm.git"
+  commit = "098e15aca8504d4747c6f62c7a845e178b7f4b10"
+
+  [[Tests.git_repos]]
+  url = "https://github.com/SemiAnalysisAI/InferenceX.git"
+  commit = "f8dff19a6898e3ee0dd95232f867151d4d2d6392"
+  init_submodules = true
+
+  [Tests.cmd_args]
+  recipe_path = "/shared/cloudai/example/recipe.yaml"
+  cluster_config = "/shared/cloudai/example/srtslurm.yaml"
+  result_format = "agentx"
+
+    [Tests.cmd_args.recipe_overrides.roles.agg.args]
+    context-length = [2048, 4096]
+
+  [Tests.extra_env_vars]
+
+  # Keep host-side uv work off shared home storage.
+  UV_CACHE_DIR = "/tmp/cloudai-agentx-uv-cache"
+  UV_PYTHON_INSTALL_DIR = "/tmp/cloudai-agentx-python"

conf/experimental/srt_slurm/agentx_dse/srtslurm.yaml

@@ -0,0 +1,15 @@
+# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+default_partition: gpu
+gpus_per_node: 4
+# Match system.toml hf_home_path and the model revision pinned in scenario.toml.
+model_paths:
+  qwen: /shared/huggingface/hub/models--Qwen--Qwen3-0.6B/snapshots/c1899de289a04d12100db370d81485cdf75e47ca
+containers: {sglang: /shared/images/sglang.sqsh}
+default_mounts:
+  # Snapshot links under /model resolve ../../blobs to /blobs inside the container.
+  /shared/huggingface/hub/models--Qwen--Qwen3-0.6B/blobs: /blobs
+  ${CLOUDAI_INSTALL_FOLDER}/InferenceX__f8dff19a6898e3ee0dd95232f867151d4d2d6392: /infmax-workspace
+  /shared/cloudai/example: /agentx-test

conf/experimental/srt_slurm/recipe.yaml

@@ -0,0 +1,23 @@
+schema: 2
+name: sglang-example
+model:
+  path: hf:Qwen/Qwen3-0.6B
+  container: docker.io/lmsysorg/sglang:v0.5.9
+resources:
+  gpu_type: h100
+  gpus_per_node: 8
+frontend:
+  type: sglang
+engine: sglang
+roles:
+  agg:
+    nodes: 1
+    workers: 1
+    gpus: 1
+    args:
+      tensor-parallel-size: 1
+benchmark:
+  type: sa-bench
+  isl: 128
+  osl: 128
+  concurrencies: "1"

conf/experimental/srt_slurm/scenario.toml

@@ -0,0 +1,41 @@
+# SPDX-FileCopyrightText: NVIDIA CORPORATION & AFFILIATES
+# Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name = "srt_slurm"
+
+[[Tests]]
+id = "srt.sglang"
+# Benchmark node allocation is defined by recipe.yaml, not this value.
+num_nodes = 1
+
+name = "srt_sglang"
+description = "SGLang through an existing srt-slurm recipe"
+test_template_name = "SRTSlurm"
+agent = "grid_search"
+agent_metrics = ["tput_per_gpu"]
+
+  [[Tests.git_repos]]
+  url = "https://github.com/NVIDIA/srt-slurm.git"
+  commit = "v2.2.1"
+
+  [Tests.cmd_args]
+  recipe_path = "conf/experimental/srt_slurm/recipe.yaml"
+  # Owns benchmark scheduling defaults; CloudAI allocation settings are not merged.
+  cluster_config = "conf/experimental/srt_slurm/srtslurm.yaml"
+  result_format = "sa-bench"
+
+    [Tests.cmd_args.recipe_overrides.benchmark]
+    concurrencies = ["1", "4", "16"]

conf/experimental/srt_slurm/srtslurm.yaml

@@ -0,0 +1,6 @@
+# Benchmark scheduling defaults belong here and in the recipe.
+# CloudAI system/scenario allocation settings are not forwarded to srt-slurm.
+# Adapt these Slurm defaults to your cluster. srtctl_root is omitted so the
+# pinned srtctl installation derives its own checkout path.
+default_partition: gpu
+gpus_per_node: 8

doc/workloads/index.rst

@@ -30,6 +30,7 @@ Available Workloads
    ":doc:`nixl_perftest`", "✅", "❌", "❌", "❌"
    ":doc:`sleep`", "✅", "✅", "❌", "✅"
    ":doc:`sglang`", "✅", "❌", "❌", "❌"
+   ":doc:`srt_slurm`", "✅", "❌", "❌", "❌"
    ":doc:`slurm_container`", "✅", "❌", "❌", "❌"
    "Triton Inference", "✅", "❌", "❌", "❌"
    ":doc:`ucc`", "✅", "❌", "❌", "❌"