SwRaw · Oct 21, 2026
@@ -85,8 +85,9 @@ How validation is timed
 
 The timing of validation is controlled by the ``alwaysValidate`` option. By default
 (``alwaysValidate = 0``), validation runs once after all timed iterations complete,
-minimizing overhead during benchmarking. When ``alwaysValidate = 1``, validation is
-performed after every iteration; any detected error immediately stops the run.
+minimizing overhead during benchmarking. When ``alwaysValidate > 0``, validation is
+performed after every iteration; any detected error immediately stops the run. Setting
+``alwaysValidate`` to a negative value disables validation entirely.
 
 .. list-table::
     :header-rows: 1
@@ -95,11 +96,15 @@ performed after every iteration; any detected error immediately stops the run.
       - When
       - Behavior
 
+    * - ``alwaysValidate < 0``
+      - Never
+      - Validation is disabled entirely; ``ValidateAllTransfers`` is never called.
+
     * - ``alwaysValidate = 0`` (default)
       - Once at the end of all iterations
       - ``ValidateAllTransfers`` called after the iteration loop.
 
-    * - ``alwaysValidate = 1``
+    * - ``alwaysValidate > 0``
       - After every timed iteration
       - ``ValidateAllTransfers`` called inside the loop; any error stops the run.
 
@@ -112,8 +117,9 @@ For each transfer and each DST, the following steps are performed:
 
 2. **Get the actual output:**
 
+   - **GPU destination** and ``validateOnDevice = 1``: Compare on the device using a validation kernel; no host copy is made.
    - **CPU destination** or ``validateDirect = 1``: Point directly at the destination memory.
-   - **GPU destination** and ``validateDirect = 0``: Copy destination to a host ``outputBuffer``
+   - **GPU destination**, ``validateOnDevice = 0``, and ``validateDirect = 0``: Copy destination to a host ``outputBuffer``
      via ``hipMemcpy``, then compare against ``outputBuffer``.
 
 3. **Comparison:** Performed using ``memcmp(output, expected, numBytes)``. On mismatch, the code finds the first differing index and returns an error with the index, expected value, and actual value.
@@ -135,7 +141,7 @@ environment variables or in a configuration file.
 
     * - ``alwaysValidate``
       - ``ALWAYS_VALIDATE``
-      - To validate after each iteration, set to ``1``. To validate once at the end, set to ``0``.
+      - To validate after each iteration, set to a value greater than ``0``. To validate once at the end (default), set to ``0``. To disable validation entirely, set to a negative value.
 
     * - ``validateDirect``
       - ``VALIDATE_DIRECT``

@@ -347,6 +347,8 @@ To modify the CMake behavior, use the following environment variables:
   cmake .. -DENABLE_NIC_EXEC=ON -DENABLE_MPI_COMM=ON
   make
 
+.. _build-troubleshooting:
+
 Troubleshooting common build errors
 ====================================
 

@@ -69,6 +69,18 @@ General options
       - Controls how many transfer sizes are sampled per power-of-2 range when ``num_bytes`` is set to ``0``. For example, ``SAMPLING_FACTOR=4`` produces four evenly spaced sizes between each power of 2.
       - ``1``
 
+    * - ``SHOW_PERCENTILES``
+      - Comma-separated percentiles of iteration duration to report. Each value must be between 1 and 99.
+      - —
+
+    * - ``SWEEP_MIN_POW2``
+      - When ``0`` is specified for data size, the starting power-of-two exponent.
+      - ``10``
+
+    * - ``SWEEP_MAX_POW2``
+      - When ``0`` is specified for data size, the ending power-of-two exponent.
+      - ``29``
+
 .. _data-validation-var:
 
 Data and validation options
@@ -225,6 +237,10 @@ GFX and kernel options
       - Preferred XCC per source-by-destination GPU pair, specified as a comma-separated list. Supported on AMD hardware only.
       - —
 
+    * - ``GFX_KERNEL``
+      - Selects the GFX kernel. ``-1`` = auto, ``0`` = force ``GpuReduceKernel``, ``1`` = force ``GpuCopyKernel`` (might error if the transfer is ineligible for this kernel).
+      - ``0``
+
 DMA options
 -----------
 
@@ -239,6 +255,28 @@ DMA options
       - DMA implementation. ``1`` = use ``hsa_amd_async_copy`` for direct HSA control, ``0`` = use ``hipMemcpy`` for the standard HIP path.
       - ``0``
 
+TDM options
+-----------
+
+.. list-table::
+    :header-rows: 1
+
+    * - Environment variable
+      - Description
+      - Default value
+
+    * - ``TDM_BLOCK_ORDER``
+      - How blocks for TDM transfers are ordered. ``0`` = sequential, ``1`` = interleaved.
+      - ``0``
+
+    * - ``TDM_BLOCK_SIZE``
+      - Threads per threadblock for TDM (async tensor) kernels. Must be a multiple of 32.
+      - ``256``
+
+    * - ``TDM_LDS_BYTES``
+      - LDS bytes per workgroup for TDM kernels. ``0`` = device max. Accepts ``K``, ``M``, and ``G`` suffixes.
+      - ``0``
+
 Variable SubExecutor options
 ------------------------------
 
@@ -293,6 +331,18 @@ The following environment variables apply only when NIC support is enabled. To e
       - RoCE version.
       - ``2``
 
+    * - ``NIC_CQ_POLL_BATCH``
+      - Number of completion queue entries to poll per ``ibv_poll_cq`` call.
+      - ``4``
+
+    * - ``NIC_SERVICE_LEVEL``
+      - InfiniBand service level (``sl``) for IB queue pairs. Must be between 0 and 15.
+      - ``0``
+
+    * - ``NIC_TRAFFIC_CLASS``
+      - DSCP / traffic-class byte for the RoCE GRH. Must be between 0 and 255.
+      - ``0``
+
 HIP runtime variables (client-read)
 -------------------------------------
 
@@ -338,6 +388,12 @@ Socket-related variables
     * - ``TB_SINGLE_LOG``
       - When set, only rank 0 produces output. Useful for multinode socket mode.
 
+    * - ``TB_MASTER_IFACE``
+      - When ``TB_MASTER_ADDR`` is unset on rank 0, the optional network interface used for IPv4 address detection (for example, ``eth0``).
+
+    * - ``TB_SEND_USLEEP``
+      - Microseconds to sleep after each socket ``SendData`` call. Default: ``0``. Useful for diagnosing socket stability.
+
 Backend variables
 -----------------
 
@@ -407,6 +463,12 @@ Backend variables
     * - ``TB_FORCE_SINGLE_POD``
       - Forces single pod mode, skipping AMD-SMI and NVML pod queries. This assumes that all GPUs are in the same pod and skips cluster membership API calls.
 
+    * - ``TB_SHOW_ALL_NUMA``
+      - Shows all CPU NUMA nodes, including those with no cores. By default, coreless nodes are hidden, which changes the reported CPU indices.
+
+    * - ``TB_WALLCLOCK_RATE``
+      - Overrides the queried GPU wallclock rate when the query returns ``0``.
+
 HSA runtime variables
 ---------------------
 

@@ -1,6 +1,6 @@
 .. meta::
   :description: Frequently asked questions about TransferBench, covering common errors, warnings, and configuration issues including IOMMU, memory types, and XGMI.
-  :keywords: TransferBench FAQ, TransferBench errors, TransferBench warnings, IOMMU, GPU_MAX_HW_QUEUES, GFX_UNROLL, validation, XGMI, UALoE, memory types
+  :keywords: TransferBench FAQ, TransferBench errors, TransferBench warnings, IOMMU, GPU_MAX_HW_QUEUES, GFX_UNROLL, validation, XGMI, UALoE, memory types, cmath, numactl, cpuset
 
 .. _faq:
 
@@ -15,6 +15,18 @@ Error and warning messages
 
 This section describes common TransferBench error and warning messages and how to resolve them.
 
+Could not find standard C++ header 'cmath'
+--------------------------------------------
+
+This occurs when the standard C++ headers aren't installed. Try installing ``g++-12`` or
+``g++-14`` based on the OS version. For example:
+
+.. code-block:: shell
+
+  apt-get install g++-12
+
+For more build troubleshooting, see :ref:`build-troubleshooting`.
+
 Unexpected mismatch at index
 -----------------------------
 
@@ -154,6 +166,57 @@ Setting ``GPU_MAX_HW_QUEUES=8``:
   ``GPU_MAX_HW_QUEUES=4`` case the CPU-timed aggregate bandwidth is roughly half the
   GPU-timed aggregate, while in the ``GPU_MAX_HW_QUEUES=8`` case the two are much closer.
 
+.. _cpuset-allocation-error:
+
+Cannot allocate memory due to process memory policy/cpuset
+-------------------------------------------------------------
+
+For example:
+
+.. code-block:: text
+
+  [ERROR] ERROR: Transfer 0: SRC 0: CPU 0 on rank 0 cannot allocate memory due to process memory policy/cpuset
+
+This indicates that the memory policy on the node doesn't allow CPU memory allocations on
+a particular NUMA node. Check the current policy:
+
+.. code-block:: shell
+
+  numactl --show
+
+.. code-block:: text
+
+  policy: default
+  preferred node: current
+  physcpubind: 0 1 2 3 4 5 6 7 8 9 10 ... 255
+  cpubind: 1       # <-- 0 is not listed
+  nodebind: 1      # <-- 0 is not listed
+  membind: 1       # <-- 0 is not listed
+  preferred:
+
+If the NUMA node that TransferBench is trying to allocate on isn't listed under
+``cpubind``, ``nodebind``, or ``membind``, the process is restricted from allocating memory
+there.
+
+This can also be caused by a hardware misconfiguration, or by a NUMA node having no RAM
+installed. Check installed memory per node with:
+
+.. code-block:: shell
+
+  numactl --hardware
+
+.. code-block:: text
+
+  available: 11 nodes (0-10)
+  node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 ... 191
+  node 0 size: 0 MB      # <-- No RAM installed on NUMA node 0
+  node 0 free: 0 MB
+  node 1 cpus: 64 65 66 67 68 69 70 71 72 73 74 ... 255
+  node 1 size: 257227 MB
+  node 1 free: 242799 MB
+
+A node with ``size: 0 MB`` has no RAM installed and can't be used for allocations.
+
 Feature questions
 ==================
 

@@ -75,6 +75,10 @@ The following table lists the presets available on TransferBench 1.67.0:
       - Stress tests by running through combinations of transfers.
       - ❌
 
+    * - :ref:`TDM sweep preset (tdmsweep) <tdmsweep>`
+      - Sweeps Tensor Data Mover (TDM) executor options (block size, LDS, order, SubExecutors) for a given TDM transfer.
+      - ✅
+
 .. note::
 
     You can modify a preset using environment variables, which are detailed when running the preset.
@@ -2615,3 +2619,131 @@ Example output
   -------------------┼--------------┼------------┼-------------------┼--------------------
      Aggregate (CPU) │ 295.108 GB/s │  12.735 ms │  3758096384 bytes │ Overhead 0.372 ms
   -------------------┴--------------┴------------┴-------------------┴--------------------
+
+.. _tdmsweep:
+
+TDM sweep preset (tdmsweep)
+=============================
+
+The tdmsweep preset sweeps every parameter that affects Tensor Data Mover (TDM) copy performance for a single user-specified Transfer, and reports the best-performing combination. It mirrors the structure of the ``gfxsweep`` preset.
+
+The TDM executor (``EXE_GPU_TDM``, the ``T`` executor in a Transfer expression) stages HBM -> LDS -> HBM without touching cache. Its throughput is governed by four parameters, and this preset sweeps their full cartesian product:
+
+- **Threadblock size:** ``TDM_BLOCK_SIZE``, threads per block (multiple of 32).
+
+- **LDS staging window:** ``TDM_LDS_BYTES``, bytes per block (``0`` = device max).
+
+- **Threadblock order:** ``TDM_BLOCK_ORDER`` (0=sequential, 1=interleaved, 2=random).
+
+- **Sub-executor count:** number of threadblocks/WGPs engaged.
+
+**Key features:**
+
+- Verifies that a TDM executor exists and that TDM copies are supported on device 0.
+
+- Parses ``TDM_TRANSFER`` into ``NUM_TRANSFERS`` Transfer(s). Every Transfer must use the ``T`` executor.
+
+- Iterates over ``BLOCK_ORDERS`` x ``BLOCKSIZES`` x ``LDS_BYTES``, and for each combination over every value in ``NUM_SUB_EXECS``.
+
+- Prints a matrix of bandwidths: one row per (order, block size, LDS bytes) triple and one column per sub-executor count.
+
+- Reports the best bandwidth per sub-executor count, the overall best combination, and a ready-to-paste command that reproduces it.
+
+- **Automatic timing mode** (``TIMING_MODE=-1``): a single Transfer uses GPU-wallclock timing; multiple Transfers sharing one executor use HIP-event timing; otherwise aggregate CPU timing is used.
+
+**Restrictions:**
+
+- **Requires TDM-capable hardware:** AMD gfx1250 or NVIDIA sm_90+.
+
+- **Hardware gate:** If no TDM executor is present, or ``IsTdmCopySupported(0)`` is false, the preset prints ``[WARN] TDM executor is not supported on this device`` and returns a fatal error.
+
+- **Executor check:** Every parsed Transfer must use the ``T`` executor. Anything else fails with ``[ERROR] tdmsweep preset only works on Transfers that are using the TDM executor``.
+
+- **Rejected combinations are skipped, not fatal:** A combination the library refuses (for example, an LDS window larger than the device maximum) prints ``N/A`` in that cell, and the sweep continues.
+
+- **No successful run:** If every combination fails, the preset reports ``[WARN] No transfers executed successfully`` and returns a fatal error.
+
+**Usage:**
+
+.. code-block:: shell
+
+    ./TransferBench tdmsweep [numBytes]
+
+Sweep the default parameter space for a local GPU 0 TDM copy:
+
+.. code-block:: shell
+
+    ./TransferBench tdmsweep
+
+Sweep a GPU0->GPU1 TDM copy of 256 MB over a reduced search space:
+
+.. code-block:: shell
+
+    TDM_TRANSFER="G0->T0->G1" BLOCKSIZES=256,512 LDS_BYTES=32768,0 ./TransferBench tdmsweep 268435456
+
+Environment variables
+-----------------------
+
+To modify the behavior of the tdmsweep preset, use the following environment variables:
+
+.. list-table::
+    :header-rows: 1
+
+    * - Environment variable
+      - Description
+      - Default value
+
+    * - ``BLOCKSIZES``
+      - Threadblock sizes to sweep. Each value must be a positive multiple of 32.
+      - ``64,128,256,512,1024``
+
+    * - ``BLOCK_ORDERS``
+      - Threadblock orderings to sweep: 0=sequential, 1=interleaved, 2=random. Values outside 0-2 are rejected.
+      - ``0``
+
+    * - ``LDS_BYTES``
+      - LDS staging window sizes (bytes per threadblock) to sweep. ``0`` means the device maximum. Must be >= 0.
+      - ``8192,16384,32768,65536,0``
+
+    * - ``NUM_SUB_EXECS``
+      - Sub-executor counts to sweep; these form the columns of the output matrix. Must not be empty.
+      - ``2,4,8,16,32,64``
+
+    * - ``NUM_TRANSFERS``
+      - Number of Transfers described by ``TDM_TRANSFER``.
+      - ``1``
+
+    * - ``TIMING_MODE``
+      - ``-1``\=auto, ``0``\=aggregate CPU time, ``1``\=executor time (HIP event), ``2``\=transfer time (GPU wallclock).
+      - ``-1``
+
+    * - ``TDM_TRANSFER``
+      - The TDM Transfer to sweep, using :ref:`Transfer definition syntax <transfer-definition-syntax>`. Must use the ``T`` (TDM) executor.
+      - ``G0->T0->G0``
+
+.. note::
+
+    The sweep sets the global ``TDM_BLOCK_ORDER``, ``TDM_BLOCK_SIZE``, and ``TDM_LDS_BYTES`` options on each iteration, so setting those directly has no effect when running the tdmsweep preset — use ``BLOCK_ORDERS``, ``BLOCKSIZES``, and ``LDS_BYTES`` instead.
+
+Example output
+----------------
+
+The sweep prints one row per (BlkO, BlkS, LDSBytes) combination and one column per sub-executor count, followed by a summary row of the best value in each column:
+
+.. code-block:: shell
+
+  BlkO    BlkS    LDSBytes  SE 002  SE 004  SE 008  SE 016  SE 032  SE 064
+   ...     ...     ...       ...     ...     ...     ...     ...     ...
+  BlkO    BlkS    LDSBytes  <best>  <best>  <best>  <best>  <best>  <best>
+  =======================================================================================
+  Highest bandwidth found:  <value> GB/s (GPU wallclock-timed)
+            BlockOrder   :      <n>  [TDM_BLOCK_ORDER=<n>]
+            BlockSize    :      <n>  [TDM_BLOCK_SIZE=<n>]
+            LDS Bytes    :      <n>  [TDM_LDS_BYTES=<n>]
+            NumSubExec   :      <n>
+  Command to run best result:
+  TDM_BLOCK_ORDER=<n> TDM_BLOCK_SIZE=<n> TDM_LDS_BYTES=<n> ./TransferBench cmdline <bytes> "<n> <n> G0->T0->G0"
+
+.. note::
+
+    A capture from real TDM-capable hardware has not been added yet.