From 6e2e6ab880189419ef1c1477ada0808117567159 Mon Sep 17 00:00:00 2001 From: srawat <120587655+SwRaw@users.noreply.github.com> Date: Wed, 21 Oct 2026 23:04:07 +0530 Subject: [PATCH] Doc updates --- .../transferbench-data-validation.rst | 16 ++- docs/install/build_from_source.rst | 2 + docs/reference/environment-variables.rst | 62 ++++++++ docs/reference/faq.rst | 65 ++++++++- docs/reference/presets.rst | 132 ++++++++++++++++++ docs/reference/transfer-definition-syntax.rst | 10 ++ docs/what-is-transferbench.rst | 4 +- 7 files changed, 284 insertions(+), 7 deletions(-) diff --git a/docs/conceptual/transferbench-data-validation.rst b/docs/conceptual/transferbench-data-validation.rst index 4d60e749..134f4eda 100644 --- a/docs/conceptual/transferbench-data-validation.rst +++ b/docs/conceptual/transferbench-data-validation.rst @@ -85,8 +85,9 @@ How validation is timed The timing of validation is controlled by the ``alwaysValidate`` option. By default (``alwaysValidate = 0``), validation runs once after all timed iterations complete, -minimizing overhead during benchmarking. When ``alwaysValidate = 1``, validation is -performed after every iteration; any detected error immediately stops the run. +minimizing overhead during benchmarking. When ``alwaysValidate > 0``, validation is +performed after every iteration; any detected error immediately stops the run. Setting +``alwaysValidate`` to a negative value disables validation entirely. .. list-table:: :header-rows: 1 @@ -95,11 +96,15 @@ performed after every iteration; any detected error immediately stops the run. - When - Behavior + * - ``alwaysValidate < 0`` + - Never + - Validation is disabled entirely; ``ValidateAllTransfers`` is never called. + * - ``alwaysValidate = 0`` (default) - Once at the end of all iterations - ``ValidateAllTransfers`` called after the iteration loop. - * - ``alwaysValidate = 1`` + * - ``alwaysValidate > 0`` - After every timed iteration - ``ValidateAllTransfers`` called inside the loop; any error stops the run. @@ -112,8 +117,9 @@ For each transfer and each DST, the following steps are performed: 2. **Get the actual output:** + - **GPU destination** and ``validateOnDevice = 1``: Compare on the device using a validation kernel; no host copy is made. - **CPU destination** or ``validateDirect = 1``: Point directly at the destination memory. - - **GPU destination** and ``validateDirect = 0``: Copy destination to a host ``outputBuffer`` + - **GPU destination**, ``validateOnDevice = 0``, and ``validateDirect = 0``: Copy destination to a host ``outputBuffer`` via ``hipMemcpy``, then compare against ``outputBuffer``. 3. **Comparison:** Performed using ``memcmp(output, expected, numBytes)``. On mismatch, the code finds the first differing index and returns an error with the index, expected value, and actual value. @@ -135,7 +141,7 @@ environment variables or in a configuration file. * - ``alwaysValidate`` - ``ALWAYS_VALIDATE`` - - To validate after each iteration, set to ``1``. To validate once at the end, set to ``0``. + - To validate after each iteration, set to a value greater than ``0``. To validate once at the end (default), set to ``0``. To disable validation entirely, set to a negative value. * - ``validateDirect`` - ``VALIDATE_DIRECT`` diff --git a/docs/install/build_from_source.rst b/docs/install/build_from_source.rst index f80d6570..1f5b77d0 100644 --- a/docs/install/build_from_source.rst +++ b/docs/install/build_from_source.rst @@ -347,6 +347,8 @@ To modify the CMake behavior, use the following environment variables: cmake .. -DENABLE_NIC_EXEC=ON -DENABLE_MPI_COMM=ON make +.. _build-troubleshooting: + Troubleshooting common build errors ==================================== diff --git a/docs/reference/environment-variables.rst b/docs/reference/environment-variables.rst index 7228e6e9..78b14b91 100644 --- a/docs/reference/environment-variables.rst +++ b/docs/reference/environment-variables.rst @@ -69,6 +69,18 @@ General options - Controls how many transfer sizes are sampled per power-of-2 range when ``num_bytes`` is set to ``0``. For example, ``SAMPLING_FACTOR=4`` produces four evenly spaced sizes between each power of 2. - ``1`` + * - ``SHOW_PERCENTILES`` + - Comma-separated percentiles of iteration duration to report. Each value must be between 1 and 99. + - — + + * - ``SWEEP_MIN_POW2`` + - When ``0`` is specified for data size, the starting power-of-two exponent. + - ``10`` + + * - ``SWEEP_MAX_POW2`` + - When ``0`` is specified for data size, the ending power-of-two exponent. + - ``29`` + .. _data-validation-var: Data and validation options @@ -225,6 +237,10 @@ GFX and kernel options - Preferred XCC per source-by-destination GPU pair, specified as a comma-separated list. Supported on AMD hardware only. - — + * - ``GFX_KERNEL`` + - Selects the GFX kernel. ``-1`` = auto, ``0`` = force ``GpuReduceKernel``, ``1`` = force ``GpuCopyKernel`` (might error if the transfer is ineligible for this kernel). + - ``0`` + DMA options ----------- @@ -239,6 +255,28 @@ DMA options - DMA implementation. ``1`` = use ``hsa_amd_async_copy`` for direct HSA control, ``0`` = use ``hipMemcpy`` for the standard HIP path. - ``0`` +TDM options +----------- + +.. list-table:: + :header-rows: 1 + + * - Environment variable + - Description + - Default value + + * - ``TDM_BLOCK_ORDER`` + - How blocks for TDM transfers are ordered. ``0`` = sequential, ``1`` = interleaved. + - ``0`` + + * - ``TDM_BLOCK_SIZE`` + - Threads per threadblock for TDM (async tensor) kernels. Must be a multiple of 32. + - ``256`` + + * - ``TDM_LDS_BYTES`` + - LDS bytes per workgroup for TDM kernels. ``0`` = device max. Accepts ``K``, ``M``, and ``G`` suffixes. + - ``0`` + Variable SubExecutor options ------------------------------ @@ -293,6 +331,18 @@ The following environment variables apply only when NIC support is enabled. To e - RoCE version. - ``2`` + * - ``NIC_CQ_POLL_BATCH`` + - Number of completion queue entries to poll per ``ibv_poll_cq`` call. + - ``4`` + + * - ``NIC_SERVICE_LEVEL`` + - InfiniBand service level (``sl``) for IB queue pairs. Must be between 0 and 15. + - ``0`` + + * - ``NIC_TRAFFIC_CLASS`` + - DSCP / traffic-class byte for the RoCE GRH. Must be between 0 and 255. + - ``0`` + HIP runtime variables (client-read) ------------------------------------- @@ -338,6 +388,12 @@ Socket-related variables * - ``TB_SINGLE_LOG`` - When set, only rank 0 produces output. Useful for multinode socket mode. + * - ``TB_MASTER_IFACE`` + - When ``TB_MASTER_ADDR`` is unset on rank 0, the optional network interface used for IPv4 address detection (for example, ``eth0``). + + * - ``TB_SEND_USLEEP`` + - Microseconds to sleep after each socket ``SendData`` call. Default: ``0``. Useful for diagnosing socket stability. + Backend variables ----------------- @@ -407,6 +463,12 @@ Backend variables * - ``TB_FORCE_SINGLE_POD`` - Forces single pod mode, skipping AMD-SMI and NVML pod queries. This assumes that all GPUs are in the same pod and skips cluster membership API calls. + * - ``TB_SHOW_ALL_NUMA`` + - Shows all CPU NUMA nodes, including those with no cores. By default, coreless nodes are hidden, which changes the reported CPU indices. + + * - ``TB_WALLCLOCK_RATE`` + - Overrides the queried GPU wallclock rate when the query returns ``0``. + HSA runtime variables --------------------- diff --git a/docs/reference/faq.rst b/docs/reference/faq.rst index 362691d0..9bc1e1f9 100644 --- a/docs/reference/faq.rst +++ b/docs/reference/faq.rst @@ -1,6 +1,6 @@ .. meta:: :description: Frequently asked questions about TransferBench, covering common errors, warnings, and configuration issues including IOMMU, memory types, and XGMI. - :keywords: TransferBench FAQ, TransferBench errors, TransferBench warnings, IOMMU, GPU_MAX_HW_QUEUES, GFX_UNROLL, validation, XGMI, UALoE, memory types + :keywords: TransferBench FAQ, TransferBench errors, TransferBench warnings, IOMMU, GPU_MAX_HW_QUEUES, GFX_UNROLL, validation, XGMI, UALoE, memory types, cmath, numactl, cpuset .. _faq: @@ -15,6 +15,18 @@ Error and warning messages This section describes common TransferBench error and warning messages and how to resolve them. +Could not find standard C++ header 'cmath' +-------------------------------------------- + +This occurs when the standard C++ headers aren't installed. Try installing ``g++-12`` or +``g++-14`` based on the OS version. For example: + +.. code-block:: shell + + apt-get install g++-12 + +For more build troubleshooting, see :ref:`build-troubleshooting`. + Unexpected mismatch at index ----------------------------- @@ -154,6 +166,57 @@ Setting ``GPU_MAX_HW_QUEUES=8``: ``GPU_MAX_HW_QUEUES=4`` case the CPU-timed aggregate bandwidth is roughly half the GPU-timed aggregate, while in the ``GPU_MAX_HW_QUEUES=8`` case the two are much closer. +.. _cpuset-allocation-error: + +Cannot allocate memory due to process memory policy/cpuset +------------------------------------------------------------- + +For example: + +.. code-block:: text + + [ERROR] ERROR: Transfer 0: SRC 0: CPU 0 on rank 0 cannot allocate memory due to process memory policy/cpuset + +This indicates that the memory policy on the node doesn't allow CPU memory allocations on +a particular NUMA node. Check the current policy: + +.. code-block:: shell + + numactl --show + +.. code-block:: text + + policy: default + preferred node: current + physcpubind: 0 1 2 3 4 5 6 7 8 9 10 ... 255 + cpubind: 1 # <-- 0 is not listed + nodebind: 1 # <-- 0 is not listed + membind: 1 # <-- 0 is not listed + preferred: + +If the NUMA node that TransferBench is trying to allocate on isn't listed under +``cpubind``, ``nodebind``, or ``membind``, the process is restricted from allocating memory +there. + +This can also be caused by a hardware misconfiguration, or by a NUMA node having no RAM +installed. Check installed memory per node with: + +.. code-block:: shell + + numactl --hardware + +.. code-block:: text + + available: 11 nodes (0-10) + node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 ... 191 + node 0 size: 0 MB # <-- No RAM installed on NUMA node 0 + node 0 free: 0 MB + node 1 cpus: 64 65 66 67 68 69 70 71 72 73 74 ... 255 + node 1 size: 257227 MB + node 1 free: 242799 MB + +A node with ``size: 0 MB`` has no RAM installed and can't be used for allocations. + Feature questions ================== diff --git a/docs/reference/presets.rst b/docs/reference/presets.rst index 285371d8..0e74353e 100644 --- a/docs/reference/presets.rst +++ b/docs/reference/presets.rst @@ -75,6 +75,10 @@ The following table lists the presets available on TransferBench 1.67.0: - Stress tests by running through combinations of transfers. - ❌ + * - :ref:`TDM sweep preset (tdmsweep) ` + - Sweeps Tensor Data Mover (TDM) executor options (block size, LDS, order, SubExecutors) for a given TDM transfer. + - ✅ + .. note:: You can modify a preset using environment variables, which are detailed when running the preset. @@ -2615,3 +2619,131 @@ Example output -------------------┼--------------┼------------┼-------------------┼-------------------- Aggregate (CPU) │ 295.108 GB/s │ 12.735 ms │ 3758096384 bytes │ Overhead 0.372 ms -------------------┴--------------┴------------┴-------------------┴-------------------- + +.. _tdmsweep: + +TDM sweep preset (tdmsweep) +============================= + +The tdmsweep preset sweeps every parameter that affects Tensor Data Mover (TDM) copy performance for a single user-specified Transfer, and reports the best-performing combination. It mirrors the structure of the ``gfxsweep`` preset. + +The TDM executor (``EXE_GPU_TDM``, the ``T`` executor in a Transfer expression) stages HBM -> LDS -> HBM without touching cache. Its throughput is governed by four parameters, and this preset sweeps their full cartesian product: + +- **Threadblock size:** ``TDM_BLOCK_SIZE``, threads per block (multiple of 32). + +- **LDS staging window:** ``TDM_LDS_BYTES``, bytes per block (``0`` = device max). + +- **Threadblock order:** ``TDM_BLOCK_ORDER`` (0=sequential, 1=interleaved, 2=random). + +- **Sub-executor count:** number of threadblocks/WGPs engaged. + +**Key features:** + +- Verifies that a TDM executor exists and that TDM copies are supported on device 0. + +- Parses ``TDM_TRANSFER`` into ``NUM_TRANSFERS`` Transfer(s). Every Transfer must use the ``T`` executor. + +- Iterates over ``BLOCK_ORDERS`` x ``BLOCKSIZES`` x ``LDS_BYTES``, and for each combination over every value in ``NUM_SUB_EXECS``. + +- Prints a matrix of bandwidths: one row per (order, block size, LDS bytes) triple and one column per sub-executor count. + +- Reports the best bandwidth per sub-executor count, the overall best combination, and a ready-to-paste command that reproduces it. + +- **Automatic timing mode** (``TIMING_MODE=-1``): a single Transfer uses GPU-wallclock timing; multiple Transfers sharing one executor use HIP-event timing; otherwise aggregate CPU timing is used. + +**Restrictions:** + +- **Requires TDM-capable hardware:** AMD gfx1250 or NVIDIA sm_90+. + +- **Hardware gate:** If no TDM executor is present, or ``IsTdmCopySupported(0)`` is false, the preset prints ``[WARN] TDM executor is not supported on this device`` and returns a fatal error. + +- **Executor check:** Every parsed Transfer must use the ``T`` executor. Anything else fails with ``[ERROR] tdmsweep preset only works on Transfers that are using the TDM executor``. + +- **Rejected combinations are skipped, not fatal:** A combination the library refuses (for example, an LDS window larger than the device maximum) prints ``N/A`` in that cell, and the sweep continues. + +- **No successful run:** If every combination fails, the preset reports ``[WARN] No transfers executed successfully`` and returns a fatal error. + +**Usage:** + +.. code-block:: shell + + ./TransferBench tdmsweep [numBytes] + +Sweep the default parameter space for a local GPU 0 TDM copy: + +.. code-block:: shell + + ./TransferBench tdmsweep + +Sweep a GPU0->GPU1 TDM copy of 256 MB over a reduced search space: + +.. code-block:: shell + + TDM_TRANSFER="G0->T0->G1" BLOCKSIZES=256,512 LDS_BYTES=32768,0 ./TransferBench tdmsweep 268435456 + +Environment variables +----------------------- + +To modify the behavior of the tdmsweep preset, use the following environment variables: + +.. list-table:: + :header-rows: 1 + + * - Environment variable + - Description + - Default value + + * - ``BLOCKSIZES`` + - Threadblock sizes to sweep. Each value must be a positive multiple of 32. + - ``64,128,256,512,1024`` + + * - ``BLOCK_ORDERS`` + - Threadblock orderings to sweep: 0=sequential, 1=interleaved, 2=random. Values outside 0-2 are rejected. + - ``0`` + + * - ``LDS_BYTES`` + - LDS staging window sizes (bytes per threadblock) to sweep. ``0`` means the device maximum. Must be >= 0. + - ``8192,16384,32768,65536,0`` + + * - ``NUM_SUB_EXECS`` + - Sub-executor counts to sweep; these form the columns of the output matrix. Must not be empty. + - ``2,4,8,16,32,64`` + + * - ``NUM_TRANSFERS`` + - Number of Transfers described by ``TDM_TRANSFER``. + - ``1`` + + * - ``TIMING_MODE`` + - ``-1``\=auto, ``0``\=aggregate CPU time, ``1``\=executor time (HIP event), ``2``\=transfer time (GPU wallclock). + - ``-1`` + + * - ``TDM_TRANSFER`` + - The TDM Transfer to sweep, using :ref:`Transfer definition syntax `. Must use the ``T`` (TDM) executor. + - ``G0->T0->G0`` + +.. note:: + + The sweep sets the global ``TDM_BLOCK_ORDER``, ``TDM_BLOCK_SIZE``, and ``TDM_LDS_BYTES`` options on each iteration, so setting those directly has no effect when running the tdmsweep preset — use ``BLOCK_ORDERS``, ``BLOCKSIZES``, and ``LDS_BYTES`` instead. + +Example output +---------------- + +The sweep prints one row per (BlkO, BlkS, LDSBytes) combination and one column per sub-executor count, followed by a summary row of the best value in each column: + +.. code-block:: shell + + BlkO BlkS LDSBytes SE 002 SE 004 SE 008 SE 016 SE 032 SE 064 + ... ... ... ... ... ... ... ... ... + BlkO BlkS LDSBytes + ======================================================================================= + Highest bandwidth found: GB/s (GPU wallclock-timed) + BlockOrder : [TDM_BLOCK_ORDER=] + BlockSize : [TDM_BLOCK_SIZE=] + LDS Bytes : [TDM_LDS_BYTES=] + NumSubExec : + Command to run best result: + TDM_BLOCK_ORDER= TDM_BLOCK_SIZE= TDM_LDS_BYTES= ./TransferBench cmdline " G0->T0->G0" + +.. note:: + + A capture from real TDM-capable hardware has not been added yet. diff --git a/docs/reference/transfer-definition-syntax.rst b/docs/reference/transfer-definition-syntax.rst index ebc1ce57..20ccbc1c 100644 --- a/docs/reference/transfer-definition-syntax.rst +++ b/docs/reference/transfer-definition-syntax.rst @@ -163,6 +163,16 @@ The following table lists the supported Executor type letters: - Queue pair - GPU. Uses the closest NIC for SRC and DST. SubIndex is optional. + * - ``B`` + - Batched DMA + - Batch item + - GPU (0 to #GPUs - 1). Requires a single SRC and at least one DST. + + * - ``T`` + - TDM + - Threadblock/CU + - GPU (0 to #GPUs - 1). Requires AMD gfx1250 or NVIDIA sm_90+. + The optional Executor fields have the following meanings: - ``Slot`` (``A``, ``B``, ...): Selects which closest NIC to use, where ``A`` is the first and ``B`` is the second, and so on. Used for ``EXE_NIC_NEAREST``. diff --git a/docs/what-is-transferbench.rst b/docs/what-is-transferbench.rst index 84f6f3df..d2b0d4b8 100644 --- a/docs/what-is-transferbench.rst +++ b/docs/what-is-transferbench.rst @@ -27,6 +27,8 @@ An Executor (EXE) is the specific device that performs a transfer — reading fr - **CPU threads** - **GPU compute kernels** (GFX) - **GPU DMA engines** (SDMA) +- **GPU batched DMA** (must have a single SRC and at least one DST) +- **GPU TDM** (Tensor Data Mover; requires hardware support: AMD gfx1250 or NVIDIA sm_90+) - **NIC RDMA** (via Remote Direct Memory Access network interface cards) Some Executors support **SubExecutors**, which allow further partitioning of the data to be transferred across multiple execution units within the same device. @@ -36,7 +38,7 @@ Features TransferBench supports the following features: -- **Multiple Executors:** CPU threads, GPU compute kernels (GFX), GPU DMA engines (SDMA), and NIC RDMA. +- **Multiple Executors:** CPU threads, GPU compute kernels (GFX), GPU DMA engines (SDMA), GPU batched DMA, GPU TDM, and NIC RDMA. - **Multi-input or multi-output (MIMO) transfers:** Element-wise sum from multiple SRCs to multiple DSTs.