High-Performance Networking & RDMA

C implementations and performance experiments spanning conventional TCP sockets, InfiniBand Verbs, and distributed ring collectives. The main system implements Reduce-Scatter, All-Gather, and All-Reduce over a custom RDMA transport, with eager and rendezvous paths for different message sizes.

Languages: C, Python (analysis)
Systems: Linux, POSIX sockets, InfiniBand / libibverbs, GNU Make

Highlights

• TCP throughput benchmark with warm-up, adaptive iteration counts, and convergence checks.
• RDMA bandwidth measurement using memory registration, queue pairs, completion queues, and verbs operations.
• Ring-based collective communication supporting INT32/FP64, sum/product, and fixed-size custom reduction callbacks.
• Two transfer modes: small-message eager sends and pipelined RDMA rendezvous for larger blocks.
• Flow control, CQ polling, completion accounting, and ring-size-aware message partitioning.

Implementations

Component	Key files	Technical focus
TCP throughput	client.c, server.c, common.c	Adaptive socket benchmarking and throughput measurement
RDMA bandwidth	bw_template.c	Low-level InfiniBand Verbs and registered-memory transfers
Ring collectives	pg.c, pg_verbs.c, pg_bootstrap.c	Ring Reduce-Scatter / All-Gather / All-Reduce, data transport and bootstrap

Ring All-Reduce architecture

A distributed All-Reduce is decomposed into two ring phases:

1. Reduce-Scatter: partial reductions circulate until each rank owns one reduced partition.
2. All-Gather: reduced partitions circulate until every rank has the full result.

For small transfers, the transport uses SEND_WITH_IMM with pre-posted receive buffers. Large transfers use RDMA_WRITE_WITH_IMM, registered memory, and pipeline slots to reduce copying and coordinate progress. The technical walkthrough covers the transport, queue-pair topology, flow control, and APIs in detail.

Measured performance

Results below are from the original Linux/InfiniBand test environment, using 65,536 FP64 elements (512 KiB per rank) and median All-Reduce latency.

Ranks	Eager	RDMA rendezvous	Speedup
2	384.61 µs	200.96 µs	1.91×
4	576.31 µs	345.04 µs	1.67×

Two ranks	Four ranks

The RDMA bandwidth experiment recorded approximately 33.85 Gbps peak throughput on the available InfiniBand hardware. These numbers describe that environment; they are not portable performance guarantees.

Detailed All-Reduce report · TCP report · RDMA bandwidth report

Build and environment

TCP (ex1) requires Linux, gcc, and make:

cd ex1
make
# Run ./server on one machine, then ./client <server-ip> on another.

RDMA (ex2 and ex3) requires Linux hosts with compatible InfiniBand/RDMA devices, working device configuration, and libibverbs development headers:

cd ex3
make

The RDMA programs were developed for a configured multi-host lab network. The original hardware and host configuration are not distributed, so reproducing the benchmarks requires equivalent infrastructure. See the Exercise 3 README for rank startup flags, transport design, and protocol thresholds.

Skills demonstrated

Low-level C systems programming, distributed algorithms, memory registration, one-sided communication, concurrency and flow-control reasoning, benchmark methodology, and performance optimization.

Contributors: Shay Morad, Avraham Klings, Yaacov Pinkert.