Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1,375 changes: 1,375 additions & 0 deletions .github/bench/report_page.py

Large diffs are not rendered by default.

229 changes: 229 additions & 0 deletions .github/workflows/benchmark-report.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,229 @@
#
# Copyright (c) 2026 Steve Gerbino
#
# Distributed under the Boost Software License, Version 1.0. (See accompanying
# file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
#
# Official repository: https://github.com/cppalliance/corosio/
#
# Comparison benchmark run for the published documentation report.
# Dispatch-only; artifacts are turned into doc pages locally — see
# bench/README.md ("Refreshing the published report").
#
# Runner prerequisites: same as benchmarks.yml.

name: benchmark-report

on:
workflow_dispatch:
inputs:
iterations:
description: "iterations per configuration"
default: "7"
duration:
description: "seconds per benchmark"
default: "2"

permissions:
contents: read

concurrency:
group: benchmark-report
cancel-in-progress: false

jobs:
bench:
strategy:
fail-fast: false
matrix:
include:
- platform: linux
labels: '["self-hosted", "Linux", "X64"]'
configs: "corosio-epoll corosio-uring asio-epoll asio-uring asio_callback-epoll asio_callback-uring"
- platform: windows
labels: '["self-hosted", "Windows", "X64"]'
configs: "corosio-iocp asio asio_callback"
- platform: macos
labels: '["self-hosted", "macOS", "ARM64"]'
configs: "corosio-kqueue asio asio_callback"
name: bench-report-${{ matrix.platform }}
runs-on: ${{ fromJSON(matrix.labels) }}
timeout-minutes: 240
env:
ITERS: ${{ inputs.iterations || '7' }}
DUR: ${{ inputs.duration || '2' }}
defaults:
run:
shell: bash
steps:
- name: Clean workspace
run: rm -rf results build-aoff build-aon ws boost-root && mkdir -p results

- name: Checkout corosio
uses: actions/checkout@v4
with:
path: ws/corosio
persist-credentials: false

- name: Resolve Capy branch
id: capy-ref
uses: ./ws/corosio/.github/actions/resolve-capy

- name: Checkout capy
uses: actions/checkout@v4
with:
repository: ${{ steps.capy-ref.outputs.repo }}
ref: ${{ steps.capy-ref.outputs.ref }}
path: ws/capy
persist-credentials: false

- name: Clone Boost
uses: alandefreitas/cpp-actions/boost-clone@v1.9.0
with:
branch: develop
boost-dir: boost-root
modules: asio
modules-exclude-paths: ''
scan-modules-dir: ws/corosio
scan-modules-ignore: corosio, capy

- name: Patch Boost
run: |
set -e
rm -rf boost-root/libs/corosio boost-root/libs/capy
cp -r ws/corosio boost-root/libs/corosio
cp -r ws/capy boost-root/libs/capy

- name: Put common toolchain dirs on PATH
run: |
for dir in /opt/homebrew/bin /usr/local/bin "$HOME/.local/bin" /snap/bin /opt/cmake/bin; do
if [ -d "$dir" ]; then echo "$dir" >> "$GITHUB_PATH"; fi
done

- name: Configure and build (epoll/default asio reactor)
run: |
set -e
cmake -S boost-root -B build-aoff \
-DCMAKE_BUILD_TYPE=Release \
-DBOOST_INCLUDE_LIBRARIES="corosio;asio" \
-DBOOST_COROSIO_BUILD_BENCH=ON \
-DBOOST_COROSIO_BUILD_TESTS=OFF \
-DBOOST_COROSIO_BUILD_EXAMPLES=OFF \
-DBOOST_COROSIO_BENCH_ASIO_IO_URING=OFF
cmake --build build-aoff --config Release --target corosio_bench --parallel

# Linux only: a second binary with asio built against io_uring, so
# the asio-uring/asio_callback-uring configs compare against a
# same-reactor asio rather than an epoll-only one. Harmless to skip
# on Windows/macOS, which have no io_uring reactor to build against.
- name: Configure and build (io_uring asio reactor)
if: matrix.platform == 'linux'
run: |
set -e
cmake -S boost-root -B build-aon \
-DCMAKE_BUILD_TYPE=Release \
-DBOOST_INCLUDE_LIBRARIES="corosio;asio" \
-DBOOST_COROSIO_BUILD_BENCH=ON \
-DBOOST_COROSIO_BUILD_TESTS=OFF \
-DBOOST_COROSIO_BUILD_EXAMPLES=OFF \
-DBOOST_COROSIO_BENCH_ASIO_IO_URING=ON
cmake --build build-aon --config Release --target corosio_bench --parallel

- name: Locate binaries
id: bin
run: |
set -e
bin_off=$(find build-aoff -type f \( -name corosio_bench -o -name corosio_bench.exe \) | head -1)
test -n "$bin_off"
"$bin_off" --library asio --list > /dev/null # hard-fail early if asio absent
if [ "${{ matrix.platform }}" = "linux" ]; then
bin_on=$(find build-aon -type f -name corosio_bench | head -1)
test -n "$bin_on"
"$bin_on" --library asio --list > /dev/null
else
bin_on="$bin_off"
fi
echo "bin_off=$bin_off" >> "$GITHUB_OUTPUT"
echo "bin_on=$bin_on" >> "$GITHUB_OUTPUT"

- name: Run interleaved comparison
run: |
set -e
configs=(${{ matrix.configs }})
n=${#configs[@]}
for i in $(seq 1 "$ITERS"); do
# rotate starting configuration each iteration so slow drift
# does not systematically favor any one implementation
for j in $(seq 0 $((n - 1))); do
cfg=${configs[$(( (j + i - 1) % n ))]}
case "$cfg" in
corosio-*) args="--library corosio --backend ${cfg#corosio-}" ;;
asio-*) args="--library asio --backend ${cfg#asio-}" ;;
asio_callback-*) args="--library asio_callback --backend ${cfg#asio_callback-}" ;;
*) args="--library $cfg" ;;
esac
# Only the io_uring-flavored asio configs need the ON binary
# (asio's reactor is fixed at compile time); corosio picks
# its backend at runtime via --backend, so every corosio
# config runs fine on the OFF binary.
case "$cfg" in
asio-uring|asio_callback-uring) bin="${{ steps.bin.outputs.bin_on }}" ;;
*) bin="${{ steps.bin.outputs.bin_off }}" ;;
esac
"$bin" $args \
--duration "$DUR" --output "results/$cfg-$i.json"
done
done

- name: Capture environment
continue-on-error: true
run: |
cpu=$( (grep -m1 'model name' /proc/cpuinfo | cut -d: -f2-) 2>/dev/null \
|| sysctl -n machdep.cpu.brand_string 2>/dev/null \
|| powershell -NoProfile -Command "(Get-CimInstance Win32_Processor).Name" 2>/dev/null \
|| echo unknown)
cores=$(nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo unknown)
ram=$( (awk '/MemTotal/ {printf "%.0f", $2/1048576}' /proc/meminfo) 2>/dev/null \
|| (sysctl -n hw.memsize 2>/dev/null | awk '{printf "%.0f", $1/1073741824}') \
|| powershell -NoProfile -Command "[math]::Round((Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory/1GB)" 2>/dev/null \
|| echo unknown)
osname=$(uname -sr 2>/dev/null || echo unknown)
kern=$(uname -v 2>/dev/null || echo unknown)
if command -v cl.exe >/dev/null 2>&1; then comp=$(cl.exe 2>&1 | head -1)
elif command -v c++ >/dev/null 2>&1; then comp=$(c++ --version | head -1)
else comp=unknown; fi
cmakev=$(cmake --version | head -1)
uringv=$( (dpkg-query -W -f '${Version}' liburing-dev) 2>/dev/null || echo n/a)
ASIO_REACTOR=$(case "${{ matrix.platform }}" in linux) echo "epoll and io_uring (matched per configuration)";; windows) echo IOCP;; macos) echo kqueue;; esac)
CPU="$cpu" CORES="$cores" RAM="$ram" OSN="$osname" KERN="$kern" \
COMP="$comp" CMAKEV="$cmakev" URINGV="$uringv" \
BOOST_SHA=$(git -C boost-root rev-parse HEAD) \
ASIO_SHA=$(git -C boost-root/libs/asio rev-parse HEAD 2>/dev/null || echo unknown) \
CAPY_SHA=$(git -C ws/capy rev-parse HEAD) \
COROSIO_SHA=${{ github.sha }} COROSIO_REF=${{ github.ref_name }} \
PLATFORM=${{ matrix.platform }} ITERS="$ITERS" DUR="$DUR" \
ASIO_REACTOR="$ASIO_REACTOR" \
RUN_URL="${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}" \
python3 - <<'EOF'
import json, os
e = os.environ
json.dump({
"platform": e["PLATFORM"], "cpu": e["CPU"].strip(),
"cores": e["CORES"], "ram_gb": e["RAM"], "os": e["OSN"],
"kernel": e["KERN"], "compiler": e["COMP"], "cmake": e["CMAKEV"],
"liburing": e["URINGV"], "boost_sha": e["BOOST_SHA"],
"asio_sha": e["ASIO_SHA"], "asio_reactor": e["ASIO_REACTOR"],
"capy_sha": e["CAPY_SHA"],
"corosio_sha": e["COROSIO_SHA"], "corosio_ref": e["COROSIO_REF"],
"date_utc": __import__("datetime").datetime.utcnow().strftime("%Y-%m-%d"),
"iterations": int(e["ITERS"]), "duration_s": float(e["DUR"]),
"run_url": e["RUN_URL"],
}, open("results/environment.json", "w"), indent=2)
EOF

- name: Upload results
if: always()
uses: actions/upload-artifact@v4
with:
name: bench-report-${{ matrix.platform }}
path: results/
150 changes: 150 additions & 0 deletions bench/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,150 @@
# Benchmarks

## What lives here

- `corosio/` — corosio benchmark suites (one `.cpp` per category).
- `asio/callback/`, `asio/coroutine/` — equivalent suites against
Boost.Asio, built only when `Boost::asio` is available at configure time.
- `common/` — shared harness: benchmark registration, timing, backend
selection, HTTP parsing helpers.

All suites build into a single `corosio_bench` binary, selected at
runtime via `--library`. Categories: `io_context`, `socket_throughput`,
`socket_latency`, `http_server`, `accept_churn`, `fan_out`,
`local_socket_throughput`, `local_socket_latency` (the last two are POSIX
only). Run `corosio_bench --list` for the exact set of benchmarks in each
category on your build — some are gated by backend or platform.

## Building

Build inside the Boost superproject with `asio` included, so
`bench/CMakeLists.txt` picks up `Boost::asio` as a sibling target and
compiles the comparison suites:

```bash
cmake -S boost-root -B build \
-DCMAKE_BUILD_TYPE=Release \
-DBOOST_INCLUDE_LIBRARIES="corosio;asio" \
-DBOOST_COROSIO_BUILD_BENCH=ON
cmake --build build --config Release --target corosio_bench --parallel
```

Building corosio standalone also works
(`-DBOOST_COROSIO_BUILD_BENCH=ON` from the corosio checkout), but then
the comparison suites need a system-installed Boost 1.84+ with Asio —
`bench/CMakeLists.txt` falls back to `find_package(Boost COMPONENTS
asio)` when no sibling target exists. Either way, if configure prints:

```
Boost.Asio not found -- comparison benchmarks disabled
```

then `corosio_bench` only accepts `--library corosio`; `asio` and
`asio_callback` are compiled out entirely, not just unavailable at
runtime.

Always build `Release`. Debug disables inlining and optimization, which
distorts relative costs between fast and slow paths.

## Running locally

The binary lands at `build/bench/<config>/corosio_bench` (or
`build/bench/corosio_bench` for a single-config generator). Run
benchmarks one process at a time — never in parallel — since concurrent
processes contend for CPU and cache and produce unreliable numbers.

Full suite, default library (corosio), platform-default backend:

```bash
build/bench/Release/corosio_bench
```

One category:

```bash
build/bench/Release/corosio_bench --category socket_throughput
```

One benchmark (`--bench` is a prefix match on the benchmark name, not
the category; combine with `--category` to disambiguate across
categories):

```bash
build/bench/Release/corosio_bench --category socket_throughput \
--bench unidirectional
```

Select a backend, comparison library, duration, and warmup:

```bash
build/bench/Release/corosio_bench --library asio_callback --backend epoll \
--duration 5 --warmup 0.5
```

`--duration` sets the measured time per benchmark (default 3s);
`--warmup` runs an unmeasured pass first (default 0, disabled) — use it
for benchmarks sensitive to cold caches or lazy connection setup.
`--library all` runs corosio and both asio variants back to back. Write
results to JSON for later aggregation:

```bash
build/bench/Release/corosio_bench --output results/corosio-epoll-1.json
```

To compare two working trees (e.g. before/after a change), build both,
then run each several times with the same flags, alternating which one
goes first each iteration — this cancels out drift from thermal
throttling or background load instead of it favoring whichever side ran
first.

## The PR benchmark workflow

`.github/workflows/benchmarks.yml` runs `corosio_bench` on dedicated
self-hosted runners for pull requests opened by the repo owner or a
collaborator, or on any PR labeled `benchmark`. It builds base and head,
runs both interleaved (ABBA order) across several iterations per
platform, and posts a single updating PR comment
(`.github/bench/compare.py`) summarizing per-category deltas with a
within-noise/faster/slower verdict. It's advisory only — see issue #343.

## Refreshing the published report

`doc/modules/ROOT/pages/benchmark-report.adoc` is a generated page —
nothing on it is hand-edited. Refresh it after a change that meaningfully
affects performance:

```bash
# 1. Run the full suite on all three platforms and collect raw JSON.
gh workflow run benchmark-report.yml --repo cppalliance/corosio --ref develop

# 2. Download the workflow's artifacts once it completes.
gh run download <run-id> --repo cppalliance/corosio -D /tmp/bench-report

# 3. Regenerate the page and its charts from the raw JSON.
python3 .github/bench/report_page.py --input-dir /tmp/bench-report \
--output-dir doc/modules/ROOT

# 4. Preview the rendered docs site.
./doc/build_antora.sh

# 5. Commit the regenerated page and charts (nothing else).
git add doc/modules/ROOT/pages/benchmark-report.adoc doc/modules/ROOT/images/bench
git commit -m "docs: regenerate benchmark report"
```

Never regenerate the published page from a reduced/smoke run (e.g.
`iterations=1`): single-iteration data has no noise estimate, so the
summary's within-noise/faster/slower classifications are meaningless.

`report_page.py` expects `<input-dir>` to contain one
`bench-report-<platform>/` directory per platform (`linux`, `windows`,
`macos`), each holding the raw `<config>-<iter>.json` files plus an
`environment.json`. A missing platform directory is not an error — the
page notes it and generation continues with what's present.

## Methodology

Metric selection, aggregation (median across iterations, CV for noise),
and the within-noise threshold are implemented once, in
`report_page.py`, and described in full on the generated page's own
Methodology section — that page is the source of truth, not this file.
8 changes: 8 additions & 0 deletions bench/asio/callback/accept_churn_bench.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,14 @@ make_churn_acceptor(asio::io_context& ioc)
ec = acc.open(tcp::v4(), ec);
if (!ec)
ec = acc.set_option(tcp_acceptor::reuse_address(true), ec);
// Accepted sockets inherit these from the listener
if (!ec)
ec = acc.set_option(asio::socket_base::send_buffer_size(1024), ec);
if (!ec)
ec = acc.set_option(
asio::socket_base::receive_buffer_size(1024), ec);
if (!ec)
ec = acc.set_option(asio::socket_base::linger(true, 0), ec);
if (!ec)
ec = acc.bind(tcp::endpoint(tcp::v4(), 0), ec);
if (!ec)
Expand Down
Loading
Loading