Skip to content

perf(ssh): relay each session's channels from one event-driven reader and drop the App Nap hold - #4248

Merged
datlechin merged 3 commits into
mainfrom
fix/perf-ssh-relay
Nov 13, 2026
Merged

datlechin merged 3 commits into
mainfrom
fix/perf-ssh-relay

Conversation

@datlechin

@datlechin datlechin commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Summary

  • One reader per SSH session (CPU-06 rest). New SSHSessionPump in TableProSSHTransport. Every libssh2 call on a session runs on its serial queue, and after each call the pump serves every channel, because ssh2_channel_read and ssh2_channel_write pull every channel's packets off the socket. Readiness comes from DispatchSource read and write sources on the transport and local sockets. No 500 ms poll and no thread per relay. The macOS tunnel, Remote SQLite, jump hops and the iOS tunnel all use it.
  • libssh2 sends one packet at a time per session, and only the call that started a packet can finish it. A relay owns its half-sent packet and ends only once it is out. Relays take turns starting packets, and opens and frees hold a send turn, so an upload cannot starve them.
  • App Nap: SSHTunnelManager and RemoteSQLiteTransportManager no longer hold an activity while a tunnel is open. TCP keepalive runs in the kernel, server requests are answered on transport read events, and the SSH keepalive sleeps until libssh2's next due time (30 s, was a 10 s tick).
  • Keepalive and channel EOF build their packets on the stack, and one that blocked wedged the whole session. They now go out only when nothing else waits to send and the socket has room. A reset or closed socket fails the keepalive at once. The keepalive asks for no reply, since libssh2 queued every reply and never removed it.
  • Remote SQLite: an exec channel given up mid-open is finished and freed through the free loop. One non-blocking free used to leak it.
  • Probes: check-ssh-relay.sh measures idle wakeups, bulk throughput and latency beside other channels. check-libssh2-keepalive-pending.sh shows the wedge. The tunnel matrix harness builds again (stale localPort: call, missing LogRedaction source) and can build with -O.
  • Tests replaced on purpose: SSHChannelRelayTests went with SSHChannelRelay; every case is ported to SSHSessionPumpTests. The relayFDState and transportPollOutcome cases in RelayPollStateTests went with those functions.

Test plan

  • apple/scripts/probes/check-ssh-relay.sh, OpenSSH 10 and Redis in Docker, harness built -O, before = origin/main:
    • Idle wakeups/s with 0, 1 and 3 relayed clients: 0.20, 2.18, 6.12 before; 0.06, 0.06, 0.06 after. Per relay 1.97/s to 0, and one parked thread per relay to none.
    • App Nap activity while a tunnel is open: held before, none after.
    • 4 clients x 500 PING: p99 2.4 ms before, 0.5 ms after. Before also showed rounds of 500.2 and 502.6 ms (the lost wakeup); after, none over 3.3 ms.
    • 64 MB SET and GET: about 200 MB/s each way before, 180 to 236 MB/s after.
    • PING beside a 16 MB GET loop on the same tunnel: see Needs the maintainer.
  • check-ssh-tunnel-matrix.sh --app macos: 87 of 87, including the loss cells (relaying tunnel notices in 0.0 s, idle one in 59.4 s and 29.8 s, CPU under 0.1 s).
  • check-libssh2-keepalive-pending.sh: after a keepalive whose send blocks, 2,141,640 channel writes are refused in 3 s with the socket writable, and a keepalive from another stack frame does not flush it.
  • swift test TableProSSHTransportTests: 203 pass. New: data another channel's read queued reaches its client with no transport event, an idle relay makes no channel call, local data flows while the client does not read, a relay's half-sent write or window adjust finishes before it ends, a message and an outside call get through beside a socket-limited upload, an upload keeps going beside a channel whose window is closed.
  • Hosted SSH suites: 129 pass, including the new LibSSH2KeepAliveTests. iOS build-for-testing succeeds; this Mac has no simulator runtime, so CI runs the iOS tests.

Needs the maintainer

  • PING beside a 16 MB GET loop on the same tunnel, where the GETs keep the session's one queue busy decrypting at full speed: p50 2.8 ms and p99 15 ms before, p50 13.5 to 17 ms and p99 40 to 46 ms after, while the GETs went from 185 to 220-234 MB/s. A smaller read budget per pass did not change it. A link slower than the decrypt rate leaves the queue idle between packets.
  • Cloudflare, Cloud SQL, tunnel command and SOCKS tunnels still hold an App Nap activity; they are not SSH relays.

@datlechin
datlechin enabled auto-merge (squash) November 12, 2026 20:30
@datlechin
datlechin merged commit 9bf8f6e into main Nov 13, 2026
64 of 67 checks passed
@datlechin
datlechin deleted the fix/perf-ssh-relay branch November 14, 2026 21:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant