Skip to content

Core/unified msg and multiplexer - #168

Open
vorjdux wants to merge 127 commits into
mainfrom
core/unified-msg-and-multiplexer
Open

vorjdux wants to merge 127 commits into
mainfrom
core/unified-msg-and-multiplexer

Conversation

@vorjdux

@vorjdux vorjdux commented Oct 30, 2026

Copy link
Copy Markdown
Owner

No description provided.

… blocks

RouterHub and PubSubHub had no users outside monocoque-core. The futures
Stream/Sink adapters only ever returned Unsupported and nothing implemented
their traits. SocketOptions::small()/large() and BufferConfig set read
buffers below the 32 KiB default and were read by nothing.

Delete the probe/debug examples, monocoque-zmtp/bench_results.txt and the
interop scripts no workflow or runner references.

Every crate now takes its lint policy from [workspace.lints] alone. The
blanket cast_possible_truncation allow is gone; each narrowing cast is now a
checked conversion. The ZAP metadata encoder clamps the value to u32::MAX so
its length prefix always matches the bytes written, and the CURVE command
writer reuses append_zmtp_cmd_frame instead of copying the frame twice.
rust-toolchain.toml pins local builds to the CI clippy toolchain (1.98).
Every other CI job sets RUSTUP_TOOLCHAIN so it keeps running on the
toolchain it installs (stable, beta, nightly, the 1.95 MSRV).

dependabot batches weekly minor/patch bumps into one PR per ecosystem and
ignores the documented chacha20poly1305/crypto_box aead-0.5 pin.

The README Safety section is now the single unsafe inventory and matches
the code; CONTRIBUTING and the crate docs point to it instead of making
contradictory claims. Drop the README roadmap and the stale phase table in
the overview blueprint.
compio-io's default write_vectored writes one iovec per call, so every
multi-frame send over IPC on the tokio backend issued one syscall per
header and body while TCP took a single writev. Route it through the same
write_vectored_from the TCP adapters use, with a test that fails without
the override (10 of 60 bytes written).
The PLAIN and CURVE server handshakes passed the literal "unknown" as the
ZAP Address frame, so any handler applying IP allow/deny policy saw the
same value for every client and the policy was silently disabled.

Add a PeerAddress trait to the runtime facade that reports the address the
way libzmq does (the peer IP for TCP, empty for IPC and inproc) and require
it on the handshake stream bound, so both server exchanges read it from the
connection they are authenticating. zap_peer_address runs the full TCP
handshake for both mechanisms against a recording ZAP handler and fails
with the old literal.
idle_wakeups was a manual RSS harness behind #[ignore], so nothing gated
the claim that an idle connection costs no wakeups. The new test parks a
recv() on every receive-capable socket of DEALER/ROUTER, REQ/REP,
PUSH/PULL, PAIR and PUB/SUB, then sums the context switches of every
thread in the process across a 500 ms idle window. Today each case costs
1 (compio), 2 (tokio) or 5 (smol) switches against a budget of 10.

XPUB's recv_subscription polls each subscriber under a 1 ms read timeout,
which measures at about 470 switches per window; that case is an ignored
guard until the poll is removed.

docs/perf/baselines/a37409b.md freezes docs/performance.md as released in
v0.4.2, alongside the earlier c7d7ead baseline.
These are the pieces the multiplexer is built from.

Msg stores one frame of up to 64 bytes inline, one larger frame as a
single Bytes, up to four frames in a SmallVec, and a fan-out as one
Arc<[Bytes]>. MsgRef is the borrowed view a receive will hand out.

LocalQueue is a same-thread SPSC queue over Rc<RefCell<VecDeque>> with
one waker per side and no atomics. Capacity is the HWM (0 is unbounded),
try_send hands the value back at the mark, send waits there, and a
conflating queue keeps only the newest value.

transport::Stream is a TCP or IPC stream whose reads and writes go
through &Stream, so a peer can keep a read and a write in flight without
splitting it. The tokio and smol adapters gain the &TcpStream and
&UnixStream impls compio already has. Stream::configure applies
SO_SNDBUF/SO_RCVBUF to IPC streams as well as TCP, since Unix sockets
have no kernel autotuning.
A socket becomes a set of peers. Each wire peer has one reader loop and
one writer loop, boxed once per connection: the reader decodes complete
messages into the peer's ready queue and stops reading at the receive
HWM, so a slow consumer backpressures only its own peer; the writer
drains the peer's outbound queue one job at a time, coalescing only what
accumulated behind a write already in flight. inproc peers move Msg
values through a LocalQueue on one thread or a flume channel across
threads, with no codec.

Every per-socket future is a child polled by whichever task touches the
socket. A child's waker wakes the user task if it is parked on the
socket and the driver otherwise, so a read completing under a waiting
recv() runs inline in the user's poll with no task in between. The
driver task handles accept, dial and reconnect with jittered backoff,
heartbeats when enabled, and linger on close; idle with no heartbeat it
takes no wakeups.

No loop is ever cancelled by a timeout, so no in-flight read or write is
dropped. The core is protocol-agnostic: a Protocol supplies the
handshake and a per-connection Codec. The error type gains the ZeroMQ
cases callers act on (Again, TimedOut, HostUnreachable, InvalidState,
Unsupported) as variants.

Tested over a toy length-prefixed protocol on all three backends: TCP
round trips, fair queueing, HWM backpressure, reconnect, inproc on one
and across threads, linger flush, and shared-encoding fan-out.
wire::Zmtp implements the core Protocol: the handshake reuses the
existing NULL/PLAIN/CURVE engine, and ZmtpCodec decodes straight out of
a peer's contiguous read buffer. A single frame of up to 64 bytes is
copied into the Msg itself, so it costs no allocation or refcount and
pins none of the read buffer; a frame longer than what has been read so
far reserves its full size up front instead of regrowing a staging
buffer. Bodies at or above the vectored threshold go out as their own
iovec. PING is answered with a PONG echoing its context, and ZMTP 3.1
SUBSCRIBE/CANCEL surface as subscription messages.

sock::Socket<K> carries what every type shares (bind and connect to any
mix of tcp/ipc/inproc, options, monitor, timeouts that never cancel I/O,
the borrowed receive slot); each kind adds its pattern: PAIR refuses a
second peer, ROUTER keeps a SipHash identity table with handover and
mandatory routing, REQ/REP enforce alternation and carry the envelope,
PUB/XPUB match through one subscription index and fan out one shared
encoding, SUB/XSUB filter locally and replay subscriptions to new peers.

SubscriptionTrie and the unused pubsub index collapse into one matcher:
a sorted prefix vector with per-peer reference counts, an early-exit
forward scan, and a per-subscriber cap on distinct prefixes.

sock_patterns covers every pattern on tcp, ipc and inproc, and talks to
libzmq for REQ->REP, DEALER->ROUTER, PUB->SUB both ways and large-frame
PUSH->PULL, on all three backends.
Measured against the old direct REQ/REP path (64 B, loopback, two threads
pinned to two cores), the first cut of the multiplexer was 15-20% slower.
Under callgrind (tokio backend) it ran 48.4M instructions for 3,200
round trips against the old path's 27.3M. Fixes, in order of effect:

- REQ and REP envelopes are encoded as a prefix by the codec instead of
  being pushed onto the user's message, which turned an inline message
  into a heap SmallVec on every request.
- Received messages are written straight into the socket's receive slot
  instead of moving a 144-byte Msg through every future layer, and
  Msg::push/push_front/pop_front work in place on multi-frame messages.
- A child woken by the user's own operation no longer schedules the
  driver task; the operation drains it before returning, and anything
  left (including a cross-thread wakeup after the last drain) is routed
  on exit.
- Wakeups keeps atomic mirrors of its lock-protected state so the common
  nothing-to-do paths skip the mutex.
- The tokio &TcpStream read clears cached readiness after a short read,
  as tokio's own poll_read does; without that every message cost one
  extra recvfrom returning EAGAIN.

The idle-wakeup gate counts only voluntary context switches: preemption
by other load is not a wakeup the socket caused, and counting it made the
gate flap on a busy machine.

Result: 40.2M instructions; p50 gap down to about 0.45-0.55 us (6-7%) on
both compio and tokio. That is still outside run-to-run noise, so the
one-peer gate is not yet met.
The monocoque::zmq facade now re-exports the multiplexer sockets directly
and the old per-connection wrappers are gone. Every test, example and
bench uses bind/connect, Msg and borrowed receives.

New on the multiplexer:
- STREAM sockets with libzmq framing, connect and disconnect notifies.
- proxy and proxy_steerable. The loop waits on socket readiness and only
  builds a receive once a message is queued, so no in-flight receive is
  dropped. PAUSE holds messages in the sockets instead of discarding them
  and STATISTICS replies with libzmq's eight counters.
- Eager delivery: REQ, DEALER, PUSH and ROUTER sends return once the
  bytes are handed to the kernel. PUB stays non-blocking.
- Subscriptions bypass the HWM and are replayed in the greeting when a
  peer comes up.

Write coalescing no longer stalls a sender. Held writes are released at
the byte threshold or when the send HWM fills, and a flush no longer
turns coalescing off for the rest of the socket's life.

Tests: REQ state machine and gate invariants rewritten against the new
API, proxy tests with real libzmq peers through the broker, regression
tests for coalescing past the HWM and for fan-out to 100 TCP subscribers
with one blocked reader.
Only the multiplexer sockets remain. The old socket modules in
monocoque-zmtp, the old proxy, the socket trait and the in-process
stream adapter are gone, as are the core modules nothing uses any more
(inproc, message, message_builder, backpressure, poison, socket_type,
config, buffer) and the dependencies they pulled in.

ZAP now runs on the multiplexer: the server is a ROUTER bound to
inproc://zeromq.zap.01 and the client a DEALER, so each reply reaches
the connection that asked. With no handler bound the handshake is still
rejected.

Fixes found while porting the remaining tests:
- A finished child's slot id could be reused, and a later cancel by the
  old id stopped the new occupant. Child ids now carry a generation.
- ROUTER dropped messages a peer had queued before disconnecting, since
  its identity was deleted at once. The identity now lives until the
  slot is reused.
- The ZMTP decoder rejects command frames with MORE set again.

Integration scenarios not already covered elsewhere are ported to
monocoque-zmtp/tests/sock_scenarios.rs, unit tests for surviving logic
moved with it, and the fuzz targets drive the new codec.
poll waits on any mix of socket kinds for readable and writable
readiness. Readiness follows each pattern's rules: REQ is readable only
for the reply it is waiting on, SUB does not report messages its filter
would drop, and writable means the routing strategy has room for one
more message. Polling never dequeues and the caller owns the PollItem
slice, so a poll call allocates nothing. Dropping a poll future leaves
every socket usable.

Each connection's properties (Socket-Type, Identity, Peer-Address, the
ZAP User-Id and any custom READY properties) are parsed once at
handshake into an Arc'd Metadata on the peer. A received MsgRef borrows
it through gets/property/metadata, so Msg stays the same size and the
hot path does no extra work. SocketOptions::with_metadata sends custom
properties, which libzmq reads with zmq_msg_gets.

64B REQ/REP round trip, interleaved runs pinned to two cores:
before 8.93/8.72/8.77 us, after 8.76/8.75/8.73 us.
The greeting now goes out in stages, as libzmq sends it: the signature,
then the major version once the peer's signature shows it is not 1.0,
then the rest once its revision shows 3.x. A 1.0 or 2.0 peer is
recognised before it is sent bytes it cannot parse. The signature
padding carries our 1.0 identity frame length so a 1.0 peer reads it as
the start of that frame.

Revision 1 peers swap ZMTP 2.0 socket type numbers and identity frames
and then use ZMTP 3 data framing without commands. 1.0 peers, and
revision 0, use the length-counts-flags framing. A publisher subscribes
a 1.0 subscriber to everything, since those filter for themselves.
Legacy peers get no heartbeat PINGs.

A peer can never talk a secured socket down: a socket using PLAIN or
CURVE, or NULL with a ZAP domain, closes a 1.0 or 2.0 peer before
reading anything past its greeting.

The ZMTP 3 decode and encode paths gain one predictable branch per call.
64B REQ/REP round trip, interleaved runs pinned to two cores:
before 8.77/9.17/8.83 us, after 8.81/8.75/8.79 us.
with_conflate set a field nothing read. Each peer's ready buffer now
keeps only its newest message when conflate is on: an arrival replaces
the waiting one, a multipart message whole, and the receive HWM stops
applying so the reader never pauses. It covers TCP, IPC and both inproc
links, since they all deliver into the same buffer. Publishers ignore
it because what they receive is subscriptions, which must all apply.

The receive HWM is enforced per peer on every transport by the
multiplexer, so its docs now say that: a peer at the mark is not read
until the application receives, and only that sender is pushed back.
Both HWMs document 0 as unbounded, as in libzmq, and the send HWM says
which patterns wait and which drop at the mark.

Interleaved runs pinned to two cores, before then after:
64B REQ/REP round trip 8.66/8.61/8.75 us vs 8.62/8.83/8.62 us;
64B coalesced PUSH/PULL 6.99/7.17/7.13 vs 7.13/7.06/7.17 Melem/s.
CLIENT/SERVER, RADIO/DISH, SCATTER/GATHER and CHANNEL, enabled with the
draft feature on monocoque-rs. All carry single-part messages: a
multipart send fails with InvalidState and a multipart message from a
peer is dropped. Each talks only to its partner type, and a ZMTP 1.0 or
2.0 peer is refused since the types need ZMTP 3.1.

SERVER hands out a nonzero 32-bit routing id per peer with every
received message and routes sends by it, failing with HostUnreachable
for an unknown id and Again at the peer's HWM. A departed peer keeps its
id until its slot is reused so its last messages still arrive addressed.

RADIO fans each message out to the DISH peers that joined its group,
reusing one scratch vector per socket. DISH joins exact group names of up
to 255 bytes and receives [group, body]. On the wire DISH sends JOIN and
LEAVE commands and RADIO sends the group as a frame ahead of the body,
as libzmq does. DISH over CURVE is refused for now: its commands would
have to be encrypted.

The installed libzmq has no draft support, so the tests run monocoque
against itself over TCP and inproc.
On the compio backend a write the socket buffer can take at once now
goes out as a non-blocking send instead of a ring submission and
completion: one syscall and no wakeup. When the buffer is full the ring
path takes over as before. Writes over 64 KiB keep going to the ring,
since copying a large coalesced batch inline measured slower.

The latency bench gains a steady-state REQ/REP group over a Unix socket
so the IPC gap against tokio is measured on a live connection, not on
connection setup.

Pinned to two cores, before then after:
64B REQ/REP TCP 8.73/8.91 us vs 8.35/8.25 us (tokio 8.09/8.13);
64B REQ/REP IPC 7.04/7.07 us vs 6.37/6.37 us (tokio 5.08/5.15);
PUSH/PULL eager 64B 310 vs 365 Kmsg/s, 4 KiB 233 vs 270 Kmsg/s;
PUSH/PULL coalesced 64B and 4 KiB unchanged within noise.

The rest of the IPC gap is compio itself: a bare compio UnixStream
64-byte ping-pong takes 5.45 us per round trip against 4.35 us for bare
tokio, and monocoque adds the same 0.6 us on either backend.
monocoque's CURVE only ever handshook with itself. Against libzmq
4.3.5 (pyzmq's bundled build) every connection failed, in five places,
each now following RFC 26 and libzmq:

- HELLO carries a 2-byte version and 72 bytes of padding, 200 bytes in
  all; we sent and expected 1 and 71.
- INITIATE carries the client's long-term key C inside its box, and the
  vouch is Box[C' + S](C->S'). We sent C in the clear and vouched for
  c' + s' to the server's long-term key.
- The READY box nonce prefix is "CurveZMQREADY---", not "READY" padded
  with zeros.
- MESSAGE nonces must exceed the last one seen, not match a counter
  starting at 1: libzmq continues its handshake counter, so its first
  client message is nonce 3 and its first server message nonce 2. Our
  senders now start at the same values.
- libzmq sends MESSAGE as a data frame, not a command frame, and after
  the handshake carries PING, PONG and the other commands inside MESSAGE
  boxes with the inner command bit. We did neither; we now do both, and
  refuse a plaintext command on a CURVE connection, which would be
  unauthenticated. DISH JOIN/LEAVE go the same way, so DISH works over
  CURVE.

The repo's Rust CURVE interop tests skip when the system libzmq is
built without CURVE, as it is here, which is how this went unseen. A
pyzmq suite (interop_tests/test_curve_interop.py, run in CI) now covers
both roles, multipart and 70 KB frames, and heartbeats crossing CURVE
in both directions, through a new curve_echo example.

Frames are also sealed and opened in place: encryption reserves the
MESSAGE body in the outbound buffer and encrypts it there, decryption
opens the box inside the read buffer and hands the payload out as a view
of it. That drops three allocations per sent frame and one per received
frame. CURVE DEALER streaming over TCP, pinned to two cores, before then
after: 64B 233/229/225 vs 237/231/226 Kmsg/s, 1 KiB 146/145/144 vs
153/150/148, 16 KiB 27/27/27 vs 28/28/27.
io_uring_setup needs physically contiguous kernel memory for the ring.
With many runtimes alive and memory fragmented it fails with ENOMEM;
the fan-out tests, which start a runtime per subscriber thread, hit
this intermittently during full test runs. LocalRuntime::new now
retries with 256 and then 64 entries instead of failing. A socket keeps
only a few operations in flight per peer and ring size does not change
the cost of an operation.
The idle gate now reports each thread's switches, so a failure says
where the wakeups came from. On smol they came from async-io's reactor
thread: while a block_on runs it answers each reactor tick with up to
ten timed sleeps (50 us to 10 ms) before blocking, so the window's one
timer wake showed as 4 to 11 switches there and occasionally pushed the
total over the budget. That is a bounded cost per wake, not polling, so
smol's budget allows for it. A 1 ms poll would still show about 500.
Passed 25 runs in a row on smol.
Both traits were generic everywhere they were used, so async_trait only
added a boxed future and an allocation per authentication. They are now
plain async fn in trait, and the async-trait dependency is gone.

A handler can no longer be stored as dyn ZapHandler or dyn
PlainAuthHandler; implementations pass it as a concrete type, which every
caller in the workspace already did except the zap_server_demo example.
With write coalescing on and bodies at or above the vectored-write
threshold (32 KiB by default), a batch can hold more segments than one
vectored write takes (64). take_job counted every queued message as in
flight when it handed out the first 64, and complete_job then zeroed the
count while the rest of the batch was still queued. From then on the
peer looked empty: the send HWM no longer pushed back, so a sender
queued without bound, and close() saw nothing pending and returned with
the remainder unwritten. Coalesced 64 KiB PUSH/PULL at the default
threshold never completed; with a 1 KiB threshold 16 KiB messages
stalled after about 6000.

Messages now stay counted as queued until the job that takes their last
segment, so the HWM and drained() only release them once written. A
unit test drives a batch over two jobs, and a socket test sends 2000
coalesced 64 KiB messages then closes; before the fix message 97 never
arrived.
With recv_timeout or send_timeout set, every operation went through the
runtime's timeout combinator, which arms a timer before polling the
operation, so a receive that found a message already queued still paid
for creating and cancelling a timer. The operation is now polled once
first and a timer is armed only if it is pending.

Pinned to two cores, 5 s timeouts on both sockets, before then after:
coalesced PUSH/PULL 64B 4.05/4.23/4.17 vs 7.43/7.35/7.45 Mmsg/s;
eager PUSH/PULL 64B 356/358/355 vs 387/390/389 Kmsg/s;
64B REQ/REP round trip 8.54/8.52/8.50 vs 8.43/8.36/8.39 us.
A PUSH sending to a PULL on another thread over inproc hung every time
once the channel filled (2M messages with the default HWM, unpinned:
12 of 12 runs). Three wakeups could be lost:

- Nothing told a sender when a cross-thread receiver made room. flume
  wakes only senders parked in its own send futures, and a socket only
  checks for room. Each direction now carries a room signal: a sender
  that finds the channel at its HWM registers its waker, and the
  receiver wakes it after taking a message (one atomic swap per
  message).
- A send that found no room parked without watching for it at all;
  only poll() did. Both the load-balanced and the addressed send paths
  now watch before parking, and if room appeared in between (the other
  thread drained meanwhile) they retry instead of parking on a wakeup
  nobody armed. That race was the hang.
- Wakeups::notify skipped waking while an operation was active, trusting
  exit() to see the queued wakeup. From another thread that is a
  store-then-load race on both sides, which acquire/release does not
  order. A notify from a foreign thread now always delivers; one on the
  socket's own thread is ordered by program order and keeps the
  shortcut. Making exit() SeqCst instead cost 7% on coalesced PUSH/PULL.

Wakeups also gets cache lines of its own (align 128): its flags are
written on every operation and read from other threads.

Cross-thread inproc PUSH/PULL 64B, pinned: 4.03/4.24 Mmsg/s before the
alignment, 4.38/4.42 after; unpinned it now completes (3.3 Mmsg/s)
where it hung. Same-thread TCP is unchanged within noise on compio:
coalesced 64B 7.13/7.65/7.37 vs 7.53/7.47/7.13 Mmsg/s, REQ/REP
7.99/8.00/7.93 vs 7.91/7.95/7.92 us. On smol, whose reactor wakes tasks
from its own thread, coalesced 64B costs about 4% (7.86/8.19/8.12 vs
7.74/7.93/7.49 Mmsg/s); eager and latency are unchanged.

New tests run 200000 messages across threads with an HWM of 16, one way
and both ways at once; before the fix they hang.
Swept 8, 16 and 32 KiB with PUSH/PULL (eager and coalesced) and REQ/REP
over TCP, pinned to two cores, three rounds. At 16 KiB bodies a 16 KiB
threshold beats 32 KiB: coalesced 441/440/456 vs 287/300/291 Kmsg/s,
round trip 10.52/10.59/10.66 vs 10.97/10.93/10.93 us, eager no worse.
At 32 KiB bodies the three are equal, and 8 KiB gains nothing over 16.

The option's docs also said it applied only with coalescing off; it
applies either way.
With heartbeats on, each read called Instant::now. A read now only
raises a flag, and the heartbeat tick turns it into a timestamp, so the
clock is read once per interval per peer. Peer timeouts can be noticed
up to one interval later than before.

PUSH/PULL over TCP with a 1 s heartbeat, 8 interleaved runs, medians:
64 B 363K -> 360K msg/s, coalesced 64 B 7.02M -> 7.04M msg/s; both
within noise.
STREAM identities are always minted by the socket as a zero byte and a
32-bit counter, so the identity table is keyed by that counter and
hashed with a single multiply. Bytes a peer never received fail the
lookup as before.

STREAM sending 64 B frames to a raw TCP reader, 8 interleaved runs,
median: 344K -> 358K msg/s (+4%).
A frame that did not fit the read buffer grew it in place, was then
split off it, which turned the buffer shared, and left the reader to
allocate a fresh one: three allocations per long frame, plus a copy
whenever the growth reallocated. Such a frame now gets a buffer of
exactly its size, the following reads fill it, and the message takes
it whole while the read buffer waits to be reused. That leaves one
allocation per long frame, the message's own memory.

The allocation gate gains PAIR 64 KiB frames, coalesced PUSH/PULL with
send timeouts, CURVE PUSH/PULL and STREAM round trips.

TCP probe, 6 interleaved runs, medians:
  64 KiB REQ/REP round trip   25.9 -> 21.3 us
  256 KiB PUSH/PULL           27.5K -> 31K msg/s
  64 KiB and 1 KiB PUSH/PULL  unchanged
The migration guide described the 0.4 API (Option<Vec<Bytes>>, bare
host:port endpoints, no poll). It now covers porting from libzmq with
the current API and lists every breaking change since 0.4.

rfc-compliance.md goes through each ZeroMQ RFC, names where libzmq
departs from the text and which side monocoque follows, and lists the
known gaps.
The option was accepted and never read. A connector now gives up once
that many dials in a row have failed, each reported as ConnectFailed; a
successful connection resets the count. The test fails without the
change: the limited socket connects once a listener appears.
Both options were accepted and never read. An XPUB now sends its welcome
message to each subscriber as it connects, ahead of anything else. A
ROUTER routes the peer of its first connect by connect_routing_id instead
of the identity the peer announces, over TCP, IPC and inproc. Each new
test fails without its change.
The docs check failed on a link from a public doc comment to a private
constant in the spin code.

The changelog, README and performance notes now describe how a socket
chooses between spinning and sleeping, with the measurements across L3
caches that led to it, and the ROUTER routing change. The deployment
guide gains a section on placing threads that talk to each other on
cores under one L3 cache.
On the Ubuntu 26.04 runner the test binary dies of SIGSEGV on compio,
from a clean build, and passes everywhere else it was tried. The log
shows only the signal, so a failed test job now runs the binary again
under gdb and prints the stack of every thread and the kernel version.
The backtrace step failed to install gdb: the runner's package index
pointed at a glibc debug package that is no longer on the mirror. It now
updates the index first and installs gdb without recommended packages.
The crash is inside libzmq's own command queue while it connects. The
instruction, the registers and the queue's fields tell an unaligned
store from a bad pointer.
On the Ubuntu 26.04 runner every test job died of SIGSEGV in the libzmq
interop tests, inside zmq_connect. The backtrace step showed an aligned
AVX store (vmovdqa) to an address aligned to 16 bytes: libzmq 4.3.4
declares its commands 64-byte aligned but takes their queue chunks from
malloc unless HAVE_POSIX_MEMALIGN is defined, and its vendored build
defines another name. The image's compiler targets x86-64-v3, so it
uses the stores the declared alignment allows.

A cargo config now defines HAVE_POSIX_MEMALIGN and the cache line size
for C++ built in this workspace, which is only the libzmq the tests
link. The backtrace step is removed again.
…r it

--cross-l3 runs the pair comparison a second time with the two threads on
cores that share no L3 cache and writes it to logs/mode-cross-l3.log. On a
CPU with one L3 cache the pass is skipped.

The mode timeout applies to each pass, so the hard cap now defaults to the
mode timeout times the number of passes, plus 90 minutes for delivery, the
build and the copy back. Without --cross-l3 that is the 240 minutes it was.
It timed waits both ways and was meant to stop spinning where sleeping
moves messages faster, as between two cores that share no L3 cache.
Measured on an EPYC 7232P against the commit before it, with the two
threads under different L3 caches, it never settled on sleeping: tokio
REQ/REP of 64 B stayed at 29.6 us where sleeping takes about 21.8 us,
tokio IPC latency was 1.9 to 5.9% slower, smol latency 9.7 to 26.6%
slower and tokio pipelined DEALER/ROUTER at 64 B 6.7% slower. Under one
L3 cache it gained nothing and cost about 1% on IPC latency.

The spin is the fixed one again, with its backoff when idle. The docs
keep the cross-L3 measurement and the advice to place two threads that
talk to each other under one L3 cache, and give the pipelined
DEALER/ROUTER gain as measured on that server.
async-executor 1.14.0 was yanked and rustix 1.1.4 has an unsoundness
advisory on its xattr functions (RUSTSEC-2026-0486). cargo audit now
reports only pin-utils as unmaintained, which the lockfile holds for
loom builds of compio-executor.
One run of scripts/bench_all.sh on the i7-1355U, all three backends, the
two-thread benchmarks pinned to cores 0 and 2, on a quiet machine. It
replaces the PUSH/PULL, pipelined, REQ/REP latency, IPC against TCP and
PUB/SUB tables in docs/performance.md and monocoque/BENCHMARKS.md, and
the worker pool tables in BENCHMARKS.md, with the ratios in the prose
recomputed.

Pipelined DEALER/ROUTER at 64 B goes from 3.2 to 3.4 M msg/s to 4.0 to
4.3 M. The worker pool benchmarks run unpinned and moved both ways: fan
out is up, fan-in with eager senders is down.

The rust-zmq columns keep their published figures. libzmq measured lower
in this run, and the higher figures are the fairer comparison.
ZAP and security
- ZAP over ipc:// refused every client: a Unix socket peer has no address
  and the ZAP server dropped a request with an empty one.
- A CURVE server asks the ZAP handler whenever one is running, not only when
  it set a ZAP domain. Without a domain it skipped ZAP, so an allowlist
  handler beside it was never asked.
- with_curve_keypair refuses a public key that does not belong to the secret.
- A peer can no longer announce User-Id, Peer-Address, Socket-Type or
  Identity in its handshake and have them reach the application as metadata.
- monocoque::zmq::security exports the key, credential and ZAP types, so an
  application needs only monocoque-rs. CurveSecretKey::to_bytes is public.

Options
- SocketOptions is non_exhaustive. Options that were stored and never read
  are removed: immediate, probe_router, router_raw, xpub_nodrop, ipv6,
  bind_to_device, rate, recovery_ivl, multicast_hops, multicast_maxtpdu and
  with_unsubscribe.
- connect_timeout and tos take effect.
- A zero handshake_timeout means no limit; it failed every handshake.
- The docs of linger, max_msg_size and the timeouts say what they do.

Wire and multiplexer
- A frame above 256 MiB is refused at its header whatever max_msg_size says;
  a length near usize::MAX overflowed.
- A message with no frames, or a ROUTER message that was only its identity,
  left the socket waiting on bytes that were never written.
- disconnect during a handshake left a peer nothing could remove.
- Heartbeats no longer time out ZMTP 1.0 and 2.0 peers, which have no PING.
- On smol, a ZMTP 1.0 frame split across two reads corrupted the stream.
- A stale peer id in the fair queue could claim a new peer's message.

API
- The public enums that will grow are non_exhaustive.
- tcp://*:port, tcp://host:* and host names are accepted.
- The crates no longer set tracing's release_max_level_info, which capped
  the log level of the whole application, and the benchmarks carry their own
  log setup in place of monocoque::dev_tracing.
curve_demo authorizes client keys through a ZAP handler; it claimed to and
did not. pipeline_worker_pool and borrowed_recv send with write coalescing,
which is what the performance guide cites them for. The paranoid pirate
examples make a worker die while it holds a request, so the client's retry
runs. The brokers use proxy. The interop examples print what they do and no
longer share ports. Receives that only read the message borrow it.

paranoid_pirate_simple and monitoring duplicated other examples and are
removed. New: coalesced_push_pull, inproc_across_threads, stream_socket,
heartbeats, draft_client_server and draft_radio_dish.

tracing and tracing-subscriber become dev-dependencies of monocoque-rs, and
the new tests are registered.
The migration guide lists the construction change and the methods and
options removed since 0.4. The security guide says when the ZAP handler is
asked for each mechanism, and the duplicate docs/security.md is gone. The
quick start samples use LocalRuntime, which is what gives the published
round-trip figures. Reconnect backoff and heartbeats are described as they
are by default. Install snippets carry the zmq feature, commands use the
real package names, and the licence line says MIT. The changelog records
the fixes and marks 1.0.0 as unreleased.
A REQ socket reads from one chosen peer at a time, so it never took the
fair-queue turn a peer was given nor the messages a peer left unread. A
peer that had gone kept its slot and its queue for as long as the socket
lived. It is now freed when it is read to the end, and at the start of
each new request, where nothing it left can still be a wanted reply.

StreamSocket gains the methods the other sockets have for managing and
inspecting themselves: disconnect, unbind, attach, monitor, options,
set_options, update_options, peer_ids, last_endpoint, peer_property,
last_metadata, last_peer, has_message and can_send. Its flush honours
send_timeout.

BENCHMARKS.md no longer describes the recv_into case, which is gone.
Heartbeats
- A PING carried the heartbeat interval as its time-to-live, while an idle
  peer was pinged only every second interval. An idle libzmq peer therefore
  closed the connection again and again. No TTL is sent now unless one is
  set, as in libzmq, and with one a PING goes out every interval.

Write coalescing
- It held back what the protocol itself sends: a PING or PONG, a
  subscription, the greeting messages and a STREAM close waited for a batch
  to fill, and flush did not release the commands. A coalescing socket
  timed its peer out, a coalescing SUB received nothing, and a STREAM close
  dropped the data queued before it. These are written at once now.

Metadata
- The last message of a peer that had already gone arrived with no
  metadata, User-Id included.
- Routing-Id joins the handshake properties a peer cannot set for itself.

Connections
- disconnect matches the endpoint text given to connect before it compares
  addresses, so a host name that now resolves elsewhere still disconnects,
  and it cancels a handshake still running on that connection.
- A connection whose options cannot be applied counts as a failed attempt
  and is reported, where the dial loop redialed it silently for ever.
- A REQ socket left one fair-queue entry behind for each departed peer.

The option doc comments now describe router_handover, routing_id,
connect_routing_id, xsub_verbose_unsubs and the reconnect intervals as the
code behaves. The interop CI job runs the new heartbeat test.
pipeline_worker_pool sent every task before it read a result, on one
task: results backed up through the bounded queues until the send waited
for ever, which a run of a few million tasks showed. The sink is read by
its own task now, and the workers stop through a control socket in place
of one STOP each over round-robin.

plain_auth_demo ran its valid client well into the server's backoff for
repeated failures; the refused client gives up sooner. zap_server_demo no
longer has an arm for NULL peers, which a handler is never asked about.
The interop examples take ports from the OS. The other review items are
accuracy and error handling.

New: conflate, xpub_manual, connect_disconnect, attach_stream,
poll_send_and_recv, draft_scatter_gather, draft_channel and stalled_peer.
The CURVE key line in the security guide compiles. The interop guide has
the curve_demo commands as they are. The migration guide covers the
non-exhaustive enums and the XPUB, XSUB, PAIR and STREAM changes. The
ZMTP version and the libzmq versions tested are stated one way. The bench
peer is marked as written against 0.4. Router handover, correlated
replies, ZAP with no handler and failed handshakes in the monitor are
described as the code behaves. The changelog records the library fixes.
A receive spins before it sleeps so that a reply coming from another
thread is seen the moment it lands. The spin keeps its thread busy, and
a busy thread does not get round to its reactor. Two sockets talking to
each other on one thread therefore each kept the other from being told
its bytes had arrived.

On tokio and smol a spin now ends when another socket on the thread
comes to wait, and stays off for the next 1024 waits. A socket alone on
its thread spins as before, and the send and receive fast paths are not
touched: the check runs only on the way to a wait.

Measured on an i7-1355U, DEALER and ROUTER echoing 64 B messages in
coalesced batches on one thread, per thousand messages:
  smol   611 us before, 219 us now (0.4.2: 236 us)
  tokio  237 us before, 185 us now (0.4.2: 206 us)
smol's coalesced PUSH/PULL at 64 B also went from 467 us to 358 us.
REQ/REP latency, pipelined DEALER/ROUTER, fan-out and fan-in moved by 2%
or less.
A message too long to be stored inline is one reference-counted frame.
Putting the identity in front of it took the frames out as a list and
put them back. It now builds the two-frame message directly.

Pipelined DEALER/ROUTER on a laptop, 256 B: compio 3.42 to 3.29 ms,
tokio 3.39 to 3.26 ms, smol 3.26 to 3.16 ms. 1 KB gains 1 to 3 percent
and 64 B, which is stored inline, is unchanged.
ROUTER send was an async fn that moved the message through its future
several times before it reached the encoder. It now routes and encodes
from a reference and only builds a future when the write has to wait.

The result of an immediate send carried the error inside the enum, so
it was written as one word and copied back as four, which stalls on
store forwarding. It is now a small Copy enum and the error goes to an
out-parameter.

The check that every send and receive makes for pending socket work is
inlined, and the work itself stays out of line.

Pipelined DEALER/ROUTER on a laptop, 10 000 messages of 256 bytes:
tokio 3.23 ms to 2.70 ms, smol 3.14 to 2.84, compio 3.28 to 2.95,
measured together with the benchmark fix that follows.
The ROUTER side asked peer_count() every millisecond before starting
its clock. That left the DEALER, with its first batch already sent,
asleep for a millisecond or two. A core idle that long goes into a deep
sleep, and waking it took about 100 microseconds of the timed run. The
ROUTER now waits for the monitor to report the accepted peer and starts
at once.
Since ROUTER send stopped using it, only the draft sockets call it, so
a build without the draft feature saw dead code and clippy failed in
CI. It now carries the same feature gate as its callers.
bench_pair.py printed a note and carried on when a benchmark did not
build, a run failed or no case matched, and exited 0 every time. A
report with a benchmark missing, or with medians taken over fewer
rounds on one side, looked the same as a whole one.

It now lists what is missing under INCOMPLETE and exits 1. The numbers
it did measure are still printed.
The hourly reaper deleted any tagged server older than four hours, and
a run may be given longer than that. Each server now carries its own
deadline as a second tag. The reaper deletes it once past that, at 24
hours whatever the tag says, and at --max-age when it has no deadline.
A creation time it cannot read still means delete, and now also a
non-zero exit.

A create call whose answer was lost left a server that nothing looked
for until the reaper. The state file is now written before the call,
the exit trap and the watchdog keep looking for the server by name for
a few minutes, --reap takes --server-name, and the final step in CI
uses it when the file holds no ID. The call has a time limit, and a 4xx
answer is taken as no server.

A remote run that died without an exit code, or a server that stopped
answering, was polled until the hard cap. Both now end the run.

Also: the exit trap no longer stops on a failing command, SSH ignores
the user's configuration and forwards nothing, scw cannot read the
reap loop's input, setsid is required and the watchdog is checked
before the server is created, the results directory carries the time
of day, numeric flags with a leading zero are read in base ten, and
the cross-L3 pass runs after an incomplete first pass.

In the workflow the delete step runs before the upload and both job
timeouts leave room for the script's own limits.

Both benchmark scripts are tested in CI against stand-ins for scw,
curl and the benchmark binaries; nothing is rented.
RFC 27 allows 255 bytes for each. With the command name and the two
length bytes that is a HELLO of 518 bytes, and the limit was 512, so
such a client was refused.
tcp://[::1]:* did not parse: the port wildcard sent it to the host
name lookup with its brackets on. connect now refuses a wildcard host
or port, which names somewhere to listen and not somewhere to dial.

A peer that names a TTL in its PING asks to be closed when it sends
nothing for that long. That was looked at only on a socket with a
heartbeat interval. A socket without one now keeps the time of the
last read from such a peer and closes it between one TTL and one and a
half later, without sending PINGs of its own.

An inproc peer's custom metadata could replace Socket-Type, Identity,
User-Id and Peer-Address, which a peer on the wire cannot. Both paths
now share one list of reserved names.
A socket without a heartbeat interval starts a tick when a peer names
a TTL in its PING. Nothing stopped it again, so one PING from a peer
that then left kept an idle socket waking twenty times a second for
good, and any peer could do that.

The tick now follows the peers: it stops when no live peer has named a
TTL, and its period is set again when a peer names a different TTL or
leaves. The idle wake-up test covers the case.
A server whose creation time cannot be read is judged by its own
deadline when it has one, and deleted as before when it has none.
Either way the reap exits non-zero.

A server now lives 40 minutes past the hard cap, not 15: a run stopped
at the cap may take 20 to copy its results back, and the watchdog must
not delete the server under it. A deletion is given 30 minutes.

A 408 or 499 from the create call leaves open whether a server exists,
so it is looked for by name. The state file is removed only for the
server it names, by whole fields: an empty name used to match any
file. The benchmark list is split without file pattern expansion.

In the workflow the credentials go to the steps that run the script
and to no others.

The tests assert that a create case reached the create call, stand in
for ssh, scp and ssh-keygen, and cover the new cases.
A coalescing writer asked the kernel for the segment size before every
batch, because a young connection reports a smaller one for its first
few writes. A pipelined round of 4 KiB messages paid about twelve
getsockopt calls for it. The size is now asked for each batch only while
the answer keeps changing, then at every second, fourth, eighth batch
and so on, up to every 64th. On two cores of a laptop this took the
pipelined DEALER/ROUTER bench from 11.4 to 11.3 ms at 4 KiB and from
38.6 to 37.6 ms at 16 KiB.

Each new socket also counted the CPUs it may run on to decide whether to
spin, which reads a file per level of the cgroup tree, about thirty
system calls. That is now done once per process.
On tokio and smol, a read of 1 KiB or more that did not fill its buffer
was followed at once by another, on the guess that it was a stream with
more on the way. A batch that is answered as a whole is that large too
and nothing follows it, so every batch cost a recv that found nothing:
two per round of a pipelined DEALER/ROUTER exchange.

The reader now goes by what asking again finds. It keeps asking while
that turns up data, takes a large short read as drained after sixteen
misses in a row, and asks again every 64th time in case a stream has
started. A miss costs one recv and a hit saves a sleep and a wakeup, so
it takes a long run of misses to give up: at two in a row a coalesced
16 KiB stream on tokio lost 3%.

On two slow cores of a laptop, tokio's pipelined bench went from 5.02 to
4.70 ms at 256 B, and about 2% faster at 64 B, 1 KiB and 4 KiB.
Streaming throughput on tokio and smol is level. compio reads through
the ring and never made the extra call.

tokio is told the socket is drained straight after the read, before
anything is awaited. Told later, it forgot news of bytes that had
arrived since, and a reader held at its receive HWM waited for an event
the kernel had already delivered.
A sender and a receiver on the two threads of one physical core take
turns on it, and a thread that spins while it waits slows the other.
One pipelined case read 20% slower pinned that way and level on two
cores. bench_pair.py now reads the CPU topology and stops, naming the
two CPUs, when --pin puts two of them on one core.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant