Repository navigation
Conversation
… blocks RouterHub and PubSubHub had no users outside monocoque-core. The futures Stream/Sink adapters only ever returned Unsupported and nothing implemented their traits. SocketOptions::small()/large() and BufferConfig set read buffers below the 32 KiB default and were read by nothing. Delete the probe/debug examples, monocoque-zmtp/bench_results.txt and the interop scripts no workflow or runner references. Every crate now takes its lint policy from [workspace.lints] alone. The blanket cast_possible_truncation allow is gone; each narrowing cast is now a checked conversion. The ZAP metadata encoder clamps the value to u32::MAX so its length prefix always matches the bytes written, and the CURVE command writer reuses append_zmtp_cmd_frame instead of copying the frame twice.
rust-toolchain.toml pins local builds to the CI clippy toolchain (1.98). Every other CI job sets RUSTUP_TOOLCHAIN so it keeps running on the toolchain it installs (stable, beta, nightly, the 1.95 MSRV). dependabot batches weekly minor/patch bumps into one PR per ecosystem and ignores the documented chacha20poly1305/crypto_box aead-0.5 pin. The README Safety section is now the single unsafe inventory and matches the code; CONTRIBUTING and the crate docs point to it instead of making contradictory claims. Drop the README roadmap and the stale phase table in the overview blueprint.
compio-io's default write_vectored writes one iovec per call, so every multi-frame send over IPC on the tokio backend issued one syscall per header and body while TCP took a single writev. Route it through the same write_vectored_from the TCP adapters use, with a test that fails without the override (10 of 60 bytes written).
The PLAIN and CURVE server handshakes passed the literal "unknown" as the ZAP Address frame, so any handler applying IP allow/deny policy saw the same value for every client and the policy was silently disabled. Add a PeerAddress trait to the runtime facade that reports the address the way libzmq does (the peer IP for TCP, empty for IPC and inproc) and require it on the handshake stream bound, so both server exchanges read it from the connection they are authenticating. zap_peer_address runs the full TCP handshake for both mechanisms against a recording ZAP handler and fails with the old literal.
idle_wakeups was a manual RSS harness behind #[ignore], so nothing gated the claim that an idle connection costs no wakeups. The new test parks a recv() on every receive-capable socket of DEALER/ROUTER, REQ/REP, PUSH/PULL, PAIR and PUB/SUB, then sums the context switches of every thread in the process across a 500 ms idle window. Today each case costs 1 (compio), 2 (tokio) or 5 (smol) switches against a budget of 10. XPUB's recv_subscription polls each subscriber under a 1 ms read timeout, which measures at about 470 switches per window; that case is an ignored guard until the poll is removed. docs/perf/baselines/a37409b.md freezes docs/performance.md as released in v0.4.2, alongside the earlier c7d7ead baseline.
These are the pieces the multiplexer is built from. Msg stores one frame of up to 64 bytes inline, one larger frame as a single Bytes, up to four frames in a SmallVec, and a fan-out as one Arc<[Bytes]>. MsgRef is the borrowed view a receive will hand out. LocalQueue is a same-thread SPSC queue over Rc<RefCell<VecDeque>> with one waker per side and no atomics. Capacity is the HWM (0 is unbounded), try_send hands the value back at the mark, send waits there, and a conflating queue keeps only the newest value. transport::Stream is a TCP or IPC stream whose reads and writes go through &Stream, so a peer can keep a read and a write in flight without splitting it. The tokio and smol adapters gain the &TcpStream and &UnixStream impls compio already has. Stream::configure applies SO_SNDBUF/SO_RCVBUF to IPC streams as well as TCP, since Unix sockets have no kernel autotuning.
A socket becomes a set of peers. Each wire peer has one reader loop and one writer loop, boxed once per connection: the reader decodes complete messages into the peer's ready queue and stops reading at the receive HWM, so a slow consumer backpressures only its own peer; the writer drains the peer's outbound queue one job at a time, coalescing only what accumulated behind a write already in flight. inproc peers move Msg values through a LocalQueue on one thread or a flume channel across threads, with no codec. Every per-socket future is a child polled by whichever task touches the socket. A child's waker wakes the user task if it is parked on the socket and the driver otherwise, so a read completing under a waiting recv() runs inline in the user's poll with no task in between. The driver task handles accept, dial and reconnect with jittered backoff, heartbeats when enabled, and linger on close; idle with no heartbeat it takes no wakeups. No loop is ever cancelled by a timeout, so no in-flight read or write is dropped. The core is protocol-agnostic: a Protocol supplies the handshake and a per-connection Codec. The error type gains the ZeroMQ cases callers act on (Again, TimedOut, HostUnreachable, InvalidState, Unsupported) as variants. Tested over a toy length-prefixed protocol on all three backends: TCP round trips, fair queueing, HWM backpressure, reconnect, inproc on one and across threads, linger flush, and shared-encoding fan-out.
wire::Zmtp implements the core Protocol: the handshake reuses the existing NULL/PLAIN/CURVE engine, and ZmtpCodec decodes straight out of a peer's contiguous read buffer. A single frame of up to 64 bytes is copied into the Msg itself, so it costs no allocation or refcount and pins none of the read buffer; a frame longer than what has been read so far reserves its full size up front instead of regrowing a staging buffer. Bodies at or above the vectored threshold go out as their own iovec. PING is answered with a PONG echoing its context, and ZMTP 3.1 SUBSCRIBE/CANCEL surface as subscription messages. sock::Socket<K> carries what every type shares (bind and connect to any mix of tcp/ipc/inproc, options, monitor, timeouts that never cancel I/O, the borrowed receive slot); each kind adds its pattern: PAIR refuses a second peer, ROUTER keeps a SipHash identity table with handover and mandatory routing, REQ/REP enforce alternation and carry the envelope, PUB/XPUB match through one subscription index and fan out one shared encoding, SUB/XSUB filter locally and replay subscriptions to new peers. SubscriptionTrie and the unused pubsub index collapse into one matcher: a sorted prefix vector with per-peer reference counts, an early-exit forward scan, and a per-subscriber cap on distinct prefixes. sock_patterns covers every pattern on tcp, ipc and inproc, and talks to libzmq for REQ->REP, DEALER->ROUTER, PUB->SUB both ways and large-frame PUSH->PULL, on all three backends.
Measured against the old direct REQ/REP path (64 B, loopback, two threads pinned to two cores), the first cut of the multiplexer was 15-20% slower. Under callgrind (tokio backend) it ran 48.4M instructions for 3,200 round trips against the old path's 27.3M. Fixes, in order of effect: - REQ and REP envelopes are encoded as a prefix by the codec instead of being pushed onto the user's message, which turned an inline message into a heap SmallVec on every request. - Received messages are written straight into the socket's receive slot instead of moving a 144-byte Msg through every future layer, and Msg::push/push_front/pop_front work in place on multi-frame messages. - A child woken by the user's own operation no longer schedules the driver task; the operation drains it before returning, and anything left (including a cross-thread wakeup after the last drain) is routed on exit. - Wakeups keeps atomic mirrors of its lock-protected state so the common nothing-to-do paths skip the mutex. - The tokio &TcpStream read clears cached readiness after a short read, as tokio's own poll_read does; without that every message cost one extra recvfrom returning EAGAIN. The idle-wakeup gate counts only voluntary context switches: preemption by other load is not a wakeup the socket caused, and counting it made the gate flap on a busy machine. Result: 40.2M instructions; p50 gap down to about 0.45-0.55 us (6-7%) on both compio and tokio. That is still outside run-to-run noise, so the one-peer gate is not yet met.
The monocoque::zmq facade now re-exports the multiplexer sockets directly and the old per-connection wrappers are gone. Every test, example and bench uses bind/connect, Msg and borrowed receives. New on the multiplexer: - STREAM sockets with libzmq framing, connect and disconnect notifies. - proxy and proxy_steerable. The loop waits on socket readiness and only builds a receive once a message is queued, so no in-flight receive is dropped. PAUSE holds messages in the sockets instead of discarding them and STATISTICS replies with libzmq's eight counters. - Eager delivery: REQ, DEALER, PUSH and ROUTER sends return once the bytes are handed to the kernel. PUB stays non-blocking. - Subscriptions bypass the HWM and are replayed in the greeting when a peer comes up. Write coalescing no longer stalls a sender. Held writes are released at the byte threshold or when the send HWM fills, and a flush no longer turns coalescing off for the rest of the socket's life. Tests: REQ state machine and gate invariants rewritten against the new API, proxy tests with real libzmq peers through the broker, regression tests for coalescing past the HWM and for fan-out to 100 TCP subscribers with one blocked reader.
Only the multiplexer sockets remain. The old socket modules in monocoque-zmtp, the old proxy, the socket trait and the in-process stream adapter are gone, as are the core modules nothing uses any more (inproc, message, message_builder, backpressure, poison, socket_type, config, buffer) and the dependencies they pulled in. ZAP now runs on the multiplexer: the server is a ROUTER bound to inproc://zeromq.zap.01 and the client a DEALER, so each reply reaches the connection that asked. With no handler bound the handshake is still rejected. Fixes found while porting the remaining tests: - A finished child's slot id could be reused, and a later cancel by the old id stopped the new occupant. Child ids now carry a generation. - ROUTER dropped messages a peer had queued before disconnecting, since its identity was deleted at once. The identity now lives until the slot is reused. - The ZMTP decoder rejects command frames with MORE set again. Integration scenarios not already covered elsewhere are ported to monocoque-zmtp/tests/sock_scenarios.rs, unit tests for surviving logic moved with it, and the fuzz targets drive the new codec.
poll waits on any mix of socket kinds for readable and writable readiness. Readiness follows each pattern's rules: REQ is readable only for the reply it is waiting on, SUB does not report messages its filter would drop, and writable means the routing strategy has room for one more message. Polling never dequeues and the caller owns the PollItem slice, so a poll call allocates nothing. Dropping a poll future leaves every socket usable. Each connection's properties (Socket-Type, Identity, Peer-Address, the ZAP User-Id and any custom READY properties) are parsed once at handshake into an Arc'd Metadata on the peer. A received MsgRef borrows it through gets/property/metadata, so Msg stays the same size and the hot path does no extra work. SocketOptions::with_metadata sends custom properties, which libzmq reads with zmq_msg_gets. 64B REQ/REP round trip, interleaved runs pinned to two cores: before 8.93/8.72/8.77 us, after 8.76/8.75/8.73 us.
The greeting now goes out in stages, as libzmq sends it: the signature, then the major version once the peer's signature shows it is not 1.0, then the rest once its revision shows 3.x. A 1.0 or 2.0 peer is recognised before it is sent bytes it cannot parse. The signature padding carries our 1.0 identity frame length so a 1.0 peer reads it as the start of that frame. Revision 1 peers swap ZMTP 2.0 socket type numbers and identity frames and then use ZMTP 3 data framing without commands. 1.0 peers, and revision 0, use the length-counts-flags framing. A publisher subscribes a 1.0 subscriber to everything, since those filter for themselves. Legacy peers get no heartbeat PINGs. A peer can never talk a secured socket down: a socket using PLAIN or CURVE, or NULL with a ZAP domain, closes a 1.0 or 2.0 peer before reading anything past its greeting. The ZMTP 3 decode and encode paths gain one predictable branch per call. 64B REQ/REP round trip, interleaved runs pinned to two cores: before 8.77/9.17/8.83 us, after 8.81/8.75/8.79 us.
with_conflate set a field nothing read. Each peer's ready buffer now keeps only its newest message when conflate is on: an arrival replaces the waiting one, a multipart message whole, and the receive HWM stops applying so the reader never pauses. It covers TCP, IPC and both inproc links, since they all deliver into the same buffer. Publishers ignore it because what they receive is subscriptions, which must all apply. The receive HWM is enforced per peer on every transport by the multiplexer, so its docs now say that: a peer at the mark is not read until the application receives, and only that sender is pushed back. Both HWMs document 0 as unbounded, as in libzmq, and the send HWM says which patterns wait and which drop at the mark. Interleaved runs pinned to two cores, before then after: 64B REQ/REP round trip 8.66/8.61/8.75 us vs 8.62/8.83/8.62 us; 64B coalesced PUSH/PULL 6.99/7.17/7.13 vs 7.13/7.06/7.17 Melem/s.
CLIENT/SERVER, RADIO/DISH, SCATTER/GATHER and CHANNEL, enabled with the draft feature on monocoque-rs. All carry single-part messages: a multipart send fails with InvalidState and a multipart message from a peer is dropped. Each talks only to its partner type, and a ZMTP 1.0 or 2.0 peer is refused since the types need ZMTP 3.1. SERVER hands out a nonzero 32-bit routing id per peer with every received message and routes sends by it, failing with HostUnreachable for an unknown id and Again at the peer's HWM. A departed peer keeps its id until its slot is reused so its last messages still arrive addressed. RADIO fans each message out to the DISH peers that joined its group, reusing one scratch vector per socket. DISH joins exact group names of up to 255 bytes and receives [group, body]. On the wire DISH sends JOIN and LEAVE commands and RADIO sends the group as a frame ahead of the body, as libzmq does. DISH over CURVE is refused for now: its commands would have to be encrypted. The installed libzmq has no draft support, so the tests run monocoque against itself over TCP and inproc.
On the compio backend a write the socket buffer can take at once now goes out as a non-blocking send instead of a ring submission and completion: one syscall and no wakeup. When the buffer is full the ring path takes over as before. Writes over 64 KiB keep going to the ring, since copying a large coalesced batch inline measured slower. The latency bench gains a steady-state REQ/REP group over a Unix socket so the IPC gap against tokio is measured on a live connection, not on connection setup. Pinned to two cores, before then after: 64B REQ/REP TCP 8.73/8.91 us vs 8.35/8.25 us (tokio 8.09/8.13); 64B REQ/REP IPC 7.04/7.07 us vs 6.37/6.37 us (tokio 5.08/5.15); PUSH/PULL eager 64B 310 vs 365 Kmsg/s, 4 KiB 233 vs 270 Kmsg/s; PUSH/PULL coalesced 64B and 4 KiB unchanged within noise. The rest of the IPC gap is compio itself: a bare compio UnixStream 64-byte ping-pong takes 5.45 us per round trip against 4.35 us for bare tokio, and monocoque adds the same 0.6 us on either backend.
monocoque's CURVE only ever handshook with itself. Against libzmq 4.3.5 (pyzmq's bundled build) every connection failed, in five places, each now following RFC 26 and libzmq: - HELLO carries a 2-byte version and 72 bytes of padding, 200 bytes in all; we sent and expected 1 and 71. - INITIATE carries the client's long-term key C inside its box, and the vouch is Box[C' + S](C->S'). We sent C in the clear and vouched for c' + s' to the server's long-term key. - The READY box nonce prefix is "CurveZMQREADY---", not "READY" padded with zeros. - MESSAGE nonces must exceed the last one seen, not match a counter starting at 1: libzmq continues its handshake counter, so its first client message is nonce 3 and its first server message nonce 2. Our senders now start at the same values. - libzmq sends MESSAGE as a data frame, not a command frame, and after the handshake carries PING, PONG and the other commands inside MESSAGE boxes with the inner command bit. We did neither; we now do both, and refuse a plaintext command on a CURVE connection, which would be unauthenticated. DISH JOIN/LEAVE go the same way, so DISH works over CURVE. The repo's Rust CURVE interop tests skip when the system libzmq is built without CURVE, as it is here, which is how this went unseen. A pyzmq suite (interop_tests/test_curve_interop.py, run in CI) now covers both roles, multipart and 70 KB frames, and heartbeats crossing CURVE in both directions, through a new curve_echo example. Frames are also sealed and opened in place: encryption reserves the MESSAGE body in the outbound buffer and encrypts it there, decryption opens the box inside the read buffer and hands the payload out as a view of it. That drops three allocations per sent frame and one per received frame. CURVE DEALER streaming over TCP, pinned to two cores, before then after: 64B 233/229/225 vs 237/231/226 Kmsg/s, 1 KiB 146/145/144 vs 153/150/148, 16 KiB 27/27/27 vs 28/28/27.
io_uring_setup needs physically contiguous kernel memory for the ring. With many runtimes alive and memory fragmented it fails with ENOMEM; the fan-out tests, which start a runtime per subscriber thread, hit this intermittently during full test runs. LocalRuntime::new now retries with 256 and then 64 entries instead of failing. A socket keeps only a few operations in flight per peer and ring size does not change the cost of an operation.
The idle gate now reports each thread's switches, so a failure says where the wakeups came from. On smol they came from async-io's reactor thread: while a block_on runs it answers each reactor tick with up to ten timed sleeps (50 us to 10 ms) before blocking, so the window's one timer wake showed as 4 to 11 switches there and occasionally pushed the total over the budget. That is a bounded cost per wake, not polling, so smol's budget allows for it. A 1 ms poll would still show about 500. Passed 25 runs in a row on smol.
Both traits were generic everywhere they were used, so async_trait only added a boxed future and an allocation per authentication. They are now plain async fn in trait, and the async-trait dependency is gone. A handler can no longer be stored as dyn ZapHandler or dyn PlainAuthHandler; implementations pass it as a concrete type, which every caller in the workspace already did except the zap_server_demo example.
With write coalescing on and bodies at or above the vectored-write threshold (32 KiB by default), a batch can hold more segments than one vectored write takes (64). take_job counted every queued message as in flight when it handed out the first 64, and complete_job then zeroed the count while the rest of the batch was still queued. From then on the peer looked empty: the send HWM no longer pushed back, so a sender queued without bound, and close() saw nothing pending and returned with the remainder unwritten. Coalesced 64 KiB PUSH/PULL at the default threshold never completed; with a 1 KiB threshold 16 KiB messages stalled after about 6000. Messages now stay counted as queued until the job that takes their last segment, so the HWM and drained() only release them once written. A unit test drives a batch over two jobs, and a socket test sends 2000 coalesced 64 KiB messages then closes; before the fix message 97 never arrived.
With recv_timeout or send_timeout set, every operation went through the runtime's timeout combinator, which arms a timer before polling the operation, so a receive that found a message already queued still paid for creating and cancelling a timer. The operation is now polled once first and a timer is armed only if it is pending. Pinned to two cores, 5 s timeouts on both sockets, before then after: coalesced PUSH/PULL 64B 4.05/4.23/4.17 vs 7.43/7.35/7.45 Mmsg/s; eager PUSH/PULL 64B 356/358/355 vs 387/390/389 Kmsg/s; 64B REQ/REP round trip 8.54/8.52/8.50 vs 8.43/8.36/8.39 us.
A PUSH sending to a PULL on another thread over inproc hung every time once the channel filled (2M messages with the default HWM, unpinned: 12 of 12 runs). Three wakeups could be lost: - Nothing told a sender when a cross-thread receiver made room. flume wakes only senders parked in its own send futures, and a socket only checks for room. Each direction now carries a room signal: a sender that finds the channel at its HWM registers its waker, and the receiver wakes it after taking a message (one atomic swap per message). - A send that found no room parked without watching for it at all; only poll() did. Both the load-balanced and the addressed send paths now watch before parking, and if room appeared in between (the other thread drained meanwhile) they retry instead of parking on a wakeup nobody armed. That race was the hang. - Wakeups::notify skipped waking while an operation was active, trusting exit() to see the queued wakeup. From another thread that is a store-then-load race on both sides, which acquire/release does not order. A notify from a foreign thread now always delivers; one on the socket's own thread is ordered by program order and keeps the shortcut. Making exit() SeqCst instead cost 7% on coalesced PUSH/PULL. Wakeups also gets cache lines of its own (align 128): its flags are written on every operation and read from other threads. Cross-thread inproc PUSH/PULL 64B, pinned: 4.03/4.24 Mmsg/s before the alignment, 4.38/4.42 after; unpinned it now completes (3.3 Mmsg/s) where it hung. Same-thread TCP is unchanged within noise on compio: coalesced 64B 7.13/7.65/7.37 vs 7.53/7.47/7.13 Mmsg/s, REQ/REP 7.99/8.00/7.93 vs 7.91/7.95/7.92 us. On smol, whose reactor wakes tasks from its own thread, coalesced 64B costs about 4% (7.86/8.19/8.12 vs 7.74/7.93/7.49 Mmsg/s); eager and latency are unchanged. New tests run 200000 messages across threads with an HWM of 16, one way and both ways at once; before the fix they hang.
Swept 8, 16 and 32 KiB with PUSH/PULL (eager and coalesced) and REQ/REP over TCP, pinned to two cores, three rounds. At 16 KiB bodies a 16 KiB threshold beats 32 KiB: coalesced 441/440/456 vs 287/300/291 Kmsg/s, round trip 10.52/10.59/10.66 vs 10.97/10.93/10.93 us, eager no worse. At 32 KiB bodies the three are equal, and 8 KiB gains nothing over 16. The option's docs also said it applied only with coalescing off; it applies either way.
With heartbeats on, each read called Instant::now. A read now only raises a flag, and the heartbeat tick turns it into a timestamp, so the clock is read once per interval per peer. Peer timeouts can be noticed up to one interval later than before. PUSH/PULL over TCP with a 1 s heartbeat, 8 interleaved runs, medians: 64 B 363K -> 360K msg/s, coalesced 64 B 7.02M -> 7.04M msg/s; both within noise.
STREAM identities are always minted by the socket as a zero byte and a 32-bit counter, so the identity table is keyed by that counter and hashed with a single multiply. Bytes a peer never received fail the lookup as before. STREAM sending 64 B frames to a raw TCP reader, 8 interleaved runs, median: 344K -> 358K msg/s (+4%).
A frame that did not fit the read buffer grew it in place, was then split off it, which turned the buffer shared, and left the reader to allocate a fresh one: three allocations per long frame, plus a copy whenever the growth reallocated. Such a frame now gets a buffer of exactly its size, the following reads fill it, and the message takes it whole while the read buffer waits to be reused. That leaves one allocation per long frame, the message's own memory. The allocation gate gains PAIR 64 KiB frames, coalesced PUSH/PULL with send timeouts, CURVE PUSH/PULL and STREAM round trips. TCP probe, 6 interleaved runs, medians: 64 KiB REQ/REP round trip 25.9 -> 21.3 us 256 KiB PUSH/PULL 27.5K -> 31K msg/s 64 KiB and 1 KiB PUSH/PULL unchanged
The migration guide described the 0.4 API (Option<Vec<Bytes>>, bare host:port endpoints, no poll). It now covers porting from libzmq with the current API and lists every breaking change since 0.4. rfc-compliance.md goes through each ZeroMQ RFC, names where libzmq departs from the text and which side monocoque follows, and lists the known gaps.
The option was accepted and never read. A connector now gives up once that many dials in a row have failed, each reported as ConnectFailed; a successful connection resets the count. The test fails without the change: the limited socket connects once a listener appears.
Both options were accepted and never read. An XPUB now sends its welcome message to each subscriber as it connects, ahead of anything else. A ROUTER routes the peer of its first connect by connect_routing_id instead of the identity the peer announces, over TCP, IPC and inproc. Each new test fails without its change.
The docs check failed on a link from a public doc comment to a private constant in the spin code. The changelog, README and performance notes now describe how a socket chooses between spinning and sleeping, with the measurements across L3 caches that led to it, and the ROUTER routing change. The deployment guide gains a section on placing threads that talk to each other on cores under one L3 cache.
On the Ubuntu 26.04 runner the test binary dies of SIGSEGV on compio, from a clean build, and passes everywhere else it was tried. The log shows only the signal, so a failed test job now runs the binary again under gdb and prints the stack of every thread and the kernel version.
The backtrace step failed to install gdb: the runner's package index pointed at a glibc debug package that is no longer on the mirror. It now updates the index first and installs gdb without recommended packages.
The crash is inside libzmq's own command queue while it connects. The instruction, the registers and the queue's fields tell an unaligned store from a bad pointer.
On the Ubuntu 26.04 runner every test job died of SIGSEGV in the libzmq interop tests, inside zmq_connect. The backtrace step showed an aligned AVX store (vmovdqa) to an address aligned to 16 bytes: libzmq 4.3.4 declares its commands 64-byte aligned but takes their queue chunks from malloc unless HAVE_POSIX_MEMALIGN is defined, and its vendored build defines another name. The image's compiler targets x86-64-v3, so it uses the stores the declared alignment allows. A cargo config now defines HAVE_POSIX_MEMALIGN and the cache line size for C++ built in this workspace, which is only the libzmq the tests link. The backtrace step is removed again.
…r it --cross-l3 runs the pair comparison a second time with the two threads on cores that share no L3 cache and writes it to logs/mode-cross-l3.log. On a CPU with one L3 cache the pass is skipped. The mode timeout applies to each pass, so the hard cap now defaults to the mode timeout times the number of passes, plus 90 minutes for delivery, the build and the copy back. Without --cross-l3 that is the 240 minutes it was.
It timed waits both ways and was meant to stop spinning where sleeping moves messages faster, as between two cores that share no L3 cache. Measured on an EPYC 7232P against the commit before it, with the two threads under different L3 caches, it never settled on sleeping: tokio REQ/REP of 64 B stayed at 29.6 us where sleeping takes about 21.8 us, tokio IPC latency was 1.9 to 5.9% slower, smol latency 9.7 to 26.6% slower and tokio pipelined DEALER/ROUTER at 64 B 6.7% slower. Under one L3 cache it gained nothing and cost about 1% on IPC latency. The spin is the fixed one again, with its backoff when idle. The docs keep the cross-L3 measurement and the advice to place two threads that talk to each other under one L3 cache, and give the pipelined DEALER/ROUTER gain as measured on that server.
async-executor 1.14.0 was yanked and rustix 1.1.4 has an unsoundness advisory on its xattr functions (RUSTSEC-2026-0486). cargo audit now reports only pin-utils as unmaintained, which the lockfile holds for loom builds of compio-executor.
One run of scripts/bench_all.sh on the i7-1355U, all three backends, the two-thread benchmarks pinned to cores 0 and 2, on a quiet machine. It replaces the PUSH/PULL, pipelined, REQ/REP latency, IPC against TCP and PUB/SUB tables in docs/performance.md and monocoque/BENCHMARKS.md, and the worker pool tables in BENCHMARKS.md, with the ratios in the prose recomputed. Pipelined DEALER/ROUTER at 64 B goes from 3.2 to 3.4 M msg/s to 4.0 to 4.3 M. The worker pool benchmarks run unpinned and moved both ways: fan out is up, fan-in with eager senders is down. The rust-zmq columns keep their published figures. libzmq measured lower in this run, and the higher figures are the fairer comparison.
ZAP and security - ZAP over ipc:// refused every client: a Unix socket peer has no address and the ZAP server dropped a request with an empty one. - A CURVE server asks the ZAP handler whenever one is running, not only when it set a ZAP domain. Without a domain it skipped ZAP, so an allowlist handler beside it was never asked. - with_curve_keypair refuses a public key that does not belong to the secret. - A peer can no longer announce User-Id, Peer-Address, Socket-Type or Identity in its handshake and have them reach the application as metadata. - monocoque::zmq::security exports the key, credential and ZAP types, so an application needs only monocoque-rs. CurveSecretKey::to_bytes is public. Options - SocketOptions is non_exhaustive. Options that were stored and never read are removed: immediate, probe_router, router_raw, xpub_nodrop, ipv6, bind_to_device, rate, recovery_ivl, multicast_hops, multicast_maxtpdu and with_unsubscribe. - connect_timeout and tos take effect. - A zero handshake_timeout means no limit; it failed every handshake. - The docs of linger, max_msg_size and the timeouts say what they do. Wire and multiplexer - A frame above 256 MiB is refused at its header whatever max_msg_size says; a length near usize::MAX overflowed. - A message with no frames, or a ROUTER message that was only its identity, left the socket waiting on bytes that were never written. - disconnect during a handshake left a peer nothing could remove. - Heartbeats no longer time out ZMTP 1.0 and 2.0 peers, which have no PING. - On smol, a ZMTP 1.0 frame split across two reads corrupted the stream. - A stale peer id in the fair queue could claim a new peer's message. API - The public enums that will grow are non_exhaustive. - tcp://*:port, tcp://host:* and host names are accepted. - The crates no longer set tracing's release_max_level_info, which capped the log level of the whole application, and the benchmarks carry their own log setup in place of monocoque::dev_tracing.
curve_demo authorizes client keys through a ZAP handler; it claimed to and did not. pipeline_worker_pool and borrowed_recv send with write coalescing, which is what the performance guide cites them for. The paranoid pirate examples make a worker die while it holds a request, so the client's retry runs. The brokers use proxy. The interop examples print what they do and no longer share ports. Receives that only read the message borrow it. paranoid_pirate_simple and monitoring duplicated other examples and are removed. New: coalesced_push_pull, inproc_across_threads, stream_socket, heartbeats, draft_client_server and draft_radio_dish. tracing and tracing-subscriber become dev-dependencies of monocoque-rs, and the new tests are registered.
The migration guide lists the construction change and the methods and options removed since 0.4. The security guide says when the ZAP handler is asked for each mechanism, and the duplicate docs/security.md is gone. The quick start samples use LocalRuntime, which is what gives the published round-trip figures. Reconnect backoff and heartbeats are described as they are by default. Install snippets carry the zmq feature, commands use the real package names, and the licence line says MIT. The changelog records the fixes and marks 1.0.0 as unreleased.
A REQ socket reads from one chosen peer at a time, so it never took the fair-queue turn a peer was given nor the messages a peer left unread. A peer that had gone kept its slot and its queue for as long as the socket lived. It is now freed when it is read to the end, and at the start of each new request, where nothing it left can still be a wanted reply. StreamSocket gains the methods the other sockets have for managing and inspecting themselves: disconnect, unbind, attach, monitor, options, set_options, update_options, peer_ids, last_endpoint, peer_property, last_metadata, last_peer, has_message and can_send. Its flush honours send_timeout. BENCHMARKS.md no longer describes the recv_into case, which is gone.
Heartbeats - A PING carried the heartbeat interval as its time-to-live, while an idle peer was pinged only every second interval. An idle libzmq peer therefore closed the connection again and again. No TTL is sent now unless one is set, as in libzmq, and with one a PING goes out every interval. Write coalescing - It held back what the protocol itself sends: a PING or PONG, a subscription, the greeting messages and a STREAM close waited for a batch to fill, and flush did not release the commands. A coalescing socket timed its peer out, a coalescing SUB received nothing, and a STREAM close dropped the data queued before it. These are written at once now. Metadata - The last message of a peer that had already gone arrived with no metadata, User-Id included. - Routing-Id joins the handshake properties a peer cannot set for itself. Connections - disconnect matches the endpoint text given to connect before it compares addresses, so a host name that now resolves elsewhere still disconnects, and it cancels a handshake still running on that connection. - A connection whose options cannot be applied counts as a failed attempt and is reported, where the dial loop redialed it silently for ever. - A REQ socket left one fair-queue entry behind for each departed peer. The option doc comments now describe router_handover, routing_id, connect_routing_id, xsub_verbose_unsubs and the reconnect intervals as the code behaves. The interop CI job runs the new heartbeat test.
pipeline_worker_pool sent every task before it read a result, on one task: results backed up through the bounded queues until the send waited for ever, which a run of a few million tasks showed. The sink is read by its own task now, and the workers stop through a control socket in place of one STOP each over round-robin. plain_auth_demo ran its valid client well into the server's backoff for repeated failures; the refused client gives up sooner. zap_server_demo no longer has an arm for NULL peers, which a handler is never asked about. The interop examples take ports from the OS. The other review items are accuracy and error handling. New: conflate, xpub_manual, connect_disconnect, attach_stream, poll_send_and_recv, draft_scatter_gather, draft_channel and stalled_peer.
The CURVE key line in the security guide compiles. The interop guide has the curve_demo commands as they are. The migration guide covers the non-exhaustive enums and the XPUB, XSUB, PAIR and STREAM changes. The ZMTP version and the libzmq versions tested are stated one way. The bench peer is marked as written against 0.4. Router handover, correlated replies, ZAP with no handler and failed handshakes in the monitor are described as the code behaves. The changelog records the library fixes.
A receive spins before it sleeps so that a reply coming from another thread is seen the moment it lands. The spin keeps its thread busy, and a busy thread does not get round to its reactor. Two sockets talking to each other on one thread therefore each kept the other from being told its bytes had arrived. On tokio and smol a spin now ends when another socket on the thread comes to wait, and stays off for the next 1024 waits. A socket alone on its thread spins as before, and the send and receive fast paths are not touched: the check runs only on the way to a wait. Measured on an i7-1355U, DEALER and ROUTER echoing 64 B messages in coalesced batches on one thread, per thousand messages: smol 611 us before, 219 us now (0.4.2: 236 us) tokio 237 us before, 185 us now (0.4.2: 206 us) smol's coalesced PUSH/PULL at 64 B also went from 467 us to 358 us. REQ/REP latency, pipelined DEALER/ROUTER, fan-out and fan-in moved by 2% or less.
A message too long to be stored inline is one reference-counted frame. Putting the identity in front of it took the frames out as a list and put them back. It now builds the two-frame message directly. Pipelined DEALER/ROUTER on a laptop, 256 B: compio 3.42 to 3.29 ms, tokio 3.39 to 3.26 ms, smol 3.26 to 3.16 ms. 1 KB gains 1 to 3 percent and 64 B, which is stored inline, is unchanged.
ROUTER send was an async fn that moved the message through its future several times before it reached the encoder. It now routes and encodes from a reference and only builds a future when the write has to wait. The result of an immediate send carried the error inside the enum, so it was written as one word and copied back as four, which stalls on store forwarding. It is now a small Copy enum and the error goes to an out-parameter. The check that every send and receive makes for pending socket work is inlined, and the work itself stays out of line. Pipelined DEALER/ROUTER on a laptop, 10 000 messages of 256 bytes: tokio 3.23 ms to 2.70 ms, smol 3.14 to 2.84, compio 3.28 to 2.95, measured together with the benchmark fix that follows.
The ROUTER side asked peer_count() every millisecond before starting its clock. That left the DEALER, with its first batch already sent, asleep for a millisecond or two. A core idle that long goes into a deep sleep, and waking it took about 100 microseconds of the timed run. The ROUTER now waits for the monitor to report the accepted peer and starts at once.
Since ROUTER send stopped using it, only the draft sockets call it, so a build without the draft feature saw dead code and clippy failed in CI. It now carries the same feature gate as its callers.
bench_pair.py printed a note and carried on when a benchmark did not build, a run failed or no case matched, and exited 0 every time. A report with a benchmark missing, or with medians taken over fewer rounds on one side, looked the same as a whole one. It now lists what is missing under INCOMPLETE and exits 1. The numbers it did measure are still printed.
The hourly reaper deleted any tagged server older than four hours, and a run may be given longer than that. Each server now carries its own deadline as a second tag. The reaper deletes it once past that, at 24 hours whatever the tag says, and at --max-age when it has no deadline. A creation time it cannot read still means delete, and now also a non-zero exit. A create call whose answer was lost left a server that nothing looked for until the reaper. The state file is now written before the call, the exit trap and the watchdog keep looking for the server by name for a few minutes, --reap takes --server-name, and the final step in CI uses it when the file holds no ID. The call has a time limit, and a 4xx answer is taken as no server. A remote run that died without an exit code, or a server that stopped answering, was polled until the hard cap. Both now end the run. Also: the exit trap no longer stops on a failing command, SSH ignores the user's configuration and forwards nothing, scw cannot read the reap loop's input, setsid is required and the watchdog is checked before the server is created, the results directory carries the time of day, numeric flags with a leading zero are read in base ten, and the cross-L3 pass runs after an incomplete first pass. In the workflow the delete step runs before the upload and both job timeouts leave room for the script's own limits. Both benchmark scripts are tested in CI against stand-ins for scw, curl and the benchmark binaries; nothing is rented.
RFC 27 allows 255 bytes for each. With the command name and the two length bytes that is a HELLO of 518 bytes, and the limit was 512, so such a client was refused.
tcp://[::1]:* did not parse: the port wildcard sent it to the host name lookup with its brackets on. connect now refuses a wildcard host or port, which names somewhere to listen and not somewhere to dial. A peer that names a TTL in its PING asks to be closed when it sends nothing for that long. That was looked at only on a socket with a heartbeat interval. A socket without one now keeps the time of the last read from such a peer and closes it between one TTL and one and a half later, without sending PINGs of its own. An inproc peer's custom metadata could replace Socket-Type, Identity, User-Id and Peer-Address, which a peer on the wire cannot. Both paths now share one list of reserved names.
A socket without a heartbeat interval starts a tick when a peer names a TTL in its PING. Nothing stopped it again, so one PING from a peer that then left kept an idle socket waking twenty times a second for good, and any peer could do that. The tick now follows the peers: it stops when no live peer has named a TTL, and its period is set again when a peer names a different TTL or leaves. The idle wake-up test covers the case.
A server whose creation time cannot be read is judged by its own deadline when it has one, and deleted as before when it has none. Either way the reap exits non-zero. A server now lives 40 minutes past the hard cap, not 15: a run stopped at the cap may take 20 to copy its results back, and the watchdog must not delete the server under it. A deletion is given 30 minutes. A 408 or 499 from the create call leaves open whether a server exists, so it is looked for by name. The state file is removed only for the server it names, by whole fields: an empty name used to match any file. The benchmark list is split without file pattern expansion. In the workflow the credentials go to the steps that run the script and to no others. The tests assert that a create case reached the create call, stand in for ssh, scp and ssh-keygen, and cover the new cases.
A coalescing writer asked the kernel for the segment size before every batch, because a young connection reports a smaller one for its first few writes. A pipelined round of 4 KiB messages paid about twelve getsockopt calls for it. The size is now asked for each batch only while the answer keeps changing, then at every second, fourth, eighth batch and so on, up to every 64th. On two cores of a laptop this took the pipelined DEALER/ROUTER bench from 11.4 to 11.3 ms at 4 KiB and from 38.6 to 37.6 ms at 16 KiB. Each new socket also counted the CPUs it may run on to decide whether to spin, which reads a file per level of the cgroup tree, about thirty system calls. That is now done once per process.
On tokio and smol, a read of 1 KiB or more that did not fill its buffer was followed at once by another, on the guess that it was a stream with more on the way. A batch that is answered as a whole is that large too and nothing follows it, so every batch cost a recv that found nothing: two per round of a pipelined DEALER/ROUTER exchange. The reader now goes by what asking again finds. It keeps asking while that turns up data, takes a large short read as drained after sixteen misses in a row, and asks again every 64th time in case a stream has started. A miss costs one recv and a hit saves a sleep and a wakeup, so it takes a long run of misses to give up: at two in a row a coalesced 16 KiB stream on tokio lost 3%. On two slow cores of a laptop, tokio's pipelined bench went from 5.02 to 4.70 ms at 256 B, and about 2% faster at 64 B, 1 KiB and 4 KiB. Streaming throughput on tokio and smol is level. compio reads through the ring and never made the extra call. tokio is told the socket is drained straight after the read, before anything is awaited. Told later, it forgot news of bytes that had arrived since, and a reader held at its receive HWM waited for an event the kernel had already delivered.
A sender and a receiver on the two threads of one physical core take turns on it, and a thread that spins while it waits slows the other. One pipelined case read 20% slower pinned that way and level on two cores. bench_pair.py now reads the CPU topology and stops, naming the two CPUs, when --pin puts two of them on one core.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.