Skip to content

Write far messages straight into the node buffer and read them from the envelope blob - #204

Merged
resetius merged 3 commits into
masterfrom
far-direct
Nov 11, 2026
Merged

resetius merged 3 commits into
masterfrom
far-direct

Conversation

@resetius

@resetius resetius commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Non-POD far messages are now written straight into the node's outbound buffer and read straight from the received envelope blob. The old path built each payload in an std::ostringstream, copied it out three times through oss.str(), then copied it into the node buffer. On receive it copied the blob into an std::string and then into an std::istringstream.

Why

The QumirDB distributed engine sends the rows of its shuffle as far messages between nodes; one TPC-H SF100 query moves up to 25 GB. All of it went through the stream path on the single actor thread. With this change, the actor-thread CPU of TPC-H SF100 q17 on 4 nodes x 4 threads goes from 4.8 to 2.9 s per node, and the eight shuffle-heavy queries run 9% faster in total.

What changed

Before, a non-POD message specialized:

template<> void SerializeToStream<T>(const T&, std::ostringstream&);
template<> void DeserializeFromStream<T>(T&, std::istringstream&);

Now it specializes:

template<> size_t SerializedSize<T>(const T&);                // exact payload size
template<> void SerializeTo<T>(const T&, std::span<char> out); // writes exactly that many bytes
template<> void DeserializeFrom<T>(T&, TBlob& blob);          // reads blob.Data / blob.Size, may take blob.Data
  • SerializeFarInplace reserves sizeof(THeader) + SerializedSize(obj) bytes with Acquire, writes the header, lets SerializeTo fill the payload in place, then calls Commit. out may be unaligned.
  • SerializeFar (the TMessagesFactory path) allocates SerializedSize bytes and fills them with SerializeTo.
  • A payload over 4 GiB throws std::length_error before anything is reserved, because THeader::Size is 32-bit.
  • DeserializeFar now takes TBlob&, so DeserializeFrom can take ownership of the bytes. When it does, DeserializeFar resets blob.Size to 0, so the blob that TBehavior passes to Receive is consistently empty (it cannot be forwarded).
  • Blob ownership. DeserializeFrom may move blob.Data into the message and keep pointers into the received bytes instead of copying them. The blob is then empty. If it does not take the bytes, the caller frees the blob as before. TZeroCopyEnvelopeReader allocates each payload with operator new, so the bytes are 16-byte aligned and a message can point at typed data inside them. TZeroCopyEnvelopeReaderV2 hands out blobs inside its chunks that are not aligned and must not outlive the reader. Nothing in the repository uses it.
  • TNode::Acquire fix for a throwing serializer. SerializeTo is user code, and it may throw after Acquire. Until now, an Acquire without a Commit shifted every later message: the bytes reserved for the failed message went to the peer and broke the stream. Acquire now always starts right after the committed bytes, so the next Acquire reuses a reservation that was never committed. TNode::Send and SerializeFarInplace make one Acquire per Commit, so their behaviour does not change.
  • The stream path is removed.

API break

This is a source-incompatible change for every non-POD far message. All users in this repository are updated: tests/test_actors.cpp, examples/behavior_actors.cpp, README.md and coroio/actors/README.md. Downstream code has to port its SerializeToStream/DeserializeFromStream specializations. DeserializeFar now takes TBlob& for POD messages too, so a caller that holds a const TBlob& must switch to TBlob& (QumirDB's test helper ReadMessage had to). QumirLabs/qumirdb#553 does this for QumirDB's distributed messages and keeps its draft status until this change is merged.

Tests

New tests in test_actors:

  • test_serialize_far_inplace_round_trip: two messages, one of them empty, go through the in-place path and the envelope reader, and the payload is exactly SerializedSize bytes.
  • test_deserialize_far_takes_blob: DeserializeFrom takes the blob's bytes. The message then points at the received memory and the blob is empty.
  • test_far_message_across_yield: a 5 MiB message (more than the 2 MiB budget of one inbound turn) and a small one after it are written by SerializeFarInplace, sent over a socket and decoded by TBehavior.
  • test_node_drops_failed_serialization: a serializer throws after Acquire, and the next message is still the first one on the wire. This test fails with the old Acquire.

All ctest suites pass on macOS (RelWithDebInfo) and on Linux aarch64 with ASan/LSan (-fsanitize=address, detect_leaks=1).

Micro-benchmark

Each iteration does a far round trip of a message that holds one std::string payload: SerializeFarInplace into a vector-backed stream, TZeroCopyEnvelopeReader::Push/Pop, then DeserializeFar. Apple M-series, clang -O2, median of 3 runs, GB/s of payload. The benchmark program is not committed.

payload end to end, old end to end, new send only, old send only, new
64 KiB 5.7 17.4 8.8 99.8
1 MiB 6.3 18.4 9.3 71.5
8 MiB 5.8 17.9 8.9 72.7

The new send is one memcpy into the reserved region, and the buffer stays hot in cache. The new receive still includes the envelope reader's copy out of its ring buffer, plus this message's own copy into std::string. A message that takes the blob, as QumirDB's row chunks do, skips that second copy.

Review follow-up (a964550): DeserializeFar resets the size of a taken blob; the DeserializeFrom docs state the alignment and lifetime of the received bytes; a test checks that the 4 GiB limit throws before anything is reserved; the comment on empty POD messages is corrected; SerializeFarInplace writes THeader with memcpy, because its position in the buffer may be unaligned.

TODOs left for later (b34ec22, comments only): the POD payload in SerializeFarInplace is still constructed with placement new at a position that may be unaligned; TNode::Send serializes a blob again even when it is already in Far form, e.g. a received non-POD message that is Forward-ed to another node.

@github-actions

Copy link
Copy Markdown

Unit Test Results (Linux)

329 tests  +4   329 ✅ +4   24s ⏱️ -1s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 11b800f. ± Comparison against base commit 44544ec.

@github-actions

Copy link
Copy Markdown

Unit Test Results (MacOS)

250 tests  +4   250 ✅ +4   31s ⏱️ +4s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 11b800f. ± Comparison against base commit 44544ec.

@github-actions

Copy link
Copy Markdown

Unit Test Results (Windows)

171 tests  +4   171 ✅ +4   43s ⏱️ ±0s
  4 suites ±0     0 💤 ±0 
  4 files   ±0     0 ❌ ±0 

Results for commit 11b800f. ± Comparison against base commit 44544ec.

@resetius

Copy link
Copy Markdown
Owner Author

Review: far messages written in place / read from the blob

Verdict: LGTM. No P1/P2 findings. A few P3 notes, mostly docs.

Findings

P3 — the "empty" blob keeps its size (coroio/actors/messages.hpp:349-350, :376-383, tests/test_actors.cpp:569-571)
The doc says the blob "is then empty" after DeserializeFrom takes blob.Data. But DeserializeFar does not reset blob.Size, and the test TBlobView does not reset it either. TBehavior then passes this blob to Receive (coroio/actors/actor.hpp:563-566) with Data == nullptr and Size still equal to the payload size. A Receive that reads or forwards the blob (ctx->Forward(to, id, std::move(blob))) gets an inconsistent blob. QumirDB#553 resets Size itself. Suggestion: if (!blob.Data) { blob.Size = 0; } in DeserializeFar, and one line in the doc: a message that takes the bytes cannot be forwarded through its blob.

P3 — the alignment and lifetime of the received bytes are only in the PR body (coroio/actors/messages.hpp:344-358, coroio/actors/README.md:133)
The default TZeroCopyEnvelopeReader (and TEnvelopeReader) allocate with ::operator new, so the bytes are aligned. TZeroCopyEnvelopeReaderV2 can still be selected with Serve<TSocket, TZeroCopyEnvelopeReaderV2>. Its blobs are unaligned and must not outlive the reader. That limit matters more now, because a message may keep the bytes. One sentence in the DeserializeFrom doc would make this part of the contract. QumirDB#553 already copies an unaligned blob, so this is a doc change only.

P3 — the API break also affects POD callers. DeserializeFar now takes TBlob&, so code that holds a const TBlob& no longer compiles, even for POD messages. QumirDB's test ReadMessage(const TBlob&) had to change in #553. The "API break" section could say this.

P3 — the 4 GiB check has no test (coroio/actors/messages.hpp:249-257)
This test is cheap. Make SerializedSize return size_t(1) << 32, then check that SerializeFarInplace throws std::length_error and that the stream is unchanged. Nothing is allocated.

P3 (nit, old text) coroio/actors/messages.hpp:368-370: "Non-POD or empty POD: calls DeserializeFrom<T>" is not correct. An empty POD goes through DeserializeNear and returns T{}.

P3 (already on master, optional) coroio/actors/messages.hpp:319 (and :310): new (buf.data()) THeader{...} creates the header at CommittedBytes. After a non-POD payload whose size is not a multiple of 4, that offset is not aligned for THeader. For example, the second header in test_serialize_far_inplace_round_trip is at offset 37. The standard calls this UB, but it works on x86-64 and arm64. TNode::Send already writes its header with memcpy, and the same would work here.

What was verified

  • Read the whole diff, plus TNode, TBehavior::TryHandleMessage, the three envelope readers, and TActorSystem::Send / InboundConnection.
  • SerializeFarInplace: the header Size is the payload size. The 32-bit check runs before Acquire. SerializeFar frees its buffer if SerializeTo throws.
  • Acquire/Commit: the only callers are TNode::Send and SerializeFarInplace. Each makes one Acquire per Commit. A partial Commit behaves as before. DoDrain sends only CommittedBytes, so a reservation that was not committed never reaches the wire. QumirDB's TMemoryNode also makes one Acquire per Commit.
  • DeserializeFar ownership: if DeserializeFrom throws after taking the bytes, the message destructor frees them once and blob.Data is null. There is no double free and no leak.
  • POD path: the logic is unchanged. Only the signature changed.
  • Old API: git grep finds no remaining SerializeToStream / DeserializeFromStream / stream use in coroio/, tests/, examples/, bench/ or the READMEs. Removing <sstream> from messages.hpp does not break any other header.
  • macOS, RelWithDebInfo: all targets build with no warnings, and test_actors passes 5 times in a row. Linux aarch64 (GCC 15, -fsanitize=address, detect_leaks=1): test_actors passes 5 times in a row.
  • With master's node.hpp, test_node_drops_failed_serialization fails (tests/test_actors.cpp:1274: the failed message's header goes on the wire first). With this PR it passes. The other new tests add coverage of the new API and are not expected to fail on master.
  • CI: build (Linux ASan), macos-build and the Linux/macOS/Windows unit test results are green. Every step of windows-build succeeded, and the job was still being finalized when this review was written.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant