FEATURE-ci-darwin-focused-diagnostics: capture six failures in one native batch

Add temporary scalar observations at unchanged test assertions and timer/readiness/storage boundaries. Prepare an existing dispatch-capable release workflow as a manual-only read-only Darwin diagnostic job; bind its source and archive identity and launch each selected real program once with LLDB failure evidence. This diagnostic branch must never merge into v3.

Verification: enabled/disabled C++20 syntax and format checks, logger-only errno/operand smoke checks, changed-file cpplint, actionlint, workflow shell and embedded Python/LLDB parsing, and git diff checks passed locally. Darwin compilation/execution and caller validation remain pending; no native suite, push, merge or dispatch was performed.

Author: etr

.github/workflows/release.yml

@@ -44,6 +44,9 @@
 #include <vector>
 
 #include <httpserver/server/readiness.hpp>
+#ifdef LHS_DARWIN_FOCUSED_DIAGNOSTICS
+#include "../src/detail/darwin_focused_diagnostics.hpp"
+#endif
 
 namespace v3_host {
 namespace srv = httpserver::server;
@@ -97,12 +100,20 @@ inline std::vector<srv::readiness_event> wait(
     if (snapshot.wake) interests.push_back(*snapshot.wake);
     std::vector<slot> fds;
     for (const auto& interest : interests) fds.push_back(registration(interest));
+    const auto selected_timeout = timeout(deadline);
 #if defined(_WIN32)
-    const auto ready = ::WSAPoll(fds.data(), static_cast<ULONG>(fds.size()), timeout(deadline));
+    const auto ready = ::WSAPoll(fds.data(), static_cast<ULONG>(fds.size()), selected_timeout);
     if (ready < 0 && ::WSAGetLastError() != WSAEINTR) throw std::runtime_error("WSAPoll failed");
 #else
-    const auto ready = ::poll(fds.data(), static_cast<nfds_t>(fds.size()), timeout(deadline));
+    const auto ready = ::poll(fds.data(), static_cast<nfds_t>(fds.size()), selected_timeout);
     if (ready < 0 && errno != EINTR) throw std::runtime_error("poll failed");
+#endif
+#ifdef LHS_DARWIN_FOCUSED_DIAGNOSTICS
+    LHS_DARWIN_HEADER_DIAG("header host poll now_us=%lld result=%d errno=%d deadline_us=%lld timeout_ms=%d count=%zu\n",
+        LHS_DARWIN_NOW, ready, errno, deadline ? LHS_DARWIN_TICK(*deadline) : -1LL, selected_timeout, fds.size());
+    for (const auto& fd : fds) {
+        LHS_DARWIN_HEADER_DIAG("header host poll fd=%llu events=%d revents=%d\n", LHS_DARWIN_ID(fd.fd), fd.events, fd.revents);
+    }
 #endif
     std::vector<srv::readiness_event> events;
     for (std::size_t i = 0; i < fds.size(); ++i) {

examples/host_poll.hpp

@@ -0,0 +1,37 @@
+# FEATURE-ci-darwin-focused-diagnostics
+
+Status: In Progress — implementation prepared; caller validation and the one native batch remain pending.
+
+Base: local `v3` at `a5a72356741300e5da16b94090f0f26a67a2d1f7`.
+Branch: `task/FEATURE-ci-darwin-focused-diagnostics`.
+Plan: `.groundwork-plans/FEATURE-ci-darwin-focused-diagnostics-plan.md` in the primary workspace.
+Journal: `.debug/ci-darwin-focused-diagnostics-cn_lss1c/journal.md` in the primary workspace.
+
+This is temporary diagnostic instrumentation. This branch, including its temporary replacement of `.github/workflows/release.yml`, must never merge into `v3`. The source assertions, budgets, timeout values, allocation fault countdowns and runtime decisions remain authoritative. A passing native observation establishes intermittency only. No root cause or production repair is claimed.
+
+- [x] Read literal journal, selected historical assertion logs and owning tests/boundaries. Prior batches stay closed.
+- [x] Instrument actual continuation/local noop values; outbox capacities and abort frames; QPACK allocation attempts and original error result; resource push kind/capacity/budget; peer bind/connect operation/address/errno; header watchdog, timer submission/claim/application, published readiness, actual poll and close timestamps.
+- [x] Keep private scalar tracing compile-time disabled in ordinary builds. The helper uses one fixed 512-byte stack buffer and one write, preserves errno, creates no event store/lock and performs no C++ allocation. Runtime flags narrow library traces to the header or storage processes.
+- [x] Prepare manual-only, read-only native Darwin batch under existing default-branch dispatch-capable `release.yml` filename. The required `tag` is a fixed diagnostic sentinel; it never selects a ref, creates a tag or publishes anything.
+- [x] Compile syntax, diagnostic logger smoke checks, workflow/embedded Python/shell syntax, actionlint, changed-file cpplint and diff checks locally; no native tests or full suites in this phase.
+- [ ] Caller independently validates the diagnostic diff and reviews the workflow before publishing this branch only.
+- [ ] Caller dispatches the command below ONCE and retains source-bound receipt, raw logs and LLDB abort frames.
+- [ ] Caller records actual native results, context differences and causal assessment without reopening closed probes.
+
+Proposed single dispatch after caller review/publication:
+
+```bash
+gh workflow run release.yml --ref task/FEATURE-ci-darwin-focused-diagnostics -f tag=darwin-focused-once-a5a72356
+```
+
+The workflow rejects other event/ref/sentinel values, dirty checkout and rerun attempts, binds checkout and Git source archive to `github.sha`, and uses the existing macOS 15/compiler-wrapper/provider prerequisite owners and GNU make. Root independently verified the current default-branch `release.yml` exists and supports dispatch. There are no push, tag, schedule, release or deployment triggers/jobs in this temporary replacement.
+
+The five historical TLS-no programs (`task_resume_signal_lifetime`, `http1_response_outbox`, `http3_qpack`, `resource_limits`, `consumer_v3_peer_policy_runtime`) build through their declared native make targets, retaining the target's real shared/convenience-static linkage. `external_http1_e2e` builds once against an installed TLS-yes shared library using the same real test/parity translation units and private integration include context as the historical package suite. All execute at EUID 0, matching the recurring header failure. There is no extra static-only header execution. All six execute at most once, each with a 180-second outer LLDB bound; later programs still build/run after an earlier failure or abort. Existing test case assertions and their own 300 ms/10 s deadlines remain unchanged.
+
+The artifact contains exact archive bytes/hash, commit/run identity, selected tools/environment, configure/compiler/link commands and original command logs, binaries/link map, provider preparation record and LLDB exit/stop/backtrace/local-variable evidence. A launch/setup failure remains incomplete. This is a bootstrapped Git source archive, not the historical generated release tarball. Current selected tools/provider and temporary tracing/debugging can change timing. This evidence is diagnostic only and cannot substitute for release acceptance. The existing independent CI monitor is untouched.
+
+Narrow source-storage tracing was approved by the caller because returned sink state cannot reveal a destroyed rejected allocation. It observes only existing `bound`, `actual`, `excess`, provisional charge and successful result locals in `reserve_buffer_locked`; it adds no capacity/reserve probes or accounting changes. The real host poll helper is instrumented at its syscall boundary to capture the actual submitted timeout and returned revents, without an alternate poll implementation.
+
+Local evidence: enabled C++20 syntax/format checking succeeded for the six test TUs and changed source TUs using the existing primary Linux `build/config.h` only to satisfy `connection_engine.cpp`'s generated include. The first syntax attempt lacked that header; corrected execution setup passed. Linux excludes the native kqueue body; its Darwin compilation and all native execution remain pending. Logger smoke checks confirmed enabled output, preserved errno and disabled operand elision (no test suites executed). Shell/embedded Python/LLDB-script parsing and actionlint passed. Final complete changed-file lint/diff results are recorded in the primary journal.
+
+Implementation notes for future task executors: Autotools requires out-of-tree builds; `./bootstrap`, configure with selected CC/CXX/MAKE, then GNU make. These six tests use the existing littletest harness (`-Itest` for `./littletest.hpp`), except the peer consumer's `require` checks. `src/libhttpserver_v3core.la` supplies private implementation tests; normal consumers link `src/libhttpserver.la` or installed shared library. Public headers must not include private/platform/provider headers. Worktree creation required sandbox escalation for Git metadata even though the task files are within the writable repository. No preexisting task-executor agent memory was available at the usual agent-memory locations; this record retains the setup facts.

specs/tasks/FEATURE-ci-darwin-focused-diagnostics.md

@@ -40,6 +40,7 @@
 #include <httpserver/detail/drain_scope.hpp>
 #include <httpserver/detail/diagnostics_access.hpp>
 #include <httpserver/http/outcome.hpp>
+#include "./darwin_focused_diagnostics.hpp"
 
 namespace httpserver {
 
@@ -740,6 +741,8 @@ void connection_engine::note_transport_activity(bool written) {
 }
 
 void connection_engine::enforce_timeout(std::chrono::steady_clock::time_point deadline) noexcept {
+    LHS_DARWIN_HEADER_DIAG("header enforce_timeout now_us=%lld id=%llu deadline_us=%lld\n",
+        LHS_DARWIN_NOW, LHS_DARWIN_ID(id_), LHS_DARWIN_TICK(deadline));
     bool global_expiry = false, close_expiry = false;
     {
         std::lock_guard<std::mutex> lock(mu_);
@@ -776,13 +779,17 @@ task<void> connection_engine::watchdog_loop(
             std::lock_guard<std::mutex> lock(self->mu_);
             plan = self->plan_watchdog_locked();
         }
+        LHS_DARWIN_HEADER_DIAG("header watchdog plan now_us=%lld id=%llu exit=%d defer=%d deadline_us=%lld\n",
+            LHS_DARWIN_NOW, LHS_DARWIN_ID(self->id_), plan.exit, plan.defer, LHS_DARWIN_TICK(plan.deadline));
         if (plan.exit) co_return;
         const auto deadline = plan.defer
             ? std::chrono::steady_clock::now() + k_watchdog_tick
             : plan.deadline;
         timer_operation op(self->owner_, self->id_, deadline);
         if (!self->arm_watchdog(op, plan)) continue;
         const io_result r = co_await std::move(op);
+        LHS_DARWIN_HEADER_DIAG("header watchdog resumed now_us=%lld id=%llu result=%d deadline_us=%lld\n",
+            LHS_DARWIN_NOW, LHS_DARWIN_ID(self->id_), static_cast<int>(r.code), LHS_DARWIN_TICK(deadline));
         {
             std::lock_guard<std::mutex> lock(self->mu_);
             self->pending_timer_.reset();
@@ -810,6 +817,7 @@ void connection_engine::loop_finished() {
 }
 
 void connection_engine::finalize() {
+    LHS_DARWIN_HEADER_DIAG("header connection finalize now_us=%lld id=%llu\n", LHS_DARWIN_NOW, LHS_DARWIN_ID(id_));
     // The engine's counted unit drops first: by the time the listener
     // erases its record, a drain waiting on the scope observes
     // completion with every loop finished and the writer flushed.

src/detail/connection_engine.cpp

@@ -25,6 +25,7 @@
 #include <utility>
 #include <httpserver/detail/connection_engine.hpp>
 #include <httpserver/detail/http1_websocket_handshake.hpp>
+#include "./darwin_focused_diagnostics.hpp"
 namespace httpserver::detail {
 namespace {
 bool retire_completed_front(http1_response_outbox& outbox) {
@@ -239,6 +240,9 @@ bool connection_engine::arm_watchdog(timer_operation& timer, const watchdog_plan
     if (current.exit) return false;
     if (current.defer != plan.defer || current.deadline != plan.deadline) return false;
     pending_timer_ = timer.state();
+    LHS_DARWIN_HEADER_DIAG("header watchdog submit now_us=%lld id=%llu op=%p deadline_us=%lld phase=%d head_routed=%d last_activity_us=%lld\n",
+        LHS_DARWIN_NOW, LHS_DARWIN_ID(id_), static_cast<void*>(pending_timer_.get()),
+        LHS_DARWIN_TICK(std::get<timer_payload>(pending_timer_->payload()).deadline), static_cast<int>(phase_), head_routed_, LHS_DARWIN_TICK(last_activity_));
     timer.submit(backend_);
     return true;
 }

src/detail/connection_engine_websocket.cpp

@@ -0,0 +1,47 @@
+/* Copyright (C) 2026 Sebastiano Merlino; SPDX-License-Identifier: LGPL-2.1-or-later */
+#ifndef SRC_DETAIL_DARWIN_FOCUSED_DIAGNOSTICS_HPP_
+#define SRC_DETAIL_DARWIN_FOCUSED_DIAGNOSTICS_HPP_
+#include <cstdio>
+// Temporary, noninstalled instrumentation for one authorized Darwin batch.
+#ifdef LHS_DARWIN_FOCUSED_DIAGNOSTICS
+#include <unistd.h>
+#include <cerrno>
+#include <chrono>
+#include <cstdlib>
+// One bounded write, no allocator, retry, new mutex or retained event state.
+#define LHS_DARWIN_DIAG(...) do { \
+    const int lhs_saved_errno = errno; \
+    char lhs_line[512]; \
+    const int lhs_length = std::snprintf(lhs_line, sizeof(lhs_line), __VA_ARGS__); \
+    if (lhs_length > 0) { \
+        const auto lhs_count = static_cast<std::size_t>(lhs_length) < sizeof(lhs_line) \
+            ? static_cast<std::size_t>(lhs_length) : sizeof(lhs_line) - 1; \
+        static_cast<void>(::write(STDERR_FILENO, lhs_line, lhs_count)); \
+    } \
+    errno = lhs_saved_errno; \
+} while (false)
+#define LHS_DARWIN_HEADER_DIAG(...) do { \
+    const int lhs_trace_errno = errno; \
+    if (std::getenv("LHS_DARWIN_HEADER_TRACE")) { \
+        LHS_DARWIN_DIAG(__VA_ARGS__); \
+    } \
+    errno = lhs_trace_errno; \
+} while (false)
+#define LHS_DARWIN_STORAGE_DIAG(...) do { \
+    const int lhs_storage_errno = errno; \
+    if (std::getenv("LHS_DARWIN_STORAGE_TRACE")) { \
+        LHS_DARWIN_DIAG(__VA_ARGS__); \
+    } \
+    errno = lhs_storage_errno; \
+} while (false)
+// Match printf's ll formats on both LP64 and LLP64 hosts.
+#define LHS_DARWIN_ID(value) static_cast<unsigned long long>(value)  // NOLINT(runtime/int)
+#define LHS_DARWIN_TICK(point) \
+    static_cast<long long>(std::chrono::duration_cast<std::chrono::microseconds>((point).time_since_epoch()).count())  // NOLINT(runtime/int)
+#define LHS_DARWIN_NOW LHS_DARWIN_TICK(std::chrono::steady_clock::now())
+#else
+#define LHS_DARWIN_DIAG(...) do {} while (false)
+#define LHS_DARWIN_HEADER_DIAG(...) do {} while (false)
+#define LHS_DARWIN_STORAGE_DIAG(...) do {} while (false)
+#endif
+#endif  // SRC_DETAIL_DARWIN_FOCUSED_DIAGNOSTICS_HPP_

src/detail/darwin_focused_diagnostics.hpp

@@ -7,6 +7,7 @@
 #include <vector>
 
 #include <httpserver/detail/http1_response_outbox.hpp>
+#include "./darwin_focused_diagnostics.hpp"
 
 namespace httpserver::detail {
 
@@ -147,8 +148,14 @@ auto http1_response_outbox::reserve_buffer_locked(http1_response_sink& sink, std
     if (size > maximum) return false;
     const auto capacity = sink.bytes_.capacity();
     const auto bound = std::max(size, capacity > maximum / 2 ? maximum : 2 * capacity);
+    LHS_DARWIN_STORAGE_DIAG("storage reserve candidate outbox=%p sink=%p size=%zu old_capacity=%zu bound=%zu charge=%zu queue_budget=%zu\n",
+        static_cast<void*>(this), static_cast<void*>(&sink), size, capacity, bound, storage_charge_.units(), budget_.max_queue_bytes);
     if (bound > std::numeric_limits<std::size_t>::max() - storage_charge_.units() ||
-        !storage_charge_.grow(bound)) return false;
+        !storage_charge_.grow(bound)) {
+        LHS_DARWIN_STORAGE_DIAG("storage refused provisional outbox=%p sink=%p bound=%zu charge=%zu\n",
+            static_cast<void*>(this), static_cast<void*>(&sink), bound, storage_charge_.units());
+        return false;
+    }
     // Destroy the private candidate before rolling back its admission.
     struct provisional_charge {
         resource_charge& charge;
@@ -159,14 +166,22 @@ auto http1_response_outbox::reserve_buffer_locked(http1_response_sink& sink, std
     replacement.reserve(bound);
     const auto actual = replacement.capacity();
     const auto excess = actual - bound;
+    LHS_DARWIN_STORAGE_DIAG("storage allocated candidate outbox=%p sink=%p bound=%zu actual=%zu excess=%zu provisional_charge=%zu\n",
+        static_cast<void*>(this), static_cast<void*>(&sink), bound, actual, excess, storage_charge_.units());
     if (excess > std::numeric_limits<std::size_t>::max() - storage_charge_.units() ||
-        !storage_charge_.grow(excess)) return false;
+        !storage_charge_.grow(excess)) {
+        LHS_DARWIN_STORAGE_DIAG("storage refused excess outbox=%p sink=%p actual=%zu excess=%zu charge=%zu\n",
+            static_cast<void*>(this), static_cast<void*>(&sink), actual, excess, storage_charge_.units());
+        return false;
+    }
     pending.units = actual;
     replacement.append(sink.bytes_);
     sink.bytes_.swap(replacement);
     std::string().swap(replacement);
     storage_charge_.shrink(std::exchange(sink.storage_units_, actual));
     pending.units = 0;
+    LHS_DARWIN_STORAGE_DIAG("storage accepted candidate outbox=%p sink=%p actual=%zu charge=%zu\n",
+        static_cast<void*>(this), static_cast<void*>(&sink), actual, storage_charge_.units());
     return true;
 }
 

src/detail/http1_response_outbox.cpp

@@ -25,6 +25,7 @@
 #include <memory>
 #include <exception>
 #include <utility>
+#include "./darwin_focused_diagnostics.hpp"
 
 namespace httpserver {
 namespace detail {
@@ -87,6 +88,9 @@ datagram_enqueue_code io_connection_owner::datagram_port::enqueue(
 
 void io_connection_owner::apply_record(record entry, const std::shared_ptr<shared_state>& pending) {
     if (entry.state) {
+        LHS_DARWIN_HEADER_DIAG("header owner apply now_us=%lld id=%llu op=%p kind=%d code=%d bytes=%zu\n", LHS_DARWIN_NOW,
+            LHS_DARWIN_ID(entry.state->connection()), static_cast<void*>(entry.state.get()),
+            static_cast<int>(entry.state->kind()), static_cast<int>(entry.result.code), entry.result.transferred);
         entry.state->apply(entry.result);
         return;
     }
@@ -106,6 +110,8 @@ void io_connection_owner::apply_record(record entry, const std::shared_ptr<share
 
 void io_connection_owner::enqueue(std::shared_ptr<op_state> state,
                                   io_result result) {
+    LHS_DARWIN_HEADER_DIAG("header owner enqueue now_us=%lld id=%llu op=%p kind=%d code=%d bytes=%zu\n", LHS_DARWIN_NOW,
+        LHS_DARWIN_ID(state->connection()), static_cast<void*>(state.get()), static_cast<int>(state->kind()), static_cast<int>(result.code), result.transferred);
     bool need_post = false;
     {
         std::lock_guard<std::mutex> lock(state_->mu);

src/detail/io_connection_owner.cpp

@@ -30,6 +30,7 @@
 #include <unordered_map>
 #include <utility>
 #include <vector>
+#include "./darwin_focused_diagnostics.hpp"
 
 namespace httpserver {
 namespace detail {
@@ -121,6 +122,9 @@ server::interest_snapshot io_poll_backend::interests() const {
         borrowed_handle(wake_.read_handle()), true, false};
     std::unordered_map<std::uint64_t, pollsys::event_mask> masks;
     snapshot.next_deadline = scan_interest_locked(masks);
+    LHS_DARWIN_HEADER_DIAG("header poll published now_us=%lld pending=%zu deadline_us=%lld wake_pending=%d wake_fd=%llu\n",
+        LHS_DARWIN_NOW, pending_.size(), snapshot.next_deadline ? LHS_DARWIN_TICK(*snapshot.next_deadline) : -1LL,
+        wake_pending_, LHS_DARWIN_ID(wake_.read_handle()));
     for (const auto& entry : connections_) {
         const auto& record = entry.second;
         const auto& lifetime = record.lifetime;
@@ -161,6 +165,7 @@ http::outcome io_poll_backend::dispatch(
         close();
         return status;
     }
+    LHS_DARWIN_HEADER_DIAG("header poll dispatch now_us=%lld host_now_us=%lld events=%zu\n", LHS_DARWIN_NOW, LHS_DARWIN_TICK(now), events.size());
     for (const auto& event : events) dispatch_event(event);
     expire_due_timers(now);
     return ready();
@@ -213,6 +218,9 @@ void io_poll_backend::dispatch_event(const server::readiness_event& event) {
         std::lock_guard<std::mutex> lock(mu_);
         if (closed_) return;
         std::tie(id, lease) = find_registration_locked(event);
+        LHS_DARWIN_HEADER_DIAG("header poll event now_us=%lld id=%llu generation=%llu matched=%d read=%d write=%d closed=%d error=%d\n",
+            LHS_DARWIN_NOW, LHS_DARWIN_ID(id), LHS_DARWIN_ID(event.generation),
+            static_cast<bool>(lease), event.readable, event.writable, event.closed, event.error);
         if (!lease) return;
         datagram = connections_.at(id).datagram;
         // Prepare both directions before detaching either one. A failure in