Skip to content

Fix lost Kafka catch-up signal that can hang Thunder startup - #47

Open
huxint wants to merge 1 commit into
xai-org:mainfrom
huxint:fix/thunder-catchup-signal-loss
Open

Fix lost Kafka catch-up signal that can hang Thunder startup#47
huxint wants to merge 1 commit into
xai-org:mainfrom
huxint:fix/thunder-catchup-signal-loss

Conversation

@huxint

@huxint huxint commented Aug 14, 2026

Copy link
Copy Markdown

Summary

process_tweet_events_v2 can drop the one-shot catch-up completion signal. When that happens with --is_serving set, main() blocks forever on rx.recv().await: finalize_init() never runs and the server never reports ready.

Root cause

In thunder/kafka/tweet_events_listener_v2.rs, catchup_sender is Some only on the single poll iteration where catch-up is first detected (init_data_downloaded flips to true and stays true), but sender.send(lag) sits inside the if message_buffer.len() >= batch_size block. If the buffer holds fewer than batch_size messages on that iteration, the signal is silently dropped and can never be re-sent.

A partition whose remaining backlog is smaller than one batch hits this deterministically — e.g. --skip_to_latest, a low-traffic partition, or a topic shorter than kafka_batch_size — since poll is bounded by fetch_timeout_ms and cannot be relied on to return a full batch there. main.rs waits for one signal per thread before finalize_init() and set_readiness(true).

Fix

On the detection iteration, flush whatever is buffered (if anything) and send the signal unconditionally. When no catch-up signal is pending, the control flow is unchanged. This also inserts the tail of the backlog before finalize_init() sorts and trims, instead of leaving up to batch_size - 1 messages stranded until live traffic accumulates.

Verification

  • The published thunder/ component ships no Cargo manifest and depends on unpublished crates (xai_kafka, xai_thunder_proto), so it cannot be compiled from this snapshot. The change is a minimal re-nesting of the existing statements plus Option::is_some / Vec::is_empty; rustfmt parses the file cleanly.
  • I extracted the loop's control flow into a standalone test binary and verified: (1) with a sub-batch backlog the original flow never sends and strands the buffered tail, while the fixed flow sends exactly once and drains it; (2) with no catch-up pending, original and fixed flows behave identically; (3) the signal cannot fire more than once. Happy to inline those tests if useful.

The catch-up completion signal is only sent inside the
`message_buffer.len() >= batch_size` block, but catchup_sender is Some
only on the single poll iteration where catch-up is first detected
(init_data_downloaded flips to true and stays true). If the buffer holds
fewer than batch_size messages on that iteration — guaranteed when a
partition's remaining backlog is smaller than one batch, e.g. with
--skip_to_latest or a low-traffic partition — the signal is dropped and
never re-sent. main() then blocks forever on rx.recv() when --is_serving
is set: finalize_init() never runs and the server never reports ready.

Flush whatever is buffered and send the signal on the detection
iteration itself, regardless of batch fill. When no catch-up signal is
pending the control flow is unchanged. This also ensures the tail of
the backlog is inserted before finalize_init() sorts and trims.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant