Fix lost Kafka catch-up signal that can hang Thunder startup - #47
Open
huxint wants to merge 1 commit into
Open
Conversation
The catch-up completion signal is only sent inside the `message_buffer.len() >= batch_size` block, but catchup_sender is Some only on the single poll iteration where catch-up is first detected (init_data_downloaded flips to true and stays true). If the buffer holds fewer than batch_size messages on that iteration — guaranteed when a partition's remaining backlog is smaller than one batch, e.g. with --skip_to_latest or a low-traffic partition — the signal is dropped and never re-sent. main() then blocks forever on rx.recv() when --is_serving is set: finalize_init() never runs and the server never reports ready. Flush whatever is buffered and send the signal on the detection iteration itself, regardless of batch fill. When no catch-up signal is pending the control flow is unchanged. This also ensures the tail of the backlog is inserted before finalize_init() sorts and trims.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
process_tweet_events_v2can drop the one-shot catch-up completion signal. When that happens with--is_servingset,main()blocks forever onrx.recv().await:finalize_init()never runs and the server never reports ready.Root cause
In
thunder/kafka/tweet_events_listener_v2.rs,catchup_senderisSomeonly on the single poll iteration where catch-up is first detected (init_data_downloadedflips to true and stays true), butsender.send(lag)sits inside theif message_buffer.len() >= batch_sizeblock. If the buffer holds fewer thanbatch_sizemessages on that iteration, the signal is silently dropped and can never be re-sent.A partition whose remaining backlog is smaller than one batch hits this deterministically — e.g.
--skip_to_latest, a low-traffic partition, or a topic shorter thankafka_batch_size— sincepollis bounded byfetch_timeout_msand cannot be relied on to return a full batch there.main.rswaits for one signal per thread beforefinalize_init()andset_readiness(true).Fix
On the detection iteration, flush whatever is buffered (if anything) and send the signal unconditionally. When no catch-up signal is pending, the control flow is unchanged. This also inserts the tail of the backlog before
finalize_init()sorts and trims, instead of leaving up tobatch_size - 1messages stranded until live traffic accumulates.Verification
thunder/component ships no Cargo manifest and depends on unpublished crates (xai_kafka,xai_thunder_proto), so it cannot be compiled from this snapshot. The change is a minimal re-nesting of the existing statements plusOption::is_some/Vec::is_empty;rustfmtparses the file cleanly.