Skip to content

WriteAnswers grows without bound on nodes with no waiting clients #36

Open

Description

@resetius

Problem

TRequestProcessor::ProcessCommitted pushes a TAnswer into WriteAnswers for every applied entry, on every node (raft.cpp#L634-L644). The only place that removes entries from the queue is ProcessWaiting, and only while completing a local waiting write (raft.cpp#L654-L661). A follower has no waiting writes (writes are forwarded to the leader).

Scenario

A follower applies a long stream of writes. Each applied entry adds a TAnswer that holds the TCommandResponse (and whatever the RSM returned, for example the SQL result text), and nothing ever pops it. Memory grows linearly with the number of applied entries, without bound. The same happens on a leader for entries that no local client is waiting for (for example forwarded writes), until some later local waiting write pops past them.

Expected

Replies are kept only for writes that are pending on this node, keyed by log index and term, and are dropped when the request is answered, when leadership changes, or when the client disconnects. No state grows with the number of applied entries on a node that has no waiting clients.

Fix in PR #31

PR #31 (closed, not merged) has the fix in e117c18 (write replies for local pending writes only, keyed by index and term; pending requests fail on leadership change and on disconnect). Tests that cover it: test_follower_application_does_not_retain_unused_replies, test_replaced_write_never_receives_another_commands_reply, test_stepdown_rejects_write_without_waiting_for_application, test_disconnect_*. In the split plan it comes after the FIFO fix for ProcessWaiting. Since the PR is closed, the fix would come as a separate small PR.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions