Skip to content

HDDS-15757. Streaming Write also commit PutBlock at the time of closing#10694

Open
amaliujia wants to merge 3 commits into
apache:masterfrom
amaliujia:streaming_putblock_dn
Open

HDDS-15757. Streaming Write also commit PutBlock at the time of closing#10694
amaliujia wants to merge 3 commits into
apache:masterfrom
amaliujia:streaming_putblock_dn

Conversation

@amaliujia

@amaliujia amaliujia commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

A new config (hdds.datanode.datastream.putblock.enabled) is introduced to control the behavior of PutBlock:

  1. It assumes the PutBlock proto and the proto length is append to the end of the stream.
  2. When the stream is closed. it also read the PutBlock proto from the end of the stream and commit the it.

The future PR will be the client side change that should no longer send PutBlock as a separate Raft request.

This PR is assisted by Cursor ai.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-15792

How was this patch tested?

Unit test.

@amaliujia amaliujia marked this pull request as draft July 9, 2026 01:48
@amaliujia amaliujia force-pushed the streaming_putblock_dn branch from b8e4e3e to 4902773 Compare July 9, 2026 03:07
@amaliujia amaliujia changed the title HDDS-15757.Support Stream containers as EC in datanode HDDS-15757. Streaming Write also commit PutBlock at the time of closing Jul 9, 2026
@amaliujia amaliujia marked this pull request as ready for review July 9, 2026 03:12
@amaliujia amaliujia changed the title HDDS-15757. Streaming Write also commit PutBlock at the time of closing HDDS-15463. Streaming Write also commit PutBlock at the time of closing Jul 9, 2026
@amaliujia amaliujia changed the title HDDS-15463. Streaming Write also commit PutBlock at the time of closing HDDS-15757. Streaming Write also commit PutBlock at the time of closing Jul 9, 2026
@amaliujia amaliujia changed the title HDDS-15757. Streaming Write also commit PutBlock at the time of closing HDDS-15757. Streaming Write also commit PutBlock at the time of closing Jul 9, 2026
@amaliujia

Copy link
Copy Markdown
Contributor Author

R: @szetszwo

@amaliujia amaliujia requested a review from szetszwo July 9, 2026 03:22

public ContainerCommandRequestProto getPutBlockRequest() {
return Objects.requireNonNull(putBlockRequest.get(),
() -> "putBlockRequest == null, " + this);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we have any meaningful toString() that would trace the state of the object in '+ this'?

@yandrey321

Copy link
Copy Markdown
Contributor
  1. what is the motivation for these changes?
  2. when these changes would benefit write latency/throughput?
  3. Any benchmarks that shows the performance gain?

@amaliujia

Copy link
Copy Markdown
Contributor Author

@yandrey321 you may find motivations from comments under https://issues.apache.org/jira/browse/HDDS-15463

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants