Skip to content

[Java] Support batch acknowledgements for PushConsumer - #1326

Draft
qianye1001 wants to merge 1 commit into
apache:masterfrom
qianye1001:codex/grpc-batch-ack
Draft

[Java] Support batch acknowledgements for PushConsumer#1326
qianye1001 wants to merge 1 commit into
apache:masterfrom
qianye1001:codex/grpc-batch-ack

Conversation

@qianye1001

Copy link
Copy Markdown
Contributor

Which Issue(s) This PR Fixes

Fixes #1325

Brief Description

The Java gRPC PushConsumer currently sends one acknowledgement RPC for every successfully consumed message even though AckMessageRequest supports repeated entries. Under a high consume rate, these one-entry requests add substantial client and Proxy overhead.

This change introduces one acknowledgement batcher per PushConsumer:

  • Batches are partitioned by endpoint and topic to match the request routing and protocol constraints.
  • A batch is flushed after 5 seconds or when it reaches 1,024 messages.
  • Only messages consumed within 25 seconds of receive/decode are eligible. Slow messages use the existing immediate acknowledgement path.
  • FIFO acknowledgements remain immediate because subsequent ordered messages depend on them.
  • Lite consumers retain the existing immediate acknowledgement behavior.
  • Multi-entry responses are mapped back to each message by message ID. Missing or failed results use the existing delayed individual retry path; invalid receipt handles preserve the existing no-retry behavior.
  • PushConsumer shutdown waits for consumption to finish and then flushes pending and in-flight acknowledgement batches.
  • ACK interceptors still receive all messages included in each batch.

The current RocketMQ Proxy protocol and batch-ACK path already accept repeated entries, so this change does not require a protocol update.

At approximately 8,000 TPS in the medium 2B workload, two controlled candidate runs averaged about 20.4 entries per ACK RPC, reducing ACK RPC volume by approximately 95.1%. Total Java consumer CPU decreased by approximately 20.6%, while throughput remained unchanged. A separate single-topic validation observed requests with exactly 1,024 entries.

How Did You Test This Change?

  • zsh -ic 'javaswitch 11 >/dev/null && mvn -f java/pom.xml -pl client -am test'
    • 304 tests run
    • 0 failures
    • 0 errors
    • 1 skipped
  • Checkstyle: 0 violations
  • SpotBugs: 0 errors or warnings
  • Verified the compiled batcher class remains Java 8 compatible (major version: 52).
  • Ran an end-to-end Kubernetes smoke test against a two-Broker/two-Proxy deployment with batch ACK enabled.
  • Ran a medium 2B A-B-B-A comparison with 1,000 topics, 1,000 groups, 2,000 producers, 2,000 consumers, 4 KiB messages, 120 seconds warm-up, and 300 seconds sampling per round.
Metric Baseline mean Batch ACK mean Change
Send TPS 7,981.0 7,981.5 +0.01%
Consumer total CPU 3.446 cores 2.736 cores -20.59%
Consumer CPU p95 3.658 cores 2.975 cores -18.67%
Proxy total CPU 10.998 cores 9.052 cores -17.70%
E2E latency p99 21.682 ms 20.515 ms -5.38%

@codecov-commenter

codecov-commenter commented Aug 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 76.04167% with 46 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.24%. Comparing base (4076e2b) to head (a5c68e8).
⚠️ Report is 166 commits behind head on master.

Files with missing lines Patch % Lines
...q/client/java/impl/consumer/AckMessageBatcher.java 81.45% 21 Missing and 7 partials ⚠️
...mq/client/java/impl/consumer/PushConsumerImpl.java 0.00% 12 Missing ⚠️
...mq/client/java/impl/consumer/ProcessQueueImpl.java 63.63% 1 Missing and 3 partials ⚠️
...cketmq/client/java/impl/consumer/ConsumerImpl.java 88.88% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #1326      +/-   ##
============================================
+ Coverage     53.26%   63.24%   +9.97%     
- Complexity      651      856     +205     
============================================
  Files           208      109      -99     
  Lines         14303     4807    -9496     
  Branches       5845      390    -5455     
============================================
- Hits           7619     3040    -4579     
+ Misses         6308     1558    -4750     
+ Partials        376      209     -167     
Flag Coverage Δ
cpp ?
golang ?
java 63.24% <76.04%> (+1.55%) ⬆️
rust ?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@RockteMQ-AI RockteMQ-AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Introduces AckMessageBatcher to batch PushConsumer acknowledgements by (endpoints, topic), reducing per-message RPC overhead. The design is clean: batches are flushed either when the size threshold is reached or when a delay timer expires. Partial failures (MULTIPLE_RESULTS) are correctly mapped back to individual message futures, and missing results default to INTERNAL_ERROR so the caller can retry. FIFO and lite-consumer messages correctly bypass batching to preserve ordering guarantees.

The retry path (ackMessageLater) correctly forces batchable=false to prevent re-batching of failed messages, and flushAndClose() provides a graceful shutdown mechanism.

Comprehensive test coverage (7 test cases) validates the key scenarios.

LGTM.


Automated review by github-manager

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Batch acknowledgements for Java gRPC PushConsumer

3 participants