Skip to content

[alerter]bugfix: preserve group alert recovery events - #4168

Open
hutiefang76 wants to merge 2 commits into
apache:masterfrom
hutiefang76:codex/fix-alarm-group-resolve
Open

hutiefang76 wants to merge 2 commits into
apache:masterfrom
hutiefang76:codex/fix-alarm-group-resolve

Conversation

@hutiefang76

Copy link
Copy Markdown
Contributor

What's changed

Close #4160.

This fixes two group alert lifecycle cases:

  • do not let the firing repeat interval suppress pending resolved alerts in a mixed firing group
  • recompute the persisted group status from all known member alerts, so one recovered alert does not mark the whole group resolved while another member is still firing

Regression tests cover both the in-memory group reducer path and the database store path.

Verification

  • JAVA_HOME=$(/usr/libexec/java_home -v 25 2>/dev/null || /usr/libexec/java_home) ./mvnw -pl hertzbeat-alerter -am test
  • git diff --check

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes two alert-group lifecycle edge cases in the alerter pipeline: ensuring recovery (resolved) events are not suppressed by the firing repeat-interval throttle, and ensuring persisted group status reflects the aggregate state of all known member alerts (not just the current push window).

Changes:

  • Bypass the group repeat-interval throttle when the group payload includes any resolved member alerts.
  • Recompute a group’s persisted status in the DB store layer from all member alert statuses (by fingerprint), and add regression tests for both reducer and store paths.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
hertzbeat-alerter/src/main/java/org/apache/hertzbeat/alert/reduce/AlarmGroupReduce.java Adjusts repeat-throttle behavior so mixed firing groups with resolved members are still dispatched.
hertzbeat-alerter/src/main/java/org/apache/hertzbeat/alert/notice/impl/DbAlertStoreHandlerImpl.java Recomputes persisted group status from stored member alerts before saving.
hertzbeat-alerter/src/test/java/org/apache/hertzbeat/alert/reduce/AlarmGroupReduceTest.java Adds a regression test for “resolved bypasses repeat throttle”.
hertzbeat-alerter/src/test/java/org/apache/hertzbeat/alert/notice/impl/DbAlertStoreHandlerImplTest.java Adds a regression test ensuring group status stays firing when other members still fire.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@hutiefang76

Copy link
Copy Markdown
Contributor Author

I rebased the branch onto the current master and rechecked the three earlier review points on the latest head: missing group rules now use the default repeat interval, resolved groups reload the full persisted member list, and firing groups skip the extra DB lookup. The focused AlarmGroupReduceTest and DbAlertStoreHandlerImplTest suites pass locally (11 tests, 0 failures/errors), with checkstyle passing across the reactor. The old threads are now outdated and resolved; this is ready for maintainer review.

@hutiefang76

Copy link
Copy Markdown
Contributor Author

Thanks for the review. I rechecked the current head and confirmed the three points are covered:

  • sendGroupAlert(...) now falls back to DEFAULT_REPEAT_INTERVAL when the cached group rule has been removed; cachedGroupUsesDefaultRepeatIntervalWhenGroupRuleWasRemoved covers that path.
  • Historical group-alert lookup is only performed for an incoming resolved group, so firing updates do not add the extra database query; storeFiringGroupDoesNotQueryHistoricalAlerts covers it.
  • When a resolved member arrives while another member is still firing, the persisted group is recalculated from the complete stored member set and the returned GroupAlert carries that complete alert list; storeResolvedAlertKeepsGroupFiringWhenOtherGroupAlertsStillFire covers it.

I reran AlarmGroupReduceTest and DbAlertStoreHandlerImplTest from the reactor with a Central-only Maven configuration: 11 tests passed, 0 failures/errors.

@hutiefang76
hutiefang76 force-pushed the codex/fix-alarm-group-resolve branch from 7754ef2 to a9cae8b Compare August 30, 2026 15:14
@hutiefang76

hutiefang76 commented Aug 30, 2026

Copy link
Copy Markdown
Contributor Author

Rebased this branch onto current master (8e9f8da), retaining only the follow-up behavior not already covered by #4316:

  • For a resolved group update, reload the complete persisted member-alert set before deciding the group status, so another firing member keeps the group firing.
  • Keep firing updates on the existing in-memory path without an extra database read.
  • Preserve the no-rule default throttle behavior.

Validation completed:

  • Targeted Maven reactor test: hertzbeat-alerter with its required modules
  • AlarmGroupReduceTest: 8 tests, 0 failures
  • DbAlertStoreHandlerImplTest: 6 tests, 0 failures
  • git diff --check upstream/master...HEAD

The PR head now contains the rebase and is ready for CI review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Alarm group convergence: group wrongly flips to resolved and recovery events get silently dropped

2 participants