Skip to content

Improvement/bb 605 alert for ingestion processing #2821

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
benzekrimaha wants to merge 4 commits into
base: development/9.6
Choose a base branch
from
+81 −1
Open

Improvement/bb 605 alert for ingestion processing #2821

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ce732c7
Name the critical source error rate alert after its threshold
benzekrimaha Aug 20, 2026
9077d5f
Alert when an ingestion source cannot be set up
benzekrimaha Aug 20, 2026
ecb9f8b
Test the ingestion producer alerts
benzekrimaha Aug 20, 2026
3c1e419
Word the setup-failing alert as set up, not read
benzekrimaha Sep 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .github/workflows/alerts.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,17 @@ jobs:
notification_processor_replicas=2
github_token: ${{ secrets.GIT_ACCESS_TOKEN }}

- name: Render and test ingestion producer
uses: scality/action-prom-render-test@1.0.3
with:
alert_file_path: monitoring/ingestion/ingestion-producer-alert.yaml
test_file_path: monitoring/ingestion/ingestion-producer-alert.test.yaml
alert_inputs: |
namespace=zenko
job_ingestion_producer=artesca-data-backbeat-ingestion-producer-headless
replicas=1
github_token: ${{ secrets.GIT_ACCESS_TOKEN }}

- name: Render and test oplog populator
uses: scality/action-prom-render-test@1.0.3
with:
Expand Down
53 changes: 53 additions & 0 deletions monitoring/ingestion/ingestion-producer-alert.test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
evaluation_interval: 1m
rule_files:
- ingestion-producer-alert.rendered.yaml

tests:

# A source whose setup keeps failing is retried on every cron tick, so it
# fails at a steady rate while every other source keeps reading normally.
- name: Ingestion Producer Source Setup Failing
interval: 1m
input_series:
- series: s3_ingestion_source_operations_total{namespace="zenko",job="artesca-data-backbeat-ingestion-producer-headless",origin="ingestion",op="getRaftId",status="error"}
values: 0+12x20
- series: s3_ingestion_source_operations_total{namespace="zenko",job="artesca-data-backbeat-ingestion-producer-headless",origin="ingestion",op="getRaftLog",status="success"}
values: 0+12000x20
alert_rule_test:
- alertname: IngestionProducerSourceSetupFailing
eval_time: 5m
exp_alerts: []
- alertname: IngestionProducerSourceSetupFailing
eval_time: 12m
exp_alerts:
- exp_labels:
severity: warning
exp_annotations:
zenko_service: backbeat-ingestion-producer
description: "Ingestion producer has been unable to set up a source for the last 10 minutes, so the objects of that location are not ingested. Check the producer logs for the bucket it fails to resolve."
summary: "Ingestion producer cannot set up a source"
# The same failures are 0.1% of all the source operations, which is why the
# error ratio alerts cannot report a single source that stopped ingesting.
- alertname: IngestionProducerSourceErrorRate3Percent
eval_time: 12m
exp_alerts: []
- alertname: IngestionProducerSourceErrorRate5Percent
eval_time: 12m
exp_alerts: []

# A source that fails to set up while its location restarts, then recovers,
# must not alert.
- name: Ingestion Producer Source Setup Recovering
interval: 1m
input_series:
- series: s3_ingestion_source_operations_total{namespace="zenko",job="artesca-data-backbeat-ingestion-producer-headless",origin="ingestion",op="getRaftId",status="error"}
values: 0 12 24 24x17
- series: s3_ingestion_source_operations_total{namespace="zenko",job="artesca-data-backbeat-ingestion-producer-headless",origin="ingestion",op="getRaftId",status="success"}
values: 0 0 0 1+1x17
alert_rule_test:
- alertname: IngestionProducerSourceSetupFailing
eval_time: 8m
exp_alerts: []
- alertname: IngestionProducerSourceSetupFailing
eval_time: 15m
exp_alerts: []
18 changes: 17 additions & 1 deletion monitoring/ingestion/ingestion-producer-alert.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ groups:
description: "Ingestion producer source error rate has increased to 3% in the last 5 minutes"
summary: "Ingestion producer source error rate has increased"

- alert: IngestionProducerSourceErrorRate3Percent
- alert: IngestionProducerSourceErrorRate5Percent
Expr: |
sum(rate(s3_ingestion_source_operations_total{status="error", namespace="${namespace}", job="${job_ingestion_producer}"}[5m]))
/
Expand All @@ -98,6 +98,22 @@ groups:
description: "Ingestion producer source error rate has increased to 5% in the last 5 minutes"
summary: "Ingestion producer source error rate has increased"

# The setup of a reader is retried on every cron tick, so a source that cannot
# be resolved fails continuously. It is a handful of errors against the reads
# of every healthy source, so it only shows up on the absolute rate: the ratio
# alerts above stay far below their threshold when a single source is stuck.
- alert: IngestionProducerSourceSetupFailing

@francoisferrand francoisferrand Sep 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this alert is a sub-class of the previous one (IngestionProducerSourceErrorRate5Percent) :

  • Why does it not report issues? Is this because of the threshold, or because one location's error may be lost in the other location's success, or because other calls are incorrectly reducing the ratio?
  • Should we adjust the threshold instead? Or skip some ops which are not relevant to the alert? Or sum by bucket/location?

@benzekrimaha benzekrimaha Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me it is the aggregation. A stuck source retries getRaftId every few seconds and those errors are drowned by getRaftLog successes from healthy sources. Lowering the ratio would noisy-alert on read errors and filtering the existing ratio to op="getRaftId" would fire, but then “error rate 5%” would mean setup failure...
The dedicated absolute rate on getRaftId, with for: 10m, is the smallest change that names this failure mode in my opinion

Expr: |
sum(rate(s3_ingestion_source_operations_total{op="getRaftId", status="error", namespace="${namespace}", job="${job_ingestion_producer}"}[5m]))

@francoisferrand francoisferrand Sep 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we sum by bucket or location? So the alert points to the specific bucket/location which does not work?

@benzekrimaha benzekrimaha Sep 16, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not possible today. s3_ingestion_source_operations_total only has origin, op, status. No bucket and no location sum by would be empty and locationNameRaftID exists on the journal gauges, not on this counter. We can add a locationName label, but it would apply to every source op (getRaftLog, metadata, …), not only getRaftId. I would keep this alert cluster-wide for now and add the label in a follow-up if we want the firing alert to name the location.

> 0
For: "10m"
Labels:
severity: warning
Annotations:
zenko_service: backbeat-ingestion-producer
description: "Ingestion producer has been unable to set up a source for the last 10 minutes, so the objects of that location are not ingested. Check the producer logs for the bucket it fails to resolve."
summary: "Ingestion producer cannot set up a source"

- alert: IngestionProducerZookeeperErrorRate3Percent
Expr: |
sum(rate(s3_ingestion_zookeeper_operations_total{status="error", namespace="${namespace}", job="${job_ingestion_producer}"}[5m]))
Expand Down
Loading