Skip to content

fix(aws_s3 source): ignore CloudTrail log delivery notifications - #47

Closed
pupapaik wants to merge 1 commit into
v0.54.0-exaforcefrom
jpavlik/aws-s3-ignore-cloudtrail-sns-notification
Closed

pupapaik wants to merge 1 commit into
v0.54.0-exaforcefrom
jpavlik/aws-s3-ignore-cloudtrail-sns-notification

Conversation

@pupapaik

@pupapaik pupapaik commented Sep 19, 2026

Copy link
Copy Markdown

Summary

  • Customers often point a trail's own SNS topic at the same topic their bucket publishes S3 events to, so our queue gets a second message per log file, shaped {"s3Bucket": ..., "s3ObjectKey": [...]}. It matched no SqsEvent variant, so it was logged as Failed to process SQS message and never deleted; it came back every visibility timeout until retention expired. On gainsight that is ~20-37 errors/s and ~400 undeletable messages/hour, heading for a ~39k in-flight plateau.
  • Recognise that shape and treat it like s3:TestEvent: warn, count it under sqs_s3_event_record_ignored_total{ignore_type="cloudtrail_notification"}, and delete it. No customer action needed, and no data impact: the S3 event for the same file is what gets ingested, which is also why the files are deliberately not fetched from this message (that would double every event).
  • The new variant is last in the untagged enum, so every body that parsed before resolves to the same variant as before. Bodies that match nothing still fail loudly.

Tests: cargo test --no-default-features --features sources-aws_s3 --lib -- sources::aws_s3::sqs on rust 1.92 -- 7 passed, 0 failed. Three are new: raw shape, SNS-wrapped shape, and a variant-order regression that also asserts near-miss bodies still fail to parse.

Details

A customer with only the trail topic (no S3 event notifications) moves from erroring to silent. That case stays detectable: the Grafana "no events and no errors" rules for source_(cloudtrail|gcp).* (20 min) fire instead of VectorSourceAllErrors, and the new warning plus counter say why.

Rollout after merge: build a new vector-base tag from this branch, then bump the FROM .../vector-base: line in tools/chronoforge/Dockerfile and goservices/vector_exec_sources/Dockerfile in operations. v0.54.0-exaforce has no other undeployed commits, so the bump ships only this change.

Verification on gainsight (both message kinds arrive in eu-prod-exaforce-cloudtrail-logs-queue): SQS deleted/sent should go from 0.51 to ~1.0, ApproximateNumberOfMessagesNotVisible should drain within one visibility timeout, and hourly CloudTrail event counts should not change.

A trail's own SNS topic is often the same topic the bucket publishes S3 events to, so the queue receives a second message per log file shaped {"s3Bucket", "s3ObjectKey": [...]}. It matched no SqsEvent variant, so it was logged as a processing error and never deleted, and came back every visibility timeout until retention expired.

Parse that shape as its own variant and treat it like s3:TestEvent: emit a warning and sqs_s3_event_record_ignored_total{ignore_type="cloudtrail_notification"}, then delete the message. The variant is last in the untagged enum, so every body that parsed before resolves to the same variant as before. The files are not fetched from it because the S3 event for the same file is already ingested.
@pupapaik
pupapaik marked this pull request as ready for review September 19, 2026 11:55
@pupapaik

Copy link
Copy Markdown
Author

@sundaresanr can you check thsi one if it make sense. I see it in gainsight

Comment thread src/sources/aws_s3/sqs.rs
object_count: notification.s3_object_key.len(),
});
Ok(())
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#48

@pupapaik pupapaik closed this Sep 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants