Conversation
A trail's own SNS topic is often the same topic the bucket publishes S3 events to, so the queue receives a second message per log file shaped {"s3Bucket", "s3ObjectKey": [...]}. It matched no SqsEvent variant, so it was logged as a processing error and never deleted, and came back every visibility timeout until retention expired.
Parse that shape as its own variant and treat it like s3:TestEvent: emit a warning and sqs_s3_event_record_ignored_total{ignore_type="cloudtrail_notification"}, then delete the message. The variant is last in the untagged enum, so every body that parsed before resolves to the same variant as before. The files are not fetched from it because the S3 event for the same file is already ingested.
pupapaik
marked this pull request as ready for review
September 19, 2026 11:55
Author
|
@sundaresanr can you check thsi one if it make sense. I see it in gainsight |
sundaresanr
reviewed
Sep 19, 2026
| object_count: notification.s3_object_key.len(), | ||
| }); | ||
| Ok(()) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
{"s3Bucket": ..., "s3ObjectKey": [...]}. It matched noSqsEventvariant, so it was logged asFailed to process SQS messageand never deleted; it came back every visibility timeout until retention expired. On gainsight that is ~20-37 errors/s and ~400 undeletable messages/hour, heading for a ~39k in-flight plateau.s3:TestEvent: warn, count it undersqs_s3_event_record_ignored_total{ignore_type="cloudtrail_notification"}, and delete it. No customer action needed, and no data impact: the S3 event for the same file is what gets ingested, which is also why the files are deliberately not fetched from this message (that would double every event).Tests:
cargo test --no-default-features --features sources-aws_s3 --lib -- sources::aws_s3::sqson rust 1.92 -- 7 passed, 0 failed. Three are new: raw shape, SNS-wrapped shape, and a variant-order regression that also asserts near-miss bodies still fail to parse.Details
A customer with only the trail topic (no S3 event notifications) moves from erroring to silent. That case stays detectable: the Grafana "no events and no errors" rules for
source_(cloudtrail|gcp).*(20 min) fire instead ofVectorSourceAllErrors, and the new warning plus counter say why.Rollout after merge: build a new
vector-basetag from this branch, then bump theFROM .../vector-base:line intools/chronoforge/Dockerfileandgoservices/vector_exec_sources/Dockerfilein operations.v0.54.0-exaforcehas no other undeployed commits, so the bump ships only this change.Verification on gainsight (both message kinds arrive in
eu-prod-exaforce-cloudtrail-logs-queue): SQS deleted/sent should go from 0.51 to ~1.0,ApproximateNumberOfMessagesNotVisibleshould drain within one visibility timeout, and hourly CloudTrail event counts should not change.