[WIP] [V2] Emit action events as Kubernetes Events - #8116
Draft
jeffoodchain wants to merge 4 commits into
Draft
jeffoodchain wants to merge 4 commits into
jeffoodchain wants to merge 4 commits into
Conversation
Signed-off-by: Jeff Chung <sh1001309@gmail.com>
Signed-off-by: Jeff Chung <sh1001309@gmail.com>
Signed-off-by: Jeff Chung <sh1001309@gmail.com>
Signed-off-by: Jeff Chung <sh1001309@gmail.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking issue
Closes #7829
Related to #7903
Why are the changes needed?
Flyte 1 pushed each execution event to a message broker. Flyte 2 has no such path, so
systems outside Flyte (lineage, alerting, cost accounting) cannot react to action events.
The design in #7903 solves this without a broker client in Flyte. The executor emits each
action event as a k8s Event on the TaskAction. An operator deploys a third-party
exporter (for example resmoio/kubernetes-event-exporter) and routes the events to Kafka, Loki, or a webhook.
What changes were proposed in this pull request?
Design Doc: #7903
This PR is still work in progress, and will be updated frequently.
Done:
truncateUTF8: cut a note to the 1024-byte apiserver limit and keep valid UTF-8. (in doc, it istruncateRunes)actionsk8s.IsTerminalPhaseso the executor can reuse it.EventLevel(off,terminal,info,debug) andeventLevelOf, which classifies an action event. A system retry is aQUEUEDevent in the reserved version range.buildActionEventK8s: build anevents.k8s.io/v1Event from anActionEvent.TODOs:
ReportingInstanceon the reconciler.flyte.org/info. The apiserver caps all annotations of one object at 256 KiB.parent-action-nameandtask-typeannotations. Remove the unusedlaunch-planandprincipalkeys (TaskActionSpechas neither).flyte.org/run-nameor the existing label keyflyte.org/run.How was this patch tested?
Unit tests in
executor/pkg/controller/k8s_event_test.go:TestTruncateUTF8: ASCII cases, a cut inside a multi-byte rune, invalid bytes.TestBuildActionEventK8s: fields the apiserver validates, error annotations with andwithout
ErrorInfo, andflyte.org/infodecodes back to the original event.TestEventLevelOf,TestEventType,TestEventReason,TestHumanSummary.Not tested yet: no apiserver has accepted one of these events. That is the devbox item above.
Labels
Check all the applicable boxes
Related PRs
#7903