feat: qualify dedup IDs and bound the exporter queue - #248
Merged
savme merged 7 commits intoOct 2, 2026
Merged
Conversation
This was referenced Sep 24, 2026
ecv
self-requested a review
September 26, 2026 21:35
ecv
approved these changes
Sep 26, 2026
scotwells
approved these changes
Oct 2, 2026
savme
added a commit
that referenced
this pull request
Oct 2, 2026
The processor builds an Activity from a Kubernetes event in two different places: `EventProcessor` handles the live NATS consumer path, `ActivityBuilder` handles `PolicyPreview` and the reindex job. Both independently resolve the event's actor, subject, and timestamp, and the implementations drifted somewhat apart. This PR merges both into one set of shared extractors in `internal/processor/utils.go`, so live processing, preview, and reindex all resolve an event the same way. ### Test plan - [x] go test - [x] Added `TestEventBuildersParity`, asserting the live path and the preview/reindex path produce identical output for the same event Related to datum-cloud/compute#100 Stack: - #239 👈 - #240 - #245 - #246 - #248 - #250
The exporter connects to NATS without TLS, so it cannot reach any listener requiring a client certificate - both the shared `nats-system` broker and the edge relay. It has only ever run in dev, where NATS has no TLS. Edge clusters also have no path to a central broker at all. The exporter gains mTLS and publishes to a subject carrying a cluster token, so per-PoP NATS permissions can scope publish access to that cluster's own subjects. An `activity nkey-generator` subcommand mints per-cluster NKeys from Karmada cluster state and distributes them through ESO. A Kustomize bundle deploys the exporter alongside a local core-NATS relay on edge cells, and a second processor instance consumes the federated stream from the hub while publishing activities back to the shared broker. ### Test plan - [x] `go test` - [x] Covers subject and grant derivation across prefixes, including that a cluster name containing a dot is rejected rather than silently widening a grant - [x] Covers that an unset output broker keeps the processor on a single connection - [ ] Deploy to staging and confirm `event_exporter_events_published_total` emits its first series Depends on #248 Stack: - #239 - #240 - #245 - #246 - #248 - #250 👈
savme
added a commit
that referenced
this pull request
Oct 2, 2026
The event exporter now resolves its own cell identity: plane type, cluster, and region from config, city resolved in the background from the cluster's `ServingLocation`. This identity information is used to stamp each event's `source` On edge deployments, it also recovers per-event tenant scope from the label Karmada already puts on the event's namespace, instead of a fixed scope for the whole deployment. Management-plane behavior is unchanged. **Note**: this bumps k8s dependencies to v0.36, which is required to bring in the `locations` dependency. Test plan - [x] `go test` - [x] `task generate:openapi`, `task generate:rbac` re-run Related to datum-cloud/compute#100 Depends on #245 Stack: - #239 - #240 - #245 - #246 👈 - #248 - #250
savme
added a commit
that referenced
this pull request
Oct 2, 2026
Builds on the schema from #240 to populate `source`. An event carrying `source-*` annotations (stamped by an edge exporter) now gets those values copied onto the Activity's source field. ### Test plan - [x] `go test` - [x] Table-driven coverage for source population, related-based resource resolution, and origin ID qualification, including the same-UID-different-cluster case Related to datum-cloud/compute#100 Stack: - #239 - #240 - #245 👈 - #246 - #248 - #250
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The exporter's informer callback called NATS publish directly, so a slow or unreachable NATS connection blocked the callback outright. Adds a bounded queue between the two: the callback enqueues without blocking, one goroutine drains it into NATS, and a full queue drops the newest event and counts it rather than blocking.
This also introduces
qualifiedMsgID, so that messages are deduped cross-cluster correctly.Test plan
go testDepends on #246
Stack: