Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/CLI.md
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ hold the primary role), and a target that is already active (prints a no-op noti

```
$ kubectl dba dc-dr switchover cli-test -n demo --to dc-a
Switchover of demo/cli-test to "dc-a" requested (scope primary-dc-clitest, from PlacementPolicy cli-test-pp failoverPolicy trigger (Group "clitest")).
Switchover of demo/cli-test to "dc-a" requested (scope primary-dc-clitest, from PlacementPolicy cli-test-pp failoverGroupRef "clitest").
The operator will quiesce, wait for catch-up, and hand off; zero committed rows are lost.
Monitor: kubectl dba dc-dr status cli-test -n demo
Abort: kubectl dba dc-dr abort cli-test -n demo
Expand Down Expand Up @@ -102,7 +102,7 @@ operator's own gates read, so the display cannot drift from the real decision.
$ kubectl dba dc-dr status pg-dcdr -n demo
Database: demo/pg-dcdr (Ready)
Failover scope: primary-dc
(PlacementPolicy dcdr-postgres failoverPolicy trigger (Global))
(PlacementPolicy dcdr-postgres failoverPolicy (Global))
Member DCs: dc-a, dc-b
Active DC: dc-b DR phase: Steady
Protected: true (data center "dc-a" is streaming 0 bytes behind, within the 16777216 byte budget)
Expand Down Expand Up @@ -329,13 +329,13 @@ kubectl dba dc-dr active-dc DB_NAME -n NS -q # just the DC name, for sc
```

Given a database, its scope is resolved exactly as the operator resolves it (the
PlacementPolicy's `failoverPolicy.trigger`, falling back to the legacy
`dr.kubedb.com/failover-group` annotation, else Global) and the matching Lease is read.
PlacementPolicy's `failoverPolicy.failoverGroupRef`, else Global) and the matching
Lease is read.

```
$ kubectl dba dc-dr active-dc pg-dcdr -n demo
Active DC: dc-b
Lease: dc-failover/primary-dc (scope from PlacementPolicy dcdr-postgres failoverPolicy trigger (Global))
Lease: dc-failover/primary-dc (scope from PlacementPolicy dcdr-postgres failoverPolicy (Global))
Renewed: 2s ago (lease duration 45s)
Transitions:41
Members: dc-a,dc-b
Expand Down Expand Up @@ -371,7 +371,7 @@ Diagnosing failover for demo/pg-dcdr

[OK ] database is DC-DR distributed and armed
[OK ] failover scope resolves to primary-dc
PlacementPolicy dcdr-postgres failoverPolicy trigger (Global)
PlacementPolicy dcdr-postgres failoverPolicy (Global)
[WARN] holder "dc-b" is renewing normally, so the authority will NOT move on its own
the Lease was renewed 5s ago, inside its 45s duration: that data center's agent is alive
-> this is by design: no database-level condition (client errors, QPS, lag, a crashed
Expand Down
2 changes: 1 addition & 1 deletion pkg/dcdr/activedc.go
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ func NewCmdActiveDC(f cmdutil.Factory) *cobra.Command {
authority for which data center is active.

Given a database name, its failover scope is resolved first (the
PlacementPolicy's failoverPolicy trigger, exactly as the operator
PlacementPolicy's failoverPolicy failoverGroupRef, exactly as the operator
resolves it) and the matching Lease is read. Given --lease, that Lease is
read directly, which also works for a scope whose database is gone.

Expand Down
59 changes: 20 additions & 39 deletions pkg/dcdr/dcdr.go
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,6 @@ const (
AnnSwitchoverStart = "dr.kubedb.com/switchover-started"
AnnQuiesceActive = "dr.kubedb.com/quiesce-active"
AnnAcceptDataLoss = "dr.kubedb.com/accept-failover-data-loss"
AnnFailoverGroup = "dr.kubedb.com/failover-group"
AnnMaxLagBytes = "dr.kubedb.com/switchover-max-lag-bytes"

AnnLeaseHandoffTo = "dr.open-cluster-management.io/handoff-to"
Expand Down Expand Up @@ -202,49 +201,31 @@ type Scope struct {
}

// ResolveScopeForDB mirrors the operator's scopeForDB: the PlacementPolicy's
// failoverPolicy.trigger is the source of truth (Group with a name, else Global);
// the dr.kubedb.com/failover-group annotation is consulted only when the policy
// carries no failoverPolicy; otherwise the scope is Global.
// failoverPolicy.failoverGroupRef is the source of truth (a named FailoverGroup
// follows its primary-dc-<group> Lease); no reference, or no policy, is Global.
func ResolveScopeForDB(ctx context.Context, dyn dynamic.Interface, db *unstructured.Unstructured) (*Scope, error) {
ppName, _, _ := unstructured.NestedString(db.Object, "spec", "podTemplate", "spec", "podPlacementPolicy", "name")
if ppName != "" {
pp, err := dyn.Resource(PlacementGVR).Get(ctx, ppName, metav1.GetOptions{})
if err == nil {
s := &Scope{MemberDCs: memberDCsFromPP(pp)}
trigger, found, _ := unstructured.NestedMap(pp.Object, "spec", "clusterSpreadConstraint", "failoverPolicy", "trigger")
if found {
scope, _ := trigger["scope"].(string)
group, _ := trigger["group"].(string)
if scope == "Group" && group != "" {
s.LeaseName = GroupLeasePrefix + group
s.Source = fmt.Sprintf("PlacementPolicy %s failoverPolicy trigger (Group %q)", ppName, group)
return s, nil
}
s.LeaseName = GlobalPrimaryLease
s.Source = fmt.Sprintf("PlacementPolicy %s failoverPolicy trigger (Global)", ppName)
return s, nil
}
// Policy exists but registers no failoverPolicy: back-compat annotation.
if g := db.GetAnnotations()[AnnFailoverGroup]; g != "" {
s.LeaseName = GroupLeasePrefix + g
s.Source = fmt.Sprintf("annotation %s (PlacementPolicy %s has no failoverPolicy)", AnnFailoverGroup, ppName)
return s, nil
}
s.LeaseName = GlobalPrimaryLease
s.Source = fmt.Sprintf("default Global (PlacementPolicy %s has no failoverPolicy); WARNING: this scope may not be registered, so no Lease may exist and protection may not be armed", ppName)
return s, nil
}
// The policy is referenced but unreadable: fall through to the annotation,
// but say so.
if g := db.GetAnnotations()[AnnFailoverGroup]; g != "" {
return &Scope{LeaseName: GroupLeasePrefix + g, Source: fmt.Sprintf("annotation %s (PlacementPolicy %s unreadable: %v)", AnnFailoverGroup, ppName, err)}, nil
}
if ppName == "" {
return &Scope{LeaseName: GlobalPrimaryLease, Source: "default Global (no PlacementPolicy set)"}, nil
}
pp, err := dyn.Resource(PlacementGVR).Get(ctx, ppName, metav1.GetOptions{})
if err != nil {
return &Scope{LeaseName: GlobalPrimaryLease, Source: fmt.Sprintf("default Global (PlacementPolicy %s unreadable: %v)", ppName, err)}, nil
}
if g := db.GetAnnotations()[AnnFailoverGroup]; g != "" {
return &Scope{LeaseName: GroupLeasePrefix + g, Source: "annotation " + AnnFailoverGroup}, nil
s := &Scope{MemberDCs: memberDCsFromPP(pp)}
if _, found, _ := unstructured.NestedMap(pp.Object, "spec", "clusterSpreadConstraint", "failoverPolicy"); !found {
s.LeaseName = GlobalPrimaryLease
s.Source = fmt.Sprintf("default Global (PlacementPolicy %s has no failoverPolicy); WARNING: this scope may not be registered, so no Lease may exist and protection may not be armed", ppName)
return s, nil
}
if group, _, _ := unstructured.NestedString(pp.Object, "spec", "clusterSpreadConstraint", "failoverPolicy", "failoverGroupRef", "name"); group != "" {
s.LeaseName = GroupLeasePrefix + group
s.Source = fmt.Sprintf("PlacementPolicy %s failoverGroupRef %q", ppName, group)
return s, nil
}
return &Scope{LeaseName: GlobalPrimaryLease, Source: "default Global (no PlacementPolicy set)"}, nil
s.LeaseName = GlobalPrimaryLease
s.Source = fmt.Sprintf("PlacementPolicy %s failoverPolicy (Global)", ppName)
return s, nil
}

func memberDCsFromPP(pp *unstructured.Unstructured) []string {
Expand Down
2 changes: 1 addition & 1 deletion pkg/dcdr/debug.go
Original file line number Diff line number Diff line change
Expand Up @@ -129,7 +129,7 @@ func newCmdDebugFailover(f cmdutil.Factory) *cobra.Command {
findings = append(findings, finding{ok: true, title: "database is DC-DR distributed and armed"})
}
if strings.Contains(scope.Source, "WARNING") {
findings = append(findings, finding{title: "failover scope is registered", detail: scope.Source, remedy: "register the scope in the PlacementPolicy (clusterSpreadConstraint.failoverPolicy.trigger); an unregistered scope has no Lease and no protection", blocker: true})
findings = append(findings, finding{title: "failover scope is registered", detail: scope.Source, remedy: "register the scope in the PlacementPolicy (clusterSpreadConstraint.failoverPolicy, with failoverGroupRef for a group); an unregistered scope has no Lease and no protection", blocker: true})
} else {
findings = append(findings, finding{ok: true, title: fmt.Sprintf("failover scope resolves to %s", scope.LeaseName), detail: scope.Source})
}
Expand Down
Loading