Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ home: https://github.com/validatedpatterns/aap-config-chart.git
keywords:
- pattern
name: aap-config
version: 0.2.7
version: 0.2.8
dependencies:
- name: vp-rbac
version: '0.1.*'
Expand Down
8 changes: 7 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# aap-config

![Version: 0.2.7](https://img.shields.io/badge/Version-0.2.7-informational?style=flat-square)
![Version: 0.2.8](https://img.shields.io/badge/Version-0.2.8-informational?style=flat-square)

A Helm chart to build and deploy secrets using external-secrets for ansible-edge-gitops

Expand Down Expand Up @@ -59,6 +59,12 @@ to 600 (10 min). Run the validation job as an Argo Sync hook at sync-wave 2 with
`HookSucceeded,BeforeHookCreation` so ExternalSecrets apply in wave 1 before
validation runs and each sync retry gets a fresh job without OutOfSync drift.

* v0.2.8: Make all jobs drain-cancelable: a `podFailurePolicy` ignores
DisruptionTarget drain evictions and exit code 143 (SIGTERM), so an evicted
pod does not burn `backoffLimit` and a replacement pod resumes the work, and
`terminationGracePeriodSeconds` is set to 60 so a node drain is never blocked
waiting for a job pod.

### Git authentication secret (`agof.gitAuthSecret`)

When `agof_repo` or the config-as-code repo (`agof.cac_repo` / `agof.iac_repo`) is
Expand Down
6 changes: 6 additions & 0 deletions README.md.gotmpl
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,12 @@ to 600 (10 min). Run the validation job as an Argo Sync hook at sync-wave 2 with
`HookSucceeded,BeforeHookCreation` so ExternalSecrets apply in wave 1 before
validation runs and each sync retry gets a fresh job without OutOfSync drift.

* v0.2.8: Make all jobs drain-cancelable: a `podFailurePolicy` ignores
DisruptionTarget drain evictions and exit code 143 (SIGTERM), so an evicted
pod does not burn `backoffLimit` and a replacement pod resumes the work, and
`terminationGracePeriodSeconds` is set to 60 so a node drain is never blocked
waiting for a job pod.

### Git authentication secret (`agof.gitAuthSecret`)

When `agof_repo` or the config-as-code repo (`agof.cac_repo` / `agof.iac_repo`) is
Expand Down
3 changes: 3 additions & 0 deletions templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@
{{- define "aap-config.app.configjobspec" -}}
restartPolicy: Never
serviceAccountName: {{ $.Values.serviceAccountName }}
# SIGTERM'd containers exit promptly; keep the grace period short so a node
# drain is never blocked waiting for this pod.
terminationGracePeriodSeconds: 60
volumes:
- name: agof-scratch-space
emptyDir:
Expand Down
14 changes: 14 additions & 0 deletions templates/agof-config-bootstrap-job.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,20 @@ metadata:
annotations:
argocd.argoproj.io/sync-wave: "3"
spec:
# A drain-evicted (cancelled) pod is ignored rather than counted as a
# failure: the Job stays incomplete and a replacement pod retries the
# configuration. Only genuine failures count toward backoffLimit.
# No containerName: both the agof-init (bash) and agof-config
# (timeout-wrapped make) containers exit 143 on SIGTERM.
podFailurePolicy:
rules:
- action: Ignore
onExitCodes:
operator: In
values: [143]
- action: Ignore
onPodConditions:
- type: DisruptionTarget
parallelism: 1
completions: 1
backoffLimit: 99
Expand Down
13 changes: 13 additions & 0 deletions templates/agof-config-cronjob.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,19 @@ spec:
concurrencyPolicy: Forbid
jobTemplate:
spec:
# A drain-evicted (cancelled) pod is ignored rather than counted as a
# failure; only genuine failures count toward backoffLimit.
# No containerName: both the agof-init (bash) and agof-config
# (timeout-wrapped make) containers exit 143 on SIGTERM.
podFailurePolicy:
rules:
- action: Ignore
onExitCodes:
operator: In
values: [143]
- action: Ignore
onPodConditions:
- type: DisruptionTarget
template:
spec:
activeDeadlineSeconds: {{ $.Values.configJob.activeDeadlineSeconds }}
Expand Down
22 changes: 21 additions & 1 deletion templates/external-secrets-validation-job.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,19 @@ metadata:
argocd.argoproj.io/hook: Sync
argocd.argoproj.io/hook-delete-policy: HookSucceeded,BeforeHookCreation
spec:
# A drain-evicted (cancelled) pod is ignored rather than counted as a
# failure: the Job stays incomplete and a replacement pod resumes the
# validation. Only genuine failures count toward backoffLimit.
podFailurePolicy:
rules:
- action: Ignore
onExitCodes:
containerName: secrets-validator
operator: In
values: [143]
- action: Ignore
onPodConditions:
- type: DisruptionTarget
parallelism: 1
completions: 1
backoffLimit: {{ $.Values.validationJob.backoffLimit }}
Expand All @@ -17,6 +30,9 @@ spec:
spec:
restartPolicy: Never
serviceAccountName: {{ $.Values.serviceAccountName }}
# SIGTERM trap exits promptly; keep the grace period short so a node
# drain is never blocked waiting for this pod.
terminationGracePeriodSeconds: 60
containers:
- name: secrets-validator
image: {{ .Values.configJob.image }}
Expand All @@ -26,7 +42,11 @@ spec:
- -c
- |
set -euo pipefail

# A cancelled run must never be marked successful: exit 143 on
# SIGTERM so the podFailurePolicy ignores it and a replacement
# pod resumes the wait.
trap 'echo "Validation cancelled (SIGTERM); exiting 143" >&2; exit 143' TERM

echo "Starting external secrets validation..."

# List of external secrets to validate
Expand Down
91 changes: 91 additions & 0 deletions tests/drain-cancelable_test.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
suite: Test jobs are drain-cancelable
templates:
- templates/external-secrets-validation-job.yaml
- templates/agof-config-bootstrap-job.yaml
- templates/agof-config-cronjob.yaml
release:
name: release-test
tests:
- it: Validation job ignores drain evictions and exit 143
template: templates/external-secrets-validation-job.yaml
asserts:
# Drain evictions are ignored, not counted as failures
- equal:
path: spec.podFailurePolicy.rules[0].action
value: Ignore
- equal:
path: spec.podFailurePolicy.rules[0].onExitCodes.containerName
value: secrets-validator
- equal:
path: spec.podFailurePolicy.rules[0].onExitCodes.operator
value: In
- equal:
path: spec.podFailurePolicy.rules[0].onExitCodes.values
value: [143]
- equal:
path: spec.podFailurePolicy.rules[1].action
value: Ignore
- equal:
path: spec.podFailurePolicy.rules[1].onPodConditions[0].type
value: DisruptionTarget
# Short grace period so a drain is never blocked
- equal:
path: spec.template.spec.terminationGracePeriodSeconds
value: 60
# SIGTERM trap exits 143 (nonzero) so the Job never completes on cancel
- matchRegex:
path: spec.template.spec.containers[0].command[2]
pattern: trap '.*exit 143' TERM

- it: Bootstrap job ignores drain evictions and exit 143
template: templates/agof-config-bootstrap-job.yaml
asserts:
- equal:
path: spec.podFailurePolicy.rules[0].action
value: Ignore
- equal:
path: spec.podFailurePolicy.rules[0].onExitCodes.operator
value: In
- equal:
path: spec.podFailurePolicy.rules[0].onExitCodes.values
value: [143]
- equal:
path: spec.podFailurePolicy.rules[1].action
value: Ignore
- equal:
path: spec.podFailurePolicy.rules[1].onPodConditions[0].type
value: DisruptionTarget
- equal:
path: spec.template.spec.terminationGracePeriodSeconds
value: 60

- it: Config cronjob ignores drain evictions and exit 143
template: templates/agof-config-cronjob.yaml
asserts:
- equal:
path: spec.jobTemplate.spec.podFailurePolicy.rules[0].action
value: Ignore
- equal:
path: spec.jobTemplate.spec.podFailurePolicy.rules[0].onExitCodes.operator
value: In
- equal:
path: spec.jobTemplate.spec.podFailurePolicy.rules[0].onExitCodes.values
value: [143]
- equal:
path: spec.jobTemplate.spec.podFailurePolicy.rules[1].action
value: Ignore
- equal:
path: spec.jobTemplate.spec.podFailurePolicy.rules[1].onPodConditions[0].type
value: DisruptionTarget
- equal:
path: spec.jobTemplate.spec.template.spec.terminationGracePeriodSeconds
value: 60

- it: Does not render the validation job when disabled
template: templates/external-secrets-validation-job.yaml
set:
validationJob:
disabled: true
asserts:
- hasDocuments:
count: 0
Loading