Skip to content

Make jobs drain-cancelable - #27

Merged
OAharoni-RedHat merged 1 commit into
validatedpatterns:mainfrom
mhjacks:drain_cancelable_jobs
Nov 16, 2026
Merged

OAharoni-RedHat merged 1 commit into
validatedpatterns:mainfrom
mhjacks:drain_cancelable_jobs

Conversation

@mhjacks

@mhjacks mhjacks commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

The external-secrets validation job is a poll loop and the config jobs
run long ansible playbooks, so a node drain evicting one of these pods
could burn backoffLimit retries or leave the work half-done.

  • Validation job: SIGTERM trap exits 143 so a cancelled run is never
    marked successful
  • All three Jobs (validation, bootstrap, config cronjob): podFailurePolicy
    ignores DisruptionTarget drain evictions and exit code 143, so
    evictions do not burn backoffLimit and a replacement pod resumes the
    work. The config jobs omit containerName since both the agof-init
    (bash) and agof-config (timeout-wrapped make) containers exit 143 on
    SIGTERM.
  • terminationGracePeriodSeconds: 60 on all job pods so a drain is never
    blocked waiting for a pod

Note: the exit-code rule requires operator: In, and the drain-eviction
rule must use onPodConditions with type DisruptionTarget (there is no
onDisruptionConditions field).

Co-Authored-By: Claude Code noreply@anthropic.com

The external-secrets validation job is a poll loop and the config jobs
run long ansible playbooks, so a node drain evicting one of these pods
could burn backoffLimit retries or leave the work half-done.

- Validation job: SIGTERM trap exits 143 so a cancelled run is never
  marked successful
- All three Jobs (validation, bootstrap, config cronjob): podFailurePolicy
  ignores DisruptionTarget drain evictions and exit code 143, so
  evictions do not burn backoffLimit and a replacement pod resumes the
  work. The config jobs omit containerName since both the agof-init
  (bash) and agof-config (timeout-wrapped make) containers exit 143 on
  SIGTERM.
- terminationGracePeriodSeconds: 60 on all job pods so a drain is never
  blocked waiting for a pod

Note: the exit-code rule requires operator: In, and the drain-eviction
rule must use onPodConditions with type DisruptionTarget (there is no
onDisruptionConditions field).

Co-Authored-By: Claude Code <noreply@anthropic.com>
@OAharoni-RedHat
OAharoni-RedHat merged commit 6a3270e into validatedpatterns:main Nov 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants