Skip to content

ref(cleanup): Schedule sentry cleanup with python instead of cron - #4544

Open
oioki wants to merge 2 commits into
masterfrom
alextarasov/sentry-cleanup-without-cron
Open

oioki wants to merge 2 commits into
masterfrom
alextarasov/sentry-cleanup-without-cron

Conversation

@oioki

@oioki oioki commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Run sentry-cleanup on sentry-self-hosted-local (the same image and entrypoint as web) with cron/run_daily.py, which runs sentry cleanup --days $SENTRY_EVENT_RETENTION_DAYS every day at midnight, like the old 0 0 * * * crontab. The separate cleanup image, which apt-get installed cron and gosu and used a bash entrypoint, is gone, and install removes the old sentry-cleanup-self-hosted-local image. ./cron is mounted read-only at /cron, like ./healthcheck.

Each run gets its own process group. If a run is still going at the next midnight, it is stopped (SIGTERM, then SIGKILL after 60s) before the next one starts. sentry cleanup has no lock and can hang, e.g. its JoinableQueue.join() never returns if a worker is killed mid-task; with cron, a stuck run kept running while the next day's run overlapped it. On container stop, SIGTERM is forwarded to the current run.

Cleanup runs as the sentry user directly instead of via gosu, and its output still goes to the container logs.

Behavior change: users who override the sentry-cleanup command with a crontab line in docker-compose.override.yml need to switch to the new command form.

Part of making self-hosted run on the distroless sentry image by changing only SENTRY_IMAGE.

Run sentry-cleanup on sentry-self-hosted-local with cron/run_daily.py, which runs sentry cleanup every day at midnight like the old '0 0 * * *' crontab. This drops the separate image that installed cron and gosu with apt-get and its bash entrypoint, so sentry-cleanup also works on a sentry image without a shell.
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Coverage Results 📊

✅ 22 passed | Total: 22 | Pass Rate: 100% | Execution Time: 9m 50s

📊 Comparison with Base Branch

Metric Change
Total Tests —
Passed Tests —
Failed Tests —
Skipped Tests —

✨ Test counts unchanged from base.

All tests are passing successfully.

✅ Patch coverage is 100.00% (no changed executable lines found; target 50%).
Project statement coverage is 95.54% (unchanged from base (86b6d79) to head (3e8ee18)).

Coverage diff
@@            Coverage Diff             @@
##        master     #4544       +/-##
==========================================
  Coverage    95.54%    95.54%        —%
==========================================
  Files            5         5         —
  Tracked lines       336       336         —
  Branches         0         0         —
==========================================
  Hits           321       321         —
  Misses          15        15         —
  Partials         0         0         —

Generated by Coverage Action

@BYK BYK left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this works, I'm game! I hated that extra cron image from the beginning!

…duled run

sentry cleanup has no lock and can hang (e.g. its JoinableQueue.join() never returns if a worker is killed mid-task). Run each cleanup in its own process group and stop it (SIGTERM, then SIGKILL) if it's still running at the next midnight, so a stuck run can't block the schedule and runs never overlap. Forward SIGTERM to the run on container stop. Also remove the obsolete sentry-cleanup-self-hosted-local image during install.
@oioki
oioki marked this pull request as ready for review October 1, 2026 18:05
Comment thread cron/run_daily.py


def stop(proc: subprocess.Popen[bytes]) -> None:
# The run is its own process group, so this also reaches its workers.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: An unhandled ProcessLookupError can crash the script due to a race condition where a subprocess exits before os.killpg() is called.
Severity: MEDIUM

Suggested Fix

Wrap the os.killpg() calls in a try...except ProcessLookupError block to gracefully handle cases where the process has already exited, preventing the script from crashing.

Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: cron/run_daily.py#L31

Potential issue: A race condition can occur if the subprocess exits between the
`current.poll() is None` check and the subsequent call to `os.killpg()`. If the process
group no longer exists when `os.killpg()` is called, it will raise a
`ProcessLookupError`. This exception is not handled within a `try...except` block,
causing the `run_daily.py` script to crash. This can lead to missed daily cleanup runs
or an unclean container shutdown if the race condition is met during the `on_sigterm`
signal handling.

Also affects:

  • cron/run_daily.py:36~36

Did we get this right? 👍 / 👎 to inform future reviews.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is correct.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 3e8ee18. Configure here.

Comment thread cron/run_daily.py
time.sleep(min(remaining, 60))
if current.poll() is None:
print(f"`{' '.join(command)}` still running at the next scheduled run, stopping it", flush=True)
stop(current)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Past deadlines kill freshly started runs

Low Severity

After a long suspend or clock jump past the computed deadline, run_daily.py still launches sentry cleanup and then immediately treats that run as overdue and stops it. deadline is next_midnight(run_at), so any wake-up more than a day after the missed slot makes seconds_until(deadline) already negative. Catch-up attempts are SIGTERM'd right after start until run_at lands on the current day, so the first successful cleanup can be delayed and a just-started run can be interrupted.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit 3e8ee18. Configure here.

@oioki
oioki requested review from aldy505 and aminvakil October 1, 2026 18:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

3 participants