Skip to content

DRU-611 -- Catch the errors Exec.kill raises in the first-byte killer - #1147

Merged
czpython merged 1 commit into
czpython:mainfrom
p-r-a-v-i-n:dru-611
Nov 10, 2026
Merged

czpython merged 1 commit into
czpython:mainfrom
p-r-a-v-i-n:dru-611

Conversation

@p-r-a-v-i-n

@p-r-a-v-i-n p-r-a-v-i-n commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

_first_byte_killer caught SandboxAPIError and SandboxUnavailableError around run.kill(). Exec.kill makes no drukbox SDK call. It runs the kill helper over SSH through Host.exec, so a failed kill raises an asyncssh.Error, for example asyncssh.TimeoutError when the helper passes its 10s budget. That error escaped the killer, and _watch_run suppressed it when it awaited the task, so a failed kill left no trace.

The killer now catches asyncssh.Error and OSError, the errors the SSH calls in runner.py already catch. It logs the failure with the run id, because a timed-out kill has an empty error message, and returns False, so _watch_run does not report a first-byte kill. That outcome is the same as before. _watch_run now suppresses only CancelledError when it awaits the killer, so an unexpected error in the killer is no longer hidden. The drukbox import that only this clause used is gone.

@p-r-a-v-i-n
p-r-a-v-i-n requested a review from czpython as a code owner November 9, 2026 12:51

@czpython czpython left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new except is right. It catches every error that run.kill() raises, and each test case fails on main. Two parts need to change.

1. Log the run id (host.py)

When the kill passes its 10s timeout, the error text is empty: str(asyncssh.TimeoutError(...)) is ''. The log format in setup_logging has no run id either, so nobody can match the line to a run. Log the run id, as _download_artifacts does:

logger.exception("first-byte killer failed to kill run %s", run.run_id)

The test fake then needs run_id=.

2. Remove Exception from the suppress (host.py)

_watch_run awaits the killer inside contextlib.suppress(asyncio.CancelledError, Exception). The description says this suppress hid the error. The killer now catches what the kill raises, so Exception there only hides bugs. Change it to contextlib.suppress(asyncio.CancelledError).

Tests

Drop the asyncssh.TimeoutError case. It is both an asyncssh.Error and an OSError, so it covers nothing that the other two cases do not. ConnectionLost covers asyncssh.Error, and ConnectionRefusedError covers OSError.

Description

_exec is now _watch_run.

@czpython
czpython enabled auto-merge (squash) November 10, 2026 16:13
@czpython
czpython disabled auto-merge November 10, 2026 16:13
@czpython
czpython merged commit b99c571 into czpython:main Nov 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants