Repository navigation
DRU-611 -- Catch the errors Exec.kill raises in the first-byte killer - #1147
Conversation
czpython
left a comment
There was a problem hiding this comment.
The new except is right. It catches every error that run.kill() raises, and each test case fails on main. Two parts need to change.
1. Log the run id (host.py)
When the kill passes its 10s timeout, the error text is empty: str(asyncssh.TimeoutError(...)) is ''. The log format in setup_logging has no run id either, so nobody can match the line to a run. Log the run id, as _download_artifacts does:
logger.exception("first-byte killer failed to kill run %s", run.run_id)The test fake then needs run_id=.
2. Remove Exception from the suppress (host.py)
_watch_run awaits the killer inside contextlib.suppress(asyncio.CancelledError, Exception). The description says this suppress hid the error. The killer now catches what the kill raises, so Exception there only hides bugs. Change it to contextlib.suppress(asyncio.CancelledError).
Tests
Drop the asyncssh.TimeoutError case. It is both an asyncssh.Error and an OSError, so it covers nothing that the other two cases do not. ConnectionLost covers asyncssh.Error, and ConnectionRefusedError covers OSError.
Description
_exec is now _watch_run.
_first_byte_killercaughtSandboxAPIErrorandSandboxUnavailableErroraroundrun.kill().Exec.killmakes no drukbox SDK call. It runs the kill helper over SSH throughHost.exec, so a failed kill raises anasyncssh.Error, for exampleasyncssh.TimeoutErrorwhen the helper passes its 10s budget. That error escaped the killer, and_watch_runsuppressed it when it awaited the task, so a failed kill left no trace.The killer now catches
asyncssh.ErrorandOSError, the errors the SSH calls inrunner.pyalready catch. It logs the failure with the run id, because a timed-out kill has an empty error message, and returns False, so_watch_rundoes not report a first-byte kill. That outcome is the same as before._watch_runnow suppresses onlyCancelledErrorwhen it awaits the killer, so an unexpected error in the killer is no longer hidden. The drukbox import that only this clause used is gone.