[Fix-17786][SeaTunnel] Cancel Zeta jobs via seatunnel.sh -can - #18492
richardmilles wants to merge 1 commit into
Conversation
Parse SeaTunnel Engine job id from logs and cancel the engine job after killing the local client process.
| for (String line : Files.readAllLines(Paths.get(logPath), StandardCharsets.UTF_8)) { | ||
| String jobId = findSeaTunnelJobId(line); | ||
| if (jobId != null) { | ||
| jobIds.add(jobId); | ||
| } | ||
| } |
There was a problem hiding this comment.
readAllLines method can OOM the worker.
findSeaTunnelJobIdsFromLog calls it after every Zeta task exits, batch or streaming, success or failure. Streaming tasks are exactly the target of this PR and run for weeks; the client prints a job-progress table every 60s by default, local deploy mode dumps all engine logs to stdout, and Console sinks print every row. A multi-hundred-MB log becomes several times that on the heap. The worker runs with a 4g default heap and -XX:+ExitOnOutOfMemoryError, so one such task takes down the worker and every task on it. DS's own precedent for the same job, LogUtils.getAppIdsFromLogFile, streams with Files.lines.
Minimum fix: stream line by line and stop at the first match. Better fix: capture the id from the stdout stream while the task runs instead of re-reading the file at all.
This branch has not been deployed
2 participants
Was this PR generated or assisted by AI?
Yes, assisted by AI.
Purpose of the pull request
Fixes #17786
Replaces #18469 (closed for missing PR template).
SeaTunnel Zeta streaming jobs are not cancelled after the client is killed, so tasks can keep running.
Brief change log
seatunnel.sh -can <jobId>after client killVerify this pull request
This pull request is already covered by existing tests where available.
Pull Request Notice
Pull Request Notice
If your pull request contains incompatible change, you should also add it to
docs/docs/en/guide/upgrade/incompatible.md