Repository navigation
fix: report the cgroupv2 limit with the least room left for container cpu and memory - #44
Open
david-fraley wants to merge 1 commit into
Open
david-fraley wants to merge 1 commit into
david-fraley wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #40 (ENG-3479). Also fixes #38 and #41, so this supersedes #39, #42, and #43.
On cgroup v2,
ContainerCPUandContainerMemorystopped at the nearest cgroup with a limit and read usage from the process's own cgroup. The kernel enforces every limit from the process's cgroup up to the container root, so clistat could report 2% while an ancestor limit was about to kill the process, or 0 cores while nested containers were using 4.How it works now
cpu.maxquota ormemory.max).cpu.statandmemory.currentinclude all descendants, so siblings that share the limit are counted.If a limit owner has no usage files, the process's own usage is used for that limit (the same fallback #39 and #43 used). cgroup v1 and the public API are unchanged. The README has a new "How container limits are reported" section that documents this and says that limits above the container root, such as a Kubernetes pod limit behind a private cgroup namespace, are not visible.
Verification
New
NestedLimitstests: only an ancestor sets a limit (bug: cgroup v2 memory usage ignores sibling cgroups under the limit-owning ancestor (docker-in-docker, devcontainers) #38, bug: CPU usage misses other processes that share the same CPU limit (cgroup v2) #41), an ancestor limit runs out first (bug: clistat only checks the nearest limit, but every parent's limit applies (cgroup v2) #40), and the nearest limit runs out first. The first four fail onmain. The nearest-limit cases pass on both and guard the old behavior.All existing tests pass unchanged.
go test -raceandgolangci-lintare clean, exceptTestCgroupV2Detection/OsFsandTestIsContainerized/IsCorrectlyDetected, which also fail onmainwhen run inside a container.Live, in a sysbox workspace (root
cpu.max1200000 100000, agent in/init.scope, 4 busy loops in a nested container):ContainerCPU()main0.004/12 cores (0%)4.01/12 cores (33%)Decisions
coder statneed. Returning every limit would changeResultand push the choice onto every caller. A separate API can expose all limits later if someone needs them for debugging.cgroupStatternow returnscpuCounters(): one counter per limited cgroup, sampled together over the same interval, so CPU room is compared from the same window. cgroup v1 returns a single counter with its old values.cpu.max. With no quota anywhere, the existingcpuPeriod()walk and the 100ms default still apply.Generated by Coder Agents on behalf of @david-fraley.