bug: CPU usage misses other processes that share the same CPU limit (cgroup v2) #41

Summary

This is the CPU version of #38.

clistat gets the CPU limit from the nearest parent cgroup that has one. But it reads CPU usage from the process's own cgroup. Other cgroups under that parent use the same limit, but their usage is not counted. So clistat can show the container as almost idle when it is busy.

This often happens when a container runs other containers inside it. The process is in /init.scope and the inner containers are in /docker/<id>. Both are under the container root, which has the limit.

Why this is wrong

• A cgroup's cpu.stat includes CPU time from all cgroups below it. In the kernel, each cgroup's CPU time is added to its parent's (cgroup_base_stat_flush() in kernel/cgroup/rstat.c, v6.17). If you only want a cgroup's own time, there is a separate file, cpu.stat.local.
• A limit in cpu.max applies to the cgroup and everything below it (cgroup v2 docs).

So to see how close we are to the limit, we must read usage from the cgroup that has the limit.

How to reproduce

We used a sysbox container on cgroup v2. The container root has a 12-core limit (cpu.max is 1200000 100000). /init.scope has no limit, and the process is in /init.scope. Then we ran 4 busy loops in an inner container:

docker run -d --name cpuhog alpine sh -c 'for i in 1 2 3 4; do (while :; do :; done) & done; wait'

What should happen

clistat should show about 4.3/12 cores (36%).

How to fix it

Do the same thing #39 does for memory. Find the cgroup that has the CPU limit, then read CPU usage from that same cgroup. If the process's own cgroup has the limit, or there is no limit, nothing changes.

The nearest limit is not always the one that matters. That is a separate problem, covered in #40.

Generated by Coder Agents on behalf of @DanielleMaywood.