bug: clistat only checks the nearest limit, but every parent's limit applies (cgroup v2) #40

Summary

For memory and CPU, clistat goes up from the process's cgroup and stops at the first limit it finds. But the kernel checks the limit of every parent cgroup, not just the nearest one. Whichever parent runs out of room first is the one that matters. This means a process can be killed for using too much memory while clistat says it is only using a small part of its limit.

This can happen even when every child limit is smaller than its parent's. If other cgroups have used up most of the parent's memory, the child hits the parent's limit first.

#38 and #39 fix the case where only one cgroup has a limit. This issue is about the general case.

Why this is wrong

• When memory is used, the kernel adds it to the cgroup and to every parent above it. If any of them goes over its limit, the allocation fails and that parent's limit is the one hit (page_counter_try_charge() in mm/page_counter.c, v6.17).
• memory.current is the memory used by a cgroup and everything below it (cgroup v2 docs). This is the number the kernel compares against memory.max.
• Child limits can add up to more than the parent's limit (cgroup v2 docs). So a child's limit alone does not tell you if there is room left.
• CPU limits work the same way. A limit applies to the cgroup and everything below it (cgroup v2 docs).

How to reproduce

We tested this with #39 applied. The container root has a 1 GiB limit. Cgroup A, where clistat runs, has a 600 MiB limit. Cgroup B has no limit and uses 850 MiB:

docker run --rm --privileged --memory 1g -v "$PWD/probe:/probe:ro" alpine sh -c '
  cd /sys/fs/cgroup && mkdir leaf A B
  echo $$ > leaf/cgroup.procs && echo +memory > cgroup.subtree_control
  echo 629145600 > A/memory.max
  mkdir /t && mount -t tmpfs -o size=2g tmpfs /t
  sh -c "echo \$\$ > B/cgroup.procs; dd if=/dev/zero of=/t/f bs=1M count=850"
  sh -c "echo \$\$ > A/cgroup.procs; exec /probe"
  sh -c "echo \$\$ > A/cgroup.procs; exec dd if=/dev/zero of=/t/g bs=1M count=300"'

probe is a small static binary that prints ContainerMemory(PrefixMebi).

• clistat in A shows 9.71/600 MiB (2%).
• We then used 300 MiB in A, which is half of its own limit. The process was killed for using too much memory. The root's memory.events shows oom 1, but A's own oom count is still 0. The root's limit was the one hit, not A's.

What should happen

clistat should show the limit that will actually be hit first. Here that is the root: about 895 MB of 1 GiB (83%), not 2%.

How to fix it

Go up from the process's cgroup to the top of the container. For each cgroup that has a limit, read usage from that same cgroup:

• Memory: memory.current - inactive_file, compared with memory.max.
• CPU: the change in cpu.stat usage_usec over the sample time, compared with cpu.max.

Report the cgroup with the least room left (limit minus usage). This also covers #38, #39 and #41.

Open question: should clistat report only the limit that will be hit first, or every limit it finds?

Not covered by this issue

clistat cannot see limits set outside the container. For example, a Kubernetes pod-level limit is not visible from inside a container that has its own cgroup namespace. We should document this, not try to fix it here.

Generated by Coder Agents on behalf of @DanielleMaywood.