Skip to content

bug: cgroup v2 memory usage ignores sibling cgroups under the limit-owning ancestor (docker-in-docker, devcontainers) #38

Open

Description

@alekhya-amb

Summary

In cgroup v2, memory() reads usage (memory.current - inactive_file) from the process's own cgroup but walks up to an ancestor to find memory.max. When the agent is not the only thing under that ancestor, usage is understated. The common case is docker-in-docker or devcontainer workspaces, where the agent sits in /init.scope and nested containers sit in /docker/<id>, both children of the container cgroup that holds the limit.

The result is that resources_monitoring memory alerts in Coder never fire before the container is OOM-killed, and coder stat mem reports a fraction of real usage.

Related: #21 / #22 / #28 fixed where the limit and missing usage files are read from. This is the remaining case: the usage file exists in the agent's cgroup, so no fallback happens, but it does not cover the whole domain the limit applies to.

Reproduction

  • Kubernetes pod, cgroup v2, one container with a 14 GiB memory limit, running dockerd inside (docker-in-docker). Coder v2.36.4 (clistat v1.2.1).
  • Agent PID 1 is in 0::/init.scope; the devcontainer is in /docker/<id>. init.scope/memory.max is max, the container root holds the limit.
  • Allocate ~11 GiB inside the nested container and hold it for 4 minutes.
Reading Value
container root: memory.current - inactive_file / memory.max 89%
/docker subtree 83%
/init.scope 5%
coder stat mem run by the agent 0.824/14 GiB (6%)
resources_monitoring memory alert (threshold 85%) not triggered

Expected

Usage should be read from the same cgroup that memory.max was found in, so numerator and denominator describe the same domain. That reports 89% here and the 85% alert fires.

Proposed fix

In memory(), after resolving the ancestor that owns the effective memory.max, read memory.current and memory.stat (inactive_file) from that ancestor rather than from the process's own cgroup. When the own cgroup holds the limit (the usual single-process container), behavior is unchanged. Happy to send a PR with a test using a nested cgroup fixture (/init.scope + /docker/<id> under a limited root).

Notes:

  • Reading the ancestor's files works from inside the container's cgroup namespace (the container root is /sys/fs/cgroup), no extra privileges needed.
  • CPU likely has the same shape (cpu.stat from own cgroup vs cpu.max from ancestor); not verified here.

Activity

  1. linear-code commented on Sep 25, 2026

    @linear-code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions