From 72ec51a0dbf84499fc831892ba516eef4a952592 Mon Sep 17 00:00:00 2001 From: Ashish shah Date: Thu, 8 Oct 2026 14:40:23 +0530 Subject: [PATCH 1/2] curated hands-on lab steps for Analyzing Heap and Off-Heap Settings --- ...-memory-in-containerized-environments.adoc | 319 +++++++++--------- 1 file changed, 166 insertions(+), 153 deletions(-) diff --git a/modules/risks-and-memory-management/pages/managing-off-heap-memory-in-containerized-environments.adoc b/modules/risks-and-memory-management/pages/managing-off-heap-memory-in-containerized-environments.adoc index 1c56255..1d9bc12 100644 --- a/modules/risks-and-memory-management/pages/managing-off-heap-memory-in-containerized-environments.adoc +++ b/modules/risks-and-memory-management/pages/managing-off-heap-memory-in-containerized-environments.adoc @@ -156,155 +156,201 @@ _Note: If your application is highly dependent on Off-Heap (e.g., uses large Dir This activity guides you through identifying the memory footprint of your Java process. -=== Step 1: Heap details +. Use `jcmd` to inspect the memory distribution of the running containerized application at runtime. -Use `jcmd` to inspect the memory distribution of the running containerized application at runtime. - -. Assuming the same application as before, the terminal of your running pod: +. Assuming the same application as before. + [source,bash] ---- -$ oc get deployment +oc get deployment +oc get pods +---- ++ +.Example outpput: ++ +---- +[lab-user@bastion ~]$ oc get deployment NAME READY UP-TO-DATE AVAILABLE AGE hotrod-app 1/1 1 1 2m16s -$ oc rsh +[lab-user@bastion ~]$ +[lab-user@bastion ~]$ oc get pods +NAME READY STATUS RESTARTS AGE +hotrod-app-1-build 0/1 Completed 0 12m +hotrod-app-7569b6c6cd-52dqg 1/1 Running 0 4m28s +[lab-user@bastion ~]$ ---- -. Confirming the default MaxRAMPercentage value at 80% via jcmd VM.info's `command line`: +. Connect to the running container using the `oc rsh` command: + [source,bash] ---- -$ jcmd 1 VM.info -Command Line: -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar +oc rsh ---- -+ -NOTE: Since the above deployment method does not set container limits by default, it will default to the host's limits. This directly impacts JVM heap allocation. For example, if the heap size exceeds 32 GB, Compressed Oops will automatically be disabled, and the Heap Address line will be omitted from the `VM.info` output. -. Although one may have CPU, without setting memory limits, the container will be limitless, and Java will take the limit from the host size (MaxRAMPercentage at 80% of the host): +. Confirm the default MaxRAMPercentage value is at 80% via jcmd VM.info's `Command Line`: + [source,bash] ---- -$ jcmd 1 VM.info -1: -# -# JRE version: OpenJDK Runtime Environment (Red_Hat-17.0.20.1.1-1) ---------------- S U M M A R Y ------------ +jcmd 1 VM.info | grep 'Command Line' +---- ++ +.Example output: ++ +---- +sh-5.1$ jcmd 1 VM.info | grep 'Command Line' Command Line: -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar -Host: Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz, 8 cores, 31G <----- 31G -Time: Fri Sep 18 16:07:52 2026 UTC elapsed time: 14.701644 seconds (0d 0h 0m 14s) ---------------- P R O C E S S --------------- -Heap address: 0x00000001bba00000, size: 25670 MB <------------------- 25Gb (80% of 31G) -... -... - -... -... -container (cgroup) information: -container_type: cgroupv2 <------------------------------------------ verifying cgroups v2 -cpu_cpuset_cpus: not supported -cpu_memory_nodes: not supported -active_processor_count: 8 -cpu_quota: no quota -cpu_period: 100000 -cpu_shares: 28 -memory_limit_in_bytes: unlimited <------------------ unlimited -memory_and_swap_limit_in_bytes: unlimited +sh-5.1$ ---- ++ +Note `-XX:MaxRAMPercentage=80.0` set by default in the run-java.sh script. This means that the JVM will use up to 80% of the container's memory limit for the Java Heap. -**Let's set a limit on the container memory and verify the percentage:** +. Check the Heap Max Capacity, container type, and memory limit in bytes: ++ +[source,bash] +---- +jcmd 1 VM.info | grep -e 'Heap Max Capacity' -e 'container_type' -e 'memory_limit_in_bytes' +exit +---- ++ +.Example output: +---- +sh-5.1$ jcmd 1 VM.info | grep -e 'Heap Max Capacity' -e 'container_type' -e 'memory_limit_in_bytes' + Heap Max Capacity: 103040M <------------------- 103G (80% of 128G) +container_type: cgroupv2 <------------------------------------------ verifying cgroups v2 +memory_limit_in_bytes: unlimited <------------------ no memory limit set +sh-5.1$ exit +[lab-user@bastion ~]$ +---- -. Set 2Gi at limit on the container and leave it at default 80% +. Check the memory capacity of the node running the pod using the `free` command: + [source,bash] ---- -### -### Set resources in deployment at 2Gi -### -$ oc set resources deployment/hotrod-app --limits=memory="2Gi" -deployment.apps/hotrod-app resource requirements updated -### -### Go inside the pod -### -$ oc get pod -NAME READY STATUS RESTARTS AGE -hotrod-app-ID 1/1 Running 0 9s -### -### Go inside the pod -### -$ oc rsh hotrod-app-ID -### -### jcmd with VM.info -### -sh-5.1$ jcmd 1 VM.info -1: -# -# JRE version: OpenJDK Runtime Environment (Red_Hat-17.0.20.1.1-1) (17.0.20.1+1) (build 17.0.20.1+1-LTS) -# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-17.0.20.1.1-1) (17.0.20.1+1-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, parallel gc, linux-amd64) ---------------- S U M M A R Y ------------ -Command Line: -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar ---------------- P R O C E S S --------------- -Heap address: 0x0000000099800000, size: 1640 MB <---------------------- 1640 of 2000MB +oc get pods -o wide +oc debug node/ -- bash -c "free -m" ---- ++ +Replace with the name of the node where your pod is running. ++ +.Example output: ++ +---- +[lab-user@bastion ~]$ oc get pods -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +hotrod-app-1-build 0/1 Completed 0 44m 10.232.0.221 control-plane-cluster-phzlt-1 +hotrod-app-7569b6c6cd-52dqg 1/1 Running 0 37m 10.232.0.223 control-plane-cluster-phzlt-1 +[lab-user@bastion ~]$ +[lab-user@bastion ~]$ oc debug node/control-plane-cluster-phzlt-1 -- bash -c "free -m" +Temporary namespace openshift-debug-vq288 is created for debugging node... +Starting pod/control-plane-cluster-phzlt-1-debug-72n7v ... +To use host binaries, run `chroot /host`. Instead, if you need to access host namespaces, run `nsenter -a -t 1`. + total used free shared buff/cache available +Mem: 128797 21721 55812 273 52770 107075 +Swap: 0 0 0 + +Removing debug pod ... +Temporary namespace openshift-debug-vq288 was removed. +[lab-user@bastion ~]$ +---- ++ +Note the total memory of the node is 128G, which matches the Heap Max Capacity of 103G (80% of 128G) seen in the previous step. ++ +NOTE: Although a container may have CPU limits configured, without an explicit memory limit, it can use memory without a defined container-level constraint. In this case, Java determines its memory limit based on the host's available memory, with MaxRAMPercentage set to 80% of the host memory. -**Assuming 80% is an inadequate value, let's set for a lower value:** +. Let's set a limit on the container memory and verify the percentage. -. Set 50% MaxRAMPercentage: +. Set 2Gi limit on the container and leave the MaxRAMPercentage at default 80% + [source,bash] ---- -### -### Set MaxRAMPercentage at 50% (rather than 80%) -### -$ oc set env deployment/hotrod-app JAVA_OPTS_APPEND="-XX:MaxRAMPercentage=50.0" -deployment.apps/hotrod-app updated -### -### Get the pod -### -$ oc get pod -NAME READY STATUS RESTARTS AGE -hotrod-app-ID 1/1 Running 0 3s -### -### Verify with VM.info -### -$ jcmd 1 VM.info -1: -# -# JRE version: OpenJDK Runtime Environment (Red_Hat-17.0.20.1.1-1) (17.0.20.1+1) (build 17.0.20.1+1-LTS) -# Java VM: OpenJDK 64-Bit Server VM (Red_Hat-17.0.20.1.1-1) (17.0.20.1+1-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, parallel gc, linux-amd64) +oc set resources deployment/hotrod-app --limits=memory="2Gi" +---- ---------------- S U M M A R Y ------------ +. Get the pod name and go inside the pod: ++ +[source,bash] +---- +oc get pod +oc rsh +---- -Command Line: -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=50.0 <--------------- 50% is appended -/deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar ---------------- P R O C E S S --------------- -Heap address: 0x00000000c0000000, size: 1024 MB, Compressed Oops mode: 32-bit <---- 1024MB of 2000MB +. Check the Heap Max Capacity and memory limit in bytes: ++ +[source,bash] +---- +jcmd 1 VM.info | grep -e 'Heap Max Capacity' -e 'memory_limit_in_bytes' +exit +---- ++ +.Example output: +---- +sh-5.1$ jcmd 1 VM.info | grep -e 'Heap Max Capacity' -e 'memory_limit_in_bytes' + Heap Max Capacity: 1640M <------------------- 1640M (80% of 2Gi) +memory_limit_in_bytes: 2097152 k <------------------ 2Gi limit set on the container +sh-5.1$ ---- -=== Step 2: Off-heap details +. Assuming 80% is an inadequate value, let's set the MaxRAMPercentage to 50% and verify the Heap Max Capacity again. ++ +[source,bash] +---- +oc set env deployment/hotrod-app JAVA_OPTS_APPEND="-XX:MaxRAMPercentage=50.0" +oc get pod +oc rsh +---- + +. Check the MaxRAMPercentage, Heap Max Capacity and memory limit in bytes: ++ +[source,bash] +---- +jcmd 1 VM.info | grep -e 'Command Line' -e 'Heap Max Capacity' -e 'memory_limit_in_bytes' +exit +---- ++ +.Example output: +---- +sh-5.1$ jcmd 1 VM.info | grep -e 'Command Line' -e 'Heap Max Capacity' -e 'memory_limit_in_bytes' +Command Line: -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=50.0 /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar + Heap Max Capacity: 1G +memory_limit_in_bytes: 2097152 k +sh-5.1$ exit +[lab-user@bastion ~]$ +---- ++ +Note that the `Command Line` shows both `-XX:MaxRAMPercentage=80.0` and `-XX:MaxRAMPercentage=50.0`. The latter takes precedence, resulting in a Heap Max Capacity of 1G (50% of 2Gi). -For more details on off-heap settings, deploy with NativeMemoryTracking (NMT) as below: +. For more details on off-heap settings, deploy with NativeMemoryTracking (NMT) as below: . Deploy the application with NativeMemoryTracking summary (or details) + [source,bash] ---- -$ oc set env deployment/hotrod-app JAVA_OPTS_APPEND="-XX:NativeMemoryTracking=summary" +oc set env deployment/hotrod-app JAVA_OPTS_APPEND="-XX:NativeMemoryTracking=summary" ---- +. Connect to the running container using the `oc rsh` command ++ +[source,bash] +---- +oc get pod +oc rsh +---- . Confirm the application has the correct JVM argument, NativeMemoryTracking, applied: + [source,bash] ---- -VM Arguments: -jvm_args: -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -XX:NativeMemoryTracking=summary <--------------------------- HERE -java_command: /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar -java_class_path (initial): /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar -Launcher Type: SUN_STANDARD +jcmd 1 VM.info | grep jvm_args ---- - + -Full example output for VM.native summary: +.Example output: +---- +sh-5.1$ jcmd 1 VM.info | grep jvm_args +jvm_args: -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -XX:NativeMemoryTracking=summary <-------------- HERE +sh-5.1$ +---- + +. Full example output for VM.native_memory summary: + ---- sh-5.3$ jcmd 1 VM.native_memory summary @@ -363,36 +409,16 @@ Total: reserved=28725786KB, committed=1518126KB (arena=680KB #1) (at peak) ---- + -NOTE: If you see "Native memory tracking is not enabled", you must enable it by adding the following JVM option: ----- -### Option 1 --XX:NativeMemoryTracking=summary -### Option 2 --XX:NativeMemoryTracking=details ----- +If you see "Native memory tracking is not enabled", you must enable it by adding the following JVM option: -. Update the current deployment with this setting -+ -[source,bash] ----- -$ oc set env deployment/hotrod-app JAVA_OPTS_APPEND="-XX:NativeMemoryTracking=details" ----- +* Option 1 - `-XX:NativeMemoryTracking=summary` +* Option 2 - `-XX:NativeMemoryTracking=details` -. Example: +. Update the current deployment with this setting + [source,bash] ---- -cat > hotrod-app.yaml << EOF -apiVersion: apps/v1 -kind: Deployment -metadata: - name: hotrod-app - spec: - containers: -... - env: - - name: JAVA_OPTS_APPEND - value: "-XX:NativeMemoryTracking=details" +$ oc set env deployment/hotrod-app JAVA_OPTS_APPEND="-XX:NativeMemoryTracking=detail" ---- . Access the terminal of your new running pod: @@ -436,44 +462,31 @@ Total: reserved=33545316KB, committed=288676KB ---- -. Analyze the output. For off-heap, look at sections such as: - -* **Internal:** Memory used by the JVM internal structures. -* **Symbol:** Memory used for string interning. -* **Thread:** Memory consumed by thread/stack spaces. +**Additional Notes:** -=== Step 3: Simulate Constraint Pressure -To ensure your configuration is resilient, verify that your application does not exceed the container limit. +* To analyze the output for off-heap, look at sections such as: + ** **Internal:** Memory used by the JVM internal structures. + ** **Symbol:** Memory used for string interning. + ** **Thread:** Memory consumed by thread/stack spaces. -1. View the current cgroup memory limit for your container: +* To ensure your configuration is resilient, verify that your application does not exceed the container limit. +* View the current cgroup memory limit for your container: + -[source,bash] ---- ### CGROUPS V1 cat /sys/fs/cgroup/memory/memory.limit_in_bytes ### CGROUPS V2 cat /sys/fs/cgroup/memory.max - ---- -2. Compare this value against your calculated Heap + Off-Heap usage. If your application's `Total` committed memory (as seen in NMT) + heap usage exceeds this limit, one must adjust your `MaxRAMPercentage` downward or increase the container's memory limit in OpenShift `Deployment` YAML. - -=== Troubleshooting Tip -The following risks are possible on the deployment in orchestrators like Openshift 4: - -[cols="1,3", options="header"] -|=== -| Event | Description +* Compare this value against your calculated Heap + Off-Heap usage. If your application's `Total` committed memory (as seen in NMT) + heap usage exceeds this limit, one must adjust your `MaxRAMPercentage` downward or increase the container's memory limit in OpenShift `Deployment` YAML. -| **OOMKill** -| The full memory is above the container size or cgroups v2 detection is failing. +**Troubleshooting Tip:** -| **OOMException** -| Usually off-heap allocation or metaspace. Note that if the `ExitOnOutOfMemoryError` flag is enabled, the JVM will exit. +The following risks are possible on the deployment in orchestrators like Openshift 4: -| **SIGTERM** -| In case communication is impacted or the JVM is unresponsive, SIGTERM can kick in. That will be in the events. -|=== +* **OOMKill**: The full memory is above the container size or cgroups v2 detection is failing. +* **OOMException**: Usually off-heap allocation or metaspace, not that if `ExitOnOutOfMemoryError` flag is enabled, so then the JVM will exit. +* **SIGTERM**: In case communication is impacted or the JVM is unresponsive sigterm can kick in. That will be in the events. -If you encounter `cgroup OOMKills` despite having a small Java Heap, use `jcmd` to identify the category consuming unexpected amounts of native memory. From that, investigate off-heap components such as non-container aware libraries, thread pool sizing, and do an investigation from the off-heap size perspective. -In the first example above, deployment with `oc new-app` does not set limits so the deployment will be unbounded. +If you encounter `cgroup OOMKills` despite having a small Java Heap, use `jcmd` to identify the category consuming unexpected amounts of native memory. From that, investigate off-heap components such as non-container aware libraries, thread pool sizing, and do an investigation from the off-heap size perspective. \ No newline at end of file From 61d0b7f8a56cfa8854d5e3a3582e593e580b9767 Mon Sep 17 00:00:00 2001 From: Ashish shah Date: Thu, 8 Oct 2026 14:45:55 +0530 Subject: [PATCH 2/2] Moved hands-on lab instructions for heap-settings to new page. --- modules/risks-and-memory-management/nav.adoc | 1 + .../pages/hands-on-lab-heap-settings.adoc | 338 +++++++++++++++++ ...-memory-in-containerized-environments.adoc | 341 +----------------- 3 files changed, 340 insertions(+), 340 deletions(-) create mode 100644 modules/risks-and-memory-management/pages/hands-on-lab-heap-settings.adoc diff --git a/modules/risks-and-memory-management/nav.adoc b/modules/risks-and-memory-management/nav.adoc index ffba845..3c669c4 100644 --- a/modules/risks-and-memory-management/nav.adoc +++ b/modules/risks-and-memory-management/nav.adoc @@ -1,4 +1,5 @@ * xref:risks-and-memory-management.adoc[] ** xref:implications-of-decoupling-heap-memory-from-container-limits.adoc[] ** xref:managing-off-heap-memory-in-containerized-environments.adoc[] +** xref:hands-on-lab-heap-settings.adoc[] ** xref:best-practices-for-java-memory-configuration-in-containers.adoc[] diff --git a/modules/risks-and-memory-management/pages/hands-on-lab-heap-settings.adoc b/modules/risks-and-memory-management/pages/hands-on-lab-heap-settings.adoc new file mode 100644 index 0000000..c4c6750 --- /dev/null +++ b/modules/risks-and-memory-management/pages/hands-on-lab-heap-settings.adoc @@ -0,0 +1,338 @@ += Hands-on Activity: Analyzing Heap and Off-Heap Settings + +This activity guides you through identifying the memory footprint of your Java process. + +. Use `jcmd` to inspect the memory distribution of the running containerized application at runtime. + +. Assuming the same application as before. ++ +[source,bash] +---- +oc get deployment +oc get pods +---- ++ +.Example outpput: ++ +---- +[lab-user@bastion ~]$ oc get deployment +NAME READY UP-TO-DATE AVAILABLE AGE +hotrod-app 1/1 1 1 2m16s +[lab-user@bastion ~]$ +[lab-user@bastion ~]$ oc get pods +NAME READY STATUS RESTARTS AGE +hotrod-app-1-build 0/1 Completed 0 12m +hotrod-app-7569b6c6cd-52dqg 1/1 Running 0 4m28s +[lab-user@bastion ~]$ +---- + +. Connect to the running container using the `oc rsh` command: ++ +[source,bash] +---- +oc rsh +---- + +. Confirm the default MaxRAMPercentage value is at 80% via jcmd VM.info's `Command Line`: ++ +[source,bash] +---- +jcmd 1 VM.info | grep 'Command Line' +---- ++ +.Example output: ++ +---- +sh-5.1$ jcmd 1 VM.info | grep 'Command Line' +Command Line: -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar +sh-5.1$ +---- ++ +Note `-XX:MaxRAMPercentage=80.0` set by default in the run-java.sh script. This means that the JVM will use up to 80% of the container's memory limit for the Java Heap. + +. Check the Heap Max Capacity, container type, and memory limit in bytes: ++ +[source,bash] +---- +jcmd 1 VM.info | grep -e 'Heap Max Capacity' -e 'container_type' -e 'memory_limit_in_bytes' +exit +---- ++ +.Example output: +---- +sh-5.1$ jcmd 1 VM.info | grep -e 'Heap Max Capacity' -e 'container_type' -e 'memory_limit_in_bytes' + Heap Max Capacity: 103040M <------------------- 103G (80% of 128G) +container_type: cgroupv2 <------------------------------------------ verifying cgroups v2 +memory_limit_in_bytes: unlimited <------------------ no memory limit set +sh-5.1$ exit +[lab-user@bastion ~]$ +---- + +. Check the memory capacity of the node running the pod using the `free` command: ++ +[source,bash] +---- +oc get pods -o wide +oc debug node/ -- bash -c "free -m" +---- ++ +Replace with the name of the node where your pod is running. ++ +.Example output: ++ +---- +[lab-user@bastion ~]$ oc get pods -o wide +NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES +hotrod-app-1-build 0/1 Completed 0 44m 10.232.0.221 control-plane-cluster-phzlt-1 +hotrod-app-7569b6c6cd-52dqg 1/1 Running 0 37m 10.232.0.223 control-plane-cluster-phzlt-1 +[lab-user@bastion ~]$ +[lab-user@bastion ~]$ oc debug node/control-plane-cluster-phzlt-1 -- bash -c "free -m" +Temporary namespace openshift-debug-vq288 is created for debugging node... +Starting pod/control-plane-cluster-phzlt-1-debug-72n7v ... +To use host binaries, run `chroot /host`. Instead, if you need to access host namespaces, run `nsenter -a -t 1`. + total used free shared buff/cache available +Mem: 128797 21721 55812 273 52770 107075 +Swap: 0 0 0 + +Removing debug pod ... +Temporary namespace openshift-debug-vq288 was removed. +[lab-user@bastion ~]$ +---- ++ +Note the total memory of the node is 128G, which matches the Heap Max Capacity of 103G (80% of 128G) seen in the previous step. ++ +NOTE: Although a container may have CPU limits configured, without an explicit memory limit, it can use memory without a defined container-level constraint. In this case, Java determines its memory limit based on the host's available memory, with MaxRAMPercentage set to 80% of the host memory. + +. Let's set a limit on the container memory and verify the percentage. + +. Set 2Gi limit on the container and leave the MaxRAMPercentage at default 80% ++ +[source,bash] +---- +oc set resources deployment/hotrod-app --limits=memory="2Gi" +---- + +. Get the pod name and go inside the pod: ++ +[source,bash] +---- +oc get pod +oc rsh +---- + +. Check the Heap Max Capacity and memory limit in bytes: ++ +[source,bash] +---- +jcmd 1 VM.info | grep -e 'Heap Max Capacity' -e 'memory_limit_in_bytes' +exit +---- ++ +.Example output: +---- +sh-5.1$ jcmd 1 VM.info | grep -e 'Heap Max Capacity' -e 'memory_limit_in_bytes' + Heap Max Capacity: 1640M <------------------- 1640M (80% of 2Gi) +memory_limit_in_bytes: 2097152 k <------------------ 2Gi limit set on the container +sh-5.1$ +---- + +. Assuming 80% is an inadequate value, let's set the MaxRAMPercentage to 50% and verify the Heap Max Capacity again. ++ +[source,bash] +---- +oc set env deployment/hotrod-app JAVA_OPTS_APPEND="-XX:MaxRAMPercentage=50.0" +oc get pod +oc rsh +---- + +. Check the MaxRAMPercentage, Heap Max Capacity and memory limit in bytes: ++ +[source,bash] +---- +jcmd 1 VM.info | grep -e 'Command Line' -e 'Heap Max Capacity' -e 'memory_limit_in_bytes' +exit +---- ++ +.Example output: +---- +sh-5.1$ jcmd 1 VM.info | grep -e 'Command Line' -e 'Heap Max Capacity' -e 'memory_limit_in_bytes' +Command Line: -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=50.0 /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar + Heap Max Capacity: 1G +memory_limit_in_bytes: 2097152 k +sh-5.1$ exit +[lab-user@bastion ~]$ +---- ++ +Note that the `Command Line` shows both `-XX:MaxRAMPercentage=80.0` and `-XX:MaxRAMPercentage=50.0`. The latter takes precedence, resulting in a Heap Max Capacity of 1G (50% of 2Gi). + +. For more details on off-heap settings, deploy with NativeMemoryTracking (NMT) as below: + +. Deploy the application with NativeMemoryTracking summary (or details) ++ +[source,bash] +---- +oc set env deployment/hotrod-app JAVA_OPTS_APPEND="-XX:NativeMemoryTracking=summary" +---- + +. Connect to the running container using the `oc rsh` command ++ +[source,bash] +---- +oc get pod +oc rsh +---- + +. Confirm the application has the correct JVM argument, NativeMemoryTracking, applied: ++ +[source,bash] +---- +jcmd 1 VM.info | grep jvm_args +---- ++ +.Example output: +---- +sh-5.1$ jcmd 1 VM.info | grep jvm_args +jvm_args: -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -XX:NativeMemoryTracking=summary <-------------- HERE +sh-5.1$ +---- + +. Full example output for VM.native_memory summary: ++ +---- +sh-5.3$ jcmd 1 VM.native_memory summary +1: +$ jcmd 1 VM.native_memory +1: +Native Memory Tracking: (Omitting categories weighting less than 1KB) +Total: reserved=28725786KB, committed=1518126KB + malloc: 62206KB #254051 + mmap: reserved=28663580KB, committed=1455920KB + +- Java Heap (reserved=26286080KB, committed=514048KB) + (mmap: reserved=26286080KB, committed=514048KB) + +- Class (reserved=1049363KB, committed=5779KB) + (classes #8643) + ( instance classes #8081, array classes #562) + (malloc=787KB #18256) (at peak) + (mmap: reserved=1048576KB, committed=4992KB) + ( Metadata: ) + ( reserved=65536KB, committed=32256KB) + ( used=32053KB) + ( waste=203KB =0.63%) + ( Class space:) + ( reserved=1048576KB, committed=4992KB) + ( used=4780KB) + ( waste=212KB =4.25%) + +- Thread (reserved=39016KB, committed=2836KB) + (thread #39) + (stack: reserved=38912KB, committed=2732KB) + (malloc=61KB #232) (peak=72KB #242) + (arena=43KB #74) (peak=471KB #42) + +- Code (reserved=248583KB, committed=14187KB) + (malloc=895KB #5162) (peak=895KB #5163) + (mmap: reserved=247688KB, committed=13292KB) + +- GC (reserved=989213KB, committed=905329KB) + (malloc=28853KB #134) (peak=28861KB #136) + (mmap: reserved=960360KB, committed=876476KB) + +- Compiler (reserved=195KB, committed=195KB) + (malloc=30KB #355) (peak=66KB #139) + (arena=165KB #5) (peak=54165KB #23) + +- Internal (reserved=12650KB, committed=12650KB) + (malloc=12614KB #5439) (peak=12741KB #5348) + (mmap: reserved=36KB, committed=36KB) + +- Other (reserved=4892KB, committed=4892KB) + (malloc=4892KB #20) (peak=4902KB #22) + +- Symbol (reserved=9203KB, committed=9203KB) + (malloc=8524KB #220921) (at peak) + (arena=680KB #1) (at peak) +---- ++ +If you see "Native memory tracking is not enabled", you must enable it by adding the following JVM option: + +* Option 1 - `-XX:NativeMemoryTracking=summary` +* Option 2 - `-XX:NativeMemoryTracking=details` + +. Update the current deployment with this setting ++ +[source,bash] +---- +$ oc set env deployment/hotrod-app JAVA_OPTS_APPEND="-XX:NativeMemoryTracking=detail" +---- + +. Access the terminal of your new running pod: ++ +[source,bash] +---- +oc rsh +---- + +. Inspect the memory distribution: ++ +[source,bash] +---- +jcmd 1 VM.native_memory summary +---- ++ +Example output: ++ +[source,bash] +---- +sh-5.3$ jcmd 1 VM.native_memory summary +Picked up JAVA_OPTS_APPEND: -XX:NativeMemoryTracking=summary +1: +Native Memory Tracking: +(Omitting categories weighting less than 1KB) + +Total: reserved=33545316KB, committed=288676KB + malloc: 59876KB #80575, peak=111544KB #81451 + mmap: reserved=33485440KB, committed=228800KB + +- Java Heap (reserved=31424512KB, committed=147456KB) + (mmap: reserved=31424512KB, committed=147456KB, at peak) + +. . . + +- String Deduplication (reserved=1KB, committed=1KB) + (malloc=1KB tag=String Deduplication #8) (at peak) + +- Object Monitors (reserved=75KB, committed=75KB) + (malloc=75KB tag=Object Monitors #385) (at peak) + +---- + +**Additional Notes:** + +* To analyze the output for off-heap, look at sections such as: + ** **Internal:** Memory used by the JVM internal structures. + ** **Symbol:** Memory used for string interning. + ** **Thread:** Memory consumed by thread/stack spaces. + +* To ensure your configuration is resilient, verify that your application does not exceed the container limit. +* View the current cgroup memory limit for your container: ++ +---- +### CGROUPS V1 +cat /sys/fs/cgroup/memory/memory.limit_in_bytes +### CGROUPS V2 +cat /sys/fs/cgroup/memory.max +---- + +* Compare this value against your calculated Heap + Off-Heap usage. If your application's `Total` committed memory (as seen in NMT) + heap usage exceeds this limit, one must adjust your `MaxRAMPercentage` downward or increase the container's memory limit in OpenShift `Deployment` YAML. + +**Troubleshooting Tip:** + +The following risks are possible on the deployment in orchestrators like Openshift 4: + +* **OOMKill**: The full memory is above the container size or cgroups v2 detection is failing. +* **OOMException**: Usually off-heap allocation or metaspace, not that if `ExitOnOutOfMemoryError` flag is enabled, so then the JVM will exit. +* **SIGTERM**: In case communication is impacted or the JVM is unresponsive sigterm can kick in. That will be in the events. + +If you encounter `cgroup OOMKills` despite having a small Java Heap, use `jcmd` to identify the category consuming unexpected amounts of native memory. From that, investigate off-heap components such as non-container aware libraries, thread pool sizing, and do an investigation from the off-heap size perspective. \ No newline at end of file diff --git a/modules/risks-and-memory-management/pages/managing-off-heap-memory-in-containerized-environments.adoc b/modules/risks-and-memory-management/pages/managing-off-heap-memory-in-containerized-environments.adoc index 1d9bc12..bfd8a3f 100644 --- a/modules/risks-and-memory-management/pages/managing-off-heap-memory-in-containerized-environments.adoc +++ b/modules/risks-and-memory-management/pages/managing-off-heap-memory-in-containerized-environments.adoc @@ -150,343 +150,4 @@ _Note: If your application is highly dependent on Off-Heap (e.g., uses large Dir ---- # Enable NMT in the container -XX:NativeMemoryTracking=summary (or detail) ----- - -== Hands-on Activity: Analyzing Heap and Off-Heap Settings - -This activity guides you through identifying the memory footprint of your Java process. - -. Use `jcmd` to inspect the memory distribution of the running containerized application at runtime. - -. Assuming the same application as before. -+ -[source,bash] ----- -oc get deployment -oc get pods ----- -+ -.Example outpput: -+ ----- -[lab-user@bastion ~]$ oc get deployment -NAME READY UP-TO-DATE AVAILABLE AGE -hotrod-app 1/1 1 1 2m16s -[lab-user@bastion ~]$ -[lab-user@bastion ~]$ oc get pods -NAME READY STATUS RESTARTS AGE -hotrod-app-1-build 0/1 Completed 0 12m -hotrod-app-7569b6c6cd-52dqg 1/1 Running 0 4m28s -[lab-user@bastion ~]$ ----- - -. Connect to the running container using the `oc rsh` command: -+ -[source,bash] ----- -oc rsh ----- - -. Confirm the default MaxRAMPercentage value is at 80% via jcmd VM.info's `Command Line`: -+ -[source,bash] ----- -jcmd 1 VM.info | grep 'Command Line' ----- -+ -.Example output: -+ ----- -sh-5.1$ jcmd 1 VM.info | grep 'Command Line' -Command Line: -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar -sh-5.1$ ----- -+ -Note `-XX:MaxRAMPercentage=80.0` set by default in the run-java.sh script. This means that the JVM will use up to 80% of the container's memory limit for the Java Heap. - -. Check the Heap Max Capacity, container type, and memory limit in bytes: -+ -[source,bash] ----- -jcmd 1 VM.info | grep -e 'Heap Max Capacity' -e 'container_type' -e 'memory_limit_in_bytes' -exit ----- -+ -.Example output: ----- -sh-5.1$ jcmd 1 VM.info | grep -e 'Heap Max Capacity' -e 'container_type' -e 'memory_limit_in_bytes' - Heap Max Capacity: 103040M <------------------- 103G (80% of 128G) -container_type: cgroupv2 <------------------------------------------ verifying cgroups v2 -memory_limit_in_bytes: unlimited <------------------ no memory limit set -sh-5.1$ exit -[lab-user@bastion ~]$ ----- - -. Check the memory capacity of the node running the pod using the `free` command: -+ -[source,bash] ----- -oc get pods -o wide -oc debug node/ -- bash -c "free -m" ----- -+ -Replace with the name of the node where your pod is running. -+ -.Example output: -+ ----- -[lab-user@bastion ~]$ oc get pods -o wide -NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES -hotrod-app-1-build 0/1 Completed 0 44m 10.232.0.221 control-plane-cluster-phzlt-1 -hotrod-app-7569b6c6cd-52dqg 1/1 Running 0 37m 10.232.0.223 control-plane-cluster-phzlt-1 -[lab-user@bastion ~]$ -[lab-user@bastion ~]$ oc debug node/control-plane-cluster-phzlt-1 -- bash -c "free -m" -Temporary namespace openshift-debug-vq288 is created for debugging node... -Starting pod/control-plane-cluster-phzlt-1-debug-72n7v ... -To use host binaries, run `chroot /host`. Instead, if you need to access host namespaces, run `nsenter -a -t 1`. - total used free shared buff/cache available -Mem: 128797 21721 55812 273 52770 107075 -Swap: 0 0 0 - -Removing debug pod ... -Temporary namespace openshift-debug-vq288 was removed. -[lab-user@bastion ~]$ ----- -+ -Note the total memory of the node is 128G, which matches the Heap Max Capacity of 103G (80% of 128G) seen in the previous step. -+ -NOTE: Although a container may have CPU limits configured, without an explicit memory limit, it can use memory without a defined container-level constraint. In this case, Java determines its memory limit based on the host's available memory, with MaxRAMPercentage set to 80% of the host memory. - -. Let's set a limit on the container memory and verify the percentage. - -. Set 2Gi limit on the container and leave the MaxRAMPercentage at default 80% -+ -[source,bash] ----- -oc set resources deployment/hotrod-app --limits=memory="2Gi" ----- - -. Get the pod name and go inside the pod: -+ -[source,bash] ----- -oc get pod -oc rsh ----- - -. Check the Heap Max Capacity and memory limit in bytes: -+ -[source,bash] ----- -jcmd 1 VM.info | grep -e 'Heap Max Capacity' -e 'memory_limit_in_bytes' -exit ----- -+ -.Example output: ----- -sh-5.1$ jcmd 1 VM.info | grep -e 'Heap Max Capacity' -e 'memory_limit_in_bytes' - Heap Max Capacity: 1640M <------------------- 1640M (80% of 2Gi) -memory_limit_in_bytes: 2097152 k <------------------ 2Gi limit set on the container -sh-5.1$ ----- - -. Assuming 80% is an inadequate value, let's set the MaxRAMPercentage to 50% and verify the Heap Max Capacity again. -+ -[source,bash] ----- -oc set env deployment/hotrod-app JAVA_OPTS_APPEND="-XX:MaxRAMPercentage=50.0" -oc get pod -oc rsh ----- - -. Check the MaxRAMPercentage, Heap Max Capacity and memory limit in bytes: -+ -[source,bash] ----- -jcmd 1 VM.info | grep -e 'Command Line' -e 'Heap Max Capacity' -e 'memory_limit_in_bytes' -exit ----- -+ -.Example output: ----- -sh-5.1$ jcmd 1 VM.info | grep -e 'Command Line' -e 'Heap Max Capacity' -e 'memory_limit_in_bytes' -Command Line: -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=50.0 /deployments/hotrodspringboot-0.0.1-SNAPSHOT.jar - Heap Max Capacity: 1G -memory_limit_in_bytes: 2097152 k -sh-5.1$ exit -[lab-user@bastion ~]$ ----- -+ -Note that the `Command Line` shows both `-XX:MaxRAMPercentage=80.0` and `-XX:MaxRAMPercentage=50.0`. The latter takes precedence, resulting in a Heap Max Capacity of 1G (50% of 2Gi). - -. For more details on off-heap settings, deploy with NativeMemoryTracking (NMT) as below: - -. Deploy the application with NativeMemoryTracking summary (or details) -+ -[source,bash] ----- -oc set env deployment/hotrod-app JAVA_OPTS_APPEND="-XX:NativeMemoryTracking=summary" ----- - -. Connect to the running container using the `oc rsh` command -+ -[source,bash] ----- -oc get pod -oc rsh ----- - -. Confirm the application has the correct JVM argument, NativeMemoryTracking, applied: -+ -[source,bash] ----- -jcmd 1 VM.info | grep jvm_args ----- -+ -.Example output: ----- -sh-5.1$ jcmd 1 VM.info | grep jvm_args -jvm_args: -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:+ExitOnOutOfMemoryError -XX:NativeMemoryTracking=summary <-------------- HERE -sh-5.1$ ----- - -. Full example output for VM.native_memory summary: -+ ----- -sh-5.3$ jcmd 1 VM.native_memory summary -1: -$ jcmd 1 VM.native_memory -1: -Native Memory Tracking: (Omitting categories weighting less than 1KB) -Total: reserved=28725786KB, committed=1518126KB - malloc: 62206KB #254051 - mmap: reserved=28663580KB, committed=1455920KB - -- Java Heap (reserved=26286080KB, committed=514048KB) - (mmap: reserved=26286080KB, committed=514048KB) - -- Class (reserved=1049363KB, committed=5779KB) - (classes #8643) - ( instance classes #8081, array classes #562) - (malloc=787KB #18256) (at peak) - (mmap: reserved=1048576KB, committed=4992KB) - ( Metadata: ) - ( reserved=65536KB, committed=32256KB) - ( used=32053KB) - ( waste=203KB =0.63%) - ( Class space:) - ( reserved=1048576KB, committed=4992KB) - ( used=4780KB) - ( waste=212KB =4.25%) - -- Thread (reserved=39016KB, committed=2836KB) - (thread #39) - (stack: reserved=38912KB, committed=2732KB) - (malloc=61KB #232) (peak=72KB #242) - (arena=43KB #74) (peak=471KB #42) - -- Code (reserved=248583KB, committed=14187KB) - (malloc=895KB #5162) (peak=895KB #5163) - (mmap: reserved=247688KB, committed=13292KB) - -- GC (reserved=989213KB, committed=905329KB) - (malloc=28853KB #134) (peak=28861KB #136) - (mmap: reserved=960360KB, committed=876476KB) - -- Compiler (reserved=195KB, committed=195KB) - (malloc=30KB #355) (peak=66KB #139) - (arena=165KB #5) (peak=54165KB #23) - -- Internal (reserved=12650KB, committed=12650KB) - (malloc=12614KB #5439) (peak=12741KB #5348) - (mmap: reserved=36KB, committed=36KB) - -- Other (reserved=4892KB, committed=4892KB) - (malloc=4892KB #20) (peak=4902KB #22) - -- Symbol (reserved=9203KB, committed=9203KB) - (malloc=8524KB #220921) (at peak) - (arena=680KB #1) (at peak) ----- -+ -If you see "Native memory tracking is not enabled", you must enable it by adding the following JVM option: - -* Option 1 - `-XX:NativeMemoryTracking=summary` -* Option 2 - `-XX:NativeMemoryTracking=details` - -. Update the current deployment with this setting -+ -[source,bash] ----- -$ oc set env deployment/hotrod-app JAVA_OPTS_APPEND="-XX:NativeMemoryTracking=detail" ----- - -. Access the terminal of your new running pod: -+ -[source,bash] ----- -oc rsh ----- - -. Inspect the memory distribution: -+ -[source,bash] ----- -jcmd 1 VM.native_memory summary ----- -+ -Example output: -+ -[source,bash] ----- -sh-5.3$ jcmd 1 VM.native_memory summary -Picked up JAVA_OPTS_APPEND: -XX:NativeMemoryTracking=summary -1: -Native Memory Tracking: -(Omitting categories weighting less than 1KB) - -Total: reserved=33545316KB, committed=288676KB - malloc: 59876KB #80575, peak=111544KB #81451 - mmap: reserved=33485440KB, committed=228800KB - -- Java Heap (reserved=31424512KB, committed=147456KB) - (mmap: reserved=31424512KB, committed=147456KB, at peak) - -. . . - -- String Deduplication (reserved=1KB, committed=1KB) - (malloc=1KB tag=String Deduplication #8) (at peak) - -- Object Monitors (reserved=75KB, committed=75KB) - (malloc=75KB tag=Object Monitors #385) (at peak) - ----- - -**Additional Notes:** - -* To analyze the output for off-heap, look at sections such as: - ** **Internal:** Memory used by the JVM internal structures. - ** **Symbol:** Memory used for string interning. - ** **Thread:** Memory consumed by thread/stack spaces. - -* To ensure your configuration is resilient, verify that your application does not exceed the container limit. -* View the current cgroup memory limit for your container: -+ ----- -### CGROUPS V1 -cat /sys/fs/cgroup/memory/memory.limit_in_bytes -### CGROUPS V2 -cat /sys/fs/cgroup/memory.max ----- - -* Compare this value against your calculated Heap + Off-Heap usage. If your application's `Total` committed memory (as seen in NMT) + heap usage exceeds this limit, one must adjust your `MaxRAMPercentage` downward or increase the container's memory limit in OpenShift `Deployment` YAML. - -**Troubleshooting Tip:** - -The following risks are possible on the deployment in orchestrators like Openshift 4: - -* **OOMKill**: The full memory is above the container size or cgroups v2 detection is failing. -* **OOMException**: Usually off-heap allocation or metaspace, not that if `ExitOnOutOfMemoryError` flag is enabled, so then the JVM will exit. -* **SIGTERM**: In case communication is impacted or the JVM is unresponsive sigterm can kick in. That will be in the events. - -If you encounter `cgroup OOMKills` despite having a small Java Heap, use `jcmd` to identify the category consuming unexpected amounts of native memory. From that, investigate off-heap components such as non-container aware libraries, thread pool sizing, and do an investigation from the off-heap size perspective. \ No newline at end of file +---- \ No newline at end of file