Wrong Memory Usage Number for Nodes

[sass1997]

Environmental Info:
RKE2 Version:
v1.24.7+rke2r1

Node(s) CPU architecture, OS, and Version:
amd64
Ubuntu 22.04

Cluster Configuration:
3 Server
3 Agents

Describe the bug:
My Cluster is reporting me 50% Usage for Memory on Worker-01.

I've checked the usage via kubectl top / k9s / Rancher UI
Memory Avaible 15967
Used 7980

I've also summed up the usage of the pod's which are running on this worker.
Total: 724

Then I've checked on the OS my avaible and used memory

$ free -m
               total        used        free      shared  buff/cache   available
Mem:           15967        1394        1861          16       12711       14221
Swap:              0           0           0

In my opinion there is only 20% used and not 50%
Steps To Reproduce:

• Installed RKE2:
  I've installed rke2 on already existing VM's to installed it with the installation command. No Node drivers etc.

Expected behavior:
Correct Number of Used RAM Memory

Actual behavior:
Wrong Number

[brandond]

Simply looking at free or top is not the correct way to gauge memory utilization in the same way that Kubernetes does. Depending on where you're looking, the figures you see for a Kubernetes node may be either resource-based (node allocatable memory, vs pod requests and reservations) or based on observed metrics reported by cgroups and exposed via tools like cadvisor and prometheus/grafana.

The numbers you're seeing in Rancher are probably the former. If you want to see the latter, you can run something like kubectl top pod -A or kubectl top node.

If you want something to run on the node, you could also look at systemd-cgtop and check the memory utilization for the various pod cgroups.

[sass1997]

I've fixed now for all pods that are running the limit. But still doesn't help. I have a memory usage reported 96% but used is in real life only 13.5%

[brandond]

I don't know what that screenshot is from. What makes you think this is an issue with RKE2 or Kubernetes instead of whatever tool that is?

[sass1997]

Sorry forgot to mention this. It's actually c-advisor.
The reason why I still believe there is an issue, of course also misconfiguration from my side possible, is that after I created a cluster and didn't install any additonal tools beside the normal set from rke2. The memory usage is reaching over 80% without any traffic on the cluster.

[jcmontigny]

Have you found where the issue was coming from?
I and another colleague have the same discrepancies on a rke2 1.25.9 cluster, but not to that extent, more like double to triple ram usage versus outputs from free or sar utilities, when referring to the used memory.

For instance on a RKE2 cluster: 2.4GB used reported by the free command, 6.1GB reported by metrics-server:

worker-general-0:~$ free -m
               total        used        free      shared  buff/cache   available
Mem:           32090        2385        4813           7       24891       29234
Swap:              0           0           0
worker-general-0:~$ kubectl top nodes
NAME                CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
worker-general-0    405m         2%     6065Mi          19%

On a minikube cluster, k top nodes and free report the same scale as expected:

$ kubectl -- top nodes
NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
minikube   235m         5%     601Mi           7%
$ free -m
               total        used        free      shared  buff/cache   available
Mem:            7937         859        1830          10        5247        6760
Swap:              0           0           0

[Nialit]

Bump. Experiencing the same issue. Also about x3 times exaggerating.

[devender9010]

same issue for our cluster also only one node reached around 81% and the remaining nodes under 70 % ram usage

[helletheone]

we have the same issue for v1.24.10+rke2r1 version for example: used 8gb but kubectl shows: 11566Mi used

[brandond]

The metrics come from cadvisor and are exposed via kubectl top node by way of the metrics-server project. If you have questions about how these figures are calculated, and why they might differ from what's shown by top, free, or other system resource profiling tools, I would ask on those projects. We do not maintain or modify any of that code.

[bolaum]

Hi, if you guys still have this problem, I found a workaround. It seems like there is a problem with how calculations are made using cgroupv2. Patch will probably come in 1.29. I downgraded my nodes to use cgroupv1 by adding the following kernel parameter:

systemd.unified_cgroup_hierarchy=0

This will force systemd to use cgroupv1. Metrics are a lot more coherent now. I don't know if there are more consequences for doing this, so be careful if doing this in prod envs. Memory resource management could be affected as stated in the k8s docs:

Some Kubernetes features exclusively use cgroup v2 for enhanced resource management and isolation. For example, the MemoryQoS feature improves memory QoS and relies on cgroup v2 primitives.

Related issues and links:
Azure/AKS#3443
kubernetes/kubernetes#118916
https://kubernetes.io/docs/concepts/architecture/cgroups/