RKE2 v1.30.0 +rke2r1, RKE2 agent cant renew certificates.

[cengizhanR]

Rke2's documentation explains how to renew the certificate for agents."To renew agent certificates, restart rke2-agent in agent nodes. Agent certificates are renewed every time the agent starts."
I did as said, but only the kubelet's certificate was renewed, not the kube-proxy and rke2-controller.

[root@rke2agent1 ~]# rke2 certificate check
INFO[0000] Agent detected, checking agent certificates
INFO[0000] Checking certificates for kubelet
INFO[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=system:node:rke2agent1,O=system:nodes is ok, expires at 2025-06-28T15:14:47Z
INFO[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=rke2-client-ca@1717058760 is ok, expires at 2034-05-28T08:46:00Z
INFO[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=rke2agent1 is ok, expires at 2025-06-28T15:14:46Z
INFO[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=rke2-server-ca@1717058760 is ok, expires at 2034-05-28T08:46:00Z
INFO[0000] Checking certificates for rke2-controller
INFO[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=system:rke2-controller is ok, expires at 2025-05-30T09:05:35Z
INFO[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=rke2-client-ca@1717058760 is ok, expires at 2034-05-28T08:46:00Z
INFO[0000] Checking certificates for kube-proxy
INFO[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=system:kube-proxy is ok, expires at 2025-05-30T09:05:35Z
INFO[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=rke2-client-ca@1717058760 is ok, expires at 2034-05-28T08:46:00Z
[root@rke2agent1 ~]# systemctl restart rke2-agent
[root@rke2agent1 ~]# rke2 certificate check
INFO[0000] Agent detected, checking agent certificates
INFO[0000] Checking certificates for kube-proxy
INFO[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=system:kube-proxy is ok, expires at 2025-05-30T09:05:35Z
INFO[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=rke2-client-ca@1717058760 is ok, expires at 2034-05-28T08:46:00Z
INFO[0000] Checking certificates for kubelet
INFO[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=system:node:rke2agent1,O=system:nodes is ok, expires at 2025-07-10T11:42:05Z
INFO[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=rke2-client-ca@1717058760 is ok, expires at 2034-05-28T08:46:00Z
INFO[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=rke2agent1 is ok, expires at 2025-07-10T11:42:05Z
INFO[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=rke2-server-ca@1717058760 is ok, expires at 2034-05-28T08:46:00Z
INFO[0000] Checking certificates for rke2-controller
INFO[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=system:rke2-controller is ok, expires at 2025-05-30T09:05:35Z
INFO[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=rke2-client-ca@1717058760 is ok, expires at 2034-05-28T08:46:00Z

This is config yaml for my agent node

[root@rke2agent1 bilge]# cat /etc/rancher/rke2/config.yaml
server: https://rancher.exemple.com:9345/
system-default-registry: 10.0.0.10:5000
token: fuzzybunnyslippers
write-kubeconfig-mode: "0600"
node-ip: "10.0.0.5"
kube-apiserver-arg:
- authorization-mode=RBAC,Node
kubelet-arg:
- protect-kernel-defaults=true
- read-only-port=0
- authorization-mode=Webhook
kubelet-arg:
- protect-kernel-defaults=true
- read-only-port=0
- authorization-mode=Webhook
- minimum-container-ttl-duration=10s
- maximum-dead-containers-per-container=2
- maximum-dead-containers=240
- image-gc-high-threshold=85
- image-gc-low-threshold=80

[cengizhanR]

I delete the certificate "rm -rf /var/lib/rancher/rke2/agent/client-rke2-controller.crt " and restart rke2-agent. Hovewer, certificate date is the same.

[root@rke2agent1 agent]# rke2 certificate check
INFO[0000] Agent detected, checking agent certificates
INFO[0000] Checking certificates for rke2-controller
INFO[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=system:rke2-controller is ok, expires at 2025-05-17T13:00:35Z
INFO[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=rke2-client-ca@1715943396 is ok, expires at 2034-05-15T10:56:36Z
INFO[0000] Checking certificates for kube-proxy
INFO[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=system:kube-proxy is ok, expires at 2025-05-17T13:00:35Z
INFO[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=rke2-client-ca@1715943396 is ok, expires at 2034-05-15T10:56:36Z
INFO[0000] Checking certificates for kubelet
INFO[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=system:node:rke2agent1,O=system:nodes is ok, expires at 2025-07-10T12:47:10Z
INFO[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=rke2-client-ca@1715943396 is ok, expires at 2034-05-15T10:56:36Z
INFO[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=rke2agent1 is ok, expires at 2025-07-10T12:47:10Z
INFO[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=rke2-server-ca@1715943396 is ok, expires at 2034-05-15T10:56:36Z

I stopped rke-2 agent and used rke2 certificate rotate command and it dosent work either.

[root@rke2agent1 agent]# rke2 certificate rotate
INFO[0000] Agent detected, rotating agent certificates
INFO[0000] Rotating certificates for kube-proxy
INFO[0000] Rotating certificates for kubelet
INFO[0000] Rotating certificates for rke2-controller
INFO[0000] Successfully backed up certificates to /var/lib/rancher/rke2/agent/tls-1720617032, please restart rke2 server or agent to rotate certificates
[root@rke2agent1 agent]# systemctl start rke2-agent
[root@rke2agent1 agent]# rke2 certificate check
INFO[0000] Agent detected, checking agent certificates
INFO[0000] Checking certificates for kubelet
INFO[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=system:node:rke2agent1,O=system:nodes is ok, expires at 2025-07-10T13:10:41Z
INFO[0000] /var/lib/rancher/rke2/agent/client-kubelet.crt: certificate CN=rke2-client-ca@1715943396 is ok, expires at 2034-05-15T10:56:36Z
INFO[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=rke2agent1 is ok, expires at 2025-07-10T13:10:40Z
INFO[0000] /var/lib/rancher/rke2/agent/serving-kubelet.crt: certificate CN=rke2-server-ca@1715943396 is ok, expires at 2034-05-15T10:56:36Z
INFO[0000] Checking certificates for rke2-controller
INFO[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=system:rke2-controller is ok, expires at 2025-05-17T12:13:19Z
INFO[0000] /var/lib/rancher/rke2/agent/client-rke2-controller.crt: certificate CN=rke2-client-ca@1715943396 is ok, expires at 2034-05-15T10:56:36Z
INFO[0000] Checking certificates for kube-proxy
INFO[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=system:kube-proxy is ok, expires at 2025-05-17T12:13:19Z
INFO[0000] /var/lib/rancher/rke2/agent/client-kube-proxy.crt: certificate CN=rke2-client-ca@1715943396 is ok, expires at 2034-05-15T10:56:36Z

For server, All certificates are rotated incluede client-kube-proxy.crt and client-rke2-controller.crt. However, Agent certificates can not be rotated

[brandond]

As the documentation says, you must rotate the server certificates first. Rotate the certificates on ALL of the servers, then rotate the agent certs. The updated kube-proxy and controller certs must be available on servers for agents to retrieve when they restart.

[cengizhanR]

@brandond Thank you