RKE2 Cluster Nodes IP address change

[charlieharcum]

Environmental Info:
RKE2 Version:
rke2 version v1.24.6+rke2r1

Node(s) CPU architecture, OS, and Version: 3
Linux 5.15.0-1035-azure #42~20.04.1-Ubuntu SMP Wed Mar 1 19:17:41 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
3 Nodes, 1 Master Node, 2 Worker Nodes

Describe the bug:
Not sure it's a "bug" per say but we are looking for a possible documented process to change the IP address(s) of all 3 nodes. We are looking to have an air-gapped clone of the existing cluster but it needs to be a different set of IP addresses

Expected behavior:
We would expect that after restoring the nodes from a backup we could change the host IP addresses and possibly update some RKE2 config files on the master and worker nodes and have the cluster work as expected

Actual behavior:
rke2-server service will not start on the master node once the IP address has changed

Additional context / logs:

[brandond]

Server nodes are expected to have a static IP address for the duration of their life in the cluster. If you want to change all of their addresses, you should start the first node with --cluster-reset, then once it comes back up as a single-node etcd cluster with the current address, delete the other nodes from the cluster using kubectl delete node, then rejoin them as if they were new.

[nickderoo]

Hi @brandond
I installed a standalone RKE2 cluster through the install scripts method in attempts to test this procedure, but when I attempt to run the cluster reset command I've been getting issues certificate errors and datastore connection errors that the node is not a member of the etcd cluster.

Do you have any suggestions around the correct process which is suitable for changing domain names and IP addresses post deployment?

INFO[61207] Cluster-Http-Server 2024/07/31 09:00:24 http: TLS handshake error from 127.0.0.1:53830: remote error: tls: bad certificate INFO[61207] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error WARN[61208] Unable to watch for tunnel endpoints: Get "https://127.0.0.1:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&resourceVersion=0&watch=true": dial tcp 127.0.0.1:6443: connect: connection refused I0731 09:00:28.097471 18506 client.go:360] parsed scheme: "passthrough" I0731 09:00:28.097548 18506 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>} I0731 09:00:28.097567 18506 clientconn.go:948] ClientConn switching balancer to "pick_first" I0731 09:00:28.097634 18506 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick I0731 09:00:28.373308 18506 client.go:360] parsed scheme: "passthrough" I0731 09:00:28.373428 18506 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379 <nil> 0 <nil>}] <nil> <nil>} I0731 09:00:28.373461 18506 clientconn.go:948] ClientConn switching balancer to "pick_first" I0731 09:00:28.373589 18506 clientconn.go:897] blockingPicker: the picked transport is not ready, loop back to repick INFO[61210] Failed to test data store connection: this server is a not a member of the etcd cluster. Found [f20-8-vm5-871bb5fc=https://172.18.248.48:2380], expect: f20-8-vm5-871bb5fc=172.18.249.248

[charlieharcum]

Not sure if related, but I've seen sometimes when I go through this process, the local FW of the VM (especially if it's Ubuntu) will cause an issue. LIke even if we had it disabled previously. Just recently restored a RKE2 cluster from backup and was standing up in a separate IP space and so going through the process like normal, the cluster-reset kept failing, not exactly what you have above but similar. After I disabled ufw in Ubuntu and disabled apparmor it worked as expected.

[brandond]

@nickderoo
Failed to test data store connection: this server is a not a member of the etcd cluster. Found [f20-8-vm5-871bb5fc=https://172.18.248.48:2380], expect: f20-8-vm5-871bb5fc=172.18.249.248

Is your VM IP continuing to change even after the cluster-reset? You REALLY need stable node IPs, it's not going to work if you keep getting a new IP from DHCP every time the node reboots. Use static IPs, DHCP reservations, or increase your DHCP lease time.

[nickderoo]

no it's only changing once the process I followed was

1. Stable cluster
2. set new IP
3. reboot
4. systemctl stop rke2-server
5. rke2 server --cluster-reset

[charlieharcum]

Our documented steps that we've been able to reproduce success are below. We work with a 3 node cluster for reference on Ubuntu 20.04... I will note, the last time I threw in a rke2 certificate rotate after the killall just for good measure but don't know if that changed anything.. :)

1. Run systemctl stop rke2-server on the Master Node
2. Run systemctl disable --now ufw to disable firewall
3. Run systemctl disable --now apparmor.service to disable app armor
4. Run rke2-killall.sh on the Master Node
5. Run rke2 server --cluster-reset on the Master Node (takes a while to finish)
6. Run systemctl start rke2-server on the Master Node
7. Reboot Node
8. Apply the KUBECONFIG (export KUBECONFIG=/etc/rancher/rke2/rke2.yaml)
9. Run kubectl get nodes and should see Master Ready and Node 2 and 3 not ready
10. Run kubectl delete node <nodename> to delete each worker node
11. Run systemctl stop rke2-agent on the worker node(s)
12. Edit /etc/rancher/rke2/config.yaml on the worker node(s) and change server IP to the new Master Node IP
13. Run systemctl start rke2-agent on the worker node(s)
14. Runsystemctl status rke2-agent on the worker node(s) to see that service is started or is starting
15. On Master Node, run kubectl get nodes and should now see all Nodes Ready