Fail to restart RKE2 server after LAN -> DMZ migration

[Ducatel]

Hi everyone,

I face an issue after migration of all of my node from LAN to DMZ (means IP / Gateway change + new firewall rules)

Now when I try to run /usr/bin/rke2 server --cni calico --debug
I can see many issue about TLS certificate

INFO[0022] Cluster-Http-Server 2024/07/01 16:44:37 http: TLS handshake error from 127.0.0.1:60622: remote error: tls: bad certificate
DEBU[0022] Tunnel server handing HTTP/1.1 GET request for /v1-rke2/connect from 127.0.0.1:60618
INFO[0022] Handling backend connection request [dmz-k8s-1]
INFO[0022] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error
DEBU[0027] Wrote ping
INFO[0027] Cluster-Http-Server 2024/07/01 16:44:43 http: TLS handshake error from 127.0.0.1:60650: remote error: tls: bad certificate
INFO[0027] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error
DEBU[0032] Wrote ping
INFO[0032] Cluster-Http-Server 2024/07/01 16:44:48 http: TLS handshake error from 127.0.0.1:49200: remote error: tls: bad certificate
INFO[0032] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error
{"level":"warn","ts":"2024-07-01T16:44:50.169649+0200","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of

So I suppose this is coming from the fact that my old IPs are stored somewhere But I cannot find where.

I tried to make a curl query to see why it's failing and seems I have issue with CA

curl -v  https://127.0.0.1:9345/v1-rke2/readyz
*   Trying 127.0.0.1:9345...
* Connected to 127.0.0.1 (127.0.0.1) port 9345 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, unknown CA (560):
* SSL certificate problem: self-signed certificate in certificate chain
* Closing connection 0
curl: (60) SSL certificate problem: self-signed certificate in certificate chain
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

And if I bypass the ssl check

 curl -v -k  https://127.0.0.1:9345/v1-rke2/readyz
*   Trying 127.0.0.1:9345...
* Connected to 127.0.0.1 (127.0.0.1) port 9345 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=rke2; CN=rke2
*  start date: Apr  2 12:46:57 2024 GMT
*  expire date: Jun 10 13:38:56 2025 GMT
*  issuer: CN=rke2-server-ca@1712062017
*  SSL certificate verify result: self-signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* Using Stream ID: 1 (easy handle 0x55af2e906d80)
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET /v1-rke2/readyz HTTP/2
> Host: 127.0.0.1:9345
> user-agent: curl/7.76.1
> accept: */*
>
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.2 (IN), TLS header, Unknown (23):
< HTTP/2 403
< content-type: application/json
< content-length: 129
< date: Mon, 01 Jul 2024 08:11:38 GMT
<
* TLSv1.2 (IN), TLS header, Unknown (23):
{
  "kind": "Status",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden",
  "reason": "Forbidden",
  "code": 403
* Connection #0 to host 127.0.0.1 left intact
}[

Here you can find the full log of the rke2 server startup
rke2.log

If you have any ideas, your help will be appreciated
Thanks in advance for your help

Ps: I already tried to run rke2 certificate rotate but doesn't solve anything

Environmental Info:
RKE2 Version:

On windows nodes:

rke2.exe version v1.27.12+rke2r1 (25b27b4e4709a2ac4c550609ad730a9e172d110a)
go version go1.21.8

On linux node:

rke2 version v1.27.12+rke2r1 (25b27b4e4709a2ac4c550609ad730a9e172d110a)
go version go1.21.8 X:boringcrypto

Node(s) CPU architecture, OS, and Version:

On windows nodes:

windows server 2022 21H2 Build 20348.2527

On linux node:

Linux  5.14.0-427.18.1.el9_4.x86_64 rancher/rke2#1 SMP PREEMPT_DYNAMIC Mon May 13 10:47:09 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

• 3 RHEL9 master
• 2 RHEL9 worker
• 2 windows server 2022 worker

[brandond]

You cannot change server node IPs. If you change the server node IPs, you need to reset the etcd cluster membership (rke2 server --cluster-reset) and rejoin all the server nodes with their new IPs.

[Ducatel]

Hi Brandond,

Thanks for the tricks is seems to work 👍
At least I can reconnect my slave (I remove them one by one and re-add them)
But seems to fail for masters. I will investigate a bit, I keep you in touch

Edit 1:
When I run the cluster-reset on masters, I have to remove the server URL from config.yaml

/usr/bin/rke2 server --cluster-reset
WARN[0000] not running in CIS mode
INFO[0000] Applying Pod Security Admission Configuration
INFO[0000] Static pod cleanup in progress
INFO[0000] Logging temporary containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log
INFO[0000] Running temporary containerd /var/lib/rancher/rke2/bin/containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd
INFO[0010] Static pod cleanup completed successfully
FATA[0010] cannot perform cluster-reset while server URL is set - remove server from configuration before resetting

Edit 2:

After I removed the URL from the config file the cluster-reset create an infinit loop on

INFO[0011] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6444/v1-rke2/readyz: 500 Internal Server Error
{"level":"warn","ts":"2024-07-02T15:04:16.605534+0200","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000ae9dc0/127.0.0.1:2379","attempt   ":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""}
WARN[0015] Failed to get apiserver address from etcd: context deadline exceeded
INFO[0030] Pod for etcd not synced (pod sandbox not found), retrying

Edit 3:

It seems the problem come from the fact that I removed the node from the cluster during my test.

[brandond]

You should do a --cluster-reset on the first server, and then on the others remove their db files from disk, update their --server address to point at the first server's new IP, and then start the service. That's what I meant by "rejoin all the server nodes".

[Ducatel]

OK, go it.
Thanks for the explanation.

Can you tell me which files I should remove?
should I remove /var/lib/rancher/rke2 ?

[brandond]

Just the files under server/db.
See steps 4/5 from
https://docs.rke2.io/backup_restore#restoring-a-snapshot-to-existing-nodes

[Ducatel]

Ok this was working to for the master node which was not deleted by myself previously.

On rke2 first master node I see many error like:

Jul 03 10:09:43 dmz-k8s-1 rke2[3062943]: {"level":"warn","ts":"2024-07-03T10:09:43.437051+0200","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d02fc0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: can only promote a learner member which is in sync with leader"}

So I wanted to check what is the member status inside etcd container (still first master)

# 0ad4236ef6213  is the etcd container

/var/lib/rancher/rke2/bin/crictl exec -ti 0ad4236ef6213 etcdctl member list
{"level":"warn","ts":"2024-07-03T08:11:21.152225Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0001c2000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection closed before server preface received"}
Error: context deadline exceeded
FATA[0005] execing command in container: command terminated with exit code 1
```

So I probably need credential or something like that.
I will continue digging.

[Ducatel]

Finally, I just reboot the node which refused to join the cluster and... it joined .

Really thanks you @brandond for your help ;)
(I still didn't find what I have to do to access etcd within the container. )