The connection to the server rancher.xxx.com was refused - did you specify the right host or port?

[bpopovich44]

I have a HA cluster with three head nodes and three worker nodes. I upgraded rke2 from v1.27.10 to v1.28.9 and then patched all six servers. I am able to run kubectl commands from the head nodes, but when running from other machines I get the connection refused error. Rancher UI is also not available. I've verified swap is off, firewall is off and other items I've researched. I have ran upgrades before with no issues. When I run kubectl events, I have no errors or warnings.
Head nodes -- rhel8
worker nodes --rocky8
I'm not sure where to look but I don't see any errors in logs.

This is the only error I see in the kubelet logs, but not sure what it means:

"StopPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to destroy network for sandbox "03ee58aa646e2ad2d55d66dfeb32ea76e40ff6307a3baf332dec30b842b6c6e8": plugin type="calico" failed (delete): error getting ClusterInformation: Get "https://[10.43.0.1]:443/apis/crd.projectcalico.org/v1/clusterinformations/default": dial tcp 10.43.0.1:443: i/o timeout" podSandboxID="03ee58aa646e2ad2d55d66dfeb32ea76e40ff6307a3baf332dec30b842b6c6e8"

Thank you

[bpopovich44]

kubectl get nodes
E0522 02:18:32.613533 11444 memcache.go:265] couldn't get current server API group list: Get "https://rancher.xxxr.com/k8s/clusters/local/api?timeout=32s": dial tcp 192.168xxx.xxx:443: connect: connection refused
E0522 02:18:32.628092 11444 memcache.go:265] couldn't get current server API group list: Get "https://rancher.xxx.com/k8s/clusters/local/api?timeout=32s": dial tcp 192.168.xxx.xxx:443: connect: connection refused
E0522 02:18:32.643672 11444 memcache.go:265] couldn't get current server API group list: Get "https://rancher.xxx.com/k8s/clusters/local/api?timeout=32s": dial tcp 192.168.xxx.xxx:443: connect: connection refused
E0522 02:18:32.659461 11444 memcache.go:265] couldn't get current server API group list: Get "https://rancher.xxx.com/k8s/clusters/local/api?timeout=32s": dial tcp 192.168.xxx.xxx:443: connect: connection refused
E0522 02:18:32.675392 11444 memcache.go:265] couldn't get current server API group list: Get "https://rancher.xxx.com/k8s/clusters/local/api?timeout=32s": dial tcp 192.168.xxx.xxx:443: connect: connection refused
The connection to the server rancher.xxxr.com was refused - did you specify the right host or port?

[manuelbuil]

192.168.xxx.xxx looks to me like a node-ip. In that case, you'd probably be looking at port 6443. If you use the kubenetes service instead, which normally is at 10.43.0.1, then the port is indeed 443

[manuelbuil]

Have a look at both kubeconfigs in head and worker nodes, maybe it quickly shows the problem

[brandond]

This also appears to be a duplicate of #5948 - please don't open multiple discussions for the same problem.