Unhealthy etcd
#6932
-
|
Hello, I have a problem with etcd continuously restarting. ❯ kubectl get pods
NAME READY STATUS RESTARTS AGE
cloud-controller-manager-rke2-server-1 0/1 Running 5179 (3m59s ago) 20d
etcd-rke2-server-1 0/1 Running 4453 (8s ago) 18h
kube-proxy-rke2-server-1 0/1 Running 4464 (7s ago) 20dMy cluster consists in 3 nodes : ❯ kubectl get nodes
NAME STATUS ROLES AGE VERSION
rke2-agent-1 Ready <none> 20d v1.30.4+rke2r1
rke2-loadbalancer-1 Ready control-plane,master 20d v1.30.4+rke2r1
rke2-server-1 Ready etcd 20d v1.30.4+rke2r1When I look at description of etcd pod, the kubelet can't check health of the pod : ❯ kubectl describe pods etcd-rke2-server-1
...
Warning BackOff 72m (x1876 over 17h) kubelet Back-off restarting failed container etcd in pod etcd-rke2-server-1_kube-system(5716c4a9275006798064896b8e82fcd0)
Warning Unhealthy 53m (x3790 over 18h) kubelet Startup probe failed: Get "http://localhost:2381/health?serializable=true": dial tcp [::1]:2381: connect: connection refused
Normal Killing 19m kubelet Container etcd failed startup probe, will be restarted
Warning Unhealthy 2m22s (x126 over 23m) kubelet Startup probe failed: Get "http://localhost:2381/health?serializable=true": dial tcp [::1]:2381: connect: connection refusedAccessing from the node works fine : There's no error in the logs. May it be a firewall issue? |
Beta Was this translation helpful? Give feedback.
Answered by
MelvinCou
Oct 4, 2024
Replies: 1 comment 3 replies
-
|
localhost appears to resolve to an ipv6 address in the pod when the kubelet is health-checking it. Do you perhaps have ipv6 incorrectly configured on this cluster, or on that node? Please show the output of |
Beta Was this translation helpful? Give feedback.
3 replies
-
|
You're right, ipv6 might be the issue. config.yaml files : # server
❯ cat /etc/rancher/rke2/config.yaml
tls-san:
- **.**.**.**
disable-apiserver: true
disable-controller-manager: true
disable-scheduler: true
# loadbalancer
❯ cat /etc/rancher/rke2/config.yaml
server: https://192.168.1.22:9345
token: *******
tls-san:
- **.**.**.**
disable-etcd: true❯ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
rke2-agent-1 Ready <none> 20d v1.30.4+rke2r1 192.168.1.91 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-25-amd64 containerd://1.7.20-k3s1
rke2-loadbalancer-1 Ready control-plane,master 20d v1.30.4+rke2r1 192.168.1.40 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-25-amd64 containerd://1.7.20-k3s1
rke2-server-1 Ready etcd 20d v1.30.4+rke2r1 192.168.1.22 <none> Debian GNU/Linux 12 (bookworm) 6.1.0-25-amd64 containerd://1.7.20-k3s1❯ kubectl get pod -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
gitlab gitlab-runner-5466c86cf7-ndrch 1/1 Running 0 93m 10.42.2.38 rke2-agent-1 <none> <none>
gitlab gitlab-runner-5466c86cf7-ttl4j 1/1 Running 0 93m 10.42.1.25 rke2-server-1 <none> <none>
kube-system cloud-controller-manager-rke2-loadbalancer-1 1/1 Running 4120 (2m59s ago) 20d 192.168.1.40 rke2-loadbalancer-1 <none> <none>
kube-system cloud-controller-manager-rke2-server-1 0/1 Running 5202 (6m56s ago) 20d 192.168.1.22 rke2-server-1 <none> <none>
kube-system etcd-rke2-server-1 0/1 Running 4473 (25s ago) 20h 192.168.1.22 rke2-server-1 <none> <none>
kube-system helm-install-rke2-canal-2m2h8 0/1 Completed 0 20d 192.168.1.40 rke2-loadbalancer-1 <none> <none>
kube-system helm-install-rke2-coredns-tlz74 0/1 Completed 0 20d 192.168.1.40 rke2-loadbalancer-1 <none> <none>
kube-system helm-install-rke2-ingress-nginx-nzhjr 0/1 Completed 0 20d <none> rke2-loadbalancer-1 <none> <none>
kube-system helm-install-rke2-metrics-server-gtcnf 0/1 Completed 0 20d <none> rke2-loadbalancer-1 <none> <none>
kube-system helm-install-rke2-snapshot-controller-crd-2ls6g 0/1 Completed 0 20d <none> rke2-loadbalancer-1 <none> <none>
kube-system helm-install-rke2-snapshot-controller-n2fxq 0/1 Completed 1 20d <none> rke2-loadbalancer-1 <none> <none>
kube-system helm-install-rke2-snapshot-validation-webhook-ztz27 0/1 Completed 0 20d <none> rke2-loadbalancer-1 <none> <none>
kube-system kube-apiserver-rke2-loadbalancer-1 1/1 Running 9 20d 192.168.1.40 rke2-loadbalancer-1 <none> <none>
kube-system kube-controller-manager-rke2-loadbalancer-1 1/1 Running 4448 (2m59s ago) 20d 192.168.1.40 rke2-loadbalancer-1 <none> <none>
kube-system kube-proxy-rke2-agent-1 1/1 Running 2 (20h ago) 4d8h 192.168.1.91 rke2-agent-1 <none> <none>
kube-system kube-proxy-rke2-loadbalancer-1 1/1 Running 8 (20h ago) 20d 192.168.1.40 rke2-loadbalancer-1 <none> <none>
kube-system kube-proxy-rke2-server-1 0/1 CrashLoopBackOff 4483 (44s ago) 20d 192.168.1.22 rke2-server-1 <none> <none>
kube-system kube-scheduler-rke2-loadbalancer-1 1/1 Running 2190 (9m19s ago) 20d 192.168.1.40 rke2-loadbalancer-1 <none> <none>
kube-system rke2-canal-gwjpv 2/2 Running 16 (20h ago) 20d 192.168.1.40 rke2-loadbalancer-1 <none> <none>
kube-system rke2-canal-pmfrj 2/2 Running 17 (20h ago) 20d 192.168.1.91 rke2-agent-1 <none> <none>
kube-system rke2-canal-rgpkf 2/2 Running 19 (161m ago) 20d 192.168.1.22 rke2-server-1 <none> <none>
kube-system rke2-coredns-rke2-coredns-6bb85f9dd8-v8xqs 1/1 Running 9 (20h ago) 20d 10.42.2.36 rke2-agent-1 <none> <none>
kube-system rke2-coredns-rke2-coredns-6bb85f9dd8-xff2b 1/1 Running 8 (20h ago) 20d 10.42.0.60 rke2-loadbalancer-1 <none> <none>
kube-system rke2-coredns-rke2-coredns-autoscaler-7b9c797d64-wlrl5 1/1 Running 8 (20h ago) 20d 10.42.0.59 rke2-loadbalancer-1 <none> <none>
kube-system rke2-ingress-nginx-controller-6jftf 1/1 Running 8 (20h ago) 20d 10.42.0.62 rke2-loadbalancer-1 <none> <none>
kube-system rke2-ingress-nginx-controller-wtghh 1/1 Running 8 (20h ago) 20d 10.42.2.37 rke2-agent-1 <none> <none>
kube-system rke2-ingress-nginx-controller-wxmfj 1/1 Running 9 (161m ago) 20d 10.42.1.23 rke2-server-1 <none> <none>
kube-system rke2-metrics-server-868fc8795f-bbb72 1/1 Running 9 (20h ago) 20d 10.42.0.58 rke2-loadbalancer-1 <none> <none>
kube-system rke2-snapshot-controller-7dcf5d5b46-dzgs9 1/1 Running 2214 (9m23s ago) 20d 10.42.0.63 rke2-loadbalancer-1 <none> <none>
kube-system rke2-snapshot-validation-webhook-bf7bbd6fc-7ldq6 1/1 Running 8 (20h ago) 20d 10.42.0.61 rke2-loadbalancer-1 <none> <none>
metallb-system controller-6dd967fdc7-fzgx6 1/1 Running 11 (161m ago) 14d 10.42.1.24 rke2-server-1 <none> <none>
metallb-system speaker-5j7ll 1/1 Running 7 (20h ago) 14d 192.168.1.91 rke2-agent-1 <none> <none>
metallb-system speaker-7npjh 1/1 Running 10 (20h ago) 14d 192.168.1.40 rke2-loadbalancer-1 <none> <none>
metallb-system speaker-h7kgk 1/1 Running 8 (161m ago) 14d 192.168.1.22 rke2-server-1 <none> <none> |
Beta Was this translation helpful? Give feedback.
-
|
Check your hosts file - make sure localhost points at the ipv4 loopback address. |
Beta Was this translation helpful? Give feedback.
-
|
Nevermind... The problem was very simple : ❯ cat /etc/hosts
*127.0.0.1 localhostWhen I edit my hosts file some times ago, I accidentally added a star to the locahost. Thank you for your time! |
Beta Was this translation helpful? Give feedback.
Answer selected by
MelvinCou
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Nevermind... The problem was very simple :
❯ cat /etc/hosts *127.0.0.1 localhostWhen I edit my hosts file some times ago, I accidentally added a star to the locahost.
Thank you for your time!