When using kubectl logs to view the kube-proxy pod logs on the agent node received a "400 Bad Request" error

[warnerchen]

Environmental Info:
RKE2 Version: v1.26.15+rke2r1
Cluster Configuration: 3 servers + n agents

Describe the bug:
In my Rancher, I found that I could not view the kube-proxy pod logs of the agent nodes through the "View Logs" feature (it shows disconnected), but I can view the kube-proxy pod logs of the server node normally.

After that, I tried to view the kube-proxy pod logs on the agent nodes using kubectl -n kube-system logs kube-proxy-xxx from the server node, but it didn't work either.

yyy:/root # kubectl -n kube-system logs kube-proxy-xxx -v 7
I0918 16:48:31.348464    5761 loader.go:373] Config loaded from file:  /etc/rancher/rke2/rke2.yaml
I0918 16:48:31.349079    5761 round_trippers.go:463] GET https://127.0.0.1:6443/api?timeout=32s
I0918 16:48:31.349093    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.349100    5761 round_trippers.go:473]     Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json
I0918 16:48:31.349107    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.357916    5761 round_trippers.go:574] Response Status: 200 OK in 8 milliseconds
I0918 16:48:31.384732    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis?timeout=32s
I0918 16:48:31.384766    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.384790    5761 round_trippers.go:473]     Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json
I0918 16:48:31.384820    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.391537    5761 round_trippers.go:574] Response Status: 200 OK in 6 milliseconds
I0918 16:48:31.393493    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/authentication.k8s.io/v1?timeout=32s
I0918 16:48:31.393516    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393532    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393541    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/apiregistration.k8s.io/v1?timeout=32s
I0918 16:48:31.393562    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/metrics.k8s.io/v1beta1?timeout=32s
I0918 16:48:31.393567    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393591    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393607    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393614    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393626    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393634    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393664    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/apps/v1?timeout=32s
I0918 16:48:31.393674    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393685    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393696    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393705    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/acme.cert-manager.io/v1?timeout=32s
I0918 16:48:31.393734    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393759    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393779    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393797    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/events.k8s.io/v1?timeout=32s
I0918 16:48:31.393805    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393816    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393827    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393837    5761 round_trippers.go:463] GET https://127.0.0.1:6443/api/v1?timeout=32s
I0918 16:48:31.393859    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393875    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393891    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393925    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/storage.k8s.io/v1beta1?timeout=32s
I0918 16:48:31.393933    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.393944    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.393955    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394075    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/authorization.k8s.io/v1?timeout=32s
I0918 16:48:31.394085    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394096    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394107    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394183    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/autoscaling/v2?timeout=32s
I0918 16:48:31.394191    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394201    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394212    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394318    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/batch/v1?timeout=32s
I0918 16:48:31.394325    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394335    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394346    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394387    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/admissionregistration.k8s.io/v1?timeout=32s
I0918 16:48:31.394414    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394445    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394479    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394520    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/certificates.k8s.io/v1?timeout=32s
I0918 16:48:31.394531    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394540    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394549    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394644    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/cert-manager.io/v1?timeout=32s
I0918 16:48:31.394680    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394695    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394710    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394519    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/catalog.cattle.io/v1?timeout=32s
I0918 16:48:31.394779    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394789    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394800    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394799    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/apiextensions.k8s.io/v1?timeout=32s
I0918 16:48:31.394820    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394839    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394859    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394878    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/compute.functionmesh.io/v1alpha1?timeout=32s
I0918 16:48:31.394886    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394895    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394903    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394976    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/helm.cattle.io/v1?timeout=32s
I0918 16:48:31.394982    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.394991    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.394999    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395030    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/scheduling.k8s.io/v1?timeout=32s
I0918 16:48:31.395045    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395059    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/k3s.cattle.io/v1?timeout=32s
I0918 16:48:31.395063    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395084    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395223    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/snapshot.storage.k8s.io/v1beta1?timeout=32s
I0918 16:48:31.395230    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395238    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395247    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395311    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/ui.cattle.io/v1?timeout=32s
I0918 16:48:31.395317    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395326    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395334    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395344    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/coordination.k8s.io/v1?timeout=32s
I0918 16:48:31.395352    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395364    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395373    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395391    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/upgrade.cattle.io/v1?timeout=32s
I0918 16:48:31.395398    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395406    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395415    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395442    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/node.k8s.io/v1?timeout=32s
I0918 16:48:31.395449    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395458    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395467    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395476    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/plantcon.bmwgroup.io/v1alpha?timeout=32s
I0918 16:48:31.395483    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395491    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395500    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395531    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/discovery.k8s.io/v1?timeout=32s
I0918 16:48:31.395538    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395547    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395557    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395563    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/admin.pulsar.io/v1alpha1?timeout=32s
I0918 16:48:31.395571    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395579    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395591    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395637    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/policy/v1?timeout=32s
I0918 16:48:31.395652    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395662    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395672    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395689    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/flowcontrol.apiserver.k8s.io/v1beta3?timeout=32s
I0918 16:48:31.395702    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395715    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395730    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395740    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/management.cattle.io/v3?timeout=32s
I0918 16:48:31.395748    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.395756    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.395765    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.396193    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/flowcontrol.apiserver.k8s.io/v1beta2?timeout=32s
I0918 16:48:31.396218    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.396229    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.396274    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.396395    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/autoscaling/v1?timeout=32s
I0918 16:48:31.396411    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.396440    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.396486    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.396607    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/networking.k8s.io/v1?timeout=32s
I0918 16:48:31.396623    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.396638    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.396653    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.396746    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/rbac.authorization.k8s.io/v1?timeout=32s
I0918 16:48:31.396756    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.396770    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.396784    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395066    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.396847    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.396869    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395106    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/monitoring.coreos.com/v1?timeout=32s
I0918 16:48:31.396947    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.396964    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.396985    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395139    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/monitoring.coreos.com/v1alpha1?timeout=32s
I0918 16:48:31.397065    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.397087    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.397105    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.395183    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/snapshot.storage.k8s.io/v1?timeout=32s
I0918 16:48:31.397173    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.397191    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.397206    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.397367    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/cilium.io/v2alpha1?timeout=32s
I0918 16:48:31.397391    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.397421    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.397441    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.397558    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/cilium.io/v2?timeout=32s
I0918 16:48:31.397577    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.397596    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.397615    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.397762    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/cluster.cattle.io/v3?timeout=32s
I0918 16:48:31.397782    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.397805    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.397830    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.393545    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.394762    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/storage.k8s.io/v1?timeout=32s
I0918 16:48:31.397950    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.397969    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.397993    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.398241    5761 round_trippers.go:463] GET https://127.0.0.1:6443/apis/controller-manager.dvpe-cloud.github.io/v1alpha1?timeout=32s
I0918 16:48:31.398261    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.398282    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.398303    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.398471    5761 round_trippers.go:574] Response Status: 200 OK in 4 milliseconds
I0918 16:48:31.399630    5761 round_trippers.go:574] Response Status: 200 OK in 5 milliseconds
I0918 16:48:31.399677    5761 round_trippers.go:574] Response Status: 200 OK in 5 milliseconds
I0918 16:48:31.400247    5761 round_trippers.go:574] Response Status: 200 OK in 5 milliseconds
I0918 16:48:31.400789    5761 round_trippers.go:574] Response Status: 200 OK in 7 milliseconds
I0918 16:48:31.400881    5761 round_trippers.go:574] Response Status: 200 OK in 7 milliseconds
I0918 16:48:31.401091    5761 round_trippers.go:574] Response Status: 200 OK in 6 milliseconds
I0918 16:48:31.402134    5761 round_trippers.go:574] Response Status: 200 OK in 7 milliseconds
I0918 16:48:31.402168    5761 round_trippers.go:574] Response Status: 200 OK in 6 milliseconds
I0918 16:48:31.402704    5761 round_trippers.go:574] Response Status: 200 OK in 7 milliseconds
I0918 16:48:31.402772    5761 round_trippers.go:574] Response Status: 200 OK in 8 milliseconds
I0918 16:48:31.402802    5761 round_trippers.go:574] Response Status: 200 OK in 9 milliseconds
I0918 16:48:31.402885    5761 round_trippers.go:574] Response Status: 200 OK in 7 milliseconds
I0918 16:48:31.402978    5761 round_trippers.go:574] Response Status: 200 OK in 8 milliseconds
I0918 16:48:31.403098    5761 round_trippers.go:574] Response Status: 200 OK in 8 milliseconds
I0918 16:48:31.403536    5761 round_trippers.go:574] Response Status: 200 OK in 9 milliseconds
I0918 16:48:31.405704    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.405732    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.405756    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.407919    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.407965    5761 round_trippers.go:574] Response Status: 200 OK in 12 milliseconds
I0918 16:48:31.407984    5761 round_trippers.go:574] Response Status: 200 OK in 12 milliseconds
I0918 16:48:31.407987    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408046    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408077    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.408077    5761 round_trippers.go:574] Response Status: 200 OK in 12 milliseconds
I0918 16:48:31.408088    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.408118    5761 round_trippers.go:574] Response Status: 200 OK in 14 milliseconds
I0918 16:48:31.408119    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.408201    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408219    5761 round_trippers.go:574] Response Status: 200 OK in 12 milliseconds
I0918 16:48:31.408233    5761 round_trippers.go:574] Response Status: 200 OK in 12 milliseconds
I0918 16:48:31.408244    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408353    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408396    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408412    5761 round_trippers.go:574] Response Status: 200 OK in 11 milliseconds
I0918 16:48:31.408485    5761 round_trippers.go:574] Response Status: 200 OK in 14 milliseconds
I0918 16:48:31.408525    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.408560    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.408565    5761 round_trippers.go:574] Response Status: 200 OK in 13 milliseconds
I0918 16:48:31.408576    5761 round_trippers.go:574] Response Status: 200 OK in 13 milliseconds
I0918 16:48:31.408781    5761 round_trippers.go:574] Response Status: 200 OK in 10 milliseconds
I0918 16:48:31.409022    5761 round_trippers.go:574] Response Status: 200 OK in 15 milliseconds
I0918 16:48:31.562466    5761 round_trippers.go:463] GET https://127.0.0.1:6443/api/v1/namespaces/kube-system/pods/kube-proxy-xxx
I0918 16:48:31.562508    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.562537    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.562560    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.565911    5761 round_trippers.go:574] Response Status: 200 OK in 3 milliseconds
I0918 16:48:31.571760    5761 round_trippers.go:463] GET https://127.0.0.1:6443/api/v1/namespaces/kube-system/pods/kube-proxy-xxx/log?container=kube-proxy
I0918 16:48:31.571787    5761 round_trippers.go:469] Request Headers:
I0918 16:48:31.571798    5761 round_trippers.go:473]     Accept: application/json, */*
I0918 16:48:31.571815    5761 round_trippers.go:473]     User-Agent: kubectl/v1.26.15+rke2r1 (linux/amd64) kubernetes/1649f59
I0918 16:48:31.641024    5761 round_trippers.go:574] Response Status: 400 Bad Request in 69 milliseconds
I0918 16:48:31.645923    5761 helpers.go:246] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "container \"kube-proxy\" in pod \"kube-proxy-xxx\" is not available",
  "reason": "BadRequest",
  "code": 400
}]
Error from server (BadRequest): container "kube-proxy" in pod "kube-proxy-xxx" is not available

However, when I logged into the agent node and used crictl log, I could view the logs normally, so the issue seems to be on the Kubernetes level.

xxx:/root # crictl ps -a | grep kube-proxy
d7b795fbb538b       4a2dd7d23859b       6 weeks ago         Running             kube-proxy                      0                   7f83862ecf3a1       kube-proxy-xxx

xxx:/root # ps aux | grep kube-proxy 
root     16955  0.0  0.1 1280240 55640 ?       Ssl  Aug05  30:34 kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=xxx --kubeconfig=/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig --proxy-mode=iptables
root     29007  0.0  0.0   5644   816 pts/0    S+   15:37   0:00 grep kube-proxy

xxx:/root # crictl logs $kube_proxy_container_id
I0805 05:38:27.037153       1 server.go:224] "Warning, all flags other than --config, --write-config-to, and --cleanup are deprecated, please begin using a config file ASAP"
I0805 05:38:27.054925       1 node.go:163] Successfully retrieved node IP: 10.117.158.31
I0805 05:38:27.054998       1 server_others.go:109] "Detected node IP" address="10.117.158.31"
I0805 05:38:27.105372       1 server_others.go:178] "Using iptables Proxier"
I0805 05:38:27.105491       1 server_others.go:185] "kube-proxy running in dual-stack mode" ipFamily=IPv4
I0805 05:38:27.105516       1 server_others.go:186] "Creating dualStackProxier for iptables"
I0805 05:38:27.105552       1 server_others.go:468] "Detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6"
I0805 05:38:27.105612       1 proxier.go:242] "Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.localhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses"
I0805 05:38:27.106246       1 server.go:656] "Version info" version="v1.26.15+rke2r1"
I0805 05:38:27.106280       1 server.go:658] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0805 05:38:27.107138       1 config.go:226] "Starting endpoint slice config controller"
I0805 05:38:27.107220       1 shared_informer.go:270] Waiting for caches to sync for endpoint slice config
I0805 05:38:27.107223       1 config.go:317] "Starting service config controller"
I0805 05:38:27.107256       1 shared_informer.go:270] Waiting for caches to sync for service config
I0805 05:38:27.107223       1 config.go:444] "Starting node config controller"
I0805 05:38:27.107319       1 shared_informer.go:270] Waiting for caches to sync for node config
I0805 05:38:27.207742       1 shared_informer.go:277] Caches are synced for endpoint slice config
I0805 05:38:27.207902       1 shared_informer.go:277] Caches are synced for node config
I0805 05:38:27.208044       1 shared_informer.go:277] Caches are synced for service config

xxx:/root # cat /var/log/pods/kube-system_kube-proxy-xxx_c16282c8793e789dcf001594d2941864/kube-proxy/0.log 
2024-08-05T13:38:27.037609427+08:00 stderr F I0805 05:38:27.037153       1 server.go:224] "Warning, all flags other than --config, --write-config-to, and --cleanup are deprecated, please begin using a config file ASAP"
2024-08-05T13:38:27.055400955+08:00 stderr F I0805 05:38:27.054925       1 node.go:163] Successfully retrieved node IP: 10.117.158.31
2024-08-05T13:38:27.055451729+08:00 stderr F I0805 05:38:27.054998       1 server_others.go:109] "Detected node IP" address="10.117.158.31"
2024-08-05T13:38:27.105839301+08:00 stderr F I0805 05:38:27.105372       1 server_others.go:178] "Using iptables Proxier"
2024-08-05T13:38:27.105877094+08:00 stderr F I0805 05:38:27.105491       1 server_others.go:185] "kube-proxy running in dual-stack mode" ipFamily=IPv4
2024-08-05T13:38:27.105882616+08:00 stderr F I0805 05:38:27.105516       1 server_others.go:186] "Creating dualStackProxier for iptables"
2024-08-05T13:38:27.105896168+08:00 stderr F I0805 05:38:27.105552       1 server_others.go:468] "Detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, , defaulting to no-op detect-local for IPv6"
2024-08-05T13:38:27.105902017+08:00 stderr F I0805 05:38:27.105612       1 proxier.go:242] "Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.localhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses"
2024-08-05T13:38:27.106399454+08:00 stderr F I0805 05:38:27.106246       1 server.go:656] "Version info" version="v1.26.15+rke2r1"
2024-08-05T13:38:27.106422753+08:00 stderr F I0805 05:38:27.106280       1 server.go:658] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
2024-08-05T13:38:27.107445169+08:00 stderr F I0805 05:38:27.107138       1 config.go:226] "Starting endpoint slice config controller"
2024-08-05T13:38:27.107453733+08:00 stderr F I0805 05:38:27.107220       1 shared_informer.go:270] Waiting for caches to sync for endpoint slice config
2024-08-05T13:38:27.107458336+08:00 stderr F I0805 05:38:27.107223       1 config.go:317] "Starting service config controller"
2024-08-05T13:38:27.107462883+08:00 stderr F I0805 05:38:27.107256       1 shared_informer.go:270] Waiting for caches to sync for service config
2024-08-05T13:38:27.107468516+08:00 stderr F I0805 05:38:27.107223       1 config.go:444] "Starting node config controller"
2024-08-05T13:38:27.107473363+08:00 stderr F I0805 05:38:27.107319       1 shared_informer.go:270] Waiting for caches to sync for node config
2024-08-05T13:38:27.208175431+08:00 stderr F I0805 05:38:27.207742       1 shared_informer.go:277] Caches are synced for endpoint slice config
2024-08-05T13:38:27.208214556+08:00 stderr F I0805 05:38:27.207902       1 shared_informer.go:277] Caches are synced for node config
2024-08-05T13:38:27.208231535+08:00 stderr F I0805 05:38:27.208044       1 shared_informer.go:277] Caches are synced for service config

xxx:/root # curl 127.0.0.1:10256/healthz 
{"lastUpdated": "2024-09-19 03:00:31.379390098 +0000 UTC m=+3878524.370184849","currentTime": "2024-09-19 03:00:31.379390098 +0000 UTC m=+3878524.370184849"}

After I restarted the rke2-agent and recreated the kube-proxy pod, everything returned to normal.

[brandond]

If you'd pulled the kubelet and containerd logs before restarting things, we might have been be able to figure out why it was unable to read logs from that container. Unfortunately there's not much to be done now.

[warnerchen]

I only restarted one agent node's rke2-agent in the cluster, but the issue persists on the other nodes. I tried using kubectl logs to retrieve the kube-proxy logs on one of the nodes, and it failed. Below are today's containerd and kubelet logs.

I have filtered out the information that seems unimportant.

containerd：

time="2024-09-20T10:33:36.328368393+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:dashboard-shell-gs6wf,Uid:4a4af124-8e4e-41f8-99dc-2be0d61b4703,Namespace:cattle-system,Attempt:0,}"
time="2024-09-20T10:33:36.771916600+08:00" level=info msg="loading plugin \"io.containerd.internal.v1.shutdown\"..." runtime=io.containerd.runc.v2 type=io.containerd.internal.v1
time="2024-09-20T10:33:36.772086639+08:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.pause\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-09-20T10:33:36.772129933+08:00" level=info msg="loading plugin \"io.containerd.event.v1.publisher\"..." runtime=io.containerd.runc.v2 type=io.containerd.event.v1
time="2024-09-20T10:33:36.772169050+08:00" level=info msg="loading plugin \"io.containerd.ttrpc.v1.task\"..." runtime=io.containerd.runc.v2 type=io.containerd.ttrpc.v1
time="2024-09-20T10:33:36.900441949+08:00" level=info msg="RunPodSandbox for &PodSandboxMetadata{Name:dashboard-shell-gs6wf,Uid:4a4af124-8e4e-41f8-99dc-2be0d61b4703,Namespace:cattle-system,Attempt:0,} returns sandbox id \"6a7928a27da8f12b803ce601af7cbc0c7d8bf1e7683a101ab38f1b11534e53c0\""
time="2024-09-20T10:33:36.910791743+08:00" level=info msg="CreateContainer within sandbox \"6a7928a27da8f12b803ce601af7cbc0c7d8bf1e7683a101ab38f1b11534e53c0\" for container &ContainerMetadata{Name:shell,Attempt:0,}"
time="2024-09-20T10:33:36.951330778+08:00" level=info msg="CreateContainer within sandbox \"6a7928a27da8f12b803ce601af7cbc0c7d8bf1e7683a101ab38f1b11534e53c0\" for &ContainerMetadata{Name:shell,Attempt:0,} returns container id \"8c876b3184a6fd9d38edbb9b5ceaf001ac4146907d49af4902866a79663cc98d\""
time="2024-09-20T10:33:36.952275750+08:00" level=info msg="StartContainer for \"8c876b3184a6fd9d38edbb9b5ceaf001ac4146907d49af4902866a79663cc98d\""
time="2024-09-20T10:33:37.104748716+08:00" level=info msg="StartContainer for \"8c876b3184a6fd9d38edbb9b5ceaf001ac4146907d49af4902866a79663cc98d\" returns successfully"
time="2024-09-20T10:33:37.118399610+08:00" level=info msg="CreateContainer within sandbox \"6a7928a27da8f12b803ce601af7cbc0c7d8bf1e7683a101ab38f1b11534e53c0\" for container &ContainerMetadata{Name:proxy,Attempt:0,}"
time="2024-09-20T10:33:37.152492428+08:00" level=info msg="CreateContainer within sandbox \"6a7928a27da8f12b803ce601af7cbc0c7d8bf1e7683a101ab38f1b11534e53c0\" for &ContainerMetadata{Name:proxy,Attempt:0,} returns container id \"6a3a340391cff03b852a6575deee53da98eb1f9b9bcd05634302e048e7c9b78d\""
time="2024-09-20T10:33:37.153396153+08:00" level=info msg="StartContainer for \"6a3a340391cff03b852a6575deee53da98eb1f9b9bcd05634302e048e7c9b78d\""
time="2024-09-20T10:33:37.292769256+08:00" level=info msg="StartContainer for \"6a3a340391cff03b852a6575deee53da98eb1f9b9bcd05634302e048e7c9b78d\" returns successfully"

kubelet：

I0920 10:33:36.006654    1089 topology_manager.go:210] "Topology Admit Handler" podUID=4a4af124-8e4e-41f8-99dc-2be0d61b4703 podNamespace="cattle-system" podName="dashboard-shell-gs6wf"
I0920 10:33:36.006998    1089 memory_manager.go:346] "RemoveStaleState removing state" podUID="49e842e1-4000-480c-8130-454292681246" containerName="shell"
I0920 10:33:36.007064    1089 memory_manager.go:346] "RemoveStaleState removing state" podUID="49e842e1-4000-480c-8130-454292681246" containerName="proxy"
I0920 10:33:36.179171    1089 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"user-kubeconfig\" (UniqueName: \"kubernetes.io/configmap/4a4af124-8e4e-41f8-99dc-2be0d61b4703-user-kubeconfig\") pod \"dashboard-shell-gs6wf\" (UID: \"4a4af124-8e4e-41f8-99dc-2be0d61b4703\") " pod="cattle-system/dashboard-shell-gs6wf"
I0920 10:33:36.179269    1089 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pod-impersonation-shell-ktzl9-token\" (UniqueName: \"kubernetes.io/secret/4a4af124-8e4e-41f8-99dc-2be0d61b4703-pod-impersonation-shell-ktzl9-token\") pod \"dashboard-shell-gs6wf\" (UID: \"4a4af124-8e4e-41f8-99dc-2be0d61b4703\") " pod="cattle-system/dashboard-shell-gs6wf"
I0920 10:33:36.179510    1089 reconciler_common.go:253] "operationExecutor.VerifyControllerAttachedVolume started for volume \"admin-kubeconfig\" (UniqueName: \"kubernetes.io/configmap/4a4af124-8e4e-41f8-99dc-2be0d61b4703-admin-kubeconfig\") pod \"dashboard-shell-gs6wf\" (UID: \"4a4af124-8e4e-41f8-99dc-2be0d61b4703\") " pod="cattle-system/dashboard-shell-gs6wf"
I0920 10:33:37.347854    1089 pod_startup_latency_tracker.go:102] "Observed pod startup duration" pod="cattle-system/dashboard-shell-gs6wf" podStartSLOduration=2.347789981 pod.CreationTimestamp="2024-09-20 10:33:35 +0800 CST" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2024-09-20 10:33:37.347640376 +0800 CST m=+1529048.719916478" watchObservedRunningTime="2024-09-20 10:33:37.347789981 +0800 CST m=+1529048.720066027"
``