RKE2 v.1.30.0, My kube-apiserver cant comminicate between prometheus and kube-api server cant loading v1beta1.custom.metrics.k8s.i

[cengizhanR]

Errors:
error resolving cattle-monitoring-system/rancher-monitoring-prometheus-adapter: no endpoints available for service "rancher-monitoring-prometheus-adapter
error` resolving kube-system/rke2-metrics-server: no endpoints available for service `"rke2-metrics-server"
**loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService**
Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"

Kube-Api server logs,

W0629 13:42:26.637615       1 logging.go:59] [core] [Channel #31 SubChannel #32] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:26.653790155+03:00 W0629 13:42:26.653665       1 logging.go:59] [core] [Channel #371 SubChannel #372] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:26.667768775+03:00 W0629 13:42:26.667655       1 logging.go:59] [core] [Channel #25 SubChannel #26] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:26.692205758+03:00 E0629 13:42:26.692099       1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
2024-06-29T16:42:26.692225959+03:00 E0629 13:42:26.692111       1 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 14.801µs, panicked: false, err: context deadline exceeded, panic-reason: <nil>
E0629 13:42:26.692132       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0629 13:42:26.693205       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
2024-06-29T16:42:26.693321976+03:00 I0629 13:42:26.693288       1 trace.go:236] Trace[861108839]: "Update" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:b29e9870-5ab3-4333-9d03-777b9ac32987,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:rke21,subresource:,namespace:kube-node-lease,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/rke21,user-agent:kubelet/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b,verb:PUT (29-Jun-2024 13:42:16.693) (total time: 10000ms):
2024-06-29T16:42:26.693324876+03:00 Trace[861108839]: ["GuaranteedUpdate etcd3" audit-id:b29e9870-5ab3-4333-9d03-777b9ac32987,key:/leases/kube-node-lease/rke21,type:*coordination.Lease,resource:leases.coordination.k8s.io 10000ms (13:42:16.693)
2024-06-29T16:42:26.693327276+03:00 Trace[861108839]:  ---"Txn call failed" err:context deadline exceeded 9998ms (13:42:26.691)]
Trace[861108839]: [10.000218763s] [10.000218763s] END
2024-06-29T16:42:26.693463478+03:00 E0629 13:42:26.693392       1 timeout.go:142] post-timeout activity - time-elapsed: 1.25902ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/rke21" result: <nil>
W0629 13:42:26.715236       1 logging.go:59] [core] [Channel #100 SubChannel #101] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0629 13:42:26.754085       1 logging.go:59] [core] [Channel #151 SubChannel #152] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:26.761578047+03:00 W0629 13:42:26.761466       1 logging.go:59] [core] [Channel #139 SubChannel #140] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:26.783920698+03:00 W0629 13:42:26.783800       1 logging.go:59] [core] [Channel #305 SubChannel #306] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:26.900014520+03:00 W0629 13:42:26.899877       1 logging.go:59] [core] [Channel #163 SubChannel #164] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:26.936456392+03:00 W0629 13:42:26.936324       1 logging.go:59] [core] [Channel #91 SubChannel #92] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.016429047+03:00 W0629 13:42:27.016319       1 logging.go:59] [core] [Channel #76 SubChannel #77] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: conne
ct: connection refused"
W0629 13:42:27.055269       1 logging.go:59] [core] [Channel #112 SubChannel #113] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.066418332+03:00 W0629 13:42:27.066313       1 logging.go:59] [core] [Channel #34 SubChannel #35] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.211691312+03:00 W0629 13:42:27.211572       1 logging.go:59] [core] [Channel #64 SubChannel #65] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.259436562+03:00 W0629 13:42:27.259302       1 logging.go:59] [core] [Channel #160 SubChannel #161] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.279837782+03:00 W0629 13:42:27.279719       1 logging.go:59] [core] [Channel #61 SubChannel #62] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.314231522+03:00 W0629 13:42:27.314108       1 logging.go:59] [core] [Channel #82 SubChannel #83] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: 
connection refused"
2024-06-29T16:42:27.355594771+03:00 W0629 13:42:27.355456       1 logging.go:59] [core] [Channel #130 SubChannel #131] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0629 13:42:27.406535       1 logging.go:59] [core] [Channel #46 SubChannel #47] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.407873591+03:00 W0629 13:42:27.407796       1 logging.go:59] [core] [Channel #58 SubChannel #59] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.483997786+03:00 W0629 13:42:27.483842       1 logging.go:59] [core] [Channel #79 SubChannel #80] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.523250602+03:00 W0629 13:42:27.523108       1 logging.go:59] [core] [Channel #157 SubChannel #158] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.605726997+03:00 W0629 13:42:27.605601       1 logging.go:59] [core] [Channel #154 SubChannel #155] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.721814619+03:00 W0629 13:42:27.721676       1 logging.go:59] [core] [Channel #118 SubChannel #119] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect
: connection refused"
2024-06-29T16:42:27.724252757+03:00 W0629 13:42:27.724013       1 logging.go:59] [core] [Channel #145 SubChannel #146] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.730822560+03:00 W0629 13:42:27.730743       1 logging.go:59] [core] [Channel #19 SubChannel #20] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0629 13:42:27.779188       1 logging.go:59] [core] [Channel #172 SubChannel #173] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.825278643+03:00 W0629 13:42:27.825142       1 logging.go:59] [core] [Channel #2 SubChannel #4] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.836589921+03:00 W0629 13:42:27.836477       1 logging.go:59] [core] [Channel #136 SubChannel #137] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.894435128+03:00 W0629 13:42:27.894310       1 logging.go:59] [core] [Channel #109 SubChannel #110] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0629 13:42:27.915674       1 logging.go:59] [core] [Channel #28 SubChannel #29] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.943707602+03:00 W0629 13:42:27.943573       1 logging.go:59] [core] [Channel #16 SubChannel #17] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:27.984123536+03:00 W0629 13:42:27.983906       1 logging.go:59] [core] [Channel #67 SubChannel #68] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:28.011876572+03:00 W0629 13:42:28.011760       1 logging.go:59] [core] [Channel #10 SubChannel #11] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:28.011919072+03:00 W0629 13:42:28.011878       1 logging.go:59] [core] [Channel #181 SubChannel #182] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:28.107615675+03:00 W0629 13:42:28.107428       1 logging.go:59] [core] [Channel #175 SubChannel #176] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: 
connection refused"
W0629 13:42:28.202281       1 logging.go:59] [core] [Channel #148 SubChannel #149] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:28.227080050+03:00 W0629 13:42:28.226937       1 logging.go:59] [core] [Channel #94 SubChannel #95] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
W0629 13:42:28.228202       1 logging.go:59] [core] [Channel #133 SubChannel #134] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:28.320617718+03:00 W0629 13:42:28.320456       1 logging.go:59] [core] [Channel #169 SubChannel #170] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:28.823479011+03:00 W0629 13:42:28.823340       1 logging.go:59] [core] [Channel #115 SubChannel #116] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:29.040938824+03:00 E0629 13:42:29.040800       1 status.go:71] apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded
2024-06-29T16:42:29.044312477+03:00 E0629 13:42:29.044203       1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
2024-06-29T16:42:29.044334877+03:00 E0629 13:42:29.044243       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
2024-06-29T16:42:29.045492195+03:00 E0629 13:42:29.045389       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0629 13:42:29.045444       1 trace.go:236] Trace[1704177230]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:1d74d014-001a-46ef-9d1e-5a72ecbc4702,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:rke2-cloud-controller-manager,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager,user-agent:cloud-controller-manager/v1.29.3 (linux/amd64) kubernetes/d9f75b6/leader-election,verb:GET (29-Jun-2024 13:42:24.041) (total time: 5004ms):
2024-06-29T16:42:29.045530496+03:00 Trace[1704177230]: [5.004181045s] [5.004181045s] END
2024-06-29T16:42:29.045650798+03:00 E0629 13:42:29.045590       1 timeout.go:142] post-timeout activity - time-elapsed: 1.388822ms, GET "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager" result: <nil>
2024-06-29T16:42:29.386483947+03:00 W0629 13:42:29.386353       1 logging.go:59] [core] [Channel #371 SubChannel #372] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:32.929556459+03:00 E0629 13:42:32.929409       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}: context canceled
2024-06-29T16:42:32.929573559+03:00 E0629 13:42:32.929497       1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
2024-06-29T16:42:32.931611691+03:00 E0629 13:42:32.931525       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
2024-06-29T16:42:32.931630692+03:00 E0629 13:42:32.931570       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
2024-06-29T16:42:32.932770210+0
3:00 I0629 13:42:32.932680       1 trace.go:236] Trace[439743571]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:4239795d-6c6e-467c-874f-0919368db1c8,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:kube-controller-manager,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:kube-controller-manager/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b/leader-election,verb:GET (29-Jun-2024 13:42:27.929) (total time: 5002ms):
Trace[439743571]: [5.002622421s] [5.002622421s] END
2024-06-29T16:42:32.932867311+03:00 E0629 13:42:32.932796       1 timeout.go:142] post-timeout activity - time-elapsed: 3.311252ms, GET "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager" result: <nil>
W0629 13:42:33.205657       1 logging.go:59] [core] [Channel #371 SubChannel #372] grpc: addrConn.createTransport failed to connect to {Addr: "127.0.0.1:2379", ServerName: "127.0.0.1:2379", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused"
2024-06-29T16:42:33.534312751+03:00 E0629 13:42:33.534230       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}: context canceled
2024-06-29T16:42:33.534360152+03:00 E0629 13:42:33.534321       1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
2024-06-29T16:42:33.535392968+03:00 E0629 13:42:33.535317       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
2024-06-29T16:42:33.535413869+03:00 E0629 13:42:33.535366       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0629 13:42:33.536516       1 trace.go:236] Trace[1954358434]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:825782bf-6ed6-4aa9-ba61-8d5da2892d33,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:kube-scheduler,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler,user-agent:kube-scheduler/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b/leader-election,verb:GET (29-Jun-2024 13:42:28.534) (total time: 5001ms):
2024-06-29T16:42:33.536605687+03:00 Trace[1954358434]: [5.001727206s] [5.001727206s] END
E0629 13:42:33.536631       1 timeout.go:142] post-timeout activity - time-elapsed: 2.317336ms, GET "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result: <nil>
2024-06-29T16:42:36.316766824+03:00 I0629 13:42:36.316648       1 trace.go:236] Trace[1482854175]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:7b173f9c-3c24-40b5-a497-ab036b566e83,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:rke2-cloud-controller-manager,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager,user-agent:cloud-controller-manager/v1.29.3 (linux/amd64) kubernetes/d9f75b6/leader-election,verb:GET (29-Jun-2024 13:42:32.642) (total time: 3674ms):
Trace[1482854175]: ---"About to write a response" 3674ms (13:42:36.316)
2024-06-29T16:42:36.316792025+03:00 Trace[1482854175]: [3.67421767s] [3.67421767s] END
2024-06-29T16:42:36.316951127+03:00 I0629 13:42:36.316886       1 trace.go:236] Trace[1908527761]: "Update" accept:application/vnd.kubernetes.protobuf, */*,audit-id:5b879c83-e2ae-4ed5-b811-b3f00f830358,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:apiserver-n2csatvhyzsprryaw5h6zfmm3m,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/apiserver-n2csatvhyzsprryaw5h6zfmm3m,user-agent:kube-apiserver/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b,verb:PUT (29-Jun-2024 13:42:16.748) (total time: 19568ms):
2024-06-29T16:42:36.316957027+03:00 Trace[1908527761]: ["GuaranteedUpdate etcd3" audit-id:5b879c83-e2ae-4ed5-b811-b3f00f830358,key:/leases/kube-system/apiserver-n2csatvhyzsprryaw5h6zfmm3m,type:*coordination.Lease,resource:leases.coordination.k8s.io 19568ms (13:42:16.748)
2024-06-29T16:42:36.316960227+03:00 Trace[1908527761]:  ---"Txn call completed" 19567ms (13:42:36.316)]
2024-06-29T16:42:36.316962727+03:00 Trace[1908527761]: [19.568305642s] [19.568305642s] END
I0629 13:42:36.316908       1 trace.go:236] Trace[1525458874]: "Update" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:da764c10-595e-4d07-8a71-6fc67bd4c31a,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:rke21,subresource:,namespace:kube-node-lease,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/rke21,user-agent:kubelet/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b,verb:PUT (29-Jun-2024 13:42:26.693) (total time: 9623ms):
Trace[1525458874]: ["GuaranteedUpdate etcd3" audit-id:da764c10-595e-4d07-8a71-6fc67bd4c31a,key:/leases/kube-node-lease/rke21,type:*coordination.Lease,resource:leases.coordination.k8s.io 9623ms (13:42:26.693)
2024-06-29T16:42:36.316985028+03:00 Trace[1525458874]:  ---"Txn call completed" 9622ms (13:42:36.316)]
2024-06-29T16:42:36.316987528+03:00 Trace[1525458874]: [9.623756953s] [9.623756953s] END
I0629 13:42:36.536297       1 trace.go:236] Trace[370137025]: "Get" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:7d2f05d3-4876-4daf-872f-46fcc2c7df30,client:127.0.0.1,api-group:storage.k8s.io,api-version:v1,name:csi-b63266bbe2731e9e0038c46fa542cc69497341c01dea9a2288ad3ba528d073ab,subresource:,namespace:,protocol:HTTP/2.0,resource:volumeattachments,scope:resource,url:/apis/storage.k8s.io/v1/volumeattachments/csi-b63266bbe2731e9e0038c46fa542cc69497341c01dea9a2288ad3ba528d073ab,user-agent:kubelet/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b,verb:GET (29-Jun-2024 13:42:16.102) (total time: 20433ms):
Trace[370137025]: ---"About to write a response" 20433ms (13:42:36.536)
2024-06-29T16:42:36.536444972+03:00 Trace[370137025]: [20.433967229s] [20.433967229s] END
2024-06-29T16:42:37.170549925+03:00 I0629 13:42:37.170406       1 trace.go:236] Trace[490812490]: "Update" accept:application/vnd.kubernetes.protobuf, */*,audit-id:cd2a425c-d965-4848-8c5c-65d1b478ebc2,client:127.0.0.1,api-group:apiregistration.k8s.io,api-version:v1,name:v1beta1.custom.metrics.k8s.io,subresource:status,namespace:,protocol:HTTP/2.0,resource:apiservices,scope:resource,url:/apis/apiregistration.k8s.io/v1/apiservices/v1beta1.custom.metrics.k8s.io/status,user-agent:kube-apiserver/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b,verb:PUT (29-Jun-2024 13:42:15.848) (total time: 21321ms):
2024-06-29T16:42:37.170580226+03:00 Trace[490812490]: ["GuaranteedUpdate etcd3" audit-id:cd2a425c-d965-4848-8c5c-65d1b478ebc2,key:/apiregistration.k8s.io/apiservices/v1beta1.custom.metrics.k8s.io,type:*apiregistration.APIService,resource:apiservices.apiregistration.k8s.io 21321ms (13:42:15.848)
2024-06-29T16:42:37.170585926+03:00 Trace[490812490]:  ---"Txn call completed" 21318ms (13:42:37.169)]
Trace[490812490]: [21.321643962s] [21.321643962s] END
2024-06-29T16:42:37.170954032+03:00 E0629 13:42:37.170873       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
2024-06-29T16:42:37.175620105+03:00 W0629 13:42:37.174266       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:42:37.175643005+03:00 E0629 13:42:37.174327       1 controller.go:146] Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-06-29T16:42:37.175648805+03:00 W0629 13:42:37.174377       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:42:37.175650905+03:00 E0629 13:42:37.174396       1 controller.go:146] Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
2024-06-29T16:42:37.175653105+03:00 , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0629 13:42:37.174431       1 handler_proxy.go:137] error resolving kube-system/rke2-metrics-server: no endpoints available for service "rke2-metrics-server"
2024-06-29T16:42:37.175665205+03:00 W0629 13:42:37.175473       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:42:37.175667606+03:00 E0629 13:42:37.175519       1 controller.go:146] Error updating APIService "v1beta1.custom.metrics.k8s.io" with err: failed to download v1beta1.custom.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
2024-06-29T16:42:37.175669606+03:00 , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-06-29T16:42:37.175671606+03:00 E0629 13:42:37.175542       1 handler_proxy.go:137] error resolving kube-system/rke2-metrics-server: no endpoints available for service "rke2-metrics-server"
2024-06-29T16:42:37.175674206+03:00 W0629 13:42:37.175565       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:42:37.175681606+03:00 E0629 13:42:37.175594       1 controller.go:146] Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
2024-06-29T16:42:37.175684006+03:00 , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
E0629 13:42:37.175605       1 handler_proxy.go:137] error resolving cattle-monitoring-system/rancher-monitoring-prometheus-adapter: no endpoints available for service "rancher-monitoring-prometheus-adapter"
E0629 13:42:37.176610       1 handler_proxy.go:137] error resolving kube-system/rke2-metrics-server: no endpoints available for service "rke2-metrics-server"
2024-06-29T16:42:37.179433765+03:00 E0629 13:42:37.179355       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
2024-06-29T16:42:37.613629180+03:00 I0629 13:42:37.613421       1 trace.go:236] Trace[391020489]: "Get" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:5cf34353-4593-4749-904a-bd391666d3ba,client:10.0.0.4,api-group:crd.projectcalico.org,api-version:v1,name:default,subresource:,namespace:,protocol:HTTP/2.0,resource:clusterinformations,scope:resource,url:/apis/crd.projectcalico.org/v1/clusterinformations/default,user-agent:Go-http-client/2.0,verb:GET (29-Jun-2024 13:42:14.154) (total time: 23458ms):
2024-06-29T16:42:37.613656880+03:00 Trace[391020489]: ---"About to write a response" 23458ms (13:42:37.613)
Trace[391020489]: [23.458695805s] [23.458695805s] END
2024-06-29T16:42:37.614428392+03:00 I0629 13:42:37.614184       1 trace.go:236] Trace[1543684117]: "Get" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:530dc1fe-b540-4c77-b841-8ba49d61c2bb,client:10.0.0.3,api-group:crd.projectcalico.org,api-version:v1,name:default,subresource:,namespace:,protocol:HTTP/2.0,resource:clusterinformations,scope:resource,url:/apis/crd.projectcalico.org/v1/clusterinformations/default,user-agent:Go-http-client/2.0,verb:GET (29-Jun-2024 13:42:12.621) (total time: 24992ms):
Trace[1543684117]: ---"About to write a response" 24992ms (13:42:37.613)
2024-06-29T16:42:37.614447793+03:00 Trace[1543684117]: [24.992394677s] [24.992394677s] END
I0629 13:42:37.707956       1 trace.go:236] Trace[732975119]: "Get" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:0dd13b2d-5736-481b-bdcc-257dfbf14cdc,client:127.0.0.1,api-group:,api-version:v1,name:etcd-rke21,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/etcd-rke21,user-agent:kubelet/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b,verb:GET (29-Jun-2024 13:42:11.917) (total time: 25790ms):
2024-06-29T16:42:37.709106478+03:00 Trace[732975119]: ---"About to write a response" 25789ms (13:42:37.707)
2024-06-29T16:42:37.709109478+03:00 Trace[732975119]: [25.7901581s] [25.7901581s] END
2024-06-29T16:42:37.967605336+03:00 I0629 13:42:37.967478       1 trace.go:236] Trace[245088945]: "GuaranteedUpdate etcd3" audit-id:,key:/masterleases/10.0.0.2,type:*v1.Endpoints,resource:apiServerIPInfo (29-Jun-2024 13:42:20.773) (total time: 17193ms):
Trace[245088945]: ---"initial value restored" 17186ms (13:42:37.960)
2024-06-29T16:42:37.967633036+03:00 Trace[245088945]: [17.193534868s] [17.193534868s] END
2024-06-29T16:42:37.971534297+03:00 W0629 13:42:37.971426       1 lease.go:265] Resetting endpoints for master service "kubernetes" to [10.0.0.2 10.0.0.3 10.0.0.4]
2024-06-29T16:42:38.174821788+03:00 W0629 13:42:38.174678       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:42:38.174851489+03:00 W0629 13:42:38.174691       1 handler_proxy.go:93] no RequestInfo found in the context
E0629 13:42:38.174733       1 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
I0629 13:42:38.174771       1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
2024-06-29T16:42:38.174866189+03:00 E0629 13:42:38.174783       1 controller.go:102] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
2024-06-29T16:42:38.174867689+03:00 , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-06-29T16:42:38.175881705+03:00 I0629 13:42:38.175786       1 controller.go:109] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
2024-06-29T16:42:38.175896105+03:00 W0629 13:42:38.175814       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:42:38.175898005+03:00 W0629 13:42:38.175829       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:42:38.175899905+03:00 E0629 13:42:38.175836       1 controller.go:113] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
2024-06-29T16:42:38.175905705+03:00 E0629 13:42:38.175858       1 controller.go:102] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: failed to download v1beta1.custom.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-06-29T16:42:38.175911005+03:00 I0629 13:42:38.175873       1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
I0629 13:42:38.177870       1 controller.go:109] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
2024-06-29T16:42:39.200360585+03:00 I0629 13:42:39.200047       1 trace.go:236] Trace[1907283326]: "Patch" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:d23571f0-7062-401e-81fc-2906e46e03a2,client:127.0.0.1,api-group:,api-version:v1,name:csi-provisioner-6cfd577457-9whsg.17dd7d5cb35b1e68,subresource:,namespace:longhorn-system,protocol:HTTP/2.0,resource:events,scope:resource,url:/api/v1/namespaces/longhorn-system/events/csi-provisioner-6cfd577457-9whsg.17dd7d5cb35b1e68,user-agent:kubelet/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b,verb:PATCH (29-Jun-2024 13:42:14.310) (total time: 24889ms):
2024-06-29T16:42:39.200395785+03:00 Trace[1907283326]: ["GuaranteedUpdate etcd3" audit-id:d23571f0-7062-401e-81fc-2906e46e03a2,key:/events/longhorn-system/csi-provisioner-6cfd577457-9whsg.17dd7d5cb35b1e68,type:*core.Event,resource:events 24889ms (13:42:14.310)
Trace[1907283326]:  ---"initial value restored" 24883ms (13:42:39.193)]
2024-06-29T16:42:39.200400185+03:00 Trace[1907283326]: [24.88997287s] [24.88997287s] END
2024-06-29T16:42:39.714623657+03:00 E0629 13:42:39.714499       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.42.3.25:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.42.3.25:10250/apis/metrics.k8s.io/v1beta1": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-06-29T16:42:41.811925576+03:00 I0629 13:42:41.811757       1 trace.go:236] Trace[408382000]: "Proxy via http_connect protocol over tcp" address:10.42.3.25:10250 (29-Jun-2024 13:42:34.713) (total time: 7098ms):
Trace[408382000]: [7.098015609s] [7.098015609s] END
I0629 13:42:41.811793       1 trace.go:236] Trace[1616832970]: "Proxy via http_connect protocol over tcp" address:10.42.3.25:10250 (29-Jun-2024 13:42:34.713) (total time: 7098ms):
Trace[1616832970]: [7.098087411s] [7.098087411s] END
2024-06-29T16:42:41.811967376+03:00 I0629 13:42:41.811791       1 trace.go:236] Trace[1834449896]: "Proxy via http_connect protocol over tcp" address:10.42.3.25:10250 (29-Jun-2024 13:42:34.713) (total time: 7098ms):
2024-06-29T16:42:41.811968876+03:00 Trace[1834449896]: [7.098018609s] [7.098018609s] END
2024-06-29T16:42:41.811970376+03:00 I0629 13:42:41.811791       1 trace.go:236] Trace[769577519]: "Proxy via http_connect protocol over tcp" address:10.42.3.25:10250 (29-Jun-2024 13:42:34.713) (total time: 7098ms):
2024-06-29T16:42:41.811972076+03:00 Trace[769577519]: [7.098079111s] [7.098079111s] END
2024-06-29T16:42:41.812416883+03:00 I0629 13:42:41.812343       1 trace.go:236] Trace[311667268]: "Proxy via http_connect protocol over tcp" address:10.42.3.25:10250 (29-Jun-2024 13:42:34.713) (total time: 7098ms):
Trace[311667268]: [7.098643419s] [7.098643419s] END
2024-06-29T16:43:09.192050331+03:00 E0629 13:43:09.191925       1 handler_proxy.go:137] error resolving kube-system/rke2-metrics-server: no endpoints available for service "rke2-metrics-server"
2024-06-29T16:43:09.192107632+03:00 E0629 13:43:09.191931       1 handler_proxy.go:137] error resolving cattle-monitoring-system/rancher-monitoring-prometheus-adapter: no endpoints available for service "rancher-monitoring-prometheus-adapter"
2024-06-29T16:43:11.191938321+03:00 I0629 13:43:11.191314       1 controller.go:615] quota admission added evaluator for: addons.k3s.cattle.io
2024-06-29T16:43:11.274268013+03:00 I0629 13:43:11.274071       1 controller.go:615] quota admission added evaluator for: helmchartconfigs.helm.cattle.io
2024-06-29T16:43:11.632144730+03:00 W0629 13:43:11.631976       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.secrets: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s": no endpoints available for service "rancher-webhook"
W0629 13:43:12.046738       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.secrets: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s": no endpoints available for service "rancher-webhook"
I0629 13:43:27.290889       1 trace.go:236] Trace[1985555354]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/backupbackingimages,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:26.719) (total time: 571ms):
2024-06-29T16:43:27.290993909+03:00 Trace[1985555354]: [571.033363ms] [571.033363ms] END
I0629 13:43:27.294374       1 trace.go:236] Trace[1647675674]: "List" accept:application/json, */*,audit-id:88b0b5a8-33cb-4ce8-a75c-40cf1ee2b217,client:10.0.0.4,api-group:longhorn.io,api-version:v1beta2,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:backupbackingimages,scope:cluster,url:/apis/longhorn.io/v1beta2/backupbackingimages,user-agent:longhorn-manager/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (29-Jun-2024 13:43:26.720) (total time: 573ms):
2024-06-29T16:43:27.295031873+03:00 Trace[1647675674]: ["List(recursive=true) etcd3" audit-id:88b0b5a8-33cb-4ce8-a75c-40cf1ee2b217,key:/longhorn.io/backupbackingimages,resourceVersion:0,resourceVersionMatch:,limit:500,continue: 573ms (13:43:26.720)]
2024-06-29T16:43:27.295035773+03:00 Trace[1647675674]: [573.700904ms] [573.700904ms] END
2024-06-29T16:43:27.298228223+03:00 I0629 13:43:27.295478       1 trace.go:236] Trace[1275505808]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/backupvolumes,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:26.775) (total time: 520ms):
2024-06-29T16:43:27.298263723+03:00 Trace[1275505808]: [520.038262ms] [520.038262ms] END
I0629 13:43:27.297960       1 trace.go:236] Trace[1142530264]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/volumes,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:26.765) (total time: 532ms):
2024-06-29T16:43:27.298281924+03:00 Trace[1142530264]: [532.389156ms] [532.389156ms] END
2024-06-29T16:43:27.303139800+03:00 I0629 13:43:27.299775       1 trace.go:236] Trace[1019650764]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/backingimagemanagers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:26.729) (total time: 570ms):
Trace[1019650764]: [570.402353ms] [570.402353ms] END
2024-06-29T16:43:27.307138863+03:00 I0629 13:43:27.304252       1 trace.go:236] Trace[713093646]: "List" accept:application/json, */*,audit-id:d3163cf9-c033-4fc2-b206-e1443578838a,client:10.0.0.4,api-group:longhorn.io,api-version:v1beta2,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:backingimagemanagers,scope:cluster,url:/apis/longhorn.io/v1beta2/backingimagemanagers,user-agent:longhorn-manager/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (29-Jun-2024 13:43:26.734) (total time: 569ms):
2024-06-29T16:43:27.307158263+03:00 Trace[713093646]: ["List(recursive=true) etcd3" audit-id:d3163cf9-c033-4fc2-b206-e1443578838a,key:/longhorn.io/backingimagemanagers,resourceVersion:0,resourceVersionMatch:,limit:500,continue: 569ms (13:43:26.734)]
2024-06-29T16:43:27.307163163+03:00 Trace[713093646]: [569.273335ms] [569.273335ms] END
2024-06-29T16:43:27.307167963+03:00 I0629 13:43:27.304469       1 trace.go:236] Trace[1319340440]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/backingimagemanagers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:26.734) (total time: 569ms):
2024-06-29T16:43:27.307171563+03:00 Trace[1319340440]: [569.950446ms] [569.950446ms] END
I0629 13:43:27.304751       1 trace.go:236] Trace[1707298078]: "List" accept:application/json, */*,audit-id:0dcee632-7a7e-459d-9561-ec4ca835eb46,client:10.0.0.4,api-group:longhorn.io,api-version:v1beta2,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:volumes,scope:cluster,url:/apis/longhorn.io/v1beta2/volumes,user-agent:longhorn-manager/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (29-Jun-2024 13:43:26.765) (total time: 538ms):
2024-06-29T16:43:27.307191164+03:00 Trace[1707298078]: ["List(recursive=true) etcd3" audit-id:0dcee632-7a7e-459d-9561-ec4ca835eb46,key:/longhorn.io/volumes,resourceVersion:0,resourceVersionMatch:,limit:500,continue: 538ms (13:43:26.765)]
2024-06-29T16:43:27.307194264+03:00 Trace[1707298078]: [538.893959ms] [538.893959ms] END
2024-06-29T16:43:27.307198664+03:00 I0629 13:43:27.305265       1 trace.go:236] Trace[33672064]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/backupvolumes,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:26.780) (total time: 524ms):
2024-06-29T16:43:27.307201864+03:00 Trace[33672064]: [524.781937ms] [524.781937ms] END
2024-06-29T16:43:27.312690250+03:00 I0629 13:43:27.307719       1 trace.go:236] Trace[1998510494]: "List" accept:application/json, */*,audit-id:d79fd06d-fdf1-4056-9cfd-728e592b167b,client:10.0.0.4,api-group:longhorn.io,api-version:v1beta2,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:instancemanagers,scope:cluster,url:/apis/longhorn.io/v1beta2/instancemanagers,user-agent:longhorn-manager/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (29-Jun-2024 13:43:26.794) (total time: 512ms):
2024-06-29T16:43:27.312705350+03:00 Trace[1998510494]: ["List(recursive=true) etcd3" audit-id:d79fd06d-fdf1-4056-9cfd-728e592b167b,key:/longhorn.io/instancemanagers,resourceVersion:0,resourceVersionMatch:,limit:500,continue: 512ms (13:43:26.794)]
2024-06-29T16:43:27.312708550+03:00 Trace[1998510494]: [512.752248ms] [512.752248ms] END
2024-06-29T16:43:27.312711050+03:00 I062
13:43:27.308540       1 trace.go:236] Trace[1833741765]: "List" accept:application/json, */*,audit-id:431eb7a4-5557-484f-b3b4-9e481c7cd10d,client:10.0.0.4,api-group:longhorn.io,api-version:v1beta2,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:backupvolumes,scope:cluster,url:/apis/longhorn.io/v1beta2/backupvolumes,user-agent:longhorn-manager/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (29-Jun-2024 13:43:26.780) (total time: 527ms):
2024-06-29T16:43:27.312713350+03:00 Trace[1833741765]: ["List(recursive=true) etcd3" audit-id:431eb7a4-5557-484f-b3b4-9e481c7cd10d,key:/longhorn.io/backupvolumes,resourceVersion:0,resourceVersionMatch:,limit:500,continue: 527ms (13:43:26.780)]
2024-06-29T16:43:27.312715550+03:00 Trace[1833741765]: [527.466579ms] [527.466579ms] END
2024-06-29T16:43:27.312718350+03:00 I0629 13:43:27.310735       1 trace.go:236] Trace[1698360829]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/instancemanagers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:26.794) (total time: 516ms):
2024-06-29T16:43:27.312720550+03:00 Trace[1698360829]: [516.199602ms] [516.199602ms] END
2024-06-29T16:43:27.312722450+03:00 I0629 13:43:27.311787       1 trace.go:236] Trace[1599704505]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/instancemanagers,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:26.789) (total time: 521ms):
2024-06-29T16:43:27.312724350+03:00 Trace[1599704505]: [521.955592ms] [521.955592ms] END
2024-06-29T16:43:29.396121551+03:00 W0629 13:43:29.395898       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.secrets: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s": no endpoints available for service "rancher-webhook"
2024-06-29T16:43:34.420000705+03:00 I0629 13:43:34.419880       1 trace.go:236] Trace[1327275152]: "Call conversion webhook" custom-resource-defini
tion:volumes.longhorn.io,desired-api-version:longhorn.io/v1beta1,object-count:1,UID:859f84cb-6315-4c68-a2de-4f2b29e509ec (29-Jun-2024 13:43:27.299) (total time: 7120ms):
Trace[1327275152]: ["Proxy via http_connect protocol over tcp" address:10.42.4.18:9501 7119ms (13:43:27.300)]
2024-06-29T16:43:34.420041206+03:00 Trace[1327275152]: [7.120464362s] [7.120464362s] END
I0629 13:43:34.419945       1 trace.go:236] Trace[1403682203]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/volumes,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:26.747) (total time: 7672ms):
Trace[1403682203]: [7.672183121s] [7.672183121s] END
W0629 13:43:34.419974       1 reflector.go:547] storage/cacher.go:/longhorn.io/volumes: failed to list longhorn.io/v1beta1, Kind=Volume: conversion webhook for longhorn.io/v1beta2, Kind=Volume failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.18:9501, code 502: 502 Bad Gateway
2024-06-29T16:43:34.420076807+03:00 I0629 13:43:34.419988       1 trace.go:236] Trace[112996832]: "Call conversion webhook" custom-resource-definition:engineimages.longhorn.io,desired-api-version:longhorn.io/v1beta1,object-count:1,UID:dc830f94-b50a-4123-94b3-328f63908194 (29-Jun-2024 13:43:27.301) (total time: 7117ms):
2024-06-29T16:43:34.420079307+03:00 Trace[112996832]: ["Proxy via http_connect protocol over tcp" address:10.42.4.18:9501 7117ms (13:43:27.302)]
2024-06-29T16:43:34.420081507+03:00 Trace[112996832]: [7.117990123s] [7.117990123s] END
2024-06-29T16:43:34.420083807+03:00 E0629 13:43:34.419996       1 cacher.go:475] cacher (volumes.longhorn.io): unexpected ListAndWatch error: failed to list longhorn.io/v1beta1, Kind=Volume: conversion webhook for longhorn.io/v1beta2, Kind=Volume failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.18:9501, code 502: 502 Bad Gateway; reinitializing...
I0629 13:43:34.420031       1 trace.go:236] Trace[721931098]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/engineimages,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:26.960) (total time: 7459ms):
2024-06-29T16:43:34.420121607+03:00 Trace[721931098]: [7.459137278s] [7.459137278s] END
2024-06-29T16:43:34.420214709+03:00 W0629 13:43:34.420118       1 reflector.go:547] storage/cacher.go:/longhorn.io/engineimages: failed to list longhorn.io/v1beta1, Kind=EngineImage: conversion webhook for longhorn.io/v1beta2, Kind=EngineImage failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.18:9501, code 502: 502 Bad Gateway
2024-06-29T16:43:34.420226209+03:00 E0629 13:43:34.420154       1 cacher.go:475] cacher (engineimages.longhorn.io): unexpected ListAndWatch error: failed to list longhorn.io/v1beta1, Kind=EngineImage: conversion webhook for longhorn.io/v1beta2, Kind=EngineImage failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.18:9501, code 502: 502 Bad Gateway; reinitializing...
2024-06-29T16:43:34.420548814+03:00 I0629 13:43:34.420460       1 trace.go:236] Trace[544251764]: "Call conversion webhook" custom-resource-definition:nodes.longhorn.io,desired-api-version:longhorn.io/v1beta1,object-count:1,UID:01ed12b9-bfce-4351-ba7b-2ea9d35ad421 (29-Jun-2024 13:43:27.300) (total time: 7119ms):
2024-06-29T16:43:34.420557814+03:00 Trace[544251764]: ["Proxy via http_connect protocol over tcp" address:10.42.1.6:9501 7119ms (13:43:27.301)]
Trace[544251764]: [7.119449946s] [7.119449946s] END
2024-06-29T16:43:34.420562914+03:00 I0629 13:43:34.420496       1 trace.go:236] Trace[1077800681]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/nodes,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:26.933) (total time: 7486ms):
2024-06-29T16:43:34.420570114+03:00 Trace[1077800681]: [7.486742711s] [7.486742711s] END
2024-06-29T16:43:34.420573314+03:00 W0629 13:43:34.420512       1 reflector.go:547] storage/cacher.go:/longhorn.io/nodes: failed to list longhorn.io/v1beta1, Kind=Node: conversion webhook for longhorn.io/v1beta2, Kind=Node failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.1.6:9501, code 502: 502 Bad Gateway
2024-06-29T16:43:34.420578914+03:00 E0629 13:43:34.420525       1 cacher.go:475] cacher (nodes.longhorn.io): unexpected ListAndWatch error: failed to list longhorn.io/v1beta1, Kind=Node: conversion webhook for longhorn.io/v1beta2, Kind=Node failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.1.6:9501, code 502: 502 Bad Gateway; reinitializing...
2024-06-29T16:43:34.420799318+03:00 I0629 13:43:34.420560       1 trace.go:236] Trace[549687810]: "Call conversion webhook" custom-resource-definition:backuptargets.longhorn.io,desired-api-version:longhorn.io/v1beta1,object-count:1,UID:e3548798-5196-4ffa-9828-e629e57770ff (29-Jun-2024 13:43:27.303) (total time: 7117ms):
2024-06-29T16:43:34.420807818+03:00 Trace[549687810]: ["Proxy via http_connect protocol over tcp" address:10.42.4.18:9501 7117ms (13:43:27.303)]
2024-06-29T16:43:34.420809718+03:00 Trace[549687810]: [7.117403014s] [7.117403014s] END
I0629 13:43:34.420743       1 trace.go:236] Trace[1969114419]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/backuptargets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:26.808) (total time: 7611ms):
2024-06-29T16:43:34.420862119+03:00 Trace[1969114419]: [7.611970377s] [7.611970377s] END
2024-06-29T16:43:34.420864119+03:00 W0629 13:43:34.420833       1 reflector.go:547] storage/cacher.go:/longhorn.io/backuptargets: failed to list longhorn.io/v1beta1, Kind=BackupTarget: conversion webhook for longhorn.io/v1beta2, Kind=BackupTarget failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.18:9501, code 502: 502 Bad Gateway
2024-06-29T16:43:34.420865919+03:00 E0629 13:43:34.420844       1 cacher.go:475] cacher (backuptargets.longhorn.io): unexpected ListAndWatch error: failed to list longhorn.io/v1beta1, Kind=BackupTarget: conversion webhook for longhorn.io/v1beta2, Kind=BackupTarget failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.18:9501, code 502: 502 Bad Gateway; reinitializing...
2024-06-29T16:43:38.175936358+03:00 W0629 13:43:38.175817       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:43:38.175960758+03:00 E0629 13:43:38.175853       1 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
2024-06-29T16:43:38.175963258+03:00 I0629 13:43:38.175862       1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
2024-06-29T16:43:38.175964859+03:00 W0629 13:43:38.175873       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:43:38.175972459+03:00 W0629 13:43:38.175945       1 handler_proxy.go:93] no RequestInfo found in the con
text
2024-06-29T16:43:38.175974459+03:00 E0629 13:43:38.175951       1 controller.go:102] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
2024-06-29T16:43:38.175981559+03:00 , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-06-29T16:43:38.175985659+03:00 E0629 13:43:38.175961       1 controller.go:113] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
2024-06-29T16:43:38.177314980+03:00 I0629 13:43:38.177181       1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
2024-06-29T16:43:38.177327980+03:00 I0629 13:43:38.177261       1 controller.go:109] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
2024-06-29T16:43:38.178879004+03:00 W0629 13:43:38.178639       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:43:38.178927405+03:00 E0629 13:43:38.178752       1 controller.go:102] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: failed to download v1beta1.custom.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-06-29T16:43:38.178983606+03:00 I0629 13:43:38.178775       1 controller.go:109] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
2024-06-29T16:43:40.308044123+03:00 W0629 13:43:40.307893       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:40.308093524+03:00 W0629 13:43:40.307939       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
I0629 13:43:40.310851       1 trace.go:236] Trace[754387836]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:9b7c6a0b-18f7-4a76-b787-03bafcb1e24b,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-tpxpz,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-tpxpz,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:33.218) (total time: 7092ms):
2024-06-29T16:43:40.310976869+03:00 Trace[754387836]: ["GuaranteedUpdate etcd3" audit-id:9b7c6a0b-18f7-4a76-b787-03bafcb1e24b,key:/management.cattle.io/projects/local/p-tpxpz,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7091ms (13:43:33.219)]
2024-06-29T16:43:40.310982569+03:00 Trace[754387836]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:0c5203a1-b284-4719-acea-fa3ec373ee5c 7088ms (13:43:33.221)
2024-06-29T16:43:40.310986570+03:00 Trace[754387836]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7088ms (13:43:33.222)]]
2024-06-29T16:43:40.310989770+03:00 Trace[754387836]: [7.092006316s] [7.092006316s] END
2024-06-29T16:43:40.310992570+03:00 I0629 13:43:40.310881       1 trace.go:236] Trace[43738933]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:1220e7cd-d631-4001-9fd6-e379e4d5b692,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:33.219) (total time: 7091ms):
2024-06-29T16:43:40.311002370+03:00 Trace[43738933]: ["GuaranteedUpdate etcd3" audit-id:1220e7cd-d631-4001-9fd6-e379e4d5b69
2,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7090ms (13:43:33.220)]
2024-06-29T16:43:40.311005470+03:00 Trace[43738933]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:88a2016f-8c4b-403a-8307-4ccc898e8594 7088ms (13:43:33.222)
2024-06-29T16:43:40.311008170+03:00 Trace[43738933]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7088ms (13:43:33.222)]]
2024-06-29T16:43:40.311011370+03:00 Trace[43738933]: [7.091495907s] [7.091495907s] END
2024-06-29T16:43:42.547953381+03:00 I0629 13:43:42.547821       1 trace.go:236] Trace[167122816]: "Call conversion webhook" custom-resource-definition:nodes.longhorn.io,desired-api-version:longhorn.io/v1beta1,object-count:1,UID:51e075fb-d6c8-47d2-84f7-c7cb760130a0 (29-Jun-2024 13:43:35.425) (total time: 7121ms):
2024-06-29T16:43:42.547987181+03:00 Trace[167122816]: ["Proxy via http_connect protocol over tcp" address:10.42.1.6:9501 7121ms (13:43:35.426)]
2024-06-29T16:43:42.547991281+03:00 Trace[167122816]: [7.121835184s] [7.121835184s] END
2024-06-29T16:43:42.547994781+03:00 I0629 13:43:42.547889       1 trace.go:236] Trace[1734165959]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/nodes,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:35.420) (total time: 7127ms):
2024-06-29T16:43:42.547998181+03:00 Trace[1734165959]: [7.127058365s] [7.127058365s] END
2024-06-29T16:43:42.548001581+03:00 I0629 13:43:42.547887       1 trace.go:236] Trace[1831298416]: "Call conversion webhook" custom-resource-definition:volumes.longhorn.io,desired-api-version:longhorn.io/v1beta1,object-count:1,UID:8e2d368e-ec08-4c9b-af9b-4774d128f9e0 (29-Jun-2024 13:43:35.426) (total time: 7121ms):
2024-06-29T16:43:42.548004282+03:00 Trace[1831298416]: ["Proxy via http_connect protocol over tcp" address:10.42.4.18:9501 7120ms (13:
43:35.427)]
2024-06-29T16:43:42.548006882+03:00 Trace[1831298416]: [7.121225874s] [7.121225874s] END
2024-06-29T16:43:42.548024882+03:00 W0629 13:43:42.547919       1 reflector.go:547] storage/cacher.go:/longhorn.io/nodes: failed to list longhorn.io/v1beta1, Kind=Node: conversion webhook for longhorn.io/v1beta2, Kind=Node failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.1.6:9501, code 502: 502 Bad Gateway
2024-06-29T16:43:42.548028782+03:00 I0629 13:43:42.547943       1 trace.go:236] Trace[1246011941]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/volumes,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:35.420) (total time: 7127ms):
2024-06-29T16:43:42.548031982+03:00 Trace[1246011941]: [7.127093266s] [7.127093266s] END
2024-06-29T16:43:42.548112583+03:00 W0629 13:43:42.547978       1 reflector.go:547] storage/cacher.go:/longhorn.io/volumes: failed to list longhorn.io/v1beta1, Kind=Volume: conversion webhook for longhorn.io/v1beta2, Kind=Volume failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.18:9501, code 502: 502 Bad Gateway
2024-06-29T16:43:42.548126583+03:00 E0629 13:43:42.548002       1 cacher.go:475] cacher (volumes.longhorn.io): unexpected ListAndWatch error: failed to list longhorn.io/v1beta1, Kind=Volume: conversion webhook for longhorn.io/v1beta2, Kind=Volume failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.18:9501, code 502: 502 Bad Gateway; reinitializing...
2024-06-29T16:43:42.548137984+03:00 E0629 13:43:42.547943       1 cacher.go:475] cacher (nodes.longhorn.io): unexpected ListAndWatch error: failed to list longhorn.io/v1beta1, Kind=Node: conversion webhook for longhorn.io/v1beta2, Kind=Node failed: Post "https://longh
orn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.1.6:9501, code 502: 502 Bad Gateway; reinitializing...
2024-06-29T16:43:42.548140884+03:00 I0629 13:43:42.547921       1 trace.go:236] Trace[1017972269]: "Call conversion webhook" custom-resource-definition:engineimages.longhorn.io,desired-api-version:longhorn.io/v1beta1,object-count:1,UID:d77d5540-a202-4012-9986-663d83717a7f (29-Jun-2024 13:43:35.426) (total time: 7121ms):
2024-06-29T16:43:42.548144084+03:00 Trace[1017972269]: ["Proxy via http_connect protocol over tcp" address:10.42.2.4:9501 7121ms (13:43:35.426)]
2024-06-29T16:43:42.548147184+03:00 Trace[1017972269]: [7.121555379s] [7.121555379s] END
2024-06-29T16:43:42.548150184+03:00 I0629 13:43:42.548074       1 trace.go:236] Trace[662903184]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/engineimages,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:35.420) (total time: 7127ms):
2024-06-29T16:43:42.548153184+03:00 Trace[662903184]: [7.127243168s] [7.127243168s] END
2024-06-29T16:43:42.548168684+03:00 I0629 13:43:42.548095       1 trace.go:236] Trace[1089574145]: "Call conversion webhook" custom-resource-definition:backuptargets.longhorn.io,desired-api-version:longhorn.io/v1beta1,object-count:1,UID:2f0c1f80-bc22-4eea-8ee0-cfb181e66042 (29-Jun-2024 13:43:35.426) (total time: 7121ms):
2024-06-29T16:43:42.548171084+03:00 Trace[1089574145]: ["Proxy via http_connect protocol over tcp" address:10.42.4.18:9501 7121ms (13:43:35.426)]
2024-06-29T16:43:42.548173084+03:00 Trace[1089574145]: [7.121882184s] [7.121882184s] END
2024-06-29T16:43:42.548212985+03:00 I0629 13:43:42.548152       1 trace.go:236] Trace[561993776]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/backuptargets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (29-Jun-2024 13:43:35.420) (total time: 7127ms):
2024-06-29T16:43:42.548219785+03:00 Trace[561993776]: [7.127212768s] [7.127212768s] END
2024-06-29T1
6:43:42.548223685+03:00 W0629 13:43:42.548182       1 reflector.go:547] storage/cacher.go:/longhorn.io/backuptargets: failed to list longhorn.io/v1beta1, Kind=BackupTarget: conversion webhook for longhorn.io/v1beta2, Kind=BackupTarget failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.18:9501, code 502: 502 Bad Gateway
2024-06-29T16:43:42.548226885+03:00 E0629 13:43:42.548192       1 cacher.go:475] cacher (backuptargets.longhorn.io): unexpected ListAndWatch error: failed to list longhorn.io/v1beta1, Kind=BackupTarget: conversion webhook for longhorn.io/v1beta2, Kind=BackupTarget failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.18:9501, code 502: 502 Bad Gateway; reinitializing...
2024-06-29T16:43:42.548234485+03:00 W0629 13:43:42.548099       1 reflector.go:547] storage/cacher.go:/longhorn.io/engineimages: failed to list longhorn.io/v1beta1, Kind=EngineImage: conversion webhook for longhorn.io/v1beta2, Kind=EngineImage failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.2.4:9501, code 502: 502 Bad Gateway
2024-06-29T16:43:42.548270386+03:00 E0629 13:43:42.548226       1 cacher.go:475] cacher (engineimages.longhorn.io): unexpected ListAndWatch error: failed to list longhorn.io/v1beta1, Kind=EngineImage: conversion webhook for longhorn.io/v1beta2, Kind=EngineImage failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.2.4:9501, code 502: 502 Bad Gateway; reinitializing...
2024-06-29T16:43:47.110662496+03:00 I0629 13:43:47.110515       1 controller.go:615] quota admission added evaluator for: nodes.management.cattle.io
2024-06-29T16:43:47.412500034+03:00 W0629 13:43:47.412021
       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:47.412518734+03:00 W0629 13:43:47.412111       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:47.412520834+03:00 W0629 13:43:47.412111       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:47.412522434+03:00 W0629 13:43:47.412223       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:47.412840339+03:00 W0629 13:43:47.412519       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webho
ok "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:47.412851140+03:00 W0629 13:43:47.412782       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:47.412898240+03:00 W0629 13:43:47.412848       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:47.431336930+03:00 I0629 13:43:47.431014       1 trace.go:236] Trace[708405810]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:5acadf88-c607-47b2-a0e4-3e95275cf81c,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:40.317) (total time: 7113ms):
2024-06-29T16:43:47.431371030+03:00 Trace[708405810]: ["GuaranteedUpdate etcd3" audit-id:5acadf88-c607-47b2-a0e4-3e95275cf81c,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7112ms 
(13:43:40.318)]
2024-06-29T16:43:47.431375630+03:00 Trace[708405810]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:69da2f07-c9d6-4517-8ad6-9738613375cd 7109ms (13:43:40.321)
2024-06-29T16:43:47.431378030+03:00 Trace[708405810]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7108ms (13:43:40.322)]]
2024-06-29T16:43:47.431381430+03:00 Trace[708405810]: ---"Write to database call failed" len:2750,err:Internal error occurred: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway 18ms (13:43:47.430)
2024-06-29T16:43:47.431384530+03:00 Trace[708405810]: [7.113516753s] [7.113516753s] END
2024-06-29T16:43:47.431507032+03:00 I0629 13:43:47.431396       1 trace.go:236] Trace[258393720]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:6c9d457f-e1b3-4e70-a402-c70654b11085,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:40.321) (total time: 7110ms):
2024-06-29T16:43:47.431513132+03:00 Trace[258393720]: ["GuaranteedUpdate etcd3" audit-id:6c9d457f-e1b3-4e70-a402-c70654b11085,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7109ms (13:43:40.322)]
2024-06-29T16:43:47.431515732+03:00 Trace[258393720]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resou
rce=projects,subresource:,operation:UPDATE,UID:2f27efed-0be6-4d43-94a2-afb56b1f5822 7106ms (13:43:40.325)
2024-06-29T16:43:47.431518233+03:00 Trace[258393720]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7105ms (13:43:40.325)]]
2024-06-29T16:43:47.431577433+03:00 Trace[258393720]: ---"Write to database call failed" len:2750,err:Internal error occurred: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway 19ms (13:43:47.431)
2024-06-29T16:43:47.431660335+03:00 Trace[258393720]: [7.110067599s] [7.110067599s] END
2024-06-29T16:43:47.431758936+03:00 I0629 13:43:47.431707       1 trace.go:236] Trace[1091076032]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:5accc334-d25c-4932-8677-b83a8cfe699a,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:40.317) (total time: 7114ms):
2024-06-29T16:43:47.431772537+03:00 Trace[1091076032]: ["GuaranteedUpdate etcd3" audit-id:5accc334-d25c-4932-8677-b83a8cfe699a,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7113ms (13:43:40.318)]
2024-06-29T16:43:47.431775037+03:00 Trace[1091076032]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:08ce73cc-2122-4440-98d2-611e5a4f6fb6 7110ms (13:43:40.321)
2024-06-29T16:43:47.431777037+03:00 Trace[1091076032]:  ["Proxy via http_connect protocol over tcp" address:10.42.4
7109ms (13:43:40.321)]]
2024-06-29T16:43:47.431779537+03:00 Trace[1091076032]: ---"Write to database call failed" len:2750,err:Internal error occurred: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway 18ms (13:43:47.431)
2024-06-29T16:43:47.431782137+03:00 Trace[1091076032]: [7.114388067s] [7.114388067s] END
2024-06-29T16:43:47.431870338+03:00 I0629 13:43:47.431763       1 trace.go:236] Trace[165462980]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:64a46c79-2d1c-4e5a-8510-24e92e165e31,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:40.315) (total time: 7115ms):
2024-06-29T16:43:47.431883838+03:00 Trace[165462980]: ["GuaranteedUpdate etcd3" audit-id:64a46c79-2d1c-4e5a-8510-24e92e165e31,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7114ms (13:43:40.317)]
2024-06-29T16:43:47.431891538+03:00 Trace[165462980]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:a2b219c4-247d-4802-85fe-39f240339844 7111ms (13:43:40.319)
2024-06-29T16:43:47.431895838+03:00 Trace[165462980]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7111ms (13:43:40.320)]]
2024-06-29T16:43:47.431899039+03:00 Trace[165462980]: ---"Write to database call failed" len:2750,err:Internal error occurred: failed calling webhook "rancher.cattle.io.projects.management.
cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway 19ms (13:43:47.431)
2024-06-29T16:43:47.431902539+03:00 Trace[165462980]: [7.115736488s] [7.115736488s] END
2024-06-29T16:43:47.431905739+03:00 I0629 13:43:47.431792       1 trace.go:236] Trace[740861668]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:f5d27fb2-7c0d-4c52-a19b-e2b8672014d0,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:40.317) (total time: 7114ms):
2024-06-29T16:43:47.431908939+03:00 Trace[740861668]: ["GuaranteedUpdate etcd3" audit-id:f5d27fb2-7c0d-4c52-a19b-e2b8672014d0,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7113ms (13:43:40.318)]
2024-06-29T16:43:47.431921139+03:00 Trace[740861668]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:6e84a801-09f7-4dec-81e8-c3071c9eed2c 7110ms (13:43:40.321)
2024-06-29T16:43:47.431924239+03:00 Trace[740861668]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7109ms (13:43:40.322)]]
2024-06-29T16:43:47.431927239+03:00 Trace[740861668]: ---"Write to database call failed" len:2750,err:Internal error occurred: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 50
2: 502 Bad Gateway 19ms (13:43:47.431)
2024-06-29T16:43:47.431930039+03:00 Trace[740861668]: [7.114156163s] [7.114156163s] END
2024-06-29T16:43:47.431939439+03:00 I0629 13:43:47.431794       1 trace.go:236] Trace[970826710]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:a599cbfd-928b-452a-bebf-33e99bf8bb20,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-tpxpz,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-tpxpz,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:40.315) (total time: 7116ms):
2024-06-29T16:43:47.431943039+03:00 Trace[970826710]: ["GuaranteedUpdate etcd3" audit-id:a599cbfd-928b-452a-bebf-33e99bf8bb20,key:/management.cattle.io/projects/local/p-tpxpz,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7115ms (13:43:40.316)]
2024-06-29T16:43:47.431946339+03:00 Trace[970826710]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:0ecfa289-4d12-4ae5-b393-48625ff41394 7109ms (13:43:40.322)
2024-06-29T16:43:47.431950139+03:00 Trace[970826710]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7109ms (13:43:40.322)]]
2024-06-29T16:43:47.431953439+03:00 Trace[970826710]: ---"Write to database call failed" len:2123,err:Internal error occurred: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway 18ms (13:43:47.431)
2024-06-29T16:43:47.431956339+03:00 Trace[970826710]: [7.116185795s] [7.116185795s] END
2024-06-29T16:43:47.432003640+03:00 I0629 13:43:47.431797       1 trace.go:236] Trace[1101314313]
: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:bbddff18-cc6a-4eac-9fca-02bcc9828738,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:40.315) (total time: 7115ms):
2024-06-29T16:43:47.432008940+03:00 Trace[1101314313]: ["GuaranteedUpdate etcd3" audit-id:bbddff18-cc6a-4eac-9fca-02bcc9828738,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7114ms (13:43:40.317)]
2024-06-29T16:43:47.432012240+03:00 Trace[1101314313]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:a9379b40-d412-4e6d-8dae-457f7d9b23be 7111ms (13:43:40.320)
2024-06-29T16:43:47.432019940+03:00 Trace[1101314313]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7110ms (13:43:40.321)]]
2024-06-29T16:43:47.432022540+03:00 Trace[1101314313]: ---"Write to database call failed" len:2750,err:Internal error occurred: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway 19ms (13:43:47.431)
2024-06-29T16:43:47.432024440+03:00 Trace[1101314313]: [7.11591899s] [7.11591899s] END
2024-06-29T16:43:54.580146336+03:00 W0629 13:43:54.579955       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.sv
c:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:54.580197337+03:00 W0629 13:43:54.579977       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:54.580313739+03:00 W0629 13:43:54.580215       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:54.580324739+03:00 W0629 13:43:54.580223       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:54.580327839+03:00 W0629 13:43:54.580261       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, c
ode 502: 502 Bad Gateway
2024-06-29T16:43:54.580513642+03:00 W0629 13:43:54.580137       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:54.580543643+03:00 W0629 13:43:54.580409       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:54.583032282+03:00 I0629 13:43:54.582940       1 trace.go:236] Trace[1007816024]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:dfcb8965-ed74-44d8-8412-1432e1c31cda,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:47.434) (total time: 7148ms):
2024-06-29T16:43:54.583074782+03:00 Trace[1007816024]: ["GuaranteedUpdate etcd3" audit-id:dfcb8965-ed74-44d8-8412-1432e1c31cda,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7147ms (13:43:47.435)]
2024-06-29T16:43:54.583083883+03:00 Trace[1007816024]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operati
on:UPDATE,UID:f1c54988-245e-47dd-aa45-5448f1987f99 7145ms (13:43:47.437)
2024-06-29T16:43:54.583087083+03:00 Trace[1007816024]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7144ms (13:43:47.437)]]
2024-06-29T16:43:54.583089883+03:00 Trace[1007816024]: [7.148278498s] [7.148278498s] END
2024-06-29T16:43:54.583497789+03:00 I0629 13:43:54.583402       1 trace.go:236] Trace[133059863]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:04ef86fa-a880-4e5c-80bd-050b5915f8ae,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-tpxpz,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-tpxpz,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:47.433) (total time: 7150ms):
2024-06-29T16:43:54.583512289+03:00 Trace[133059863]: ["GuaranteedUpdate etcd3" audit-id:04ef86fa-a880-4e5c-80bd-050b5915f8ae,key:/management.cattle.io/projects/local/p-tpxpz,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7149ms (13:43:47.433)]
2024-06-29T16:43:54.583518889+03:00 Trace[133059863]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:ae09f48b-b05d-4132-b45d-8f467a33aa29 7147ms (13:43:47.435)
2024-06-29T16:43:54.583524289+03:00 Trace[133059863]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7146ms (13:43:47.436)]]
2024-06-29T16:43:54.583527890+03:00 Trace[133059863]: [7.150029126s] [7.150029126s] END
2024-06-29T16:43:54.583843495+03:00 I0629 13:43:54.583769       1 trace.go:236] Trace[65422712]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:760f165d-8bd4-479b-af70-588127aa0ee1,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resourc
e,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:47.433) (total time: 7149ms):
2024-06-29T16:43:54.583861695+03:00 Trace[65422712]: ["GuaranteedUpdate etcd3" audit-id:760f165d-8bd4-479b-af70-588127aa0ee1,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7149ms (13:43:47.434)]
2024-06-29T16:43:54.583864995+03:00 Trace[65422712]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:595adc19-d202-45da-b9ae-432b564d2979 7147ms (13:43:47.436)
2024-06-29T16:43:54.583874695+03:00 Trace[65422712]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7146ms (13:43:47.436)]]
2024-06-29T16:43:54.583877495+03:00 Trace[65422712]: [7.14969402s] [7.14969402s] END
2024-06-29T16:43:54.583927396+03:00 I0629 13:43:54.583416       1 trace.go:236] Trace[950001065]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:d4cf21c4-87e7-4557-8638-cea464497fc6,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:47.433) (total time: 7149ms):
2024-06-29T16:43:54.583933096+03:00 Trace[950001065]: ["GuaranteedUpdate etcd3" audit-id:d4cf21c4-87e7-4557-8638-cea464497fc6,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7149ms (13:43:47.434)]
2024-06-29T16:43:54.583935696+03:00 Trace[950001065]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subre
source:,operation:UPDATE,UID:782b4bfc-e65e-4758-8f63-804b3f5f85b3 7146ms (13:43:47.436)
2024-06-29T16:43:54.583938096+03:00 Trace[950001065]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7146ms (13:43:47.436)]]
2024-06-29T16:43:54.583940496+03:00 Trace[950001065]: [7.149976125s] [7.149976125s] END
2024-06-29T16:43:54.584025197+03:00 I0629 13:43:54.583752       1 trace.go:236] Trace[1447454604]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:b0ddc9ed-4281-4e99-b86c-65943c70899e,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:47.434) (total time: 7149ms):
2024-06-29T16:43:54.584038198+03:00 Trace[1447454604]: ["GuaranteedUpdate etcd3" audit-id:b0ddc9ed-4281-4e99-b86c-65943c70899e,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7148ms (13:43:47.434)]
2024-06-29T16:43:54.584042598+03:00 Trace[1447454604]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:6fbb5f50-857b-47d2-bfbb-7d92c0a60797 7146ms (13:43:47.437)
2024-06-29T16:43:54.584045498+03:00 Trace[1447454604]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7145ms (13:43:47.437)]]
2024-06-29T16:43:54.584048198+03:00 Trace[1447454604]: [7.149567819s] [7.149567819s] END
2024-06-29T16:43:54.584156699+03:00 I0629 13:43:54.583756       1 trace.go:236] Trace[361193639]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:50525ab8-041a-42d5-b2cc-a0e2c9e38095,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:pro
jects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:47.433) (total time: 7149ms):
2024-06-29T16:43:54.584189000+03:00 Trace[361193639]: ["GuaranteedUpdate etcd3" audit-id:50525ab8-041a-42d5-b2cc-a0e2c9e38095,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7148ms (13:43:47.434)]
2024-06-29T16:43:54.584194000+03:00 Trace[361193639]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:d8d3fb4e-b65c-4b4d-88e6-61271689f34d 7146ms (13:43:47.436)
2024-06-29T16:43:54.584208400+03:00 Trace[361193639]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7146ms (13:43:47.437)]]
2024-06-29T16:43:54.584210900+03:00 Trace[361193639]: [7.149567418s] [7.149567418s] END
2024-06-29T16:43:54.584512705+03:00 I0629 13:43:54.584445       1 trace.go:236] Trace[1426638615]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:3021d76a-3d99-4593-b4c3-8acda52e7c01,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:47.434) (total time: 7150ms):
2024-06-29T16:43:54.584519705+03:00 Trace[1426638615]: ["GuaranteedUpdate etcd3" audit-id:3021d76a-3d99-4593-b4c3-8acda52e7c01,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7149ms (13:43:47.435)]
2024-06-29T16:43:54.584523205+03:00 Trace[1426638615]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io
/v3, Resource=projects,subresource:,operation:UPDATE,UID:5a2ff7ca-8ad0-4770-9b6c-cc06c31ac8f4 7146ms (13:43:47.437)
2024-06-29T16:43:54.584526905+03:00 Trace[1426638615]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7146ms (13:43:47.437)]]
2024-06-29T16:43:54.584530105+03:00 Trace[1426638615]: [7.150089327s] [7.150089327s] END
2024-06-29T16:43:54.707808540+03:00 W0629 13:43:54.707664       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:43:54.710991090+03:00 I0629 13:43:54.710895       1 trace.go:236] Trace[715828154]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:4fd5be2e-9ee1-4275-a966-4366c36d41c4,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:local,subresource:,namespace:,protocol:HTTP/2.0,resource:clusters,scope:resource,url:/apis/management.cattle.io/v3/clusters/local,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:47.560) (total time: 7150ms):
2024-06-29T16:43:54.711010190+03:00 Trace[715828154]: ["GuaranteedUpdate etcd3" audit-id:4fd5be2e-9ee1-4275-a966-4366c36d41c4,key:/management.cattle.io/clusters/local,type:*unstructured.Unstructured,resource:clusters.management.cattle.io 7149ms (13:43:47.561)]
2024-06-29T16:43:54.711013991+03:00 Trace[715828154]: ["Call mutating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.clusters.management.cattle.io,resource:management.cattle.io/v3, Resource=clusters,subresource:,operation:UPDATE,UID:532edda1-a04b-48bd-b5b0-70b0f8c090cf 7147ms (13:43:47.563)
2024-06-29T16:43:54.711016591+03:00 Trace[715828154]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:
7146ms (13:43:47.564)]]
2024-06-29T16:43:54.711019291+03:00 Trace[715828154]: [7.150649036s] [7.150649036s] END
2024-06-29T16:44:01.683890336+03:00 W0629 13:44:01.683748       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:44:01.684283342+03:00 W0629 13:44:01.683741       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:44:01.684303742+03:00 W0629 13:44:01.683887       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:44:01.684307642+03:00 W0629 13:44:01.683897       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:44:01
.684310443+03:00 W0629 13:44:01.683911       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:44:01.684313243+03:00 W0629 13:44:01.684191       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:44:01.684316743+03:00 W0629 13:44:01.684191       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:44:01.686833782+03:00 I0629 13:44:01.686728       1 trace.go:236] Trace[947317980]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:7eea74b2-25c6-494c-9800-9a7cdb0d3169,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.586) (total time: 7100ms):
2024-06-29T16:44:01.686854982+03:00 Trace[947317980]: ["GuaranteedUpdate etcd3"
 audit-id:7eea74b2-25c6-494c-9800-9a7cdb0d3169,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7100ms (13:43:54.586)]
2024-06-29T16:44:01.686860183+03:00 Trace[947317980]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:63351d31-18ff-46e5-b7e1-f9f315bfda34 7097ms (13:43:54.588)
2024-06-29T16:44:01.686871183+03:00 Trace[947317980]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7096ms (13:43:54.589)]]
2024-06-29T16:44:01.686877383+03:00 Trace[947317980]: [7.100631451s] [7.100631451s] END
2024-06-29T16:44:01.686992085+03:00 I0629 13:44:01.686844       1 trace.go:236] Trace[1274170684]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:b6e20855-4cba-43c3-b5ad-24b1cb6b977a,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.586) (total time: 7100ms):
2024-06-29T16:44:01.687005185+03:00 Trace[1274170684]: ["GuaranteedUpdate etcd3" audit-id:b6e20855-4cba-43c3-b5ad-24b1cb6b977a,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7099ms (13:43:54.587)]
2024-06-29T16:44:01.687009285+03:00 Trace[1274170684]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:5ad5bf54-356e-40b1-b972-0aeab3337b89 7096ms (13:43:54.590)
2024-06-29T16:44:01.687012485+03:00 Trace[1274170684]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7095ms (13:43:54.591)]]
2024-06-29T16:44:01.6
87015785+03:00 Trace[1274170684]: [7.100171043s] [7.100171043s] END
2024-06-29T16:44:01.687093486+03:00 I0629 13:44:01.686848       1 trace.go:236] Trace[1416997392]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:aa560905-158b-47b0-a0cb-b5abf934f71f,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.585) (total time: 7101ms):
2024-06-29T16:44:01.687106686+03:00 Trace[1416997392]: ["GuaranteedUpdate etcd3" audit-id:aa560905-158b-47b0-a0cb-b5abf934f71f,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7100ms (13:43:54.586)]
2024-06-29T16:44:01.687110986+03:00 Trace[1416997392]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:b54ccc77-806d-4c94-aba5-8acd80d0775c 7097ms (13:43:54.589)
2024-06-29T16:44:01.687115087+03:00 Trace[1416997392]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7096ms (13:43:54.590)]]
2024-06-29T16:44:01.687132887+03:00 Trace[1416997392]: [7.10122676s] [7.10122676s] END
2024-06-29T16:44:01.687427791+03:00 I0629 13:44:01.687310       1 trace.go:236] Trace[244916747]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:e43d3bba-ab01-476d-8c36-ddca1e9f962b,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.586) (total time: 7100ms):
2024-06-29T16:44:01.687436192+03:00 Trace[244916747
]: ["GuaranteedUpdate etcd3" audit-id:e43d3bba-ab01-476d-8c36-ddca1e9f962b,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7100ms (13:43:54.587)]
2024-06-29T16:44:01.687448192+03:00 Trace[244916747]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:421e7053-14a0-4a24-bb40-add20620f7fb 7097ms (13:43:54.589)
2024-06-29T16:44:01.687451692+03:00 Trace[244916747]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7097ms (13:43:54.590)]]
2024-06-29T16:44:01.687454892+03:00 Trace[244916747]: [7.100865555s] [7.100865555s] END
2024-06-29T16:44:01.687458492+03:00 I0629 13:44:01.687355       1 trace.go:236] Trace[2027599660]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:f8fedb04-19ab-464b-b410-1dffc91673a3,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-tpxpz,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-tpxpz,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.585) (total time: 7101ms):
2024-06-29T16:44:01.687461492+03:00 Trace[2027599660]: ["GuaranteedUpdate etcd3" audit-id:f8fedb04-19ab-464b-b410-1dffc91673a3,key:/management.cattle.io/projects/local/p-tpxpz,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7100ms (13:43:54.586)]
2024-06-29T16:44:01.687464792+03:00 Trace[2027599660]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:6ae8fc07-623d-49ca-96db-9a8f16e996c2 7098ms (13:43:54.588)
2024-06-29T16:44:01.687467692+03:00 Trace[2027599660]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7098ms (13:43:54.
589)]]
2024-06-29T16:44:01.687470392+03:00 Trace[2027599660]: [7.101615966s] [7.101615966s] END
2024-06-29T16:44:01.687656195+03:00 I0629 13:44:01.687554       1 trace.go:236] Trace[1296692068]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:b53bfb5b-de1a-46e1-bdee-5a9d05ba48ff,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.586) (total time: 7100ms):
2024-06-29T16:44:01.687663095+03:00 Trace[1296692068]: ["GuaranteedUpdate etcd3" audit-id:b53bfb5b-de1a-46e1-bdee-5a9d05ba48ff,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7099ms (13:43:54.587)]
2024-06-29T16:44:01.687665695+03:00 Trace[1296692068]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:b44d8e60-6e96-4db8-a03e-9ca482a43942 7096ms (13:43:54.590)
2024-06-29T16:44:01.687668095+03:00 Trace[1296692068]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7096ms (13:43:54.590)]]
2024-06-29T16:44:01.687670795+03:00 Trace[1296692068]: [7.100702952s] [7.100702952s] END
2024-06-29T16:44:01.687750497+03:00 I0629 13:44:01.687672       1 trace.go:236] Trace[983609095]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:2e8b8020-8a63-4c68-a07b-5529f79e6587,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.586) (total time: 7101ms):
2024-06-29T16:44:01.6
87756797+03:00 Trace[983609095]: ["GuaranteedUpdate etcd3" audit-id:2e8b8020-8a63-4c68-a07b-5529f79e6587,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7100ms (13:43:54.587)]
2024-06-29T16:44:01.687782497+03:00 Trace[983609095]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:a633182d-bbb9-44a0-947c-b0f77cb74ef0 7097ms (13:43:54.589)
2024-06-29T16:44:01.687786497+03:00 Trace[983609095]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7097ms (13:43:54.590)]]
2024-06-29T16:44:01.687789497+03:00 Trace[983609095]: [7.101374362s] [7.101374362s] END
2024-06-29T16:44:01.702355026+03:00 I0629 13:44:01.702241       1 controller.go:615] quota admission added evaluator for: projects.management.cattle.io
2024-06-29T16:44:05.020902413+03:00 I0629 13:44:05.020815       1 controller.go:615] quota admission added evaluator for: replicas.longhorn.io
2024-06-29T16:44:05.160711808+03:00 I0629 13:44:05.160592       1 controller.go:615] quota admission added evaluator for: engines.longhorn.io
2024-06-29T16:44:05.315099631+03:00 I0629 13:44:05.314993       1 controller.go:615] quota admission added evaluator for: volumes.longhorn.io
2024-06-29T16:44:07.679868348+03:00 I0629 13:44:07.679653       1 controller.go:615] quota admission added evaluator for: poddisruptionbudgets.policy
2024-06-29T16:44:07.679889948+03:00 I0629 13:44:07.679653       1 controller.go:615] quota admission added evaluator for: poddisruptionbudgets.policy
2024-06-29T16:44:07.679893048+03:00 I0629 13:44:07.679653       1 controller.go:615] quota admission added evaluator for: poddisruptionbudgets.policy
2024-06-29T16:44:09.192121384+03:00 E0629 13:44:09.191953       1 handler_proxy.go:137] error resolving kube-system/rke2-metrics-server: no endpoints available for service "rke2-metrics-server"
2024-06-29T16:44:09.19215068
5+03:00 E0629 13:44:09.191977       1 handler_proxy.go:137] error resolving cattle-monitoring-system/rancher-monitoring-prometheus-adapter: no endpoints available for service "rancher-monitoring-prometheus-adapter"
2024-06-29T16:44:56.548568085+03:00 I0629 13:44:56.548391       1 trace.go:236] Trace[27991602]: "Get" accept:,audit-id:a36c994f-bbcf-403c-a62e-e28977b7fdd2,client:10.0.0.1,api-group:,api-version:v1,name:rke2-metrics-server-c9dcd6cc7-wsb5c,subresource:log,namespace:kube-system,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/rke2-metrics-server-c9dcd6cc7-wsb5c/log,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36,verb:CONNECT (29-Jun-2024 13:44:54.979) (total time: 1569ms):
2024-06-29T16:44:56.548620086+03:00 Trace[27991602]: ---"Writing http response done" 1565ms (13:44:56.548)
2024-06-29T16:44:56.548626386+03:00 Trace[27991602]: [1.569176529s] [1.569176529s] END
2024-06-29T16:45:08.348671898+03:00 I0629 13:45:08.348561       1 controller.go:615] quota admission added evaluator for: snapshots.longhorn.io
2024-06-29T16:45:08.354083283+03:00 I0629 13:45:08.353963       1 controller.go:615] quota admission added evaluator for: volumeattachments.longhorn.io
2024-06-29T16:45:09.192528243+03:00 E0629 13:45:09.192394       1 handler_proxy.go:137] error resolving kube-system/rke2-metrics-server: no endpoints available for service "rke2-metrics-server"
2024-06-29T16:45:09.192567244+03:00 E0629 13:45:09.192452       1 handler_proxy.go:137] error resolving cattle-monitoring-system/rancher-monitoring-prometheus-adapter: no endpoints available for service "rancher-monitoring-prometheus-adapter"
2024-06-29T16:45:20.465205177+03:00 I0629 13:45:20.465092       1 handler.go:286] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T16:45:21.699563652+03:00 W0629 13:45:21.699419       1 aggregator.go:166] failed to download v1beta1.custom.metrics.k8s.io: failed to retrieve openAPI spec, http
 error: ResponseCode: 503, Body: service unavailable
2024-06-29T16:45:21.699611953+03:00 , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-06-29T16:45:22.905398178+03:00 W0629 13:45:22.905244       1 aggregator.go:166] failed to download v1beta1.custom.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
2024-06-29T16:45:22.905436479+03:00 , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-06-29T16:45:38.178298100+03:00 W0629 13:45:38.177955       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:45:38.178335700+03:00 E0629 13:45:38.178052       1 controller.go:113] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
2024-06-29T16:45:38.178339701+03:00 I0629 13:45:38.178157       1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
2024-06-29T16:45:38.179158213+03:00 W0629 13:45:38.179088       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:45:38.179305016+03:00 E0629 13:45:38.179243       1 controller.go:102] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: failed to download v1beta1.custom.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
2024-06-29T16:45:38.179318916+03:00 , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-06-29T16:45:38.179324316+03:00 I0629 13:45:38.179260       1 controller.go:109] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
2024-06-29T16:45:54.320650768+03:00 E0629 13:45:54.320536       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io": the object has been modified; p
lease apply your changes to the latest version and try again
2024-06-29T16:45:54.343556328+03:00 I0629 13:45:54.343442       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T16:46:09.156124924+03:00 I0629 13:46:09.155972       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T16:48:09.267199972+03:00 I0629 13:48:09.267050       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T16:48:09.267855982+03:00 E0629 13:48:09.267766       1 writers.go:68] error encountered while streaming results via websocket: use of closed network connection
2024-06-29T16:48:09.268113286+03:00 I0629 13:48:09.268019       1 trace.go:236] Trace[1548674291]: "Get" accept:,audit-id:8223336c-03e6-4551-a1cd-3be6af76f732,client:10.0.0.1,api-group:,api-version:v1,name:kube-apiserver-rke21,subresource:log,namespace:kube-system,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/kube-apiserver-rke21/log,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36,verb:CONNECT (29-Jun-2024 13:46:49.216) (total time: 80051ms):
2024-06-29T16:48:09.268134287+03:00 Trace[1548674291]: ---"Writing http response done" 80046ms (13:48:09.267)
2024-06-29T16:48:09.268138787+03:00 Trace[1548674291]: [1m20.051471978s] [1m20.051471978s] END
2024-06-29T16:50:09.253365860+03:00 I0629 13:50:09.253226       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T16:50:09.253981769+03:00 E0629 13:50:09.253867       1 writers.go:68] error encountered while streaming results via websocket: use of closed network connection
2024-06-29T16:50:09.254076471+03:00 I0629 13:50:09.253995       1 trace.go:236] Trace[12298559]: "Get" accept:,audit-id:59d0698c-3d1b-4f1b-8924-1329b89a99e5,client:10.0.0.1,api-group:,api-version:v1,name:kube-apiserver-rke21,subresource:log,namespace:kube-system,protocol:H
TTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/kube-apiserver-rke21/log,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36,verb:CONNECT (29-Jun-2024 13:47:04.509) (total time: 184744ms):
2024-06-29T16:50:09.254100771+03:00 Trace[12298559]: ---"Writing http response done" 184738ms (13:50:09.253)
2024-06-29T16:50:09.254104771+03:00 Trace[12298559]: [3m4.744911033s] [3m4.744911033s] END

rke2-metrics-server logs
`

I0629 13:44:57.000550       1 handler.go:275] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T16:44:57.123332227+03:00 I0629 13:44:57.122356       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
2024-06-29T16:44:57.123355627+03:00 I0629 13:44:57.122384       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
2024-06-29T16:44:57.123361228+03:00 I0629 13:44:57.122423       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
2024-06-29T16:44:57.123365628+03:00 I0629 13:44:57.122438       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
2024-06-29T16:44:57.123369528+03:00 I0629 13:44:57.122460       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0629 13:44:57.122468       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
2024-06-29T16:44:57.123395628+03:00 I0629 13:44:57.122869       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
I0629 13:44:57.123761       1 secure_serving.go:213] Serving securely on [::]:10250
I0629 13:44:57.123811       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
2024-06-29T16:44:57.223303697+03:00 I0629 13:44:57.223070       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I0629 13:44:57.223070       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
2024-06-29T16:44:57.223328798+03:00 I0629 13:44:57.223088       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file

[cengizhanR]

My hosts file on every server and agent node following below, I dont have firewalls, all of them disabled.

127.0.0.1               localhost.localdomain    localhost
::1                     localhost6.localdomain6    localhost6
# cluster nodes.
10.0.0.2          rke21.rancher.exemple.com    rke21
10.0.0.3          rke22.rancher.exemple.com    rke22
10.0.0.4          rke23.rancher.exemple.com    rke23
10.0.0.5          rke2agent1.rancher.exemple.com    rke2agent1
10.0.0.6          rke2agent2.rancher.exemple.com    rke2agent2

When ı run this command,

[root@rke21 bin]# kubectl describe apiservice v1beta1.metrics.k8s.io
Name:         v1beta1.metrics.k8s.io
Namespace:
Labels:       app=rke2-metrics-server
              app.kubernetes.io/instance=rke2-metrics-server
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=rke2-metrics-server
              app.kubernetes.io/version=0.7.1
              helm.sh/chart=rke2-metrics-server-3.12.002
Annotations:  meta.helm.sh/release-name: rke2-metrics-server
              meta.helm.sh/release-namespace: kube-system
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2024-05-17T10:59:07Z
  Resource Version:    22519723
  UID:                 1828cf41-0c76-49ae-afc3-c91c852ae9fa
Spec:
  Group:                     metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            rke2-metrics-server
    Namespace:       kube-system
    Port:            443
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2024-06-29T13:23:35Z
    Message:               all checks passed
    Reason:                Passed
    Status:                True
    Type:                  Available
Events:                    <none>

[cengizhanR]

This is last logs from kubeapi-server

`715828154]: [7.150649036s] [7.150649036s] END
2024-06-29T16:44:01.683890336+03:00 W0629 13:44:01.683748       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:44:01.684283342+03:00 W0629 13:44:01.683741       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:44:01.684303742+03:00 W0629 13:44:01.683887       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
W0629 13:44:01.683897       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:44:01.684310443+03:00 W0629 13:44:01.683911       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:44:01.684313243+03:00 W0629 13:44:01.684191       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:44:01.684316743+03:00 W0629 13:44:01.684191       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": proxy error from 127.0.0.1:9345 while dialing 10.42.4.16:9443, code 502: 502 Bad Gateway
2024-06-29T16:44:01.686833782+03:00 I0629 13:44:01.686728       1 trace.go:236] Trace[947317980]: "Update" accept:application/vnd.kubern
etes.protobuf, application/json,audit-id:7eea74b2-25c6-494c-9800-9a7cdb0d3169,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.586) (total time: 7100ms):
Trace[947317980]: ["GuaranteedUpdate etcd3" audit-id:7eea74b2-25c6-494c-9800-9a7cdb0d3169,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7100ms (13:43:54.586)]
2024-06-29T16:44:01.686860183+03:00 Trace[947317980]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:63351d31-18ff-46e5-b7e1-f9f315bfda34 7097ms (13:43:54.588)
Trace[947317980]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7096ms (13:43:54.589)]]
2024-06-29T16:44:01.686877383+03:00 Trace[947317980]: [7.100631451s] [7.100631451s] END
2024-06-29T16:44:01.686992085+03:00 I0629 13:44:01.686844       1 trace.go:236] Trace[1274170684]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:b6e20855-4cba-43c3-b5ad-24b1cb6b977a,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.586) (total time: 7100ms):
2024-06-29T16:44:01.687005185+03:00 Trace[1274170684]: ["GuaranteedUpdate etcd3" audit-id:b6e20855-4cba-43c3-b5ad-24b1cb6b977a,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7099ms (13:43:54.587)]
2024-06-29T16:44:01.687009285+03:00 Trace[1274170684]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:5ad5bf54-356e-40b1-b972-0aeab3337b89 7096ms (13:43:54.590)
2024-06-29T16:44:01.687012485+03:00 Trace[1274170684]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7095ms (13:43:54.591)]]
Trace[1274170684]: [7.100171043s] [7.100171043s] END
2024-06-29T16:44:01.687093486+03:00 I0629 13:44:01.686848       1 trace.go:236] Trace[1416997392]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:aa560905-158b-47b0-a0cb-b5abf934f71f,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.585) (total time: 7101ms):
2024-06-29T16:44:01.687106686+03:00 Trace[1416997392]: ["GuaranteedUpdate etcd3" audit-id:aa560905-158b-47b0-a0cb-b5abf934f71f,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7100ms (13:43:54.586)]
2024-06-29T16:44:01.687110986+03:00 Trace[1416997392]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:b54ccc77-806d-4c94-aba5-8acd80d0775c 7097ms (13:43:54.589)
2024-06-29T16:44:01.687115087+03:00 Trace[1416997392]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7096ms (13:43:54.590)]]
2024-06-29T16:44:01.687132887+03:00 Trace[1416997392]: [7.10122676s] [7.10122676s] END
I0629 13:44:01.687310       1 trace.go:236] Trace[244916747]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:e43d3bba-ab01-476d-8c36-ddca1e9f962b,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.586) (total time: 7100ms):
2024-06-29T16:44:01.687436192+03:00 Trace[244916747]: ["GuaranteedUpdate etcd3" audit-id:e43d3bba-ab01-476d-8c36-ddca1e9f962b,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7100ms (13:43:54.587)]
2024-06-29T16:44:01.687448192+03:00 Trace[244916747]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:421e7053-14a0-4a24-bb40-add20620f7fb 7097ms (13:43:54.589)
Trace[244916747]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7097ms (13:43:54.590)]]
2024-06-29T16:44:01.687454892+03:00 Trace[244916747]: [7.100865555s] [7.100865555s] END
2024-06-29T16:44:01.687458492+03:00 I0629 13:44:01.687355       1 trace.go:236] Trace[2027599660]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:f8fedb04-19ab-464b-b410-1dffc91673a3,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-tpxpz,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-tpxpz,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.585) (total time: 7101ms):
2024-06-29T16:44:01.687461492+03:00 Trace[2027599660]: ["GuaranteedUpdate etcd3" audit-id:f8fedb04-19ab-464b-b410-1dffc91673a3,key:/management.cattle.io/projects/local/p-tpxpz,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7100ms (13:43:54.586)]
2024-06-29T16:44:01.687464792+03:00 Trace[2027599660]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:6ae8fc07-623d-49ca-96db-9a8f16e996c2 7098ms (13:43:54.588)
2024-06-29T16:44:01.687467692+03:00 Trace[2027599660]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7098ms (13:43:54.589)]]
2024-06-29T16:44:01.687470392+03:00 Trace[2027599660]: [7.101615966s] [7.101615966s] END
2024-06-29T16:44:01.687656195+03:00 I0629 13:44:01.687554       1 trace.go:236] Trace[1296692068]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:b53bfb5b-de1a-46e1-bdee-5a9d05ba48ff,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8
x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.586) (total time: 7100ms):
Trace[1296692068]: ["GuaranteedUpdate etcd3" audit-id:b53bfb5b-de1a-46e1-bdee-5a9d05ba48ff,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7099ms (13:43:54.587)]
Trace[1296692068]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:b44d8e60-6e96-4db8-a03e-9ca482a43942 7096ms (13:43:54.590)
2024-06-29T16:44:01.687668095+03:00 Trace[1296692068]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7096ms (13:43:54.590)]]
Trace[1296692068]: [7.100702952s] [7.100702952s] END
2024-06-29T16:44:01.687750497+03:00 I0629 13:44:01.687672       1 trace.go:236] Trace[983609095]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:2e8b8020-8a63-4c68-a07b-5529f79e6587,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:p-wfv8x,subresource:,namespace:local,protocol:HTTP/2.0,resource:projects,scope:resource,url:/apis/management.cattle.io/v3/namespaces/local/projects/p-wfv8x,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (29-Jun-2024 13:43:54.586) (total time: 7101ms):
2024-06-29T16:44:01.687756797+03:00 Trace[983609095]: ["GuaranteedUpdate etcd3" audit-id:2e8b8020-8a63-4c68-a07b-5529f79e6587,key:/management.cattle.io/projects/local/p-wfv8x,type:*unstructured.Unstructured,resource:projects.management.cattle.io 7100ms (13:43:54.587)]
2024-06-29T16:44:01.687782497+03:00 Trace[983609095]: ["Call validating webhook" configuration:rancher.cattle.io,webhook:rancher.cattle.io.projects.management.cattle.io,resource:management.cattle.io/v3, Resource=projects,subresource:,operation:UPDATE,UID:a633182d-bbb9-44a0-947c-b0f77cb74ef0 7097ms (13:43:54.589)
Trace[983609095]:  ["Proxy via http_connect protocol over tcp" address:10.42.4.16:9443 7097ms (13:43:54.590)]]
2024-06-29T16:44:01.687789497+03:00 Trace[983609095]: [7.101374362s] [7.101374362s] END
2024-06-29T16:44:01.702355026+03:00 I0629 13:44:01.702241       1 controller.go:615] quota admission added evaluator for: projects.management.cattle.io
I0629 13:44:05.020815       1 controller.go:615] quota admission added evaluator for: replicas.longhorn.io
I0629 13:44:05.160592       1 controller.go:615] quota admission added evaluator for: engines.longhorn.io
2024-06-29T16:44:05.315099631+03:00 I0629 13:44:05.314993       1 controller.go:615] quota admission added evaluator for: volumes.longhorn.io
2024-06-29T16:44:07.679868348+03:00 I0629 13:44:07.679653       1 controller.go:615] quota admission added evaluator for: poddisruptionbudgets.policy
I0629 13:44:07.679653       1 controller.go:615] quota admission added evaluator for: poddisruptionbudgets.policy
2024-06-29T16:44:07.679893048+03:00 I0629 13:44:07.679653       1 controller.go:615] quota admission added evaluator for: poddisruptionbudgets.policy
E0629 13:44:09.191953       1 handler_proxy.go:137] error resolving kube-system/rke2-metrics-server: no endpoints available for service "rke2-metrics-server"
2024-06-29T16:44:09.192150685+03:00 E0629 13:44:09.191977       1 handler_proxy.go:137] error resolving cattle-monitoring-system/rancher-monitoring-prometheus-adapter: no endpoints available for service "rancher-monitoring-prometheus-adapter"
2024-06-29T16:44:56.548568085+03:00 I0629 13:44:56.548391       1 trace.go:236] Trace[27991602]: "Get" accept:,audit-id:a36c994f-bbcf-403c-a62e-e28977b7fdd2,client:10.0.0.1,api-group:,api-version:v1,name:rke2-metrics-server-c9dcd6cc7-wsb5c,subresource:log,namespace:kube-system,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/rke2-metrics-server-c9dcd6cc7-wsb5c/log,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36,verb:CONNECT (29-Jun-2024 13:44:54.979) (total time: 1569ms):
2024-06-29T16:44:56.548620086+03:00 Trace[27991602]: ---"Writing http response done" 1565ms (13:44:56.548)
Trace[27991602]: [1.569176529s] [1.569176529s] END
2024-06-29T16:45:08.348671898+03:00 I0629 13:45:08.348561       1 controller.go:615] quota admission added evaluator for: snapshots.longhorn.io
I0629 13:45:08.353963       1 controller.go:615] quota admission added evaluator for: volumeattachments.longhorn.io
2024-06-29T16:45:09.192528243+03:00 E0629 13:45:09.192394       1 handler_proxy.go:137] error resolving kube-system/rke2-metrics-server: no endpoints available for service "rke2-metrics-server"
2024-06-29T16:45:09.192567244+03:00 E0629 13:45:09.192452       1 handler_proxy.go:137] error resolving cattle-monitoring-system/rancher-monitoring-prometheus-adapter: no endpoints available for service "rancher-monitoring-prometheus-adapter"
2024-06-29T16:45:20.465205177+03:00 I0629 13:45:20.465092       1 handler.go:286] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T16:45:21.699563652+03:00 W0629 13:45:21.699419       1 aggregator.go:166] failed to download v1beta1.custom.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
W0629 13:45:22.905244       1 aggregator.go:166] failed to download v1beta1.custom.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
W0629 13:45:38.177955       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:45:38.178335700+03:00 E0629 13:45:38.178052       1 controller.go:113] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
I0629 13:45:38.178157       1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
2024-06-29T16:45:38.179158213+03:00 W0629 13:45:38.179088       1 handler_proxy.go:93] no RequestInfo found in the context
E0629 13:45:38.179243       1 controller.go:102] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: failed to download v1beta1.custom.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
2024-06-29T16:45:38.179318916+03:00 , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-06-29T16:45:38.179324316+03:00 I0629 13:45:38.179260       1 controller.go:109] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
2024-06-29T16:45:54.320650768+03:00 E0629 13:45:54.320536       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
2024-06-29T16:45:54.343556328+03:00 I0629 13:45:54.343442       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0629 13:46:09.155972       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0629 13:48:09.267050       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T16:48:09.267855982+03:00 E0629 13:48:09.267766       1 writers.go:68] error encountered while streaming results via websocket: use of closed network connection
I0629 13:48:09.268019       1 trace.go:236] Trace[1548674291]: "Get" accept:,audit-id:8223336c-03e6-4551-a1cd-3be6af76f732,client:10.0.0.1,api-group:,api-version:v1,name:kube-apiserver-rke21,subresource:log,namespace:kube-system,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/kube-apiserver-rke21/log,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36,verb:CONNECT (29-Jun-2024 13:46:49.216) (total time: 80051ms):
Trace[1548674291]: ---"Writing http response done" 80046ms (13:48:09.267)
2024-06-29T16:48:09.268138787+03:00 Trace[1548674291]: [1m20.051471978s] [1m20.051471978s] END
2024-06-29T16:50:09.253365860+03:00 I0629 13:50:09.253226       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T16:50:09.253981769+03:00 E0629 13:50:09.253867       1 writers.go:68] error encountered while streaming results via websocket: use of closed network connection
I0629 13:50:09.253995       1 trace.go:236] Trace[12298559]: "Get" accept:,audit-id:59d0698c-3d1b-4f1b-8924-1329b89a99e5,client:10.0.0.1,api-group:,api-version:v1,name:kube-apiserver-rke21,subresource:log,namespace:kube-system,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/kube-apiserver-rke21/log,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36,verb:CONNECT (29-Jun-2024 13:47:04.509) (total time: 184744ms):
2024-06-29T16:50:09.254100771+03:00 Trace[12298559]: ---"Writing http response done" 184738ms (13:50:09.253)
Trace[12298559]: [3m4.744911033s] [3m4.744911033s] END
I0629 13:52:09.273705       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0629 13:54:09.253458       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T16:54:25.175606683+03:00 W0629 13:54:25.175456       1 handler_proxy.go:93] no RequestInfo found in the context
2024-06-29T16:54:25.175700584+03:00 E0629 13:54:25.175538       1 controller.go:146] Error updating APIService "v1beta1.metrics.k8s.io" with err: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
2024-06-29T16:54:25.175705284+03:00 , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-06-29T16:54:25.175717884+03:00 E0629 13:54:25.175597       1 handler_proxy.go:137] error resolving kube-system/rke2-metrics-server: no endpoints available for service "rke2-metrics-server"
2024-06-29T16:54:25.175767785+03:00 I0629 13:54:25.175700       1 handler.go:286] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
E0629 13:54:25.184816       1 writers.go:68] error encountered while streaming results via websocket: use of closed network connection
2024-06-29T16:54:25.185123632+03:00 I0629 13:54:25.184935       1 trace.go:236] Trace[912262168]: "Get" accept:,audit-id:bb32bdc6-970f-4285-8786-84bc12316673,client:10.0.0.1,api-group:,api-version:v1,name:rke2-metrics-server-c9dcd6cc7-wsb5c,subresource:log,namespace:kube-system,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/rke2-metrics-server-c9dcd6cc7-wsb5c/log,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36,verb:CONNECT (29-Jun-2024 13:49:28.005) (total time: 297179ms):
2024-06-29T16:54:25.185136632+03:00 Trace[912262168]: ---"Writing http response done" 297175ms (13:54:25.184)
Trace[912262168]: [4m57.179187086s] [4m57.179187086s] END
2024-06-29T16:54:25.185990946+03:00 E0629 13:54:25.185913       1 available_controller.go:460] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
I0629 13:54:55.605745       1 handler.go:286] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
I0629 13:55:09.248679       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0629 13:57:09.251975       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0629 13:59:09.252926       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T17:01:09.232664611+03:00 I0629 14:01:09.232514       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
W0629 14:01:23.641377       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-06-29T17:02:26.817322770+03:00 W0629 14:02:26.817165       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-06-29T17:03:09.253594544+03:00 I0629 14:03:09.253445       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
W0629 14:03:28.676428       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
W0629 14:04:02.870467       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-06-29T17:04:09.254384209+03:00 I0629 14:04:09.254241       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
E0629 14:04:39.253338       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}: context canceled
E0629 14:04:39.254578       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}: context canceled
E0629 14:04:39.255247       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}: context canceled
E0629 14:04:39.255470       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}: context canceled
2024-06-29T17:04:39.255816507+03:00 E0629 14:04:39.255693       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}: context canceled
I0629 14:06:09.178643       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0629 14:08:09.264563       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0629 14:10:09.265939       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0629 14:11:35.861611       1 controller.go:615] quota admission added evaluator for: deployments.apps
I0629 14:11:35.905862       1 controller.go:615] quota admission added evaluator for: issuers.cert-manager.io
2024-06-29T17:12:09.265796007+03:00 I0629 14:12:09.265657       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
W0629 14:12:23.369980       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
W0629 14:12:58.554029       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-06-29T17:14:09.258182666+03:00 I0629 14:14:09.258037       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T17:15:46.317847087+03:00 W0629 14:15:46.317715       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-06-29T17:16:09.202921802+03:00 I0629 14:16:09.202680       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T17:17:09.264276718+03:00 I0629 14:17:09.264133       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T17:18:19.597405009+03:00 W0629 14:18:19.597265       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
I0629 14:19:09.259311       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0629 14:21:09.211533       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-06-29T17:23:09.260689873+03:00 I0629 14:23:09.260556       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
W0629 14:23:35.137568       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-06-29T17:25:09.262233301+03:00 I0629 14:25:09.262086       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0629 14:27:09.261188       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
W0629 14:27:31.198238       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-06-29T17:28:09.263713480+03:00 I0629 14:28:09.263579       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
W0629 14:28:36.972550       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-06-29T17:30:09.255306452+03:00 I0629 14:30:09.255162       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager`

[brandond]

error resolving cattle-monitoring-system/rancher-monitoring-prometheus-adapter: no endpoints available for service "rancher-monitoring-prometheus-adapter
error resolving kube-system/rke2-metrics-server: no endpoints available for service "rke2-metrics-server"

There are no ready pods for these services. Why are the pods not running?

[cengizhanR]

Hi @brandond, thank you for your interest and return.
I restore etcd and getting same error and Rancher monitoring and Rancher Webhook is runining in my cluster, I will share pictures and thier logs too.
Kube-apiserver logs:

`
Container: kube-apiserver
Filter
Connected
W0701 13:26:05.041750       1 handler_proxy.go:93] no RequestInfo found in the context
2024-07-01T16:26:05.041878997+03:00 E0701 13:26:05.041784       1 controller.go:113] loading OpenAPI spec for "v1beta1.custom.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
2024-07-01T16:26:05.041883597+03:00 I0701 13:26:05.041746       1 controller.go:109] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
2024-07-01T16:26:05.043035415+03:00 I0701 13:26:05.042945       1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.custom.metrics.k8s.io: Rate Limited Requeue.
2024-07-01T16:26:06.186533933+03:00 E0701 13:26:06.186398       1 handler_proxy.go:137] error resolving cattle-monitoring-system/rancher-monitoring-prometheus-adapter: no endpoints available for service "rancher-monitoring-prometheus-adapter"
2024-07-01T16:26:07.854874791+03:00 E0701 13:26:07.854579       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
2024-07-01T16:26:08.275862841+03:00 I0701 13:26:08.275732       1 trace.go:236] Trace[518961721]: "Proxy via http_connect protocol over tcp" address:10.42.3.10:6443 (01-Jul-2024 13:26:01.185) (total time: 7090ms):
2024-07-01T16:26:08.275897341+03:00 Trace[518961721]: [7.090302321s] [7.090302321s] END
I0701 13:26:09.940535       1 trace.go:236] Trace[295829371]: "Proxy via http_connect protocol over tcp" address:10.42.3.10:6443 (01-Jul-2024 13:26:02.850) (total time: 7090ms):
Trace[295829371]: [7.090411822s] [7.090411822s] END
2024-07-01T16:26:09.941076151+03:00 I0701 13:26:09.940657       1 trace.go:236] Trace[1773236773]: "Proxy via http_connect protocol over tcp" address:10.42.3.10:6443 (01-Jul-2024 13:26:02.850) (total time: 7090ms):
Trace[1773236773]: [7.090624425s] [7.090624425s] END
2024-07-01T16:26:09.941081051+03:00 I0701 13:26:09.940737       1 trace.go:236] Trace[1914615679]: "Proxy via http_connect protocol over tcp" address:10.42.3.10:6443 (01-Jul-2024 13:26:02.849) (total time: 7090ms):
2024-07-01T16:26:09.941083451+03:00 Trace[1914615679]: [7.090773428s] [7.090773428s] END
2024-07-01T16:26:09.941085951+03:00 I0701 13:26:09.940756       1 trace.go:236] Trace[1047191158]: "Proxy via http_connect protocol over tcp" address:10.42.3.10:6443 (01-Jul-2024 13:26:02.849) (total time: 7090ms):
2024-07-01T16:26:09.941088051+03:00 Trace[1047191158]: [7.090750628s] [7.090750628s] END
2024-07-01T16:26:09.941097352+03:00 I0701 13:26:09.940890       1 trace.go:236] Trace[2101513089]: "Proxy via http_connect protocol over tcp" address:10.42.3.10:6443 (01-Jul-2024 13:26:02.849) (total time: 7090ms):
Trace[2101513089]: [7.09090233s] [7.09090233s] END
2024-07-01T16:26:10.174889833+03:00 I0701 13:26:10.174767       1 controller.go:615] quota admission added evaluator for: addons.k3s.cattle.io
2024-07-01T16:26:10.255237364+03:00 I0701 13:26:10.255092       1 controller.go:615] quota admission added evaluator for: helmchartconfigs.helm.cattle.io
2024-07-01T16:26:10.598109817+03:00 W0701 13:26:10.597934       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.secrets: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s": no endpoints available for service "rancher-webhook"
W0701 13:26:10.629735       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.secrets: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s": no endpoints available for service "rancher-webhook"
2024-07-01T16:26:10.863637985+03:00 W0701 13:26:10.863507       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.secrets: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s": no endpoints available for service "rancher-webhook"
2024-07-01T16:26:10.972809557+03:00 W0701 13:26:10.972667       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.secrets: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s": no endpoints available for service "rancher-webhook"
2024-07-01T16:26:11.131648790+03:00 W0701 13:26:11.131501       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.secrets: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=15s": no endpoints available for service "rancher-webhook"
2024-07-01T16:26:25.052014045+03:00 I0701 13:26:25.049925       1 trace.go:236] Trace[579377030]: "Patch" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:125b0545-1181-4b13-8196-64b33a3a6738,client:127.0.0.1,api-group:,api-version:v1,name:rke2-canal-67pb9,subresource:status,namespace:kube-system,protocol:HTTP/2.0,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/rke2-canal-67pb9/status,user-agent:kubelet/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b,verb:PATCH (01-Jul-2024 13:26:24.474) (total time: 575ms
):
Trace[579377030]: ["GuaranteedUpdate etcd3" audit-id:125b0545-1181-4b13-8196-64b33a3a6738,key:/pods/kube-system/rke2-canal-67pb9,type:*core.Pod,resource:pods 575ms (13:26:24.474)
2024-07-01T16:26:25.052081846+03:00 Trace[579377030]:  ---"Txn call completed" 567ms (13:26:25.048)]
2024-07-01T16:26:25.052085246+03:00 Trace[579377030]: ---"Object stored in database" 568ms (13:26:25.048)
2024-07-01T16:26:25.052088246+03:00 Trace[579377030]: [575.830721ms] [575.830721ms] END
2024-07-01T16:26:27.006183582+03:00 I0701 13:26:27.006071       1 trace.go:236] Trace[348997973]: "Patch" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:f269a80f-8ce1-480d-8249-e760538eb759,client:127.0.0.1,api-group:,api-version:v1,name:csi-provisioner-6cfd577457-9whsg,subresource:status,namespace:longhorn-system,protocol:HTTP/2.0,resource:pods,scope:resource,url:/api/v1/namespaces/longhorn-system/pods/csi-provisioner-6cfd577457-9whsg/status,user-agent:kubelet/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b,verb:PATCH (01-Jul-2024 13:26:26.500) (total time: 505ms):
2024-07-01T16:26:27.006208983+03:00 Trace[348997973]: ["GuaranteedUpdate etcd3" audit-id:f269a80f-8ce1-480d-8249-e760538eb759,key:/pods/longhorn-system/csi-provisioner-6cfd577457-9whsg,type:*core.Pod,resource:pods 505ms (13:26:26.500)
2024-07-01T16:26:27.006212983+03:00 Trace[348997973]:  ---"Txn call completed" 501ms (13:26:27.005)]
2024-07-01T16:26:27.006216583+03:00 Trace[348997973]: ---"Object stored in database" 501ms (13:26:27.005)
2024-07-01T16:26:27.006226683+03:00 Trace[348997973]: [505.913151ms] [505.913151ms] END
2024-07-01T16:26:27.027943616+03:00 I0701 13:26:27.027726       1 trace.go:236] Trace[2000318347]: "Call conversion webhook" custom-resource-definition:nodes.longhorn.io,desired-api-version:longhorn.io/v1beta1,object-count:1,UID:2c08159d-9f05-40d9-8e11-5eb20bba3137 (01-Jul-2024 13:26:19.898) (total time: 7129ms):
2024-07-01T16:26:27.027961616+03:00 Trace[2000318347]: ["Proxy via http_connect protocol over tcp"
 address:10.42.2.7:9501 7128ms (13:26:19.899)]
2024-07-01T16:26:27.027965616+03:00 Trace[2000318347]: [7.12939362s] [7.12939362s] END
2024-07-01T16:26:27.028009217+03:00 I0701 13:26:27.027937       1 trace.go:236] Trace[505007152]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/nodes,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Jul-2024 13:26:19.896) (total time: 7131ms):
2024-07-01T16:26:27.028014417+03:00 Trace[505007152]: [7.131739155s] [7.131739155s] END
2024-07-01T16:26:27.028027317+03:00 W0701 13:26:27.027985       1 reflector.go:547] storage/cacher.go:/longhorn.io/nodes: failed to list longhorn.io/v1beta1, Kind=Node: conversion webhook for longhorn.io/v1beta2, Kind=Node failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.2.7:9501, code 502: 502 Bad Gateway
2024-07-01T16:26:27.028033917+03:00 E0701 13:26:27.028004       1 cacher.go:475] cacher (nodes.longhorn.io): unexpected ListAndWatch error: failed to list longhorn.io/v1beta1, Kind=Node: conversion webhook for longhorn.io/v1beta2, Kind=Node failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.2.7:9501, code 502: 502 Bad Gateway; reinitializing...
2024-07-01T16:26:27.091957196+03:00 I0701 13:26:27.091827       1 trace.go:236] Trace[1411966677]: "Call conversion webhook" custom-resource-definition:backuptargets.longhorn.io,desired-api-version:longhorn.io/v1beta1,object-count:1,UID:ee023f73-c0e1-4b6f-a120-45950799b4ad (01-Jul-2024 13:26:19.993) (total time: 7098ms):
2024-07-01T16:26:27.091988297+03:00 Trace[1411966677]: ["Proxy via http_connect protocol over tcp" address:10.42.1.6:9501 7098ms (13:26:19.993)]
2024-07-01T16:26:27.091991797+03:00 Trace[1411966677]: [7.098642849s] [7.098642849s] END
2024-07-01T16:26:27.091995397+03:00 I0701 13:26:27.091893       1 trace.go:236] Trace[358635160]: "List(recursive=true
) etcd3" audit-id:,key:/longhorn.io/backuptargets,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Jul-2024 13:26:19.990) (total time: 7101ms):
2024-07-01T16:26:27.091997597+03:00 Trace[358635160]: [7.101414891s] [7.101414891s] END
2024-07-01T16:26:27.092014397+03:00 W0701 13:26:27.091928       1 reflector.go:547] storage/cacher.go:/longhorn.io/backuptargets: failed to list longhorn.io/v1beta1, Kind=BackupTarget: conversion webhook for longhorn.io/v1beta2, Kind=BackupTarget failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.1.6:9501, code 502: 502 Bad Gateway
E0701 13:26:27.091948       1 cacher.go:475] cacher (backuptargets.longhorn.io): unexpected ListAndWatch error: failed to list longhorn.io/v1beta1, Kind=BackupTarget: conversion webhook for longhorn.io/v1beta2, Kind=BackupTarget failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.1.6:9501, code 502: 502 Bad Gateway; reinitializing...
2024-07-01T16:26:27.156690388+03:00 I0701 13:26:27.156466       1 trace.go:236] Trace[1336269757]: "Call conversion webhook" custom-resource-definition:engineimages.longhorn.io,desired-api-version:longhorn.io/v1beta1,object-count:1,UID:b3659902-7d29-4bb1-bae0-c8639dfb0da8 (01-Jul-2024 13:26:20.069) (total time: 7086ms):
2024-07-01T16:26:27.156724689+03:00 Trace[1336269757]: ["Proxy via http_connect protocol over tcp" address:10.42.1.6:9501 7086ms (13:26:20.070)]
2024-07-01T16:26:27.156729289+03:00 Trace[1336269757]: [7.086684865s] [7.086684865s] END
2024-07-01T16:26:27.157177496+03:00 I0701 13:26:27.156551       1 trace.go:236] Trace[2044528698]: "Call conversion webhook" custom-resource-definition:volumes.longhorn.io,desired-api-version:longhorn.io/v1beta1,object-count:1,UID:a416e960-bf16-4fc7-b3ef-2ac157f4d888 (01-Jul-2024 13:26:20.022) (total time: 7133ms):
2024-07-01T16:26:27.157222496+03:00 Trace[2044528698]: ["Proxy via http_connect protocol over tcp" address:10.42.1.6:9501 7133ms (13:26:20.023)]
2024-07-01T16:26:27.157232496+03:00 Trace[2044528698]: [7.133893189s] [7.133893189s] END
2024-07-01T16:26:27.157239697+03:00 I0701 13:26:27.156782       1 trace.go:236] Trace[422974506]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/engineimages,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Jul-2024 13:26:20.066) (total time: 7089ms):
2024-07-01T16:26:27.157246797+03:00 Trace[422974506]: [7.089895715s] [7.089895715s] END
2024-07-01T16:26:27.157274197+03:00 W0701 13:26:27.157111       1 reflector.go:547] storage/c
acher.go:/longhorn.io/engineimages: failed to list longhorn.io/v1beta1, Kind=EngineImage: conversion webhook for longhorn.io/v1beta2, Kind=EngineImage failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.1.6:9501, code 502: 502 Bad Gateway
2024-07-01T16:26:27.157284097+03:00 E0701 13:26:27.157173       1 cacher.go:475] cacher (engineimages.longhorn.io): unexpected ListAndWatch error: failed to list longhorn.io/v1beta1, Kind=EngineImage: conversion webhook for longhorn.io/v1beta2, Kind=EngineImage failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.1.6:9501, code 502: 502 Bad Gateway; reinitializing...
2024-07-01T16:26:27.157292397+03:00 I0701 13:26:27.157131       1 trace.go:236] Trace[1050093478]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/volumes,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Jul-2024 13:26:20.019) (total time: 7137ms):
2024-07-01T16:26:27.157299397+03:00 Trace[1050093478]: [7.137679246s] [7.137679246s] END
W0701 13:26:27.157269       1 reflector.go:547] storage/cacher.go:/longhorn.io/volumes: failed to list longhorn.io/v1beta1, Kind=Volume: conversion webhook for longhorn.io/v1beta2, Kind=Volume failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.1.6:9501, code 502: 502 Bad Gateway
2024-07-01T16:26:27.157393799+03:00 E0701 13:26:27.157312       1 cacher.go:475] cacher (volumes.longhorn.io): unexpected ListAndWatch error: failed to list longhorn.io/v1beta1, Kind=Volume: conversion webhook for longhorn.io/v1beta2, Kind=Volume failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.1.6:9501, code 502: 502 Bad Gateway; reinitializing...
2024-07-01T16:26:32.863928421+03:00 E0701 13:26:32.863812       1 handler_proxy.go:137] error resolving cattle-monitoring-system/rancher-monitoring-prometheus-adapter: no endpoints available for service "rancher-monitoring-prometheus-adapter"
2024-07-01T16:26:32.864030222+03:00 E0701 13:26:32.863881       1 handler_proxy.go:137] error resolving kube-system/rke2-metrics-server: no endpoints available for service "rke2-metrics-server"
2024-07-01T16:26:33.860406387+03:00 W0701 13:26:33.860280       1 handler_proxy.go:93] no RequestInfo found in the context
2024-07-01T16:26:33.860459387+03:00 E0701 13:26:33.860384       1 controller.go:102] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to download v1beta1.metrics.k8s.io: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
2024-07-01T16:26:33.860472888+03:00 , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
2024-07-01T16:26:33.860476688+03:00 I0701 13:26:33.860398       1 controller.go:109] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
2024-07-01T
16:26:33.955443843+03:00 W0701 13:26:33.955321       1 handler_proxy.go:93] no RequestInfo found in the context
2024-07-01T16:26:33.955474943+03:00 E0701 13:26:33.955367       1 controller.go:113] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: Error, could not get list of group versions for APIService
2024-07-01T16:26:33.955477543+03:00 I0701 13:26:33.955377       1 controller.go:126] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
2024-07-01T16:26:35.156041935+03:00 I0701 13:26:35.155904       1 trace.go:236] Trace[1593364189]: "Call conversion webhook" custom-resource-definition:nodes.longhorn.io,desired-api-version:longhorn.io/v1beta1,object-count:1,UID:74590745-8075-46d1-b791-8792b708e6fb (01-Jul-2024 13:26:28.030) (total time: 7125ms):
2024-07-01T16:26:35.156098836+03:00 Trace[1593364189]: ["Proxy via http_connect protocol over tcp" address:10.42.2.7:9501 7124ms (13:26:28.031)]
2024-07-01T16:26:35.156108336+03:00 Trace[1593364189]: [7.125161955s] [7.125161955s] END
2024-07-01T16:26:35.156111836+03:00 I0701 13:26:35.155949       1 trace.go:236] Trace[674556275]: "List(recursive=true) etcd3" audit-id:,key:/longhorn.io/nodes,resourceVersion:,resourceVersionMatch:,limit:10000,continue: (01-Jul-2024 13:26:28.028) (total time: 7127ms):
2024-07-01T16:26:35.156113836+03:00 Trace[674556275]: [7.127618993s] [7.127618993s] END
2024-07-01T16:26:35.156130137+03:00 W0701 13:26:35.155981       1 reflector.go:547] storage/cacher.go:/longhorn.io/nodes: failed to list longhorn.io/v1beta1, Kind=Node: conversion webhook for longhorn.io/v1beta2, Kind=Node failed: Post "https://longhorn-conversion-webhook.longhorn-system.svc:9501/v1/webhook/conversion?timeout=30s": proxy error from 127.0.0.1:9345 while dialing 10.42.2.7:9501, code 502: 502 Bad Gateway
2024-07-01T16:26:35.156133737+03:00 E0701 13:26:35.155999       1 cacher.go:475] cacher (nodes.longhorn.io): unexpected ListAndWatch error: failed to list longhorn.io/v1beta1, Kind=Node: conversion webhook for longhorn.io/
v1beta
2024-07-01T16:27:42.367914897+03:00 W0701 13:27:42.367831       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:42.369962128+03:00 W0701 13:27:42.369880       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:42.379919181+03:00 W0701 13:27:42.379827       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:42.383393034+03:00 W0701 13:27:42.383124       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.manage
ment.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:42.383406034+03:00 W0701 13:27:42.383281       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:42.383408034+03:00 W0701 13:27:42.383317       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:42.384836656+03:00 W0701 13:27:42.384719       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:42.385368664+03:00 W0701 13:27:42.385284       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:42.390906549+03:00 W0701 13:27:42.390814       1 disp
atcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:42.391794763+03:00 W0701 13:27:42.391716       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:42.392705576+03:00 W0701 13:27:42.392630       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.projects.management.cattle.io: failed calling webhook "rancher.cattle.io.projects.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/projects.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:42.476808765+03:00 W0701 13:27:42.476720       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:42.576823697+03:00 W0701 13:27:42.576713       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.
management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:45.876954754+03:00 W0701 13:27:45.876824       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:45.976803284+03:00 W0701 13:27:45.976651       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
W0701 13:27:46.076815       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:46.177757862+03:00 W0701 13:27:46.177490       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:46.277472390+03:00 W0701 13:27:46.277307       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:46.376843612+03:00 W0701 13:27:46.376696       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
W0701 13:27:46.379184       1 aggregator.go:166] failed to download v1beta1.custom.metrics.k8s.io: failed to retrieve openAPI 
2024-07-01T16:27:56.476615237+03:00 W0701 13:27:56.476460       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:56.577088876+03:00 W0701 13:27:56.576928       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:56.676354697+03:00 W0701 13:27:56.676215       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "r
ancher-webhook"
W0701 13:27:56.777330       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:56.877211574+03:00 W0701 13:27:56.877121       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:56.977496610+03:00 W0701 13:27:56.977364       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:57.078422956+03:00 W0701 13:27:57.078279       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:57.177854580+03:00 W0701 13:27:57.177631       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://ran
cher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:57.276866396+03:00 W0701 13:27:57.276729       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
W0701 13:27:57.376190       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:57.476923561+03:00 W0701 13:27:57.476764       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:57.577239998+03:00 W0701 13:27:57.577088       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:57.677281431+03:00 W0701 13:27:57.677115       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:57.777615168+03:00 W0701 13:27:57.777474       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://ran
cher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
E0701 13:27:58.985083       1 dispatcher.go:213] failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:58.997465427+03:00 W0701 13:27:58.997373       1 dispatcher.go:205] Failed calling webhook, failing open rancher.cattle.io.features.management.cattle.io: failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:58.997480527+03:00 E0701 13:27:58.997406       1 dispatcher.go:213] failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:59.004915344+03:00 W0701 13:27:59.004826       1 dispatcher.go:205] Failed calling webhook, failing open rancher.cattle.io.features.management.cattle.io: failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
E0701 13:27:59.004855       1 dispatcher.go:213] failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
W0701 13:27:59.010984       1 dispatcher.go:205] Failed calling webhook, failing open rancher.cattle.io.features.management.cattle.io: failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:59.011095341+03:00 E0701 13:27:59.011015       1 dispatcher.go:213] failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:59.021414703+03:00 W0701 13:27:59.020809       1 dispatcher.go:205] Failed calling webhook, failing open rancher.cattle.io.features.management.cattle.io: failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:59.021430003+03:00 E0701 13:27:59.020843       1 dispatcher.go:213] failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
W0701 13:27:59.030090       1 dispatcher.go:205] Failed calling webhook, failing open rancher.cattle.io.features.management.cattle.io: failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:59.030209941+03:00 E0701 13:27:59.030152       1 dispatcher.go:213] failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:59.044231262+03:00 W0701 13:27:59.044143       1 dispatcher.go:205] Failed calling webhook, failing open rancher.cattle.io.features.management.cattle.io: failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:27:59.044247262+03:00 E0701 13:27:59.044177       1 dispatcher.go:213] failed calling webhook "rancher.cattle.io.features.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/features.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
I0701 13:28:13.855882       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 13:28:14.090366       1 controller.go:615] quota admission added evaluator for: replicasets.apps
2024-07-01T16:28:14.832821818+03:00 I0701 13:28:14.832702       1 trace.go:236] Trace[2035208871]: "List" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:53705152-ff45-4fcd-a766-3d06f5214559,client:10.0.0.6,api-group:apiextensions.k8s.io,api-version:v1,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:customresourcedefinitions,scope:cluster,url:/apis/apiextensions.k8s.io/v1/customresourcedefinitions,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (01-Jul-2024 13:28:14.258) (total time: 574ms):
2024-07-01T16:28:14.832841718+03:00 Trace[2035208871]: ---"Writing http response done" count:171 216ms (13:28:14.832)
Trace[2035208871]: [574.241819ms] [574.241819ms] END
2024-07-01T16:28:14.883340512+03:00 I0701 13:28:14.883210       1 trace.go:236] Trace[401408932]: "List" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:ffcbae6e-e090-4266-8931-ddf2d1559c46,client:10.0.0.6,api-group:management.cattle.io,api-version:v3,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:rkeaddons,scope:cluster,url:/apis/management.cattle.io/v3/rkeaddons,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (01-Jul-2024 13:28:14.247) (total time: 635ms):
Trace[401408932]: ---"Writing http response done" count:96 293ms (13:28:14.883)
2024-07-01T16:28:14.883382612+03:00 Trace[401408932]: [635.550182ms] [635.550182ms] END
2024-07-01T16:28:21.371694610+03:00 I0701 13:28:21.371572       1 controller.go:615] quota admission added evaluator for: poddisruptionbudgets.policy
2024-07-01T16:28:29.013456522+03:00 I0701 13:28:29.013274       1 metrics.go:299] "Failed to get storage metrics" storage_cluster_id="etcd-0" err="context deadline exceeded"
2024-07-01T16:28:29.549818146+03:00 E0701 13:28:29.549711       1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
E0701 13:28:29.549751       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
E0701 13:28:29.549781       1 finisher.go:175] FinishRequest: post-timeout activity - time-elapsed: 9.4µs, panicked: false, err: context canceled, panic-reason: <nil>
2024-07-01T16:28:29.551003164+03:00 E0701 13:28:29.550914       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
2024-07-01T16:28:29.551106866+03:00 I0701 13:28:29.550990       1 trace.go:236] Trace[2121085333]: "Update" accept:application/vnd.kubernetes.protobuf, */*,audit-id:ba2fdf2e-b0f0-406d-a5aa-d9a9879eac78,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:kube-controller-manager,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:kube-controller-manager/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b/leader-election,verb:PUT (01-Jul-2024 13:28:24.550) (total time: 5000ms):
2024-07-01T16:28:29.551147466+03:00 Trace[2121085333]: ["GuaranteedUpdate etcd3" audit-id:ba2fdf2e-b0f0-406d-a5aa-d9a9879eac78,key:/leases/kube-system/kube-controller-manager,type:*coordination.Lease,resource:leases.coordination.k8s.io 5000ms (13:28:24.550)
2024-07-01T16:28:29.551151267+03:00 Trace[2121085333]:  ---"Txn call failed" err:context canceled 4998ms (13:28:29.549)]
Trace[2121085333]: [5.000577533s] [5.000577533s] END
2024-07-01T16:28:29.551165067+03:00 E0701 13:28:29.551090       1 timeout.go:142] post-timeout activity - time-elapsed: 1.29132ms, PUT "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager" result: <nil>
2024-07-01T16:28:30.083695830+03:00 I0701 13:28:30.083600       1 trace.go:236] Trace[1408237751]: "Delete" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:2579c5d2-4246-4fdf-a773-7cb1afe7bf62,client:127.0.0.1,api-group:apps,api-version:v1,name:fleet-agent-596955f45,subresource:,namespace:cattle-fleet-local-system,protocol:HTTP/2.0,resource:replicasets,scope:resource,url:/apis/apps/v1/namespaces/cattle-fleet-local-system/replicasets/fleet-agent-596955f45,user-agent:kube-controller-manager/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b/system:serviceaccount:kube-system:generic-garbage-collector,verb:DELETE (01-Jul-2024 13:28:24.354) (total time: 5729ms):
Trace[1408237751]: ---"Object deleted from database" 5729ms (13:28:30.083)
2024-07-01T16:28:30.083712130+03:00 Trace[1408237751]: [5.729221176s] [5.729221176s] END
2024-07-01T16:28:30.085161053+03:00 I0701 13:28:30.085032       1 trace.go:236] Trace[916545628]: "Update" accept:application/json, */*,audit-id:75650e69-d577-4c31-8880-62d02aaa5c88,client:10.0.0.6,api-group:,api-version:v1,name:alertmanager-rancher-monitoring-alertmanager-generated,subresource:,namespace:cattle-monitoring-system,protocol:HTTP/2.0,resource:secrets,scope:resource,url:/api/v1/namespaces/cattle-monitoring-system/secrets/alertmanager-rancher-monitoring-alertmanager-generated,user-agent:PrometheusOperator/0.65.1,verb:PUT (01-Jul-2024 13:28:24.358) (total time: 5726ms):
2024-07-01T16:28:30.085183353+03:00 Trace[916545628]: ["GuaranteedUpdate etcd3" audit-id:75650e69-d577-4c31-8880-62d02aaa5c88,key:/secrets/cattle-monitoring-system/alertmanager-rancher-monitoring-alertmanager-generated,type:*core.Secret,resource:secrets 5726ms (13:28:24.358)]
Trace[916545628]: [5.72689744s] [5.72689744s] END
I0701 13:28:30.085990       1 trace.go:236] Trace[1890275310]: "Update" accept:application/json, */*,audit-id:cdac06cd-5e1f-4741-8948-1d0b7de8323e,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:rke2,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2,user-agent:rke2-supervisor@rke22/v1.30.0+rke2r1 (linux/amd64) rke2/60e06c4dbccff996f717af8f4c532971f57264b4,verb:PUT (01-Jul-2024 13:28:24.578) (total time: 5506ms):
2024-07-01T16:28:30.086101068+03:00 Trace[1890275310]: ["GuaranteedUpdate etcd3" audit-id:cdac06cd-5e1f-4741-8948-1d0b7de8323e,key:/leases/kube-system/rke2,type:*coordination.Lease,resource:leases.coordination.k8s.io 5506ms (13:28:24.579)
2024-07-01T16:28:30.086104968+03:00 Trace[1890275310]:  ---"Txn call completed" 5505ms (13:28:30.085)]
2024-07-01T16:28:30.086133668+03:00 Trace[1890275310]: [5.506931885s] [5.506931885s] END
2024-07-01T16:28:30.090196032+03:00 I0701 13:28:30.087418       1 trace.go:236] Trace[965202379]: "Update" accept:application/json, */*,audit-id:638b4375-843b-4ea5-b265-1c2eb968f7a3,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:rke2-etcd,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-etcd,user-agent:rke2-supervisor@rke22/v1.30.0+rke2r1 (linux/amd64) rke2/60e06c4dbccff996f717af8f4c532971f57264b4,verb:PUT (01-Jul-2024 13:28:26.148) (total time: 3938ms):
2024-07-01T16:28:30.090206332+03:00 Trace[965202379]: ["GuaranteedUpdate etcd3" audit-id:638b4375-843b-4ea5-b265-1c2eb968f7a3,key:/leases/kube-system/rke2-etcd,type:*coordination.Lease,resource:leases.coordination.k8s.io 3938ms (13:28:26.148)
Trace[965202379]:  ---"Txn call completed" 3937ms (13:28:30.087)]
Trace[965202379]: [3.938770857s] [3.938770857s] END
2024-07-01T16:28:30.289626864+03:00 E0701 13:28:30.289215       1 status.go:71] apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded
2024-07-01T16:28:30.289637264+03:00 E0701 13:28:30.289320       1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
2024-07-01T16:28:30.290713681+03:00 E0701 13:28:30.290608       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
2024-07-01T16:28:30.290730382+03:00 E0701 13:28:30.290657       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
I0701 13:28:30.291732       1 trace.go:236] Trace[133879617]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:b3b2563f-faf3-4903-b678-5f99c71880d0,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:kube-scheduler,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler,user-agent:kube-scheduler/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b/leader-election,verb:GET (01-Jul-2024 13:28:25.289) (total time: 5001ms):
2024-07-01T16:28:30.291846099+03:00 Trace[133879617]: [5.001976455s] [5.001976455s] END
2024-07-01T16:28:30.292015102+03:00 E0701 13:28:30.291878       1 timeout.go:142] post-timeout activity - time-elapsed: 2.443838ms, GET "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result: <nil>
I0701 13:28:30.381168       1 trace.go:236] Trace[1637741352]: "Update" accept:application/vnd.kubernetes.protobuf, */*,audit-id:94ab654d-96c5-4757-9fc3-da30f80b1fc0,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:apiserver-vzadqw356hykimwwk4zcvw5eba,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/apiserver-vzadqw356hykimwwk4zcvw5eba,user-agent:kube-apiserver/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b,verb:PUT (01-Jul-2024 13:28:28.508) (total time: 1872ms):
2024-07-01T16:28:30.383135633+03:00 Trace[1637741352]: ["GuaranteedUpdate etcd3" audit-id:94ab654d-96c5-4757-9fc3-da30f80b1fc0,key:/leases/kube-system/apiserver-vzadqw356hykimwwk4zcvw5eba,type:*coordination.Lease,resource:leases.coordination.k8s.io 1872ms (13:28:28.508)
Trace[1637741352]:  ---"Txn call completed" 1872ms (13:28:30.380)]
2024-07-01T16:28:30.383141833+03:00 Trace[1637741352]: [1.872897114s] [1.872897114s] END
E0701 13:28:30.622301       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"context canceled"}: context canceled
2024-07-01T16:28:30.622485292+03:00 E0701 13:28:30.622377       1 writers.go:122] apiserver was unable to write a JSON response: http: Handler timeout
2024-07-01T16:28:30.624086517+03:00 E0701 13:28:30.623975       1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http: Handler timeout"}: http: Handler timeout
2024-07-01T16:28:30.624209319+03:00 E0701 13:28:30.624011       1 writers.go:135] apiserver was unable to write a fallback JSON response: http: Handler timeout
2024-07-01T16:28:30.625324036+03:00 I0701 13:28:30.625092       1 trace.go:236] Trace[1405655480]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:464b19e4-4df5-4b76-92b3-61439e2d2722,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:rke2-cloud-controller-manager,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager,user-agent:cloud-controller-manager/v1.29.3 (linux/amd64) kubernetes/d9f75b6/leader-election,verb:GET (01-Jul-2024 13:28:25.622) (total time: 5002ms):
Trace[1405655480]: [5.002232759s] [5.002232759s] END
2024-07-01T16:28:30.625449638+03:00 E0701 13:28:30.625377       1 timeout.go:142] post-timeout activity - time-elapsed: 3.023348ms, GET "/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager" result: <nil>
2024-07-01T16:28:30.646500369+03:00 I0701 13:28:30.646406       1 trace.go:236] Trace[796875325]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:84a2adbf-2854-4594-bbc3-aa8dd7c31199,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:kube-controller-manager,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:kube-controller-manager/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b/leader-election,verb:GET (01-Jul-2024 13:28:29.550) (total time: 1095ms):
Trace[796875325]: ---"About to write a response" 1095ms (13:28:30.646)
2024-07-01T16:28:30.646521669+03:00 Trace[796875325]: [1.095621507s] [1.095621507s] END
2024-07-01T16:28:30.646532569+03:00 I0701 13:28:30.646467       1 trace.go:236] Trace[880013760]: "Get" accept:application/json, */*,audit-id:e95e1adf-0775-42e3-b53e-369a2d82d8d4,client:10.0.0.5,api-group:coordination.k8s.io,api-version:v1,name:external-attacher-leader-driver-longhorn-io,subresource:,namespace:longhorn-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/longhorn-system/leases/external-attacher-leader-driver-longhorn-io,user-agent:csi-attacher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:27.910) (total time: 2736ms):
2024-07-01T16:28:30.646534969+03:00 Trace[880013760]: ---"About to write a response" 2735ms (13:28:30.646)
Trace[880013760]: [2.73606667s] [2.73606667s] END
I0701 13:28:30.646627       1 trace.go:236] Trace[1234306176]: "Get" accept:application/json, */*,audit-id:fced656a-5b23-4097-a07c-f1dbf6b691e0,client:10.0.0.6,api-group:coordination.k8s.io,api-version:v1,name:cert-manager-controller,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cert-manager-controller,user-agent:cert-manager-leader-election/v1.14.5 (linux/amd64) cert-manager/6a0915206f61a9116bb605e58e749c2aab2d43e2,verb:GET (01-Jul-2024 13:28:25.575) (total time: 5071ms):
2024-07-01T16:28:30.646725972+03:00 Trace[1234306176]: ---"About to write a response" 5071ms (13:28:30.646)
2024-07-01T16:28:30.646728572+03:00 Trace[1234306176]: [5.071398245s] [5.071398245s] END
I0701 13:28:30.646657       1 trace.go:236] Trace[138168740]: "Get" accept:application/json, */*,audit-id:23df3090-0783-44b3-88e4-3665f09ce49b,client:10.0.0.2,api-group:coordination.k8s.io,api-version:v1,name:external-snapshotter-leader-driver-longhorn-io,subresource:,namespace:longhorn-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/longhorn-system/leases/external-snapshotter-leader-driver-longhorn-io,user-agent:csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:28.071) (total time: 2575ms):
2024-07-01T16:28:30.646732972+03:00 Trace[138168740]: ---"About to write a response" 2575ms (13:28:30.646)
Trace[138168740]: [2.575569349s] [2.575569349s] END
2024-07-01T16:28:30.648513300+03:00 I0701 13:28:30.646477       1 trace.go:236] Trace[1931705648]: "Get" accept:application/json, */*,audit-id:87eeeab4-4c3c-4cb6-81ae-766ec11ffe3f,client:10.0.0.6,api-group:coordination.k8s.io,api-version:v1,name:cattle-controllers,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cattle-controllers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:25.606) (total time: 5039ms):
2024-07-01T16:28:30.648530601+03:00 Trace[1931705648]: ---"About to write a response" 5039ms (13:28:30.646)
2024-07-01T16:28:30.648533601+03:00 Trace[1931705648]: [5.039909451s] [5.039909451s] END
2024-07-01T16:28:30.648539801+03:00 I0701 13:28:30.646798       1 trace.go:236] Trace[1810772599]: "Get" accept:application/json, */*,audit-id:06362686-821e-4202-90c1-e67ed0beee15,client:10.0.0.6,api-group:coordination.k8s.io,api-version:v1,name:external-snapshotter-leader-driver-longhorn-io,subresource:,namespace:longhorn-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/longhorn-system/leases/external-snapshotter-leader-driver-longhorn-io,user-agent:csi-snapshotter/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:25.894) (total time: 4751ms):
Trace[1810772599]: ---"About to write a response" 4751ms (13:28:30.646)
Trace[1810772599]: [4.751804026s] [4.751804026s] END
2024-07-01T16:28:30.648553501+03:00 I0701 13:28:30.646943       1 trace.go:236] Trace[352601910]: "Get" accept:application/json, */*,audit-id:138c5c87-3ff0-4a66-ab1b-43832eb43013,client:10.0.0.6,api-group:coordination.k8s.io,api-version:v1,name:rke2-ingress-nginx-leader,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-ingress-nginx-leader,user-agent:nginx-ingress-controller/nginx-1.9.6-hardened1 (linux/amd64) ingress-nginx/git-05efe494a,verb:GET (01-Jul-2024 13:28:29.976) (total time: 670ms):
Trace[352601910]: ---"About to write a response" 670ms (13:28:30.646)
2024-07-01T16:28:30.648557801+03:00 Trace[352601910]: [670.49343ms] [670.49343ms] END
2024-07-01T16:28:30.648560101+03:00 I0701 13:28:30.646968       1 trace.go:236] Trace[1612765615]: "Get" accept:application/json, */*,audit-id:9a24c514-dcf4-4d96-a92f-04514b0a21f7,client:10.0.0.6,api-group:coordination.k8s.io,api-version:v1,name:external-resizer-driver-longhorn-io,subresource:,namespace:longhorn-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/longhorn-system/leases/external-resizer-driver-longhorn-io,user-agent:csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:29.544) (total time: 1102ms):
Trace[1612765615]: ---"About to write a response" 1102ms (13:28:30.646)
2024-07-01T16:28:30.648564101+03:00 Trace[1612765615]: [1.102739419s] [1.102739419s] END
2024-07-01T16:28:30.650334929+03:00 I0701 13:28:30.650231       1 trace.go:236] Trace[570955919]: "List" accept:application/json, */*,audit-id:69664986-0db1-40b8-abdf-7e3a4da4c960,client:10.0.0.6,api-group:,api-version:v1,name:,subresource:,namespace:cattle-monitoring-system,protocol:HTTP/2.0,resource:configmaps,scope:namespace,url:/api/v1/namespaces/cattle-monitoring-system/configmaps,user-agent:PrometheusOperator/0.65.1,verb:LIST (01-Jul-2024 13:28:24.474) (total time: 6175ms):
Trace[570955919]: ["List(recursive=true) etcd3" audit-id:69664986-0db1-40b8-abdf-7e3a4da4c960,key:/configmaps/cattle-monitoring-system,resourceVersion:,resourceVersionMatch:,limit:0,continue: 6175ms (13:28:24.474)]
2024-07-01T16:28:30.650350629+03:00 Trace[570955919]: [6.175964392s] [6.175964392s] END
2024-07-01T16:28:30.915161888+03:00 I0701 13:28:30.914843       1 trace.go:236] Trace[592743277]: "Get" accept:application/vnd.kubernetes.protobuf;as=PartialObjectMetadata;g=meta.k8s.io;v=v1,application/json;as=PartialObjectMetadata;g=meta.k8s.io;v=v1,application/json,audit-id:1b40b09c-cd25-4eef-a770-0f9a501204f4,client:127.0.0.1,api-group:,api-version:v1,name:fleet-agent-596955f45-bng97,subresource:,namespace:cattle-fleet-local-system,protocol:HTTP/2.0,resource:pods,scope:resource,url:/api/v1/namespaces/cattle-fleet-local-system/pods/fleet-agent-596955f45-bng97,user-agent:kube-controller-manager/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b/system:serviceaccount:kube-system:generic-garbage-collector,verb:GET (01-Jul-2024 13:28:30.090) (total time: 824ms):
Trace[592743277]: ---"About to write a response" 824ms (13:28:30.914)
2024-07-01T16:28:30.915186988+03:00 Trace[592743277]: [824.258145ms] [824.258145ms] END
2024-07-01T16:28:30.918073234+03:00 I0701 13:28:30.915455       1 trace.go:236] Trace[701158841]: "Get" accept:application/json, */*,audit-id:35102913-2e71-41b2-95ed-74c020839c60,client:10.0.0.6,api-group:,api-version:v1,name:alertmanager-rancher-monitoring-alertmanager-tls-assets-0,subresource:,namespace:cattle-monitoring-system,protocol:HTTP/2.0,resource:secrets,scope:resource,url:/api/v1/namespaces/cattle-monitoring-system/secrets/alertmanager-rancher-monitoring-alertmanager-tls-assets-0,user-agent:PrometheusOperator/0.65.1,verb:GET (01-Jul-2024 13:28:30.087) (total time: 827ms):
Trace[701158841]: ---"About to write a response" 826ms (13:28:30.914)
2024-07-01T16:28:30.918095834+03:00 Trace[701158841]: [827.831801ms] [827.831801ms] END
2024-07-01T16:28:30.982135140+03:00 I0701 13:28:30.982016       1 trace.go:236] Trace[56076880]: "Patch" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:99fe72da-2b37-4a8f-9cec-1cf88957739e,client:127.0.0.1,api-group:,api-version:v1,name:csi-provisioner-6cfd577457-8xb68.17de19abc59c177a,subresource:,namespace:longhorn-system,protocol:HTTP/2.0,resource:events,scope:resource,url:/api/v1/namespaces/longhorn-system/events/csi-provisioner-6cfd577457-8xb68.17de19abc59c177a,user-agent:kubelet/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b,verb:PATCH (01-Jul-2024 13:28:27.825) (total time: 3156ms):
Trace[56076880]: ["GuaranteedUpdate etcd3" audit-id:99fe72da-2b37-4a8f-9cec-1cf88957739e,key:/events/longhorn-system/csi-provisioner-6cfd577457-8xb68.17de19abc59c177a,type:*core.Event,resource:events 3156ms (13:28:27.825)
Trace[56076880]:  ---"initial value restored" 2821ms (13:28:30.646)
2024-07-01T16:28:30.982155640+03:00 Trace[56076880]:  ---"Txn call completed" 333ms (13:28:30.981)]
Trace[56076880]: ---"Object stored in database" 334ms (13:28:30.981)
Trace[56076880]: [3.156406071s] [3.156406071s] END
2024-07-01T16:28:31.195218286+03:00 I0701 13:28:31.190460       1 trace.go:236] Trace[1517678949]: "Create" accept:application/vnd.kubernetes.protobuf, */*,audit-id:c503362c-f47c-41c9-9df8-9c6fb1c26915,client:127.0.0.1,api-group:apps,api-version:v1,name:,subresource:,namespace:cattle-fleet-local-system,protocol:HTTP/2.0,resource:replicasets,scope:resource,url:/apis/apps/v1/namespaces/cattle-fleet-local-system/replicasets,user-agent:kube-controller-manager/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b/system:serviceaccount:kube-system:deployment-controller,verb:POST (01-Jul-2024 13:28:30.682) (total time: 508ms):
Trace[1517678949]: ["Create etcd3" audit-id:c503362c-f47c-41c9-9df8-9c6fb1c26915,key:/replicasets/cattle-fleet-local-system/fleet-agent-596955f45,type:*apps.ReplicaSet,resource:replicasets.apps 500ms (13:28:30.690)
2024-07-01T16:28:31.195242987+03:00 Trace[1517678949]:  ---"Txn call succeeded" 499ms (13:28:31.190)]
2024-07-01T16:28:31.195256787+03:00 Trace[1517678949]: [508.12808ms] [508.12808ms] END
2024-07-01T16:28:31.310422996+03:00 I0701 13:28:31.310228       1 trace.go:236] Trace[1132528431]: "Update" accept:application/vnd.kubernetes.protobuf, */*,audit-id:2bb30499-105b-4719-849d-97d97e20b749,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:kube-controller-manager,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:kube-controller-manager/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b/leader-election,verb:PUT (01-Jul-2024 13:28:30.650) (total time: 659ms):
2024-07-01T16:28:31.310440296+03:00 Trace[1132528431]: ["GuaranteedUpdate etcd3" audit-id:2bb30499-105b-4719-849d-97d97e20b749,key:/leases/kube-system/kube-controller-manager,type:*coordination.Lease,resource:leases.coordination.k8s.io 659ms (13:28:30.650)]
2024-07-01T16:28:31.310443496+03:00 Trace[1132528431]: [659.894264ms] [659.894264ms] END
2024-07-01T16:28:31.310446796+03:00 I0701 13:28:31.310265       1 trace.go:236] Trace[537692620]: "Get" accept:application/json, */*,audit-id:ea9378ec-c5f5-4f7e-b74d-9fc9350b0204,client:10.0.0.6,api-group:,api-version:v1,name:rancher-monitoring-admission,subresource:,namespace:cattle-monitoring-system,protocol:HTTP/2.0,resource:secrets,scope:resource,url:/api/v1/namespaces/cattle-monitoring-system/secrets/rancher-monitoring-admission,user-agent:PrometheusOperator/0.65.1,verb:GET (01-Jul-2024 13:28:30.656) (total time: 653ms):
2024-07-01T16:28:31.310456196+03:00 Trace[537692620]: ---"About to write a response" 653ms (13:28:31.310)
202
4-07-01T16:28:31.310458496+03:00 Trace[537692620]: [653.532963ms] [653.532963ms] END
2024-07-01T16:28:31.311027405+03:00 I0701 13:28:31.310611       1 trace.go:236] Trace[1029308395]: "Get" accept:application/vnd.kubernetes.protobuf,application/json,audit-id:68f26162-6fec-4c5b-bd9b-ba6562b9fbcc,client:10.0.0.6,api-group:crd.projectcalico.org,api-version:v1,name:default,subresource:,namespace:,protocol:HTTP/2.0,resource:clusterinformations,scope:resource,url:/apis/crd.projectcalico.org/v1/clusterinformations/default,user-agent:Go-http-client/2.0,verb:GET (01-Jul-2024 13:28:30.692) (total time: 617ms):
Trace[1029308395]: ---"About to write a response" 617ms (13:28:31.310)
2024-07-01T16:28:31.311039805+03:00 Trace[1029308395]: [617.718801ms] [617.718801ms] END
2024-07-01T16:28:31.311042905+03:00 I0701 13:28:31.310602       1 trace.go:236] Trace[1989096868]: "Update" accept:application/vnd.kubernetes.protobuf, */*,audit-id:e8b34c15-8445-4996-8dd0-a792f4b83a0e,client:127.0.0.1,api-group:apps,api-version:v1,name:cert-manager-webhook,subresource:status,namespace:cert-manager,protocol:HTTP/2.0,resource:deployments,scope:resource,url:/apis/apps/v1/namespaces/cert-manager/deployments/cert-manager-webhook/status,user-agent:kube-controller-manager/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b/system:serviceaccount:kube-system:deployment-controller,verb:PUT (01-Jul-2024 13:28:30.700) (total time: 610ms):
2024-07-01T16:28:31.311045805+03:00 Trace[1989096868]: ["GuaranteedUpdate etcd3" audit-id:e8b34c15-8445-4996-8dd0-a792f4b83a0e,key:/deployments/cert-manager/cert-manager-webhook,type:*apps.Deployment,resource:deployments.apps 610ms (13:28:30.700)
2024-07-01T16:28:31.311048305+03:00 Trace[1989096868]:  ---"Txn call completed" 606ms (13:28:31.310)]
2024-07-01T16:28:31.311050305+03:00 Trace[1989096868]: [610.346485ms] [610.346485ms] END
I0701 13:28:31.357459       1 trace.go:236] Trace[1379426810]: "GuaranteedUpdate etcd3" audit-id:,key:/masterleases/10.0.0.3,type:*v1.Endpoints,resource:apiServerIPInfo (01-Jul-2024 13:28:25.809) (total time: 5547ms):
Trace[1379426810]: ---"initial value restored" 4836ms (13:28:30.646)
2024-07-01T16:28:31.357625537+03:00 Trace[1379426810]: ---"Transaction prepared" 334ms (13:28:30.980)
2024-07-01T16:28:31.357646237+03:00 Trace[1379426810]: ---"Txn call completed" 332ms (13:28:31.313)
2024-07-01T16:28:31.357648637+03:00 Trace[1379426810]: ---"Txn call completed" 33ms (13:28:31.357)
Trace[1379426810]: [5.547775026s] [5.547775026s] END
2024-07-01T16:28:31.739882640+03:00 I0701 13:28:31.739735       1 controller.go:615] quota admission added evaluator for: projects.management.cattle.io
2024-07-01T16:28:36.047920397+03:00 I0701 13:28:36.047761       1 controller.go:615] quota admission added evaluator for: apps.catalog.cattle.io
2024-07-01T16:28:36.852917439+03:00 I0701 13:28:36.852790       1 controller.go:615] quota admission added evaluator for: roles.rbac.authorization.k8s.io
2024-07-01T16:28:44.347656243+03:00 I0701 13:28:44.347549       1 controller.go:615] quota admission added evaluator for: nodes.management.cattle.io
2024-07-01T16:30:54.054623765+03:00 I0701 13:30:54.054487       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:31:54.060442044+03:00 I0701 13:31:54.060319       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 13:33:54.051531       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:35:32.867726631+03:00 E0701 13:35:32.867580       1 writers.go:68] error encountered while streaming results via websocket: use of closed network connection
2024-07-01T16:35:32.867845533+03:00 I0701 13:35:32.867752       1 trace.go:236] Trace[786073451]: "Get" accept:,audit-id:06df436a-39c9-44b1-90a8-6e5474f56b45,client:10.0.0.1,api-group:,api-version:v1,name:kube-apiserver-rke21,subresource:log,namespace:kube-system,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/kube-apiserver-rke21/log,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36,verb:CONNECT (01-Jul-2024 13:34:12.719) (total time: 80147ms):
2024-07-01T16:35:32.867899933+03:00 Trace[786073451]: ---"Writing http response done" 80144ms (13:35:32.867)
Trace[786073451]: [1m20.14796147s] [1m20.14796147s] END
2024-07-01T16:35:32.868365641+03:00 E0701 13:35:32.868258       1 writers.go:68] error encountered while streaming results via websocket: use of closed network connection
2024-07-01T16:35:32.869370956+03:00 I0701 13:35:32.869281       1 trace.go:236] Trace[1489989362]: "Get" accept:,audit-id:6c2b53bc-6a68-4014-9f5c-64182a54ca7d,client:10.0.0.1,api-group:,api-version:v1,name:kube-apiserver-rke22,subresource:log,namespace:kube-system,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/kube-apiserver-rke22/log,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36,verb:CONNECT (01-Jul-2024 13:33:21.687) (total time: 131181ms):
2024-07-01T16:35:32.869402057+03:00 Trace[1489989362]: ---"Writing http response done" 131178ms (13:35:32.869)
Trace[1489989362]: [2m11.181894947s] [2m11.181894947s] END
2024-07-01T16:35:54.057695915+03:00 I0701 13:35:54.057580       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:37:54.054656011+03:00 I0701 13:37:54.054514       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 13:39:54.026599       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:40:54.075488553+03:00 I0701 13:40:54.075348       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:42:54.053843857+03:00 I0701 13:42:54.053710       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:44:54.053656497+03:00 I0701 13:44:54.053534       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
`

Rahncer-webhook logs:

Container: rancher-webhook
Filter
Connected
time="2024-07-01T13:27:23Z" level=info msg="Rancher-webhook version v0.4.5 (b6b5ef9) is starting"
2024-07-01T16:27:53.670770330+03:00 time="2024-07-01T13:27:53Z" level=info msg="Waiting for server to become available: Get \"https://10.43.0.1:443/version\": dial tcp 10.43.0.1:443: i/o timeout"
time="2024-07-01T13:27:55Z" level=info msg="could not read client CA file at /tmp/k8s-webhook-server/client-ca/ca.crt, incoming requests will not be authenticated"
2024-07-01T16:27:55.685291559+03:00 time="2024-07-01T13:27:55Z" level=info msg="Active TLS secret cattle-system/cattle-webhook-tls (ver=1076152) (count 1): map[listener.cattle.io/cn-rancher-webhook.cattle-system.svc:rancher-webhook.cattle-system.svc listener.cattle.io/fingerprint:SHA1=B3CC32F195B2BF556ABBFCA978A80EE552214120]"
2024-07-01T16:27:55.685497362+03:00 time="2024-07-01T13:27:55Z" level=info msg="Listening on :9443"
2024-07-01T16:27:56.200850654+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller"
2024-07-01T16:27:56.200886355+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller"
2024-07-01T16:27:56.200981156+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller"
time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=PodSecurityAdmissionConfigurationTemplate controller"
2024-07-01T16:27:56.201153459+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting apiregistration.k8s.io/v1, Kind=APIService controller"
2024-07-01T16:27:56.201232160+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=Cluster controller"
2024-07-01T16:27:56.201347362+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=Role controller"
2024-07-01T16:27:56.201377262+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=RoleTemplate controller"
2024-07-01T16:27:56.201566465+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=GlobalRole controller"
2024-07-01T16:27:56.201621366+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting provisioning.cattle.io/v1, Kind=Cluster controller"
2024-07-01T16:27:56.201634366+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller"
2024-07-01T16:27:56.201665867+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting /v1, Kind=Secret controller"
2024-07-01T16:27:56.201779469+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller"
2024-07-01T16:27:56.201784469+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=Project controller"
time="2024-07-01T13:27:56Z" level=info msg="Starting apiextensions.k8s.io/v1, Kind=CustomResourceDefinition controller"
time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=Node controller"
2024-07-01T16:27:56.202260876+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller"
time="2024-07-01T13:27:56Z" level=info msg="Sleeping for 15 seconds then applying webhook config"
2024-07-01T16:27:56.206044736+03:00 time="2024-07-01T13:27:56Z" level=info msg="Updating TLS secret for cattle-system/cattle-webhook-tls (count: 1): map[listener.cattle.io/cn-rancher-webhook.cattle-system.svc:rancher-webhook.cattle-system.svc listener.cattle.io/fingerprint:SHA1=B3CC32F195B2BF556ABBFCA978A80EE552214120]"
2024-07-01T16:27:56.719661400+03:00 E0701 13:27:56.719525       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.720737       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.734959       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.735893955+03:00 E0701 13:27:56.735806       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.754171842+03:00 E0701 13:27:56.754052       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.757199       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.771330       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.778096217+03:00 E0701 13:27:56.773579       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.792699646+03:00 E0701 13:27:56.792494       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.792734147+03:00 E0701 13:27:56.792506       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.806061       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.807459378+03:00 E0701 13:27:56.807348       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.819189962+03:00 E0701 13:27:56.819071       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.820806188+03:00 E0701 13:27:56.820708       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.833936894+03:00 E0701 13:27:56.833793       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.841750       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.851662872+03:00 E0701 13:27:56.851542       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.857063857+03:00 E0701 13:27:56.856957       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.869677855+03:00 E0701 13:27:56.869498       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.879600311+03:00 E0701 13:27:56.879487       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.888912       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.894916151+03:00 E0701 13:27:56.894539       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.902915977+03:00 E0701 13:27:56.902813       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.907763253+03:00 E0701 13:27:56.907640       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.915775       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.919080       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.927324560+03:00 E0701 13:27:56.927188       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.930428709+03:00 E0701 13:27:56.930328       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.939745       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.947681380+03:00 E0701 13:27:56.947582       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.963482       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.969499722+03:00 E0701 13:27:56.969382       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.978763       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.981258       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.993530900+03:00 E0701 13:27:56.993430       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.996138341+03:00 E0701 13:27:56.995972       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.008891441+03:00 E0701 13:27:57.008792       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.010801       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.022405253+03:00 E0701 13:27:57.022287       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.024484       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.035359556+03:00 E0701 13:27:57.035133       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.037720694+03:00 E0701 13:27:57.036970       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.055321370+03:00 E0701 13:27:57.055224       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.055891       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.067156956+03:00 E0701 13:27:57.067048       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.067422       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.078890       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.082322594+03:00 E0701 13:27:57.082237       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.092135648+03:00 E0701 13:27:57.092037       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.094538086+03:00 E0701 13:27:57.094440       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.105565959+03:00 E0701 13:27:57.105457       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.106392       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.119430       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.119577279+03:00 E0701 13:27:57.119435       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.133612199+03:00 E0701 13:27:57.133505       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.133628599+03:00 E0701 13:27:57.133547       1 gvks.go:69] failed to sync schemas: unable to retrieve the .metrics.k8s.io/v1beta1
2024-07-01T16:27:57.168356345+03:00 E0701 13:27:57.168272       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.179861025+03:00 E0701 13:27:57.179747       1 gvks.go:69] fa
iled to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.181245       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.197162597+03:00 E0701 13:27:57.197064       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.197621204+03:00 E0701 13:27:57.197554       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.209121385+03:00 E0701 13:27:57.209015       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.210874612+03:00 E0701 13:27:57.210788       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.223573012+03:00 E0701 13:27:57.223483       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.225367140+03:00 E0701 13:27:57.225267       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.236050707+03:00 E0701 13:27:57.235945       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.240585679+03:00 E0701 13:27:57.240464       1 gvks.go:69] fa
iled to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.255015705+03:00 E0701 13:27:57.254927       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.257487344+03:00 E0701 13:27:57.257369       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.267743605+03:00 E0701 13:27:57.267643       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs:om.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.284567269+03:00 E0701 13:27:57.284482       1 gvks.go:69] failed to sync schemas: unable to retrieve the
  1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
 schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.829040718+03:00 E0701 13:27:57.828584       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.839651184+03:00 E0701 13:27:57.839505       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.841310111+03:00 E0701 13:27:57.841236       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of 
server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.851089364+03:00 E0701 13:27:57.851010       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1

2024-07-01T16:27:57.887594937+03:00 E0701 13:27:57.887499       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.897724996+03:00 E0701 13:27:57.897643       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom
.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.899065117+03:00 E0701 13:27:57.898974       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.912447427+03:00 E0701 13:27:57.912349       1 gvks.go:69] failed to sync schemas: unable to retrieve the  GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.936776809+03:00 E0701 13:27:57.936597       1 gvks.go:69] failed to sync schemas: unable to retrieve the 
2024-07-01T16:27:58.320809339+03:00 E0701 13:27:58.320708       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.329123370+03:00 E0701 13:27:58.329039       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of sev1beta1
2024-07-01T16:27:58.361017670+03:00 E0701 13:27:58.360938       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.370437818+03:00 E0701 13:27:58.370243       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.373107960+03:00 E0701 13
:27:58.372977       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.382817413+03:00 E0701 13:27:58.382714       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.383484023+03:00 E0701 13:27:58.383401       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
k8s.io/v1beta1
2024-07-01T16:27:58.410922654+03:00 E0701 13:27:58.410844       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.425937590+03:00 E0701 13:27:58.425830       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.426204694+03:00 E0701 13:27:58.426139       1 gvks.go:69] failed to sync schemas: unable to retrieve the
nable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.468092652+03:00 E0701 13:27:58.468029       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.480450946+03:00 E0701 13:27:58.480318       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.480783       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.493057       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.493058       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.504655       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.506293851+03:00 E0701 13:27:58.506202       1 gvks.go:69] failed to sync schemas: unable to retrieve the  of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.941604086+03:00 E0701 13:27:58.941523       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.954794       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1

Rke2-metric-server logs:

I0701 13:27:24.149467       1 handler.go:275] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:27:24.265149541+03:00 I0701 13:27:24.265007       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
2024-07-01T16:27:24.265178941+03:00 I0701 13:27:24.265023       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
2024-07-01T16:27:24.265186041+03:00 I0701 13:27:24.265035       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
2024-07-01T16:27:24.265189542+03:00 I0701 13:27:24.265046       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
2024-07-01T16:27:24.265192142+03:00 I0701 13:27:24.265055       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
2024-07-01T16:27:24.265211842+03:00 I0701 13:27:24.265046       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0701 13:27:24.265670       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
2024-07-01T16:27:24.266779866+03:00 I0701 13:27:24.266455       1 secure_serving.go:213] Serving securely on [::]:10250
2024-07-01T16:27:24.266802267+03:00 I0701 13:27:24.266662       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
2024-07-01T16:27:24.366056723+03:00 I0701 13:27:24.365900       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
2024-07-01T16:27:24.366074524+03:00 I0701 13:27:24.365893       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
2024-07-01T16:27:24.366079424+03:00 I0701 13:27:24.365941       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController

[cengizhanR]

@brandond, This is service pictures, All pods are running, I cant observe metric for kube api server on grafana too(Kubernetes/API server). Could this be due to errors? I can access apiserver metrics from Prometheus, but Grafana does not show these metrics. Thank you very much for your interest and concern.

[manuelbuil]

In your deployed rke2 cluster, can you provide the output of the commands:
1 - kubectl get pods -A
2 - kubectl get services -A
3 - kubectl get endpoints -A

[cengizhanR]

@manuelbuil Thank you very much for your interest and concern. I shared below them,
kubectl get pods -A

[root@rke21 home]# kubectl get pods -A
NAMESPACE                         NAME                                                      READY   STATUS      RESTARTS        AGE
cattle-fleet-local-system         fleet-agent-596955f45-flcp8                               1/1     Running     0               40m
cattle-fleet-system               fleet-controller-867864bc49-5dhcm                         1/1     Running     44 (41m ago)    34d
cattle-fleet-system               gitjob-755f97fbf7-94vpz                                   1/1     Running     20 (42m ago)    42d
cattle-monitoring-system          alertmanager-rancher-monitoring-alertmanager-0            2/2     Running     18 (42m ago)    34d
cattle-monitoring-system          prometheus-rancher-monitoring-prometheus-0                3/3     Running     23 (45m ago)    34d
cattle-monitoring-system          pushprox-kube-controller-manager-client-h2ppn             1/1     Running     9 (45m ago)     34d
cattle-monitoring-system          pushprox-kube-controller-manager-client-j5m98             1/1     Running     10 (44m ago)    34d
cattle-monitoring-system          pushprox-kube-controller-manager-client-tscdt             1/1     Running     9 (45m ago)     34d
cattle-monitoring-system          pushprox-kube-controller-manager-proxy-7cdd86d77d-k9dc4   1/1     Running     9 (42m ago)     34d
cattle-monitoring-system          pushprox-kube-etcd-client-467vs                           1/1     Running     10 (44m ago)    34d
cattle-monitoring-system          pushprox-kube-etcd-client-btpq4                           1/1     Running     9 (45m ago)     34d
cattle-monitoring-system          pushprox-kube-etcd-client-zk4jm                           1/1     Running     9 (45m ago)     34d
cattle-monitoring-system          pushprox-kube-etcd-proxy-6bf9dfb758-9m8mj                 1/1     Running     9 (45m ago)     34d
cattle-monitoring-system          pushprox-kube-proxy-client-dklsr                          1/1     Running     9 (45m ago)     34d
cattle-monitoring-system          pushprox-kube-proxy-client-jqwxt                          1/1     Running     9 (42m ago)     34d
cattle-monitoring-system          pushprox-kube-proxy-client-mprlm                          1/1     Running     9 (42m ago)     34d
cattle-monitoring-system          pushprox-kube-proxy-client-mvlxh                          1/1     Running     10 (44m ago)    34d
cattle-monitoring-system          pushprox-kube-proxy-client-r2945                          1/1     Running     9 (45m ago)     34d
cattle-monitoring-system          pushprox-kube-proxy-proxy-874d5c99d-pgnlv                 1/1     Running     7 (45m ago)     34d
cattle-monitoring-system          pushprox-kube-scheduler-client-7hndn                      1/1     Running     9 (45m ago)     34d
cattle-monitoring-system          pushprox-kube-scheduler-client-kk24v                      1/1     Running     9 (45m ago)     34d
cattle-monitoring-system          pushprox-kube-scheduler-client-wf2z7                      1/1     Running     10 (44m ago)    34d
cattle-monitoring-system          pushprox-kube-scheduler-proxy-8d54cf848-kjc5q             1/1     Running     9 (42m ago)     34d
cattle-monitoring-system          rancher-monitoring-grafana-7b6454647b-xrljt               4/4     Running     24 (42m ago)    34d
cattle-monitoring-system          rancher-monitoring-kube-state-metrics-575886b657-wmw9s    1/1     Running     27 (40m ago)    34d
cattle-monitoring-system          rancher-monitoring-operator-665bd495f7-g5sck              1/1     Running     26 (40m ago)    34d
cattle-monitoring-system          rancher-monitoring-prometheus-adapter-78874ddf98-mb4j8    1/1     Running     18 (41m ago)    34d
cattle-monitoring-system          rancher-monitoring-prometheus-node-exporter-gmjc6         1/1     Running     9 (42m ago)     34d
cattle-monitoring-system          rancher-monitoring-prometheus-node-exporter-l2fnl         1/1     Running     9 (45m ago)     34d
cattle-monitoring-system          rancher-monitoring-prometheus-node-exporter-ssdw9         1/1     Running     9 (45m ago)     34d
cattle-monitoring-system          rancher-monitoring-prometheus-node-exporter-wvpcv         1/1     Running     10 (42m ago)    34d
cattle-monitoring-system          rancher-monitoring-prometheus-node-exporter-z7mwc         1/1     Running     10 (44m ago)    34d
cattle-provisioning-capi-system   capi-controller-manager-745f484f4d-qln7p                  1/1     Running     25 (41m ago)    42d
cattle-system                     rancher-7f5977f44c-f6hx7                                  1/1     Running     18 (42m ago)    42d
cattle-system                     rancher-7f5977f44c-fr796                                  1/1     Running     14 (44m ago)    42d
cattle-system                     rancher-7f5977f44c-jtdxk                                  1/1     Running     16 (42m ago)    42d
cattle-system                     rancher-webhook-6cf88c8fc7-s7z8m                          1/1     Running     449 (42m ago)   42d
cengiz                            cengiz-6d95db646b-s26vv                                   1/1     Running     1 (44m ago)     17d
cengiz                            mypod                                                     1/1     Running     6 (45m ago)     17d
cert-manager                      cert-manager-745965d64b-c47xt                             1/1     Running     20 (42m ago)    42d
cert-manager                      cert-manager-cainjector-6ff697db96-l4vvs                  1/1     Running     35 (41m ago)    42d
cert-manager                      cert-manager-startupapicheck-2nm2l                        0/1     Completed   0               42d
cert-manager                      cert-manager-webhook-5d5dfb9b94-zqvkw                     1/1     Running     26 (40m ago)    42d
kube-system                       cloud-controller-manager-rke21                            1/1     Running     3 (45m ago)     6d20h
kube-system                       cloud-controller-manager-rke22                            1/1     Running     34 (44m ago)    43m
kube-system                       cloud-controller-manager-rke23                            1/1     Running     31 (45m ago)    42d
kube-system                       etcd-rke21                                                1/1     Running     1               100m
kube-system                       etcd-rke22                                                1/1     Running     0               43m
kube-system                       etcd-rke23                                                1/1     Running     0               43m
kube-system                       helm-install-rke2-canal-gmrf9                             0/1     Completed   0               42d
kube-system                       helm-install-rke2-coredns-s4tnk                           0/1     Completed   0               42d
kube-system                       helm-install-rke2-ingress-nginx-n56fr                     0/1     Completed   4 (40m ago)     42m
kube-system                       helm-install-rke2-metrics-server-swdwb                    0/1     Completed   0               42d
kube-system                       helm-install-rke2-snapshot-controller-crd-2p5kq           0/1     Completed   0               42d
kube-system                       helm-install-rke2-snapshot-controller-r9kbd               0/1     Completed   0               42d
kube-system                       helm-install-rke2-snapshot-validation-webhook-99vw4       0/1     Completed   0               42d
kube-system                       kube-apiserver-rke21                                      1/1     Running     1               99m
kube-system                       kube-apiserver-rke22                                      1/1     Running     1               98m
kube-system                       kube-apiserver-rke23                                      1/1     Running     1               98m
kube-system                       kube-controller-manager-rke21                             1/1     Running     1 (45m ago)     100m
kube-system                       kube-controller-manager-rke22                             1/1     Running     1 (44m ago)     43m
kube-system                       kube-controller-manager-rke23                             1/1     Running     1 (45m ago)     98m
kube-system                       kube-proxy-rke21                                          1/1     Running     1 (45m ago)     42m
kube-system                       kube-proxy-rke22                                          1/1     Running     12 (44m ago)    43m
kube-system                       kube-proxy-rke23                                          1/1     Running     11 (45m ago)    43m
kube-system                       kube-proxy-rke2agent1                                     1/1     Running     0               41m
kube-system                       kube-proxy-rke2agent2                                     1/1     Running     1 (42m ago)     98m
kube-system                       kube-scheduler-rke21                                      1/1     Running     1 (45m ago)     100m
kube-system                       kube-scheduler-rke22                                      1/1     Running     1 (44m ago)     43m
kube-system                       kube-scheduler-rke23                                      1/1     Running     1 (45m ago)     98m
kube-system                       rke2-canal-67pb9                                          2/2     Running     18 (45m ago)    42d
kube-system                       rke2-canal-7vsnj                                          2/2     Running     22 (44m ago)    42d
kube-system                       rke2-canal-ggtt4                                          2/2     Running     18 (42m ago)    42d
kube-system                       rke2-canal-gvh9z                                          2/2     Running     18 (42m ago)    42d
kube-system                       rke2-canal-qwd2j                                          2/2     Running     18 (45m ago)    42d
kube-system                       rke2-coredns-rke2-coredns-7646547544-5c2h9                1/1     Running     9 (42m ago)     42d
kube-system                       rke2-coredns-rke2-coredns-7646547544-kgbrf                1/1     Running     18 (42m ago)    42d
kube-system                       rke2-coredns-rke2-coredns-autoscaler-77f7d75b74-wzqs9     1/1     Running     18 (42m ago)    42d
kube-system                       rke2-ingress-nginx-controller-5lh5g                       1/1     Running     24 (45m ago)    39d
kube-system                       rke2-ingress-nginx-controller-7sswg                       1/1     Running     25 (44m ago)    39d
kube-system                       rke2-ingress-nginx-controller-hbtw4                       1/1     Running     19 (42m ago)    39d
kube-system                       rke2-ingress-nginx-controller-kjlsk                       1/1     Running     24 (45m ago)    39d
kube-system                       rke2-ingress-nginx-controller-wddct                       1/1     Running     21 (42m ago)    39d
kube-system                       rke2-metrics-server-c9dcd6cc7-xxzf9                       1/1     Running     18 (41m ago)    42d
kube-system                       rke2-snapshot-controller-7ff4687584-wgpj5                 1/1     Running     29 (42m ago)    42d
kube-system                       rke2-snapshot-validation-webhook-74bbbd4949-8269h         1/1     Running     29 (40m ago)    42d
longhorn-system                   csi-attacher-5dc54bcccc-5lt6p                             1/1     Running     16 (42m ago)    17d
longhorn-system                   csi-attacher-5dc54bcccc-g2kxd                             1/1     Running     28 (40m ago)    17d
longhorn-system                   csi-attacher-5dc54bcccc-lx52k                             1/1     Running     23 (41m ago)    17d
longhorn-system                   csi-provisioner-6cfd577457-8xb68                          1/1     Running     4 (40m ago)     17d
longhorn-system                   csi-provisioner-6cfd577457-9whsg                          1/1     Running     31 (40m ago)    17d
longhorn-system                   csi-provisioner-6cfd577457-cwdpw                          1/1     Running     15 (42m ago)    17d
longhorn-system                   csi-resizer-bc8db45d-dqjz9                                1/1     Running     16 (42m ago)    17d
longhorn-system                   csi-resizer-bc8db45d-fdh96                                1/1     Running     23 (41m ago)    17d
longhorn-system                   csi-resizer-bc8db45d-frv2h                                1/1     Running     4 (40m ago)     17d
longhorn-system                   csi-snapshotter-75c94ccf4d-2x472                          1/1     Running     28 (40m ago)    17d
longhorn-system                   csi-snapshotter-75c94ccf4d-nsb6m                          1/1     Running     16 (42m ago)    17d
longhorn-system                   csi-snapshotter-75c94ccf4d-qvm26                          1/1     Running     4 (40m ago)     17d
longhorn-system                   engine-image-ei-432c5210-cdhdg                            1/1     Running     9 (42m ago)     17d
longhorn-system                   engine-image-ei-432c5210-kmv8w                            1/1     Running     10 (42m ago)    17d
longhorn-system                   engine-image-ei-432c5210-sqwfv                            1/1     Running     10 (44m ago)    17d
longhorn-system                   engine-image-ei-432c5210-wcphg                            1/1     Running     9 (45m ago)     17d
longhorn-system                   engine-image-ei-432c5210-wswpj                            1/1     Running     9 (45m ago)     17d
longhorn-system                   instance-manager-0d432bc67f9c3c1053e561c687f02872         1/1     Running     0               41m
longhorn-system                   instance-manager-1ab5836542c0fd42fc6df6de7dabbd6a         1/1     Running     0               40m
longhorn-system                   instance-manager-c3797ec036a289a19ee69b539570d91a         1/1     Running     0               41m
longhorn-system                   instance-manager-c96c134ffcb41f15abe2445293be32dd         1/1     Running     0               40m
longhorn-system                   instance-manager-e8fc75b7098eb2befb1de415f680a744         1/1     Running     0               41m
longhorn-system                   longhorn-csi-plugin-6fl7v                                 3/3     Running     52 (41m ago)    17d
longhorn-system                   longhorn-csi-plugin-ml2m4                                 3/3     Running     54 (41m ago)    17d
longhorn-system                   longhorn-csi-plugin-qcrmz                                 3/3     Running     75 (40m ago)    17d
longhorn-system                   longhorn-csi-plugin-qf8pj                                 3/3     Running     77 (41m ago)    17d
longhorn-system                   longhorn-csi-plugin-r72tw                                 3/3     Running     81 (40m ago)    17d
longhorn-system                   longhorn-driver-deployer-d9b97f779-c2w2m                  1/1     Running     11 (45m ago)    17d
longhorn-system                   longhorn-manager-2btc8                                    1/1     Running     19 (41m ago)    17d
longhorn-system                   longhorn-manager-msgd6                                    1/1     Running     9 (42m ago)     17d
longhorn-system                   longhorn-manager-s8kzq                                    1/1     Running     14 (41m ago)    17d
longhorn-system                   longhorn-manager-scccm                                    1/1     Running     9 (42m ago)     17d
longhorn-system                   longhorn-manager-w2p9v                                    1/1     Running     16 (41m ago)    17d
longhorn-system                   longhorn-ui-9584c465f-9mxp6                               1/1     Running     34 (41m ago)    17d
longhorn-system                   longhorn-ui-9584c465f-sgml9                               1/1     Running     19 (41m ago)    17d

kubectl get services -A

cattle-fleet-system               gitjob                                        ClusterIP      10.43.104.180   <none>        80/TCP                         42d
cattle-monitoring-system          alertmanager-operated                         ClusterIP      None            <none>        9093/TCP,9094/TCP,9094/UDP     34d
cattle-monitoring-system          prometheus-operated                           ClusterIP      None            <none>        9090/TCP                       34d
cattle-monitoring-system          pushprox-kube-controller-manager-client       ClusterIP      10.43.174.151   <none>        10257/TCP                      34d
cattle-monitoring-system          pushprox-kube-controller-manager-proxy        ClusterIP      10.43.58.55     <none>        8080/TCP                       34d
cattle-monitoring-system          pushprox-kube-etcd-client                     ClusterIP      10.43.103.82    <none>        2381/TCP                       34d
cattle-monitoring-system          pushprox-kube-etcd-proxy                      ClusterIP      10.43.185.165   <none>        8080/TCP                       34d
cattle-monitoring-system          pushprox-kube-proxy-client                    ClusterIP      10.43.38.214    <none>        10249/TCP                      34d
cattle-monitoring-system          pushprox-kube-proxy-proxy                     ClusterIP      10.43.235.198   <none>        8080/TCP                       34d
cattle-monitoring-system          pushprox-kube-scheduler-client                ClusterIP      10.43.134.140   <none>        10259/TCP                      34d
cattle-monitoring-system          pushprox-kube-scheduler-proxy                 ClusterIP      10.43.135.16    <none>        8080/TCP                       34d
cattle-monitoring-system          rancher-monitoring-alertmanager               ClusterIP      10.43.93.52     <none>        9093/TCP                       34d
cattle-monitoring-system          rancher-monitoring-grafana                    ClusterIP      10.43.24.235    <none>        80/TCP                         34d
cattle-monitoring-system          rancher-monitoring-kube-state-metrics         ClusterIP      10.43.84.95     <none>        8080/TCP                       34d
cattle-monitoring-system          rancher-monitoring-operator                   ClusterIP      10.43.32.237    <none>        443/TCP                        34d
cattle-monitoring-system          rancher-monitoring-prometheus                 ClusterIP      10.43.74.112    <none>        9090/TCP                       34d
cattle-monitoring-system          rancher-monitoring-prometheus-adapter         ClusterIP      10.43.20.98     <none>        443/TCP                        34d
cattle-monitoring-system          rancher-monitoring-prometheus-node-exporter   ClusterIP      10.43.225.98    <none>        9796/TCP                       34d
cattle-provisioning-capi-system   capi-webhook-service                          ClusterIP      10.43.244.117   <none>        443/TCP                        42d
cattle-system                     rancher                                       ClusterIP      10.43.106.42    <none>        80/TCP,443/TCP                 42d
cattle-system                     rancher-webhook                               ClusterIP      10.43.224.165   <none>        443/TCP                        42d
cengiz                            cengiz                                        ClusterIP      10.43.138.138   <none>        8080/TCP                       34d
cert-manager                      cert-manager                                  ClusterIP      10.43.197.100   <none>        9402/TCP                       44d
cert-manager                      cert-manager-webhook                          ClusterIP      10.43.128.178   <none>        443/TCP                        44d
default                           kubernetes                                    ClusterIP      10.43.0.1       <none>        443/TCP                        45d
kube-system                       pushprox-ingress-nginx-client                 ClusterIP      10.43.111.40    <none>        10254/TCP                      34d
kube-system                       rancher-monitoring-coredns                    ClusterIP      None            <none>        9153/TCP                       34d
kube-system                       rancher-monitoring-kubelet                    ClusterIP      None            <none>        10250/TCP,10255/TCP,4194/TCP   34d
kube-system                       rke2-coredns-rke2-coredns                     ClusterIP      10.43.0.10      <none>        53/UDP,53/TCP                  45d
kube-system                       rke2-ingress-nginx-controller                 LoadBalancer   10.43.36.82     <pending>     80:32645/TCP,443:31745/TCP     39d
kube-system                       rke2-ingress-nginx-controller-admission       ClusterIP      10.43.220.172   <none>        443/TCP                        39d
kube-system                       rke2-ingress-nginx-controller-metrics         ClusterIP      10.43.103.64    <none>        10254/TCP                      39d
kube-system                       rke2-metrics-server                           ClusterIP      10.43.58.68     <none>        443/TCP                        45d
kube-system                       rke2-snapshot-validation-webhook              ClusterIP      10.43.13.188    <none>        443/TCP                        45d
longhorn-system                   longhorn-admission-webhook                    ClusterIP      10.43.30.204    <none>        9502/TCP                       17d
longhorn-system                   longhorn-backend                              ClusterIP      10.43.125.155   <none>        9500/TCP                       17d
longhorn-system                   longhorn-conversion-webhook                   ClusterIP      10.43.223.221   <none>        9501/TCP                       17d
longhorn-system                   longhorn-engine-manager                       ClusterIP      None            <none>        <none>                         17d
longhorn-system                   longhorn-frontend                             ClusterIP      10.43.83.98     <none>        80/TCP                         17d
longhorn-system                   longhorn-recovery-backend                     ClusterIP      10.43.102.58    <none>        9503/TCP                       17d
longhorn-system                   longhorn-replica-manager                      ClusterIP      None            <none>        <none>                         17d

kubectl get endpoints -A

NAMESPACE                         NAME                                          ENDPOINTS                                                      AGE
cattle-fleet-system               gitjob                                        10.42.3.8:8080                                                 42d
cattle-monitoring-system          alertmanager-operated                         10.42.3.12:9094,10.42.3.12:9094,10.42.3.12:9093                34d
cattle-monitoring-system          prometheus-operated                           10.42.2.4:9090                                                 34d
cattle-monitoring-system          pushprox-kube-controller-manager-client       10.0.0.2:10257,10.0.0.3:10257,10.0.0.4:10257                   34d
cattle-monitoring-system          pushprox-kube-controller-manager-proxy        10.42.3.17:8080                                                34d
cattle-monitoring-system          pushprox-kube-etcd-client                     10.0.0.2:2381,10.0.0.3:2381,10.0.0.4:2381                      34d
cattle-monitoring-system          pushprox-kube-etcd-proxy                      10.42.0.11:8080                                                34d
cattle-monitoring-system          pushprox-kube-proxy-client                    10.0.0.2:10249,10.0.0.3:10249,10.0.0.4:10249 + 2 more...       34d
cattle-monitoring-system          pushprox-kube-proxy-proxy                     10.42.2.2:8080                                                 34d
cattle-monitoring-system          pushprox-kube-scheduler-client                10.0.0.2:10259,10.0.0.3:10259,10.0.0.4:10259                   34d
cattle-monitoring-system          pushprox-kube-scheduler-proxy                 10.42.3.16:8080                                                34d
cattle-monitoring-system          rancher-monitoring-alertmanager               10.42.3.12:9093                                                34d
cattle-monitoring-system          rancher-monitoring-grafana                    10.42.4.16:8080                                                34d
cattle-monitoring-system          rancher-monitoring-kube-state-metrics         10.42.4.12:8080                                                34d
cattle-monitoring-system          rancher-monitoring-operator                   10.42.4.8:8443                                                 34d
cattle-monitoring-system          rancher-monitoring-prometheus                 10.42.2.4:8081                                                 34d
cattle-monitoring-system          rancher-monitoring-prometheus-adapter         10.42.3.9:6443                                                 34d
cattle-monitoring-system          rancher-monitoring-prometheus-node-exporter   10.0.0.2:9796,10.0.0.3:9796,10.0.0.4:9796 + 2 more...          34d
cattle-provisioning-capi-system   capi-webhook-service                          10.42.3.6:9443                                                 42d
cattle-system                     rancher                                       10.42.1.5:80,10.42.3.15:80,10.42.4.13:80 + 3 more...           42d
cattle-system                     rancher-webhook                               10.42.4.3:9443                                                 42d
cengiz                            cengiz                                        10.42.1.2:8080                                                 34d
cert-manager                      cert-manager                                  10.42.4.11:9402                                                44d
cert-manager                      cert-manager-webhook                          10.42.4.9:10250                                                44d
default                           kubernetes                                    10.0.0.2:6443,10.0.0.3:6443,10.0.0.4:6443                      45d
kube-system                       pushprox-ingress-nginx-client                 10.42.0.4:10254,10.42.1.10:10254,10.42.2.6:10254 + 2 more...   34d
kube-system                       rancher-monitoring-coredns                    10.42.3.11:9153,10.42.4.14:9153                                34d
kube-system                       rancher-monitoring-kubelet                    10.0.0.2:10250,10.0.0.3:10250,10.0.0.4:10250 + 12 more...      34d
kube-system                       rke2-coredns-rke2-coredns                     10.42.3.11:53,10.42.4.14:53,10.42.3.11:53 + 1 more...          45d
kube-system                       rke2-ingress-nginx-controller                 10.42.0.4:443,10.42.1.10:443,10.42.2.6:443 + 7 more...         39d
kube-system                       rke2-ingress-nginx-controller-admission       10.42.0.4:8443,10.42.1.10:8443,10.42.2.6:8443 + 2 more...      39d
kube-system                       rke2-ingress-nginx-controller-metrics         10.42.0.4:10254,10.42.1.10:10254,10.42.2.6:10254 + 2 more...   39d
kube-system                       rke2-metrics-server                           10.42.3.2:10250                                                45d
kube-system                       rke2-snapshot-validation-webhook              10.42.4.10:8443                                                45d
longhorn-system                   longhorn-admission-webhook                    10.42.0.3:9502,10.42.1.6:9502,10.42.2.7:9502 + 2 more...       17d
longhorn-system                   longhorn-backend                              10.42.0.3:9500,10.42.1.6:9500,10.42.2.7:9500 + 2 more...       17d
longhorn-system                   longhorn-conversion-webhook                   10.42.0.3:9501,10.42.1.6:9501,10.42.2.7:9501 + 2 more...       17d
longhorn-system                   longhorn-engine-manager                       <none>                                                         17d
longhorn-system                   longhorn-frontend                             10.42.1.4:8000,10.42.4.4:8000                                  17d
longhorn-system                   longhorn-recovery-backend                     10.42.0.3:9503,10.42.1.6:9503,10.42.2.7:9503 + 2 more...       17d
longhorn-system                   longhorn-replica-manager                      <none>                                                         17d

[manuelbuil]

Thanks for the output. As you can see endpoints are there and they seem correct. Can you confirm that the no endpoints available for service errors are still popping up in the logs? I wonder if it was something temporary

[cengizhanR]

@manuelbuil Thank you for your return, I am sharing my logs from the last 30 minutes below. As you said, it may be a temporary problem.However, rancher-weebhooks logs is a bit scary. I am sharing rancher-weebhooks logs too.

`Kube-api server logs

W0701 13:28:12.477357       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:28:12.578714215+03:00 W0701 13:28:12.578477       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:28:12.677159323+03:00 W0701 13:28:12.676993       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
W0701 13:28:12.777080       1 dispatcher.go:225] Failed calling webhook, failing closed rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/clusters.management.cattle.io?timeout=10s": no endpoints available for service "rancher-webhook"
2024-07-01T16:28:12.928086167+03:00 I0701 13:28:12.927941       1 controller.go:615] quota admission added evaluator for: namespaces
2024-07-01T16:28:13.367503199+03:00 I0701 13:28:13.367325       1 controller.go:615] quota admission added evaluator for: projects.management.cattle.io
2024-07-01T16:28:13.708736927+03:00 E0701 13:28:13.708598       1 available_controller.go:460] v1beta1.custom.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.custom.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
2024-07-01T16:28:13.853516245+03:00 I0701 13:28:13.853376       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 13:28:14.075080       1 controller.go:615] quota admission added evaluator for: deployments.apps
2024-07-01T16:28:14.075268142+03:00 I0701 13:28:14.075078       1 controller.go:615] quota admission added evaluator for: deployments.apps
I0701 13:28:14.239752       1 controller.go:615] quota admission added evaluator for: apps.catalog.cattle.io
I0701 13:28:14.931569       1 controller.go:615] quota admission added evaluator for: ingresses.networking.k8s.io
I0701 13:28:21.372080       1 controller.go:615] quota admission added evaluator for: poddisruptionbudgets.policy
2024-07-01T16:28:21.372210528+03:00 I0701 13:28:21.372087       1 controller.go:615] quota admission added evaluator for: poddisruptionbudgets.policy
I0701 13:28:27.669641       1 trace.go:236] Trace[1053553635]: "Update" accept:application/json, */*,audit-id:042f9f29-54df-4d44-8daa-086efd7147dd,client:10.0.0.3,api-group:coordination.k8s.io,api-version:v1,name:cattle-controllers,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cattle-controllers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (01-Jul-2024 13:28:24.354) (total time: 3315ms):
2024-07-01T16:28:27.669755604+03:00 Trace[1053553635]: ["GuaranteedUpdate etcd3" audit-id:042f9f29-54df-4d44-8daa-086efd7147dd,key:/leases/kube-system/cattle-controllers,type:*coordination.Lease,resource:leases.coordination.k8s.io 3315ms (13:28:24.354)
Trace[1053553635]:  ---"Txn call completed" 3314ms (13:28:27.669)]
2024-07-01T16:28:27.669762504+03:00 Trace[1053553635]: [3.31532299s] [3.31532299s] END
2024-07-01T16:28:27.671859937+03:00 I0701 13:28:27.670389       1 trace.go:236] Trace[1560741819]: "Get" accept:application/json, */*,audit-id:73d96afc-d493-40f3-a7f1-ede8c8c7478a,client:10.0.0.6,api-group:coordination.k8s.io,api-version:v1,name:driver-longhorn-io,subresource:,namespace:longhorn-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/longhorn-system/leases/driver-longhorn-io,user-agent:csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:24.424) (total time: 3245ms):
Trace[1560741819]: ---"About to write a response" 3245ms (13:28:27.670)
2024-07-01T16:28:27.671883337+03:00 Trace[1560741819]: [3.245674823s] [3.245674823s] END
2024-07-01T16:28:27.671886937+03:00 I0701 13:28:27.670773       1 trace.go:236] Trace[1892764832]: "Get" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:7d84dc0a-391b-4642-9db8-42bb709e1622,client:10.0.0.3,api-group:management.cattle.io,api-version:v3,name:u-mo773yttt4,subresource:,namespace:,protocol:HTTP/2.0,resource:userattributes,scope:resource,url:/apis/management.cattle.io/v3/userattributes/u-mo773yttt4,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:24.355) (total time: 3315ms):
Trace[1892764832]: [3.31537079s] [3.31537079s] END
2024-07-01T16:28:27.672132341+03:00 I0701 13:28:27.672025       1 trace.go:236] Trace[1380168316]: "List" accept:application/json, */*,audit-id:11902988-f543-4063-8dca-06158b0c5f61,client:10.42.3.5,api-group:,api-version:v1,name:,subresource:,namespace:cattle-fleet-local-system,protocol:HTTP/2.0,resource:pods,scope:namespace,url:/api/v1/namespaces/cattle-fleet-local-system/pods,user-agent:fleetcontroller/v0.0.0 (linux/amd64) kubernetes/$Format,verb:LIST (01-Jul-2024 13:28:24.355) (total time: 3316ms):
2024-07-01T16:28:27.672146641+03:00 Trace[1380168316]: ["List(recursive=true) etcd3" audit-id:11902988-f543-4063-8dca-06158b0c5f61,key:/pods/cattle-fleet-local-system,resourceVersion:,resourceVersionMatch:,limit:0,continue: 3316ms (13:28:24.355)]
2024-07-01T16:28:27.672150041+03:00 Trace[1380168316]: [3.316097401s] [3.316097401s] END
I0701 13:28:29.915994       1 trace.go:236] Trace[1341462078]: "Update" accept:application/json, */*,audit-id:fa82a29b-5332-4830-8c57-ae21de4682b9,client:10.42.0.3,api-group:longhorn.io,api-version:v1beta2,name:instance-manager-c96c134ffcb41f15abe2445293be32dd,subresource:status,namespace:longhorn-system,protocol:HTTP/2.0,resource:instancemanagers,scope:resource,url:/apis/longhorn.io/v1beta2/namespaces/longhorn-system/instancemanagers/instance-manager-c96c134ffcb41f15abe2445293be32dd/status,user-agent:longhorn-manager/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (01-Jul-2024 13:28:25.045) (total time: 4870ms):
2024-07-01T16:28:29.916171519+03:00 Trace[1341462078]: ["GuaranteedUpdate etcd3" audit-id:fa82a29b-5332-4830-8c57-ae21de4682b9,key:/longhorn.io/instancemanagers/longhorn-system/instance-manager-c96c134ffcb41f15abe2445293be32dd,type:*unstructured.Unstructured,resource:instancemanagers.longhorn.io 4869ms (13:28:25.046)
2024-07-01T16:28:29.916175719+03:00 Trace[1341462078]:  ---"Txn call completed" 4867ms (13:28:29.915)]
Trace[1341462078]: [4.870327612s] [4.870327612s] END
2024-07-01T16:28:29.968140315+03:00 I0701 13:28:29.967955       1 trace.go:236] Trace[781367366]: "Get" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:64039d1d-5bf7-4214-8707-db6b3d7d05f8,client:10.0.0.5,api-group:management.cattle.io,api-version:v3,name:token-2h685,subresource:,namespace:,protocol:HTTP/2.0,resource:tokens,scope:resource,url:/apis/management.cattle.io/v3/tokens/token-2h685,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:24.996) (total time: 4971ms):
2024-07-01T16:28:29.968160815+03:00 Trace[781367366]: [4.971673164s] [4.971673164s] END
I0701 13:28:29.968172       1 trace.go:236] Trace[1357902404]: "Get" accept:application/json, */*,audit-id:a4bd83ea-a907-4a60-9549-4e1b3a1ef648,client:10.0.0.3,api-group:coordination.k8s.io,api-version:v1,name:rke2-ingress-nginx-leader,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-ingress-nginx-leader,user-agent:nginx-ingress-controller/nginx-1.9.6-hardened1 (linux/amd64) ingress-nginx/git-05efe494a,verb:GET (01-Jul-2024 13:28:25.440) (total time: 4528ms):
Trace[1357902404]: ---"About to write a response" 4527ms (13:28:29.967)
Trace[1357902404]: [4.528000168s] [4.528000168s] END
2024-07-01T16:28:29.969044729+03:00 I0701 13:28:29.968292       1 trace.go:236] Trace[913699919]: "Get" accept:application/json, */*,audit-id:57399819-9d13-462f-b1f4-79f2aaf82e0a,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:rke2,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2,user-agent:rke2-supervisor@rke21/v1.30.0+rke2r1 (linux/amd64) rke2/60e06c4dbccff996f717af8f4c532971f57264b4,verb:GET (01-Jul-2024 13:28:25.675) (total time: 4292ms):
2024-07-01T16:28:29.969075329+03:00 Trace[913699919]: ---"About to write a response" 4292ms (13:28:29.968)
2024-07-01T16:28:29.969086529+03:00 Trace[913699919]: [4.292273756s] [4.292273756s] END
2024-07-01T16:28:29.969159530+03:00 I0701 13:28:29.968475       1 trace.go:236] Trace[1528589676]: "Get" accept:application/json, */*,audit-id:ebc7ded6-1805-43cb-b220-4355fe9af540,client:10.0.0.5,api-group:coordination.k8s.io,api-version:v1,name:fleet-controller-lock,subresource:,namespace:cattle-fleet-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/cattle-fleet-system/leases/fleet-controller-lock,user-agent:fleetcontroller/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:24.559) (total time: 5409ms):
2024-07-01T16:28:29.969165830+03:00 Trace[1528589676]: ---"About to write a response" 5409ms (13:28:29.968)
2024-07-01T16:28:29.969170331+03:00 Trace[1528589676]: [5.409298568s] [5.409298568s] END
2024-07-01T16:28:29.970959058+03:00 I0701 13:28:29.970856       1 trace.go:236] Trace[1162492182]: "Get" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:58a94389-661c-4d9e-8634-2664c6fcdc2d,client:10.0.0.5,api-group:management.cattle.io,api-version:v3,name:token-2h685,subresource:,namespace:,protocol:HTTP/2.0,resource:tokens,scope:resource,url:/apis/management.cattle.io/v3/tokens/token-2h685,use
r-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:25.572) (total time: 4398ms):
Trace[1162492182]: [4.398420282s] [4.398420282s] END
2024-07-01T16:28:29.971502666+03:00 I0701 13:28:29.971421       1 trace.go:236] Trace[1512449378]: "Update" accept:application/json, */*,audit-id:72429f0b-9d0f-403e-a22f-3b0bf58ffd9b,client:10.0.0.6,api-group:coordination.k8s.io,api-version:v1,name:driver-longhorn-io,subresource:,namespace:longhorn-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/longhorn-system/leases/driver-longhorn-io,user-agent:csi-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (01-Jul-2024 13:28:27.672) (total time: 2298ms):
2024-07-01T16:28:29.971515566+03:00 Trace[1512449378]: ["GuaranteedUpdate etcd3" audit-id:72429f0b-9d0f-403e-a22f-3b0bf58ffd9b,key:/leases/longhorn-system/driver-longhorn-io,type:*coordination.Lease,resource:leases.coordination.k8s.io 2298ms (13:28:27.672)
Trace[1512449378]:  ---"Txn call completed" 2297ms (13:28:29.971)]
Trace[1512449378]: [2.298674715s] [2.298674715s] END
I0701 13:28:29.971801       1 trace.go:236] Trace[367860627]: "Get" accept:application/json, */*,audit-id:6ae67da9-9bb5-4445-9bf8-722e1434f677,client:10.0.0.5,api-group:coordination.k8s.io,api-version:v1,name:cattle-controllers,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cattle-controllers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:26.289) (total time: 3682ms):
Trace[367860627]: ---"About to write a response" 3682ms (13:28:29.971)
2024-07-01T16:28:29.971891272+03:00 Trace[367860627]: [3.68221071s] [3.68221071s] END
I0701 13:28:29.971817       1 trace.go:236] Trace[17126681]: "Get" accept:application/json, */*,audit-id:081e2143-1944-4748-afae-b5a095cd1751,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:rke2-etcd,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-etcd,user-agent:rke2-supervisor@rke21/v1.30.0+rke2r1 (linux/amd64) rke2/60e06c4dbccff996f717af8f4c532971f57264b4,verb:GET (01-Jul-2024 13:28:26.670) (total time: 3300ms):
Trace[17126681]: ---"About to write a response" 3300ms (13:28:29.971)
2024-07-01T16:28:29.971904872+03:00 Trace[17126681]: [3.300906369s] [3.300906369s] END
2024-07-01T16:28:29.971907772+03:00 I0701 13:28:29.971844       1 trace.go:236] Trace[335858773]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:d3b18cd2-a6b6-444b-b51a-cd401e9e3f58,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:rke2-cloud-controller-manager,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-cloud-controller-manager,user-agent:cloud-controller-manager/v1.29.3 (linux/amd64) kubernetes/d9f75b6/leader-election,verb:GET (01-Jul-2024 13:28:26.362) (total time: 3609ms):
Trace[335858773]: ---"About to write a response" 3609ms (13:28:29.971)
Trace[335858773]: [3.609287893s] [3.609287893s] END
2024-07-01T16:28:29.972243878+03:00 I0701 13:28:29.972178       1 trace.go:236] Trace[1815815429]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:33f9c4a7-bd5d-4ba7-80d3-59eaee511df1,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:kube-scheduler,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler,user-agent:kube-scheduler/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b/leader-election,verb:GET (01-Jul-2024 13:28:26.129) (total time: 3842ms):
2024-07-01T16:28:29.972264078+03:00 Trace[1815815429]: ---"About to write a response" 3842ms (13:28:29.972)
2024-07-01T16:28:29.972266878+03:00 Trace[1815815429]: [3.842648668s] [3.842648668s] END
2024-07-01T16:28:29.972566083+03:00 I0701 13:28:29.972507       1 trace.go:236] Trace[180707811]: "Update" accept:application/vnd.kubernetes.protobuf, */*,audit-id:1ac12f2b-ab1d-496b-8e4e-1232fd86654a,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:apiserver-n2csatvhyzsprryaw5h6zfmm3m,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/apiserver-n2csatvhyzsprryaw5h6zfmm3m,user-agent:kube-apiserver/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b,verb:PUT (01-Jul-2024 13:28:26.089) (total time: 3883ms):
2024-07-01T16:28:29.972573883+03:00 Trace[180707811]: ["GuaranteedUpdate etcd3" audit-id:1ac12f2b-ab1d-496b-8e4e-1232fd86654a,key:/leases/kube-system/apiserver-n2csatvhyzsprryaw5h6zfmm3m,type:*coordination.Lease,resource:leases.coordination.k8s.io 3883ms (13:28:26.089)
2024-07-01T16:28:29.972576683+03:00 Trace[180707811]:  ---"Txn call completed" 3882ms (13:28:29.972)]
Trace[180707811]: [3.883168589s] [3.883168589s] END
2024-07-01T16:28:29.972874687+03:00 I0701 13:28:29.972812       1 trace.go:236] Trace[646211192]: "Get" accept:application/json, */*,audit-id:3130416f-b3a4-454c-9254-02cf51329a13,client:10.0.0.5,api-group:coordination.k8s.io,api-version:v1,name:snapshot-controller-leader,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/snapshot-controller-leader,user-agent:snapshot-controller/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:24.514) (total time: 5457ms):
Trace[646211192]: ---"About to write a response" 5457ms (13:28:29.972)
2024-07-01T16:28:29.972882587+03:00 Trace[646211192]: [5.457804112s] [5.457804112s] END
2024-07-01T16:28:29.973228393+03:00 I0701 13:28:29.973161       1 trace.go:236] Trace[1663202292]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:f633f634-8e4f-4f02-932c-47b28639bab4,client:10.0.0.3,api-group:,api-version:v1,name:rke22,subresource:,namespace:,protocol:HTTP/2.0,resource:nodes,scope:resource,url:/api/v1/nodes/rke22,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (01-Jul-2024 13:28:29.319) (total time: 653ms):
2024-07-01T16:28:29.973245493+03:00 Trace[1663202292]: ["GuaranteedUpdate etcd3" audit-id:f633f634-8e4f-4f02-932c-47b28639bab4,key:/minions/rke22,type:*core.Node,resource:nodes 652ms (13:28:29.320)
Trace[1663202292]:  ---"Txn call completed" 644ms (13:28:29.971)]
2024-07-01T16:28:29.973266893+03:00 Trace[1663202292]: [653.852617ms] [653.852617ms] END
2024-07-01T16:28:29.973269693+03:00 I0701 13:28:29.973182       1 trace.go:236] Trace[2038336456]: "Update" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:f6d9e344-4a04-403b-9790-74b511ac84d1,client:10.0.0.3,api-group:,api-version:v1,name:rke2agent1,subresource:,namespace:,protocol:HTTP/2.0,resource:nodes,scope:resource,url:/api/v1/nodes/rke2agent1,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:PUT (01-Jul-2024 13:28:29.319) (total time: 653ms):
2024-07-01T16:28:29.973271893+03:00 Trace[2038336456]: ["GuaranteedUpdate etcd3" audit-id:f6d9e344-4a04-403b-9790-74b511ac84d1,key:/minions/rke2agent1,type:*core.Node,resource:nodes 652ms (13:28:29.320)
2024-07-01T16:28:29.973274093+03:00 Trace[2038336456]:  ---"Txn call completed" 645ms (13:28:29.970)]
2024-07-01T16:28:29.973276693+03:00 Trace[2038336456]: [653.969018ms] [653.969018ms] END
I0701 13:28:30.184258       1 trace.go:236] Trace[291992444]: "Get" accept:application/json, */*,audit-id:16f2d4b0-19f6-4f15-86d1-85471c0b55ea,client:10.0.0.5,api-group:coordination.k8s.io,api-version:v1,name:external-resizer-driver-longhorn-io,subresource:,namespace:longhorn-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/longhorn-system/leases/external-resizer-driver-longhorn-io,user-agent:csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:28.945) (total time: 1238ms):
2024-07-01T16:28:30.184518730+03:00 Trace[291992444]: ---"About to write a response" 1238ms (13:28:30.184)
2024-07-01T16:28:30.184522130+03:00 Trace[291992444]: [1.238412672s] [1.238412672s] END
2024-07-01T16:28:30.184616231+03:00 I0701 13:28:30.184567       1 trace.go:236] Trace[2144264374]: "Get" accept:application/json, */*,audit-id:71ebbf63-6f46-4679-aee7-c4a40e1255e8,client:10.0.0.3,api-group:coordination.k8s.io,api-version:v1,name:cattle-controllers,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cattle-controllers,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:29.673) (total time: 511ms):
Trace[2144264374]: ---"About to write a response" 511ms (13:28:30.184)
Trace[2144264374]: [511.409535ms] [511.409535ms] END
2024-07-01T16:28:30.184706032+03:00 I0701 13:28:30.184661       1 trace.go:236] Trace[1600624230]: "Get" accept:application/vnd.kubernetes.protobuf, application/json,audit-id:6e758cea-cd7f-49ed-bfd3-7722f5ac3547,client:10.0.0.3,api-group:management.cattle.io,api-version:v3,name:u-mo773yttt4,subresource:,namespace:,protocol:HTTP/2.0,resource:userattributes,scope:resource,url:/apis/management.cattle.io/v3/userattributes/u-mo773yttt4,user-agent:rancher/v0.0.0 (linux/amd64) kubernetes/$Format,verb:GET (01-Jul-2024 13:28:27.679) (total time: 2505ms):
Trace[1600624230]: [2.505363282s] [2.505363282s] END
2024-07-01T16:28:30.184941836+03:00 I0701 13:28:30.184842       1 trace.go:236] Trace[544932985]: "Get" accept:application/json, */*,audit-id:b3a10873-0323-4b75-b707-c51147b92727,client:10.0.0.5,api-group:coordination.k8s.io,api-version:v1,name:rke2-ingress-nginx-leader,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-ingress-nginx-leader,user-agent:nginx-ingress-controller/nginx-1.9.6-hardened1 (linux/amd64) ingress-nginx/git-05efe494a,verb:GET (01-Jul-2024 13:28:28.892) (total time: 1292ms):
2024-07-01T16:28:30.184963836+03:00 Trace[544932985]: ---"About to write a response" 1292ms (13:28:30.184)
Trace[544932985]: [1.292605402s] [1.292605402s] END
2024-07-01T16:28:30.185611146+03:00 I0701 13:28:30.185509       1 trace.go:236] Trace[1241503644]: "Get" accept:application/vnd.kubernetes.protobuf, */*,audit-id:be323e2a-0c29-47b0-b125-407b58090998,client:127.0.0.1,api-group:coordination.k8s.io,api-version:v1,name:kube-controller-manager,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager,user-agent:kube-controller-manager/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b/leader-election,verb:GET (01-Jul-2024 13:28:28.344) (total time: 1841ms):
Trace[1241503644]: ---"About to write a response" 1841ms (13:28:30.185)
2024-07-01T16:28:30.185634047+03:00 Trace[1241503644]: [1.841324908s] [1.841324908s] END
2024-07-01T16:28:30.185929851+03:00 I0701 13:28:30.185718       1 trace.go:236] Trace[1669531896]: "Delete" accept:application/json, */*,audit-id:ca15aa93-ac33-4906-b4c2-043ef6bfc5de,client:10.42.3.5,api-group:,api-version:v1,name:fleet-agent-596955f45-2dmh6,subresource:,namespace:cattle-fleet-local-system,protocol:HTTP/2.0,resource:pods,scope:resource,url:/api/v1/namespaces/cattle-fleet-local-system/pods/fleet-agent-596955f45-2dmh6,user-agent:fleetcontroller/v0.0.0 (linux/amd64) kubernetes/$Format,verb:DELETE (01-Jul-2024 13:28:27.680) (total time: 2505ms):
2024-07-01T16:28:30.185941451+03:00 Trace[1669531896]: ---"Object deleted from database" 2504ms (13:28:30.185)
Trace[1669531896]: [2.505446682s] [2.505446682s] END
2024-07-01T16:28:30.186247956+03:00 I0701 13:28:30.185543       1 trace.go:236] Trace[1527018277]: "Get" accept:application/json, */*,audit-id:ce0aed02-faa1-44e9-b9fd-4a8ef3d67873,client:10.42.0.4,api-group:coordination.k8s.io,api-version:v1,name:rke2-ingress-nginx-leader,subresource:,namespace:kube-system,protocol:HTTP/2.0,resource:leases,scope:resource,url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/rke2-ingress-nginx-leader,user-agent:nginx-ingress-controller/nginx-1.9.6-hardened1 (linux/amd64) ingress-nginx/git-05efe494a,verb:GET (01-Jul-2024 13:28:28.958) (total time: 1226ms):
2024-07-01T16:28:30.186263356+03:00 Trace[1527018277]: ---"About to write a response" 1226ms (13:28:30.185)
2024-07-01T16:28:30.186266456+03:00 Trace[1527018277]: [1.226819595s] [1.226819595s] END
I0701 13:28:30.220630       1 trace.go:236] Trace[311132634]: "List" accept:application/json;as=Table;v=v1;g=meta.k8s.io,application/json;as=Table;v=v1beta1;g=meta.k8s.io,application/json,audit-id:83472523-9c96-4074-8b2e-a21f53acbef3,client:127.0.0.1,api-group:,api-version:v1,name:,subresource:,namespace:,protocol:HTTP/2.0,resource:pods,scope:cluster,url:/api/v1/pods,user-agent:kubectl/v1.30.0+rke2r1 (linux/amd64) kubernetes/7c48c2b,verb:LIST (01-Jul-2024 13:28:29.295) (total time: 925ms):
2024-07-01T16:28:30.220778185+03:00 Trace[311132634]: ["List(recursive=true) etcd3" audit-id:83472523-9c96-4074-8b2e-a21f53acbef3,key:/pods,resourceVersion:,resourceVersionMatch:,limit:500,continue: 924ms (13:28:29.295)]
Trace[311132634]: [925.026571ms] [925.026571ms] END
2024-07-01T16:28:31.711273219+03:00 I0701 13:28:31.711131       1 controller.go:615] quota admission added evaluator for: clusterregistrations.fleet.cattle.io
2024-07-01T16:28:34.979991495+03:00 I0701 13:28:34.979881       1 controller.go:615] quota admission added evaluator for: snapshots.longhorn.io
I0701 13:28:34.989655       1 controller.go:615] quota admission added evaluator for: volumeattachments.longhorn.io
I0701 13:28:37.238921       1 controller.go:615] quota admission added evaluator for: jobs.batch
2024-07-01T16:28:49.880483365+03:00 I0701 13:28:49.880352       1 controller.go:615] quota admission added evaluator for: daemonsets.apps
2024-07-01T16:28:49.897582527+03:00 I0701 13:28:49.897412       1 controller.go:615] quota admission added evaluator for: servicemonitors.monitoring.coreos.com
2024-07-01T16:30:32.841247486+03:00 I0701 13:30:32.841128       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
E0701 13:30:32.841793       1 writers.go:68] error encountered while streaming results via websocket: use of closed network connection
2024-07-01T16:30:32.842018098+03:00 I0701 13:30:32.841934       1 trace.go:236] Trace[1034370929]: "Get" accept:,audit-id:a5553a6d-7134-463f-ba0f-5df7e26c6a0e,client:10.0.0.1,api-group:,api-version:v1,name:kube-apiserver-rke21,subresource:log,namespace:kube-system,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/kube-apiserver-rke21/log,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36,verb:CONNECT (01-Jul-2024 13:29:21.688) (total time: 71153ms):
2024-07-01T16:30:32.842039498+03:00 Trace[1034370929]: ---"Writing http response done" 71150ms (13:30:32.841)
2024-07-01T16:30:32.842044398+03:00 Trace[1034370929]: [1m11.15376725s] [1m11.15376725s] END
2024-07-01T16:32:32.932210936+03:00 I0701 13:32:32.932075       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:34:32.928751439+03:00 I0701 13:34:32.928589       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
E0701 13:34:53.060149       1 writers.go:68] error encountered while streaming results via websocket: use of closed network connection
2024-07-01T16:34:53.060489350+03:00 I0701 13:34:53.060429       1 trace.go:236] Trace[364773538]: "Get" accept:,audit-id:dabeb6f5-786c-4a06-9a66-6ff0a0948b5a,client:10.0.0.1,api-group:,api-version:v1,name:etcd-rke21,subresource:log,namespace:kube-system,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/etcd-rke21/log,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36,verb:CONNECT (01-Jul-2024 13:34:02.999) (total time: 50060ms):
2024-07-01T16:34:53.060498750+03:00 Trace[364773538]: ---"Writing http response done" 50057ms (13:34:53.060)
2024-07-01T16:34:53.060501950+03:00 Trace[364773538]: [50.060741212s] [50.060741212s] END
2024-07-01T16:35:32.867129273+03:00 I0701 13:35:32.866945       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 13:37:32.929521       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:39:32.935326767+03:00 I0701 13:39:32.935113       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:40:24.303993928+03:00 W0701 13:40:24.303881       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
I0701 13:41:32.929809       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
W0701 13:42:57.185673       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-07-01T16:43:32.931495357+03:00 I0701 13:43:32.931365       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:44:32.931090688+03:00 I0701 13:44:32.930924       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:46:32.926890097+03:00 I0701 13:46:32.926763       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:46:54.058131509+03:00 E0701 13:46:54.057982       1 writers.go:68] error encountered while streaming results via websocket: use of closed network connection
I0701 13:46:54.058155       1 trace.go:236] Trace[1874148400]: "Get" accept:,audit-id:995bb308-5a2b-402a-8370-692eae072351,client:10.0.0.1,api-group:,api-version:v1,name:kube-apiserver-rke22,subresource:log,namespace:kube-system,protocol:HTTP/1.1,resource:pods,scope:resource,url:/api/v1/namespaces/kube-system/pods/kube-apiserver-rke22/log,user-agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36,verb:CONNECT (01-Jul-2024 13:45:14.311) (total time: 99746ms):
2024-07-01T16:46:54.058200710+03:00 Trace[1874148400]: ---"Writing http response done" 99742ms (13:46:54.058)
2024-07-01T16:46:54.058204310+03:00 Trace[1874148400]: [1m39.746281363s] [1m39.746281363s] END
I0701 13:48:32.929667       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 13:50:32.863348       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 13:52:32.934981       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:52:51.516968447+03:00 W0701 13:52:51.516750       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
I0701 13:54:32.929337       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:55:32.884674877+03:00 I0701 13:55:32.884496       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T16:56:06.937859466+03:00 W0701 13:56:06.937714       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
I0701 13:57:32.923559       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 13:59:32.934312       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:01:32.934759077+03:00 I0701 14:01:32.934632       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:01:48.388624796+03:00 W0701 14:01:48.388480       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-07-01T17:03:32.930869991+03:00 I0701 14:03:32.930727       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:04:31.507292964+03:00 I0701 14:04:31.507127       1 controller.go:615] quota admission added evaluator for: networkpolicies.networking.k8s.io
2024-07-01T17:05:32.887145688+03:00 I0701 14:05:32.886870       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 14:06:32.933740       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:08:32.925956301+03:00 I0701 14:08:32.925809       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:10:30.101197595+03:00 W0701 14:10:30.101017       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
I0701 14:10:32.894778       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:12:32.933793549+03:00 I0701 14:12:32.933662       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 14:14:32.935356       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
W0701 14:16:08.848673       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-07-01T17:16:32.926939802+03:00 I0701 14:16:32.926754       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 14:17:32.933661       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:19:32.926886672+03:00 I0701 14:19:32.926756       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:21:32.936191198+03:00 I0701 14:21:32.936021       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:23:32.923385878+03:00 I0701 14:23:32.923254       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
W0701 14:25:12.272008       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-07-01T17:25:32.913908210+03:00 I0701 14:25:32.913726       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 14:27:32.924035       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:28:32.928775388+03:00 I0701 14:28:32.928640       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 14:30:32.927649       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:32:32.924760032+03:00 I0701 14:32:32.924621       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:34:32.933989781+03:00 I0701 14:34:32.933848       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:36:32.927491483+03:00 I0701 14:36:32.927353       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
E0701 14:37:33.189170       1 watch.go:250] http2: stream closed
2024-07-01T17:38:32.935841717+03:00 I0701 14:38:32.935694       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:39:32.932105511+03:00 I0701 14:39:32.931870       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
W0701 14:40:00.327434       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-07-01T17:41:32.930993397+03:00 I0701 14:41:32.930877       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
I0701 14:43:32.952929       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:44:14.915136448+03:00 W0701 14:44:14.914955       1 watcher.go:338] watch chan error: etcdserver: mvcc: required revision has been compacted
2024-07-01T17:45:32.945422530+03:00 I0701 14:45:32.945285       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:47:32.938758132+03:00 I0701 14:47:32.938511       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
2024-07-01T17:49:32.948193418+03:00 I0701 14:49:32.948027       1 handler.go:286] Adding GroupVersion custom.metrics.k8s.io v1beta1 to ResourceManager
``
This is rancher-weebhook logs below,
^+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

``
time="2024-07-01T13:27:23Z" level=info msg="Rancher-webhook version v0.4.5 (b6b5ef9) is starting"
2024-07-01T16:27:53.670770330+03:00 time="2024-07-01T13:27:53Z" level=info msg="Waiting for server to become available: Get \"https://10.43.0.1:443/version\": dial tcp 10.43.0.1:443: i/o timeout"
2024-07-01T16:27:55.679094862+03:00 time="2024-07-01T13:27:55Z" level=info msg="could not read client CA file at /tmp/k8s-webhook-server/client-ca/ca.crt, incoming requests will not be authenticated"
2024-07-01T16:27:55.685291559+03:00 time="2024-07-01T13:27:55Z" level=info msg="Active TLS secret cattle-system/cattle-webhook-tls (ver=1076152) (count 1): map[listener.cattle.io/cn-rancher-webhook.cattle-system.svc:rancher-webhook.cattle-system.svc listener.cattle.io/fingerprint:SHA1=B3CC32F195B2BF556ABBFCA978A80EE552214120]"
2024-07-01T16:27:55.685497362+03:00 time="2024-07-01T13:27:55Z" level=info msg="Listening on :9443"
2024-07-01T16:27:56.200850654+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller"
2024-07-01T16:27:56.200886355+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller"
2024-07-01T16:27:56.200981156+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=ProjectRoleTemplateBinding controller"
2024-07-01T16:27:56.201061257+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=PodSecurityAdmissionConfigurationTemplate controller"
2024-07-01T16:27:56.201153459+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting apiregistration.k8s.io/v1, Kind=APIService controller"
2024-07-01T16:27:56.201232160+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=Cluster controller"
2024-07-01T16:27:56.201347362+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=Role controller"
2024-07-01T16:27:56.201377262+03:00 t
ime="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=RoleTemplate controller"
2024-07-01T16:27:56.201566465+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=GlobalRole controller"
2024-07-01T16:27:56.201621366+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting provisioning.cattle.io/v1, Kind=Cluster controller"
2024-07-01T16:27:56.201634366+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=ClusterRoleTemplateBinding controller"
2024-07-01T16:27:56.201665867+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting /v1, Kind=Secret controller"
2024-07-01T16:27:56.201779469+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller"
2024-07-01T16:27:56.201784469+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=Project controller"
2024-07-01T16:27:56.201802669+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting apiextensions.k8s.io/v1, Kind=CustomResourceDefinition controller"
time="2024-07-01T13:27:56Z" level=info msg="Starting management.cattle.io/v3, Kind=Node controller"
2024-07-01T16:27:56.202260876+03:00 time="2024-07-01T13:27:56Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller"
2024-07-01T16:27:56.202417679+03:00 time="2024-07-01T13:27:56Z" level=info msg="Sleeping for 15 seconds then applying webhook config"
2024-07-01T16:27:56.206044736+03:00 time="2024-07-01T13:27:56Z" level=info msg="Updating TLS secret for cattle-system/cattle-webhook-tls (count: 1): map[listener.cattle.io/cn-rancher-webhook.cattle-system.svc:rancher-webhook.cattle-system.svc listener.cattle.io/fingerprint:SHA1=B3CC32F195B2BF556ABBFCA978A80EE552214120]"
2024-07-01T16:27:56.719661400+03:00 E0701 13:27:56.719525       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.720818818+03:00 E0701 13:27:56.720737       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.735079242+03:00 E0701 13:27:56.734959       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.735893955+03:00 E0701 13:27:56.735806       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.754171842+03:00 E0701 13:27:56.754052       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.757324791+03:00 E0701 13:27:56.757199       1 gvks.
go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.778027216+03:00 E0701 13:27:56.771330       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.778096217+03:00 E0701 13:27:56.773579       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.792699646+03:00 E0701 13:27:56.792494       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.792734147+03:00 E0701 13:27:56.792506       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.806210559+03:00 E0701 13:27:56.806061       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.807459378+03:00 E0701 13:27:56.807348       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.819189962+03:00 E0701 13:27:56.819071       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:56.820708       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.833936894+03:00 E0701 13:27:56.833793       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:56.841888519+03:00 E0701 13:27:56.841750       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
1
2024-07-01T16:27:57.092135648+03:00 E0701 13:27:57.092037       1 gvks.go:69] failed to sync schemas: unable to retrieve the s.k8s.io/v1beta1
2024-07-01T16:27:57.223573012+03:00 E0701 13:27:57.223483       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
mas: unable to retrieve the complete list of server
 APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.325346509+03:00 E0701 13:27:57.325233       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.332648       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.341191958+03:00 E0701 13:27:57.341099       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.348308       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.357344012+03:00 E0701 13:27:57.357254       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.360537362+03:00 E0701 13:27:57.360126       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.370843924+03:00 E0701 13:27:57.370752       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.371898440+03:00 E0701 13:27:57.371793       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.383013       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.383310820+03:00 E0701 13:27:57.383239       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.396358224+03:00 E0701 13:27:57.396271       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.396379225+03:00 E0701 13:27:57.396329       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.412358176+03:00 E0701 13:27:57.412272       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.413048686+03:00 E0701 13:27:57.412943       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.424634       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.425639884+03:00 E0701 13:27:57.425392       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.439913       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.442139443+03:00 E0701 13:27:57.442060       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.453497422+03:00 E0701 13:27:57.453373       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.456016361+03:00 E0701 13:27:57.455919       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.471431603+03:00 E0701 13:27:57.471323       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.471530005+03:00 E0701 13:27:57.471447       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.485739728+03:00 E0701 13:27:57.485629       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.485763428+03:00 E0701 13:27:57.485637       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.502187286+03:00 E0701 13:27:57.502088       1 gvks.go:69] fa
iled to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.502702494+03:00 E0701 13:27:57.502642       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.517152721+03:00 E0701 13:27:57.516948       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.518516442+03:00 E0701 13:27:57.518394       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.532754566+03:00 E0701 13:27:57.532633       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.534575095+03:00 E0701 13:27:57.534483       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.544539651+03:00 E0701 13:27:57.544442       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.546369980+03:00 E0701 13:27:57.546284       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.557246150+03:00 E0701 13:27:57.557149       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custo
m.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.560051195+03:00 E0701 13:27:57.559951       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.570321056+03:00 E0701 13:27:57.570238       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.576205448+03:00 E0701 13:27:57.576102       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.586770214+03:00 E0701 13:27:57.586643       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.594535536+03:00 E0701 13:27:57.594395       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.602614463+03:00 E0701 13:27:57.602339       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.608240551+03:00 E0701 13:27:57.608119       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.616957488+03:00 E0701 13:27:57.616858       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1
beta1
2024-07-01T16:27:57.621422358+03:00 E0701 13:27:57.621341       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.634920370+03:00 E0701 13:27:57.634813       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.640874864+03:00 E0701 13:27:57.640764       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.647118362+03:00 E0701 13:27:57.647020       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.654870183+03:00 E0701 13:27:57.654786       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.658687043+03:00 E0701 13:27:57.658612       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.667695685+03:00 E0701 13:27:57.667611       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.669534       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.681950508+03:00 E0701 13:27:57.681695       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.683005525+03:00 E0701 13:27:57.682757       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.699356382+03:00 E0701 13:27:57.699230       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.700749704+03:00 E0701 13:27:57.700653       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.715809140+03:00 E0701 13:27:57.715709       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.716001943+03:00 E0701 13:27:57.715945       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.727954631+03:00 E0701 13:27:57.727869       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.728513       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.745093       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.745854       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.757319492+03:00 E0701 13:27:57.757104       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.757726398+03:00 E0701 13:27:57.757645       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.770393097+03:00 E0701 13:27:57.770148       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.773353044+03:00 E0701 13:27:57.773241       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.783773       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.786974257+03:00 E0701 13:27:57.786870       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.795763       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.799345352+03:00 E0701 13:27:57.799164       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.808282892+03:00 E0701 13:27:57.808166       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.812699961+03:00 E0701 13:27:57.812607       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.824005039+03:00 E0701 13:27:57.823884       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.829040718+03:00 E0701 13:27:57.828584       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.839651184+03:00 E0701 13:27:57.839505       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.841310111+03:00 E0701 13:27:57.841236       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.851089364+03:00 E0701 13:27:57.851010       1 gvks.go:69] fa
iled to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.853703605+03:00 E0701 13:27:57.853627       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.863502959+03:00 E0701 13:27:57.863419       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.866109       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.876401861+03:00 E0701 13:27:57.876322       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.881996749+03:00 E0701 13:27:57.881891       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.887594937+03:00 E0701 13:27:57.887499       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.897724996+03:00 E0701 13:27:57.897643       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.899065117+03:00 E0701 13:27:57.898974       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.912349       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.914171454+03:00 E0701 13:27:57.914082       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.933530458+03:00 E0701 13:27:57.933273       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.936776809+03:00 E0701 13:27:57.936597       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.951240637+03:00 E0701 13:27:57.951134       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.955584705+03:00 E0701 13:27:57.955324       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.967753796+03:00 E0701 13:27:57.967639       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:57.974566       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.983301040+03:00 E0701 13:27:57.983181       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:57.986581391+03:00 E0701 13:27:57.986511       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.000859616+03:00 E0701 13:27:58.000769       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.001446425+03:00 E0701 13:27:58.001378       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.015168       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.016363059+03:00 E0701 13:27:58.016244       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.034548245+03:00 E0701 13:27:58.033478       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.043539986+03:00 E0701 13:27:58.041821       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.052729230+03:00 E0701 13:27:58.052440       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.059367834+03:00 E0701 13:27:58.059271       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.066030839+03:00 E0701 13:27:58.065950       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.071181020+03:00 E0701 13:27:58.071099       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.080535767+03:00 E0701 13:27:58.080431       1 gvks.go:69] fa
iled to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.083813       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.092274351+03:00 E0701 13:27:58.092194       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.094835191+03:00 E0701 13:27:58.094764       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.104614       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.107494290+03:00 E0701 13:27:58.107402       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.116400230+03:00 E0701 13:27:58.116149       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.120094088+03:00 E0701 13:27:58.120000       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.138119       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.141837129+03:00 E0701 13:27:58.141747       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.152932203+03:00 E0701 13:27:58.149915       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.155371       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.165489       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.168235944+03:00 E0701 13:27:58.168126       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.179446520+03:00 E0701 13:27:58.179199       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.180090630+03:00 E0701 13:27:58.180019       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.235200995+03:00 E0701 13:27:58.235067       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.235916206+03:00 E0701 13:27:58.235844       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.247929695+03:00 E0701 13:27:58.247795       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.250842       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.259476776+03:00 E0701 13:27:58.259379       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.268517518+03:00 E0701 13:27:58.268379       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.277568       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.282877743+03:00 E0701 13:27:58.282776       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.292023487+03:00 E0701 13:27:58.291932       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.295313839+03:00 E0701 13:27:58.295228       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.306169809+03:00 E0701 13:27:58.306066       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.308247042+03:00 E0701 13:27:58.308155       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.318301800+03:00 E0701 13:27:58.318190       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.320708       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.329123370+03:00 E0701 13:27:58.329039       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.332458122+03:00 E0701 13:27:58.332360       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.341378462+03:00 E0701 13:27:58.341280       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.350258801+03:00 E0701 13:27:58.350159       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.356917006+03:00 E0701 13:27:58.356831       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.361017670+03:00 E0701 13:27:58.360938       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.370437818+03:00 E0701 13:27:58.370243       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.373107960+03:00 E0701 13:27:58.372977       1 gvks.go:69] fa
iled to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.382817413+03:00 E0701 13:27:58.382714       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.383401       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.396416626+03:00 E0701 13:27:58.396307       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.396626       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.410807852+03:00 E0701 13:27:58.410702       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.410922654+03:00 E0701 13:27:58.410844       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.425937590+03:00 E0701 13:27:58.425830       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.426204694+03:00 E0701 13:27:58.426139       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.442359147+03:00 E0701 13:27:58.442241       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.442393248+03:00 E0701 13:27:58.442363       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.455554555+03:00 E0701 13:27:58.455414       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.455647756+03:00 E0701 13:27:58.455570       1 gvks.go:69] fa
iled to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.467413541+03:00 E0701 13:27:58.467215       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.468092652+03:00 E0701 13:27:58.468029       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.480450946+03:00 E0701 13:27:58.480318       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.480783       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.493168245+03:00 E0701 13:27:58.493057       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.493190346+03:00 E0701 13:27:58.493058       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.504777027+03:00 E0701 13:27:58.504655       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.506293851+03:00 E0701 13:27:58.506202       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.516366309+03:00 E0701 13:27:58.516267       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.518509043+03:00 E0701 13:27:58.518409       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.533956186+03:00 E0701 13:27:58.533861       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.535797115+03:00 E0701 13:27:58.535726       1 gvks.go:69] fa
iled to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.546986190+03:00 E0701 13:27:58.546862       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.547861       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.559451486+03:00 E0701 13:27:58.559234       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.560850308+03:00 E0701 13:27:58.560778       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.575300035+03:00 E0701 13:27:58.575201       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.575657       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.600064       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.601441145+03:00 E0701 13:27:58.601344       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.616854       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.619283025+03:00 E0701 13:27:58.619079       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.633977656+03:00 E0701 13:27:58.633891       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.635932287+03:00 E0701 13:27:58.635686       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.652994       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.653863968+03:00 E0701 13:27:58.653804       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.668694801+03:00 E0701 13:27:58.668594       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.668967       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.685858       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.686017873+03:00 E0701 13:27:58.685956       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.702704335+03:00 E0701 13:27:58.702611       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.704361       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.718441582+03:00 E0701 13:27:58.718341       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.719857404+03:00 E0701 13:27:58.719760       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.731931894+03:00 E0701 13:27:58.731837       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.733579820+03:00 E0701 13:27:58.733505       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.747880       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.752131211+03:00 E0701 13:27:58.752041       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.763107       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.767528453+03:00 E0701 13:27:58.767418       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.777021402+03:00 E0701 13:27:58.776859       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.780238       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.793287557+03:00 E0701 13:27:58.792956       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.797793028+03:00 E0701 13:27:58.797705       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.806840270+03:00 E0701 13:27:58.806746       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.809154606+03:00 E0701 13:27:58.808974       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.818906660+03:00 E0701 13:27:58.818818       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.822123       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.830184537+03:00 E0701 13:27:58.829884       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.837212       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.848898430+03:00 E0701 13:27:58.848807       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.855638       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.861776033+03:00 E0701 13:27:58.861683       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.869644756+03:00 E0701 13:27:58.869570       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.873251       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.881244438+03:00 E0701 13:27:58.881156       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.885482805+03:00 E0701 13:27:58.885352       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.895589364+03:00 E0701 13:27:58.895485       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.903739692+03:00 E0701 13:27:58.903644       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.908833       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.917134402+03:00 E0701 13:27:58.917031       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.923290398+03:00 E0701 13:27:58.923191       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.938236533+03:00 E0701 13:27:58.938023       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
E0701 13:27:58.941523       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
2024-07-01T16:27:58.954896395+03:00 E0701 13:27:58.954794       1 gvks.go:69] failed to sync schemas: unable to retrieve the complete list of server APIs: custom.metrics.k8s.io/v1beta1: stale GroupVersion discovery: custom.metrics.k8s.io/v1beta1
```

[manuelbuil]

How did you install rke2? Can you share the config?

[cengizhanR]

Of course, @manuelbuil Thank you for your return.
Server configration

`[root@rke21 neuVector]# cat /etc/rancher/rke2/config.yaml
system-default-registry: 10.0.0.10:5000
secrets-encryption: true
token: fuzzybunnyslippers
tls-san:
  - rancher.exemple.com
node-ip: "10.0.0.2"
etcd-expose-metrics: true
write-kubeconfig-mode: "0600"
use-service-account-credentials: true
kube-controller-manager-arg:
- bind-address=127.0.0.1
- use-service-account-credentials=true
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
kube-scheduler-arg:
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
kube-apiserver-arg:
- tls-min-version=VersionTLS12
- tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
- authorization-mode=RBAC,Node
- anonymous-auth=false
- audit-policy-file=/etc/rancher/rke2/audit-policy.yaml
- audit-log-mode=blocking-strict
- audit-log-maxage=30
- audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log
kubelet-arg:
- protect-kernel-defaults=true
- read-only-port=0
- authorization-mode=Webhook
- streaming-connection-idle-timeout=5m
- minimum-container-ttl-duration=10s
- maximum-dead-containers-per-container=2
- maximum-dead-containers=240
- image-gc-high-threshold=85
- image-gc-low-threshold=80
`

Agent configrations

`[root@rke2agent1 bin]# cat /etc/rancher/rke2/config.yaml
server: https://rancher.exemple.com:9345
system-default-registry: 10.0.0.10:5000
token: fuzzybunnyslippers
write-kubeconfig-mode: "0600"
node-ip: "10.0.0.5"
kube-apiserver-arg:
- authorization-mode=RBAC,Node
kubelet-arg:
- protect-kernel-defaults=true
- read-only-port=0
- authorization-mode=Webhook
kubelet-arg:
- protect-kernel-defaults=true
- read-only-port=0
- authorization-mode=Webhook
- minimum-container-ttl-duration=10s
- maximum-dead-containers-per-container=2
- maximum-dead-containers=240
- image-gc-high-threshold=85
- image-gc-low-threshold=80
`