Repeated errors in kueue-manager log when Kueue 0.9 is installed on Kubernetes 1.29

[dgrove-oss]

What happened:

I created a fresh kind 0.24 cluster with Kubernetes 1.27.17. I installed the Kueue 0.9 release on the cluster.

Every 30 seconds (+/-) the kueue-controller-manager log contains the following stanza:

W1108 22:04:17.191809       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.FlowSchema: the server could not find the requested resource
E1108 22:04:17.191842       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.FlowSchema: failed to list *v1.FlowSchema: the server could not find the requested resource" logger="UnhandledError"
W1108 22:04:23.165653       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E1108 22:04:23.165688       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource" logger="UnhandledError"
W1108 22:04:23.370784       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource
E1108 22:04:23.370836       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicyBinding: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource" logger="UnhandledError"
W1108 22:04:52.216220       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource
E1108 22:04:52.216261       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicy: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource" logger="UnhandledError"
W1108 22:04:58.002480       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource
E1108 22:04:58.002507       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.PriorityLevelConfiguration: failed to list *v1.PriorityLevelConfiguration: the server could not find the requested resource" logger="UnhandledError"
W1108 22:05:02.844004       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource
E1108 22:05:02.844044       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicyBinding: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource" logger="UnhandledError"

What you expected to happen:

I do not expect to see error logs when running Kueue 0.9.0 on Kubernetes 1.27. This is a regression vs. Kueue 0.8.3.

How to reproduce it (as minimally and precisely as possible):

Create 1.27 cluster with kind.

kubectl apply --server-side -f https://github.com/kubernetes-sigs/kueue/releases/download/v0.9.0/manifests.yaml

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.27.17
• Kueue version (use git describe --tags --dirty --always): 0.9.0
• Cloud provider or hardware configuration: MacOS (arm64)

[kannon92]

1.27 is out of support in open source. Kueue 0.9.0 was probably built with 1.31 apis so I’m not sure the skew will even be supporting at this point.

[dgrove-oss]

On Kubernetes 1.29 (which I believe is still in support until 2025-02-28), I get a reduced set of repeating error messages but there is still a problem:

W1111 18:40:52.560632       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource
E1111 18:40:52.560667       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicy: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource" logger="UnhandledError"
W1111 18:41:35.548617       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource
E1111 18:41:35.548652       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicy: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource" logger="UnhandledError"
W1111 18:41:36.038435       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource
E1111 18:41:36.038474       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicyBinding: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource" logger="UnhandledError"
W1111 18:42:11.505717       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource
E1111 18:42:11.505754       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicy: failed to list *v1.ValidatingAdmissionPolicy: the server could not find the requested resource" logger="UnhandledError"
W1111 18:42:15.087195       1 reflector.go:561] k8s.io/client-go/informers/factory.go:160: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource
E1111 18:42:15.087231       1 reflector.go:158] "Unhandled Error" err="k8s.io/client-go/informers/factory.go:160: Failed to watch *v1.ValidatingAdmissionPolicyBinding: failed to list *v1.ValidatingAdmissionPolicyBinding: the server could not find the requested resource" logger="UnhandledError"

[dgrove-oss]

I verified that on Kubernetes 1.30 there are no error messages in the log.

[dgrove-oss]

retitled issue to only mention supported Kubernetes version (1.29).

[mbobrovskyi]

As I can see here https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/, ValidatingAdmissionPolicy is stable starting from version v1.30. Kueue v0.9 build with v1.31 apis. That's why we have this error on 1.29.

[mimowo]

@mbobrovskyi do you know if this is just a weird logging, or ValidatingAdmissionPolicy does not work in Kueue in 1.29?
I would suppose it should still work even while in Beta, but then it is still surprising to see this error.

[dgrove-oss]

A quick glance at the 1.29 documentation seems to imply that although VAP was in Beta in 1.29, the feature gate still defaulted to false.

https://v1-29.docs.kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/

[kannon92]

Are you able to see this with just kind?

[dgrove-oss]

I was just using kind. (kind 0.25 with --image v1.29.10@sha256:3b2d8c31753e6c8069d4fc4517264cd20e86fd36220671fb7d0a5855103aa84b).

[mimowo]

A quick glance at the 1.29 documentation seems to imply that although VAP was in Beta in 1.29, the feature gate still defaulted to false.

So, the question if you can enable it on 1.29 to mitigate the issue.
Another question would be if Kueue can do something to stop logging or make it less often - I don't know for now.

[kannon92]

I reached out to @benluddy about this issue. This seems like an upstream issue and not really related to Kueue at the moment.

[benluddy]

Looks like kueue-controller-manager has started informers for the v1 VAP APIs, which did not exist in 1.27.