Admission check on free resources

[bgedik]

I have a cluster where the batch workloads share nodes with non-batch workloads. When Kueue admits a workload, it is possible that the quota specified in the Kueue queue config indicates sufficient resources are available, but some of the nodes might be used by other workloads that do not use the batch scheduler. Is there an admission check that I can introduce to check availability of resources (based on free resources and not just quota declared) before admission?

[tenzen-y]

Kueue can check the actual Node usages before the Workloads are admitted by ProvisioningRequest.
Even if you use the fixed size cluster, you can use the capacity-check class ProvisioningRequest so that you can check the actual cluster resources.

[bgedik]

Thanks @tenzen-y . Where can I find more about the ProvisioningRequest classes? We are running on GKE.

[bgedik]

Basically, I need an AC like this I assume:

apiVersion: kueue.x-k8s.io/v1beta1
kind: AdmissionCheck
metadata:
  name: training-cluster-queue-ac
spec:
  controllerName: kueue.x-k8s.io/provisioning-request
  parameters:
    apiGroup: kueue.x-k8s.io
    kind: ProvisioningRequestConfig
    name: training-cluster-resource-prc
---
apiVersion: kueue.x-k8s.io/v1beta1
kind: ProvisioningRequestConfig
metadata:
  name: training-cluster-resource-prc
spec:
  provisioningClassName: capacity-check.???
  managedResources:
  - nvidia.com/gpu

Not sure what the provisioningClassName should be.

Is it correct to assume that capacity-check check whether there are sufficient free resources to deploy all pods of the workload at this time?

[tenzen-y]

I'm not familiar with GKE, so I'm not sure how to perform ProvisioningRequest on GKE.
But, It seems that we can find the documentation here

• https://cloud.google.com/kubernetes-engine/docs/how-to/provisioningrequest
• https://github.com/kubernetes/autoscaler/blob/c27e40db0867fdfafaff54ee8d03e06a3180ebc0/cluster-autoscaler/FAQ.md?plain=1#L607-L621

@alculquicondor @mimowo @PBundyra @mwielgus @gabesaba @yaroslava-serdiuk

Note that ProvisioningClass name is check-capacity.autoscaling.x-k8s.io.

[tenzen-y]

Additionally, this troubleshooting guide may be helpful for you: https://kueue.sigs.k8s.io/docs/tasks/troubleshooting/troubleshooting_provreq/

[bgedik]

Looks like I need to have autoscaling enabled, even though I will have min=max setting, as I don’t want true autoscaling and only want a capacity check to be performed.

[bgedik]

Thanks @tenzen-y for all the help. I got this mostly working, but have a question on how frequently provisoning requests are refreshed.

I have a workload with the following state:

  Admission Checks:
    Last Transition Time:  2024-08-29T16:11:17Z
    Message:               Capacity is not found, CA will try to find it later.
    Name:                  training-cluster-queue-ac
    State:                 Pending
  Conditions:
    Last Transition Time:  2024-08-29T16:11:17Z
    Message:               Quota reserved in ClusterQueue training-cluster-queue
    Observed Generation:   1
    Reason:                QuotaReserved
    Status:                True
    Type:                  QuotaReserved
Events:
  Type    Reason                      Age    From                                   Message
  ----    ------                      ----   ----                                   -------
  Normal  QuotaReserved               9m19s  kueue-admission                        Quota reserved in ClusterQueue training-cluster-queue, wait time since queued was 1s
  Normal  ProvisioningRequestCreated  9m19s  kueue-provisioning-request-controller  Created ProvisioningRequest: "pytorchjob-bugra-gedik-64dde0e56887459e85c-n0-0-991a8-training-cluster-queue-ac-1"

You can see that "Capacity is not found, CA will try to find it later". This is true, as the capacity was not available (I was testing the capacity check so had another workload taking up resources). Now the capacity is available, but it took ~10m for the admission request to find the capacity.

I want that time reduced. Is it a configuration of the ProvisioningRequestConfig? I don't see it there.

The documentation at https://kueue.sigs.k8s.io/docs/concepts/admission_check/ has:

retryDelayMinutes (deprecated) - specifies how long to keep the workload suspended after a failed check (after it transitioned to False). After that the check state goes to “Unknown”. The default is 15 min.

But this seems to be about failed check and not a check that didn't find resources yet.

[alculquicondor]

That retry is entirely on the CA side, not Kueue. I don't think that can be configured, but we are working on improving the throughput.

cc @aleksandra-malinowska