[Feature] Identify and apply changes on ray-cluster

[AvihaiSam]

Search before asking

• I had searched in the issues and found no similar feature requirement.

Description

when using helm upgrade on ray-cluster, the operator does not identify+apply changes.
instead - it does nothing.

workaround:
helm uninstall + helm install

Use case

change helm values -> helm upgrade

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[win5923]

Do you have a reproduction script?

[AvihaiSam]

helm install --set image.tag=X ray-cluster ...
helm upgrade --set image.tag=Y ....

head pod doesn't get replaced...

[win5923]

When I tried to perform helm upgrade to modify the image tag, I noticed the following log:

{"level":"info","ts":"2024-11-16T02:14:55.421Z","logger":"controllers.RayCluster","msg":"reconcileHeadService","RayCluster":{"name":"raycluster-kuberay","namespace":"default"},"reconcileID":"a6147489-026b-411c-bc99-c4cd92ad2d0a","1 head service found":"raycluster-kuberay-head-svc"}
{"level":"info","ts":"2024-11-16T02:14:55.422Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-kuberay","namespace":"default"},"reconcileID":"a6147489-026b-411c-bc99-c4cd92ad2d0a","Found 1 head Pod":"raycluster-kuberay-head-knskh","Pod status":"Running","Pod status reason":"","Pod restart policy":"Always","Ray container terminated status":"nil"}
{"level":"info","ts":"2024-11-16T02:14:55.422Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-kuberay","namespace":"default"},"reconcileID":"a6147489-026b-411c-bc99-c4cd92ad2d0a","head Pod":"raycluster-kuberay-head-knskh","shouldDelete":false,"reason":"KubeRay does not need to delete the head Pod raycluster-kuberay-head-knskh. The Pod status is Running, and the Ray container terminated status is nil."}
{"level":"info","ts":"2024-11-16T02:14:55.422Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-kuberay","namespace":"default"},"reconcileID":"a6147489-026b-411c-bc99-c4cd92ad2d0a","desired workerReplicas (always adhering to minReplicas/maxReplica)":1,"worker group":"workergroup","maxReplicas":3,"minReplicas":1,"replicas":1}
{"level":"info","ts":"2024-11-16T02:14:55.422Z","logger":"controllers.RayCluster","msg":"reconcilePods","RayCluster":{"name":"raycluster-kuberay","namespace":"default"},"reconcileID":"a6147489-026b-411c-bc99-c4cd92ad2d0a","worker Pod":"raycluster-kuberay-workergroup-worker-4xsf9","shouldDelete":false,"reason":"KubeRay does not need to delete the worker Pod raycluster-kuberay-workergroup-worker-4xsf9. The Pod status is Running, and the Ray container terminated status is nil."}

This causes the old RayCluster pod to not be deleted. @kevin85421 Do we need to implement this functionality?

[andrewsykim]

It is expected beheavior that RayCluster does not automatically recreate Pods on changes to the API. However, it is supported to delete the old Pods and have KubeRay automatically recreate with the new spec. RayCluster automatically recreating Pods would be a new feature request we can consider.

[andrewsykim]

We are introducing a spec.upgradeStrategy API to RayService, I think we can adapt something similar for RayCluster?

[AvihaiSam]

We are introducing a spec.upgradeStrategy API to RayService, I think we can adapt something similar for RayCluster?

That sounds like a good plan.
I'd expect the operator to replace the cluster on every change...

[andrewsykim]

@kevin85421 @MortalHappiness what do we think about having the new upgrade strategy API in RayCluster too?

[MortalHappiness]

I think strategy==Recreate can be easily implemented, but it may be a little bit tricky to implement RollingUpgrade

Another solution is to introduce a feature gate or a boolean toggle in the RayCluster spec, indicating to the KubeRay operator whether the underlying pods should be recreated whenever the spec changes.