Proposal: Third-Party CRD Support For Armada API

[kannon92]

Motivation

Users want to be able to submit and monitor various types of Kubernetes objects. The Kubernetes community has started pushing for the concept of an “Operator”, which is a third-party custom resource definition with a corresponding controller. The community of these projects maintains and supports best practices for running this objects. We should consider how to design Armada to take third party objects.

There are way too many third party objects to support and GR/GR-OSS may not have experts on staff to enable Spark, Dask, Ray, MPI, etc on Kubernetes so we should turn to open source solutions to enable more use cases of Armada.

Community CRDs of Interest to Armada/GR

• MPIOperatorV2: https://github.com/kubeflow/mpi-operator
• KubeFlowOperators: https://github.com/kubeflow/training-operator
  • MPIV1
  • TensorFlowJobs
  • PyTorchJobs
  • PaddlePaddle
  • MXNet
  • XGBoost
• DaskOperator: https://github.com/dask/dask-kubernetes
• KubeRay: https://github.com/ray-project/kuberay
• SparkOperator: https://github.com/GoogleCloudPlatform/spark-on-k8s-operator
• ArgoWorkflows: https://github.com/argoproj/argo-workflows/

All of these have active community support (>50 contributors) and are being actively developed. These are not small projects and we should not try to replicate this with a Pod first API.

Kubeflow is in the process of migrating all their training operators to a single operator but the original implementation had each operator as a separate controller.

Issues on our repo:

• TFJob Support: How can I submit tf-job in armada? #536
• Dask Operator: Investigate the possibility of using published Helm charts to deploy Dask/Ray clusters #1212

Community Impact:

Batch-Processing-Gateway from Apple (https://github.com/apple/batch-processing-gateway) demonstrates how to run Spark jobs across multiple kubernetes cluster. However, this is a common pattern from many other communities that want to take these objects and run them in a multi-cluster way. Argo-Workflows are working on multi-cluster solution (argoproj/argo-workflows#3523)

In general, we should be thinking of ways to bring other batch-compute frameworks into the multi-cluster world.

High Level User Diagram

Sketch of Implementation:

Our Armada API can take a general object and it is sent to a Kubernetes cluster that has the corresponding controller deployed. I would think you would want the arbitrary object and a corresponding tag that says what type of object this is so that our executors could understand this object and our scheduler can send this object to the Kubernetes cluster that has the capability of running this object.

MCAD (https://github.com/IBM/multi-cluster-app-dispatcher) provides a nice idea of this in practice. They released a blog post talking about how to enable Ray and MCAD: https://www.anyscale.com/blog/gang-scheduling-ray-clusters-on-kubernetes-with-multi-cluster-app-dispatcher

[severinson]

I agree with the goal, but I think the proposed solution is hard to implement. Fundamentally, Armada has to understand all objects submitted to it, including the implications of scheduling them, e.g., the resources consumed by each CRD considered for scheduling. Recall that MCAD is purpose-built for Spark (I don't think it even does pods?). As I've suggested before, I recommend chatting with Sam re this. He has an idea based on hijacking the K8s scheduler that could achieve this goal without us having to manually implement support for every CRD we want to support.

[kannon92]

MCAD is a project from IBM focused on general scheduling/orchestrating of any object across multiple kubernetes clusters.

The have support for a general CRD which is sent to the worker kubernetes clusters. I believe you are thinking about the batch-processing-gateway from Apple.

I am hoping to find time to discuss Sam's idea. I'd like to get to using open source community operators for doing third party integrations so the UK team doesn't get saddled with maintaining homespun Dask/Ray integrations that are not ideal. I discussed with the dask-kubernetes creator and he suggested that we use the dask-operator for dask work. I'd really like to get to a pattern where GR users are using a open-source solution as the API so we can start funneling issues/bug-fixes to the open source community.

Having internal ways of submitting Dask/Ray/Spark/MPI jobs means that the internal team will have to support these ways or the users of Armada have to write their own custom tooling to do this. This isn't scalable in the long run.

[severinson]

Ah, sorry. You're quite right, I was thinking of the Apple project.
I haven't looked into MCAD much. I'll need to have a more careful look at it.