Proposal: Convert Armada objects to CRD/Kubernetes Native

[kannon92]

Queue Conversion to CRD

A major goal is to make deployment/support of Armada much easier. Currently, our queue object is a custom object defined in our submit.proto file. We are missing key lifecycle methods. Common operations such as editing or getting a list of created queues are missing from our API. We implement this API separately and store it in our internal data store for Armada. This epic should aim to start storing these objects in the api-server in Kubernetes.

This epic will include designing the new queue api and storing it as part of Kubernetes. This should allow us to support creation/managing of queue objects with standard kubernetes tooling rather than having to implement these features ourselves.

Long Term benefits could be tighter integration of queues with Kubernetes RBAC. It will also play nicer with creation of objects since we can use Kubernets APIs as is for deploying queue objects.

Convert Armada Job to be a Kubernetes Job

A bad practice for running Kubernetes workloads is to schedule naked pods on your cluster. Kubernetes is designed to work with higher level controllers that verify the contract of your application. Think of a deployment object with x amount of replicas. Kubernetes will always make sure your object has x amount of replicas so if a node goes down, kubernetes will restart those objects.

This pattern can be extended to batch jobs by using the job controller. There is some momentum around the Job API in native kubernetes. The job controller does provide retrying ability and should allow scheduling of pods on healthy nodes. This also allows us to use Job features that are being released to better support batch workloads. We also can take advantage of features like indexed jobs, retriably jobs, automatic retry of preemptions jobs. Moving to a Job API allows us to take advantage of future features that we can add to the kubernetes project.

I see two options for implementation:

1. We allow the submission of a batch job in our submit.proto.
2. We auto convert our pod spec to a job in the executor.

[d80tb7]

Thanks for raising this. I'm going to comment here only about moving from Armada jobs to a Kubernetes jobs- I'll leave the discussion on Queue Conversion to CRD to someone more knowledgeable than myself.

As you note, there are two places in the code where we currently use pods and could move to using jobs; the podspec present in our public api and the pods that we schedule onto the K8s clusters. Note that these two uses are independent. For example we could move our public api to take job objects while continuing to schedule pods or vice versa. As a result I'll discuss each use case separately.

Use of Job Object As The K8s Object Armada Schedules On The Cluster
I think in this case, this isn't something we want to do in the near future. In running Armada, we've seen that etcd experiences performance issues under situations of high pod churn, such as can be encountered when running batch workloads. As the failure of etcd can be catastrophic for the K8s cluster, we have to be mindful of its performance within Armada to the extent where we actually have code inside Armada to monitor its health and back off from scheduling more jobs if we think we're running jobs at a rate it can't sustain. If we were to move to job objects then we would be increasing the load on etcd each armada jobs would now consist of two K8s objects (a K8s Job and a K8s Pod) along with a job controller that would be querying etcd to reconcile the state between the two. The extra load on etcd here is likely to cause us pain and until this issue is resolved, it doesn't seem that it's worth switching until there is a compelling reason.

Use of Job Objects In the Armada Api
In this case the argument is more persuasive. Moving to an public api where users submit K8s jobs rather than armada jobs is an attractive proposition as it's likely to make third party integrations easier and reduce the barrier to entry for using Armada as a job submission tool. Unfortunately though, I'd still argue this isn't something we'd want to do right now for the following two reasons:

1. The current K8s Job object is a basic and underdeveloped api which in it's current form is more suited to running small scale cron-like jobs than large scale batch-compute jobs. As a result, we'd almost certainly have to implement our own Armada-specific extensions to it, which would largely negate the benefit of using a standard K8s object.
2. This API is likely to evolve relatively quickly in the future- as you point out the K8s batch community is well aware of its shortcomings and are looking to add new features to it in order to meet their needs. In general this is good for us, however moving to a new way of submitting jobs will be disruptive to our users and doubly so if it's to an api that is not yet mature.

I think in this case the best thing to do is to monitor what happens with K8s Jobs (and also with other related concepts such as podgroups) to see if they evolve to a point where they offer enough benefits/are stable enough that using them in the Armada API pffers a compelling case. Once this happens (and I'm hopeful in this case), it should be relatively straightforward to provide an API making use of them, especially if the core Armada functionality has reached a point where we no longer have any big outstanding issues.

[kannon92]

Use of Job Object As The K8s Object Armada Schedules On The Cluster
I think in this case, this isn't something we want to do in the near future. In running Armada, we've seen that etcd experiences performance issues under situations of high pod churn, such as can be encountered when running batch workloads. As the failure of etcd can be catastrophic for the K8s cluster, we have to be mindful of its performance within Armada to the extent where we actually have code inside Armada to monitor its health and back off from scheduling more jobs if we think we're running jobs at a rate it can't sustain. If we were to move to job objects then we would be increasing the load on etcd each armada jobs would now consist of two K8s objects (a K8s Job and a K8s Pod) along with a job controller that would be querying etcd to reconcile the state between the two. The extra load on etcd here is likely to cause us pain and until this issue is resolved, it doesn't seem that it's worth switching until there is a compelling reason.

I do understand that but it seems like there is maybe some issue with our infrastructure. Kubernetes docs state that they can support up to 5000 nodes and GKE does in fact support up to 15000 nodes on their clusters (https://cloud.google.com/kubernetes-engine/quotas). I believe there is some engineering effort in ETCD and others to get it to scale this high. But from what I understand, we aren't even close to either of these limits so it seems that maybe there is a discussion we should have in the community about how to run etcd/kubernetes for high throughput workflows. OpenAPI has two good blog posts mentioning what they did to scale to 2500 nodes (https://openai.com/blog/scaling-kubernetes-to-2500-nodes/) and 7500 nodes (https://openai.com/blog/scaling-kubernetes-to-7500-nodes/). It seems that there could be more work done to allow us to no longer worry about putting pressure on ETCD for the cluster we currently have.

And to your second point, our team is working with this Kubernetes community to better support Jobs in batch. I will be pushing this issue hopefully in the next few months: kubernetes/kubernetes#113211

One ask I have of you is what features would you like to see in the job api? If it isn't satisfactory for our use case, then I would love to hear what features you'd like to see from the Job-API to make adoption a no brainer.

[jimbobby5]

FWIW even at KubeCon when we heard of people stretching K8s to 5,000+ nodes they very quickly noted that as soon as you run anything meaningful on it, it can fall apart under its own weight. I've always found those kind of announcements about scale relating to nodes a little disingenuous. For us the scaling factor is much more about numbers of pods, and the rate of churn of these. I don't know for sure, but I'd be willing to bet OpenAI on those large clusters don't fill them with 1-core jobs that turn over relatively quickly.

[jakexks]

More specifically it's about the number of Kubernetes objects being ListAndWatched, which is basically the number of controllers and their scope. When a watch drops, controllers re-establish themselves by listing, which is an expensive etcd request. Once you reach breaking point, the thundering herd effect of these lists takes down your cluster and it's extremely difficult to recover. For example, we are using Calico internally. If calico loses track of nodes due to etcd being slow, it can garbage collect routes, taking down your entire cluster networking.

Something to note about the OpenAI blog post is their job pattern is very simple. A single pod owns an entire node - so they really scaled K8s to 7500 pods rather than nodes - which we're already exceeding. Despite this they had to shard the K8s API across different etcd clusters and run it on dedicated local fast storage to get it to work.

This ties in a little to your proposal, as doubling the number of watched objects to run a job could mean a large decrease in scheduling performance.

[jakexks]

Having Armada queues as K8s native objects seems like a nice feature. My thought is that it will require a fair bit of work.

The source of truth for generating CRDs is Go structs with extra metadata encoded in comments. Currently, as you say, it's protobuf, so this will need to be translated. The clients would have to be regenerated, everything using them would have to be updated.
I'm not familiar with the current queue state storage of Armada, but this would have to be updated to use K8s native status and conditions, rather than whatever the internal datastore is. Armada's queue manager would switch from an API server to a controller.

I think it is worth doing, but it needs careful planning.

[severinson]

Armada currently works like this:
system creating pod specs => Armada => per-cluster scheduler (e.g., kube-scheduler) => pod running on node

A job controller (such as Kueue) is a system that creates pods. With a bit of extra plumbing, there's no reason those pods can't be passed through Armada instead of being sent straight to kube-scheduler. In this way, we'd benefit from work done on job controllers without needing to implement our own.

I think Armada's niche is and should be efficiently scheduling lots of pods across many clusters.

I think having jobs be the unit that Armada schedules is wrong for the reasons given by @d80tb7 and @jakexks and I think Armada accepting jobs instead of pods is wrong for the reason given above.

I think having queues be CRDs would be nice. However, I think it may require a fair bit of work (as suggested by @jakexks ) and I think we can significantly improve how easy it is to setup and use Armada without making queues be CRDs.

[severinson]

Re etcd performance, I believe improving it is far from trivial and I doubt us spending time on it would be time well spent.

Recall that etcd implements total order broadcast via the Raft protocol, and key-value storage is implemented on top of total order broadcast. As a result, etcd provides excellent consistency, durability, and availability guarantees, but low throughput. That's a deliberate design trade-off. Other technologies make other design trade-offs to achieve higher throughput.

[suprjinx]

There is a project, "kine"(https://github.com/k3s-io/kine), which aims to provide alternatives to etcd by maintaining an etcd api in front of postgres, etc.

[jakexks]

Watches in kine are implemented as sql select statements polled at a fixed time interval. My guess is this scales more poorly with large numbers of controllers than etcd but it would require benchmarking to confirm

[dave-gantenbein]

This seems related, and possibly an area in which we could contribute: k3s-io/kine#20

[kannon92]

So from what I am hearing: Job API is out. Too much pressure on ETCD and that is bad.

For queueing, I hear that it would be nice but it may be a lot of work as queue's are already used.

Some areas that could make queue CRD more interesting is that I'd like to start implementing CRUD APIs for Queue (Create/Delete/Upgrade/Patch/List) and I think we can get away from storing our queue objects in Redis. I don't see any reason why we can't convert this to a CRD and get all the benefits of REST-APIs. Right now I was told implementing a list of queues is difficult with our current data structure and I don't think we have patch/apply either. These APIs should be simple to implement with a CRD.

[kannon92]

@d80tb7 followed up with me offline and gave me a list of features that he would like from the JobApi:

• Job Priority- there is priorityclass on the podspec but I don't think this is suitable for representing a job priority.
• Marking jobs (or even pods within jobs) as non-preemptible
• A first-class representation of distributed jobs (this is a big topic and includes things like gang scheduling, job discovery, failure conditions etc)
• Parameters to enable jobs to scale up and down.
• How one might run an array style job where a large number of jobs are spawned which are identical except for their command lines. Specifically we don't want the user to have to submit such jobs by copy-pasting the job yaml 100k times.
• Parameters to control job queuing- there's a lot encompassed by this, but discussion should start as to whether job queue should be a first class job object.
• Some way of specifying whether it's safe to have multiple instances of the job running in parallel.
• Some way of specifying whether a job must be run at most once, or at least once.
• Some sort of story around how Kubernetes objects associated with a job (services, secrets etc) fit in with the job lifecycle.
• How jobs could be associated with other jobs- think of a job that spawns other jobs.

[kannon92]

Some of these are already possible in Kubernetes 1.25:

• Marking jobs (or even pods within jobs) as non-preemptible
  • PreemptionPolicy: Never
• Some way of specifying whether it's safe to have multiple instances of the job running in parallel.
  • Parallelism flag is specifically designed for this. 1 means only one pod at once and more can be how many pods are allowed to run in parallel
• Some way of specifying whether a job must be run at most once, or at least once.
  • BackoffLimit: 1 or BackoffLimit > 1 would satisfy this.

I followed up with people in the k8-batch group to discuss the other options. One suggestion is to attend the batch-working group to discuss some of these in more details. There is interest in formulating these as feature ideas.