How can I adjust redis-ha with nodeAffinity?

[cychoi34]

Hi, Team!
Here’s a quick question: I’d like to set my Redis HA pods to be scheduled with nodeAffinity, but it’s not working as I expected.
All the other pods except for the Redis HA are deployed with the nodeAffinity.

  set {
    name  = "redis-ha.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key"
    value = "name"
  }
  set {
    name  = "redis-ha.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].operator"
    value = "In"
  }
  set {
    name  = "redis-ha.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].values[0]"
    value = "name-of-node"
  }

I’ve tried to deploy the pods with the set phrase using Terraform.
But all I got after executing terraform apply is the following error:
│ Error: template: argo-cd/charts/redis-ha/templates/redis-ha-statefulset.yaml:67:7: executing "argo-cd/charts/redis-ha/templates/redis-ha-statefulset.yaml" at <.>: wrong type for value; expected string; got map[string]interface {}

I’ve set redis-ha.affinity as follows, but I’m still stuck with the problem:

  set {
    name  = "redis-ha.affinity"
    value = <<EOF
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: Name
            operator: In
            values:
            - "name-of-node"
    EOF
  }

The following is the Kubernetes events log. There’s no Pod-AntiAffinity configuration I’ve set.

10s (x239 over 19h) Warning FailedScheduling Pod/argocd-redis-ha-haproxy-6cd77968c8-bhct2 0/18 nodes are available: 1 node(s) had untolerated taint {karpenter.sh/disruption: disrupting}, 15 node(s) didn't match Pod's node affinity/selector, 2 node(s) didn't match pod anti-affinity rules. preemption: 0/18 nodes are available: 16 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.
10s (x239 over 19h) Warning FailedScheduling Pod/argocd-redis-ha-server-0 0/18 nodes are available: 1 node(s) had untolerated taint {karpenter.sh/disruption: disrupting}, 15 node(s) didn't match Pod's node affinity/selector, 2 node(s) didn't match pod anti-affinity rules. preemption: 0/18 nodes are available: 16 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod.

After setting the replicas to 3 and adjusting the rolling update strategy to have a minimum unavailable pods of 1, I tried again, but encountered the following errors:

│ Error: Get "https://1OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO.sk1.ap-northeast-2.eks.amazonaws.com/api/v1/namespaces/argocd/services/argocd-redis-ha-announce-0": context deadline exceeded
│
│ with helm_release.argocd,
│ on eks_helm.tf line 135, in resource "helm_release" "argocd":
│ 135: resource "helm_release" "argocd" {
│
╵
╷
│ Error: karpenter/xpert-node-pool failed to fetch resource from kubernetes: the server could not find the requested resource
│
│ with kubectl_manifest.karpenter_node_pool,
│ on eks_kubectl_karpenter.tf line 9, in resource "kubectl_manifest" "karpenter_node_pool":
│ 9: resource "kubectl_manifest" "karpenter_node_pool" {
│
╵
╷
│ Error: karpenter/xpert-node-class failed to fetch resource from kubernetes: the server could not find the requested resource
│
│ with kubectl_manifest.karpenter_node_class,
│ on eks_kubectl_karpenter.tf line 25, in resource "kubectl_manifest" "karpenter_node_class":
│ 25: resource "kubectl_manifest" "karpenter_node_class" {

Thanks for your help!