Manual sync does not respect sync waves?

[patrickjane]

I made an ArgoCD application which points to a repository containing a helm chart (app of apps pattern). This helm chart contains several sub-apps as well as custom resources, such as:

• Strimzi kafka operator (kind: Application / helm)
• Strimzi custom resources for Kafka cluster (kind: KafkaNodePool / kind: Kafka / kind: KafkaTopic)
• and several more (Elasticsearch, Prometheus, Kafka-UI, ...)

I defined the YAMLs & sync-waves so that the strimzi-operator is deployed first, as it will also pull in the corresponding CRDs for the kafka cluster definitions. I set the SkipDryRunOnMissingResource flag for the kafka cluster resources, as otherwise ArgoCD will complain about missing CRDs right away.

Now, when I have everything set on auto-sync (the root-application, as well as the sub-apps in the git repo under /apps), everything is working well.
However, what I actually want, is that the root-application is set on manual sync, but whenever it is manually synced, all sub-apps shall be automatically deployed, without any more manual interaction. I thus disabled auto-sync on the root-app, but kept auto-sync on the sub-apps.

This leads to the issue that once I press the sync button for the root-app, the strimzi-operator will popup and start to sync, however ArgoCD seems to disregard the sync waves, and instead continue to deploy the kafka cluster right away. Since it takes around a minute or so for the strimzi-operator (& the CRDs) to be deployed, ArgoCD will bail out and fail instantly when trying to sync the kafka cluster, complaining about missing CRDs (which is correct at this point).

So, unlike in full-auto mode, ArgoCD does not seem to honor the sync waves when the root-app is manually synced. Is this behaviour intended, or do I have a bug in my setup somewhere?

The root application looks like this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: support-services
  namespace: argocd
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: argocd
    name: in-cluster
  project: support-services
  source:
    path: apps
    repoURL: https://gitlab.mydomain.com/myproject/mypath/support-services.git
    targetRevision: HEAD
  syncPolicy: {}
    # automated:
    #   prune: true

In the referenced git repository, I have:

\apps
   \templates
      \strimzi-operator.yml         # kind: Application           sync-wave: 1
      \kafka-cluster.yml              # kind: KafkaNodePool, kind: Kafka  sync-wave: 2 / 3
      ... many more

   \values.yaml
   \Chart.yaml

The strimzi operator YAML looks like this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: strimzi-kafka-operator-{{ .Values.project_env_name }}
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
  finalizers:
  - resources-finalizer.argocd.argoproj.io
spec:
  project: support-services
  source:
    chart: strimzi-kafka-operator
    repoURL: https://strimzi.io/charts
    targetRevision: 0.42.0
    helm:
      releaseName: strimzi-kafka-operator
      parameters:
        - name: watchNamespaces
          value: "{ {{ .Values.project_prefix }}-{{ .Values.project_env_name }} }"
  destination:
    server: {{ .Values.spec.destination.server }}
    namespace: "{{ .Values.project_prefix }}-{{ .Values.project_env_name }}"
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
    automated:
      prune: true

The kafka-cluster.yml looks like this:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: kafka-nodepool
  labels:
    strimzi.io/cluster: "kafka-cluster"
  annotations:
    argocd.argoproj.io/sync-wave: "2"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
  namespace: "{{ .Values.project_prefix }}-{{ .Values.project_env_name }}"
spec:
  replicas: 3
  roles:
    - controller
    - broker
  storage:
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 10Gi
        deleteClaim: true
        kraftMetadata: shared
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: "kafka-cluster"
  annotations:
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
  namespace: "{{ .Values.project_prefix }}-{{ .Values.project_env_name }}"
spec:
  kafka:
    version: 3.7.1
    metadataVersion: 3.7-IV4
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    config:
      offsets.topic.replication.factor: 2
      transaction.state.log.replication.factor: 2
      default.replication.factor: 2
      min.insync.replicas: 1
  entityOperator:
    topicOperator: {}
    userOperator: {}