Jenkins initial admin password secret gets overwritten on every Kustomize build, losing the real login credential - this breaks the login shortly after deployment in GitOps CD systems like ArgoCD

[HariSekhon]

Describe the bug

When installing the Jenkins helm chart via Kustomize it generates a new admin password each time a kustomize build is run.

This overwrites the jenkins secret with a new random jenkins-admin-password, losing the real initial admin login password.

Version of Helm and Kubernetes

- Helm: version.BuildInfo{Version:"v3.14.0", GitCommit:"3fc9f4b2638e76f26739cd77c7017139be81d0ea", GitTreeState:"clean", GoVersion:"go1.21.6"}

- Kubernetes: Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.8-gke.1067004

Chart version

4.12.1

What happened?

When deploying Jenkins helm chart via ArgoCD + Kustomize, ArgoCD will `kustomize build`
periodically to diff and then apply the changes. 

This means the `jenkins` secret is getting replaced every few minutes which is wasteful and
so doing the usual trick to get it from the secret is invalidated within minutes of deployment,
locking out the human.

What you expected to happen?

I expected the Jenkins password to be idempotent

Perhaps the initial admin password should just be the hash of the entire config so that it is the same value on each run, at least for a while until the config is changed so that at least it's not overwriting the initial admin password so quickly when it is deployed via GitOps style CD systems like ArgoCD?

How to reproduce it

git clone https://github.com/HariSekhon/Kubernetes-configs.git k8s

cd k8s/jenkins/overlay

kustomize build --enable-helm | kubectl apply -f -

see the admin password:

kubectl get secret -n jenkins jenkins -o 'jsonpath={.data.jenkins-admin-user}' | base64 --decode

Then do another run of kustomize build like ArgoCD does every few minutes

See it will change the jenkins admin password:

kustomize build --enable-helm | kubectl diff -f -

apply:

kustomize build --enable-helm | kubectl apply -f -

see you have now lost the real jenkins admin password as the secret has been overwritten but this new password will not work to log in:

kubectl get secret -n jenkins jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode

It looks like a restart of the Jenkins pod overwrites the admin password with whatever the latest in the jenkins secret is, likely due to JCasC.

This appears to be a tricky transitory problem between mismatching secret and jenkins controller where there is no restart of the pod, made worse by using a GitOps system like ArgoCD which is reapplying so frequently rotating the secret password key but not restarting the pod (not that you'd want your jenkins controller going down frequently either).

Anything else we need to know?

I've disabled the Jenkins helm chart from recreating the secret by setting this in the chart values.yaml:

controller:
  admin:
    existingSecret: jenkins

[HariSekhon]

Related to helm/helm-www#1259 (comment)

The problem there is that Kustomize materialized Helm in such a way without the chart being able to check for an existing secret to reuse it.

Found from this relevant bit of code showing the logic:

charts/jenkins-4.12.1/jenkins/templates/_helpers.tpl :

{{/*
Returns the admin password
https://github.com/helm/charts/issues/5167#issuecomment-619137759
*/}}
{{- define "jenkins.password" -}}
  {{ if .Values.controller.adminPassword -}}
    {{- .Values.controller.adminPassword | b64enc | quote }}
  {{- else -}}
    {{- $secret := (lookup "v1" "Secret" .Release.Namespace (include "jenkins.fullname" .)).data -}}
    {{- if $secret -}}
      {{/*
        Reusing current password since secret exists
      */}}
      {{- index $secret ( .Values.controller.admin.passwordKey | default "jenkins-admin-password" ) -}}
    {{- else -}}
      {{/*
          Generate new password
      */}}
      {{- randAlphaNum 22 | b64enc | quote }}
    {{- end -}}
  {{- end -}}
{{- end -}}

[arms11]

We upgraded from 4.4.1 to 4.12.1 and while we are not using ArgoCD, I confirm the same behavior with our deployment. I will try the suggestion @HariSekhon thanks.

[arms11]

I am not able to successfully use the suggestion unfortunately. I get the error below:

0s          Warning   FailedMount              pod/jenkins-0                                        MountVolume.SetUp failed for volume "jenkins-secrets" : secret "jenkins" not found
0s          Warning   FailedMount              pod/jenkins-0                                        Unable to attach or mount volumes: unmounted volumes=[jenkins-secrets], unattached volumes=[jenkins-cache tmp-volume sc-config-volume jenkins-home kube-api-access-rsk85 jenkins-config jenkins-secrets]: timed out waiting for the condition
0s          Warning   FailedMount              pod/jenkins-0                                        MountVolume.SetUp failed for volume "jenkins-secrets" : secret "jenkins" not found

What is odd on the top of that is there is already a secret called jenkins which has the expected value. Also, setting value adminPassword: <desired or expected> also does not seem to work.

[HariSekhon]

Is it in the right namespace?

[arms11]

I have a jenkins named secret in namespace jenkins which is Release.Namespace.
I believe the below statement should be able to find it in the namespace:
{{- $secret := (lookup "v1" "Secret" .Release.Namespace (include "jenkins.fullname" .)).data -}}

My observation is if I do not apply the work-around then my admin user password becomes admin which is quite interesting!

The only other alternate for me is to update my Jenkins container (image) which has vulnerabilities patched and keep the helm chart version 4.4.1.

[HariSekhon]

Only if controller.adminPassword isn't defined in values.yaml by the look of the code:

{{- define "jenkins.password" -}}
  {{ if .Values.controller.adminPassword -}}
    {{- .Values.controller.adminPassword | b64enc | quote }}
  {{- else -}}
    {{- $secret := (lookup "v1" "Secret" .Release.Namespace (include "jenkins.fullname" .)).data -}}
    {{- if $secret -}}

[arms11]

Yup I have not defined that. Running the upgrade with controller.adminPassword overridden also does not rectify the behavior once in this state!