credentials.xml is kept from boot to boot even if the encryption keys seems to change, thus secrets inside becomes useless #1224
Comments
|
Can you supply some more detailed steps please. What is in your JCasC? JCasC should manage secrets like this just fine. Also Azure Key Vault can manage some basic secret types so you don't even need to load them into Jenkins at all: |
|
Hello,
Jcasc and Azure key vault are not changed between the boots. The only way I found to get a right credentials.xml (ie that jenkins get good credentials) and access to the credential inside jenkins script, is to delete credentials.xml before rebooting (thus casc generate it just fine, on subsequent boot). It is difficult for me to push here the casc file, not sure have the right for that (it is a production data) - I need to setup a specific test. Should casc have been overriding the credentials.xml at every boot ? Maybe there is a setting that is preventing that behavior to operate properly ? Thanks for your help ! |
|
Maybe I push the issue in the wrong place, it might be more an issue with the helm-chart ? Is there a way to move that ticket over there ? |
|
I've moved it for you. Can you provide a minimal case please? Yes on every startup the credentials should be reset. We need to see the config variables you've used, you can remove anything unrelated and anonymise it but just make sure it still reproduces the issue |
|
@timja please find attached our helm config, I removed any string linked to our infra (hoppe I did not let something critical). Logic should not be changed, but I removed cloud, permanent node and view definition from jcasc. |
|
you have a client id and client secret in the helm values, I hope those aren't valid! |
|
The Azure Key Vault credentials provider could be used for a number of these credentials so you don't actually need to load them into Jenkins: https://github.com/jenkinsci/azure-keyvault-plugin#azure-key-vault-credentials-provider You add tags to the credentials for things like the username and the type of Jenkins credential you want it to be. in terms of simplifying this its normally recommended you create your own values file that only has your customisations in it which then makes it easy to see whats changed. For an improvement, its better that you don't supply a service principal in the java opts and instead use a managed identity since its credential less and just needs to be assigned to the pod via workload identity. I can't see specifically anything causing your issue though. |
|
Is it normal that the helm chart is not removing the credentials.xml file at startup ? Or is not normal that the Jcasc is not updating it when rebooting (encoded value does not change while they change for the config.xml). |
Fuck I missed those. |
no it should update it on boot, are there logs on startup of it running? If you click reload on the configuration as code page what happens? |
|
@timja thanks to your simple suggestion to try to load the jcasc from the GUI. I figure out the source of the problem partially. Now that is fixed, I noticed a few other error message and upgrade a few plugin. Things start to operate properly, but not on first boot. On first boot, Azure Key vault data are loaded, but not the jenkins credentials build from it. I mean if I have credential declaration like that: - "basicSSHUserPrivateKey":
"description": "My key"
"id": "ValueEnvFromAzureKeyVault"
"privateKeySource":
"directEntry":
"privateKey": "${KeyEnvFromAzureKeyVault}"
"scope": "GLOBAL"
"username": "Jenkins"
"usernameSecret": true"privateKey" is loaded as null KeyEnvFromAzureKeyVault is at expected value (loaded from keyvault). If I load the jcasc again from the GUI, then privateKey is not empty. I need to understand what happens now on the initial boot (which after the plugin upgrade, is different than with the plugin version above). |
|
To be a little bit more precise, only the very first boot after deployment does not load properly the credentials. Azure credentials are there but jenkins one created from Azure credentials are empty. I notice following error in the log: 2024-10-23 09:43:14.396+0000 [id=37] INFO o.j.p.a.AzureKeyVaultSecretSource#reveal: No AzureKeyVault credentials found, skipping jcasc secret resolution If I just 'restart safely', every credentials are fine. |
Jenkins and plugins versions report
Jenkins with casc on kubernetes, credentials strings are empty when requested by jenkins scripts
I am using jenkins inside kubernetes, I use casc to load the controler configuration.
At first boot, everything is fine.
At second boot, all credentials are lost.
I noticed that the config.xml is regenerated at each boot, and that the hash of the secrets inside are different each time.
Concerning the credentials.xml file, it is only generate one time at first boot.
On second boot any credentials are empty when requested from the script console inside jenkins.
It is strange but it seems jenkins is regenerating the encryption keys at each boot. Thus, any configuration that is not newly generated contains hash that cannot be decoded by jenkins again. A I right ?
In addition I had a look to appy_config.sh (from init container), it is erasing all xml config file but not the credentials.xml... is it expected, shouldn't it be erased so that casc get a chance to create it newly (with its new encryption key) ?
Can you confirm the analyse is right: jenkins encryption key are generated again at each kubernetes deployment (each time I delete the pod) - I am not kubernetes specialist, sorry ?
Why is credentials.xml not removed by apply_config.sh of the init container ?
Or maybe I miss something in the configuration. But all our JCASC settings were working fine inside a VM. Not sure why casc is not overring credentials.xml at each boot. But on VM it is performing a kind of merge if I remember well, I guess that because the keys are already defined it is not generating again with is weird, but that's another problem probably.
What Operating System are you using (both controller, and any agents involved in the problem)?
jenkins/jenkins:2.440.1-jdk11
Reproduction steps
Expected Results
Step 5 should be like step 2
Actual Results
On step 5, credentials are empty string when requested from jenkins, even if the credentials.xml contains a hash for it. The hash is the same at each boot, but hash' in config.xml is changing at each boot.
Anything else?
Our secrets are stored in a Azure Key vault, we rely on jenkins casc mecanism to gather the secrets at boot time from the key vault. We are not using kubernetes secrets for that as, it is shared with existing jenkins on VM.
The only solution I am thinking about is to add one line in the apply_config.sh, or another init container.
I know jdk11 is old, migration to a more recent jdk is planned as soon as this behavior is fixed (we are stuck today with this problem).
The text was updated successfully, but these errors were encountered: