Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disabling certs creation in v2.32.2 helm chart breaks the jaeger deployment #375

Open
AshutoshNirkhe opened this issue Jun 20, 2022 · 12 comments · May be fixed by #376
Open

Disabling certs creation in v2.32.2 helm chart breaks the jaeger deployment #375

AshutoshNirkhe opened this issue Jun 20, 2022 · 12 comments · May be fixed by #376
Labels
bug Something isn't working

Comments

@AshutoshNirkhe
Copy link

AshutoshNirkhe commented Jun 20, 2022

Describe the bug
v2.32.x seems to be introducing admission webhooks and associated certificates. But if I want to disable them all, its not possible to do so as per https://github.com/jaegertracing/helm-charts/blob/jaeger-operator-2.32.2/charts/jaeger-operator/templates/deployment.yaml#L52

Are they mandatory going forward ?
Ref - https://www.jaegertracing.io/docs/1.35/operator/#prerequisite

To Reproduce
Steps to reproduce the behavior:

  1. Disable webhooks and certificates via values.yaml
certs:
  certificate:
    create: false
  issuer:
    create: false
webhooks:
  mutatingWebhook:
    create: false
  service:
    create: false
  validatingWebhook:
    create: false
  1. Try to upgrade jaeger helm chart from 2.27.0 to 2.32.2

  2. Pod will stay in Waiting/ContainerCreating state forever and will show below message if we do kubectl describe on pod,

  Warning  FailedMount  50s (x10 over 5m)    kubelet            MountVolume.SetUp failed for volume "cert" : secret "jaeger-operator-service-cert" not found
  Warning  FailedMount  42s (x2 over 2m57s)  kubelet            Unable to attach or mount volumes: unmounted volumes=[cert], unattached volumes=[cert kube-api-access-7bkf8]: timed out waiting for the condition

Expected behavior
Currently deployment.yaml tries to load volume from certificate secret even if we disable cert creation. It needs to have the condition to check for that flag before volume/volume mount creation. That way, webhooks/certificates creation won't be a mandatory configuration.

Screenshots
If applicable, add screenshots to help explain your problem.

Version (please complete the following information):

  • OS: Linux Ubuntu
  • Jaeger version: 1.34.1
  • Deployment: Kubernetes v1.22

What troubleshooting steps did you try?
Try to follow https://www.jaegertracing.io/docs/latest/troubleshooting/ and describe how far you were able to progress and/or which steps did not work.

Additional context

@AshutoshNirkhe AshutoshNirkhe added the bug Something isn't working label Jun 20, 2022
AshutoshNirkhe added a commit to AshutoshNirkhe/jaegertracing that referenced this issue Jun 20, 2022
AshutoshNirkhe added a commit to AshutoshNirkhe/jaegertracing that referenced this issue Jun 20, 2022
@rotarur
Copy link

rotarur commented Jun 24, 2022

is there an ETA for this bug fix? We're facing the same issue

@AshutoshNirkhe
Copy link
Author

is there an ETA for this bug fix? We're facing the same issue

Seems like they don't want us to disable those. See #376 for details.

@avishefi
Copy link

avishefi commented Jul 10, 2022

There should be a way to disable creating the certificates and to provide the names of the relevant secrets created before running the jaeger-operator Helm chart.

@mjnagel
Copy link
Contributor

mjnagel commented Aug 26, 2022

There should be a way to disable creating the certificates and to provider the names of the relevant secrets created before running the jaeger-operator Helm chart.

@avishefi This is possible, reference the below values:

certs:
  issuer:
    create: false # defaults to true, but you can disable the chart creation of the issuer
    name: "" # leave empty
  certificate:
    create: false # defaults to true but you can disable the chart creation of the cert
    namespace: "jaeger" # namespace for the secret
    secretName: "my-secret" # secret with the cert

The other thing you may need depending on how you create the certs would be the clientConfig.caBundle value set in the webhook spec - which isn't currently exposed in values. I think if you make use of cert-manager this will all be handled for you via the annotations. Another option that we use is the webhook cert gen to dynamically generate these certs and then patch the webhooks (although this requires an extra job tied to the lifecycle of the chart).

You will need to have the cert no matter what, but you can absolutely disable the chart creating it.

@avishefi
Copy link

avishefi commented Aug 27, 2022

@mjnagel Thanks, I have raised it on the PR that introduced it.

As for the CA bundle - you are right, it is handled by the cert-manager annotation and the Helm chart currently doesn't support other implementations.

@razorsk8jz
Copy link

razorsk8jz commented Nov 7, 2022

There should be a way to disable creating the certificates and to provider the names of the relevant secrets created before running the jaeger-operator Helm chart.

@avishefi This is possible, reference the below values:

certs:
  issuer:
    create: false # defaults to true, but you can disable the chart creation of the issuer
    name: "" # leave empty
  certificate:
    create: false # defaults to true but you can disable the chart creation of the cert
    namespace: "jaeger" # namespace for the secret
    secretName: "my-secret" # secret with the cert

The other thing you may need depending on how you create the certs would be the clientConfig.caBundle value set in the webhook spec - which isn't currently exposed in values. I think if you make use of cert-manager this will all be handled for you via the annotations. Another option that we use is the webhook cert gen to dynamically generate these certs and then patch the webhooks (although this requires an extra job tied to the lifecycle of the chart).

You will need to have the cert no matter what, but you can absolutely disable the chart creating it.

I attempted to do this but I appear to get an error when i try to use the cert - x509: certificate signed by unknown authority"

The cert/secret are created using these commands

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=jaeger-operator-webhook-service.monitoring.svc" -addext "subjectAltName = DNS:jaeger-operator-webhook-service.monitoring.svc"

kubectl -n monitoring create secret tls jaeger-operator-cert --key="tls.key" --cert="tls.crt" --dry-run=client -o yaml | kubectl apply -f -

Any ideas are appreciated

@avishefi
Copy link

avishefi commented Nov 9, 2022

@razorsk8jz this is a self-signed certificate so it won't be recognized. You can either use a known CA or use a self-signed CA certificate as a flag to Jaeger's skip-host-verify flags.

@sergiomcalzada
Copy link

sergiomcalzada commented Jul 20, 2023

@razorsk8jz I'm stuck too in that part. where should set the flag skip-host-verify? can't find the exact value in the values.yaml from the operator chart

I can provide also my custom certificates generated externally for my own managed domain [using lets-encrypt]. But the required host jaeger-operator-webhook-service.[namespace].svc prevent generating.

@sergiomcalzada
Copy link

I was investigating, an has created a certificate using the self kubernetes CA [https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/]

The issue here is that the operator, don't accept the kubernetes GA /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Will be great to have a flag to include that ca as valid. Or at least being able to customize the valid ca

@shicli
Copy link

shicli commented Jul 28, 2023

Hi @mjnagel ,I am deploying the jaeger operator service through helm and would like to disable cert-magger as we have our own TLS service. But I disabled them in Value, but I still check Cert when starting the service. May I know how to handle this change?
certs:
issuer:
create: false
name: ""
certificate:
create: false
namespace: ""
secretName: ""
issuerKind: Issuer

webhooks:
mutatingWebhook:
create: false
validatingWebhook:
create: false
port: 9443
service:
annotations: {}
create: false
name: ""

@shicli
Copy link

shicli commented Jul 28, 2023

@mjnagel I have submitted a question. Could you please help me take a look #492

@oxycoder
Copy link

oxycoder commented Mar 5, 2024

The operator should allow us to use existing issuer.
Example I'm using cert-manager and had cluster issuer.
It's can be done easy in open telemetry

        --set admissionWebhooks.certManager.enabled=false \
        --set admissionWebhooks.certManager.issuerRef.kind=ClusterIssuer \
        --set admissionWebhooks.certManager.issuerRef.name=letsencrypt-prod \

But it's not possible in jaeger, is there anything relate to #376 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants