Certificate issues

[kylecarbs]

Note
See this comment for the latest troubleshooting information and fixes.

---

VS Code uses a version of Electron that does not support registering self-signed intermediate certificates with extensions on Windows. This unfortunately blocks our ability to perform HTTPS requests securely to a Coder deployment with self-signed certificates.

See electron/electron#38527 for updates an information. We'll fix this as soon as we can!

Warning
"unable to verify the first certificate" is the error message you'll see.

[bpmct]

It seems there are still cases where allow-insecure: true does not allow the extension to connect. @kylecarbs is investigating this.

[code-asher]

How are folks reproducing this? I tried a self-signed certificate and an untrusted authority but both work on Windows after I add the relevant certificates to the system (I am doing so via mmc). I tested against the following:

• https://untrusted-root.badssl.com
• https://self-signed.badssl.com
• https://localhost:8081 with a Caddy instance proxying to dev.coder.com using a self-signed certificate

All three cases work after I add the certificates (and restart VS Code) so VS Code is indeed reading certificates installed on the system (here by "works" I mean the request goes through without any certificate errors---since the first two are not actual Coder deployments logging in fails with an "invalid token" error).

But those three cases give me variants of "self signed certificate" errors, not the reported error. This domain can trigger the "unable to verify the first certificate" error though:

• https://incomplete-chain.badssl.com

However this ("incomplete chain") implies to me that the server is misconfigured and nothing can be done on the client to fix this?

I could not get the coder binary nor Firefox to work against this domain either but the report seem to suggest that people are getting this error while the coder binary still works. If I can figure out how to reproduce that specifically I can continue debugging.

Going to take a look at the Electron bug in the meantime.

[code-asher]

When I generate my own self-signed certificate it works for the Electron test case. This leads me to believe that there is a way to generate a certificate that is accepted by Node but not Electron, maybe because Node is missing a check that should cause it to reject the certificate.

Either way, so far this seems to imply that the issue is a certificate that is invalid in some way. It could be helpful to understand how people are generating the certificates that result in this error so I can reproduce a case where it does work for the coder binary, curl, etc, but not for VS Code.

[code-asher]

Another note: I am able to reproduce "unable to get local issuer certificate" in VS Code 1.78.1. This is a bug that was fixed in 1.78.2 but it does not match the reported error so it might be irrelevant anyway. However, when getting a reproduction we should be careful to note the VS Code version just in case.

[code-asher]

Figured out a way to reproduce:

1. Create a certificate on Windows via Powershell using the New-SelfSignedCertificate command
2. Export the certificate via mmc
3. Convert from .pfx to .pem files using openssl

The resulting certificate and key will trigger the Electron error, and if I use them in Caddy then Firefox, Chrome, and the coder binary all accept the certificate but in VS Code I get the "unable to verify the first certificate" error, as expected since Electron gives the same error.

So we come back to needing to figure out why Electron's behavior is different and what is different about these certificates compared to generating them with openssl, for example. The fix might either be in Electron or in certificate generation, or both.

Edit: here is a way to generate a certificate that reproduces the issue using openssl.

    openssl req -x509 -nodes -newkey rsa:2048 -sha256 -days 365 \
            -keyout localhost.key -out localhost.crt \
            -addext "keyUsage = digitalSignature, keyEncipherment" \
            -addext "extendedKeyUsage = clientAuth, serverAuth" \
            -addext "authorityKeyIdentifier = none" \
            -addext "subjectAltName=DNS:localhost" \
            -subj "/CN=localhost"

If you add -addext "basicConstraints = CA:FALSE" then it will work on Firefox as well (but still not in Electron). The default is CA:TRUE. The Windows-generated certificate has no basic constraints listed at all but I could not figure out how to omit that using openssl.

More details can be found in the Electron issue.

[sreya]

Un-assigning @code-asher in favor of #115 since that is specifically tracking the investigation of the root cause of the bug since it's a little uncertain where in the upstream dependency chain things are breaking.

[bpmct]

@code-asher gave me a great run-down of what is going on here (and helped with edits to this summary). There are 2 issues that users are running into that we are aware of.

Case 1) Partial certificate chain on server

You are likely running into this if Coder warns you the certificate is self-signed, and when you press "allow insecure," you get stuck in an infinite refresh loop and Coder doesn't let you connect.

Here's why:

• golang, curl, and Google Chrome all accept only a "leaf" certificate if it is trusted on the client machine (so they do not require any intermediates or roots)
• node requires a leaf + intermediate certificate server certs which must be signed by a root certificate trusted by the system

The fix for Coder admins: Update the certificates on the server to include both the leaf certificate + intermediate cert. You can cat them together into one file, starting at the leaf and going up the chain (so leaf first, then next intermediate, and so on)

To improve the UX: The Coder extension should attempt to detect (by parsing the server certificate) whether it is a partial chain. If so, we should hide the "allow insecure" button (since it does not help in this case). Instead, we link to docs on a GitHub issue on how to change the server certificate to work with node. #116

If it's not easy to detect if the problem is case 1 or case 2, we should link to both issues in the "insecure" popup that the extension creates.

Case 2) A certificate does not have signing capabilities

You are likely running into this if Coder warns you the certificate is self-signed, and when you press "allow insecure," you are able to connect to the workspace 🎉

Here's why:

• openssl accept these certificates anyways
• boringssl, the SSL library that node uses does not accept certificates if the system certificate does not have signing capabilities.

More details + the fix for Coder users and admins are outlined here: #115 (comment)

To improve the UX: We are working on a patch upstream to boringssl + Electron (electron/electron#38527), but this will take some time. In the meantime, the "allow insecure" button should work in the pop-up dialog or you can follow the docs above to regenerate your local certificates with signing capabilities.

[pjmckee]

Hey @kylecarbs @bpmct just checking to see if any permanent solution has been found for this issue? We are evaluating coder for our platform, eventually enterprise, and ran into this issue when launching vscode from coder. I saw @code-asher's reply #115 but unfortunately we work in a restricted environment and our CA will not allow the coder server's certificate to have keyUsage: Certificate Sign, so we are without a way around it. We have loaded our entire bundle onto the coder server and tried allow insecure but no luck.

[code-asher]

Has the root certificate also been added to the client machine's trust store? Unfortunately the allow insecure option is bugged, for some reason it just does nothing in VS Code so that is something we need to investigate, but for now it means the certificates need to be trusted on the client.

I opened #151

[pjmckee]

hey @code-asher thanks for the follow-up. Yes the root and intermediates are installed on the client's trust store. Tried adding those to the coder image as a test as well, but am still met with the error saying "Your certificate is not capable of signing...".

[code-asher]

Ah OK bummer. That does seem weird though. That error happens when we get unable to verify leaf signature, and that error happens when the leaf certificate is the only one available (if I am reading the code right).

Does it look like the certificates are properly chained? Like the leaf certificate has an issuer and key id fields pointing to the next certificate, etc. For example (some parts omitted for brevity):

coder@dev27:~/vscode-coder/fixtures$ openssl x509 -text -noout -in chain-leaf.crt 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 4096 (0x1000)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = TEST-intermediate
        Validity
            Not Before: Oct 13 20:35:09 2023 GMT
            Not After : Sep 19 20:35:09 2123 GMT
        Subject: CN = localhost
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
        X509v3 extensions:
            X509v3 Key Usage: 
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Alternative Name: 
                DNS:localhost
            X509v3 Subject Key Identifier: 
                5A:7A:13:6C:1C:53:64:9E:A9:D8:C5:D3:BF:8A:3C:29:94:59:1E:90
            X509v3 Authority Key Identifier: 
                12:32:21:AD:6E:A1:48:3E:36:77:DB:DF:78:DB:53:D4:4C:FA:79:A7

If we can figure out how to generate a chain similar to the ones used in your setup I can attempt to reproduce.

[Kira-Pilot]

Closing this out as we'd have to push a fix to boringssl and this has limited scope. We can re-open if we hear more feedback on this in the future.