Specifying an EncryptedKey in a LocalWorkspace does not take effect

[dsarosi]

What happened?

When using an s3 storage with an AWS KMS secrets provider in the Automation API using a LocalWorkSpace, then the following issue arises. A stack can be created, but any subsequent updates to the stack throws the following error.

[EncryptedKey](error: getting stack configuration: get stack secrets manager: passphrase must be set with PULUMI_CONFIG_PASSPHRASE or PULUMI_CONFIG_PASSPHRASE_FILE environment variables)

I noticed that the EncryptedKey setting in my code is not used for the stack. When I open the stored stack on S3, I get a different output. Hence, I assume that for each subsequent update a new EncryptedKey gets generated, which doesn't match the EncryptedKey placed into the state file.

Example

var localWorkspaceOptions = new LocalWorkspaceOptions
        {
            ProjectSettings = new ProjectSettings("myproject", ProjectRuntimeName.Dotnet),
            Program = program,
            StackSettings = new Dictionary<string, StackSettings>()
            {
                {
                    "testing", 
                    new StackSettings
                    {
                        SecretsProvider = "awskms://da524141-XXXX-XXX-XXXX-XXXXX?region=eu-central-1",
                        EncryptedKey = "AQICAHioRE7kely3Y0APYnfykSFlm9omthRRQFVxeLpvmR6wFAEo3wydNS27n4rb2Xcrhg6eAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM9Z4loL0JxVTkKf5jAgEQgDurHRNz6FFm7Bk6L4mdkGH5yV4P4uqEcr5smTBVcd/kNEFMGu+avg7VlhGUVCngpjkRRjNFvmw5+afMrw=="
                    }
                }
            }
        };
        var localWorkSpace = await LocalWorkspace.CreateAsync(localWorkspaceOptions);
        
        var stack = await WorkspaceStack.CreateOrSelectAsync("testing", localWorkSpace);

When I grab the stack file from S3, I get the following. As one can see, the EncrypedKey in the stack file and the EncryptedKey in the code don't match.

"checkpoint": {
        "stack": "organization/kmon.company.infra.service/2a87a0e8-c1c7-46d9-8da3-b142e5ca3688_54b89cf7-400b-47eb-bf9e-cb2c4e7687a9",
        "latest": {
            "manifest": {
                "time": "2024-11-14T11:59:30.752042+08:00",
                "magic": "668ca25d7f85ccc6ac87bbb638bb84f209eef6d0a451c1879fb6018ee75ce8ea",
                "version": "v3.137.0"
            },
            "secrets_providers": {
                "type": "cloud",
                "state": {
                    "url": "awskms://da524141-XXXX-XXX-XXXX-XXXXX?region=eu-central-1",
                    "encryptedkey": "AQICAHioRE7kely3Y0APYnfykSFlm9omthRRQFVxeLpvmR6wFAHUHV6Bh8EbCb6Z6q/O+GEvAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMPbXcomEvaQOYG9xTAgEQgDu8BYonp58Dcg2G3jg5jHVWuHaY1gtLAxY0iZSg5hXO204ECI1SRfSLjRt+3VPBw1GNB/A8knCBRctVhQ=="
                }
            },
...
}

Output of pulumi about

running 'dotnet build -nologo .'
Determining projects to restore...

All projects are up-to-date for restore.

pattern.data.encryption.library -> /Users/.../Debug/net8.0/pattern.data.encryption.library.dll

/usr/local/share/dotnet/sdk/8.0.403/Microsoft.Common.CurrentVersion.targets(2412,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "Microsoft.IdentityModel.Tokens". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [/Users/.../xxx.csproj]

/usr/local/share/dotnet/sdk/8.0.403/Microsoft.Common.CurrentVersion.targets(2412,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "System.IdentityModel.Tokens.Jwt". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [/Users/.../xxx.csproj]

pattern.data.shared.services -> /Users/.../bin/Debug/net8.0/pattern.data.shared.services.dll

pattern.data.service.iam -> /Users/.../bin/Debug/net8.0/pattern.data.service.iam.dll

kmon.company.infra.service -> /Users/.../bin/Debug/net8.0/kmon.company.infra.service.dll

Build succeeded.

/usr/local/share/dotnet/sdk/8.0.403/Microsoft.Common.CurrentVersion.targets(2412,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "Microsoft.IdentityModel.Tokens". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [/Users/.../xxx.csproj]
/usr/local/share/dotnet/sdk/8.0.403/Microsoft.Common.CurrentVersion.targets(2412,5): warning MSB3245: Could not resolve this reference. Could not locate the assembly "System.IdentityModel.Tokens.Jwt". Check to make sure the assembly exists on disk. If this reference is required by your code, you may get compilation errors. [/Users/.../xxx.csproj]
2 Warning(s)
0 Error(s)

Time Elapsed 00:00:01.17

'dotnet build -nologo .' completed successfully
CLI
Version 3.137.0
Go Version go1.23.2
Go Compiler gc

Plugins
KIND NAME VERSION
resource aws 6.57.0
language dotnet unknown

Host
OS darwin
Version 14.6.1
Arch arm64

This project is written in dotnet: executable='/usr/local/bin/dotnet' version='8.0.403'

Backend
Name xxx-MBP
URL s3://xxxx-pstate
User dxxxxxxx
Organizations
Token type personal

Dependencies:
NAME VERSION
Microsoft.AspNetCore.Authentication.JwtBearer 8.0.8
Microsoft.AspNetCore.OpenApi 8.0.10
NSwag.Annotations 14.1.0
NSwag.AspNetCore 14.1.0
Pulumi.Automation 3.68.1-alpha.b035300
Pulumi.Aws 6.57.0
Swashbuckle.AspNetCore 6.6.2

Pulumi locates its logs in /var/folders/wg/xwyrm5ld5rz1cqpqmsb20zgm0000gn/T/ by default
warning: Failed to get information about the current stack: No current stack

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[dsarosi]

As a workaround, I need to apply the following.

stack.Workspace.EnvironmentVariables = new Dictionary<string, string?>()
        {
            ["PULUMI_CONFIG_PASSPHRASE"] = _encryptedKey
        };

[Frassle]

So setting PULUMI_CONFIG_PASSPHRASE isn't really fixing this, it must just be causing the passphrase secret provider to kick in and use your encrypted key string as the passphrase.

Your error is on "getting stack configuration" are you also setting stack configuration via automation api somehow?

[dsarosi]

I don't set any stack configuration via api. By the way, my example above wasn't quite correct. I also get the PULUMI_CONFIG_PASSPHRASE message when I put the SecretProvider in the StackSettings. It only works for the initial run if I place it in the LocalWorkspaceOptions.

This works for creating the very first stack.

var localWorkspaceOptions = new LocalWorkspaceOptions
        {
            SecretsProvider = "awskms://da524141-XXXX-XXX-XXXX-XXXXX?region=eu-central-1",
            ProjectSettings = new ProjectSettings("myproject", ProjectRuntimeName.Dotnet),
            Program = program,
            StackSettings = new Dictionary<string, StackSettings>()
            {
                {
                    "testing", 
                    new StackSettings
                    {
                        EncryptedKey = "AQICAHioRE7kely3Y0APYnfykSFlm9omthRRQFVxeLpvmR6wFAEo3wydNS27n4rb2Xcrhg6eAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM9Z4loL0JxVTkKf5jAgEQgDurHRNz6FFm7Bk6L4mdkGH5yV4P4uqEcr5smTBVcd/kNEFMGu+avg7VlhGUVCngpjkRRjNFvmw5+afMrw=="
                    }
                }
            }
        };
        var localWorkSpace = await LocalWorkspace.CreateAsync(localWorkspaceOptions);
        
        var stack = await WorkspaceStack.CreateOrSelectAsync("testing", localWorkSpace);

This doesn't, and it requests the PULUMI_CONFIG_PASSPHRASE. It's almost like specifying StackSettings is irrelevant.

var localWorkspaceOptions = new LocalWorkspaceOptions
        {
            ProjectSettings = new ProjectSettings("myproject", ProjectRuntimeName.Dotnet),
            Program = program,
            StackSettings = new Dictionary<string, StackSettings>()
            {
                {
                    "testing", 
                    new StackSettings
                    {
                        SecretsProvider = "awskms://da524141-XXXX-XXX-XXXX-XXXXX?region=eu-central-1",
                        EncryptedKey = "AQICAHioRE7kely3Y0APYnfykSFlm9omthRRQFVxeLpvmR6wFAEo3wydNS27n4rb2Xcrhg6eAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQM9Z4loL0JxVTkKf5jAgEQgDurHRNz6FFm7Bk6L4mdkGH5yV4P4uqEcr5smTBVcd/kNEFMGu+avg7VlhGUVCngpjkRRjNFvmw5+afMrw=="
                    }
                }
            }
        };
        var localWorkSpace = await LocalWorkspace.CreateAsync(localWorkspaceOptions);
        
        var stack = await WorkspaceStack.CreateOrSelectAsync("testing", localWorkSpace);

This is my ~/.pulumi/credentials file.

{
    "current": "s3://xxx-pstate",
    "accessTokens": {
        "s3://xxx-pstate": ""
    },
    "accounts": {
        "s3://xxx-pstate": {
            "lastValidatedAt": "0001-01-01T00:00:00Z"
        }
    }
}