apko_config -> apko_build loses information about the original request

[imjasonh]

Our normal usage of the provider, as codified in terraform-publisher-apko, passes an unresolved config to apko_config, which resolves the package versions and emits the resolved config as an attribute. We then pass that resolved config to apko_build to build the image. We do this mainly to isolate resolution from building, and so we can attest and attach the resolved config to the built image.

However, this means that apko_build produces an image with an /etc/apk/world with fully resolved packages, locking them in place. As far as apko_build is concerned, it was asked to build an image with exactly those packages at those versions, so it did that. The original request -- which we should write to the resulting image's /etc/apk/world -- was lost in the resolving process.

The apko CLI by itself doesn't exhibit this behavior:

$ crane export $(apko publish examples/wolfi-base.yaml ttl.sh/jason) - | tar -tvf - | grep world
wolfi-base

An image produced by apko_config -> apko_build does:

$ crane export cgr.dev/chainguard/wolfi-base - | tar -Oxf - etc/apk/world
apk-tools=2.14.0-r0
busybox=1.36.1-r4
ca-certificates-bundle=20230506-r1
glibc-locale-posix=2.38-r8
glibc=2.38-r8
ld-linux=2.38-r8
libcrypt1=2.38-r8
libcrypto3=3.2.0-r0
libssl3=3.2.0-r0
openssl-config=3.2.0-r0
wolfi-base=1-r3
wolfi-baselayout=20230201-r7
wolfi-keys=1-r5
zlib=1.3-r2

This matters because the original user's request was not "exactly all these packages at these versions" -- it was "whatever it takes to install wolfi-base please". That unresolved request is what /etc/apk/world communicates.

This is practically a problem when users run apk add in an image built in this way, with a fully locked APK world. Take a months-old wolfi-base image and try to apk add curl

$ docker run --rm -it cgr.dev/chainguard/wolfi-base@sha256:a8c9c2888304e62c133af76f520c9c9e6b3ce6f1a45e3eaa57f6639eb8053c90
# cat /etc/apk/world
apk-tools=2.14.0-r0
busybox=1.36.1-r2
ca-certificates-bundle=20230506-r0
glibc-locale-posix=2.38-r5
glibc=2.38-r5
ld-linux=2.38-r5
libcrypt1=2.38-r5
libcrypto3=3.1.4-r0
libssl3=3.1.4-r0
openssl-config=3.1.4-r0
wolfi-base=1-r3
wolfi-baselayout=20230201-r6
wolfi-keys=1-r5
zlib=1.3-r1

Note the locked world, with openssl libraries locked at 3.1.4, the latest available at the time the image was built.

Now let's add curl:

# apk add curl
fetch https://packages.wolfi.dev/os/aarch64/APKINDEX.tar.gz
(1/5) Installing libbrotlicommon1 (1.1.0-r1)
(2/5) Installing libbrotlidec1 (1.1.0-r1)
(3/5) Installing libnghttp2-14 (1.58.0-r1)
(4/5) Installing libcurl-openssl4 (8.5.0-r0)
(5/5) Installing curl (8.5.0-r0)
OK: 14 MiB in 19 packages
de3f7a5efe0d:/# cat /etc/apk/world
apk-tools=2.14.0-r0
busybox=1.36.1-r2
ca-certificates-bundle=20230506-r0
curl
glibc=2.38-r5
glibc-locale-posix=2.38-r5
ld-linux=2.38-r5
libcrypt1=2.38-r5
libcrypto3=3.1.4-r0
libssl3=3.1.4-r0
openssl-config=3.1.4-r0
wolfi-base=1-r3
wolfi-baselayout=20230201-r6
wolfi-keys=1-r5
zlib=1.3-r1

apk add succeeded, but it did the wrong thing, because the curl I just installed needs a newer openssl than the one we locked into the image.

# curl https://google.com
curl: /usr/lib/libssl.so.3: version `OPENSSL_3.2.0' not found (required by /usr/lib/libcurl.so.4)

This could have been avoided if the world hadn't been locked to the set of packages that was appropriate at the time the image was built, but instead reflected the original request.

Unpinning the world and apk upgradeing seems to fix the glitch:

# cat < EOF > /etc/apk/world
apk-tools
busybox
ca-certificates-bundle
curl
glibc
glibc-locale-posix
ld-linux
libcrypt1
libcrypto3
libssl3
openssl-config
wolfi-base
wolfi-baselayout
wolfi-keys
zlib
EOF
# apk upgrade
Upgrading critical system libraries and apk-tools:
(1/1) Upgrading apk-tools (2.14.0-r0 -> 2.14.0-r1)
Continuing the upgrade transaction with new apk-tools:
(1/11) Upgrading ca-certificates-bundle (20230506-r0 -> 20230506-r1)
(2/11) Upgrading wolfi-baselayout (20230201-r6 -> 20230201-r7)
(3/11) Upgrading ld-linux (2.38-r5 -> 2.38-r8)
(4/11) Upgrading glibc-locale-posix (2.38-r5 -> 2.38-r8)
(5/11) Upgrading glibc (2.38-r5 -> 2.38-r8)
(6/11) Upgrading openssl-config (3.1.4-r0 -> 3.2.0-r1)
(7/11) Upgrading libcrypto3 (3.1.4-r0 -> 3.2.0-r1)
(8/11) Upgrading libssl3 (3.1.4-r0 -> 3.2.0-r1)
(9/11) Upgrading zlib (1.3-r1 -> 1.3-r3)
(10/11) Upgrading libcrypt1 (2.38-r5 -> 2.38-r8)
(11/11) Upgrading busybox (1.36.1-r2 -> 1.36.1-r4)
Executing glibc-2.38-r8.trigger
Executing busybox-1.36.1-r4.trigger
OK: 15 MiB in 19 packages
# curl https://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>

🎉

We also could have made the world just wolfi-base and curl and that would work too -- all the packages in the world come from the original request for the wolfi-base package.

Back to tf-apko:

This locking behavior is not exhibited by apko itself, as demonstrated above with apko publish. It's only an artifact of how we resolve-then-build using tf-apko.

If apko_config's output retained some information about its original request that it could feed to apko_build to populate its unlocked world, we could correctly populate the unlocked world. It may even make sense to attest both the locked package-versions, and the original requested packages (after all TF variables/provider overrides were taken into account), since this could be useful information, and we lose it today.

Another alternative would be to have apko_build produce a locked resolved config output and take the unresolved config as an input.

Instead of

data "apko_config" "this" {
  config_contents     = file("foo.apko.yaml")
  extra_packages      = var.extra_packages
  default_annotations = var.default_annotations
}

resource "apko_build" "this" {
  repo   = var.target_repository
  config = data.apko_config.this.config
}

resource "cosign_attest" "this" {
  predicates {
    type = "https://apko.dev/image-configuration"
    json = jsonencode(data.apko_config.this.config)
  }
}

...we'd have:

resource "apko_build" "this" {
  repo   = var.target_repository
  extra_packages      = var.extra_packages
  default_annotations = var.default_annotations
  config_contents     = file("foo.apko.yaml") # <-- take the unresolved input
}

resource "cosign_attest" "this" {
  predicates {
    type = "https://apko.dev/image-configuration"
    json = jsonencode(data.apko_build.this.resolved_config) # <-- take the build's resolved config output
  }
}

(We already attest after building, since attesting depends on check-sbom)

[jonjohnsonjr]

Looking at the newer libcurl, we see:

curl -L https://packages.wolfi.dev/os/aarch64/libcurl-openssl4-8.5.0-r0.apk | tar -Oxz usr/lib/libcurl.so.4.8.0 | objdump -x -

Dynamic Section:
  NEEDED       libnghttp2.so.14
  NEEDED       libssl.so.3
  NEEDED       libcrypto.so.3
  NEEDED       libbrotlidec.so.1
  NEEDED       libz.so.1
  NEEDED       libc.so.6
  NEEDED       ld-linux-aarch64.so.1
  SONAME       libcurl.so.4

...

Version References:
  required from ld-linux-aarch64.so.1:
    0x06969197 0x00 05 GLIBC_2.17
  required from libc.so.6:
    0x069691b3 0x00 07 GLIBC_2.33
    0x069691b4 0x00 06 GLIBC_2.34
    0x06969197 0x00 04 GLIBC_2.17
  required from libcrypto.so.3:
    0x06702b20 0x00 03 OPENSSL_3.0.0
  required from libssl.so.3:
    0x06702d20 0x00 08 OPENSSL_3.2.0
    0x06702b20 0x00 02 OPENSSL_3.0.0

Compare to the previous version:

curl -L https://packages.wolfi.dev/os/aarch64/libcurl-openssl4-8.4.0-r2.apk | tar -Oxz usr/lib/libcurl.so.4.8.0 | objdump -x -

Dynamic Section:
  NEEDED       libnghttp2.so.14
  NEEDED       libssl.so.3
  NEEDED       libcrypto.so.3
  NEEDED       libbrotlidec.so.1
  NEEDED       libz.so.1
  NEEDED       libc.so.6
  NEEDED       ld-linux-aarch64.so.1
  SONAME       libcurl.so.4

...

Version References:
  required from ld-linux-aarch64.so.1:
    0x06969197 0x00 05 GLIBC_2.17
  required from libc.so.6:
    0x069691b3 0x00 08 GLIBC_2.33
    0x069691b4 0x00 07 GLIBC_2.34
    0x069691b6 0x00 06 GLIBC_2.36
    0x06969197 0x00 04 GLIBC_2.17
  required from libcrypto.so.3:
    0x06702b20 0x00 03 OPENSSL_3.0.0
  required from libssl.so.3:
    0x06702b20 0x00 02 OPENSSL_3.0.0

We have this new requirement of OPENSSL_3.2.0:

  required from libssl.so.3:
    0x06702d20 0x00 08 OPENSSL_3.2.0

Unfortunately, both new and old libcurl express the dependency in the same way:

curl -s -L https://packages.wolfi.dev/os/aarch64/libcurl-openssl4-8.4.0-r2.apk | tar -Oxz .PKGINFO | grep libssl
depend = so:libssl.so.3

curl -s -L https://packages.wolfi.dev/os/aarch64/libcurl-openssl4-8.5.0-r0.apk | tar -Oxz .PKGINFO | grep libssl
depend = so:libssl.so.3

So anything that provides so:libssl.so.3 is fine to use to satisfy that dependency, even though the newer libcurl really actually needs something like so:libssl.so>=3.2.

I'm not sure if the dependency information in APKs is rich enough to express this kind of thing without human intervention... but maybe we could do more sophisticated things in our SCA?