[v0.14] invalid cached file sent to context with buildx

[dwmh]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

When using files in build context with same size, filename and timestamp information, new buildx versions use cli side cache incorrectly. I have verified v0.13.1 is the last version to work correctly, v0.14.0 and v0.15.1 have the issue present.

Issue can be reproduced with a simple shell script in Configuration section.

Expected behaviour

Both app1 and app2 in the example provided should get correct input files as they are different.

Actual behaviour

Build of app1 caches code file and app2 incorrectly uses it.

Buildx version

github.com/docker/buildx v0.15.1 1c1dbb2

Docker info

Client:
 Version:    26.1.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.15.1
    Path:     REDACTED

Server:
 Containers: 89
  Running: 0
  Paused: 0
  Stopped: 89
 Images: 77
 Server Version: 26.1.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 926c9586fe4a6236699318391cd44976a98e31f1
 runc version: 51d5e94601ceffbbd85688df1c928ecccbfa4685
 init version: de40ad007797e0dcd8b7126f27bb87401d224240
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.9.4-gentoo
 Operating System: Gentoo Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 31.26GiB
 Name: REDACTED
 ID: Z27R:WA35:4HYF:B4VA:3TJF:CPFY:GAQI:RX3B:UISW:IDVF:QYBQ:TFH2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE     DRIVER/ENDPOINT   STATUS    BUILDKIT   PLATFORMS
default*      docker
 \_ default    \_ default       running   v0.13.1    linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

Configuration

#!/bin/sh

# Enable buildkit
export DOCKER_BUILDKIT=1

# Generate dockerfile that builds both apps
cat > Dockerfile <<EOF
FROM alpine:3.20

COPY code /code

CMD /bin/sh /code
EOF

# Generate application code with identical timestamps and sizes
mkdir -p app1 app2
rm -f app1/code app2/code
touch -d 01010101 app1/code app2/code
cat > app1/code <<EOF
#!/bin/sh
echo app1 output
EOF
cat > app2/code <<EOF
#!/bin/sh
echo app2 output
EOF
touch -d 01010101 app1/code app2/code

# Build application images without server side cache
docker build --no-cache -f ./Dockerfile ./app1 --tag localhost/app1:latest
docker build --no-cache -f ./Dockerfile ./app2 --tag localhost/app2:latest

# Verify that incorrect code is used
docker run localhost/app1:latest
docker run localhost/app2:latest

Build logs

+ export DOCKER_BUILDKIT=1
+ DOCKER_BUILDKIT=1
+ cat
+ mkdir -p app1 app2
+ rm -f app1/code app2/code
+ touch -d 01010101 app1/code app2/code
+ cat
+ cat
+ touch -d 01010101 app1/code app2/code
+ docker build --no-cache -f ./Dockerfile ./app1 --tag localhost/app1:latest
[+] Building 1.1s (7/7) FINISHED                                                                                                                                                                                                                                                                                                                                                                                           docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 90B                                                                                                                                                                                                                                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/alpine:3.20                                                                                                                                                                                                                                                                                                                                                                       1.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 26B                                                                                                                                                                                                                                                                                                                                                                                                     0.0s
 => CACHED [1/2] FROM docker.io/library/alpine:3.20@sha256:b89d9c93e9ed3597455c90a0b88a8bbb5cb7188438f70953fede212a0c4394e0                                                                                                                                                                                                                                                                                                          0.0s
 => [2/2] COPY code /code                                                                                                                                                                                                                                                                                                                                                                                                            0.0s
 => exporting to image                                                                                                                                                                                                                                                                                                                                                                                                               0.0s
 => => exporting layers                                                                                                                                                                                                                                                                                                                                                                                                              0.0s
 => => writing image sha256:735f91cf114dcd6919cd01d0bca0f9ce754d60e71cb16f53b9b43e9095f2e2c6                                                                                                                                                                                                                                                                                                                                         0.0s
 => => naming to localhost/app1:latest                                                                                                                                                                                                                                                                                                                                                                                               0.0s
+ docker build --no-cache -f ./Dockerfile ./app2 --tag localhost/app2:latest
[+] Building 0.4s (7/7) FINISHED                                                                                                                                                                                                                                                                                                                                                                                           docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                                                                                                 0.0s
 => => transferring dockerfile: 90B                                                                                                                                                                                                                                                                                                                                                                                                  0.0s
 => [internal] load metadata for docker.io/library/alpine:3.20                                                                                                                                                                                                                                                                                                                                                                       0.2s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                                                                                                      0.0s
 => [internal] load build context                                                                                                                                                                                                                                                                                                                                                                                                    0.0s
 => => transferring context: 26B                                                                                                                                                                                                                                                                                                                                                                                                     0.0s
 => CACHED [1/2] FROM docker.io/library/alpine:3.20@sha256:b89d9c93e9ed3597455c90a0b88a8bbb5cb7188438f70953fede212a0c4394e0                                                                                                                                                                                                                                                                                                          0.0s
 => [2/2] COPY code /code                                                                                                                                                                                                                                                                                                                                                                                                            0.0s
 => exporting to image                                                                                                                                                                                                                                                                                                                                                                                                               0.0s
 => => exporting layers                                                                                                                                                                                                                                                                                                                                                                                                              0.0s
 => => writing image sha256:7bd53c686e8f29a63b4ec3b1bcc2a0785b450f3aee4fc3f8a0597a534080e5e4                                                                                                                                                                                                                                                                                                                                         0.0s
 => => naming to localhost/app2:latest                                                                                                                                                                                                                                                                                                                                                                                               0.0s
+ docker run localhost/app1:latest
app1 output
+ docker run localhost/app2:latest
app1 output

Additional info

Originally the issue was discovered in git repositories that happened to add files with same name and size to multiple folders, after which timestamps were aligned by others cloning that repository.

There are at least three workarounds known to me at the moment:

• sleep 0.1 between touching each file before building, this makes timestamps unique
• docker buildx prune -f between builds (slow)
• use named build contexts as advertised in https://www.docker.com/blog/dockerfiles-now-support-multiple-build-contexts/

[dwmh]

Found a similar old issue from buildkit side, which was related to Dockerfile handling: moby/buildkit#1368

Filenames and timestamps suggest a similar underlying issue with this regression.

[tonistiigi]

I don't know why you've marked it as regression. This is expected behavior moby/buildkit#4817 if you reset the timestamps of files. Same as rsync and git.

[dwmh]

The reasoning is that older versions used to work, however if this is expected we must have relied on a bug.

I would be interested to know how monorepos are expected to setup docker image builds, as you can very easily end up with identical timestamps for different folders if those files are created or modified by same checkouts. Providing the entire monorepo as build context seems wasteful and slow, so my guess would be to utilize named build contexts per subproject. But this feels like a hack since you anyways need to provide the default build context in addition to that, which will basically be an empty folder.

[tonistiigi]

Sorry, this is indeed a regression and fixed in #2558

While the metadata based transfer is correct, this should not affect your example case as you are transferring from a different directory. We have validation that the incremental transfers are used only when the same source directory (basename+node identifier) is reused. That part broke in v0.14 .

You can still have a similar case if you make a non-metadata change to a file and then reset its timestamp, but you shouldn't be able to get this when building two different directories that both contain a file with the same timestamp.