Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added support for building images with buildkit #2482

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Added support for building images with buildkit #2482

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions docker.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,11 @@ import (
"encoding/json"
"errors"
"fmt"
"github.com/docker/docker/api/types/versions"
"github.com/moby/buildkit/session"
"io"
"io/fs"
"net"
"net/url"
"os"
"path/filepath"
Expand Down Expand Up @@ -877,6 +880,28 @@ var _ ContainerProvider = (*DockerProvider)(nil)
func (p *DockerProvider) BuildImage(ctx context.Context, img ImageBuildInfo) (string, error) {
buildOptions, err := img.BuildOptions()

const minBuildKitApiVersion = "1.39"
Copy link
Member

@mdelapenya mdelapenya Apr 15, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @silh, thanks for submitting this change. I need to talk about it more in depth with the docker-build team, as I have some concerns regarding the HTTP endpoints and the options that will be available for buildkit.

In the meantime, we have the BuildOptionsModifier concept in the FromDockerfile struct (see https://golang.testcontainers.org/features/build_from_dockerfile/#advanced-usage). I think client code could write a modifier function when buildkit is needed. Something like this:

// BuildKitOptionsModifier is a function that modifies the build options to use buildkit.
// It checks if the docker client supports buildkit and if it does, it creates a new build session.
// You can use this function as a BuildOptionsModifier in the ContainerRequest.FromDockerfile
// to build images using buildkit.
func BuildKitOptionsModifier(buildOptions *types.ImageBuildOptions) {
	ctx := context.Background()

	cli, err := testcontainers.NewDockerClientWithOpts(ctx)
	if err != nil {
		testcontainers.Logger.Printf("🛠️ Could not access the docker client: %s", err)
		return
	}
	defer cli.Close()

	clientApiVersion := cli.ClientVersion()

	if versions.GreaterThanOrEqualTo(clientApiVersion, minBuildKitApiVersion) {
		s, err := session.NewSession(ctx, "testcontainers", "")
		if err == nil {
			testcontainers.Logger.Printf("🛠️ Building using buildkit")
			dialSession := func(ctx context.Context, proto string, meta map[string][]string) (net.Conn, error) {
				return cli.DialHijack(ctx, "/session", proto, meta)
			}
			go func() {
				if err := s.Run(ctx, dialSession); err != nil {
					testcontainers.Logger.Printf("🛠️ Failed to run the build session: %s", err)
				}
			}()
			defer s.Close()
			buildOptions.SessionID = s.ID()
			buildOptions.Version = types.BuilderBuildKit
		} else {
			testcontainers.Logger.Printf("🛠️ Could not create a build session, building without buildkit: %s", err)
		}
	}
}

From testcontainers-go standpoint, if this feature is widely used, we could create a separate Go module exposing that function, which would come with the benefit of not pushing the buildkit dependency to all consumers, only to those interested in using buildkit.

Wdyt?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something like this: https://github.com/mdelapenya/testcontainers-go/pull/new/builkit-module

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like modern docker doesn't require setting up the grpc cionnection to build with v2 builder. I'll try to update the PR in the near future.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using buildkit via From FromDockerfile causes image pulls to fail due to auth credentials seemingly not being processed. See the following for more details moby/moby#48112

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stevenh did you give it a try to my branch for buildkit options modifier? https://github.com/testcontainers/testcontainers-go/compare/main...mdelapenya:testcontainers-go:builkit-module?expand=1

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately that fails as the session is closed before the build is complete resulting in errors like:

failed to resolve source metadata for docker.io/krakend/builder:latest: no active session for 1tcg31xrn1qyt6bmvq1ev9yoh: context deadline exceeded: failed to create container.

Had a play and got the basic flow seeming to work, but things like print output doesn't work. Looks like its going to pretty complex to get all buildkit functionality.

clientApiVersion := p.client.ClientVersion()
if versions.GreaterThanOrEqualTo(clientApiVersion, minBuildKitApiVersion) {
s, err := session.NewSession(ctx, "testcontainers", "")
if err == nil {
Logger.Printf("Building using buildkit")
dialSession := func(ctx context.Context, proto string, meta map[string][]string) (net.Conn, error) {
return p.client.DialHijack(ctx, "/session", proto, meta)
}
go func() {
if err := s.Run(ctx, dialSession); err != nil {
Logger.Printf("Failed to run the build session: %s", err)
}
}()
defer s.Close()
buildOptions.SessionID = s.ID()
buildOptions.Version = types.BuilderBuildKit
} else {
Logger.Printf("Couldnot create a build session, building witout buildkit: %s", err)
}
}

var buildError error
var resp types.ImageBuildResponse
err = backoff.Retry(func() error {
Expand Down
42 changes: 42 additions & 0 deletions from_dockerfile_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,3 +215,45 @@ func TestBuildImageFromDockerfile_TargetDoesNotExist(t *testing.T) {
})
require.Error(t, err)
}

func TestBuildImageFromDockerfileBuildkit(t *testing.T) {
provider, err := NewDockerProvider()
if err != nil {
t.Fatal(err)
}
defer provider.Close()

cli := provider.Client()

ctx := context.Background()

testArg := "testFile"

tag, err := provider.BuildImage(ctx, &ContainerRequest{
FromDockerfile: FromDockerfile{
Context: "testdata",
Dockerfile: "buildx.Dockerfile",
Repo: "test-repo",
Tag: "test-tag",
BuildArgs: map[string]*string{
"FILENAME": &testArg,
},
PrintBuildLog: true,
},
})
require.NoError(t, err)
assert.Equal(t, "test-repo:test-tag", tag)

_, _, err = cli.ImageInspectWithRaw(ctx, tag)
require.NoError(t, err)

t.Cleanup(func() {
_, err := cli.ImageRemove(ctx, tag, types.ImageRemoveOptions{
Force: true,
PruneChildren: true,
})
if err != nil {
t.Fatal(err)
}
})
}
19 changes: 13 additions & 6 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,14 @@ go 1.21
require (
dario.cat/mergo v1.0.0
github.com/cenkalti/backoff/v4 v4.2.1
github.com/containerd/containerd v1.7.12
github.com/containerd/containerd v1.7.13
github.com/cpuguy83/dockercfg v0.3.1
github.com/docker/docker v25.0.5+incompatible
github.com/docker/go-connections v0.5.0
github.com/docker/go-units v0.5.0
github.com/google/uuid v1.6.0
github.com/magiconair/properties v1.8.7
github.com/moby/buildkit v0.13.1
github.com/moby/patternmatcher v0.6.0
github.com/moby/term v0.5.0
github.com/opencontainers/image-spec v1.1.0
Expand All @@ -22,19 +23,23 @@ require (
)

require (
cloud.google.com/go/compute/metadata v0.2.3 // indirect
github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 // indirect
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/Microsoft/hcsshim v0.11.4 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/typeurl/v2 v2.1.1 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/distribution/reference v0.5.0 // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/go-logr/logr v1.4.1 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-ole/go-ole v1.2.6 // indirect
github.com/gogo/googleapis v1.4.1 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/klauspost/compress v1.16.0 // indirect
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0 // indirect
github.com/klauspost/compress v1.17.4 // indirect
github.com/kr/pretty v0.3.0 // indirect
github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 // indirect
github.com/moby/sys/sequential v0.5.0 // indirect
Expand All @@ -50,15 +55,17 @@ require (
github.com/tklauser/go-sysconf v0.3.12 // indirect
github.com/tklauser/numcpus v0.6.1 // indirect
github.com/yusufpapurcu/wmi v1.2.3 // indirect
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.46.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
go.opentelemetry.io/otel v1.24.0 // indirect
go.opentelemetry.io/otel/metric v1.24.0 // indirect
go.opentelemetry.io/otel/trace v1.24.0 // indirect
golang.org/x/mod v0.16.0 // indirect
golang.org/x/time v0.3.0 // indirect
golang.org/x/tools v0.13.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230711160842-782d3b101e98 // indirect
google.golang.org/grpc v1.58.3 // indirect
golang.org/x/net v0.18.0 // indirect
golang.org/x/text v0.14.0 // indirect
golang.org/x/tools v0.14.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20231016165738-49dd2c1f3d0b // indirect
google.golang.org/grpc v1.59.0 // indirect
google.golang.org/protobuf v1.33.0 // indirect
gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
Expand Down
Loading
Loading