Do more of the container build process locally

[SgtCoDFish]

See https://www.docker.com/blog/faster-multi-platform-builds-dockerfile-cross-compilation-guide/

This PR adds that blogpost for this repo. It also allows me to build trust-manager on macOS, when I currently cannot because of bugs in emulation when building for different platforms.

This will also improve performance as well as fixing the build on macOS.

We might want to use a different process down the road, but I'm confident this is an improvement to the status quo.

[jetstack-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[maelvls]

Hi! When running make image, I get an error but I assume that this problem isn't new to the changes you introduced.

$ make image
docker buildx build --builder trust-manager-builder --platform=linux/amd64,linux/arm64,linux/arm/v7,linux/ppc64le -t quay.io/jetstack/trust-manager:v0.7.0 --build-arg GOPROXY=https://proxy.golang.org,direct --output type=local,dest=/Users/mvalais/code/cert-manager/trust-manager/bin/trust-manager -f ./Dockerfile .
ERROR: no builder "trust-manager-builder" found
make: *** [trust-manager-save] Error 1

I realized that the Makefile talks about that issue:

# See wait-for-buildx.sh for an explanation of why it's needed
.PHONY: provision-buildx
provision-buildx:  ## set up docker buildx for multiarch building
ifeq ($(OS), linux)
	# This step doesn't work on macOS and doesn't seem to be required (at least with docker desktop)
	# It did seem to be needed in Linux, at least in certain configurations when running on amd64
	# TODO: it might be preferable to move away from docker buildx in the long term to avoid the dependency on Docker
	docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
endif
	docker buildx rm $(BUILDX_BUILDER) &>/dev/null || :
	./hack/wait-for-buildx.sh $(BUILDX_BUILDER) gone
	docker buildx create --name $(BUILDX_BUILDER) --bootstrap --driver docker-container --platform $(IMAGE_PLATFORMS)
	./hack/wait-for-buildx.sh $(BUILDX_BUILDER) exists
	docker buildx inspect --bootstrap --builder $(BUILDX_BUILDER)

Do I need to run make provision-buildx before running make image? It isn't mentioned anywhere 😅

[wallrj]

Would be worth adding a thumbs-up to the following issue, if that's the reason we can't use ko:

• Renameable binaries / path ko-build/ko#944

Alternatively we could use ko and remove the command field from the Deployment and rely on the default entrypoint of the image:

• trust-manager/deploy/charts/trust-manager/templates/deployment.yaml

  Line 64 in 443fc94

  command: ["trust-manager"]

[SgtCoDFish]

Do I need to run make provision-buildx before running make image? It isn't mentioned anywhere 😅

Yep you do! It'd be a good PR to raise 😁

[SgtCoDFish]

Would be worth adding a thumbs-up to the following issue, if that's the reason we can't use ko ...

I think there are a few issues with ko.

Like, I'm not religiously opposed to using ko - but it seems like the tool currently basically forces us to make a breaking change for our users, and that makes me worry a bit about adopting it more widely because if ko forces us to break to adopt it, it seems likely it'll forcibly break us later after we've adopted it.

I think Tim's experience has been that ko also makes life hard when it comes to other dependencies from the base image. Like ko works if you've got a greenfield go binary, but once you've got an established project or you've got more complicated things going on, it seems to be really tricky.

I don't love buildx but at least using dockerfiles gives us the control over what happens, you know? And like I said this morning - with these changes it at least works!

[hawksight]

@SgtCoDFish - I tried to run this locally although perhaps I ran the wrong command?

(⎈|kind-argocd-demo:cert-manager)➜  cert-manager-trust-manager git:(more-local-buildx) make image
docker buildx build --builder trust-manager-builder --platform=linux/amd64,linux/arm64,linux/arm/v7,linux/ppc64le -t quay.io/jetstack/trust-manager:v0.7.0 --build-arg GOPROXY=https://proxy.golang.org,direct --output type=local,dest=/home/pfiddes/projects/jetstack-open-source/cert-manager-trust-manager/bin/trust-manager -f ./Dockerfile .
[+] Building 382.9s (27/27) FINISHED                                                        docker-container:trust-manager-builder
 => [internal] booting buildkit                                                                                              36.6s
 => => pulling image moby/buildkit:buildx-stable-1                                                                           36.0s
 => => creating container buildx_buildkit_trust-manager-builder0                                                              0.5s
 => [internal] load build definition from Dockerfile                                                                          0.0s
 => => transferring dockerfile: 1.35kB                                                                                        0.0s
 => [linux/amd64 internal] load metadata for docker.io/library/golang:1.21                                                    1.7s
 => [internal] load .dockerignore                                                                                             0.0s
 => => transferring context: 2B                                                                                               0.0s
 => [internal] load build context                                                                                             0.0s
 => => transferring context: 402.54kB                                                                                         0.0s
 => [linux/amd64 builder  1/11] FROM docker.io/library/golang:1.21@sha256:672a2286da3ee7a854c3e0a56e0838918d0dbb1c18652992  136.8s
 => => resolve docker.io/library/golang:1.21@sha256:672a2286da3ee7a854c3e0a56e0838918d0dbb1c18652992930293312de898a6          0.0s
 => => sha256:49fe8346018f3c4b4a77420527e6ea9e36e332519ade61067af355526d85b4db 156B / 156B                                    0.1s
 => => sha256:76a498520fee85f05c762d5075ba3814f09ee1c123f14a54dc68c2f14652ec12 66.98MB / 66.98MB                             87.0s
 => => sha256:bed956b3d08c0b1903d9af3e6850930eacf865b23f62bd8cf46eb1de1325c39b 92.36MB / 92.36MB                            127.8s
 => => sha256:917ee5330e73737d6095a802333d311648959399ff2c067150890162e720f863 64.13MB / 64.13MB                            116.3s
 => => sha256:b5de22c0f5cd2ea2bb6c0524478db95bff5a294c99419ccd4a9d3ccc9873bad9 24.05MB / 24.05MB                             49.0s
 => => sha256:bc0734b949dcdcabe5bfdf0c8b9f44491e0fce04cb10c9c6e76282b9f6abdf01 49.56MB / 49.56MB                             78.0s
 => => extracting sha256:bc0734b949dcdcabe5bfdf0c8b9f44491e0fce04cb10c9c6e76282b9f6abdf01                                     1.6s
 => => extracting sha256:b5de22c0f5cd2ea2bb6c0524478db95bff5a294c99419ccd4a9d3ccc9873bad9                                     0.5s
 => => extracting sha256:917ee5330e73737d6095a802333d311648959399ff2c067150890162e720f863                                     2.1s
 => => extracting sha256:bed956b3d08c0b1903d9af3e6850930eacf865b23f62bd8cf46eb1de1325c39b                                     2.1s
 => => extracting sha256:76a498520fee85f05c762d5075ba3814f09ee1c123f14a54dc68c2f14652ec12                                     3.2s
 => => extracting sha256:49fe8346018f3c4b4a77420527e6ea9e36e332519ade61067af355526d85b4db                                     0.0s
 => [linux/amd64 builder  2/11] WORKDIR /workspace                                                                            0.4s
 => [linux/amd64 builder  3/11] COPY go.mod go.mod                                                                            0.0s
 => [linux/amd64 builder  4/11] COPY go.sum go.sum                                                                            0.0s
 => [linux/amd64 builder  5/11] COPY Makefile Makefile                                                                        0.0s
 => [linux/amd64 builder  6/11] COPY make/ make/                                                                              0.0s
 => [linux/amd64 builder  7/11] COPY cmd/ cmd/                                                                                0.0s
 => [linux/amd64 builder  8/11] COPY pkg/ pkg/                                                                                0.0s
 => [linux/amd64 builder  9/11] COPY hack/ hack/                                                                              0.0s
 => [linux/amd64->arm/v7 builder 10/11] RUN GOPROXY=https://proxy.golang.org,direct go mod download                          83.4s
 => [linux/amd64->ppc64le builder 10/11] RUN GOPROXY=https://proxy.golang.org,direct go mod download                         82.0s
 => [linux/amd64 builder 10/11] RUN GOPROXY=https://proxy.golang.org,direct go mod download                                  78.9s
 => [linux/amd64->arm64 builder 10/11] RUN GOPROXY=https://proxy.golang.org,direct go mod download                           84.4s
 => [linux/amd64 builder 11/11] RUN make build-linux-amd64                                                                  116.1s
 => [linux/amd64->ppc64le builder 11/11] RUN make build-linux-ppc64le                                                       124.0s
 => [linux/amd64->arm/v7 builder 11/11] RUN make build-linux-arm                                                            122.4s
 => [linux/amd64->arm64 builder 11/11] RUN make build-linux-arm64                                                           121.2s
 => [linux/amd64 stage-1 1/2] COPY --from=builder /workspace/bin/trust-manager-linux-amd64 /usr/bin/trust-manager             0.1s
 => [linux/arm64 stage-1 1/2] COPY --from=builder /workspace/bin/trust-manager-linux-arm64 /usr/bin/trust-manager             0.1s
 => [linux/arm/v7 stage-1 1/2] COPY --from=builder /workspace/bin/trust-manager-linux-arm /usr/bin/trust-manager              0.1s 
 => [linux/ppc64le stage-1 1/2] COPY --from=builder /workspace/bin/trust-manager-linux-ppc64le /usr/bin/trust-manager         0.1s 
 => exporting to client directory                                                                                             0.8s
 => => copying files linux/ppc64le 56.41MB                                                                                    0.8s
 => => copying files linux/arm64 55.11MB                                                                                      0.8s
 => => copying files linux/amd64 56.92MB                                                                                      0.8s
 => => copying files linux/arm/v7 54.07MB                                                                                     0.8s
make/trust-package-debian.mk:33: *** DEBIAN_TRUST_PACKAGE_VERSION must be set for trust-package-debian-save. Stop.

I couldn't see what to set DEBIAN_TRUST_PACKAGE_VERSION to, perhaps you can give me a default to test the full make image.

I ran make local-images and that ran through.
Working on Ubuntu Linux 22.04.3 LTS.

[inteon]

The commit is still called "WIP: ...".

[inteon]

Thanks @SgtCoDFish.
/approve
/lgtm

[jetstack-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inteon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [inteon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[SgtCoDFish]

@hawksight

I couldn't see what to set DEBIAN_TRUST_PACKAGE_VERSION to, perhaps you can give me a default to test the full make image.
I ran make local-images and that ran through.

I totally understand the confusion here.

I think it would make sense to have make image not build trust package stuff by default to be honest. I think it mostly exists as a helper for developing trust-manager itself. I raised #269 for this!