Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: Add sds and envoyinit to the release pipeline #10509

Merged
merged 3 commits into from
Jan 27, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .github/workflows/release.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,9 @@ jobs:
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}

- uses: "docker/setup-qemu-action@v3"
- uses: "docker/setup-buildx-action@v3"
Comment on lines +101 to +102
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needed for the SDS container build. Otherwise, the release action was failing when I was testing this out in my fork: https://github.com/timflannagan/kgateway/actions/runs/12942255998/job/36099680490.


- name: Run goreleaser
run: make release
env:
Expand Down
100 changes: 94 additions & 6 deletions .goreleaser.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -20,34 +20,122 @@ builds:
goarch:
- amd64
- arm64
- id: sds
main: ./projects/sds/cmd/
binary: sds-linux-{{ .Arch }}
gcflags: "{{ .Env.GCFLAGS }}"
ldflags: "{{ .Env.LDFLAGS }}"
env:
- CGO_ENABLED=0
- GO111MODULE=on
- GOARCH={{ .Arch }}
- GOOS={{ .Os }}
goos:
- linux
goarch:
- amd64
- arm64
- id: envoyinit
main: ./projects/envoyinit/cmd/
binary: envoyinit-linux-{{ .Arch }}
gcflags: "{{ .Env.GCFLAGS }}"
ldflags: "{{ .Env.LDFLAGS }}"
env:
- CGO_ENABLED=0
- GO111MODULE=on
- GOARCH={{ .Arch }}
- GOOS={{ .Os }}
goos:
- linux
goarch:
- amd64
- arm64
dockers:
- image_templates:
- &arm_image "{{ .Env.IMAGE_REGISTRY }}/{{ .Env.GLOO_IMAGE_REPO }}:{{ .Env.VERSION }}-arm64"
- &controller_arm_image "{{ .Env.IMAGE_REGISTRY }}/{{ .Env.GLOO_IMAGE_REPO }}:{{ .Env.VERSION }}-arm64"
use: buildx
dockerfile: &controller_dockerfile projects/gateway2/cmd/Dockerfile
goos: linux
goarch: arm64
build_flag_templates:
- "--pull"
- "--platform=linux/arm64"
- "--build-arg=GOARCH=arm64"
- "--build-arg=ENVOY_IMAGE={{ .Env.ENVOY_GLOO_IMAGE }}"
- image_templates:
- &controller_amd_image "{{ .Env.IMAGE_REGISTRY }}/{{ .Env.GLOO_IMAGE_REPO }}:{{ .Env.VERSION }}-amd64"
use: buildx
dockerfile: *controller_dockerfile
goos: linux
goarch: amd64
build_flag_templates:
- "--pull"
- "--platform=linux/amd64"
- "--build-arg=GOARCH=amd64"
- "--build-arg=ENVOY_IMAGE={{ .Env.ENVOY_GLOO_IMAGE }}"
- image_templates:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can we put the images in the same order as controller image (i.e. arm first), so it's easier to read?

- &sds_arm_image "{{ .Env.IMAGE_REGISTRY }}/{{ .Env.SDS_IMAGE_REPO }}:{{ .Env.VERSION }}-arm64"
use: buildx
dockerfile: &sds_dockerfile projects/sds/cmd/Dockerfile
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking at the controller image above, there's the extra dockerfile arg for arm but not amd; why is it the other way around here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jenshu Hmm, I'm not sure I follow. I didn't see any skew in the diff.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, it's fixed now that you swapped amd and arm :)

goos: linux
goarch: arm64
build_flag_templates:
- "--pull"
- "--platform=linux/arm64"
- "--build-arg=GOARCH=arm64"
- "--build-arg=BASE_IMAGE={{ .Env.ALPINE_BASE_IMAGE }}"
- image_templates:
- &sds_amd_image "{{ .Env.IMAGE_REGISTRY }}/{{ .Env.SDS_IMAGE_REPO }}:{{ .Env.VERSION }}-amd64"
use: buildx
dockerfile: &dockerfile projects/gateway2/cmd/Dockerfile
dockerfile: *sds_dockerfile
goos: linux
goarch: amd64
build_flag_templates:
- "--pull"
- "--platform=linux/amd64"
- "--build-arg=GOARCH=amd64"
- "--build-arg=BASE_IMAGE={{ .Env.ALPINE_BASE_IMAGE }}"
- image_templates:
- &envoyinit_arm_image "{{ .Env.IMAGE_REGISTRY }}/{{ .Env.ENVOYINIT_IMAGE_REPO }}:{{ .Env.VERSION }}-arm64"
use: buildx
dockerfile: &envoyinit_dockerfile projects/envoyinit/cmd/Dockerfile.envoyinit
goos: linux
goarch: arm64
build_flag_templates:
- "--pull"
- "--platform=linux/arm64"
- "--build-arg=GOARCH=arm64"
- "--build-arg=ENTRYPOINT_SCRIPT=/projects/envoyinit/cmd/docker-entrypoint.sh"
- "--build-arg=ENVOY_IMAGE={{ .Env.ENVOY_GLOO_IMAGE }}"
extra_files:
- projects/envoyinit/cmd/docker-entrypoint.sh
- image_templates:
- &amd_image "{{ .Env.IMAGE_REGISTRY }}/{{ .Env.GLOO_IMAGE_REPO }}:{{ .Env.VERSION }}-amd64"
- &envoyinit_amd_image "{{ .Env.IMAGE_REGISTRY }}/{{ .Env.ENVOYINIT_IMAGE_REPO }}:{{ .Env.VERSION }}-amd64"
use: buildx
dockerfile: *dockerfile
dockerfile: *envoyinit_dockerfile
goos: linux
goarch: amd64
build_flag_templates:
- "--pull"
- "--platform=linux/amd64"
- "--build-arg=GOARCH=amd64"
- "--build-arg=ENTRYPOINT_SCRIPT=/projects/envoyinit/cmd/docker-entrypoint.sh"
- "--build-arg=ENVOY_IMAGE={{ .Env.ENVOY_GLOO_IMAGE }}"
extra_files:
- projects/envoyinit/cmd/docker-entrypoint.sh
docker_manifests:
- name_template: "{{ .Env.IMAGE_REGISTRY }}/{{ .Env.GLOO_IMAGE_REPO }}:{{ .Env.VERSION }}"
image_templates:
- *amd_image
- *arm_image
- *controller_arm_image
- *controller_amd_image
- name_template: "{{ .Env.IMAGE_REGISTRY }}/{{ .Env.SDS_IMAGE_REPO }}:{{ .Env.VERSION }}"
image_templates:
- *sds_arm_image
- *sds_amd_image
- name_template: "{{ .Env.IMAGE_REGISTRY }}/{{ .Env.ENVOYINIT_IMAGE_REPO }}:{{ .Env.VERSION }}"
image_templates:
- *envoyinit_arm_image
- *envoyinit_amd_image
changelog:
disable: false
release:
Expand Down
16 changes: 9 additions & 7 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -106,8 +106,8 @@ DISTROLESS_BASE_IMAGE ?= gcr.io/distroless/base-debian11:latest
GLOO_DISTROLESS_BASE_IMAGE ?= $(IMAGE_REGISTRY)/distroless-base:$(VERSION)
# GLOO_DISTROLESS_BASE_IMAGE + utility binaries (sh, wget, sleep, nc, echo, ls, cat, vi)
GLOO_DISTROLESS_BASE_WITH_UTILS_IMAGE ?= $(IMAGE_REGISTRY)/distroless-base-with-utils:$(VERSION)
# BASE_IMAGE used in non distroless variants
ALPINE_BASE_IMAGE ?= alpine:3.17.6
# BASE_IMAGE used in non distroless variants. Exported for use in goreleaser.yaml.
export ALPINE_BASE_IMAGE ?= alpine:3.17.6
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We talked about migrating away from alpine in favor of distroless or some other base image. I held off for now as I want to focus on getting the full release pipeline working e2e before standardizing on our approach for base images.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we track removing alpine explicitly in an issue? Either to the release automation or the cleanup epic, whichever makes more sense to you.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, will do.


#----------------------------------------------------------------------------------
# Macros
Expand Down Expand Up @@ -504,12 +504,13 @@ $(GLOO_RACE_OUT_DIR)/.gloo-race-docker: $(GLOO_RACE_OUT_DIR)/gloo-linux-amd64 $(
touch $@

#----------------------------------------------------------------------------------
# SDS Server - gRPC server for serving Secret Discovery Service config for Gloo Edge MTLS
# SDS Server - gRPC server for serving Secret Discovery Service config
#----------------------------------------------------------------------------------

SDS_DIR=projects/sds
SDS_SOURCES=$(call get_sources,$(SDS_DIR))
SDS_OUTPUT_DIR=$(OUTPUT_DIR)/$(SDS_DIR)
export SDS_IMAGE_REPO ?= sds

$(SDS_OUTPUT_DIR)/sds-linux-$(GOARCH): $(SDS_SOURCES)
$(GO_BUILD_FLAGS) GOOS=linux go build -ldflags='$(LDFLAGS)' -gcflags='$(GCFLAGS)' -o $@ $(SDS_DIR)/cmd/main.go
Expand All @@ -525,7 +526,7 @@ sds-docker: $(SDS_OUTPUT_DIR)/sds-linux-$(GOARCH) $(SDS_OUTPUT_DIR)/Dockerfile.s
docker buildx build --load $(PLATFORM) $(SDS_OUTPUT_DIR) -f $(SDS_OUTPUT_DIR)/Dockerfile.sds \
--build-arg GOARCH=$(GOARCH) \
--build-arg BASE_IMAGE=$(ALPINE_BASE_IMAGE) \
-t $(IMAGE_REGISTRY)/sds:$(VERSION)
-t $(IMAGE_REGISTRY)/$(SDS_IMAGE_REPO):$(VERSION)

$(SDS_OUTPUT_DIR)/Dockerfile.sds.distroless: $(SDS_DIR)/cmd/Dockerfile.distroless
cp $< $@
Expand All @@ -535,7 +536,7 @@ sds-distroless-docker: $(SDS_OUTPUT_DIR)/sds-linux-$(GOARCH) $(SDS_OUTPUT_DIR)/D
docker buildx build --load $(PLATFORM) $(SDS_OUTPUT_DIR) -f $(SDS_OUTPUT_DIR)/Dockerfile.sds.distroless \
--build-arg GOARCH=$(GOARCH) \
--build-arg BASE_IMAGE=$(GLOO_DISTROLESS_BASE_WITH_UTILS_IMAGE) \
-t $(IMAGE_REGISTRY)/sds:$(VERSION)-distroless
-t $(IMAGE_REGISTRY)/$(SDS_IMAGE_REPO):$(VERSION)-distroless

#----------------------------------------------------------------------------------
# Envoy init (BASE/SIDECAR)
Expand All @@ -544,6 +545,7 @@ sds-distroless-docker: $(SDS_OUTPUT_DIR)/sds-linux-$(GOARCH) $(SDS_OUTPUT_DIR)/D
ENVOYINIT_DIR=projects/envoyinit/cmd
ENVOYINIT_SOURCES=$(call get_sources,$(ENVOYINIT_DIR))
ENVOYINIT_OUTPUT_DIR=$(OUTPUT_DIR)/$(ENVOYINIT_DIR)
export ENVOYINIT_IMAGE_REPO ?= gloo-envoy-wrapper

$(ENVOYINIT_OUTPUT_DIR)/envoyinit-linux-$(GOARCH): $(ENVOYINIT_SOURCES)
$(GO_BUILD_FLAGS) GOOS=linux go build -ldflags='$(LDFLAGS)' -gcflags='$(GCFLAGS)' -o $@ $(ENVOYINIT_DIR)/main.go
Expand All @@ -562,7 +564,7 @@ gloo-envoy-wrapper-docker: $(ENVOYINIT_OUTPUT_DIR)/envoyinit-linux-$(GOARCH) $(E
docker buildx build --load $(PLATFORM) $(ENVOYINIT_OUTPUT_DIR) -f $(ENVOYINIT_OUTPUT_DIR)/Dockerfile.envoyinit \
--build-arg GOARCH=$(GOARCH) \
--build-arg ENVOY_IMAGE=$(ENVOY_GLOO_IMAGE) \
-t $(IMAGE_REGISTRY)/gloo-envoy-wrapper:$(VERSION)
-t $(IMAGE_REGISTRY)/$(ENVOYINIT_IMAGE_REPO):$(VERSION)

$(ENVOYINIT_OUTPUT_DIR)/Dockerfile.envoyinit.distroless: $(ENVOYINIT_DIR)/Dockerfile.envoyinit.distroless
cp $< $@
Expand All @@ -574,7 +576,7 @@ gloo-envoy-wrapper-distroless-docker: $(ENVOYINIT_OUTPUT_DIR)/envoyinit-linux-$(
--build-arg GOARCH=$(GOARCH) \
--build-arg ENVOY_IMAGE=$(ENVOY_GLOO_IMAGE) \
--build-arg BASE_IMAGE=$(GLOO_DISTROLESS_BASE_WITH_UTILS_IMAGE) \
-t $(IMAGE_REGISTRY)/gloo-envoy-wrapper:$(VERSION)-distroless
-t $(IMAGE_REGISTRY)/$(ENVOYINIT_IMAGE_REPO):$(VERSION)-distroless

#----------------------------------------------------------------------------------
# Certgen - Job for creating TLS Secrets in Kubernetes
Expand Down
9 changes: 5 additions & 4 deletions projects/envoyinit/cmd/Dockerfile.envoyinit
Original file line number Diff line number Diff line change
Expand Up @@ -11,15 +11,16 @@ ENV DEBIAN_FRONTEND=noninteractive
#install wget for our default probes
RUN apt-get update \
&& apt-get upgrade -y \
&& apt-get install wget -y \
&& apt-get install wget -y \
&& rm -rf /var/log/*log /var/lib/apt/lists/* /var/log/apt/* /var/lib/dpkg/*-old /var/cache/debconf/*-old

COPY envoyinit-linux-$GOARCH /usr/local/bin/envoyinit

# SDS-specific setup, only used if ENVOY_SIDECAR=true
COPY docker-entrypoint.sh /
ARG ENTRYPOINT_SCRIPT=/docker-entrypoint.sh
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was needed as goreleaser's extra_files argument adds the path (e.g. projects/envoyinit/cmd/docker-entrypoint.sh in this case) to the build context, while this regular envoyinit-docker Makefile target fails if I hardcode the COPY projects/envoyinit/cmd/docker-entrypoint.sh / instruction.

COPY $ENTRYPOINT_SCRIPT /

USER 10101

ENTRYPOINT [ "/docker-entrypoint.sh"]
CMD []
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD []