Add build support for arm64-linux #1055

kaboing · 2024-12-23T13:04:23Z

Adds support for arm64-linux.

Uses buildx to build multi-platform image.

Spawned by: #611

What type of PR is this?

Uncomment only one /kind <> line, press enter to put that in a new line, and remove leading whitespace from that line:

/kind breaking
/kind bug
/kind cleanup
/kind documentation

/kind feature

What this PR does / Why we need it:
Adds support for arm64/linux. Builds multi-platform using buildx.

Which issue(s) this PR fixes:

Closes #611

Special notes for your reviewer:

Adds support for arm64-linux. Uses buildx to build multi-platform image. Spawned by: googleforgames#611

kaboing · 2025-01-02T12:26:28Z

/gcbrun

markmandel · 2025-01-03T18:30:46Z

/gcbrun

markmandel · 2025-01-03T18:50:19Z

Was make build working for you locally?

kaboing · 2025-01-07T09:48:25Z

Yeah, it works on my machine ™️ . I build on an apple m2 pro with docker-desktop.

I noticed that the error messages complains about building ring, but I don't know if that is relevant.

Step #5 - "build": Caused by:
Step #5 - "build":   process didn't exit successfully: `/workspace/target/build-image/lto/build/ring-f1972dd2a537ad30/build-script-build` (exit status: 1)
Step #5 - "build":   --- stdout
Step #5 - "build":   cargo:rerun-if-env-changed=RING_PREGENERATE_ASM
Step #5 - "build":   cargo:rustc-env=RING_CORE_PREFIX=ring_core_0_17_8_
Step #5 - "build":   OPT_LEVEL = Some(3)
Step #5 - "build":   OUT_DIR = Some(/workspace/target/build-image/aarch64-unknown-linux-gnu/lto/build/ring-02bad55d8023c288/out)
Step #5 - "build":   TARGET = Some(aarch64-unknown-linux-gnu)
Step #5 - "build":   HOST = Some(x86_64-unknown-linux-gnu)
Step #5 - "build":   cargo:rerun-if-env-changed=CC_aarch64-unknown-linux-gnu
Step #5 - "build":   CC_aarch64-unknown-linux-gnu = None
Step #5 - "build":   cargo:rerun-if-env-changed=CC_aarch64_unknown_linux_gnu
Step #5 - "build":   CC_aarch64_unknown_linux_gnu = None
Step #5 - "build":   cargo:rerun-if-env-changed=TARGET_CC
Step #5 - "build":   TARGET_CC = None
Step #5 - "build":   cargo:rerun-if-env-changed=CC
Step #5 - "build":   CC = None
Step #5 - "build":   cargo:rerun-if-env-changed=CROSS_COMPILE
Step #5 - "build":   CROSS_COMPILE = None
Step #5 - "build":   RUSTC_LINKER = Some(/usr/bin/aarch64-linux-gnu-gcc)
Step #5 - "build":   cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
Step #5 - "build":   cargo:warning=Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `/usr/bin/aarch64-linux-gnu-gcc` installed?
Step #5 - "build":   RUSTC_WRAPPER = None
Step #5 - "build":   cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
Step #5 - "build":   CRATE_CC_NO_DEFAULTS = None
Step #5 - "build":   DEBUG = Some(false)
Step #5 - "build":   CARGO_CFG_TARGET_FEATURE = Some(aes,neon)
Step #5 - "build":   cargo:rerun-if-env-changed=CFLAGS_aarch64-unknown-linux-gnu
Step #5 - "build":   CFLAGS_aarch64-unknown-linux-gnu = None
Step #5 - "build":   cargo:rerun-if-env-changed=CFLAGS_aarch64_unknown_linux_gnu
Step #5 - "build":   CFLAGS_aarch64_unknown_linux_gnu = None
Step #5 - "build":   cargo:rerun-if-env-changed=TARGET_CFLAGS
Step #5 - "build":   TARGET_CFLAGS = None
Step #5 - "build":   cargo:rerun-if-env-changed=CFLAGS
Step #5 - "build":   CFLAGS = None
Step #5 - "build":   cargo:warning=Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is `/usr/bin/aarch64-linux-gnu-gcc` installed?
Step #5 - "build": 
Step #5 - "build":   --- stderr
Step #5 - "build": 
Step #5 - "build": 
Step #5 - "build":   error occurred: Failed to find tool. Is `aarch64-linux-gnu-gcc` installed?

…lation issue with Ring

kaboing · 2025-01-07T14:47:00Z

/gcbrun

kaboing · 2025-01-07T14:49:11Z

Added setting CC explicitly, as suggested by briansmith/ring#2127.
Seems I need someone to do a /gcbrun for the PR

kaboing · 2025-01-07T14:49:46Z

I'm new to the Rust eco-system so I'm mainly searching the internet to find people who's had similar issues.

XAMPPRocky · 2025-01-07T15:40:21Z

/gcbrun

markmandel · 2025-01-07T18:13:29Z

Yeah, it works on my machine ™️ . I build on an apple m2 pro with docker-desktop.

Ah fun, so you would be building on ARM, and Cloud Build is building on Intel.

My suggestion would be to add the package g++-aarch64-linux-gnu to

quilkin/build/build-image/Dockerfile

Lines 28 to 31 in 00345be

    
           RUN set -eux && \ 
        
               apt-get update && \ 
        
               apt-get install -y lsb-release jq curl wget zip git build-essential software-properties-common protobuf-compiler \ 
        
                   libssl-dev pkg-config nodejs npm bash-completion g++-x86-64-linux-gnu g++-mingw-w64-x86-64 && \

Which should give the system what it needs, when it says:

Compiler family detection failed due to error: ToolNotFound: Failed to find tool. Is aarch64-linux-gnu-gcc installed?

kaboing · 2025-01-07T20:06:30Z

Gotcha. Let's test this version then!

markmandel · 2025-01-07T21:31:39Z

/gcbrun

XAMPPRocky · 2025-01-08T00:28:22Z

ERROR: multiple platforms feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")

kaboing · 2025-01-08T16:33:43Z

Updated PR to use docker-container as build driver. However, according to docs,

Unlike docker driver, built images will not automatically appear in docker images and build --load needs to be used to achieve that.

Docs for build --load

Shorthand for --output=type=docker. Will automatically load the single-platform build result to docker images.

=> does not seem to work for multi-arch images. So we perhaps need to restructure a bit to push directly, or configure some other reasonable output. Any thoughts on this?

markmandel · 2025-01-08T17:00:47Z

does not seem to work for multi-arch images

In my experience the multi arch image is stored temporarily in the build cache, so a subsequent build with --push will be almost immediate, because it won't rebuild.

markmandel · 2025-01-08T17:00:54Z

/gcbrun

markmandel · 2025-01-08T17:32:17Z

Well it built! But the image could not be found after the fact.

kaboing · 2025-01-08T18:21:41Z

Well, it's progress at least! And I see in the logs that it build for both architectures, so that's nice.
When trying --load locally (latest commit), the entire thing hanged on sending the resulting tarball, but let's at least try it in the gcb environment and see what happens.

XAMPPRocky · 2025-01-08T20:18:38Z

/gcbrun

Restructured test-agones to require build-image directly since it's already doing the pushing.

XAMPPRocky · 2025-01-09T08:29:49Z

/gcbrun

XAMPPRocky · 2025-01-09T10:05:02Z

So it successfully built, but the image uploaded to CI doesn't look multi arch to me.

docker buildx imagetools inspect us-docker.pkg.dev/quilkin/ci/quilkin:0.10.0-dev-0aa6b83

Name:      us-docker.pkg.dev/quilkin/ci/quilkin:0.10.0-dev-0aa6b83
MediaType: application/vnd.docker.distribution.manifest.v2+json
Digest:    sha256:5a56ba3b94663f46b4e0aa673735c662ad474c21dcb6a1ad908000f75b474985

kaboing · 2025-01-09T11:42:09Z

No, I concur. One of the other manifests uploaded to the registry at the same time do have different entries for arm64 and amd64:
https://console.cloud.google.com/artifacts/docker/quilkin/us/ci/quilkin/sha256:d9d655c2d817e95d85b56286cee74a633cb1efb6d32650fdcb145702454485aa;tab=manifest?inv=1&invt=AbmXyg

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.oci.image.index.v1+json",
   "manifests": [
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "digest": "sha256:3a22f7b558eb554c10c351db637d438803472ac59a6af5e5fd6c4ba357c3df3c",
         "size": 3895,
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.oci.image.manifest.v1+json",
         "digest": "sha256:4f2a123d6da1396fd987f71ae63f7597c180c76bde2c78199aec5d017f27cff1",
         "size": 3895,
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      },
      ...
}

I'm trying to figure out what more needs to be done.

Just to make sure, I've tried to run the image created on arm64 and it does not work.

kaboing · 2025-01-09T12:30:03Z

It does work to run on arm64 if I explicitly use a different manifest-file, the one I referred to above. So need to figure out why the manifests get messed up.

... 
containers:
        - name: quilkin-arm64
          image: us-docker.pkg.dev/quilkin/ci/quilkin:0.10.0-dev-0aa6b83@sha256:d9d655c2d817e95d85b56286cee74a633cb1efb6d32650fdcb145702454485aa
          args:
            ["proxy", "--management-server", "http://quilkin-manage-agones:80"]
...

XAMPPRocky · 2025-01-17T12:59:07Z

/gcbrun

XAMPPRocky · 2025-01-17T13:42:20Z

Seeing the same result unfortunately

docker buildx imagetools inspect us-docker.pkg.dev/quilkin/ci/quilkin:0.10.0-dev-09bbe46
Name:      us-docker.pkg.dev/quilkin/ci/quilkin:0.10.0-dev-09bbe46
MediaType: application/vnd.docker.distribution.manifest.v2+json
Digest:    sha256:cf0e208da1ecd98500284826b1cfc20a8b04e3d8f857dc36d3d7a13073c550aa

kaboing · 2025-01-17T15:12:23Z

Thanks for running the gcbruns for me on this PR, I'll investigate further where the problem is.

kaboing · 2025-01-17T15:17:28Z

This digest should have the tag, it is the manifest list that points to the other manifests. Weird.

Minor fix for quoting issue on package_version

XAMPPRocky · 2025-01-22T12:43:43Z

/gcbrun

XAMPPRocky · 2025-01-22T14:01:26Z

/gcbrun

quilkin-bot · 2025-01-22T14:34:26Z

Build Succeeded 🥳

Build Id: 68ec68b6-20ee-4397-88af-36d008f94c96

The following development images have been built, and will exist for the next 30 days:

us-docker.pkg.dev/quilkin/ci/quilkin:0.10.0-dev-1616d8c

To build this version:

git fetch [email protected]:googleforgames/quilkin.git pull/1055/head:pr_1055 && git checkout pr_1055
cargo build

maggie44 · 2025-02-08T12:14:58Z

Is this now a working build and deploy ready for merging? Happy to jump in and take a look if not.

kaboing · 2025-02-11T12:15:14Z

@maggie44 Thanks for the ping!

It's not ready for merging yet.

What happens is that the manifest list is created and it's populated with the arm64 and amd64 builds. However, in the repository, the manifest list image isn't tagged, but instead one of the arm64/amd64 images is tagged.

I haven't figured out why. On my local machine, I've run the make build with both a personal dockerhub repository and a different gcb repository, and in both cases the manifest list was properly tagged. I have yet failed to figure out what is different when we run it through the complete cloudbuild in this project.

maggie44 · 2025-02-11T13:08:50Z

I see there were a few attempts at different versions, did we try using buildx to build, tag and push in one?

docker buildx build --platform=linux/amd64,linux/arm64 -t $(IMAGE_TAG) \
-f $(project_path)/image/Dockerfile $(project_path) --push

This should generate the relevant manifests and push both architecture images and manifest all at once. I think the missing step might be the tagging of the manifest file, which should be done by the above.

kaboing · 2025-02-11T13:39:07Z

Yes, that didn't work because the driver didn't support the multiplatform part in buildx build

maggie44 · 2025-02-12T09:07:44Z

Creating and enabling the driver first?

# Create and enable the driver
docker buildx create --use --name multi-platform-builder --driver docker-container
# Run the build using the enabled driver
docker buildx build --platform=linux/amd64,linux/arm64 -t $(IMAGE_TAG) \
-f $(project_path)/image/Dockerfile $(project_path) --push

kaboing · 2025-02-12T11:42:25Z

Yes, it's been tried and it ended up with the manifest list being wrongly tag. The explicit creation of manifest list was created in the hopes of fixing that problem, but we've ended up in the same situation.

(also, usual caveats, I might have stumbled earlier and the proposed solution might work, but as far as I know, it's been tested)

maggie44 · 2025-02-12T12:03:36Z

I wonder how the makefile is called. It seems the modification was made to the build section?

quilkin/build/Makefile

Line 186 in 0aa6b83

build-image:

So the makefile build is called, which then pushed. But then push was called again which called docker push. Not sure what calling it twice would do if that is what happened.

maggie44 · 2025-02-21T09:43:23Z

How about separating the steps. Build stays as build was before and builds for the local environment (which it will choose automatically). And then add a deploy to the make file with the deploy buildx:

# Create and enable the driver
docker buildx create --use --name multi-platform-builder --driver docker-container
# Run the build using the enabled driver
docker buildx build --platform=linux/amd64,linux/arm64 -t $(IMAGE_TAG) \
-f $(project_path)/image/Dockerfile $(project_path) --push

There is some value in doing the build from source for a deploy anyway, especially when using tagged GitHub commits.

I see the challenge @kaboing, without access to the runners it’s hard to debug it.

Add build support for arm64-linux

66448e0

Adds support for arm64-linux. Uses buildx to build multi-platform image. Spawned by: googleforgames#611

kaboing requested review from markmandel, XAMPPRocky, koslib and Jake-Shadle as code owners December 23, 2024 13:04

github-actions bot added kind/feature New feature or request size/s labels Dec 23, 2024

Add explicitly setting CC for aarch64-linux builds to fix cross-compi…

7f0262a

…lation issue with Ring

Add package g++-aarch64-linux-gnu to quilkin-build

fc8dad5

Update multiarch build to use docker-container as build driver

cd6b5a1

Add --load to load image into Docker

78cea0a

Added --push to docker build command

0aa6b83

Restructured test-agones to require build-image directly since it's already doing the pushing.

Merge branch 'main' into build-for-arm64

de3dfc0

kaboing added 2 commits January 17, 2025 13:04

Explicitly creating docker manifest for multi-arch build

55deff7

Explicitly create manifest and push

09bbe46

Potential fix for docker manifest list tagging issue

306c963

Minor fix for quoting issue on package_version

Reinstate buildx builds again

1616d8c

Add build support for arm64-linux #1055

Are you sure you want to change the base?

Add build support for arm64-linux #1055

Conversation

kaboing commented Dec 23, 2024

kaboing commented Jan 2, 2025

markmandel commented Jan 3, 2025

markmandel commented Jan 3, 2025

kaboing commented Jan 7, 2025 • edited Loading

kaboing commented Jan 7, 2025

kaboing commented Jan 7, 2025

kaboing commented Jan 7, 2025

XAMPPRocky commented Jan 7, 2025

markmandel commented Jan 7, 2025 • edited Loading

kaboing commented Jan 7, 2025

markmandel commented Jan 7, 2025

XAMPPRocky commented Jan 8, 2025 • edited Loading

kaboing commented Jan 8, 2025

markmandel commented Jan 8, 2025

markmandel commented Jan 8, 2025

markmandel commented Jan 8, 2025

kaboing commented Jan 8, 2025

XAMPPRocky commented Jan 8, 2025

XAMPPRocky commented Jan 9, 2025

XAMPPRocky commented Jan 9, 2025

kaboing commented Jan 9, 2025

kaboing commented Jan 9, 2025

XAMPPRocky commented Jan 17, 2025

XAMPPRocky commented Jan 17, 2025

kaboing commented Jan 17, 2025

kaboing commented Jan 17, 2025

XAMPPRocky commented Jan 22, 2025

XAMPPRocky commented Jan 22, 2025

quilkin-bot commented Jan 22, 2025

maggie44 commented Feb 8, 2025 • edited Loading

kaboing commented Feb 11, 2025

maggie44 commented Feb 11, 2025 • edited Loading

kaboing commented Feb 11, 2025

maggie44 commented Feb 12, 2025

kaboing commented Feb 12, 2025

maggie44 commented Feb 12, 2025

maggie44 commented Feb 21, 2025

kaboing commented Jan 7, 2025 •

edited

Loading

markmandel commented Jan 7, 2025 •

edited

Loading

XAMPPRocky commented Jan 8, 2025 •

edited

Loading

maggie44 commented Feb 8, 2025 •

edited

Loading

maggie44 commented Feb 11, 2025 •

edited

Loading