Non-deterministic multi-architecture image build

[apcheamitru]

This is a possible duplicate of #669.

Apko version:

GitVersion:    v0.20.1
GitCommit:     cee37c74b11f336e99656b55778aeeed833601c8
GitTreeState:  clean
BuildDate:     '1970-01-01T00:00:00Z'
GoVersion:     go1.23.3
Compiler:      gc
Platform:      linux/amd64

I'm using the following apko definition to build a multi-architecture busybox image:

$ cat images/busybox.yaml
---
archs:
  - x86_64
  - aarch64

contents:
  packages:
    - busybox

entrypoint:
  command: /bin/sh

And this command to build the image:

apko build \
    --keyring-append https://packages.wolfi.dev/os/wolfi-signing.rsa.pub \
    --repository-append https://packages.wolfi.dev/os \
    images/busybox.yaml busybox:latest dist/images/busybox.tar

The sha256 checksum for each platform-specific image variant -- aarch64 and x86_64 -- does not change. But after running the build multiple times I can see that the tarball contents are laid down in a non-deterministic manner.

For example:

bash-5.2# tar tvf dist/images/busybox.tar
-rw-r--r-- 0/0       588 1970-01-01 00:00:00 sha256:64049bf8dca67400619f5af8aa672f30cce1ef53d9784e2c258f84aec9566d6f
-rw-r--r-- 0/0   3058747 1970-01-01 00:00:00 e0cf86e54d11d290134698408c5714bf98231a4f12016c5ac18141e9b0bc6590.tar.gz
-rw-r--r-- 0/0       588 1970-01-01 00:00:00 sha256:3a7fc89740acd31cb9907b3b7e476db08817e48cf1745a863b5eb2feace0a51c
-rw-r--r-- 0/0   3846471 1970-01-01 00:00:00 19c69267e4a969c8cf18afebaab9e3ef2b671f4cf88cb28d7a30e43e89e7f856.tar.gz
-rw-r--r-- 0/0       461 1970-01-01 00:00:00 manifest.json
-rw-r--r-- 0/0       479 1970-01-01 00:00:00 sha256:1c75b4e9c7b8e8784ed666418683e733abd10ff258d404a752964b6b4cb8e359
-rw-r--r-- 0/0       479 1970-01-01 00:00:00 sha256:757cec052a240e75a2b0c89609090c9eafbfc61d71f1f9a58d2ee8e1a85fc796
-rw-r--r-- 0/0       738 1970-01-01 00:00:00 index.json

vs.

bash-5.2# tar tvf dist/images/busybox.tar
-rw-r--r-- 0/0       588 1970-01-01 00:00:00 sha256:3a7fc89740acd31cb9907b3b7e476db08817e48cf1745a863b5eb2feace0a51c
-rw-r--r-- 0/0   3846471 1970-01-01 00:00:00 19c69267e4a969c8cf18afebaab9e3ef2b671f4cf88cb28d7a30e43e89e7f856.tar.gz
-rw-r--r-- 0/0       588 1970-01-01 00:00:00 sha256:64049bf8dca67400619f5af8aa672f30cce1ef53d9784e2c258f84aec9566d6f
-rw-r--r-- 0/0   3058747 1970-01-01 00:00:00 e0cf86e54d11d290134698408c5714bf98231a4f12016c5ac18141e9b0bc6590.tar.gz
-rw-r--r-- 0/0       461 1970-01-01 00:00:00 manifest.json
-rw-r--r-- 0/0       479 1970-01-01 00:00:00 sha256:1c75b4e9c7b8e8784ed666418683e733abd10ff258d404a752964b6b4cb8e359
-rw-r--r-- 0/0       479 1970-01-01 00:00:00 sha256:757cec052a240e75a2b0c89609090c9eafbfc61d71f1f9a58d2ee8e1a85fc796
-rw-r--r-- 0/0       738 1970-01-01 00:00:00 index.json

As far as I can tell, both manifest.json and index.json are identical. The problem is that the platform-specific layers themselves are added inconsistently.

Please let me know if there's any additional information I can provide.

[sanmai-NL]

@apcheamitru Do you have a work-around?

Perhaps putting out to a directory rather than tar archive, and then doing the tar archiving yourself?

apko build <config.yaml> <tag> <output.tar|oci-layout-dir/>