Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run ARM64 tests in CI #94

Merged
merged 2 commits into from
Aug 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 26 additions & 3 deletions .github/workflows/env/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,34 @@ runs:
steps:
- name: Install dependencies
shell: bash
run: sudo apt-get install -y llvm clang dwz cmake curl unzip
- name: Set up Go ${{matrix.go}}
run: |
# Ubuntu has ARM64 packages in an entirely different repo, so we can't just
# `dpkg --add-architecture arm64` here: we have to add a custom sources.list first.
sudo tee /etc/apt/sources.list.d/ubuntu.sources <<EOF > /dev/null
Types: deb
URIs: http://azure.archive.ubuntu.com/ubuntu/
Suites: noble noble-updates noble-backports
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Architectures: amd64

Types: deb
URIs: http://azure.ports.ubuntu.com/ubuntu-ports/
Suites: noble noble-updates noble-backports
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
Architectures: arm64
EOF

sudo dpkg --add-architecture arm64
sudo apt-get update -y

sudo apt-get install -y llvm clang dwz curl unzip gcc-aarch64-linux-gnu \
libc6-arm64-cross qemu-user-binfmt libc6:arm64
- name: Set up Go
uses: actions/setup-go@v5
with:
go-version: ${{matrix.go}}
go-version: stable
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Using stable as version for Go comes with the risk of breaking CI if there is an unexpected change in Go. E.g. there could be changes similar to golang/go#67401, that could break assumptions.
Manually updating the Go version allows to make sure CI does not break on events, that are happening outside of the scope and control of this repo/maintainership.

Ideally, this action should receive some version information as argument. Using $matrix.go here assumes that in the executed job matrix.go is defined and makes it hard to follow/review.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Using stable as version for Go comes with the risk of breaking CI if there is an unexpected change in Go. E.g. there could be changes similar to golang/go#67401, that could break assumptions.

I'm merely mirroring the previous logic here, but I'd also be OK with hardcoding a particular Go version globally. IMO we should just have one global Go version for all CI workflows and adding it as an argument would force us to duplicate the version everywhere.

Using $matrix.go here assumes that in the executed job matrix.go is defined and makes it hard to follow/review.

Yeah, that's why I opted to just hardcode it directly in the action. Also, all callers either passed "stable" already or forgot to set it entirely.

check-latest: true
cache-dependency-path: go.sum
id: go
85 changes: 49 additions & 36 deletions .github/workflows/unit-test-on-pull-request.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,11 @@ on:

jobs:
lint:
name: Lint
name: Lint (${{ matrix.target_arch }})
runs-on: ubuntu-24.04
strategy:
fail-fast: true
max-parallel: 2
matrix:
go: ["stable"]
target_arch: [amd64, arm64]
steps:
- name: Clone code
uses: actions/checkout@v4
Expand All @@ -26,79 +24,94 @@ jobs:
make lint

test:
name: Test
name: Test (${{ matrix.target_arch }})
runs-on: ubuntu-24.04
strategy:
fail-fast: true
max-parallel: 2
matrix:
go: ["stable"]
target_arch: [amd64, arm64]
steps:
- name: Clone code
uses: actions/checkout@v4
- name: Set up environment
uses: ./.github/workflows/env
- name: Build
run: |
echo $PATH
make test
- name: Tests
run: |
make test
run: make test TARGET_ARCH=${{ matrix.target_arch }}

build-integration-test-binaries:
name: Build integration test binaries
name: Build integration test binaries (${{ matrix.target_arch }})
runs-on: ubuntu-24.04
timeout-minutes: 10
strategy:
matrix:
target_arch: [amd64, arm64]
steps:
- name: Clone code
uses: actions/checkout@v4
- name: Set up environment
uses: ./.github/workflows/env
- name: Prepare integration test binaries for qemu tests
run: |
make integration-test-binaries
run: make integration-test-binaries TARGET_ARCH=${{ matrix.target_arch }}
- name: Upload integration test binaries
uses: actions/upload-artifact@v4
with:
name: integration-test-binaries
name: integration-test-binaries-${{ matrix.target_arch }}
path: support/*.test

integration-tests:
name: Integration tests (kernel v${{ matrix.kernel-version }})
name: Integration tests (v${{ matrix.kernel }} ${{ matrix.target_arch }})
runs-on: ubuntu-24.04
needs: build-integration-test-binaries
timeout-minutes: 10
strategy:
matrix:
kernel-version:
- "4.19.314"
- "5.4.276"
- "5.10.217"
- "5.15.159"
- "6.1.91"
- "6.6.31"
- "6.8.10"
- "6.9.1"
include:
# List of available kernels here:
# https://github.com/cilium/ci-kernels/pkgs/container/ci-kernels/versions?filters%5Bversion_type%5D=tagged

# AMD64
- { target_arch: amd64, kernel: 4.19.314 }
- { target_arch: amd64, kernel: 5.4.276 }
- { target_arch: amd64, kernel: 5.10.217 }
- { target_arch: amd64, kernel: 5.15.159 }
- { target_arch: amd64, kernel: 6.1.91 }
- { target_arch: amd64, kernel: 6.6.31 }
- { target_arch: amd64, kernel: 6.8.10 }
- { target_arch: amd64, kernel: 6.9.1 }

# ARM64 (NOTE: older ARM64 kernels are not available in Cilium repos)
- { target_arch: arm64, kernel: 6.6.31 }
- { target_arch: arm64, kernel: 6.8.4 }
- { target_arch: arm64, kernel: 6.9.1 }
steps:
- name: Clone code
uses: actions/checkout@v4
- name: Set up environment
uses: ./.github/workflows/env
- name: Install dependencies
run: |
sudo apt-get -y install qemu-system-x86
case "${{ matrix.target_arch }}" in
amd64) sudo apt-get -y install qemu-system-x86;;
arm64) sudo apt-get -y install qemu-system-arm;;
*) echo >&2 "bug: bad arch selected"; exit 1;;
esac
go install github.com/florianl/[email protected]
sudo mv ~/go/bin/bluebox /usr/local/bin/.
- name: Fetch integration test binaries
uses: actions/download-artifact@v4
with: { name: integration-test-binaries }
- name: Fetch precompiled kernels
with: { name: "integration-test-binaries-${{ matrix.target_arch }}" }
- name: Fetch precompiled kernel
run: |
install -d ci-kernels
echo "FROM ghcr.io/cilium/ci-kernels:${{ matrix.kernel-version }}" \
| docker buildx build --quiet --pull --output="ci-kernels" -
mv ci-kernels/boot/ ci-kernels/${{ matrix.kernel-version }}/
- name: Test on kernel ${{ matrix.kernel-version }}
echo "FROM ghcr.io/cilium/ci-kernels:${{ matrix.kernel }}" \
| docker buildx build --platform linux/${{ matrix.target_arch }} \
--quiet --pull --output="ci-kernels" -
mv ci-kernels/boot/ ci-kernels/${{ matrix.kernel }}/
- name: Test on kernel ${{ matrix.kernel }}
run: |
chmod a+rx *.test
support/run-tests.sh ${{ matrix.kernel-version }}
case "${{ matrix.target_arch }}" in
amd64) export QEMU_ARCH=x86_64;;
arm64) export QEMU_ARCH=aarch64;;
*) echo >&2 "bug: bad arch selected"; exit 1;;
esac
support/run-tests.sh ${{ matrix.kernel }}
2 changes: 0 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,6 @@ ebpf:

GOLANGCI_LINT_VERSION = "v1.59.1"
lint: generate
# We don't want to build the tracers here, so we stub them for linting
touch support/ebpf/tracer.ebpf.x86
go run github.com/golangci/golangci-lint/cmd/golangci-lint@$(GOLANGCI_LINT_VERSION) version
go run github.com/golangci/golangci-lint/cmd/golangci-lint@$(GOLANGCI_LINT_VERSION) run --build-tags integration,linux --timeout 10m

Expand Down
25 changes: 20 additions & 5 deletions support/run-tests.sh
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
#!/usr/bin/env bash
# Test the current package under a different kernel.
# Requires qemu-system-x86_64 and bluebox to be installed.
# Requires qemu-system-$QEMU_ARCH and bluebox to be installed.

set -eu
set -o pipefail

qemu_arch="${QEMU_ARCH:-x86_64}"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively we could make this a positional argument to run-tests.sh. No strong preference either way.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Env variable seems good to me.

color_green=$'\033[32m'
color_red=$'\033[31m'
color_default=$'\033[39m'
Expand Down Expand Up @@ -40,17 +41,31 @@ done < <(find . -name '*.test' -print0)

additionalQemuArgs=""

supportKVM=$(grep -c -E 'vmx|svm' /proc/cpuinfo || echo "0")
if [ "$supportKVM" -ne 0 ]; then
supportKVM=$(grep -E 'vmx|svm' /proc/cpuinfo || true)
if [ ! "$supportKVM" ] && [ "$qemu_arch" = "$(uname -m)" ]; then
additionalQemuArgs="-enable-kvm"
fi

case "$qemu_arch" in
x86_64)
additionalQemuArgs+=" -append console=ttyS0"
bb_args+=(-a amd64)
;;
aarch64)
additionalQemuArgs+=" -machine virt -cpu max"
bb_args+=(-a arm64)
;;
esac

if [ "$qemu_arch" = "aarch64" ]; then
additionalQemuArgs+=" -machine virt -cpu max"
fi

bluebox "${bb_args[@]}" || (echo "failed to generate initramfs"; exit 1)

echo Testing on "${kernel_version}"
$sudo qemu-system-x86_64 ${additionalQemuArgs} \
$sudo qemu-system-${qemu_arch} ${additionalQemuArgs} \
-nographic \
-append "console=ttyS0" \
-monitor none \
-serial file:"${output}/test.log" \
-no-user-config \
Expand Down