Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci: generic improvement and cleanup of workflow #20

Merged
merged 3 commits into from
Nov 14, 2023
Merged

ci: generic improvement and cleanup of workflow #20

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fd2bf9d
ci: move git short sha length to ENV
Ansuel Nov 14, 2023
0dd0d06
ci: generalize container test step
Ansuel Nov 14, 2023
15bb6df
ci: split container push steps to separate job and add deploy tag
Ansuel Nov 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 37 additions & 20 deletions .github/workflows/build-push.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ on:

env:
BUILDBOT_VERSION: 3.8.0
GITHUB_SHA_LEN: 8

concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
Expand Down Expand Up @@ -42,8 +43,8 @@ jobs:
- name: Stylecheck with black
run: black phase1/master.cfg

build-test-push:
name: Build, test and push containers
build-test:
name: Build and Test container
runs-on: ubuntu-latest
needs: test-lint

Expand All @@ -53,17 +54,20 @@ jobs:
strategy:
fail-fast: ${{ github.event_name == 'pull_request' }}
matrix:
container_flavor:
- master
- worker
include:
- container_flavor: master
container_verify_string: "buildmaster configured in /master"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't this approach get hairy once we would like to add more checks/tests?

Copy link
Member Author

@Ansuel Ansuel Nov 14, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mhhh yes...

I wonder if the test will be done all on logs tho...

If we will check that then we might consider adding all the check on a shell script and just run it here.

But you are right if the intention is to add additional stuff then I think the current way of keep the master and worker split is the only way...

MAYBE I can split the get log part/generalize the command to run the container?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well its actually very lame testing, but better then no testing.

In the long term I planned to look into labgrid's DockerStrategy and use that for more advanced QA, the similar approach could be then re-used for SDK/ImageBuilder/rootfs container testing in the openwrt/docker repository etc.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we could keep it as it is proposed now and improve later (tm)

Copy link
Member

@ynezz ynezz Nov 14, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

buildbot.errors.PluginDBError: Requirements are not satisfied for buildbot.www:base: The 'zope-interface>=5' distribution was not found and is required by Twisted

Murphy's law? :P https://github.com/openwrt/buildbot/actions/runs/6867270540/job/18678271015#step:5:51

(BTW its not caused by this PR)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I notice the same while testing this pr. Thing it's there from a lot :(

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About Murphy's law... I had to rerun the test as for some reason the github instance lost connection ahahaha

- container_flavor: worker
container_test_command: "--env BUILDWORKER_NAME=X --env BUILDWORKER_PASSWORD=Y"
container_verify_string: "worker configured in /builder"

steps:
- name: Checkout
uses: actions/checkout@v3

- name: Environment variables
run: |
echo "GIT_SHA_SHORT=${GITHUB_SHA::8}" >> $GITHUB_ENV
echo "GIT_SHA_SHORT=${GITHUB_SHA::${{ env.GITHUB_SHA_LEN }}}" >> $GITHUB_ENV

- name: Build container and export it to local Docker
uses: docker/build-push-action@v4
Expand All @@ -75,40 +79,53 @@ jobs:
BUILDBOT_VERSION=${{ env.BUILDBOT_VERSION }}
OPENWRT_VERSION=${{ env.GIT_SHA_SHORT }}

- name: Test master Docker container
if: matrix.container_flavor == 'master'
- name: Test ${{ matrix.container_flavor }} Docker container
run: |
docker run --detach --name test-master local/master
docker run --detach ${{ matrix.container_test_command }} --name test-${{ matrix.container_flavor }} local/${{ matrix.container_flavor }}
sleep 5
docker logs test-master | tee master.log
grep "buildmaster configured in /master" master.log
docker logs test-${{ matrix.container_flavor }} | tee ${{ matrix.container_flavor }}.log
grep "${{ matrix.container_verify_string }}" ${{ matrix.container_flavor }}.log

deploy:
name: Push Container
if: github.event_name != 'pull_request' || github.repository_owner != 'openwrt'
runs-on: ubuntu-latest
needs: build-test

environment: production

- name: Test worker Docker container
if: matrix.container_flavor == 'worker'
permissions:
packages: write

strategy:
matrix:
container_flavor:
- master
- worker

steps:
- name: Checkout
uses: actions/checkout@v3

- name: Environment variables
run: |
docker run --detach --env BUILDWORKER_NAME=X --env BUILDWORKER_PASSWORD=Y --name test-worker local/worker
sleep 5
docker logs test-worker | tee worker.log
grep "worker configured in /builder" worker.log
echo "GIT_SHA_SHORT=${GITHUB_SHA::${{ env.GITHUB_SHA_LEN }}}" >> $GITHUB_ENV

- name: Docker meta
id: meta
if: github.event_name != 'pull_request' || github.repository_owner != 'openwrt'
uses: docker/metadata-action@v4
with:
images: name=ghcr.io/${{ github.repository }}/build${{ matrix.container_flavor }}-v${{ env.BUILDBOT_VERSION }}

- name: Login to GitHub Container Registry
uses: docker/login-action@v2
if: github.event_name != 'pull_request' || github.repository_owner != 'openwrt'
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Build container again and push it
uses: docker/build-push-action@v4
if: github.event_name != 'pull_request' || github.repository_owner != 'openwrt'
with:
push: true
tags: ${{ steps.meta.outputs.tags }}
Expand Down