Skip to content

Commit

Permalink
Refactor workflow for common java and maven build steps (#29137)
Browse files Browse the repository at this point in the history
This change combines and removes a lot of common functionality and steps
in our workflow process making the whole thing easier to understand and
work with. Our main workflow relies on an initial maven build step that
generates artifacts that can be then pulled in and used in subsequent
test, deploy and release steps. We do this to allow these test stages to
be run in parallel and we can then aggregate the results. To achieve
this we need to use consistent naming conventions for the artefacts and
to provide artefacts and caches that enable the steps to run quickly
without redoing work. There are therefore common steps that we needed to
add to each job and the amount of duplication of these steps causes many
issues to understand or to make consistent changes system-wide. To use
common DRY principles we can pull out these common functionalities into
separate functions (GitHub composite actions) that allow for consistent
behavior.

Part of the push for this are:
1. In moving to Java 21 we want a consistent way to set the Java version
to use by default across the build, whether a major build or a minor
java version without having a mix of versions.
2. Some caches have not been consistently used throughout the build
requiring the config for cache load and save to be the same, and to know
when we should save vs load caches.



### Proposed Changes
* Create setup-java composite action handling all java installation
using sdkman.
   * allow override of java version
   * specify if graalvm is required (for native builds)
* uses default grallvm 21.0.2-graalce, but enables override from a
parameter
* merge separate cleanup-runner actions into a single action that covers
multiple OS
* create prepare-runner composite action handling common tasks for all
runners
   * option to setup master branch reference 
   * option to cleanup runner
   * setup java by default but option to not 
* create maven-job composite action providing common functionality to
all workflow steps running maven tasks
   * Handle setup native builds
   * Handle License if required for step
* Control of caches, maven, nodejs ( node cache restore can be disabled
if no node build steps are required )
* Generate or pull in artifacts to use between initial build and
subsequent steps e.g. .m2/repository
   * generation and restore of docker image artifacts
* Handle build report artifact generation automatically using naming
convention
* Flag to say that test results are generated and creation of report
artifact using naming convention
* Use .sdkmanrc as default for obtaining jvm to use in runner
* Update workflows and reusable workflows to use updated composite
actions
* Fix mac os runners (do not use -latest versions that could change )
x86 build not using x86 runner. Sdkman uses version appropriate for the
architecture it is running on.
* Combine 2 stage "validate" profile into single build and validate, fix
validate steps and ordering.
* pull release runner jar from artifacts created in Initial Build and
stored in .m2/repository artifact
* Changes to support use of common build steps with a non snapshot
version that can be commited to .mvn/maven.config
* supports locally created docker image to use release tag not just
snapshot
* Safety check to ensure that PR cannot be commited with
.mvn/maven.config as this should only be created in the release process.
* Protection added to ensure the initial build does not update source
files that have not been committed with the PR. This prevents issues
where other developers merge from master and end up with uncommitted
changes without any code changes of their own.
* Adds generation of .nvmrc file in project root to set the defaults at
the project level like .sdkmanrc.
* Note: .nvmrc file is generated from values in the
nodejs-parent/pom.xml file whereas .sdkmanrc is currently the manually
updated source file used within the build. We may want to choose which
direction is best for both. The protection in initial build for changed
files during build will prevent the .nvmrc file from being merged that
does not match the version defined in Maven in case someone manually
edits the file or does not run a build after modifying the version in
the pom file

Example usage of maven-job
```

- uses: ./.github/actions/maven-job
        with:
          stage-name: "Build Native Image ${{ matrix.label }}"
          maven-args: "package -Pnative -Pdist -DskipTests=$SKIP_TESTS -pl :dotcms-cli"
          native: true
          generates-test-results: false
          artifacts-from: ${{ env.ARTIFACT_RUN_ID }}
          version: ${{ inputs.version }}
          
 ```
This resolves multiple issues

#28736
#27998

and combines the changes in #28916

### Checklist
- [ ] Tests
- [ ] Translations
- [ ] Security Implications Contemplated (add notes if applicable)

### Additional Info
Tests of post PR workflows have been done on https://github.com/dotCMS/core-workflow-test including Master Checks, dotCLI Release

### Screenshots
Original             |  Updated
:-------------------------:|:-------------------------:
** original screenshot **  |  ** updated screenshot **
  • Loading branch information
spbolton authored Jul 15, 2024
1 parent 430f4c7 commit c6c0834
Show file tree
Hide file tree
Showing 20 changed files with 823 additions and 1,015 deletions.
53 changes: 0 additions & 53 deletions .github/actions/cleanup-macos-runner/action.yml

This file was deleted.

122 changes: 79 additions & 43 deletions .github/actions/cleanup-runner/action.yml
Original file line number Diff line number Diff line change
@@ -1,72 +1,108 @@
# action.yml
name: 'Cleanup Runner'
description: 'Cleans up runner resources'
author: 'victoralfaro-dotcms'
runs:
using: 'composite'
steps:
- name: Show Disk Usage
- name: Show Initial Disk Usage
shell: bash
run: |
echo '# df -h'
echo '# Initial Disk Usage'
df -h
echo "Runner OS: $RUNNER_OS"
echo '# docker images'
docker images || true
echo '# du -sh /home/runner'
sudo du -sh /home/runner || true
if [[ "$RUNNER_OS" == "macOS" ]]; then
HOME_DIR="/Users/runner"
echo '# du -sh $HOME_DIR'
sudo du -sh $HOME_DIR || true
elif [[ "$RUNNER_OS" == "Linux" ]]; then
HOME_DIR="/home/runner"
echo '# docker images'
docker images || true
echo '# du -sh $HOME_DIR'
sudo du -sh $HOME_DIR || true
fi
- name: Clean apt cache
- name: Clean Cache
shell: bash
run: |
echo 'Clean apt cache'
time sudo apt-get clean -y
time sudo apt-get autoclean -y
time sudo apt-get autoremove -y
time sudo rm -rf /var/lib/apt/lists/*
if [[ "$RUNNER_OS" == "macOS" ]]; then
echo 'Clean Homebrew cache'
brew cleanup -s
rm -rf "$(brew --cache)"
elif [[ "$RUNNER_OS" == "Linux" ]]; then
echo 'Clean apt cache'
sudo apt-get clean -y
sudo apt-get autoclean -y
sudo apt-get autoremove -y
sudo rm -rf /var/lib/apt/lists/*
fi
- name: Cleanup Docker
shell: bash
run: |
echo 'Cleanup Docker'
time docker system prune -f
time docker volume prune -f
time docker image prune -f
time docker container prune -f
time docker network prune -f
if [[ "$RUNNER_OS" == "Linux" ]] || [[ "$RUNNER_OS" == "macOS" ]]; then
echo 'Cleanup Docker'
docker system prune -f
docker volume prune -f
docker image prune -f
docker container prune -f
docker network prune -f
fi
- name: Reclaim Disk Space
shell: bash
run: |
removeIfExists() {
local file=$1
[[ -e $file ]] && time sudo rm -rf $1 || true
[[ -e $file ]] && sudo rm -rf $1 || true
}
removeIfExists /usr/share/dotnet
removeIfExists /usr/share/swift
removeIfExists /usr/local/lib/android
removeIfExists /opt/ghc
removeIfExists /opt/pipx
removeIfExists /opt/hostedtoolcache/CodeQL
removeIfExists /imagegeneration/installers/go-*
removeIfExists /imagegeneration/installers/node-*
removeIfExists /imagegeneration/installers/python-*
removeIfExists /home/runner/work/_temp/*
removeIfExists /home/runner/work/_tool/*
removeIfExists /home/runner/work/_config/*
if [[ "$RUNNER_OS" == "macOS" ]]; then
HOME_DIR="/Users/runner"
removeIfExists ${HOME_DIR}/.cache
removeIfExists ${HOME_DIR}/Library/Logs
removeIfExists ${HOME_DIR}/Library/Caches
removeIfExists ${HOME_DIR}/work/_temp
removeIfExists ${HOME_DIR}/.cargo/bin/cargo
removeIfExists ${HOME_DIR}/.ghcup/bin/ghc
removeIfExists ${HOME_DIR}/Library/Android/sdk
removeIfExists /usr/local/bin/pipx
removeIfExists /usr/bin/swift
removeIfExists /usr/local/bin/dotnet
removeIfExists /usr/local/Caskroom/
elif [[ "$RUNNER_OS" == "Linux" ]]; then
HOME_DIR="/home/runner"
removeIfExists /usr/share/dotnet
removeIfExists /usr/share/swift
removeIfExists /usr/local/lib/android
removeIfExists /opt/ghc
removeIfExists /opt/pipx
- name: Show Disk Usage
removeIfExists /opt/hostedtoolcache/CodeQL
removeIfExists /imagegeneration/installers/go-*
removeIfExists /imagegeneration/installers/node-*
removeIfExists /imagegeneration/installers/python-*
removeIfExists ${HOME_DIR}/work/_temp/*
removeIfExists ${HOME_DIR}/work/_tool/*
removeIfExists ${HOME_DIR}/work/_config/*
fi
- name: Show Final Disk Usage
shell: bash
run: |
echo "# df -h"
echo '# Final Disk Usage'
df -h
echo '# docker images'
docker images || true
echo '# du -sh /home/runner'
sudo du -sh /home/runner || true
if [[ "$RUNNER_OS" == "macOS" ]]; then
HOME_DIR="/Users/runner"
echo '# du -sh $HOME_DIR'
sudo du -sh $HOME_DIR || true
elif [[ "$RUNNER_OS" == "Linux" ]]; then
HOME_DIR="/home/runner"
echo '# docker images'
docker images || true
echo '# du -sh $HOME_DIR'
sudo du -sh $HOME_DIR || true
fi
55 changes: 7 additions & 48 deletions .github/actions/deploy-artifact-jfrog/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,61 +37,20 @@ runs:
steps:
- uses: actions/checkout@v4

- name: 'Set up JDK'
uses: actions/setup-java@v4
with:
java-version: 11
distribution: temurin
- name: Setup Java # Uses .sdkmanrc file
id: sdkman-java
uses: sdkman/sdkman-action@b1f9b696c79148b66d3d3a06f7ea801820318d0f

- name: Install xmllint
run: |
sudo apt-get update && sudo apt-get install -y libxml2-utils
shell: bash

- name: 'Get Date'
id: get-date
run: |
echo "date=$(/bin/date -u "+%Y-%m")" >> $GITHUB_OUTPUT
shell: bash

- name: Download Build Artifact
id: data-download
uses: dawidd6/[email protected]
- uses: ./.github/actions/maven-job
with:
github_token: ${{ inputs.github-token }}
workflow_search: true
commit: ${{ github.sha }}
workflow_conclusion: success
search_artifacts: true
dry_run: true
name: maven-repo
path: .
if_no_artifact_found: warn

- name: 'Check if artifact exists'
id: check
run: |
build_artifact_exists=${{ steps.data-download.outputs.found_artifact }}
if [[ ${build_artifact_exists} == "true" ]]; then
run_id=`echo '${{ steps.data-download.outputs.artifacts }}' | jq -r '.[0].workflow_run.id'`
found_artifacts=true
echo "Artifact Run id: $run_id"
else
echo "No artifact found"
run_id="${{ github.run_id }}"
found_artifacts=false
fi
echo "run_id=$run_id" >> $GITHUB_OUTPUT
echo "found_artifacts=$found_artifacts" >> $GITHUB_OUTPUT
shell: bash
stage-name: "Deploy Artifacts Validate"
maven-args: "validate" # We don't need to build just get the repo and use validate to check everything exists

- name: 'Download Maven Repo'
uses: actions/download-artifact@v4
with:
run-id: ${{ steps.check.outputs.run_id }}
github-token: ${{ inputs.github-token }}
name: maven-repo
path: ~/.m2/repository

- uses: jfrog/setup-jfrog-cli@v4

Expand Down Expand Up @@ -213,7 +172,7 @@ runs:
echo "::endgroup::"
shell: bash

- name:
- name: 'JFrog CLI context'
env:
ARTIFACTORY_URL: ${{ inputs.artifactory-url || steps.maven-artifact-repository.outputs.url }}
ARTIFACTORY_ACCESS_TOKEN: ${{ inputs.artifactory-access-token }}
Expand Down
Loading

0 comments on commit c6c0834

Please sign in to comment.