github-action: Allow to specify files in artefacts as input argument

[PatrickRudolph]

On Github Action files are passed between jobs using build artifacts.

Add support for directly accessing files not only in the local file system, but also in a named build artifact of a previous job, for files passed as input argument.

Files in artifacts should follow a specific syntax.
For example:

artifact://name-of-previous-build-step-artifact@/home/github/some/path/within/the/artifact

[AtomicFS]

I was thinking about this for some time now. Downloading artifact for local-use might be challenging, requiring a user to supply token, possibly also installing github-cli (I am not really stoked about the idea of writing my own code to interact with GitHub API), also specifying branch, maybe commit, maybe which run (in case of CI re-runs) to use ... there is a lot of complexity here.

But let's assume that we do not support interaction between GitHub CI and local runs. I think it would be rather interesting to allow firmware-action automatically download and upload artifacts (and cache them) all on it's own.

GitHub limits and APIs

As far as I checked, there is not simple and straight forward way to work with GitHub artifacts or caches (upload and download), be it in golang or in Dagger. I found google/go-github (using REST) and shurcooL/githubv4 (using GraphQL), but did not look deeper into it.

The problem is that the REST does not support artifact upload, there is probably no cache operations available either. And as for GraphQL (used also by github-ci), I would assume it has more-less feature-parity.

So at the moment it seems that the best and simplest way is to just stick to official actions/upload-artifact@v4/actions/download-artifact@v4 and actions/cache/restore@v4/actions/cache/save@v4.

Simple enough projects

For simple enough projects we can have firmware-action in single job, building recursively. This is possible when all of the following conditions are met:

• the firmware stack is not too wide, and all the code and Docker containers fit into the GitHub runner (they have only 14 GB of disk space), we have issue open about this in GitHub CI delete container on success #560
• the matrix build (DEBUG, RELEASE, ...) is not too complex.

Then we have multiple actions/upload-artifact@v4 calls to upload all of the artifacts. We also have multiple actions/cache/restore@v4 and actions/cache/save@v4 calls to speed up the execution.

This is a perfect candidate for simplification, to greatly reduce copy-pasting of code. I think we could expand our composite action (right now I mean the action.yml code) to automatically store/restore caches and upload artifacts.

Improve composite action

GitHub does not allow repeating single step multiple times, but maybe we could leverage the power of composite actions, and simply call another workflow which would take list of artifacts as input and then matrix over them.

Our current action.yml, when simplified, looks like this:

name: 'Compile open firmware solution'
description: ...
author: '9elements'
inputs:
  config:
    ...
runs:
  using: 'composite'
  steps:
    - name: Run executable
      shell: bash
      run: ./firmware-action
      env:
        INPUT_CONFIG: ${{ inputs.config }}
        INPUT_TARGET: ${{ inputs.target }}
        INPUT_RECURSIVE: ${{ inputs.recursive }}

What I would propose would look something like this:

name: 'Compile open firmware solution'
description: ...
author: '9elements'
inputs:
  config:
    ...
runs:
  using: 'composite'
  steps:
      # Get list of artifacts that need to be downloaded (do not build)
    - name: Run executable
      if: inputs.recursive == false
      id: artifacts-to-download
      shell: bash
      run: ./firmware-action --list-artifacts-to-download
      env:
        INPUT_CONFIG: ${{ inputs.config }}
        INPUT_TARGET: ${{ inputs.target }}
        INPUT_RECURSIVE: false

      # Call another workflow / action to iterate over the list and download all artifacts (and cache)
    - name: Download artifacts
      if: inputs.recursive == false
      uses: ./mass-download
      with:
        list: ${{ steps.artifacts-to-download.outputs.list }}

      # Run firmware-action and build the firmware
    - name: Run executable
      id: artifacts-to-upload
      shell: bash
      run: ./firmware-action
      env:
        INPUT_CONFIG: ${{ inputs.config }}
        INPUT_TARGET: ${{ inputs.target }}
        INPUT_RECURSIVE: ${{ inputs.recursive }}

      # Upload all artifacts that were produced (as artifacts and as cache)
    - name: Upload artifacts
      uses: ./mass-upload
      with:
        list: ${{ steps.artifacts-to-upload.outputs.list }}

I think this could also work OK for more complex matrix jobs, hence why the if: inputs.recursive == false.