Skip to content

Commit

Permalink
feat: Update Rust builder, add target flag (#140)
Browse files Browse the repository at this point in the history
* feat(rust): add support for wasm32 targets

* feat(rust): Add verify-component-adapter from wasmtime as rust tooling

* update

* replace rust std_checks.sh with python version

* update

* fix std_checks.py

* add std_build.py

* fix spelling

* fix dbviz build

* remove uneeded TARGETARCH usage

* remove std_build.sh

* fix docs

* add target flag to std_build.py

* wip

* fix

* fix

* add package argument

* wip

* cleanup

* add std_docs.py script

* fix spelling

* wip

* wip

* disable cargo progress bar

* fix

* removed --locked flag

* fix

* remove SETUP udc

* update docs

* fix markdown

* fix spell check

* update config file

* fix comments

* wip

* remove std_docs.py file

* add TRY/FINALLY usage

* fix spellcheck

* fix

* fix

* add nightly channel section

* add with_bench flag

* Move SMOKE_TEST udc to the std_build.py script

* remove nushell

* fix markdown check

---------

Co-authored-by: Mr-Leshiy <[email protected]>
  • Loading branch information
stevenj and Mr-Leshiy authored Jan 18, 2024
1 parent 33c07b6 commit 9b5e173
Show file tree
Hide file tree
Showing 24 changed files with 472 additions and 508 deletions.
162 changes: 89 additions & 73 deletions docs/src/guides/languages/rust.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ tags:
# :simple-rust: Rust
<!-- markdownlint-enable single-h1 -->

<!-- cspell: words USERARCH TARGETARCH toolsets fmtchk stdcfgs rustfmt nextest testunit testdocs depgraph -->
<!-- cspell: words USERARCH TARGETARCH toolsets fmtchk stdcfgs rustfmt nextest testunit testdocs depgraph testcov -->

## Introduction

Expand Down Expand Up @@ -80,47 +80,29 @@ By default `toolchain` setup to `rust-toolchain.toml`.
### Running checks

```Earthfile
# Test rust build container - Use best architecture host tools.
hosted-check:
# Run check using the most efficient host tooling
# CI Automated Entry point.
check:
FROM +builder
DO ./../../earthly/rust+CHECK
RUN /scripts/std_checks.py
# Test which runs check with all supported host tooling. Needs qemu or rosetta to run.
# Only used to validate tooling is working across host toolsets.
all-hosts-check:
BUILD --platform=linux/amd64 --platform=linux/arm64 +hosted-check
## Standard CI targets.
##
## These targets are discovered and executed automatically by CI.
# Run check using the most efficient host tooling
# CI Automated Entry point.
check:
FROM busybox
# This is necessary to pick the correct architecture build to suit the native machine.
# It primarily ensures that Darwin/Arm builds work as expected without needing x86 emulation.
# All target implementation of this should follow this pattern.
ARG USERARCH
IF [ "$USERARCH" == "arm64" ]
BUILD --platform=linux/arm64 +hosted-check
ELSE
BUILD --platform=linux/amd64 +hosted-check
END
all-hosts-check:
BUILD --platform=linux/amd64 --platform=linux/arm64 +check
```

With prepared environment and all data, we're now ready to start operating with the source code and configuration files.
The `hosted-check` target which actually performs all checks and validation
with the help of `+CHECK` Function.
The `+CHECK` Function performs static checks of the Rust project as
The `check` target which actually performs all checks and validation
with the help of `std_checks.py` script.
This script performs static checks of the Rust project as
`cargo fmt`, `cargo machete`, `cargo deny` which will validate formatting,
find unused dependencies and any supply chain issues with dependencies.
Here is the list of steps (look at `./earthly/rust/scripts/std_checks.sh`):
Here is the list of steps (look at `./earthly/rust/scripts/std_checks.py`):

1. `cargo fmtchk` ([cargo alias](https://doc.rust-lang.org/cargo/reference/config.html#alias),
look at `./earthly/rust/stdcfgs/config.toml`)Checking Rust Code Format.
look at `./earthly/rust/stdcfgs/cargo_config.toml`)Checking Rust Code Format.
2. Checking configuration files for consistency.
3. `cargo machete` - Checking for Unused Dependencies.
4. `cargo deny check` - Checking for Supply Chain Issues.
Expand All @@ -131,8 +113,8 @@ to be the same as defined in `earthly/rust/stdcfgs` directory of the `catalyst-c
So when you are going to setup a new Rust project copy these configuration files
described above to the appropriate location of your Rust project.

Another targets as `all-hosts-check` and `check` (running on CI) just invoke `hosted-check`
with the specified `--platform`.
Another target as `all-hosts-check` just invokes `check` with the specified `--platform`.
It is needed for the local development to double check that everything is works for different platforms.
It is important to define a `linux` target platform with a proper cpu architecture
for the Rust project when you are building it inside Docker
and check the build process with different scenarios.
Expand All @@ -141,56 +123,84 @@ The same approach we will see for the another targets of this guide.
### Build

```Earthfile
# Build the service.
hosted-build:
# Run build using the most efficient host tooling
# CI Automated Entry point.
build:
FROM +builder
DO ./../../earthly/rust+BUILD --libs="bar" --bins="foo/foo"
TRY
RUN /scripts/std_build.py --build_flags="" \
--with_test \
--with_bench \
--cov_report="coverage-report.info" \
--libs="bar" \
--bins="foo/foo"
FINALLY
SAVE ARTIFACT target/nextest/ci/junit.xml example.junit-report.xml AS LOCAL
SAVE ARTIFACT coverage-report.info example.coverage-report.info AS LOCAL
END
DO ./../../earthly/rust+SMOKE_TEST --bin="foo"
SAVE ARTIFACT target/doc doc
SAVE ARTIFACT target/release/foo foo
SAVE ARTIFACT target/$TARGETARCH/doc doc
SAVE ARTIFACT target/$TARGETARCH/release/foo foo
DO ./../../earthly/rust+SMOKE_TEST --bin="foo"
# Test which runs check with all supported host tooling. Needs qemu or rosetta to run.
# Only used to validate tooling is working across host toolsets.
all-hosts-build:
BUILD --platform=linux/amd64 --platform=linux/arm64 +build-hosted
# Run build using the most efficient host tooling
# CI Automated Entry point.
build:
FROM busybox
# This is necessary to pick the correct architecture build to suit the native machine.
# It primarily ensures that Darwin/Arm builds work as expected without needing x86 emulation.
# All target implementation of this should follow this pattern.
ARG USERARCH
IF [ "$USERARCH" == "arm64" ]
BUILD --platform=linux/arm64 +hosted-build
ELSE
BUILD --platform=linux/amd64 +hosted-build
END
all-hosts-build:
BUILD --platform=linux/amd64 --platform=linux/arm64 +build
```

After successful performing checks of the Rust project we can finally build artifacts.
As it was discussed in the previous section, actual job is done with `hosted-build` target,
other targets needs to configure different platform running options.
So we will focus on `hosted-build` target.
After successful performing checks of the Rust project we can finally `build` artifacts.
Obviously it inherits `builder` target environment and than performs build of the binary.
Important to note that in this particular example we are dealing with the executable Rust project,
so it produces binary as a final artifact.
Another case of the building Rust library we will consider later.
Actual build process is done with `+BUILD` Function.
The `+BUILD` Function have few arguments `libs` and `bins`,
they should be specified to properly generate `cargo-modules` docs (see description below).
The `libs` argument takes a list of library crate's names in your Rust project, e.g.
Actual build process is done with the `std_build.py` script.
Here is the full list of configuration of this script:

```bash
usage: std_build.py [-h] [--build_flags BUILD_FLAGS]
[--doctest_flags DOCTEST_FLAGS] [--test_flags TEST_FLAGS]
[--bench_flags BENCH_FLAGS] [--with_test]
[--cov_report COV_REPORT] [--with_bench] [--libs LIBS]
[--bins BINS]

Rust build processing.

options:
-h, --help show this help message and exit
--build_flags BUILD_FLAGS
Additional command-line flags that can be passed to
the `cargo build` command.
--doctest_flags DOCTEST_FLAGS
Additional command-line flags that can be passed to
the `cargo testdocs` command.
--test_flags TEST_FLAGS
Additional command-line flags that can be passed to
the `cargo testunit` command.
--bench_flags BENCH_FLAGS
Additional command-line flags that can be passed to
the `cargo bench` command.
--with_test Flag to indicate whether to run tests (including unit
tests and doc tests).
--cov_report COV_REPORT
The output coverage report file path. If omitted,
coverage will not be run.
--with_bench Flag to indicate whether to run benchmarks.
--libs LIBS The list of lib crates `cargo-modules` docs to build
separated by comma.
--bins BINS The list of binaries `cargo-modules` docs to build and
made a smoke tests on them. .
```

Note that the `libs` argument takes a list of library crate's names in your Rust project, e.g.
`--libs="crate1 crate2"`.
The `bins` argument takes a list of binary crate's names and binary names in your Rust project, e.g.
`--bins="crate1/bin1 crate1/bin2 crate2/bin1"`, note that each binary name correspond to each crate
and separated in this list with `/` symbol.
Under this build process we perform different steps of compiling and validating of our Rust project,
here is the list of steps (look at `./earthly/rust/scripts/std_build.sh`):
here is the list of steps (look at `./earthly/rust/scripts/std_build.py` and `./earthly/rust/scripts/std_docs.py`):

1. `cargo build` - Building all code in the workspace.
2. `cargo lint` ([cargo alias](https://doc.rust-lang.org/cargo/reference/config.html#alias),
Expand All @@ -202,19 +212,17 @@ look at `./earthly/rust/stdcfgs/config.toml`)Checking Documentation can be gener
look at `./earthly/rust/stdcfgs/config.toml`)Checking Self contained Unit tests all pass.
5. `cargo testdocs` ([cargo alias](https://doc.rust-lang.org/cargo/reference/config.html#alias),
look at `./earthly/rust/stdcfgs/config.toml`)Checking Documentation tests all pass.
6. `cargo bench` - Checking Benchmarks all run to completion.
7. `cargo depgraph` - Generating dependency graph based on the Rust code.
6. `cargo testcov` ([cargo alias](https://doc.rust-lang.org/cargo/reference/config.html#alias),
look at `./earthly/rust/stdcfgs/config.toml`)Checking Self contained Unit tests all pass and collect coverage.
7. `cargo bench` - Checking Benchmarks all run to completion.
8. `cargo depgraph` - Generating dependency graph based on the Rust code.
Generated artifacts are `doc/workspace.dot`, `doc/full.dot`, `doc/all.dot` files.
8. `cargo modules` - Generating modules trees and graphs based on the Rust code.
9. `cargo modules` - Generating modules trees and graphs based on the Rust code.
Generated artifacts are `doc/$crate.$bin.bin.modules.tree`, `doc/$crate.$bin.bin.modules.dot`
for the specified `--bins="crate1/bin1"` argument
and `target/doc/$crate.lib.modules.tree`, `target/doc/$crate.lib.modules.dot`
for the specified `--libs="crate1"` argument of the `+BUILD` Function.

Next steps is mandatory if you are going to produce a binary as an artifact,
for Rust libraries the are not mandatory and could be omitted.
The `+SMOKE_TEST` Function checks that produced binary with the specified name (`--bin` argument)
is executable, isn't a busted mess.
for the specified `--libs="crate1"` argument.
10. Running smoke tests on provided binary names (`--bins` argument).

Final step is to provide desired artifacts: docs and binary.

Expand All @@ -231,6 +239,14 @@ please follow this [guide](./../../onboarding/index.md).
It is pretty strait forward for this builder process,
because as a part of `+build` target we already creating a docker image.

## Rust `nightly` channel

Be aware that we are running some tools in the `nightly` channel, such as `cargo fmt` and `cargo docs`.
It is highly likely that the `nightly` toolchain version on the CI machines differs from what you have locally.
Unfortunately, Rust tooling does not have the capability to preserve and maintain consistency between
`stable` and `nightly` toolchains simultaneously.
In our builds, we only preserve the `stable` toolchain version (`rust-toolchain.toml` file).

## Conclusion

You can see the final `Earthfile` [here](https://github.com/input-output-hk/catalyst-ci/blob/master/examples/rust/Earthfile)
Expand Down
8 changes: 1 addition & 7 deletions earthly/postgresql/Earthfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Common PostgreSQL Earthly builders
VERSION --global-cache --use-function-keyword 0.7

# cspell: words colordiff nushell psycopg dbviz
# cspell: words colordiff psycopg dbviz

postgres-base:
FROM postgres:16.0-alpine3.18
Expand Down Expand Up @@ -38,9 +38,6 @@ postgres-base:
COPY ../../utilities/dbviz+build/dbviz /bin
RUN dbviz --help

# Get nushell
COPY ../../utilities/nushell+nushell-build/nu /bin

# Copy our set SQL files
COPY --dir sql /sql

Expand Down Expand Up @@ -134,9 +131,6 @@ integration-test:
bash \
postgresql15-client # We use PostgreSQL 16, but v15 clients will work OK with it.

# Get nushell
COPY ../../utilities/nushell+nushell-build/nu /bin

COPY +postgres-base/scripts /scripts

WORKDIR /test
Expand Down
1 change: 0 additions & 1 deletion earthly/postgresql/scripts/python

This file was deleted.

53 changes: 8 additions & 45 deletions earthly/rust/Earthfile
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Common Rust UDCs and Builders.
VERSION --global-cache --use-function-keyword 0.7

# cspell: words rustup miri nextest ripgrep colordiff rustfmt stdcfgs toolset readelf depgraph lcov
# cspell: words rustup miri nextest ripgrep colordiff rustfmt stdcfgs toolset depgraph lcov psycopg
# cspell: words TARGETPLATFORM TARGETOS TARGETARCH TARGETVARIANT USERPLATFORM USEROS USERARCH USERVARIANT

# Base Rustup build container.
Expand Down Expand Up @@ -34,10 +34,13 @@ rust-base:

# Install necessary packages
# Expand this list as needed, rather than adding more tools in later containers.
RUN apk add --no-cache \
RUN apk add --no-cache \
musl-dev \
mold \
clang \
py3-pip \
py3-rich \
py3-psycopg \
ripgrep \
bash \
colordiff \
Expand Down Expand Up @@ -70,14 +73,16 @@ rust-base:
cargo install cargo-deny --version=0.14.3 && \
cargo install cargo-modules --version=0.10.2 && \
cargo install cargo-depgraph --version=1.5.0 && \
cargo install cargo-llvm-cov --version=0.5.39
cargo install cargo-llvm-cov --version=0.5.39 && \
cargo install --git https://github.com/bytecodealliance/wasmtime --tag v16.0.0 verify-component-adapter

SAVE ARTIFACT $CARGO_HOME/bin/refinery refinery

# Universal build scripts we will always need and are not target dependent.
COPY --dir scripts /scripts
# Copy our common scripts so we can use them inside the container.
DO ../../utilities/scripts+ADD_BASH_SCRIPTS
DO ../../utilities/scripts+ADD_PYTHON_SCRIPTS

# Standardized Rust configs.
# Build will refuse to proceed if the projects rust configs do not match these.
Expand Down Expand Up @@ -113,45 +118,3 @@ SETUP:
rustup show && \
cargo --version && \
cargo +nightly --version

# Steps we do during the `check` CI phase for all Rust programs
CHECK:
FUNCTION

# This is set up so that ALL checks are run and it will fail if any fail.
# This improves visibility into all issues that need to be corrected for `check`
# to pass without needing to iterate excessively.
RUN /scripts/std_checks.sh

# Step we do during the `build` CI phase for all Rust programs
# Parameters:
# * libs : The list of lib crates `cargo-modules` docs to build.
# * bins : The list of binaries `cargo-modules` docs to build.
BUILD:
FUNCTION
ARG libs=""
ARG bins=""

RUN /scripts/std_build.sh "$libs" "$bins"

# Run unit tests and generates test and coverage report artifacts
UNIT_TEST:
FUNCTION
# remove artifacts that may affect the coverage results
RUN cargo llvm-cov clean --workspace
# run unit tests and display test result and test coverage
RUN cargo llvm-cov nextest --release --bins --lib --workspace --locked -P ci
# generate lcov report
RUN cargo llvm-cov report --release --output-path coverage-report.info

# Check if the build executable, isn't a busted mess.
SMOKE_TEST:
FUNCTION
ARG --required bin

RUN ldd target/$TARGETARCH/release/$bin
RUN readelf -p .comment target/$TARGETARCH/release/$bin
RUN strip -v target/$TARGETARCH/release/$bin

# ALL executables MUST have `--help` as an option.
RUN target/$TARGETARCH/release/$bin --help
Loading

0 comments on commit 9b5e173

Please sign in to comment.