Add testing and contrib changes

CONTRIBUTING.md

@@ -1,3 +1,5 @@
+# Overview
+
 * If you have found a discrepancy in documented and observed behaviour, that
   is a bug. Feel free to [report it as an
   issue](https://github.com/mpalmer/action-validator/issues), providing
@@ -9,8 +11,83 @@
 
 * At all times, abide by the Code of Conduct (CODE_OF_CONDUCT.md).
 
+---
+
 # Environment Setup
 
+## Install Rust
+Firstly, you'll need make any changes to the core functionality of this project. We recommend use `rustup`, on the recommendation of the rust team. You can find the installation instructions [here](https://www.rust-lang.org/tools/install).
+
+To confirm that rust is installed, run the `cargo` command. If you don't receive the help docs output, you may need to add rust to your shell rc file.
+
+## Git Submodule Setup
+This repository uses [git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules). Specifically for the use of [schemastore](https://github.com/SchemaStore/schemastore). 
+
+To setup the git submodule after cloning this repo to your local, you'll want to run the following commands:
+1. `git submodule init`
+2. `git submodule update`
+
+It should look similar to the output below.
+
+```bash
+❯ git submodule init
+Submodule 'src/schemastore' (https://github.com/SchemaStore/schemastore) registered for path 'src
+/schemastore'
+❯ git submodule update
+Cloning into '/Users/someuser/action-validator/src/schemastore'...
+Submodule path 'src/schemastore': checked out 'd3e6ab7727380b214acbab05570fb09a3e5d2dfc'
+```
+
+At this point, you should be all set to `cargo run`! If you run into any issues here, please [create an issue](https://github.com/mpalmer/action-validator/issues/new/choose).
+
 # Running the Validator Locally
 
+## `cargo run [FILE] -- [OPTIONS]`
+`cargo run` will create a _debug_ executable and run the project. If this is your first time running the command, cargo will compile the development binary with `cargo build`. This will install all of the dependencies and create the debug binary `action-validator` in the `/target/debug/` directory. `cargo run` will then invoke this binary after creation.
+
+One caveat if you're running with `cargo run`: if you want to supply the program with options, you need to use the `--` operator between `cargo run` and your provided options. This let's cargo know which flags are meant for cargo, and which are meant for the executable.
+
+## `cargo build` && `./target/debug/action-validator [OPTIONS]`
+As discussed in the prior section, `cargo build` install dependencies (if they're not cached) and build the development binary. This binary can then be invoked directly by running `./target/debug/action-validator`. This does **not** require the use of the `--` operator in between the binary and any provided options.
+
+## Try It Yourself!
+
+Run the command `cargo run -- --help`. You should see an output similar to the following.
+```bash
+❯ cargo run -- --help
+    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
+     Running `target/debug/action-validator --help`
+A validator for GitHub Action and Workflow YAML files
+
+Usage: action-validator [OPTIONS] [path_to_action_yaml]...
+
+Arguments:
+  [path_to_action_yaml]...  Input file
+
+Options:
+  -v, --verbose  Be more verbose
+  -h, --help     Print help information
+  -V, --version  Print version information
+```
+
 # Writing Tests
+All tests live in the `tests` directory. Currently, this project implements snapshot testing,
+but that's not to say you couldn't write unit or integration tests with the current structure.
+To run the tests, simply run `cargo test` from the root directory. If you want to test a specific
+feature, you can add the `-F {feature}` flag (e.g. `cargo test -F remote-checks`).
+
+## Unit/Integration Tests
+As of this writing, there are no unit or integration tests. If you are looking to write some, please
+follow the directions in [this guide](https://doc.rust-lang.org/book/ch11-01-writing-tests.html).
+
+## Snapshot Tests
+A snapshot test is performed when we execute the cli and capture `stdout`, `stderr`, and/or an exit code.
+When the tests is run, the results of the test must exactly match those of the previous run. For this project,
+the snapshot tests are named in the format `{next_id}_{whats_being_tested}` (e.g. `011_remote_checks_failure`).
+
+If you have made changes which will change the output of the program and cause snapshots to fail, you can run
+`cargo test -F save-snapshots`. This feature causes the executed command to save the `stdout`, `stderr`, and/or
+exit code to the specified testing directory.
+
+If you are writing a net new test, you will need to create the test directory with your workflow or action file.
+Once you're done, you can save the results to that directy by running `cargo test -F save-snapshots`.

Cargo.toml

@@ -11,7 +11,11 @@ include = [
 	"/src/schemastore/src/schemas/json/github-action.json"
 ]
 version = "0.0.0-git"
-authors = ["Matt Palmer <[email protected]>"]
+authors = [
+	"Matt Palmer <[email protected]>",
+	"Ben Heidemann <[email protected]>",
+	"Austin Ward <[email protected]>"
+]
 edition = "2021"
 # If this is changed, .github/workflows/qa.yml build matrix needs updating as well
 rust-version = "1.60.0"
@@ -20,7 +24,8 @@ rust-version = "1.60.0"
 crate-type = ["cdylib", "rlib"]
 
 [features]
-js = ["console_error_panic_hook", "wee_alloc", "valico/js"]
+js = ["console_error_panic_hook", "valico/js"]
+save-snapshots = []
 
 [dependencies]
 clap = { version = "4.0", features = ["derive"] }
@@ -39,14 +44,12 @@ wasm-bindgen = "0.2.63"
 # code size when deploying.
 console_error_panic_hook = { version = "0.1.6", optional = true }
 
-# `wee_alloc` is a tiny allocator for wasm that is only ~1K in code size
-# compared to the default allocator's ~10K. It is slower than the default
-# allocator, however.
-wee_alloc = { version = "0.4.5", optional = true }
 serde-wasm-bindgen = "0.4.5"
 
 [dev-dependencies]
 wasm-bindgen-test = "0.3.13"
+rstest = "0.16.0"
+assert_cmd = "2.0.8"
 
 [profile.release]
 # Tell `rustc` to optimize for small code size.

README.md

@@ -104,7 +104,7 @@ Update your .pre-commit-config.yaml:
 ```
 repos:
 - repo: https://github.com/mpalmer/action-validator
-  rev: v0.4.0
+  rev: v0.5.1
   hooks:
     - id: action-validator
 ```

test/003_successful_globs/glob.yml → tests/003_successful_globs/glob.yml

@@ -3,7 +3,7 @@ name: Test
 on:
   push:
     paths:
-      - 003_successful_globs/*
+      - ./tests/003_successful_globs/*
 
 defaults:
   run:

test/004_failing_globs/glob.yml → tests/004_failing_globs/glob.yml

@@ -3,7 +3,7 @@ name: Bad globs, no biscuit
 on:
   push:
     paths:
-      - 004_bad_globs/*.txt
+      - ./tests/004_bad_globs/*.txt
 
 defaults:
   run:

test/004_failing_globs/stderr → tests/004_failing_globs/stderr

@@ -3,13 +3,13 @@ Validation failed: ValidationState {
         Workflow,
     ),
     file_path: Some(
-        "004_failing_globs/glob.yml",
+        "./tests/004_failing_globs/glob.yml",
     ),
     errors: [
         NoFilesMatchingGlob {
             code: "glob_not_matched",
             detail: Some(
-                "Glob \"004_bad_globs/*.txt\" in /on/push/paths does not match any files",
+                "Glob \"./tests/004_bad_globs/*.txt\" in /on/push/paths does not match any files",
             ),
             path: "/on/push/paths",
             title: "Glob does not match any files",

tests/004_failing_globs/stdout

@@ -0,0 +1 @@
+Fatal error validating ./tests/004_failing_globs/glob.yml

test/007_funky_syntax/stderr → tests/007_funky_syntax/stderr

@@ -3,7 +3,7 @@ Validation failed: ValidationState {
         Workflow,
     ),
     file_path: Some(
-        "007_funky_syntax/rust-check.yml",
+        "./tests/007_funky_syntax/rust-check.yml",
     ),
     errors: [
         Parse {

tests/007_funky_syntax/stdout

@@ -0,0 +1 @@
+Fatal error validating ./tests/007_funky_syntax/rust-check.yml

test/008_job_dependencies/stderr → tests/008_job_dependencies/stderr

@@ -3,7 +3,7 @@ Validation failed: ValidationState {
         Workflow,
     ),
     file_path: Some(
-        "008_job_dependencies/test.yml",
+        "./tests/008_job_dependencies/test.yml",
     ),
     errors: [
         UnresolvedJob {

tests/008_job_dependencies/stdout

@@ -0,0 +1 @@
+Fatal error validating ./tests/008_job_dependencies/test.yml

test/009_multi_file/stderr → tests/009_multi_file/stderr

@@ -3,7 +3,7 @@ Validation failed: ValidationState {
         Workflow,
     ),
     file_path: Some(
-        "009_multi_file/xinvalid.yml",
+        "./tests/009_multi_file/xinvalid.yml",
     ),
     errors: [
         Parse {

tests/009_multi_file/stdout

@@ -0,0 +1 @@
+Fatal error validating ./tests/009_multi_file/xinvalid.yml

tests/010_remote_checks_ok/test.yml

@@ -0,0 +1,78 @@
+name: Test
+
+on:
+  push:
+  pull_request:
+    branches:
+      - main
+
+defaults:
+  run:
+    shell: bash
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Setup Rust
+        uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          override: true
+          default: true
+
+      - name: Check Formatting
+        uses: actions-rs/cargo@v1
+        with:
+          command: fmt
+          args: -- --check
+
+      - name: Check with Clippy
+        uses: actions-rs/clippy-check@v1
+        with:
+          args: -- -Dwarnings
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Shellcheck
+        uses: ludeeus/action-shellcheck@master
+
+      - name: Install shfmt
+        uses: mfinelli/setup-shfmt@master
+
+      - name: Run shfmt
+        run: shfmt -d bin/*
+
+
+  build:
+    strategy:
+      matrix:
+        rust-toolchain:
+          - stable
+          - nightly
+        os:
+          - ubuntu-latest
+          - macos-latest
+          - windows-latest
+
+    runs-on: ${{ matrix.os }}
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Setup Rust
+        uses: actions-rs/toolchain@v1
+        with:
+          toolchain: ${{ matrix.rust-toolchain }}
+          override: true
+          default: true
+
+      - name: Build
+        uses: actions-rs/cargo@v1
+        with:
+          command: build
+          args: --release --all-features

tests/011_remote_checks_failure/exitcode

@@ -0,0 +1 @@
+1

tests/011_remote_checks_failure/stderr

@@ -0,0 +1,18 @@
+Validation failed: ValidationState {
+    action_type: Some(
+        Workflow,
+    ),
+    file_path: Some(
+        "./tests/011_remote_checks_failure/test.yml",
+    ),
+    errors: [
+        Unknown {
+            code: "action_not_found",
+            detail: Some(
+                "Could not find action: actions/checkouts@v2",
+            ),
+            path: "jobs/build/steps/uses/actions/checkouts@v2",
+            title: "Action Not Found",
+        },
+    ],
+}

tests/011_remote_checks_failure/stdout

@@ -0,0 +1 @@
+Fatal error validating ./tests/011_remote_checks_failure/test.yml

tests/011_remote_checks_failure/test.yml

@@ -0,0 +1,78 @@
+name: Test
+
+on:
+  push:
+  pull_request:
+    branches:
+      - main
+
+defaults:
+  run:
+    shell: bash
+
+jobs:
+  check:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - name: Setup Rust
+        uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+          override: true
+          default: true
+
+      - name: Check Formatting
+        uses: actions-rs/cargo@v1
+        with:
+          command: fmt
+          args: -- --check
+
+      - name: Check with Clippy
+        uses: actions-rs/clippy-check@v1
+        with:
+          args: -- -Dwarnings
+          token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Shellcheck
+        uses: ludeeus/action-shellcheck@master
+
+      - name: Install shfmt
+        uses: mfinelli/setup-shfmt@master
+
+      - name: Run shfmt
+        run: shfmt -d bin/*
+
+
+  build:
+    strategy:
+      matrix:
+        rust-toolchain:
+          - stable
+          - nightly
+        os:
+          - ubuntu-latest
+          - macos-latest
+          - windows-latest
+
+    runs-on: ${{ matrix.os }}
+
+    steps:
+      - name: Checkout
+        uses: actions/checkouts@v2
+
+      - name: Setup Rust
+        uses: actions-rs/toolchain@v1
+        with:
+          toolchain: ${{ matrix.rust-toolchain }}
+          override: true
+          default: true
+
+      - name: Build
+        uses: actions-rs/cargo@v1
+        with:
+          command: build
+          args: --release --all-features