This repository is a template repository for creating test runners for Exercism tracks.
- Follow GitHub's documentation for creating a repository from a template repository
- Name your new repository based on your language track (i.e. if your track is for Python, your test runner repo name is
python-test-runner
) - In your new repository:
- Remove the Exercism Test Runner Template section from README.md
- Modify
Dockerfile
,.dockerignore
andbin/run.sh
to match your track's needs
- The interface that test runners must conform to is defined in this document.
Once you're happy with your test runner, open an issue on the exercism/automated-tests repo to request an official test runner repository for your track.
The Docker image to automatically run tests on TRACK_NAME_HERE solutions submitted to Exercism.
To run all of the tests, do the following:
- Open a terminal in the project's root
- Run
bin/run-tests.sh
These are golden tests that compare the results.json
generated by running the current state of the code against the "known good" test/<test-name>/results.json
. All files created during the test run itself are discarded.
When you've made modifications to the code that will result in a new "golden" state, you'll need to generate and commit a new test/<test-name>/results.json
file.
To run the tests of an arbitrary exercise, do the following:
- Open a terminal in the project's root
- Run
./bin/run.sh <exercise-slug> <path/to/solution/dir/> </path/to/output/dir/>
Once the test runner has finished, its results will be written to /path/to/output/dir/results.json
.
This script is provided for testing purposes, as it mimics how test runners run in Exercism's production environment.
To run the tests of an arbitrary exercise using the Docker container, do the following:
- Open a terminal in the project's root
- Run
./bin/run-in-docker.sh <exercise-slug> <./relative/path/to/solution/dir/> <./relative/path/to/output/dir/>
Once the test runner has finished, its results will be written to ./relative/path/to/output/dir/results.json
.
Note: the solution and output directory's MUST be relative paths!
To run the tests to verify the behavior of the test runner, do the following:
- Open a terminal in the project's root
- Run
./bin/run-tests.sh
These are golden tests that compare the results.json
generated by running the current state of the code against the "known good" test/<test-name>/results.json
. All files created during the test run itself are discarded.
When you've made modifications to the code that will result in a new "golden" state, you'll need to generate and commit a new test/<test-name>/results.json
file.