The system tests consist of a series of automated tests for this repository that test it in a way similar to how it would be used in production.
It uses Docker containers to create containerised instances of Kafka and other components.
Requires Python 3.6+.
[optional] Set up a Python virtual environment and activate it (see here)
-
Install Docker
-
Install the requirements using pip:
pip install -r system-tests/requirements.txt
-
Stop and remove any containers that may interfere with the system tests, e.g IOC or Kafka containers and containers from previous runs. To stop and remove all containers use
docker stop $(docker ps -a -q) && docker rm $(docker ps -a -q)
-
If you have a local conan server then set environment variable
local_conan_server=<ADDRESS OF SERVER>
, if there are binaries for conan packages on the server this makes a huge difference in time taken to build the file writer docker image. -
Run
python -m pytest -s .
from thesystem-tests/
directory -
Optionally use
--local-build <PATH_TO_BUILD_DIR>
to run against a local build of the file writer rather than rebuilding in a docker container. Note that the build directory is the one containing thebin
directory. -
To run a single test, use the
-k
argument (e.g.)python -m pytest -s . -k 'test_two_different_writer_modules_with_same_flatbuffer_id'
. -
Can also use
--wait-to-attach-debugger true
to cause the system tests to display the process ID of the file writer and give opportunity for you to attach a debugger before continuing.
Note: these tests may take up to 30 minutes to run.
The system tests use pytest for the test runner, and use separate fixtures for different configurations of the file-writer.
Firstly, the system tests attempt to build and tag the latest file-writer image. This can take a lot of time especially if something in conan has changed, as it has to reinstall all of the conan packages. The image build is layer-cached with three main layers - pulling the base image, dependency gathering (from apt, pip and conan) and building the file-writer. This strikes a balance between time taken to rebuild the image when something changes, and the disk space used by the cached layers.
The Kafka and Zookeeper containers are started with docker-compose
and persist throughout all of the tests, and when finished will be stopped and removed.
Each fixture starts the file-writer with an ini
config file (found in /config-files
). In some cases the fixtures use a JSON command at startup and in others the command is sent over Kafka as part of the test. Some fixtures start the file-writer multiple times or use the NeXus-streamer image as a source of streamed data.
In some tests, command messages in JSON
form are sent to Kafka to change the configuration of the file-writer during testing.
Most tests check the NeXus file created by the file-writer contains the correct static and streamed data, however, some tests instead test that the status of the file writer matches expectation, by consuming status messages from Kafka.
Log files are placed in the logs
folder in system-tests
provided that the ini
file is using the --log-file
flag and the docker-compose file mounts the logs
directory.
For performance reasons the docker image containing the filewriter is generated using a Python script (create_filewriter_image.py) rather than a Dockerfile. By default, this script will leave behind a container which can be re-used by the script for, relatively speaking, quickly generating a new image. If you for whatever reason want a smaller docker image, you can run the script for generating the image like this:
python create_filewriter_image.py --do-cleanup
Note that this will delete the container when it is done and thus require more time next time it is executed.
To create a new fixture, a new function should be added in conftest.py
as well as a docker compose file in compose/
and a startup ini
config file. The test itself should be created in a file with the prefix test_
, for example test_idle_pv_updates
, so that file can be picked up by pytest.
The fixture name must be used as the first parameter to the test like so:
def test_data_reaches_file(docker_compose):
black
formatting tool should be used to keep formatting of the system-test scripts consistent. A particular version of black is specified in the requirements file.
It can be run from the root of the repository like this:
black system-tests
pyproject.toml
has been configured to tell black
to exclude python scripts which were generated by flatc
, so that newly generated files can be compared if necessary.