-
Notifications
You must be signed in to change notification settings - Fork 17
Tests
This document is a collaborative space where we are going to specify the tests for the OSIRIS Pipeline.
Decisions to make:
- What testing framework should we use?
Things to collect:
- Real data tests (from OSIRIS outputs to final products) with simple documentation.
Code to write:
- Individual test functions in IDL for new/changed code.
Our tests should scale from individual functions to modules to pipeline data products. Tests should be easily run by the end-user, e.g. via make test. Tests should work with a continuous integration service, if possible (e.g. travis-ci.org). Some tests will work with data bundled with the pipeline, others may require external downloads.
We should select a testing framework. Many good testing frameworks exist, but I'm not familiar with any testing frameworks in IDL other than the built-in include tests at the bottom of each .pro file, which is probably too simple for our case. Some do exist (e.g. https://github.com/mgalloy/mgunit) but I'm not sure of their development status.
Alternatively we could use a testing framework from another language to help us write and run consistent tests. I'm happy to adapt one for our purposes with IDL (I've done something similar before for C code.). If that is the case, I propose we use py.test in python, wrapping IDL. This is helpful because we can use the test framework to abstract out code constants and settings, and to better report failures to the user.
- Individual functions. These should be small, and should test individual critcal functions to ensure that they work as advertized.
- Pipeline modules: These will require example pipeline inputs and outputs, and will be used to ensure each module works as intended.
- Synthetic Pipeline products: At least one case for each reduction template, where we generate the inputs and outputs so that we know that they should match exactly. I'm not sure if we have a technique for generating synthetic products, but we should probably make one.
- "Real" Pipeline products: These should be real data, preferably as simple as possible
Synthetic data can be done for just the simplest cases, but allows us to check our assumptions about the pipeline. We should have scripts to generate the synthetic data and the expceted output (essentially, something that can run the pipeline in reverse.). The trickiest part of this will be undoing the work of the rectification matrix / deconvolution step to put light back onto the CCD.
Real data tests should provide preferably simple cases of real data to be reduced by the OSIRIS pipeline, and where the products can be judged for quality by end users. Data should include the raw data products to be used as input, and the expected output from the pipeline. Hopefully these tests can be turned into tutorials other users can use to learn the OSIRIS pipeline.