Validation Suite Design

Components

Model

A model represents, abstractly, a particular physical system that is going to be the basis of a simulation run. It may define a set of global parameters that can be customized.

An example would be a model representing a single compartment RC-circuit for stability testing, with parameters describing the minimum and maximum δt to test.

Simulator model implementation

A script that implements a given model for a specific simulator, takes model parameters as arguments, and emits numerical simulation results in an accepted data representation format.

Per-model reference data generation

A script that creates the reference test data for analysis, if any is required. It takes model parameters as arguments. It might possibly also examine the simulator output in order to determine sample points or times or similar. Some reference data may be pre-generated, making the script implementation trivial.

Analysis

A model-specific or generic analysis script that compares simulator output with reference data and produces summary data.

Validation

A pass/fail criterion applied to the generated summary data.

Procedure

• For each simulator do:
  • For each model configuration (model + pre-defined model parameter set) with an implementation for this simulator do:
    1. Run implementation with model parameters.
    2. Generate analysis data for model and model parameters (this can be cached).
    3. Run analysis script.
    4. Run validation test on analysis summary data.