Running Tests

Introduction

Each test in the test suite runs Celeriac on a source program and compares the generated output to a known correct result. Tests exist for specific features (e.g. the MultipleExits test) as well as for general programs (e.g. the GeoPoint test).

Setting up Daikon

Executing the tests requires installation of Daikon. Follow the installation instructions for Daikon, including setting the DAIKONDIR environment variable.

If using the Windows Java executable, add the following line to your .bashrc file translate the Java classpath to the Window's format:

export CLASSPATH=`cygpath -wp "$CLASSPATH"`

Additionally, add the path to the Celeriac test folder using the Windows directory convention, e.g.:

export CELERIAC_TESTS="C:/Projects/celeriac-source/CeleriacTests"

Setting up Celeriac and Test System

• Copy Makefile.user.template to Makefile.user
• In Makefile.user, change the value of CELERIAC_EXE to the path to CeleriacLauncher.exe. Note: you must use the Cygpath (e.g., /cygdrive/c/...) since colons on the path confuse Make.
• Ensure that the CCI Metadata DLLs are in the same folder as CeleriacLauncher.exe

Running the Tests

All tests are in the CeleriacTests directory. Change into this directory and execute make. This will run all the tests, including the GUI tests, and print the result of each to the screen.

Graphical User Interface Testing with Sikuli

Many programs have a Graphical User Interface (GUI), and exercising their features requires interacting with the GUI. To ensure Celeriac runs properly on these programs the test suite contains test programs with GUI's/ To perform automated testing on these kind of applications the Celeriac test framework uses the Sikuli tool.

Sikuli is a script based automation framework. Sikuli scripts are python programs, but can perform image based matching to click on certain button or other visual elements.

Getting Sikuli

Sikuli can be downloaded here; tutorials demonstrating how to use Suikuli are located here.

Adding tests of GUI applications, using Sikuli

The Celeriac testing framework usually runs by executing the CeleriacLauncher on a test source program. Sikuli seems to have errors executing from Cygwin trying to execute on a program launched from Cygwin, so the Sikuli tests launch the CeleriacLauncher and source program in the test script. For GUI applications tests should be run with Sikuli instead of the traditional setup (where a program is simply execture). To specify this add the line

SIKULI_RUN = 1

to the Makefile for that test. A Sikuli executable named {TestName}.skl must be placed in the test folder and added to the repository. As well, all Sikuli resources, usually in a folder ending in .sikuli, must be added to the repository as well.

Common Problems

Perl "Can't execute sort-dtrace.pl"

Ensure that permissions for the sort-dtrace.pl allow it to be executed by Perl. Setting the permissions from Windows may not work, so instead use the chmod command, e.g.,:

chmod 777 sort-dtrace.pl

The System Hangs with 100% CPU Utilization

For some users, running Celeriac from Cygwin causes the Windows Defender real-time protection service to run at 100% utilization. If this occurs, temporarily disable real-time protection while running the test suite.

Known Issues

• The MultipleThreads test fails the dtrace.diff and out.diff, this is expected because thread execution order is not deterministic.