This is the SYCL Conformance Test Suite for the Khronos Group SYCL standard.
The test suite comes in the form of multiple binary executables that must be
compiled before running, a step that requires CMake. All test cases inside the
tests directory are grouped into categories. Compilation produces
executables for each category and one fat executable that contains the tests
from all categories.
Important: Due to its large size, by default the CTS is compiled in a lighter configuration that omits some combinations of testing logic for the sake of reduced compilation and execution time. This can be useful during development, however is not sufficient to establish conformance. See the sections on CMake Configuration Options and Generating a Conformance Report for more information.
To compile the CTS, the following dependencies are required:
- Python 3.7 or higher
- CMake 3.15 or higher
- A conformant OpenCL implementation
- A SYCL implementation
- The CTS currently supports ComputeCpp, DPC++ and hipSYCL
- See the AddSYCLExecutable.cmake module on how to add support for additional SYCL implementations
Configuration and compilation then follow standard CMake procedures. Begin by cloning this repository and its submodules:
git clone --recurse-submodules https://github.com/KhronosGroup/SYCL-CTS.git
Then enter the SYCL-CTS folder and configure the build using CMake:
cmake -S . -B build -DSYCL_IMPLEMENTATION=<ComputeCpp|DPCPP|hipSYCL>
See CMake Configuration Options for additional configuration options that can be passed here.
Finally, start the compilation:
cmake --build ./build
After the compilation has finished, test executables for each category will be
placed in the build/bin directory. The test_all executable contains tests
for all categories.
The CTS can be configured using the following CMake configuration options:
SYCL_IMPLEMENTATION (default: ComputeCpp)
One of ComputeCpp, DPCPP or hipSYCL.
SYCL_CTS_EXCLUDE_TEST_CATEGORIES (default: None)
Optional file specifying a list of test categories to be excluded from the build.
SYCL_CTS_ENABLE_FULL_CONFORMANCE (default: OFF)
Enable extended type coverage and testing logic with significantly increased
compilation and execution time. This mode is required to establish the
conformance of a SYCL implementation.
SYCL_CTS_VERBOSE_LOG (default: OFF)
Enable verbose debug-level logging.
SYCL_CTS_ENABLE_DOUBLE_TESTS (default: ON)
Enable tests that require double precision floating point capabilities.
SYCL_CTS_ENABLE_HALF_TESTS (default: ON)
Enable tests that require half precision floating point capabilities.
SYCL_CTS_ENABLE_DEPRECATED_FEATURES_TESTS (default ON)
Enable tests for legacy SYCL features. Should be switched on for conformance.
SYCL_CTS_ENABLE_OPENCL_INTEROP_TESTS (default: ON)
Enable OpenCL interoperability tests.
Additionally, the following SYCL implementation-specific options can be used:
COMPUTECPP_INSTALL_DIR (default: None)
DPCPP_INSTALL_DIR (default: None)
Each of the executables produced in the build/bin directory acts as a
standalone test runner that can be used to launch tests for a particular test
category (or all tests in the case of build/bin/test_all).
The --device argument is used to specify which device to run the tests on.
Selection is based on substring matching of the device name. ECMAScript regular
expression syntax is supported. To get a list of all available devices, use
--list-devices.
Please see <test_executable> --help for a complete list of available filtering
and output formatting options.
To generate a conformance report, use the run_conformance_tests.py script.
This script automates the configuration, compilation and execution of the CTS,
generating a report file conformance_report.xml. By default, the script will
enable the SYCL_CTS_ENABLE_FULL_CONFORMANCE option, resulting in long
compilation and execution times.
Please see run_conformance_tests.py --help for a complete list of available
options.