Skip to content

Latest commit

 

History

History
129 lines (105 loc) · 6.42 KB

tests.md

File metadata and controls

129 lines (105 loc) · 6.42 KB
Raw

ONNX Frontend tests

ONNX Frontend testing places

How to run the tests locally

C++ tests

  1. Build OpenVINO with the -DENABLE_TESTS=ON flag.
  2. Run <OV_REPO_DIR>/bin/intel64/<OV_BUILD_TYPE>/ov_onnx_frontend_tests, where <OV_REPO_DIR> is your workspace folder, and OV_BUILD_TYPE can be Debug or Release, depending on the -DCMAKE_BUILD_TYPE CMake build option. You can filter tests using the --gtest_filter flag like any other gtest-based tests.

For example:

<OV_REPO_DIR>/bin/intel64/<OV_BUILD_TYPE>/ov_onnx_frontend_tests --gtest_filter=*add*

Pre-steps for all Python tests

  1. Build OpenVINO with -DENABLE_PYTHON=ON, preferably in a Python virtual environment. To avoid problems with too many Python interpreters installed on the host, you can also set the -DPython3_EXECUTABLE=<PYTHON_INTERPRETER_PATH> build option (requires cmake 3.16 and higher).

NOTE: If you want to run the tests from the installation directory (like in the CI), add the -P cmake_install.cmake and -DCOMPONENT=tests CMake build options, and install OpenVINO via cmake --build . --target install as additional steps.

  1. Set up Python paths via source <OV_INSTALL_DIR>/setupvars.sh for Linux, or sh <INSTALL_DIR>\setupvars.bat for Windows.
  2. Install Python dependencies:
pip install -r <OV_REPO_DIR>/src/bindings/python/requirements.txt
pip install -r <OV_REPO_DIR>/src/bindings/python/requirements_test.txt

Python frontend tests

You can run ONNX Frontend Python tests using the following commands:

  • For the build layout:
pytest <OV_REPO_DIR>/src/bindings/python/tests/test_frontend/test_frontend_onnx*
  • For the installation layout:
pytest <OV_INSTALL_DIR>/tests/pyopenvino/tests/test_frontend/test_frontend_onnx*

Python ONNX operators tests

You can run ONNX operators Python tests using the following commands:

  • For the build layout:
pytest <OV_REPO_DIR>/src/bindings/python/tests/test_onnx \
    --ignore=<OV_REPO_DIR>/src/bindings/python/tests/test_onnx/test_zoo_models.py \
    --ignore=<OV_REPO_DIR>/src/bindings/python/tests/test_onnx/test_backend.py
  • For the installation layout:
pytest <OV_INSTALL_DIR>/tests/pyopenvino/tests/test_onnx \
    --ignore=<OV_INSTALL_DIR>/tests/pyopenvino/tests/test_onnx/test_zoo_models.py \
    --ignore=<OV_INSTALL_DIR>/tests/pyopenvino/tests/test_onnx/test_backend.py

Python tests to confirm operator compliance with the ONNX standard

  • For the build layout:
pytest <OV_REPO_DIR>/src/bindings/python/tests/test_onnx/test_backend.py -sv -k 'not cuda'
  • For the installation layout:
pytest <OV_INSTALL_DIR>/tests/pyopenvino/tests/test_onnx/test_backend.py -sv -k 'not cuda'

Python OpenModelZoo tests

Preparation: Download models. The current OpenVINO ONNX Frontend uses this version or the model_zoo_preprocess.sh script. The second approach is preferable, because some test data requires preprocessing.

model_zoo_preprocess.sh -d <ONNX_MODELS_DIR> -o

Note that the current CI also tests models from the MSFT package (the -m flag of the model_zoo_preprocess.sh script can enable the processing of it), but it is a deprecated pipeline. If you encounter problems with this package, contact openvino-onnx-frontend-maintainers.

Commands to run OpenModelZoo tests:

  • For the build layout:
pytest --backend=CPU <OV_REPO_DIR>/src/bindings/python/tests/test_onnx/test_zoo_models.py -v -n 4 --forked -k 'not _cuda' --model_zoo_dir=<ONNX_MODELS_DIR>

Note that you can also run tests for a single model only, for example:

pytest --backend=CPU <OV_REPO_DIR>/src/bindings/python/tests/test_onnx/test_zoo_models.py -v -n 4 --forked -k 'not _cuda' --model_zoo_dir=<ONNX_MODELS_DIR> -k test_onnx_model_zoo_vision_classification_alexnet_model_bvlcalexnet_9_bvlc_alexnet_model_cpu
  • For the installation layout:
pytest --backend=CPU <OV_INSTALL_DIR>/tests/pyopenvino/tests/test_onnx/test_zoo_models.py -v -n 4 --forked -k 'not _cuda' --model_zoo_dir=<ONNX_MODELS_DIR>

Be aware that each model is tested in two stages: importing (OnnxBackendModelImportTest) and inference (OnnxBackendModelExecutionTest).

TIP: A useful pytest flag during working with OpenModelZoo tests is --collect-only. It enables you to list all the possible tests (based on available models) without running them. An example output:

<TestCaseFunction test_onnx_model_zoo_vision_style_transfer_fast_neural_style_model_rain_princess_8_rain_princess_model_cpu>
<TestCaseFunction test_onnx_model_zoo_vision_style_transfer_fast_neural_style_model_rain_princess_9_rain_princess_rain_princess_cpu>
...

Tests for OpenVINO Execution Provider with ONNX Runtime

  1. Clone the ONNX Runtime repository (the rel-1.8.1 version is used in the current CI configuration):
git clone --branch rel-1.8.1 --single-branch --recursive https://github.com/microsoft/onnxruntime.git
  • (Optional) To build ONNX Runtime with a non-release OpenVINO version, an additional step is required for the rel-1.8.1 version:
cd <OV_INSTALL_DIR>/deployment_tools/inference_engine && touch version.txt && echo "2021.4" > version.txt
  1. Set up the OpenVINO environment:
source <OV_INSTALL_DIR>/setupvars.sh
  1. Build ONNX Runtime with OpenVINO enabled:
<ONNX_RUNTIME_REPO_DIR>/build.sh --config RelWithDebInfo --use_openvino CPU_FP32 --build_shared_lib --parallel

After the building is finished, the tests will run automatically if the --skip_tests flag is not set. Note that ONNX Runtime version from the master branch requires CMake version being greater than 3.25. You can parametrize the build.sh script with your custom CMake version by passing the path via the --cmake_path argument.