CNTK Evaluation Overview

This page presents an overview of model evaluation using CNTK.

##CNTK model evaluation methods Aside from training a model, CNTK also provides two different ways of evaluating the model. The first method is using the CNTK executable itself (similar to the training process), but instead of using the "train" command, the "eval" command is placed in the configuration file. The alternate method for evaluating a CNTK model is to use the evaluation library, which is currently available as C++ library (in Windows and Linux) and C# library (only in Windows). The CNTK source repository has several examples consuming this library (from C# and C++).

##Evaluating a model using CNTK.exe Using the CNTK executable for evaluation has the following advantages: ###CPU/GPU capability Like training, CNTK can leverage the GPU during evaluation. Refer to the Config File Overview page for more details. ###Readers (and their transformations)
Similar to model training, the reader plugins (e.g. ImageReader) may perform some data transforms on the input data prior to feeding it to the network during training. These transforms are not part of CNTK (per se), but of the readers. In order to feed the same transformed data during evaluation, the transformations need to occur prior to feeding. When evaluating using the CNTK executable, the same reader (as used during evaluation) can be used, and thus the same transformation can be applied. As we will cover later in this page, when using the programmatic approach, these transforms will need to be performed programmatically outside of evaluation engine prior to submitting the data for evaluation (assuming the model was trained with transformed data).
###Model tweaking When using CNTK for evaluation, there is a possibility of modifying the model's layout using BrainScript. This enables additional capabilities, such as exposing hidden layers for evaluation. Refer to the BrainScript page for more information.

Using the CNTK executable for evaluation has the following disadvantages: ###Process spin-up time The CNTK executable (by nature) runs as a process, and thus will take some time to spin up. For services where many requests need to be dynamically processed, the better option would be to use the EvalDLL (or EvalWrapper) in a service. ###File based The CNTK executable reads the input data from file(s) and writes the output data to a file. For services running in the cloud, this may cause some performance issues.

Note: If you do go the route of evaluating a CNTK model with the CNTK executable, make sure your parameters are adequate for the evaluation. In particular specifying an appropriate size for the MiniBatchSize. Please refer to the Troubleshoot CNTK page for more information.

##Evaluating a model programmatically in Windows The CNTK evaluation library in Windows is provided both as C++ and C# library. A Nuget package is also available at Nuget.org.

Using the C++ library

The EvalDLL.dll is a Windows dll that enables programmatic model evaluation. The dll can only be used in CPU only mode (no GPU is used).
The usage pattern for this dll is the following:

1. Link the EvalDLL.lib import library into the application.
2. Include the evaluation header file "Eval.h"
3. Get an instance of the evaluation engine specific for the model's data type (float or double).
4. Load the model (or create the network) in the evaluation engine.
5. Evaluate some input against the model and obtain the corresponding output.
6. Dispose the model when done.

The CNTK source code solution contains a C++ example (named CPPEvalClient) illustrating the usage pattern above. The sample is located under the Examples/Evaluation/CPPEvalClient folder in the source code. Please make sure the path to CNTK Eval dlls and its dependencies are included in the search path of dlls for your application.

Note: The Windows C++ project must be compiled with an x64 target configuration platform, otherwise some issues arise when calling the EvalDLL library. Refer to the Troubleshoot CNTK page for more information.

For details on the C++ API provided by the EvalDLL.dll, please refer to the C++ Evaluation Interface page in this wiki.

Using the C# library

CNTK provides a managed (.Net) library wrapper named EvalWrapper.dll. This library wraps the native EvalDLL library and exposes a managed interface. This interface provides the same functionality as the native interface, with the addition of some convenience methods.
Alike its native counterpart, this library can only perform evaluations using the CPU (no GPU used). The library is written in CLI/C++ and thus forms the bridge between .Net (e.g. C#) and the native C++ side.

For details regarding the managed API provided by EvalWrapper.DLL, refer to the Managed Evaluation Interface page.

The usage pattern for the managed wrapper is simple:

using Microsoft.MSR.CNTK.Extensibility.Managed;
...
try
{
    using (var model = new IEvaluateModelManagedF())
    {
        // Load model
        model.CreateNetwork(...);
        model.Evaluate(...);
    }
}
catch (CNTKException ex)
{
...
}
catch (Exception ex)
{
...
}

There are several examples of using a programmatic CNTK model evaluation in C# inside the CSEvalClient project.

Note: The Windows .Net project must be compiled with an x64 target configuration platform, otherwise some issues arise when calling the EvalWrapper (and EvalDLL) library. Refer to the Troubleshoot CNTK page for more information.

###NuGet Package There is currently a NuGet package at Nuget.org (search for CNTK) that provides both the native and managed versions for Debug and Release for the CNTK evaluation libraries (CPU only using MKL). With the NuGet it is possible to simply add the CNTK Eval NuGet to a .Net or Win32 project and call the APIs. Please refer to the NuGet Package page for details on how to get started with CNTK and NuGet.

If you do not want to use NuGet Package, you can add EvalWrapper.dll as reference to your project. Please make sure in this case that the path to EvalWrapper.dll and its dependencies are included in the search path of dlls for your application.

###Eval samples in CNTK binary download package for Windows The CNTK binary download package for Windows also includes samples for using the eval library in C++ and C#. Please refer to the Binary Download page for details.

###Shipping Eval functionality with your Windows application If you own application uses CNTK Eval you need to distribute these DLLs with your application:

• EvalDll.dll
• EvalWrapper.dll
• Math.dll
• libiomp5md.dll
• mkl_cntk_p.dll

All these dlls can be found in the CNTK Binary Download Package.

##Evaluating a model programmatically in Linux The CNTK evaluation library in Linux is provided as a C++ library.

The usage pattern for evaluation is the following:

1. Get an instance of the evaluation engine either using GetEvalF() (for the float data type) or GetEvalD() (for the double data type).
2. Load the model (or create the network) in the evaluation engine.
3. Evaluate some input against the model and obtain the corresponding output.
4. Dispose the model when done.

The evaluation library, libeval.so, can be found under cntk\lib in the CNTK binary package. If you build CNTK from source code, the libeval.so is available in the lib folder of the build directory.

Any program using the evaluation library needs to link libeval.so and libcntkmath.so when compiling the program, for example by specifying "-leval -lcntkmath" as well as appropriate search path for both libraries. Please use the same compiler version as that used to build libraries. The Examples/Evaluation/CPPEvalClient in the CNTK source code illustrates the usage pattern in Linux. The Makefile contains the target EVAL_SAMPLE_CLIENT showing how to build the example.

For details on the C++ API provided by the libeval.so, please refer to the C++ Evaluation Interface page in this wiki.

##Evaluating different data types and layers Currently the Eval library supports vectors for input and output. That means that the input vector must match the input nodes in the model (features). Some models are trained with images (e.g. CIFAR-10) however, these images are vectorized first then fed into the network. For example, the CIFAR-10 data set is composed of small images (32 pixels by 32 pixels) or RGB values. Although each is a 3 dimensional coordinate (width, height, color), the data is vectorized into a 1-dimensional vector. It is important thus, to convert the raw data to the vector format prior to evaluation. This conversion should be done in the same manner as when fed to the network for training.

Please refer to the Evaluate Image Transforms page for more information, particularly when dealing with images.

Although an already trained model has a specific set of output nodes, it is sometimes desirable to obtain the values of other nodes during evaluation (e.g. hidden layers). This is possible using the programmatic interface, please refer to the Evaluate Hidden Layers page for more information.

##Current interface The current evaluation interface is described in the following pages:

• For the C++ API refer to the C++ Evaluation Interface page.
• For the C# (.Net) API refer to the C# Evaluation Interface page.

##Future 2.0 interface We are currently working on more capability exposed through the Evaluation APIs.

##Current limitations

• Single threaded evaluation
  The CNTK evaluation library, and by extension the managed EvalWrapper library, are single threaded and single re-entrancy. That means that a single model can only be evaluated by a single thread at a time. However, it is possible to load multiple models in the same process and query each model using a single thread.
• Any program that links the pre-built evaluation libraries (EvalDLL.dll, EvalWrapper.dll in Windows, and libeval.so in Linux) of the CNTK binary package should use the same compiler version as that is used to build the pre-built libraries.