CNTK Evaluation Overview

This page presents an overview of model evaluation using CNTK.

##CNTK model evaluation methods Aside from training a model, CNTK also provides two different ways of evaluating the model. The first method is using the CNTK executable itself (similar to the training process), but instead of using the "train" command, the "eval" command is placed in the configuration file. The alternate method for evaluating a CNTK model is to use the evluation library, which is currently available as C++ and C# library in Windows and C++ library in Linux. The CNTK source repository has several examples consuming this library (from C# and C++).

##Evaluating a model using CNTK.exe Using the CNTK executable for evaluation has the following advantages: ###CPU/GPU capability Like training, CNTK can leverage the GPU during evaluation. Refer to the Config File Overview page for more details. ###Readers (and their transformations)
Similar to model training, the reader plugins (e.g. ImageReader) may perform some data transforms on the input data prior to feeding it to the network during training. These transforms are not part of CNTK (per se), but of the readers. In order to feed the same transformed data during evaluation, the transformations need to occur prior to feeding. When evaluating using the CNTK executable, the same reader (as used during evaluation) can be used, and thus the same transformation can be applied. As we will cover later in this page, when using the programmatic approach, these transforms will need to be performed programmatically outside of evaluation engine prior to submitting the data for evaluation (assuming the model was trained with transformed data).
###Model tweaking When using CNTK for evaluation, there is a possibility of modifying the model's layout using BrainScript. This enables additional capabilities, such as exposing hidden layers for evaluation. Refer to the BrainScript page for more information.

Using the CNTK executable for evaluation has the following disadvantages: ###Process spin-up time The CNTK executable (by nature) runs as a process, and thus will take some time to spin up. For services where many requests need to be dynamically processed, the better option would be to use the EvalDLL (or EvalWrapper) in a service. ###File based The CNTK executable reads the input data from file(s) and writes the output data to a file. For services running in the cloud, this may cause some performance issues.

Note: If you do go the route of evaluating a CNTK model with the CNTK executable, make sure your parameters are adequate for the evaluation. In particular specifying an appropriate size for the MiniBatchSize. Please refer to the Common Pitfalls page for more information.

##Evaluating a model programmatically using C++ in Windows The EvalDLL.dll is a Windows dll that enables programmatic model evaluation. The dll can only be used in CPU only mode (no GPU is used).
The usage pattern for this dll is the following:

1. Load the EvalDLL.dll library into the process.
2. Get an instance of the evaluation engine specific for the model's data type (float or double).
3. Load the model (or create the network) in the evaluation engine.
4. Evaluate some input against the model and obtain the corresponding output.
5. Dispose of the model when done.

The CNTK source code solution contains a C++ example (named CPPEvalClient) illustrating the usage pattern above. The sample is located under the Examples/Evaluation/CPPEvalClient folder in the source code.

Note: The Windows C++ project must be compiled with an x64 target configuration platform, otherwise some issues arise when calling the EvalDLL library. Refer to the Common Pitfalls page for more information.

For details on the C++ API provided by the EvalDLL.dll, please refer to the C++ Evaluation Interface page in this wiki.

##Evaluating a model programmatically using C# in Windows CNTK provides a managed (.Net) library wrapper named EvalWrapper.dll. This library wraps the native EvalDLL library and exposes a managed interface. This interface provides the same functionality as the native interface, with the addition of some convenience methods.
Alike its native counterpart, this library can only perform evaluations using the CPU (no GPU used). The library is written in CLI/C++ and thus forms the bridge between .Net (e.g. C#) and the native C++ side.

For details regarding the managed API provided by EvalWrapper.DLL, refer to the Managed Evaluation Interface page.

The usage pattern for the managed wrapper is simple:

using Microsoft.MSR.CNTK.Extensibility.Managed;
...
try
{
    using (var model = new IEvaluateModelManagedF())
    {
        // Load model
        model.CreateNetwork(...);
        model.Evaluate(...);
    }
}
catch (CNTKException ex)
{
...
}
catch (Exception ex)
{
...
}

There are several examples of using a programmatic CNTK model evaluation in C# inside the CSEvalClient project.

Note: The Windows .Net project must be compiled with an x64 target configuration platform, otherwise some issues arise when calling the EvalWrapper (and EvalDLL) library. Refer to the Common Pitfalls page for more information.

##NuGet Package There is currently a NuGet package at Nuget.org (search for CNTK) that provides both the native and managed versions for Debug and Release for the CNTK evaluation libraries (CPU only using MKL). With the NuGet it is possible to simply add the CNTK Eval NuGet to a .Net or Win32 project and call the APIs. Please refer to the NuGet Package page for details on how to get started with CNTK and NuGet.

##Evaluating a model programmatically using C++ in Linux The libeval.so is the library for model evaluation in Linux.

The usage pattern of libeval.so is the following:

1. Get an instance of the evaluation engine either using GetEvalF() (for the float data type) or GetEvalD() (for the double data type).
2. Load the model (or create the network) in the evaluation engine.
3. Evaluate some input against the model and obtain the corresponding output.
4. Dispose of the model when done.

Any program using the evaluation library needs to link libeval.so and libcntkmath.so when compiling the program, for example by specifying "-leval -lcntkmath" as well as appropriate search path for both libraries.

The C++ example CPPEvalClient, availble in the CNTK source code under the Examples/Evaluation/CPPEvalClient folder illustrates the usage pattern in Linux. The Makefile under the root folder in the source code contains the target EVAL_SAMPLE_CLIENT showing how to build the example.

For details on the C++ API provided by the libeval.so, please refer to the C++ Evaluation Interface page in this wiki.

##Evaluating different data types and layers Currently the Eval library supports vectors for input and output. That means that the input vector must match the input nodes in the model (features). Some models are trained with images (e.g. CIFAR-10) however, these images are vectorized first then fed into the network. For example, the CIFAR-10 data set is composed of small images (32 pixels by 32 pixels) or RGB values. Although each is a 3 dimensional coordinate (width, height, color), the data is vectorized into a 1-dimensional vector. It is important thus, to convert the raw data to the vector format prior to evaluation. This conversion should be done in the same manner as when fed to the network for training.

Please refer to the Evaluate Image Transforms page for more information, particularly when dealing with images.

Although an already trained model has a specific set of output nodes, it is sometimes desirable to obtain the values of other nodes during evaluation (e.g. hidden layers). This is possible using the programmatic interface, please refer to the Evaluate Hidden Layers page for more information.

##Current interface The current evaluation interface is described in the following pages:

• For the C++ API refer to the C++ Evaluation Interface page.
• For the C# (.Net) API refer to the C# Evaluation Interface page.

##Future 2.0 interface We are currently working on more capability exposed through the Evaluation APIs.

##Current limitations

• Single threaded evaluation
  The CNTK EvalDLL library (and by extension the managed EvalWrapper library) is single threaded. That means that a single model can only be evaluated by a single thread at a time. However, it is possible to load multiple models in the same process and query each model using a single thread. Refer to the CSEvalClient for an example.