CNTK Evaluation Overview

Once you have trained a model, you need the functionality to evaluate the model in your target environment. As a reminder, there are several ways to create models with CNTK, and there are two different formats to store the model in.

The original CNTK (prior to the CNTK 2.0 version) only supports a format we call now the model-V1 format. It was originally created by using CNTK.EXE with BrainScript. With CNTK 2.0 a Protobuf based format was introduced, which is now known as the model-V2 format. In CNTK 2.0, depending on parameters, the output model can be in either model-V1 or model-V2 format. The following table presents an overview on creating the different model formats:

model-creation	model-v1	model-v2
CNTK.EXE (BrainScript)	YES	NO
CNTK-library (Python, C++)	YES	YES

For more details on creating the different model formats refer to the CNTK model format page.

CNTK model evaluation methods

Aside from training a model, CNTK provides different ways of evaluating the model:

• CNTK.EXE allows loading models and evaluate through BrainScript
• CNTK-EvalDLL allows evalution of models through C++ and C#. They are distributed through the binary download package or the available NUGET package.
• CNTK-library allows evaluation throught C++ and Python. The C# support is under development.
  • Support both CPU and GPU device.
  • Support evaluation of multiple requests in parallel by multi-threading.
  • Share model parameters among multiple threads if the same model is loaded. This will significantly reduce memory usage when running evaluation in a service environment.

The following table gives an overview about the evaluation alternatives and the supported model formats

model-evaluation	model-v1	model-v2
CNTK.EXE (BrainScript)	YES	NO
EvalDLL (C++, C#, ASP and Azure)	YES	NO
CNTK-libaray (Python, C++)	YES	YES

The following pages provide detailed information about model evaluation in different scenarios:

• Evaluating a model using CNTK.EXE
  
• EvalDLL evaluation on Windows
• EvalDLL evaluation on Linux
• EvalDLL evaluation in Azure
• EvalDLL evaluation using the NuGet-Package
  
• CNTK-library evaluation on Windows
• CNTK-library evaluation on Linux
• CNTK-library evaluation with Python