Skip to content

Setup CNTK on Linux

Jump to bottom Edit New page
Alexey Orlov edited this page Feb 8, 2016 · 43 revisions

Linux Setup

64-bit OS

You need a 64-bit Linux installation to use CNTK.

C++ Compiler

Ensure your installation has a C++ compiler. Many distributions do not include it by default. Refer to your platform documentation on how to check for and obtain a C++ compiler.

Example: for Ubuntu, run the following command:

dpkg --list | grep compiler

if in the output you do not see something like g++-4.8 4.8.4-2ubuntu1~14.04 amd64 GNU C++ compiler then a C++ compiler is not installed. Install it with:

sudo apt-get install g++

Git

Install Git on your system as described here.

ACML

Install the AMD Core Math Library (ACML):

tar -xzvf ./acml-5-3-1-ifort-64bit.tgz
sudo ./install-acml-5-3-1-ifort-64bit.sh
  • Accept the default installation path
  • Add the following environment variable to your current session and your .bashrc profile:
export ACML_FMA=0
export LD_LIBRARY_PATH=/opt/acml5.3.1/ifort64/lib:$LD_LIBRARY_PATH

Open MPI

Install Open Message Passing Interface Library (Open MPI):

  • Get the installation sources:
wget https://www.open-mpi.org/software/ompi/v1.10/downloads/openmpi-1.10.1.tar.gz
  • Unpack, build and install Open MPI to $HOME/mpi:
tar -xzvf ./openmpi-1.10.1.tar.gz
cd openmpi-1.10.1
./configure --prefix=$HOME/mpi
make -j all
make install
  • Add the following environment variable to your current session and your .bashrc profile (Prepending the new path, ensures this version is used as opposed to a default version available through the OS):
export PATH=$HOME/mpi/bin:$PATH

GPU Specific Packages

This section outlines the packages you need to setup in order for CNTK to leverage GPUs.

If you do NOT intent to use any GPUs with CNTK and will rely on the CPU only, you may skip directly to the next general section. Otherwise proceed further.

Checking your GPU compatibility and getting the latest driver

You need a CUDA-compatible graphic card available to use CNTK GPU capabilities. You can check whether your card is CUDA-compatible here and here (for older cards).

If you have the required card, install the latest driver:

  • Select your card and download the driver pack from here (usually available as .run file)
  • If your X Windows manager is running, the driver installation will likely fail. Open a remote terminal session to your machine and stop the X Windows manager. Refer to your platform documentation for the exact commands.

Example: for Ubuntu, use the following command:

sudo stop lightdm
  • Install the driver as in the example below (note that the file name may be different for your system):
sudo chmod +x ./NVIDIA-Linux-x86_64-352.63.run
sudo ./NVIDIA-Linux-x86_64-352.63.run

We recommend accepting the default installation options

Note that the driver installation program may complain about nouveau kernel driver. Refer to your platform documentation for instructions for disabling it. For Ubuntu you may use this set of instructions

  • If stopped during the steps above, start X Windows manager. Refer to your platform documentation for the exact commands.

Example: for Ubuntu, use the following command:

sudo start lightdm

CUDA 7.0

Download and install the NVIDIA CUDA 7.0 Toolkit:

  • Find the .run file for your platform here and download it either with the browser or copy the link to the required file and use the ```wget`` command like in the example below (note that the file name may be different for your system):
wget http://developer.download.nvidia.com/compute/cuda/7_0/Prod/local_installers/cuda_7.0.28_linux.run
  • If your X Windows manager is running, the installation will likely fail. Open a remote terminal session to your machine and stop the X Windows manager. Refer to your platform documentation for the exact commands.

Example: for Ubuntu use the following command:

sudo stop lightdm
  • Install CUDA 7.0 Toolkit (note that the .run file name may be different for your system):
sudo chmod +x ./cuda_7.0.28_linux.run
sudo ./cuda_7.0.28_linux.run

IMPORTANT! When prompted by the installer:

Install NVIDIA Accelerated Graphics Driver for Linux-x86_64 346.46? ((y)es/(n)o/(q)uit):

select no if you have already installed the latest driver by performing the steps in the previous section. If you have not done it, select yes, but we strongly recommend updating to the latest driver after installing CUDA toolkit.

If you declined the driver installation from the CUDA 7.0 package as described in the previous section, you will get the following warning at the end of the installation:

***WARNING: Incomplete installation! This installation did not install the CUDA Driver. A driver of version at least 346.00 is required for CUDA 7.0 functionality to work.
To install the driver using this installer, run the following command, replacing <CudaInstaller> with the name of this run file:
    sudo <CudaInstaller>.run -silent -driver

Ignore it

  • If stopped during the steps above, start X Windows manager. Refer to your platform documentation for exact commands.

Example: for Ubuntu use the following command:

sudo start lightdm
  • Add the following environment variable to your current session and your .bashrc profile (if you modified the default paths during the CUDA installation, change the values below accordingly):
export PATH=/usr/local/cuda-7.0/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-7.0/lib64:$LD_LIBRARY_PATH

The next step is optional. You may skip further to the next section.

OPTIONAL. Verifying CUDA 7.0 installation

You may verify your CUDA installation by compiling the CUDA samples (we assume the default paths were used during the CUDA installation). Note that building all samples is a lengthy operation:

cd ~/NVIDIA_CUDA-7.0_Samples/
make

After the successful build invoke deviceQuery utility:

~/NVIDIA_CUDA-7.0_Samples/1_Utilities/deviceQuery/deviceQuery

If everything works well, you should get an output similar to the one below:

/home/alexey/NVIDIA_CUDA-7.0_Samples/1_Utilities/deviceQuery/deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Quadro 600"
  CUDA Driver Version / Runtime Version          7.5 / 7.0
  CUDA Capability Major/Minor version number:    2.1
  Total amount of global memory:                 1016 MBytes (1065734144 bytes)
  ( 2) Multiprocessors, ( 48) CUDA Cores/MP:     96 CUDA Cores
  GPU Max Clock rate:                            1280 MHz (1.28 GHz)
  Memory Clock rate:                             800 Mhz
  Memory Bus Width:                              128-bit
  L2 Cache Size:                                 131072 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65535), 3D=(2048, 2048, 2048)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 32768
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  1536
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (65535, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 2 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.5, CUDA Runtime Version = 7.0, NumDevs = 1, Device0 = Quadro 600
Result = PASS

GPU Deployment kit

Get and install the latest version of NVIDIA GPU Deployment kit:

  • Find the .run file for your platform here. Select the latest version. Download the file either with the browser or copy the link to the required file and use ```wget`` command as shown in the example below (note that the file name may be different for your system):
wget http://developer.download.nvidia.com/compute/cuda/7.5/Prod/gdk/gdk_linux_amd64_352_55_release.run
  • If your X Windows manager is running, the installation will likely fail. Open a remote terminal session to your machine and stop the X Windows manager. Refer to your platform documentation for the exact commands.

Example: for Ubuntu use the following command:

sudo stop lightdm
  • Install the GPU Deployment kit (note that the .run file name may be different for your system):
sudo chmod +x ./gdk_linux_amd64_352_55_release.run
sudo ./gdk_linux_amd64_352_55_release.run

We recommend accepting the default values during installation.

  • If stopped during the steps above, start X Windows manager. Refer to your platform documentation for exact commands.

Example: for Ubuntu use the following command:

sudo start lightdm

CUB

Get and install NVIDIA CUB.

IMPORTANT! Install NVIDIA CUB using the exact version and target path as specified below. This is necessary because it is expected by the CNTK build configuration program.

Use the following commands:

wget https://github.com/NVlabs/cub/archive/1.4.1.zip
unzip ./1.4.1.zip
sudo cp -r cub-1.4.1 /usr/local

cuDNN

If you would like to use CNTK together with the NVIDIA CUDA Deep Neural Network library (cuDNN), you'll need to get and install the cuDNN library during this step.

IMPORTANT! Install cuDNN using the exact version and target path as specified below. This is necessary because it is expected by the CNTK build configuration program.

  • Use the following commands:
wget http://developer.download.nvidia.com/compute/redist/cudnn/v4/cudnn-7.0-linux-x64-v4.0-rc.tgz
tar -xzvf ./cudnn-7.0-linux-x64-v4.0-rc.tgz
sudo mkdir /usr/local/cudnn-4.0
sudo cp -r cuda /usr/local/cudnn-4.0
  • Add the following environment variable to your current session and your .bashrc profile:
export LD_LIBRARY_PATH=/usr/local/cudnn-4.0/cuda/lib64:$LD_LIBRARY_PATH

This completes GPU Specific section

OPTIONAL. OpenCV

If you want to build the CNTK Image Reader, you need to install Open Source Computer Vision (OpenCV).

Some aspects to consider:

  • OpenCV requires at least 5.5 GB of free disk space
  • Building OpenCV is a lengthy process and can take a couple of hours
  • If you plan to use both CUDA and OpenCV, install CUDA first

OpenCV can have many interfaces and options. In this section we cover only parts necessary to build CNTK. Read more about installing OpenCV here and more generally here.

IMPORTANT! Install OpenCV using the exact version and target path as specified below. This is necessary because it is expected by the CNTK build configuration program.

  • Install OpenCV prerequisites. See OpenCV installation instructions for the full package list and explanations. See your platform documentation on how to install the packages.

Example: for Ubuntu use the following command:

sudo apt-get install cmake libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
  • Get OpenCV and install it. Use the following commands:
wget https://github.com/Itseez/opencv/archive/3.0.0.zip
unzip 3.0.0.zip
cd opencv-3.0.0
mkdir release
cd release
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local/opencv-3.0.0 ..
make all
sudo make install

Note that in the instructions above we suggest using "plain" make rather than make -j. We found that using make -j with OpenCV results in unstable system behavior and may result in a build failure and system crash.

Getting CNTK Source code

Before proceeding further, please note, that if you plan on making modifications to the CNTK code you should read the information on Developing and Testing in this Wiki.

Use Git to clone the CNTK Repository and access the source code:

git clone https://github.com/Microsoft/CNTK

IMPORTANT! The command above does not provide you with the access to 1-bit Stochastic Gradient Descent (1bit-SGD) component. If you want to build CNTK with 1bit-SGD functionality enabled, please read the instructions on this page and then proceed with installation.

Building CNTK

To build CNTK use the following commands (we assume that the CNTK repository was cloned to ~/Repos/cntk):

cd ~/Repos/cntk
mkdir build/release -p
cd build/release
../../configure

Ensure that the configure output corresponds to the packages you installed in the previous sections. I.e. ensure that configure finds CUDA if installed, etc.

make -j all

This should produce a release build of CNTK. In case you would like to get a debug build use the following parameter when invoking configure:

../../configure --with-buildtype=debug

Quick test of CNTK build functionality

To ensure that CNTK is working properly in your system, you can quickly run Simple2d example from CNTK Examples set. This example trains a simple network and can be directed to use either CPU or GPU, which helps quickly ensure that CNTK is functioning properly.

Below we assume that the CNTK repository is cloned to ~/Repos/cntk and build/release was used as a sub-directory for the build.

  • Provide the path to the CNTK binaries and switch to the Simple2d directory:
export PATH=$HOME/Repos/cntk/build/release/bin:$PATH
cd ~/Repos/cntk/Examples/Other/Simple2d/Data

First try the example:

cntk configFile=../Config/Simple.cntk

If the sample runs, i.e. if there are no error messages, you will get output related first to reading the configuration, followed by the output of the actual network training. Running the command a second time will report that the model has already been trained. To perform the training again, you must first remove the existing models by using the following command:

rm ../Output/Models/*

Trying CNTK with CPU

As seen previously, the Simple2d example produces an extensive output. Among this output is information regarding what processor was used for the Network Builder. If you perform the following set of commands:

cntk configFile=../Config/Simple.cntk &> out
cat out | grep Builder

you should get the following output

    SimpleNetworkBuilder = [
    SimpleNetworkBuilder = [
SimpleNetworkBuilder Using CPU

The last string is the informative one for our purposes. It tells us that CNTK was indeed using CPU.

Trying CNTK with GPU

If you build CNTK for GPU usage, try using GPU by executing the following commands:

cntk configFile=../Config/Simple.cntk deviceId=auto &> out
cat out | grep Builder

Expected output in this case is:

    SimpleNetworkBuilder = [
    SimpleNetworkBuilder = [
SimpleNetworkBuilder Using GPU 0

Note that GPU id may be different. deviceId parameter defines what processor to use for Network Builder:

  • deviceId=-1 means use CPU. Default value
  • deviceId=X where X is an integer >=0 means use GPU X, i.e. deviceId=0 means GPU 0, etc.
  • deviceId=auto means use GPU, select GPU automatically

Next steps

Add a custom footer

Getting started

Configuring CNTK

Describing Networks with BrainScript

Data readers

Evaluating CNTK Models

Advanced topics

Licenses

Clone this wiki locally