Repository for the course "Efficient Deep Learning" at IMT Atlantique

Course organisation / Syllabus

Here is a detailed schedule, session by session:

Introduction / Refresher on Deep Learning
1. General Intro - Why do we need to optimize deep learning ? Introduction of the MicroNet Challenge.
2. Course - Deep Learning and Transfer Learning.
3. Practical session - introduction to PyTorch, transfer learning.
4. Short project - exploring hyper parameters on a fixed architecture
5. Presentation of the Challenge Objectives and Rules (Long Project)
Quantification
1. Short evaluation on Deep Learning Essentials
2. Course - Quantifying Deep neural networks
3. Practical session - quantification on a small convolutional network
4. Long project 1 - Challenge
Pruning
1. Short evaluation on Quantification
2. Student's presentation of short project - exploring hyper parameters on a fixed architecture
3. Course - Pruning Deep neural networks
4. Practical session - pruning on a small convolutional network.
5. Long project 2 - Challenge
Data Augmentation, Regularization and Self-Supervised Learning
1. Short evaluation on Pruning
2. Course - Data Augmentation, Regularization and Self-supervised Learning
3. Practical session - Data Augmentation
4. Long Project 3 - Challenge
Factorization
1. Short evaluation on Data Augmentation
2. Student's presentation on current work on Challenge
3. Course - Factorizing Deep neural networks
4. Practical session - factorizing a small convolutional network
5. Long Project 4 - Challenge
Distillation
1. Long evaluation on Factorization and previous courses
2. Course - Distillation of knowledge and features between neural networks
3. Long Project 5 - Challenge
Embedded Software and Hardware for Deep Learning
1. Short evaluation on Distillation
2. Course - Embedded Software and Hardware for Deep Learning
3. Long Project 6 - Challenge
Final Session
1. Short evaluation on embedded software and hardware for Deep Learning
2. Long Project 7 - Challenge
3. Student's presentation - Challenge final results

What is expected for the Long Project

Short version : Exploration of the accuracy / complexity tradeoff

Long version : this course is mostly based on the long project, and you have a lot of freedom, which we expect you to use. The overarching goal is to explore the tradeoff between the performances of neural networks (= Accuracy on the test set) and complexity. Complexity can be either computational complexity (number of arithmetic operations), or memory complexity (memory size necessary to use the network).

We encourage students to get creative and test combinations of the various ideas that we present. Starting from the end of Session 1, you already have enough knowledge to explore the tradeoff between architecture, number of parameters, and accuracy. Then, we study new notions that open new avenues to explore this tradeoff : quantization, pruning, factorization, distillation. In session 7, you'll have a deeper insight on how to thing about specific software or hardware architecture in order to fully exploit all the optimizations that can be done.

Evaluation in this course

There are short written evaluations during the first 10 minutes of each session starting from session 2. Don't be late!

For the final session, we ask you to prepare a 15 minutes presentation, that will be followed by 5 Minutes of question.

What we expect for the presentations :

Explain your strategy to explore the complexity / accuracy tradeoff. We will judge whether you understood the goal, and whether the proposed strategy follows a rigourous approach.
The clarity of your exposition and quality of your support (slides)

Importantly : We will NOT judge you based on accuracy scores, as long as you have baselines (non optimized) that are not too far from the usual accuracy for the considered dataset.

General References

List of references IMT Atlantique and AI

Amazon Book - Dive into Deep learning

Tutorial presentation on Efficient Deep Learning from NeurIPS'19

Training Deep Networks

Here are some academic papers discussing learning rate strategies :

Cyclic learning rates
Demystifying Learning Rate Policies for High Accuracy Training of Deep Neural Networks
A Closer Look at Deep Learning Heuristics: Learning rate restarts, Warmup and Distillation