University project on the optimization of neural networks through hypergradient based algorithms.
Comparison between algorithms on a fully connected neural network (MINST dataset)
The original paper is from Atılım Güneş Baydin, Robert Cornish, David Martı́nez Rubio, Mark Schmidt, and Frank Wood (2018). Online learning rate adaptation with hypergradient descent.
Link to the original paper: link
Link to the original author's github: link github
- Tensorflow guide: (https://www.tensorflow.org/api_docs/python/tf/keras/optimizers/Optimizer#creating_a_custom_optimizer_2)
- Online example: (https://towardsdatascience.com/custom-optimizer-in-tensorflow-d5b41f75644a)
- Class inherited: (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/optimizer.py)
- Non slot variable example: (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/adam.py)
- VGG Net for CIFAR10:
- PyTorch (original): (http://torch.ch/blog/2015/07/30/cifar.html)
- TensorFlow: (https://github.com/geifmany/cifar-vgg/blob/master/cifar10vgg.py)