Skip to content

Commit

Permalink
Add some federated learning papers
Browse files Browse the repository at this point in the history
  • Loading branch information
Yuduo Wu committed Jan 31, 2019
1 parent 5200596 commit 61f72dc
Showing 1 changed file with 19 additions and 17 deletions.
36 changes: 19 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -636,11 +636,11 @@ accuracy-guaranteed compression.

## Survey / Reviews

##### [A Survey on Compiler Autotuning using Machine Learning](https://arxiv.org/abs/1801.04405)
#### [A Survey on Compiler Autotuning using Machine Learning](https://arxiv.org/abs/1801.04405)

> This survey summarizes and classifies the recent advances in using machine learning for the compiler optimization field, particularly on the two major problems of (1) selecting the best optimizations and (2) the phase-ordering of optimizations. The survey highlights the approaches taken so far, the obtained results, the fine-grain classification among different approaches and finally, the influential papers of the field.
##### [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://arxiv.org/abs/1710.09282)
#### [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://arxiv.org/abs/1710.09282)

> In this paper, we survey the recent advanced techniques for compacting and accelerating CNNs model developed. These techniques are roughly categorized into four schemes: parameter pruning and sharing, low-rank factorization, transfered/compact convolutional filters and knowledge distillation. Methods of parameter pruning and sharing will be described at the beginning, after that the other techniques will be introduced. For each scheme, we provide insightful analysis regarding the performance, related applications, advantages and drawbacks etc. Then we will go through a few very recent additional successful methods, for example, dynamic networks and stochastic depths networks. After that, we survey the evaluation matrix, main datasets used for evaluating the model performance and recent bench-marking efforts. Finally we conclude this paper, discuss remaining challenges and possible directions in this topic.
Expand All @@ -661,45 +661,45 @@ Major milestones for "[ImageNet](http://www.image-net.org/) in X nanoseconds" :r
| Nov 2018 | 2176 NVIDIA V100 GPUs | 3.7 mins | 75.03% | 69632 | https://arxiv.org/abs/1811.06992 |
| Nov 2018 | 1024 Google TPUs (v3) | 2.2 mins | 76.3% | 32768 | https://arxiv.org/abs/1811.06992 |

##### [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://arxiv.org/abs/1706.02677)
#### [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://arxiv.org/abs/1706.02677)

- Learning rate linear scaling rule
- Learning rate warmup (constant, gradual)
- Communication: [recursive halving and doubling algorithm](https://pdfs.semanticscholar.org/8d44/e92b3597d9e3f5245e152c9e0ce55b3e68a4.pdf)

##### [PowerAI DDL](https://arxiv.org/abs/1708.02188)
#### [PowerAI DDL](https://arxiv.org/abs/1708.02188)

- Topology-aware communication

##### [ImageNet Training in Minutes](https://arxiv.org/abs/1709.05011)
#### [ImageNet Training in Minutes](https://arxiv.org/abs/1709.05011)

- Layer-wise Adaptive Rate Scaling (LARS)

##### [Don't Decay the Learning Rate, Increase the Batch Size](https://arxiv.org/abs/1711.00489)
#### [Don't Decay the Learning Rate, Increase the Batch Size](https://arxiv.org/abs/1711.00489)

- Decaying the learning rate is simulated annealing
- Instead of decaying the learning rate, increase the
batch size during training

##### [Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes](https://arxiv.org/abs/1711.04325)
#### [Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes](https://arxiv.org/abs/1711.04325)

- RMSprop Warm-up
- Slow-start learning rate schedule
- Batch normalization without moving averages

##### [Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes](https://arxiv.org/abs/1807.11205)
#### [Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes](https://arxiv.org/abs/1807.11205)

- Mixed precision
- Layer-wise Adaptive Rate Scaling (LARS)
- Improvements on model architecture
- Communication: tensor fusion, hierarchical all-reduce, hybrid all-reduce

##### [ImageNet/ResNet-50 Training in 224 Seconds](https://arxiv.org/abs/1811.05233)
#### [ImageNet/ResNet-50 Training in 224 Seconds](https://arxiv.org/abs/1811.05233)

- Batch size control to reduce accuracy degradation with mini-batch size exceeding 32K
- Communication: 2D-torus all-reduce

##### [Image Classification at Supercomputer Scale](https://arxiv.org/abs/1811.06992)
#### [Image Classification at Supercomputer Scale](https://arxiv.org/abs/1811.06992)

- Mixed precision
- Layer-wise Adaptive Rate Scaling (LARS)
Expand All @@ -709,16 +709,18 @@ batch size during training

## Federated Learning

##### [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://arxiv.org/abs/1602.05629)
#### [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://arxiv.org/abs/1602.05629)

##### [Practical Secure Aggregation for Privacy-Preserving Machine Learning](https://eprint.iacr.org/2017/281.pdf)
#### [Practical Secure Aggregation for Privacy-Preserving Machine Learning](https://eprint.iacr.org/2017/281.pdf)

##### [Federated Multi-Task Learning](https://arxiv.org/abs/1705.10467)
#### [Federated Multi-Task Learning](https://arxiv.org/abs/1705.10467)

##### [Protection Against Reconstruction and Its Applications in Private Federated Learning](https://arxiv.org/abs/1812.00984)
#### [Protection Against Reconstruction and Its Applications in Private Federated Learning](https://arxiv.org/abs/1812.00984)

##### [Differentially Private Distributed Learning for Language Modeling Tasks](https://arxiv.org/abs/1712.07473)
#### [Differentially Private Distributed Learning for Language Modeling Tasks](https://arxiv.org/abs/1712.07473)

##### [Federated Learning for Mobile Keyboard Prediction](https://arxiv.org/abs/1811.03604)
#### [Differentially Private Federated Learning: A Client Level Perspective](https://arxiv.org/abs/1712.07557)

##### [A Generic Framework for Privacy Preserving Deep Learning](https://arxiv.org/abs/1811.04017)
#### [Federated Learning for Mobile Keyboard Prediction](https://arxiv.org/abs/1811.03604)

#### [A Generic Framework for Privacy Preserving Deep Learning](https://arxiv.org/abs/1811.04017)

0 comments on commit 61f72dc

Please sign in to comment.