low rank deep neural networks

[wenwei202]

Issue summary

I am working on low rank deep neural networks, to speedup the testing for better deployability. Anyone is working on similar stuff?

Steps to reproduce

Code is in https://github.com/wenwei202/caffe/tree/sfm.
Related publication in ICCV 2017

[bachml]

Yes.
But I didn't find anything new in the sfm branch. Would you mind offering some documents about this branch? Thanks

[wenwei202]

@bachml We have a simple tutorial on usage of this code. We will update more details recently.

[bachml]

@wenwei202 thanks to your remarkable work.
But there's a problem comes up. There's no speedups was observed when a convolution layer with rank M = 1 (high layer in ResNet) was decomposed. Also I didn't find any experiment about speedups of ResNet by force regularization.
Did you meet this issue?

[wenwei202]

@bachml I did not measure speedup by ResNet. Decomposing to rank 1 should have some benefits. It is the issue of the implementation?

[bachml]

@wenwei202 More test in my baseline case(a 27 layers ResNet) shows that the issue is related to multi-threaded blas performance (Caffe with CPU).
It did has 1.20x speedup, which is not significant, with 8 threads openblas backend. When run with single-threaded openblas, it has 2.00x speedup. I guess it's because rank 1 convolution corresponds small matrix multiplication, which do not have much benefits from multi-threaded operation with im2col trick.

[wenwei202]

@bachml In the rank one case, the conv layer is decomposed to a conv layer with only one filter plus a linear combination layer which essentially is a conv layer with kernels of 1x1. Some code optimization may be required to fully exploit this kind of compactness.

[wenwei202]

In case you still have interest in this research topic, the details are covered in the paper which is just accepted by ICCV 2017.