Loss-aware binarization of deep networks

5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings

Volumn , Issue , 2017, Pages

  Hou, Lu ^a   Yao, Quanming ^a   Kwok, James T ^a  

^a HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

[No Author keywords available]

Indexed keywords

APPROXIMATION ALGORITHMS; RECURRENT NEURAL NETWORKS;

BINARIZATION ALGORITHM; CLOSED FORM SOLUTIONS; FEEDFORWARD AND RECURRENT NETWORKS; MATRIX APPROXIMATION; NETWORK WEIGHTS; NEURAL NETWORK MODEL; NEWTON ALGORITHM; SECOND MOMENTS;

DEEP NEURAL NETWORKS;

EID: 85050926955     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. M. Courbariaux, Y. Bengio, and J.P. David. BinaryConnect: Training deep neural networks with binary weights during propagations. In NIPS, pp. 3105-3113, 2015.
2. Y. Dauphin, H. de Vries, and Y. Bengio. Equilibrated adaptive learning rates for non-convex optimization. In NIPS, pp. 1504-1512, 2015a.
3. Y. Dauphin, H. de Vries, J. Chung, and Y. Bengio. RMSprop and equilibrated adaptive learning rates for non-convex optimization. Technical Report arXiv:1502.04390, 2015b.
4. J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, 2011.
5. X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTAT, pp. 249-256, 2010.
6. Y. Gong, L. Liu, M. Yang, and L. Bourdev. Compressing deep convolutional networks using vector quantization. Technical Report arXiv:1412.6115, 2014.
7. I.J. Goodfellow, D. Warde-Farley, P. Lamblin, V. Dumoulin, M. Mirza, R. Pascanu, J. Bergstra, F. Bastien, and Y. Bengio. Pylearn2: a machine learning research library. arXiv preprint arXiv:1308.4214, 2013.
8. S. Han, H. Mao, and W.J. Dally. Deep compression: Compressing deep neural network with pruning, trained quantization and Huffman coding. In ICLR, 2016.
9. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, pp. 1735-1780, 1997.
10. I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio. Binarized neural networks. In NIPS, pp. 4107-4115, 2016.
11. A. Karpathy, J. Johnson, and F.-F. Li. Visualizing and understanding recurrent networks. In ICLR, 2016.
12. Y.-D. Kim, E. Park, S. Yoo, T. Choi, L. Yang, and D. Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. In ICLR, 2016.
13. D. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
14. Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature, 521(7553):436-444, 2015.
15. J.D. Lee, Y. Sun, and M.A. Saunders. Proximal Newton-type methods for minimizing composite functions. SIAM Journal on Optimization, 24(3):1420-1443, 2014.
16. F. Li and B. Liu. Ternary weight networks. Technical Report arXiv:1605.04711, 2016.
17. Z. Lin, M. Courbariaux, R. Memisevic, and Y. Bengio. Neural networks with few multiplications. In ICLR, 2016.
18. J. Martens and I. Sutskever. Training deep and recurrent networks with Hessian-free optimization. In Neural Networks: Tricks of the trade, pp. 479-535. Springer, 2012.
19. A. Novikov, D. Podoprikhin, A. Osokin, and D.P. Vetrov. Tensorizing neural networks. In NIPS, pp. 442-450, 2015.
20. R. Pascanu and Y. Bengio. Revisiting natural gradient for deep networks. In ICLR, 2014.
21. R. Pascanu, T. Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In ICLR, pp. 1310-1318, 2013.
22. A. Rakotomamonjy, R. Flamary, and G. Gasso. DC proximal Newton for nonconvex optimization problems. IEEE Transactions on Neural Networks and Learning Systems, 27(3):636-647, 2016.
23. M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi. XNOR-Net: ImageNet classification using binary convolutional neural networks. In ECCV, 2016.
24. Theano Development Team. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints, abs/1605.02688, May 2016. URL http://arxiv.org/abs/1605.02688.
25. T. Tieleman and G. Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude, 2012.
26. A.L. Yuille and A. Rangarajan. The concave-convex procedure (CCCP). NIPS, 2:1033-1040, 2002.
27. M.D. Zeiler. ADADELTA: An adaptive learning rate method. Technical Report arXiv:1212.5701, 2012.
28. S. Zhou, Z. Ni, X. Zhou, H. Wen, Y. Wu, and Y. Zou. DoReFa-Net: Training low bitwidth convolutional neural networks with low bitwidth gradients. Technical Report arXiv:1606.06160, 2016.