Binarized neural networks

Advances in Neural Information Processing Systems

Volumn , Issue , 2016, Pages 4114-4122

  Hubara, Itay ^a   Courbariaux, Matthieu ^b   Soudry, Daniel ^c   El Yaniv, Ran ^a   Bengio, Yoshua ^b,d  

^a TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

^b UNIVERSITÉ DE MONTRÉAL   (Canada)

^c Och Spine at New York Presbyterian Hospitals   (United States)

^d CIFAR   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CHEMICAL ACTIVATION; GRAPHICS PROCESSING UNIT;

ARITHMETIC OPERATIONS; BINARY MATRIX; CLASSIFICATION ACCURACY; MEMORY SIZE; POWER EFFICIENCY; RUNTIMES; STATE OF THE ART;

BINS;

EID: 85013626529     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (34)

1. Baldassi, C., Ingrosso, A., Lucibello, C., Saglietti, L., and Zecchina, R. Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses. Physical Review Letters, 115(12): 1-5, 2015.
2. Bastien, F., Lamblin, P., Pascanu, R., et al. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.
3. Beauchamp, M. J., Hauck, S., Underwood, K. D., and Hemmert, K. S. Embedded floating-point units in FPGAs. In Proceedings of the 2006 ACM/SIGDA 14th international symposium on Field programmable gate arrays, pp. 12-20. ACM, 2006.
4. Bengio, Y. Estimating or propagating gradients through stochastic neurons. Technical Report arXiv: 1305.2982, Universite de Montreal, 2013.
5. Bergstra, J., Breuleux, O., Bastien, F., et al. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June 2010. Oral Presentation.
6. Chen, T., Du, Z., Sun, N., et al. Diannao: A small-footprint high-throughput accelerator for ubiquitous machine-learning. In Proceedings of the 19th international conference on Architectural support for programming languages and operating systems, pp. 269-284. ACM, 2014.
7. Cheng, Z., Soudry, D., Mao, Z., and Lan, Z. Training binary multilayer neural networks for image classification using expectation backpropgation. arXiv preprint arXiv: 1503.03562, 2015.
8. Coates, A., Huval, B., Wang, T., et al. Deep learning with COTS HPC systems. In Proceedings of the 30th international conference on machine learning, pp. 1337-1345, 2013.
9. Collobert, R., Kavukcuoglu, K., and Farabet, C. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, 2011.
10. Courbariaux, M., Bengio, Y., and David, J.-P. Training deep neural networks with low precision multiplications. ArXiv e-prints, abs/1412.7024, December 2014.
11. Courbariaux, M., Bengio, Y., and David, J.-P. Binaryconnect: Training deep neural networks with binary weights during propagations. ArXiv e-prints, abs/1511.00363, November 2015.
12. Dieleman, S., Schlüter, J., Raffel, C., et al. Lasagne: First release, August 2015.
13. Esser, S. K., Appuswamy, R., Merolla, P., Arthur, J. V., and Modha, D. S. Backpropagation for energy-efficient neuromorphic computing. In Advances in Neural Information Processing Systems, pp. 1117-1125, 2015.
14. Glorot, X. and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In AISTATS'2010, 2010.
15. Gong, Y., Liu, L., Yang, M., and Bourdev, L. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv: 1412.6115, 2014.
16. Goodfellow, I. J., Warde-Farley, D., Lamblin, P., et al. Pylearn2: a machine learning research library. arXiv preprint arXiv: 1308.4214, 2013.
17. Govindu, G., Zhuo, L., Choi, S., and Prasanna, V. Analysis of high-performance floating-point arithmetic on FPGAs. In Parallel and Distributed Processing Symposium, 2004. Proceedings. 18th International, pp. 149. IEEE, 2004.
18. Graves, A. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pp. 2348-2356, 2011.
19. Han, S., Mao, H., and Dally, W. J. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 1510.00149, 2015a.
20. Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pp. 1135-1143, 2015b.
21. Hinton, G. Neural networks for machine learning. Coursera, video lectures, 2012.
22. Horowitz, M. Computing's Energy Problem (and what we can do about it). IEEE Interational Solid State Circuits Conference, pp. 10-14, 2014.
23. Hubara, I., Courbariaux, M., Soudry, D., El-Yaniv, R., and Bengio, Y. Quantized neural networks: Training neural networks with low precision weights and activations. arXiv preprint arXiv: 1609.07061, 2016.
24. Hwang, K. and Sung, W. Fixed-point feedforward deep neural network design using weights+ 1, 0, and- 1. In Signal Processing Systems (SiPS), 2014 IEEE Workshop on, pp. 1-6. IEEE, 2014.
25. Ioffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. 2015.
26. Kim, M. and Smaragdis, P. Bitwise Neural Networks. ArXiv e-prints, January 2016.
27. Kingma, D. and Ba, J. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
28. LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. Nature, 521(7553): 436-444, 2015.
29. Lee, C.-Y., Gallagher, P. W., and Tu, Z. Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. arXiv preprint arXiv: 1509.08985, 2015.
30. Lin, Z., Courbariaux, M., Memisevic, R., and Bengio, Y. Neural networks with few multiplications. ArXiv e-prints, abs/1510.03009, October 2015.
31. Soudry, D., Hubara, I., and Meir, R. Expectation backpropagation: Parameter-free training of multilayer neural networks with continuous or discrete weights. In NIPS'2014, 2014.
32. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15: 1929-1958, 2014.
33. Szegedy, C., Liu, W., Jia, Y., et al. Going deeper with convolutions. Technical report, arXiv: 1409.4842, 2014.
34. Wan, L., Zeiler, M., Zhang, S., LeCun, Y., and Fergus, R. Regularization of neural networks using dropconnect. In ICML'2013, 2013.