Compressing neural networks with the hashing trick

32nd International Conference on Machine Learning, ICML 2015

Volumn 3, Issue , 2015, Pages 2275-2284

  Chen, Wenlin ^a   Wilson, James T ^a   Tyree, Stephen ^a,b   Weinberger, Kilian Q ^a   Chen, Yixin ^a  

^a WASHINGTON UNIVERSITY IN ST LOUIS   (United States)

^b NVIDIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; BENCHMARKING; DIGITAL STORAGE; HASH FUNCTIONS; LEARNING SYSTEMS; MEMORY ARCHITECTURE; MOBILE DEVICES;

BENCHMARK DATA; CONNECTION WEIGHTS; GENERALIZATION PERFORMANCE; INHERENT REDUNDANCY; MEMORY OVERHEADS; SHARING ARCHITECTURES; SINGLE PARAMETER; STORAGE REQUIREMENTS;

NETWORK ARCHITECTURE;

EID: 84969930652     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (53)

1. Agarwal, Alekh, Chapelle, Olivier, Dudík, Miroslav, and Langford, John. A reliable effective terascale linear learning system. The Journal of Machine Learning Research, 15(1):1111-1133, 2014.
2. Ba, Jimmy and Caruana, Rich. Do deep nets really need to be deep? In NIPS, pp. 2654-2662, 2014.
3. Bishop, Christopher M. Neural Networks for Pattern Recognition. Oxford University Press, Inc., 1995.
4. Boureau, Ylan, Cun, Yann L, et al. Sparse feature learning for deep belief networks. In NIPS, pp. 1185-1192,2008.
5. Bucilu, Cristian, Caruana, Rich, and Niculescu-Mizil, Alexandru. Model compression. In KDD, 2006.
6. Chen, Minmin, Weinberger, Kilian Q., Sha, Fei, and Bengio, Yoshua. Marginalized denoising auto-encoders for nonlinear representations. In ICML, pp. 1476-1484, 2014a.
7. Chen, Wenlin, Chen, Yixin, and Weinberger, Kilian Q. Fast flux discriminant for large-scale sparse nonlinear classification. In KDD, pp. 621-630, 2014b.
8. Chun, Byung-Gon and Maniatis, Petros. Augmented smartphone applications through clone cloud execution. In HotOS, 2009.
9. Cireşan, Dan C, Meier, Ueli, Masci, Jonathan, Gambardella, Luca M, and Schmidhuber, Jürgen. High-performance neural networks for visual object classification. arXiv preprint arXiv:l 102.0183, 2011.
10. Coates, Adam, Ng, Andrew Y, and Lee, Honglak. An analysis of single-layer networks in unsupervised feature learning. In AISTATS, 2011.
11. Coates, Adam, Huval, Brody, Wang, Tao, Wu, David, Catanzaro, Bryan, and Andrew, Ng. Deep learning with cots hpc systems. In ICML, pp. 1337-1345, 2013.
12. Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Clément. Torch7: A matlab-like environment for machine learning. In Big Learn, NIPS Workshop, 2011.
13. Courbariaux, M., Bengio, Y., and David, J.-P. Low precision storage for deep learning. arXiv preprint arXiv: 1412.7024, 2014.
14. Denil, Misha, Shakibi, Babak, Dinh, Laurent, de Freitas, Nando, et al. Predicting parameters in deep learning. In NIPS, 2013.
15. Denton, Emily, Zaremba, Wojciech, Bruna, Joan, Le Cun, Yann, and Fergus, Rob. Exploiting linear structure within convolutional networks for efficient evaluation. arXiv preprint arXiv:1404.0736, 2014.
16. Donahue, Jeff, Jia, Yangqing, Vinyals, Oriol, Hoffman, Judy, Zhang, Ning, Tzeng, Eric, and Darrell, Trevor. Decaf: A deep convolutional activation feature for generic visual recognition. arXiv preprint arXiv:1310.1531, 2013.
17. Ganchev, Kuzman and Dredze, Mark. Small statistical models by random feature mixing. In Workshop on Mobile NLP at ACL, 2008.
18. Gardner, Jacob, Kusner, Matt, Weinberger, Kilian, Cunningham, John, et al. Bayesian optimization with inequality constraints. In ICML, 2014.
19. Girshick, Ross, Donahue, Jeff, Darrell, Trevor, and Malik, Jitendra. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
20. Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua. Deep sparse rectifier networks. In AISTATS, 2011a.
21. Glorot, Xavier, Bordes, Antoine, and Bengio, Yoshua. Domain adaptation for large-scale sentiment classification: A deep learning approach. In ICML, pp. 513-520, 2011b.
22. Graves, Alex, Mohamed, A-R, and Hinton, Geoffrey. Speech recognition with deep recurrent neural networks. In ICASSP, 2013.
23. Gupta, Suyog, Agrawal, Ankur, Gopalakrishnan, Kailash, and Narayanan, Pritish. Deep learning with limited numerical precision. arXiv preprint arXiv: 1502.02551, 2015.
24. Hinton, Geoffrey, Deng, Li, Yu, Dong, Dahl, George E, Mohamed, Abdelrahman, Jaitly, Navdeep, Senior, Andrew, Vanhoucke, Vincent, Nguyen, Patrick, Sainath, Tara N, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Processing Magazine, IEEE, 29 (6): 82-97, 2012.
25. Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff. Distilling the knowledge in a neural network. NIPS workshop, 2014.
26. Karpathy, Andrej and Fei-Fei, Li. Deep visual-semantic alignments for generating image descriptions. arXiv preprint arXiv: 1412.2306, 2014.
27. Kosner, A.W. Client vs. server architecture: Why google voice search is also much faster than siri @ONLINE, October 2012. URL http://tinyurl.com/c2d2otr.
28. Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
29. Larochelle, Hugo, Erhan, Dumitru, Courville, Aaron C, Bergstra, James, and Bengio, Yoshua. An empirical evaluation of deep architectures on problems with many factors of variation. In ICML, pp. 473-480, 2007.
30. Le, Quoc V. Building high-level features using large scale unsupervised learning. In ICASSP, pp. 8595-8598. IEEE, 2013.
31. Le Cun, Yann, Denker, John S, Solla, Sara A, Howard, Richard E, and Jackel, Lawrence D. Optimal brain damage. In NIPS, 1989.
32. Le Cun, Yann A, Bottou, Léon, Orr, Genevieve B, and Mtiller, Klaus-Robert. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-48. Springer, 2012.
33. Lee, Kyong Ho and Verma, Naveen. A low-power processor with configurable embedded machine-learning accelerators for high-order and adaptive analysis of medical-sensor signals. Solid-State Circuits, IEEE Journal of, 48 (7): 1625-1637, 2013.
34. Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg S, and Dean, Jeff. Distributed representations of words and phrases and their compositionality. In NIPS, 2013.
35. Mohamed, Abdelrahman, Sainath, Tara N, Dahl, George, Ramabhadran, Bhuvana, Hinton, Geoffrey E, and Picheny, Michael A. Deep belief networks using discriminative features for phone recognition. In ICASSP, 2011.
36. Montemerlo, Michael, Becker, Jan, Bhat, Suhrid, Dahlkamp, Hendrik, Dolgov, Dmitri, Ettinger, Scott, Haehnel, Dirk, Hilden, Tim, Hoffmann, Gabe, Huhnke, Burkhard, et al. Junior: The Stanford entry in the urban challenge. Journal of field Robotics, 25(9): 569-597, 2008.
37. Nair, Vinod and Hinton, Geoffrey E. Rectified linear units improve restricted boltzmann machines. In ICML, pp. 807-814, 2010.
38. Nowlan, Steven J and Hinton, Geoffrey E. Simplifying neural networks by soft weight-sharing. Neural computation, 4(4):473-493, 1992.
39. Pineda, Fernando J. Generalization of back-propagation to recurrent neural networks. Physical review letters, 59 (19):2229, 1987.
40. Razavian, Ali Sharif, Azizpour, Hossein, Sullivan, Josephine, and Carlsson, Stefan. Cnn features off-the-shelf: an astounding baseline for recognition. In CVPR Workshop, 2014.
41. Rippel, Oren, Gelbart, Michael A, and Adams, Ryan P. Learning ordered representations with nested dropout. arXiv preprint arXiv:1402.0915, 2014.
42. Salton, Gerard and Buckley, Christopher. Term-weighting approaches in automatic text retrieval. Information processing & management, 24(5):513-523, 1988.
43. Schuster, Mike. Speech recognition for mobile devices at google. In PR1CAI 2010: Trends in Artificial Intelligence, pp. 8-10. Springer, 2010.
44. Sermanet, Pierre, Eigen, David, Zhang, Xiang, Mathieu, Michael, Fergus, Rob, and Le Cun, Yann. Overfeat: Integrated recognition, localization and detection using cpn-volutional networks. arXiv preprint arXiv: 1312.6229, 2013.
45. Shi, Qinfeng, Petterson, James, Dror, Gideon, Langford, John, Smola, Alex, and Vishwanathan, S.V.N. Hash kernels for structured data. Journal of Machine Learning Research, 10:2615-2637, December 2009.
46. Simard, Patrice Y, Steinkraus, Dave, and Piatt, John C. Best practices for convolutional neural networks applied to visual document analysis. In ICDAR, volume 2, pp. 958-958. IEEE Computer Society, 2003.
47. Snoek, Jasper, Larochelle, Hugo, and Adams, Ryan P. Practical Bayesian optimization of machine learning algorithms. In NIPS, 2012.
48. Srivastava, Nitish, Hinton, Geoffrey, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1): 1929-1958, 2014.
49. Vinyals, Oriol, Toshev, Alexander, Bengio, Samy, and Erhan, Dumitru. Show and tell: A neural image caption generator. arXiv preprint arXiv:1411.4555, 2014.
50. Weinberger, Kilian, Dasgupta, Anirban, Langford, John, Smola, Alex, and Attenberg, Josh. Feature hashing for large scale multitask learning. In ICML, 2009.
51. Xu, Zhixiang, Gardner, Jacob R, Tyree, Stephen, and Weinberger, Kilian Q. Compressed support vector machines. arXiv preprint arXiv:1501.06478, 2015.
52. Zeiler, Matthew D and Fergus, Rob. Stochastic pooling for regularization of deep convolutional neural networks. arXiv preprint arXiv:1301.3557, 2013.
53. Zeiler, Matthew D and Fergus, Rob. Visualizing and understanding convolutional networks. In ECCV, 2014.