FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Volumn 2016-December, Issue , 2016, Pages 2592-2600

  Iandola, Forrest N ^a   Moskewicz, Matthew W ^a   Ashraf, Khalid ^a   Keutzer, Kurt ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; NEURAL NETWORKS; PROGRAM PROCESSORS;

BEST PRACTICES; COMMUNICATION ALGORITHMS; COMMUNICATION OVERHEADS; DEEP NEURAL NETWORKS; HIGH BANDWIDTH; HYPERPARAMETERS; LINEAR ACCELERATIONS; REDUCTION TREES;

PATTERN RECOGNITION;

EID: 84986325668     PISSN: 10636919     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2016.284     Document Type: Conference Paper

References (34)

1. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Víegas, O. Vinyals, P. Warden, M.Wattenberg, M.Wicke, Y. Yu, and X. Zheng. Tensor-Flow: Large-scale machine learning on heterogeneous systems. Google Technical Report, 2015.
2. V. Anantharaj, F. Foertter, W. Joubert, and J. Wells. Approaching exascale: application requirements for olcf leadership computing. https://www.olcf.ornl. gov/wp-content/uploads/2013/01/OLCF Requirements-TM-2013-Final1.pdf, 2013. p43-45.
3. H. Azizpour, A. S. Razavian, J. Sullivan, A. Maki, and S. Carlsson. From generic to specific deep representations for visual recognition. In CVPR Deep Vision Workshop, 2015.
4. T. M. Breuel. The effects of hyperparameters on SGD training of neural networks. arXiv:1508.02788, 2015.
5. S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer. cuDNN: efficient primitives for deep learning. arXiv:1410.0759, 2014.
6. T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman. Project Adam: building an efficient and scalable deep learning training system. In OSDI, 2014.
7. J. Dean. Keynote: Large scale deep learning. In ACM Conference on Information and Knowledge Management (CIKM), 2014.
8. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, Q. V. Le, and A. Y. Ng. Large scale distributed deep networks. In NIPS, 2012.
9. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
10. W. Ding, R. Wang, F. Mao, and G. W. Taylor. Theanobased large-scale visual recognition with multiple gpus. arXiv:1412.2302, 2014.
11. C. Farabet. Real-time, content-driven representations at twitter. In GPU Technology Conference, 2015.
12. X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. JMLR, 2010.
13. S. Guadarrama. BVLC googlenet. https: //github.com/BVLC/caffe/tree/master/models/bvlc-googlenet, 2015.
14. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. JMLR, 2015.
15. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. arXiv:1408.5093, 2014.
16. A. Krizhevsky. One weird trick for parallelizing convolutional neural networks. arXiv:1404.5997, 2014.
17. A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In NIPS, 2012.
18. A. Lavin. maxDNN: an efficient convolution kernel for deep learning with maxwell gpus. arXiv:1501.06633, 2015.
19. S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. arXiv:1504.00702, 2015.
20. M. Lin, Q. Chen, and S. Yan. Network in network. arXiv:1312.4400, 2013.
21. L. Mai, C. Hong, and P. Costa. Optimizing network performance in distributed machine learning. In HotCloud, 2015.
22. C. Noel, J. Shi, and A. Feng. Large scale distributed deep learning on hadoop clusters. 2015. http://yahoohadoop.tumblr.com/post/129872361846.
23. Oak Ridge Leadership Computing Facility (OLCF). Introducing titan: Advancing the era of accelerated computing. http://www.olcf.ornl.gov/titan, 2015.
24. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556, 2014.
25. N. Strom. Scalable distributed dnn training using commodity gpu cloud computing. In INTERSPEECH, 2015.
26. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. arXiv:1409.4842, 2014.
27. R. Thakur, R. Rabenseifner, and W. Gropp. Optimization of collective communication operations in mpich. International Journal of High Performance Computing Applications, 2005.
28. N. Vasilache, J. Johnson, M. Mathieu, S. Chintala, S. Piantino, and Y. LeCun. Fast convolutional nets with fbfft: A gpu performance evaluation. arXiv:1412.7580, 2014.
29. J. Wawrzynek, K. Asanovic, B. Kingsbury, D. Johnson, J. Beck, and N. Morgan. Spert-ii: A vector microprocessor system. Computer, 1996.
30. R. Wu, S. Yan, Y. Shan, Q. Dang, and G. Sun. Deep image: Scaling up image recognition. arXiv:1501.02876, 2015.
31. Z. Wu, Y. Zhang, F. Yu, and J. Xiao. A gpu implementation of googlenet. http://vision.princeton.edu/pvt/GoogLeNet, 2014.
32. B. Xu, N. Wang, T. Chen, and M. Li. Empirical evaluation of rectified activations in convolutional network. arXiv:1505.00853, 2015.
33. O. Yadan, K. Adams, Y. Taigman, and M. Ranzato. Multigpu training of convnets. arXiv:1312.5853, 2013.
34. Y. Zou, X. Jin, Y. Li, Z. Guo, E. Wang, and B. Xiao. Mariana: Tencent deep learning platform and its applications. In VLDB, 2014.