S-caffe: Co-designing MPI runtimes and caffe for scalable deep learning on modern GPU clusters

Proceedings of the ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPOPP

Volumn , Issue , 2017, Pages 193-205

  Awan, Ammar Ahmad ^a   Hamidouche, Khaled ^a   Hashmi, Jahanzeb Maqbool ^a   Panda, Dhabaleswar K ^a  

^a Ohio State University   (United States)

Author keywords

Caffe; CUDA aware MPI; Deep learning; Distributed training; MPI reduce

Indexed keywords

COBALT; COBALT COMPOUNDS; DESIGN; GRAPHICS PROCESSING UNIT; PARALLEL PROGRAMMING; PROGRAM PROCESSORS;

AGGREGATION SCHEMES; CAFFE; CO-DESIGN METHODOLOGY; CUDA-AWARE MPI; HIERARCHICAL REDUCTION; IN-DEPTH ANALYSIS; MASSIVELY PARALLELS; MPI REDUCE;

DEEP LEARNING;

EID: 85014452127     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/3018743.3018769     Document Type: Conference Paper

References (49)

1. Caffe: Multi-GPU Usage and Performance. https://github. com/yahoo/caffe/blob/master/docs/multigpu.md.
2. KESCH: Cray CS-Storm System. http://www.cscs.ch/computers/kesch escha/index.html.
3. Intel Caffe. https://github.com/intelcaffe.
4. A Unified Runtime System for Heterogeneous Multicore Architectures. http://starpu.gforge.inria.fr.
5. ILSVRC2012 Dataset. http://image-net.org/challenges/LSVRC/2012/index, 2012. [Online; accessed Dec-2016].
6. Caffe Website. http://caffe.berkeleyvision.org/, 2015. [Online; accessed Dec-2016].
7. CaffeNet. http://papers.nips.cc/book/advances-in-neuralinformation-processing-systems-25-2012, 2015. [Online; accessed Dec-2016].
8. GPU Direct RDMA. http://docs.nvidia.com/cuda/gpudirectrdma/, 2015. [Online; accessed Dec-2016].
9. HPC: Powering Deep Learning. http://computing.ornl. gov/workshops/SMC15/docs/bcatanzaro smcc.pdf, 2015. [Online; accessed Dec-2016].
10. LMDB. http://symas.com/mdb/, 2015. [Online; accessed Dec-2016].
11. Nvidia Development Platform for Autonomous Cars. http://www.nvidia.com/object/drive-px.html, 2016. [Online; accessed Dec-2016].
12. CNTK. http://www.cntk.ai/, 2016. [Online; accessed Dec-2016].
13. Nvidia GPUs Comparison. http://www.extremetech.com/computing/194391-nvidias-new-tesla-k80-doubles-up-ongpu-horsepower, 2016. [Online; accessed Dec-2016].
14. M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems, 2015. Software available from tensorflow.org.
15. J. A. Anderson, C. D. Lorenz, and A. Travesset. General Purpose Molecular Dynamics Simulations Fully Implemented on Graphics Processing Units. Journal of Computational Physics, 227(10):5342-5359, 2008.
16. S. Bahrampour, N. Ramakrishnan, L. Schott, and M. Shah. Comparative Study of Caffe, Neon, Theano, and Torch for Deep Learning. CoRR, abs/1511.06435, 2016.
17. F. Bastien, P. Lamblin, R. Pascanu, J. Bergstra, I. Goodfellow, A. Bergeron, N. Bouchard, D. Warde-Farley, and Y. Bengio. Theano: New Features and Speed Improvements. arXiv preprint arXiv:1211.5590, 2012.
18. D. Case, J. Berryman, R. Betz, D. Cerutti, T. Cheatham III, T. Darden, R. Duke, T. Giese, H. Gohlke, A. Goetz, et al. AMBER 2015. University of California, San Francisco, 2015.
19. T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman. Project adam: Building an efficient and scalable deep learning training system. In Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation, OSDI'14, pages 571-582, Berkeley, CA, USA, 2014. USENIX Association. ISBN 978-1-931971-16-4. URL http://dl.acm.org/citation. cfm?id=2685048.2685094.
20. A. Coates, B. Huval, T. Wang, D. Wu, B. Catanzaro, and N. Andrew. Deep Learning with COTS HPC Systems. In Proceedings of the 30th international conference on machine learning, pages 1337-1345, 2013.
21. R. Collobert, S. Bengio, and J. Mariéthoz. Torch: A Modular Machine Learning Software Library. Technical report, IDIAP, 2002.
22. Cray. http://docs.cray.com/books/004-3689-001/html-004-3689-001/004-3689-001-toc.html, 2016. [Online; accessed Dec-2016].
23. H. Cui, H. Zhang, G. R. Ganger, P. B. Gibbons, and E. P. Xing. Geeps: Scalable deep learning on distributed gpus with a gpuspecialized parameter server. In Proceedings of the Eleventh European Conference on Computer Systems, EuroSys'16, pages 4:1-4:16, New York, NY, USA, 2016. ACM. ISBN 978-1-4503-4240-7. doi: 10.1145/2901318.2901323. URL http://doi.acm.org/10.1145/2901318.2901323.
24. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, M. Mao, A. Senior, P. Tucker, K. Yang, Q. V. Le, et al. Large Scale Distributed Deep Networks. In Advances in Neural Information Processing Systems, pages 1223-1231, 2012.
25. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A Large-Scale Hierarchical Image Database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. IEEE, 2009.
26. Google. Google's Remote Procedure Call Library (gRPC). http://www.grpc.io,.
27. Google. Distributed TensorFlow: Github Issues. https://github.com/tensorflow/models/issues/698,.
28. R. L. Graham, S. Poole, P. Shamis, G. Bloch, N. Bloch, H. Chapman, M. Kagan, A. Shahar, I. Rabinovitz, and G. Shainer. Overlapping Computation and Communication: Barrier Algorithms and ConnectX-2 CORE-Direct Capabilities. In Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW), 2010 IEEE International Symposium on, pages 1-8. IEEE, 2010.
29. T. Hoefler, A. Lumsdaine, and W. Rehm. Implementation and Performance Analysis of Non-Blocking Collective Operations for MPI. In Supercomputing, 2007. SC'07. Proceedings of the 2007 ACM/IEEE Conference on, pages 1-10. IEEE, 2007.
30. F. N. Iandola, K. Ashraf, M. W. Moskewicz, and K. Keutzer. FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters. arXiv preprint arXiv:1511.00175, y2015.
31. Inspur. https://github.com/Caffe-MPI/Caffe-MPI.github.io, 2016.
32. J. Dean. Keynote: Large Scale Deep Learning.
33. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional Architecture for Fast Feature Embedding. arXiv preprint arXiv:1408.5093, 2014.
34. A. Krizhevsky. One weird trick for parallelizing convolutional neural networks. CoRR, abs/1404.5997, 2014.
35. A. Krizhevsky and G. Hinton. Learning Multiple Layers of Features from Tiny Images, 2009.
36. A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 1097-1105. Curran Associates, Inc., 2012. URL http://papers.nips.cc/paper/4824-imagenet-classificationwith-deep-convolutional-neural-networks.pdf.
37. S. Lee, S. Purushwalkam, M. Cogswell, D. J. Crandall, and D. Batra. Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks. arXiv, 2015. URL http://arxiv.org/abs/1511.06314.
38. M. Lin, Q. Chen, and S. Yan. Network in Network. arXiv preprint arXiv:1312.4400, 2013.
39. Lustre. Parallel File System. http://lustre.org.
40. H. Meuer, E. Strohmaier, J. Dongarra, and H. Simon. TOP 500 Supercomputer Sites. http://www.top500.org.
41. MVAPICH2: MPI over InfiniBand, 10GigE/iWARP and RoCE. https://mvapich.cse.ohio-state.edu/.
42. Network Based Computing Laboratory. OSU Micro-Benchmarks. http://mvapich.cse.ohio-state.edu/benchmarks/, 2016.
43. C. Nvidia. Programming Guide, 2008.
44. K. Simonyan and A. Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556, 2014.
45. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going Deeper with Convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1-9, 2015.
46. The HiDL Team. High Performance Deep Learning (HiDL) Project. http://hidl.cse.ohio-state.edu.
47. The Open MPI Development Team. Open MPI: Open Source High Performance Computing. http://www.open-mpi.org.
48. A. Vishnu, C. Siegel, and J. Daily. Distributed TensorFlow with MPI. arXiv preprint arXiv:1603.02339, 2016.
49. D. Wang, A. Khosla, R. Gargeya, H. Irshad, and A. H. Beck. Deep Learning for Identifying Metastatic Breast Cancer. ArXiv e-prints, June 2016.