A map-reduce method for training autoencoders on Xeon Phi

Proceedings - 15th IEEE International Conference on Computer and Information Technology, CIT 2015, 14th IEEE International Conference on Ubiquitous Computing and Communications, IUCC 2015, 13th IEEE International Conference on Dependable, Autonomic and Secure Computing, DASC 2015 and 13th IEEE International Conference on Pervasive Intelligence and Computing, PICom 2015

Volumn , Issue , 2015, Pages 1330-1337

  Yao, Qiongjie ^a   Liao, Xiaofei ^a   Jin, Hai ^a  

^a HUAZHONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

Author keywords

Autoencoder; BSP; Deep learning; Mapreduce; Xeon Phi

Indexed keywords

ARTIFICIAL INTELLIGENCE; LEARNING ALGORITHMS; MULTIPROCESSING SYSTEMS; SCHEDULING ALGORITHMS; UBIQUITOUS COMPUTING;

AUTO ENCODERS; BULK SYNCHRONOUS PARALLEL; DEEP LEARNING; MACHINE LEARNING APPLICATIONS; MAP-REDUCE; PERFORMANCE DEGRADATION; SEQUENTIAL IMPLEMENTATION; XEON PHI;

LEARNING SYSTEMS;

EID: 84964307446     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CIT/IUCC/DASC/PICOM.2015.197     Document Type: Conference Paper

References (24)

1. "Math kernel library", http://software.intel.com/en-us/intel-mkl.
2. K. Ahmed, Q. Qiu, P. Malani, and M. Tamhankar, "Accelerating pattern matching in neuromorphic text recognition system using intel xeon phi coprocessor", in Proceedings of International Joint Conference on Neural Networks (IJCNN), 2014, pp. 4272-4279.
3. S. Cadambi, G. Coviello, C. Li, R. Phull, K. Rao, M. Sankaradass, and S. T. Chakradhar, "Cosmic: middleware for high performance and reliable multiprocessing on xeon phi coprocessors", in Proceedings of the 22nd International Symposium on High-Performance Parallel and Distributed Computing (HPDC), 2013, pp. 215-226.
4. T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman, "Project adam: Building an efficient and scalable deep learning training system", in Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2014, pp. 571-582.
5. C. T. Chu, S. K. Kim, and Lin, "Map-reduce for machine learning on multicore", in Proceedings of the 20th Annual Conference on Neural Information Processing Systems (NIPS), 2006, pp. 281-288.
6. A. Coates, B. Huval, T. Wang, D. Wu, B. Catanzaro, and N. Andrew, "Deep learning with cots hpc systems", in Proceedings of the 30th International Conference on Machine Learning (ICML), 2013, pp. 1337-1345.
7. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. W. Senior, P. A. Tucker, K. Yang, and A. Y. Ng, "Large scale distributed deep networks", in Proceedings of the 26th Annual Conference on Neural Information Processing Systems (NIPS), 2012, pp. 1223-1231.
8. L. Dong, F. Wei, M. Zhou, and K. Xu, "Adaptive multi-compositionality for recursive neural models with applications to sentiment analysis", in Proceedings of the 28th AAAI Conference on Artificial Intelligence (AAAI), 2014, pp. 1537-1543.
9. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, and T. N. Sainath, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups", IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82-97, 2012.
10. G. E. Hinton, S. Osindero, and Y. W. Teh, "A fast learning algorithm for deep belief nets", Neural Computation, vol. 18, no. 7, pp. 1527-1554, 2006.
11. L. Jin, Z. Wang, R. Gu, C. Yuan, and Y. Huang, "Training large scale deep neural networks on the intel xeon phi many-core coprocessor", in Proceedings of the IEEE International Parallel & Distributed Processing Symposium Workshops (IPDPSW), 2014, pp. 1622-1630.
12. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks", in Proceedings of the 26th Annual Conference on Neural Information Processing Systems (NIPS), 2012, pp. 1097-1105.
13. Q. V. Le, "Building high-level features using large scale unsupervised learning", in Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2013, pp. 8595-8598.
14. Y. Le Cun, B. E. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. E. Hubbard, and L. D. Jackel, "Backpropagation applied to handwritten zip code recognition", Neural Computation, vol. 1, no. 4, pp. 541-551, 1989.
15. M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su, "Scaling distributed machine learning with the parameter server", in Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2014, pp. 583-598.
16. W. Li, R. Zhao, T. Xiao, and X. Wang, "Deepreid: Deep filter pairing neural network for person re-identification", in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 152-159.
17. X. Liu, M. Smelyanskiy, E. Chow, and P. Dubey, "Efficient sparse matrix-vector multiplication on x86-based many-core processors", in Proceedings of the 27th ACM International Conference on Supercomputing (ICS), 2013, pp. 273-282.
18. J. Martens, "Deep learning via hessian-free optimization", in Proceedings of the 30th International Conference on Machine Learning (ICML), 2010, pp. 1337-1345.
19. J. Ngiam, A. Coates, A. Lahiri, B. Prochnow, Q. V. Le, and A. Y. Ng, "On optimization methods for deep learning", in Proceedings of the 28th International Conference on Machine Learning (ICML), 2011, pp. 265-272.
20. R. Raina, A. Madhavan, and A. Y. Ng, "Large-scale deep unsupervised learning using graphics processors", in Proceedings of the 26th International Conference on Machine Learning (ICML), 2009, pp. 873-880.
21. Y. You, S. L. Song, H. Fu, A. Marquez, M. M. Dehnavi, K. Barker, K. W. Cameron, A. P. Randles, and G. Yang, "Mic-svm: Designing a highly efficient support vector machine for advanced modern multi-core and many-core architectures", in Proceedings of the 28th International Parallel and Distributed Processing Symposium (IPDPS), 2014, pp. 809-818.
22. Y. Zhi and Y. Yang, "Discrete control of longitudinal dynamics for hypersonic flight vehicle using neural networks", SCIENCE CHINA Information Sciences, vol. 58, no. 7, pp. 1-10, 2015.
23. M. Zinkevich, M. Weimer, L. Li, and A. J. Smola, "Parallelized stochastic gradient descent", in Proceedings of the 24th Annual Conference on Neural Information Processing Systems (NIPS), 2010, pp. 2595-2603.
24. Y. Zou, X. Jin, Y. Li, Z. Guo, E. Wang, and B. Xiao, "Mariana: Tencent deep learning platform and its applications", PVLDB, vol. 7, no. 13, pp. 1772-1777, 2014.