Project ADAM: Building an efficient and scalable deep learning training system

Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation, OSDI 2014

Volumn , Issue , 2014, Pages 571-582

  Chilimbi, Trishul ^a   Suzue, Yutaka ^a   Apacible, Johnson ^a   Kalyanaraman, Karthik ^a  

^a MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

DISTRIBUTED COMPUTER SYSTEMS; SYSTEMS ANALYSIS;

DESIGN AND IMPLEMENTATIONS; DISTRIBUTED SYSTEMS; IMPROVE PERFORMANCE; NEURAL NETWORK MODEL; STATE OF THE ART; TRAINING ALGORITHMS; TRAINING SYSTEMS; VISUAL RECOGNITION;

DEEP NEURAL NETWORKS;

EID: 85069497682     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. Bengio, Y., and LeCun, Y. 2007. Scaling Learning Algorithms towards AI. In Large-Scale Kernel Machines, Bottou, L. et al. (Eds), MIT Press.
2. Bottou, L., 1991. Stochastic gradient learning in neural networks. In Proceedings of Neuro-Nîmes 91, EC2, Nimes, France.
3. S. Chakradhar, M. Sankaradas, V. Jakkula, and S. Cadambi. 2010. A dynamically configurable coprocessor for convolutional neural networks. In International symposium on Computer Architecture, ISCA'10.
4. Chen, T., Du, Z., Sun, N., Wang, J., Wu, C., Chen, Y., and Temam, O. 2014. DianNao: A Small-Footprint High-Throughput Accelerator for Ubiquitous Machine-Learning. In International Conference on Architectural Support for Programming Languages and Operating Systems. ASPLOS'14.
5. Ciresan, D. C, Meier, U., and Schmidhuber, J. 2012. Multicolumn deep neural networks for image classification. In Computer Vision and Pattern Recognition. CVPR'12.
6. Coates, A., Huval, B., Wang, T., Wu, D., Ng, A., and Catanzaro, B. 2013. Deep Learning with COTS HPC. In International Conference on Machine Learning. ICML'13.
7. Dean, J., Corrado, G., Monga, R., Chen, K., Devin, M., Mao, M., Ranzato, M., Senior, A., Tucker, P., Yang, K., Le, Q., and Ng, A. 2012. Large Scale Distributed Deep Networks. In Advances in Neural Information Processing Systems. NIPS'12.
8. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In Computer Vision and Pattern Recognition. CVPR '09.
9. Faerber, P., and Asanović, K. 1997. Parallel neural network training on Multi-Spert. In IEEE 3rd International Conference on Algorithms and Architectures for Parallel Processing (Melbourne, Australia, December 1997).
10. Farabet, C., Martini, B., Corda, B., Akselrod, P., Culurciello, E., and LeCun, Y. 2011. NeuFlow: A runtime reconfigurable dataflow processor for vision. In Computer Vision and Pattern Recognition Workshop (June 2011), pages 109-116.
11. Fukushima, K. 1980. Neocognitron: A self-organizing neural network for a mechanism of pattern recognition unaffected by shift in position. In Biological Cybernetics, 36(4): 93-202.
12. Goodfellow, I., Warde-Farley, D., Mirza, M., Courville, A., and Bengio, Y. 2013. Maxout Networks. In International Conference on Machine Learning. ICML'13.
13. Hahnloser, R. 2003. Permitted and Forbidden Sets in Symmetric Threshold-Linear Networks. In Neural Computing. (Mar. 2003), 15(3):621-38.
14. Hinton, G., Deng, L., Yu, D., Dahl, G., Mohamed, A., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T., and Kingsbury, B. 2012. Deep neural networks for acoustic modeling in speech recognition. In IEEE Signal Processing Magazine, 2012.
15. Hubel, D. and Wiesel, T. 1968. Receptive fields and functional architecture of monkey striate cortex. In Journal of Physiology (London), 195, 215-243.
16. Kim, J., Member, S., Kim, M., Lee, S., Oh, J., Kim, K. and Yoo, H. 2010. A 201.4 GOPS 496 mW Real-Time Multi-Object Recognition Processor with Bio-Inspired Neural Perception Engine. In IEEE Journal of Solid-State Circuits, (Jan. 2010), 45(1):32-45.
17. Krizhevsky, A., Sutskever, I., and Hinton, G. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems. NIPS'12.
18. Le, Q., Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G., Dean, J., and Ng, A. 2012. Building high-level features using large scale unsupervised learning. In International Conference on Machine Learning. ICML'12.
19. LeCun, Y., Boser, B., Denker, J., Henderson, D., Howard, R., Hubbard, W., and Jackel, L. 1989. Backpropagation Applied to Handwritten Zip Code Recognition. In Neural Computation, 1(4):541-551.
20. LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. 1998. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, 86(11):2278-2324, (Nov. 1998).
21. Low, Y., Gonzalez, J., Kyrola, A., Bickson, D., Guestrin, C., and Hellerstein, J. 2012. Distributed GraphLab: A framework for machine learning in the cloud. In International Conference on Very Large Databases. VLDB'12.
22. Maashri, A., Debole, M., Cotter, M., Chandramoorthy, N., Xiao, Y., Narayanan, V., and Chakrabarti, C. 2012. Accelerating neuromorphic vision algorithms for recognition. In Proceedings of the 49th Annual Design Automation Conference, DAC'12.
23. Niu, F., Retcht, B., Re, C., and Wright, S. 2011. Hogwild! A lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems. NIPS'11.
24. Raina, R., Madhavan, A., and Ng., A. 2009. Large-scale deep unsupervised learning using graphics processors. In International Conference on Machine Learning. ICML'09.
25. Rumelhart, D., Hinton, G., and Williams, R. 1986. Learning representations by back-propagating errors. In Nature 323 (6088): 533-536.
26. Simard, P., Steinkraus, D., and Platt, J. 2003. Best Practices for Convolutional Neural Networks applied to Visual Document Analysis. In ICDAR, vol. 3, pp. 958-962.
27. Zeiler, M. and Fergus, R. 2013. Visualizing and Understanding Convolutional Networks. In Arxiv 1311.2901. http://arxiv.org/abs/1311.2901