High-performance distributed ML at scale through parameter server consistency models

Proceedings of the National Conference on Artificial Intelligence

Volumn 1, Issue , 2015, Pages 79-87

  Dai, Wei ^a   Kumar, Abhimanu ^a   Wei, Jinliang ^a   Ho, Qirong ^b   Gibson, Garth ^a   Xing, Eric P ^a  

^a Carnegie Mellon University   (United States)

^b Institute for Infocomm Research   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ITERATIVE METHODS; LEARNING SYSTEMS;

COMMUNICATION MECHANISMS; CONSISTENCY MODEL; DISTRIBUTED CLUSTERS; DISTRIBUTED CODES; DISTRIBUTED PROGRAMMING; SINGLE- MACHINES; THEORETICAL GUARANTEES; THEORETICAL STUDY;

ARTIFICIAL INTELLIGENCE;

EID: 84959524143     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (20)

1. Ahmed, A.; Aly, M.; Gonzalez, J.; Narayanamurthy, S.; and Smola, A. J. 2012. Scalable inference in latent variable models. In WSDM, 123-132.
2. Borthakur, D. 2007. The hadoop distributed file system: Architecture and design. Hadoop Project Website 11:21.
3. Chilimbi, T.; Suzue, Y.; Apacible, J.; and Kalyanaraman, K. 2014. Project adam: Building an efficient and scalable deep learning training system. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI14), 571-582. Broomfield, CO: USENIX Association.
4. Cipar, J.; Ho, Q.; Kim, J. K.; Lee, S.; Ganger, G. R.; Gibson, G.; Keeton, K.; and Xing, E. 2013. Solving the straggler problem with bounded staleness. In HotOS '13. Usenix.
5. Cui, H.; Cipar, J.; Ho, Q.; Kim, J. K.; Lee, S.; Kumar, A.; Wei, J.; Dai, W.; Ganger, G. R.; Gibbons, P. B.; Gibson, G. A.; and Xing, E. P. 2014. Exploiting bounded staleness to speed up big data analytics. In 2014 USENIX Annual Technical Conference (USENIX ATC 14), 37-48. Philadelphia, PA: USENIX Association.
6. Dean, J.; Corrado, G.; Monga, R.; Chen, K.; Devin, M.; Le, Q.; Mao, M.; Ranzato, M.; Senior, A.; Tucker, P.; Yang, K.; and Ng, A. 2012. Large scale distributed deep networks. In NIPS 2012.
7. Gemulla, R.; Nijkamp, E.; Haas, P. J.; and Sismanis, Y. 2011. Large-scale matrix factorization with distributed stochastic gradient descent. In KDD, 69-77. ACM.
8. Ho, Q.; Cipar, J.; Cui, H.; Kim, J.-K.; Lee, S.; Gibbons, P. B.; Gibson, G.; Ganger, G. R.; and Xing, E. P. 2013. More effective distributed ml via a stale synchronous parallel parameter server. In NIPS.
9. Koren, Y. 2009. Matrix factorization techniques for recom-mender systems. IEEE Computer 42(8):30-37.
10. Kumar, A.; Beutel, A.; Ho, Q.; and Xing, E. P. 2014. Fugue: Slow-worker-agnostic distributed learning for big models on big data. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics, 531-539.
11. Langford, J.; Li, L.; and Strehl, A. 2007. Vowpal wabbit online learning project.
12. Lee, S.; Kim, J. K.; Zheng, X.; Ho, Q.; Gibson, G. A.; and Xing, E. P. 2014. Primitives for dynamic big model parallelism. In Advances in Neural Information Processing Systems (NIPS).
13. Li, M.; Yang, L. Z. Z.; Xia, A. L. F.; Andersen, D. G.; and Smola, A. 2013. Parameter server for distributed machine learning. NIPS workshop.
14. Li, M.; Andersen, D. G.; Park, J. W.; Smola, A. J.; Ahmed, A.; Josifovski, V.; Long, J.; Shekita, E. J.; and Su, B.-Y. 2014. Scaling distributed machine learning with the parameter server. In Operating Systems Design and Implementation (OSDI).
15. Low, Y; Gonzalez, J.; Kyrola, A.; Bickson, D.; Guestrin, C.; and Hellerstein, J. M. 2010. Graphlab: A new parallel framework for machine learning. In Conference on Uncertainty in Artificial Intelligence (UAI).
16. Niu, F.; Recht, B.; Ré, G; and Wright, S. J. 2011. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. In NIPS.
17. Wang, Y; Bai, H.; Stanton, M.; Chen, W.-Y; and Chang, E. Y. 2009. Pida: Parallel latent dirichlet allocation for large-scale applications. In Proceedings of the 5th International Conference on Algorithmic Aspects in Information and Management, AAIM '09, 301-314. Berlin, Heidelberg: Springer-Verlag.
18. Yao, L.; Mimno, D.; and McCallum, A. 2009. Efficient methods for topic model inference on streaming document collections. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '09, 937-946. New York, NY, USA: ACM.
19. Yu, H.-F; Hsieh, C.-J.; Si, S.; and Dhillon, I. S. 2012. Scalable coordinate descent approaches to parallel matrix factorization for recommender systems. In ICDM, 765-774.
20. Zaharia, M.; Chowdhury, N. M. M.; Franklin, M.; Shenker, S.; and Stoica, I. 2010. Spark: Cluster computing with working sets. Technical Report UCB/EECS-2010-53, EECS Department, University of California, Berkeley.