On model parallelization and scheduling strategies for distributed machine learning

Advances in Neural Information Processing Systems

Volumn 4, Issue January, 2014, Pages 2834-2842

  Lee, Seunghak ^a   Kim, Jin Kyu ^a   Zheng, Xun ^a   Ho, Qirong ^b   Gibson, Garth A ^a   Xing, Eric P ^a  

^a Carnegie Mellon University   (United States)

^b INSTITUTE FOR INFOCOMM RESEARCH   (Singapore)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER PROGRAMMING; FACTORIZATION; INFORMATION SCIENCE; ITERATIVE METHODS; LEARNING SYSTEMS; SCHEDULING;

DISTRIBUTED MACHINE LEARNING; MATRIX FACTORIZATIONS; PARALLEL EXECUTIONS; PROGRAMMING ABSTRACTIONS; SCHEDULING EFFICIENCIES; SCHEDULING PARAMETERS; SCHEDULING STRATEGIES; SYNCHRONIZATION SCHEME;

BIG DATA;

EID: 84937822418     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. A. Ahmed, M. Aly, J. Gonzalez, S. Narayanamurthy, and A. J. Smola. Scalable inference in latent variable models. In WSDM, 2012.
2. J. Bennett and S. Lanning. The Netflix prize. In Proceedings of KDD cup and workshop, 2007.
3. D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. The Journal of Machine Learning Research, 3: 993-1022, 2003.
4. J. K. Bradley, A. Kyrola, D. Bickson, and C. Guestrin. Parallel coordinate descent for l1-regularized loss minimization. In ICML, 2011.
5. W. Dai, A. Kumar, J. Wei, Q. Ho, G. Gibson, and E. P. Xing. High-performance distributed ML at scale through parameter server consistency models. In AAAI, 2014.
6. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. W. Senior, P. A. Tucker, et al. Large scale distributed deep networks. In NIPS, 2012.
7. J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters. Communications of the ACM, 51(1): 107-113, 2008.
8. J. Fan, R. Samworth, and Y. Wu. Ultrahigh dimensional feature selection: beyond the linear model. The Journal of Machine Learning Research, 10: 2013-2038, 2009.
9. J. Friedman, T. Hastie, H. Hofling, and R. Tibshirani. Pathwise coordinate optimization. Annals of Applied Statistics, 1(2): 302-332, 2007.
10. R. Gemulla, E. Nijkamp, P. J. Haas, and Y. Sismanis. Large-scale matrix factorization with distributed stochastic gradient descent. In SIGKDD, 2011.
11. G. Gibson, G. Grider, A. Jacobson, and W. Lloyd. PRObE: A thousand-node experimental cluster for computer systems research. USENIX; login, 38, 2013.
12. J. Gonzalez, Y. Low, A. Gretton, and C. Guestrin. Parallel gibbs sampling: From colored fields to thin junction trees. In AISTATS, 2011.
13. J. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin. PowerGraph: Distributed graph-parallel computation on natural graphs. In OSDI, 2012.
14. T. L. Griffiths and M. Steyvers. Finding scientific topics. Proceedings of the National Academy of Sciences of the United States of America, 101(Suppl 1): 5228-5235, 2004.
15. Q. Ho, J. Cipar, H. Cui, J. Kim, S. Lee, P. B. Gibbons, G. Gibson, G. R. Ganger, and E. P. Xing. More effective distributed ML via a stale synchronous parallel parameter server. In NIPS, 2013.
16. Jey Han Lau, Timothy Baldwin, and David Newman. On collocations and topic models. ACM Transactions on Speech and Language Processing (TSLP), 10(3): 10, 2013.
17. Q. V. Le, M. A. Ranzato, R. Monga, M. Devin, K. Chen, G. S. Corrado, J. Dean, and A. Y. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012.
18. M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B. Su. Scaling distributed machine learning with the parameter server. In OSDI, 2014.
19. Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and J. M. Hellerstein. Distributed GraphLab: A framework for machine learning and data mining in the cloud. In VLDB, 2012.
20. D. Newman, A. Asuncion, P. Smyth, and M. Welling. Distributed algorithms for topic models. The Journal of Machine Learning Research, 10: 1801-1828, 2009.
21. C. Scherrer, A. Tewari, M. Halappanavar, and D. Haglin. Feature clustering for accelerating parallel coordinate descent. In NIPS, 2012.
22. Y. Wang, X. Zhao, Z. Sun, H. Yan, L. Wang, Z. Jin, L. Wang, Y. Gao, J. Zeng, Q. Yang, et al. Towards topic modeling for big data. arXiv: 1405.4402 [cs.IR], 2014.
23. J. Wei, W. Dai, A. Kumar, X. Zheng, Q. Ho, and E. P. Xing. Consistent bounded-asynchronous parameter servers for distributed ML. arXiv: 1312.7869 [stat.ML], 2013.
24. H. Yu, C. Hsieh, S. Si, and I. Dhillon. Scalable coordinate descent approaches to parallel matrix factorization for recommender systems. In ICDM, 2012.
25. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. Spark: Cluster computing with working sets. In HotCloud, 2010.
26. Y. Zhou, D. Wilkinson, R. Schreiber, and R. Pan. Large-scale parallel collaborative filtering for the netflix prize. In AAIM, 2008.
27. M. Zinkevich, J. Langford, and A. J. Smola. Slow learners are fast. In NIPS, 2009.
28. M. Zinkevich, M. Weimer, L. Li, and A. J. Smola. Parallelized stochastic gradient descent. In NIPS, 2010.