Petuum: A new platform for distributed machine learning on big data

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Volumn 2015-August, Issue , 2015, Pages 1335-1344

  Xing, Eric P ^a   Ho, Qirong ^b   Dai, Wei ^a   Kim, Jin Kyu ^a   Wei, Jinliang ^a   Lee, Seunghak ^a   Zheng, Xun ^a   Xie, Pengtao ^a   Kumar, Abhimanu ^a   Yu, Yaoliang ^a  

^a Carnegie Mellon University   (United States)

^b INSTITUTE FOR INFOCOMM RESEARCH   (Singapore)

Author keywords

Big data; Big model; Data parallelism; Distributed systems; Machine learning; Model parallelism; Theory

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER SUPPORTED COOPERATIVE WORK; DATA MINING; ITERATIVE METHODS; LEARNING SYSTEMS; NETWORK MANAGEMENT; SCHEDULING;

DATA PARALLELISM; DISTRIBUTED MACHINE LEARNING; DISTRIBUTED SYSTEMS; GENERAL PURPOSE FRAMEWORK; GRAPHICAL REPRESENTATIONS; MATHEMATICAL DEFINITIONS; PARALLELIZATION STRATEGIES; THEORY;

BIG DATA;

EID: 84954113279     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2783258.2783323     Document Type: Conference Paper

References (36)

1. A. Agarwal and J. C. Duchi. Distributed delayed stochastic optimization. In NIPS, 2011.
2. A. Ahmed, M. Aly, J. Gonzalez, S. Narayanamurthy, and A. J. Smola. Scalable inference in latent variable models. In WSDM, 2012.
3. L. Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT'2010, pages 177-186. Springer, 2010.
4. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3:1-124, 2011.
5. J. K. Bradley, A. Kyrola, D. Bickson, and C. Guestrin. Parallel coordinate descent for l1-regularized loss minimization. In ICML, 2011.
6. X. Chen, Q. Lin, S. Kim, J. Carbonell, and E. Xing. Smoothing proximal gradient method for general structured sparse learning. In UAI, 2011.
7. W. Dai, A. Kumar, J. Wei, Q. Ho, G. Gibson, and E. P. Xing. High-performance distributed ml at scale through parameter server consistency models. In AAAI. 2015.
8. J. V. Davis, B. Kulis, P. Jain, S. Sra, and I. S. Dhillon. Information-theoretic metric learning. In Proceedings of the 24th international conference on Machine learning, pages 209-216. ACM, 2007.
9. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Ng. Large scale distributed deep networks. In NIPS 2012, 2012.
10. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
11. H. B. M. et. al. Ad click prediction: a view from the trenches. In KDD, 2013.
12. J. Friedman, T. Hastie, H. Höfling, and R. Tibshirani. Pathwise coordinate optimization. Annals of Applied Statistics, 1(2):302-332, 2007.
13. T. L. Griffiths and M. Steyvers. Finding scientific topics. PNAS, 101(Suppl 1):5228-5235, 2004.
14. Q. Ho, J. Cipar, H. Cui, J.-K. Kim, S. Lee, P. B. Gibbons, G. Gibson, G. R. Ganger, and E. P. Xing. More effective distributed ml via a stale synchronous parallel parameter server. In NIPS, 2013.
15. M. D. Hoffman, D. M. Blei, C. Wang, and J. Paisley. Stochastic variational inference. JMLR, 14, 2013.
16. A. Kumar, A. Beutel, Q. Ho, and E. P. Xing. Fugue: Slow-worker-agnostic distributed learning for big models on big data. In AISTATS, 2014.
17. Q. Le, M. Ranzato, R. Monga, M. Devin, K. Chen, G. Corrado, J. Dean, and A. Ng. Building high-level features using large scale unsupervised learning. In ICML, 2012.
18. S. Lee, J. K. Kim, X. Zheng, Q. Ho, G. Gibson, and E. P. Xing. On model parallelism and scheduling strategies for distributed machine learning. In NIPS. 2014.
19. M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su. Scaling distributed machine learning with the parameter server. In OSDI, 2014.
20. Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and J. M. Hellerstein. Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud. PVLDB, 2012.
21. G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: a system for large-scale graph processing. In ACM SIGMOD International Conference on Management of data, 2010.
22. F. Niu, B. Recht, C. Ré, and S. J. Wright. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. In NIPS, 2011.
23. R. Power and J. Li. Piccolo: building fast, distributed programs with partitioned tables. In OSDI. USENIX Association, 2010.
24. P. Richtárik and M. Takáč. Parallel coordinate descent methods for big data optimization. arXiv:1212.0873, 2012.
25. C. Scherrer, A. Tewari, M. Halappanavar, and D. Haglin. Feature clustering for accelerating parallel coordinate descent. NIPS, 2012.
26. Y. Wang, X. Zhao, Z. Sun, H. Yan, L. Wang, Z. Jin, L. Wang, Y. Gao, J. Zeng, Q. Yang, et al. Towards topic modeling for big data. arXiv:1405.4402, 2014.
27. T. White. Hadoop: The definitive guide. O'Reilly Media, Inc., 2012.
28. S. A. Williamson, A. Dubey, and E. P. Xing. Parallel markov chain monte carlo for nonparametric mixture models. In ICML, 2013.
29. E. P. Xing, M. I. Jordan, S. Russell, and A. Y. Ng. Distance metric learning with application to clustering with side-information. In NIPS, 2002.
30. H.-F. Yu, C.-J. Hsieh, S. Si, and I. Dhillon. Scalable coordinate descent approaches to parallel matrix factorization for recommender systems. In ICDM, 2012.
31. J. Yuan, F. Gao, Q. Ho, W. Dai, J. Wei, X. Zheng, E. P. Xing, T.-Y. Liu, and W.-Y. Ma. Lightlda: Big topic models on modest compute clusters. In WWW. 2015.
32. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica. Spark: cluster computing with working sets. In HotCloud, 2010.
33. Y. Zhang, Q. Gao, L. Gao, and C. Wang. Priter: A distributed framework for prioritized iterative computations. In SOCC, 2011.
34. Y. Zhang, Q. Gao, L. Gao, and C. Wang. Priter: A distributed framework for prioritizing iterative computations. IEEE Transactions on Parallel and Distributed Systems, 24(9):1884-1893, 2013.
35. Y. Zhou, D. Wilkinson, R. Schreiber, and R. Pan. Large-scale parallel collaborative filtering for the netflix prize. In Algorithmic Aspects in Information and Management, 2008.
36. M. Zinkevich, J. Langford, and A. J. Smola. Slow learners are fast. In NIPS, 2009.