Fugue: Slow-worker-agnostic distributed learning for big models on big data

Journal of Machine Learning Research

Volumn 33, Issue , 2014, Pages 531-539

  Kumar, Abhimanu ^a   Beutel, Alex ^a   Ho, Qirong ^a   Xing, Eric P ^a  

^a Carnegie Mellon University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; DATA HANDLING; LEARNING SYSTEMS;

DISTRIBUTED CLUSTERS; DISTRIBUTED LEARNING; HIGH-DIMENSIONAL; LATENT SPACE MODELS; LEARNING STRATEGY; MACHINE LEARNING APPLICATIONS; MODEL PARTITIONING; SYNCHRONIZATION STRATEGIES;

BIG DATA;

EID: 84955497549     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (19)

1. Alekh Agarwal and John C Duchi. Distributed delayed stochastic optimization. In Decision and Control (CDC), 2012 IEEE 51st Annual Conference on, pages 5451-5452. IEEE, 2012.
2. Amr Ahmed, Moahmed Aly, Joseph Gonzalez, Shravan Narayanamurthy, and Alexander J. Smola. Scalable inference in latent variable models. In WSDM, pages 123-132, 2012.
3. Animashree Anandkumar, Dean P Foster, Daniel Hsu, Sham M Kakade, and Yi-Kai Liu. Two svds suffice: Spectral decompositions for probabilistic topic modeling and latent dirichlet allocation. arXiv preprint arXiv:1204.6703, 2012.
4. David M Blei and J Lafferty. Topic models. Text mining: classification, clustering, and applications, 10:71 2009.
5. James Cipar, Qirong Ho, Jin Kyu Kim, Seunghak Lee, Gregory R Ganger, Garth Gibson, Kimberly Keeton, and Eric Xing. Solving the straggler problem with bounded stale-ness. In To appear in HotOS 14, 2013.
6. J Dean, G Corrado, R Monga, K Chen, M Devin, Q Le, M Mao, M Ranzato, A Senior, P Tucker, K Yang, and A Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems 25, 2012.
7. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
8. Rainer Gemulla, Erik Nijkamp, Peter J. Haas, and Yannis Sismanis. Large-scale matrix factorization with distributed stochastic gradient descent. In Proceedings of the 17th ACM SIGKDD international conference on Knowledge discovery and data mining, KDD '11, pages 69-77, New York, NY, USA, 2011. ACM.
9. Q. Ho, J. Cipar, H. Cui, J.-K. Kim, S. Lee, P. B. Gibbons, G. Gibson, G. R. Ganger, and E. P. Xing. More effective distributed ml via a stale synchronous parallel parameter server. In Advances in Neural Information Processing Systems 26, 2013.
10. Matt Hoffman, David M Blei, Chong Wang, and John Paisley. Stochastic variational inference. arXiv preprint arXiv:1206.7051, 2012.
11. Kenneth Kreutz-Delgado, Joseph F. Murray, Bhaskar D. Rao, Kjersti Engan, Te-Won Lee, and Terrence J. Sejnowski. Dictionary learning algorithms for sparse representation. Neural Comput., 15(2):349-396, February 2003.
12. H. Kushner and G. Yin. Stochastic Approximation and Recursive Algorithms and Applications. Springer, 2003.
13. John Langford, Alex J Smola, and Martin Zinkevich. Slow learners are fast. In Advances in Neural Information Processing Systems, pages 2331-2339, 2009.
14. Jure Leskovec, Kevin J. Lang, Anirban Dasgupta, and Michael W. Mahoney. Community structure in large networks: Natural cluster sizes and the absence of large well-defined clusters, 2008.
15. Yucheng Low, Joseph Gonzalez, Aapo Kyrola, Danny Bickson, Carlos Guestrin, and Joseph M. Hellerstein. Distributed GraphLab: A Framework for Machine Learning and Data Mining in the Cloud. PVLDB, 2012.
16. Noboru Murata. A statistical study on on-line learning. In Online Learning and Neural Networks. Cambridge University Press, 1998.
17. Feng Niu, Benjamin Recht, Christopher Ré, and Stephen J Wright. Hogwild!: A lock-free approach to parallelizing stochastic gradient descent. In NIPS, 2011.
18. Russell Power and Jinyang Li. Piccolo: building fast, distributed programs with partitioned tables. In Proceedings of the 9th USENIX conference on Operating systems design and implementation, pages 1-14. USENIX Association, 2010.
19. Martin Zinkevich, Markus Weimer, Alex Smola, and Lihong Li. Parallelized stochastic gradient descent. Advances in Neural Information Processing Systems, 23(23):1-9, 2010.