Taming the wild: A unified analysis of HOGWILD!-Style algorithms

Advances in Neural Information Processing Systems

Volumn 2015-January, Issue , 2015, Pages 2674-2682

  De Sa, Christopher ^a   Zhang, Ce ^a   Olukotun, Kunle ^a   Ré, Christopher ^a  

^a STANFORD UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; INFORMATION SCIENCE; LEARNING SYSTEMS; STOCHASTIC SYSTEMS;

ASYNCHRONOUS EXECUTIONS; CONVERGENCE RATES; MACHINE LEARNING PROBLEM; MATRIX COMPLETION; REDUCED PRECISION; RUN-TIME PERFORMANCE; STOCHASTIC GRADIENT DESCENT; UNIFIED ANALYSIS;

ALGORITHMS;

EID: 84965170302     PISSN: 10495258     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. Léon Bottou. Large-scale machine learning with stochastic gradient descent. In COMPSTAT'2010, pages 177-186. Springer, 2010.
2. Léon Bottou. Stochastic gradient descent tricks. In Neural Networks: Tricks of the Trade, pages 421-436. Springer, 2012.
3. Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. In J. C. Platt, D. Koller, Y. Singer, and S. Roweis, editors, NIPS, volume 20, pages 161-168. NIPS Foundation, 2008.
4. Christopher De Sa, Kunle Olukotun, and Christopher Ré. Global convergence of stochastic gradient descent for some nonconvex matrix problems. ICML, 2015.
5. John C Duchi, Peter L Bartlett, and Martin J Wainwright. Randomized smoothing for stochastic optimization. SIAM Journal on Optimization, 22(2):674-701, 2012.
6. Olivier Fercoq and Peter Richtárik. Accelerated, parallel and proximal coordinate descent. arXiv preprint arXiv:1312.5799, 2013.
7. Thomas R Fleming and David P Harrington. Counting processes and survival analysis. volume 169, pages 56-57. John Wiley & Sons, 1991.
8. Pankaj Gupta, Ashish Goel, Jimmy Lin, Aneesh Sharma, Dong Wang, and Reza Zadeh. WTF: The who to follow service at twitter. WWW'13, pages 505-514, 2013.
9. Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. ICML, 2015.
10. Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. In STOC, pages 665-674. ACM, 2013.
11. Björn Johansson, Maben Rabi, and Mikael Johansson. A randomized incremental subgradient method for distributed optimization in networked systems. SIAM Journal on Optimization, 20(3):1157-1170, 2009.
12. Jakub Konecnỳ, Zheng Qu, and Peter Richtárik. S2cd: Semi-stochastic coordinate descent. In NIPS Optimization in Machine Learning workshop, 2014.
13. Yann Le Cun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. Efficient backprop. In Neural Networks, Tricks of the Trade. 1998.
14. Ji Liu and Stephen J. Wright. Asynchronous stochastic coordinate descent: Parallelism and convergence properties. SIOPT, 25(1):351-376, 2015.
15. Ji Liu, Stephen J Wright, Christopher Ré, Victor Bittorf, and Srikrishna Sridhar. An asynchronous parallel stochastic coordinate descent algorithm. JMLR, 16:285-322, 2015.
16. Ioannis Mitliagkas, Michael Borokhovich, Alexandros G. Dimakis, and Constantine Caramanis. Frogwild!: Fast pagerank approximations on graph engines. PVLDB, 2015.
17. Feng Niu, Benjamin Recht, Christopher Re, and Stephen Wright. Hogwild: A lock-free approach to parallelizing stochastic gradient descent. In NIPS, pages 693-701, 2011.
18. Cyprien Noel and Simon Osindero. Dogwild!-Distributed Hogwild for CPU & GPU. 2014.
19. Shameem Ahamed Puthiya Parambath. Matrix factorization methods for recommender systems. 2013.
20. Alexander Rakhlin, Ohad Shamir, and Karthik Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. ICML, 2012.
21. Peter Richtárik and Martin Takáč. Parallel coordinate descent methods for big data optimization. Mathematical Programming, pages 1-52, 2012.
22. Qing Tao, Kang Kong, Dejun Chu, and Gaowei Wu. Stochastic coordinate descent methods for regularized smooth and nonsmooth losses. In Machine Learning and Knowledge Discovery in Databases, pages 537-552. Springer, 2012.
23. Rachael Tappenden, Martin Takáč, and Peter Richtárik. On the complexity of parallel coordinate descent. arXiv preprint arXiv:1503.03033, 2015.
24. Hsiang-Fu Yu, Cho-Jui Hsieh, Si Si, and Inderjit S Dhillon. Scalable coordinate descent approaches to parallel matrix factorization for recommender systems. In ICDM, pages 765-774, 2012.
25. Ce Zhang and Christopher Re. Dimmwitted: A study of main-memory statistical analytics. PVLDB, 2014.