Communication-efficient algorithms for statistical optimization

Journal of Machine Learning Research

Volumn 14, Issue , 2013, Pages 3321-3363

  Zhang, Yuchen ^a   Duchi, John C ^a   Wainwright, Martin J ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Averaging; Distributed learning; Stochastic optimization; Subsampling

Indexed keywords

AVERAGING; CENTRALIZED ALGORITHMS; DISTRIBUTED LEARNING; DISTRIBUTED OPTIMIZATION; EXPERIMENTAL EVALUATION; STATISTICAL OPTIMIZATION; STOCHASTIC OPTIMIZATIONS; SUBSAMPLING;

COMMUNICATION; MEAN SQUARE ERROR; OPTIMIZATION; SEARCH ENGINES;

ALGORITHMS;

EID: 84890032023     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (33)

1. A. Agarwal and J. C. Duchi. Distributed delayed stochastic optimization. In Advances in Neural Information Processing Systems 24, 2011.
2. A. Agarwal, P. L. Bartlett, P. Ravikumar, andM. J.Wainwright. Information-theoretic lower bounds on the oracle complexity of convex optimization. IEEE Transactions on Information Theory, 58 (5):3235-3249, May 2012.
3. S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
4. C.-C. Chang and C.-J. Lin. Libsvm: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2(3):27, 2011.
5. R. Chen, A. Gittens, and J. A. Tropp. The masked sample covariance estimator: an analysis using matrix concentration inequalities. Information and Inference, to appear, 2012.
6. A. de Acosta. Inequalities for b-valued random vectors with applications to the strong law of large numbers. The Annals of Probability, 9:157-161, 1981.
7. O. Dekel, R. Gilad-Bachrach, O. Shamir, and L. Xiao. Optimal distributed online prediction using mini-batches. Journal of Machine Learning Research, 13:165-202, 2012.
8. J. C. Duchi, A. Agarwal, and M. J. Wainwright. Dual averaging for distributed optimization: convergence analysis and network scaling. IEEE Transactions on Automatic Control, 57(3):592-606, 2012a.
9. J. C. Duchi, P. L. Bartlett, and M. J. Wainwright. Randomized smoothing for stochastic optimization. SIAM Journal on Optimization, 22(2):674-701, 2012b.
10. B. Efron and R. J. Tibshirani. An Introduction to the Bootstrap. Chapman & Hall, 1993.
11. P. Hall. The Bootstrap and Edgeworth Expansion. Springer, 1992.
12. E. Hazan, A. Kalai, S. Kale, and A. Agarwal. Logarithmic regret algorithms for online convex optimization. In Proceedings of the Nineteenth Annual Conference on Computational Learning Theory, 2006.
13. B. Johansson, M. Rabi, and M. Johansson. A randomized incremental subgradient method for distributed optimization in networked systems. SIAMJournal on Optimization, 20(3):1157-1170, 2009.
14. R. W. Keener. Theoretical Statistics: Topics for a Core Course. Springer, 2010.
15. M. Ledoux and M. Talagrand. Probability in Banach Spaces. Springer, 1991.
16. E. L. Lehmann and G. Casella. Theory of Point Estimation, Second Edition. Springer, 1998.
17. G. Mann, R. McDonald, M. Mohri, N. Silberman, and D. Walker. Efficient large-scale distributed training of conditional maximum entropy models. In Advances in Neural Information Processing Systems 22, pages 1231-1239, 2009.
18. C. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.
19. R. McDonald, K. Hall, and G. Mann. Distributed training strategies for the structured perceptron. In North American Chapter of the Association for Computational Linguistics (NAACL), 2010.
20. A. Nedić and A. Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control, 54:48-61, 2009.
21. A. Nemirovski, A. Juditsky, G. Lan, and A. Shapiro. Robust stochastic approximation approach to stochastic programming. SIAM Journal on Optimization, 19(4):1574-1609, 2009.
22. J. Nocedal and S. J. Wright. Numerical Optimization. Springer, 2006.
23. D. N. Politis, J. P. Romano, and M. Wolf. Subsampling. Springer, 1999.
24. B. T. Polyak and A. B. Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Optimization, 30(4):838-855, 1992.
25. S. Rakhlin, O. Shamir, and K. Sridharan. Making gradient descent optimal for strongly convex stochastic optimization. In Proceedings of the 29th International Conference on Machine Learning, 2012.
26. S. S. Ram, A. Nedić, and V. V. Veeravalli. Distributed stochastic subgradient projection algorithms for convex optimization. Journal of Optimization Theory and Applications, 147(3):516-545, 2010.
27. B. Recht, C. Re, S. Wright, and F. Niu. Hogwild: a lock-free approach to parallelizing stochastic gradient descent. In Advances in Neural Information Processing Systems 24, 2011.
28. H. Robbins and S. Monro. A stochastic approximation method. Annals of Mathematical Statistics, 22:400-407, 1951.
29. S. Shalev-Shwartz and T. Zhang. Stochastic dual coordinate ascent methods for regularized loss minimization. arXiv preprint arXiv:1209.1873, 2012.
30. G. Sun. KDD cup track 2 soso.com ads prediction challenge, 2012. URL http://www.kddcup2012.org/c/kddcup2012-track2. Accessed August 1, 2012.
31. A. W. van der Vaart. Asymptotic Statistics. Cambridge series in statistical and probabilistic mathematics. Cambridge University Press, 1998. ISBN 0-521-49603-9.
32. Y. Zhang, J. C. Duchi, and M. J. Wainwright. Divide and conquer kernel ridge regression. In Proceedings of the Twenty Sixth Annual Conference on Computational Learning Theory, Princeton, NJ, July 2013.
33. M. A. Zinkevich, A. Smola, M. Weimer, and L. Li. Parallelized stochastic gradient descent. In Advances in Neural Information Processing Systems 23, 2010.