A reliable effective terascale linear learning system

Journal of Machine Learning Research

Volumn 15, Issue , 2014, Pages 1111-1133

  Agarwal, Alekh ^a   Chapelle, Olivier ^b   Dudík, Miroslav ^a   Langford, John ^a  

^a MICROSOFT RESEARCH   (United States)

^b Criteo   (United States)

Author keywords

AllReduce; Distributed L BFGS; Distributed machine learning; Hadoop; Repeated online averaging

Indexed keywords

ARTIFICIAL INTELLIGENCE; SOFTWARE ENGINEERING;

ALL-REDUCE; DISTRIBUTED L-BFGS; DISTRIBUTED MACHINE LEARNING; HADOOP; REPEATED ONLINE AVERAGING;

LEARNING SYSTEMS;

EID: 84899845537     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (35)

1. A. Agarwal and J. Duchi. Distributed delayed stochastic optimization. In Advances in Neural Information Processing Systems 24. 2011.
2. P. L. Bartlett and S. Mendelson. Rademacher and Gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3:463-482, 2002.
3. R. Bekkerman, M. Bilenko, and J. Langford. A tutorial on scaling up machine learning. Technical report, KDD, 2011. URL http://hunch.net/-large- scale-survey/.
4. D. P. Bertsekas and J. N. Tsitsiklis. Parallel and Distributed Computation: Numerical Methods. Prentice-Hall, Inc., 1989.
5. S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eckstein. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine Learning, 3:1-122, 2011.
6. K. Canini, T. Chandra, E. Ie, J. McFadden, K. Goldman, M. Gunter, J. Harmsen, K. LeFevre, D. Lepikhin, T. L. Llinares, I. Mukherjee, F. Pereira, J. Redstone, T. Shaked, and Y. Singer. Sibyl: A system for large scale supervised machine learning. In MLSS Santa Cruz, 2012. URL http://users.soe.ucsc.edu/- niejiazhong/slides/chandra.pdf. A short presentation.
7. T. Chandra, E. Ie, K. Goldman, T. L. Llinares, J. McFadden, F. Pereira, J. Redstone, T. Shaked, and Y. Singer. Sibyl: a system for large scale machine learning. In LADIS 2010: The 4th ACM SIGOPS/SIGACT Workshop on Large Scale Distributed Systems and Middleware, 2010. URL http://www.magicbroom.info/Papers/ Ladis10.pdf. A keynote talk.
8. O. Chapelle, E. Manavoglu, and R. Rosales. Simple and scalable response prediction for display advertising. Transactions on Intelligent Systems and Technology, 2013. In press.
9. C.T. Chu, S.K. Kim, Y.A. Lin, Y.Y. Yu, G. Bradski, A.Y. Ng, and K. Olukotun. Mapreduce for machine learning on multicore. In Advances in Neural Information Processing Systems 19, volume 19, page 281, 2007.
10. J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. Com- munications of the ACM, 51:107-113, 2008.
11. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. A. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems 25, pages 1232-1240. 2012.
12. O. Dekel, R. Gilad-Bachrach, O. Shamir, and L. Xiao. Optimal distributed online prediction using mini-batches. Journal of Machine Learning Research, 13:165-202, 2012.
13. L. Devroye, L. Györ, and G. Lugosi. A Probabilistic Theory of Pattern Recognition. Springer, 1996
14. J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, 2010.
15. J.C. Duchi, A. Agarwal, and M.J. Wainwright. Dual averaging for distributed optimization: Convergence analysis and network scaling. Automatic Control, IEEE Transactions on, 57(3):592-606, 2012.
16. K. Hall, S. Gilpin, and G. Mann. Mapreduce/bigtable for distributed optimization. In Workshop on Learning on Cores, Clusters, and Clouds, 2010.
17. D. Hsu, N. Karampatziakis, J. Langford, and A. Smola. Parallel online learning. In Scaling Up Machine Learning, 2011.
18. N. Karampatziakis and J. Langford. Online importance weight aware updates. In Uncer- tainty in Artificial Intelligence, 2011.
19. M. Kearns. Eficient noise-tolerant learning from statistical queries. In Proceedings of the Twenty-Fifth Annual ACM Symposium on the Theory of Computing, 1993.
20. J. Langford, L. Li, and A. Strehl. Vowpal wabbit open source project. Technical report, Yahoo!, 2007.
21. J. Langford, A. Smola, and M. Zinkevich. Slow learners are fast. In Advances in Neural Information Processing Systems 22, 2009.
22. D. C. Liu and J. Nocedal. On the limited memory bfgs method for large scale optimization. Mathematical Programming, 45:503-528, 1989. (Pubitemid 20660315)
23. Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and J. M. Hellerstein. Graphlab: A new framework for parallel machine learning. In Uncertainty in Artificial Intelligence, 2010.
24. O. L. Mangasarian. Parallel gradient distribution in unconstrained optimization. SIAM Journal on Optimization, 33:1916-1925, 1995.
25. R. McDonald, K. Hall, and G. Mann. Distributed training strategies for the structured perceptron. In North American Chapter of the Association for Computational Linguistics (NAACL), 2010.
26. H. B. McMahan and M. Streeter. Adaptive bound optimization for online convex optimization. In Proceedings of the Twenty Third Annual Conference on Computational Learning Theory, 2010.
27. S. G. Nash and A. Sofer. Block truncated-newton methods for parallel optimization. Math- ematical Programming, 45:529-546, 1989. (Pubitemid 20660316)
28. J. Nocedal. Updating quasi-Newton matrices with limited storage. Math. Comp., 35(151): 773-782, 1980.
29. S. Sonnenburg and V. Franc. COFFIN: a computational framework for linear SVMs. In International Conference on Machine Learning, 2010.
30. S. Sonnenburg, G. Rätsch, and K. Rieck. Large scale learning with string kernels. In Large Scale Kernel Machines, pages 73-103. 2007.
31. C. Teo, Q. Le, A. Smola, and SVN Vishwanathan. A scalable modular convex solver for regularized risk minimization. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2007.
32. K. Weinberger, A. Dasgupta, J. Langford, A. Smola, and J. Attenberg. Feature hashing for large scale multitask learning. In International Conference on Machine Learning, 2009.
33. J. Ye, J.-H. Chow, J. Chen, and Z. Zheng. Stochastic gradient boosted distributed decision trees. In Proceeding of the 18th ACM Conference on Information and Knowledge Management, 2009.
34. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. Technical Report UCB/EECS-2011-82, EECS Department, University of California, Berkeley, 2011.
35. M. Zinkevich, M. Weimer, A. Smola, and L. Li. Parallelized stochastic gradient descent. In Advances in Neural Information Processing Systems 23. 2010.