Managed communication and consistency for fast data-parallel iterative analytics

ACM SoCC 2015 - Proceedings of the 6th ACM Symposium on Cloud Computing

Volumn , Issue , 2015, Pages 381-394

  Wei, Jinliang ^a   Dai, Wei ^a   Qiao, Aurick ^a   Ho, Qirong ^b   Cui, Henggang ^a   Ganger, Gregory R ^a   Gibbons, Phillip B ^c   Gibson, Garth A ^a   Xing, Eric P ^a  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b Institute for Infocomm Research   (United States)

^c INTEL CORPORATION   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; BANDWIDTH; BUDGET CONTROL; CLOUD COMPUTING; DISTRIBUTED COMPUTER SYSTEMS; ERRORS; LEARNING SYSTEMS; STOCHASTIC SYSTEMS;

ACCUMULATED ERRORS; ALGORITHM CONVERGENCE; DISTRIBUTED IMPLEMENTATION; NETWORK COMMUNICATIONS; OPTIMAL PERFORMANCE; ORDERS OF MAGNITUDE; REVISION ALGORITHMS; STOCHASTIC GRADIENT DESCENT ALGORITHM;

ITERATIVE METHODS;

EID: 84959036260     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2806777.2806778     Document Type: Conference Paper

References (36)

1. Apache Mahout. http://mahout.apache.org/.
2. Apache Spark MLLib. https://spark.apache.org/mllib/.
3. A. Ahmed, M. Aly, J. Gonzalez, S. Narayanamurthy, and A. J. Smola. Scalable inference in latent variable models. In WSDM'12: Proceedings of the fifth ACM international conference on Web search and data mining, pages 123-132, New York, NY, USA, 2012. ACM.
4. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris. Reining in the outliers in map-reduce clusters using mantri. In Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation, OSDI'10, pages 1-16, Berkeley, CA, USA, 2010. USENIX Association.
5. G. Ananthanarayanan, A. Ghodsi, S. Shenker, and I. Stoica. Effective straggler mitigation: Attack of the clones. In Presented as part of the 10th USENIX Symposium on Networked Systems Design and Implementation (NSDI 13), pages 185-198, Lombard, IL, 2013. USENIX.
6. C. M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2006.
7. J. K. Bradley, A. Kyrola, D. Bickson, and C. Guestrin. Parallel coordinate descent for l1-regularized loss minimization. In International Conference on Machine Learning (ICML 2011), Bellevue, Washington, June 2011.
8. T. Chilimbi, Y. Suzue, J. Apacible, and K. Kalyanaraman. Project adam: Building an efficient and scalable deep learning training system. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pages 571-582, Broomfield, CO, Oct. 2014. USENIX Association.
9. J. Cipar, Q. Ho, J. K. Kim, S. Lee, G. R. Ganger, G. Gibson, K. Keeton, and E. Xing. Solving the straggler problem with bounded staleness. In Presented as part of the 14th Workshop on Hot Topics in Operating Systems, Berkeley, CA, 2013. USENIX.
10. H. Cui, J. Cipar, Q. Ho, J. K. Kim, S. Lee, A. Kumar, J. Wei, W. Dai, G. R. Ganger, P. B. Gibbons, G. A. Gibson, and E. P. Xing. Exploiting bounded staleness to speed up big data analytics. In 2014 USENIX Annual Technical Conference (USENIX ATC 14), pages 37-48, Philadelphia, PA, June 2014. USENIX Association.
11. H. Cui, A. Tumanov, J. Wei, L. Xu, W. Dai, J. Haber-Kucharsky, Q. Ho, G. R. Ganger, P. B. Gibbons, G. A. Gibson, and E. P. Xing. Exploiting iterative-ness for parallel ml computations. In Proceedings of the ACM Symposium on Cloud Computing, SOCC'14, pages 5:1-5:14, New York, NY, USA, 2014. ACM.
12. W. Dai, A. Kumar, J. Wei, Q. Ho, G. A. Gibson, and E. P. Xing. High-performance distributed ML at scale through parameter server consistency models. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA., pages 79-87, 2015.
13. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. V. Le, M. Z. Mao, M. Ranzato, A. W. Senior, P. A. Tucker, K. Yang, and A. Y. Ng. Large scale distributed deep networks. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States., pages 1232-1240, 2012.
14. J. H. Friedman, T. Hastie, and R. Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of Statistical Software, 33(1):1-22, 2010.
15. R. Gemulla, E. Nijkamp, P. J. Haas, and Y. Sismanis. Largescale matrix factorization with distributed stochastic gradient descent. In Proceedings of the 17th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD'11, pages 69-77, New York, NY, USA, 2011. ACM. .
16. A. Genkin, D. D. Lewis, and D. Madigan. Large-scale bayesian logistic regression for text categorization. Technometrics, page 2007.
17. G. Gibson, G. Grider, A. Jacobson, and W. Lloyd. Probe: A thousand-node experimental cluster for computer systems research. volume 38, June 2013.
18. W. R. Gilks. Markov chain monte carlo. Wiley Online Library, 2005.
19. J. E. Gonzalez, Y. Low, H. Gu, D. Bickson, and C. Guestrin. Powergraph: Distributed graph-parallel computation on natural graphs. In Presented as part of the 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI 12), pages 17-30, Hollywood, CA, 2012. USENIX.
20. T. L. Griffiths and M. Steyvers. Finding scientific topics. PNAS, 101(suppl. 1):5228-5235, 2004.
21. Q. Ho, J. Cipar, H. Cui, S. Lee, J. K. Kim, P. B. Gibbons, G. A. Gibson, G. Ganger, and E. P. Xing. More effective distributed ml via a stale synchronous parallel parameter server. In C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 1223-1231. Curran Associates, Inc., 2013.
22. Y. Koren, R. Bell, and C. Volinsky. Matrix factorization techniques for recommender systems. Computer, 42(8):30-37, Aug. 2009.
23. A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. Burges, L. Bottou, and K. Weinberger, editors Advances in Neural Information Processing Systems 25, pages 1097-1105. Curran Associates, Inc., 2012.
24. M. Li, D. G. Andersen, J. W. Park, A. J. Smola, A. Ahmed, V. Josifovski, J. Long, E. J. Shekita, and B.-Y. Su. Scaling distributed machine learning with the parameter server. In 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI 14), pages 583-598, Broomfield, CO, Oct. 2014. USENIX Association.
25. J. Liu, J. Chen, and J. Ye. Large-scale sparse logistic regression. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD'09, pages 547-556, New York, NY, USA, 2009. ACM.
26. Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, and J. M. Hellerstein. Distributed graphlab: A framework for machine learning and data mining in the cloud. Proc. VLDB Endow., 5(8):716-727, Apr. 2012.
27. G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: A system for large-scale graph processing. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data, SIGMOD'10, pages 135-146, New York, NY, USA, 2010. ACM.
28. H. B. McMahan and M. Streeter. Delay-tolerant algorithms for asynchronous distributed online learning. Advances in Neural Information Processing Systems (NIPS), 2014.
29. H. B. McMahan, G. Holt, D. Sculley, M. Young, D. Ebner, J. Grady, L. Nie, T. Phillips, E. Davydov, D. Golovin, S. Chikkerur, D. Liu, M. Wattenberg, A. M. Hrafnkelsson, T. Boulos, and J. Kubica. Ad click prediction: A view from the trenches. In Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2013.
30. D. G. Murray, F. McSherry, R. Isaacs, M. Isard, P. Barham, and M. Abadi. Naiad: A timely dataflow system. In Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, SOSP'13, pages 439-455, New York, NY, USA, 2013. ACM.
31. R. Power and J. Li. Piccolo: Building fast, distributed programs with partitioned tables. In Proceedings of the 9th USENIX Conference on Operating Systems Design and Implementation, OSDI'10, pages 1-14, Berkeley, CA, USA, 2010. USENIX Association.
32. L. G. Valiant. A bridging model for parallel computation. Commun. ACM, 33(8):103-111, Aug. 1990.
33. L. Yao, D. Mimno, and A. McCallum. Efficient methods for topic model inference on streaming document collections. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD'09, pages 937-946, New York, NY, USA, 2009. ACM.
34. J. W. Young. A first order approximation to the optimum checkpoint interval. Commun. ACM, 17(9):530-531, Sept. 1974.
35. H.-F. Yu, H.-Y. Lo, H.-P. Hsieh, J.-K. Lou, T. G. McKenzie, J.-W. Chou, P.-H. Chung, C.-H. Ho, Y.-H. Chang, Chun-Fu an d Wei, et al. Feature engineering and classifier ensemble for kdd cup 2010. KDD Cup, 2010.
36. M. Zinkevich, J. Langford, and A. J. Smola. Slow learners are fast. In Y. Bengio, D. Schuurmans, J. Lafferty, C. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 2331-2339. Curran Associates, Inc., 2009.