Exploiting iterative-ness for parallel ML computations

Proceedings of the 5th ACM Symposium on Cloud Computing, SOCC 2014

Volumn , Issue , 2014, Pages

  Cui, Henggang ^a   Tumanov, Alexey ^a   Wei, Jinliang ^a   Xu, Lianghong ^a   Dai, Wei ^a   Haber Kucharsky, Jesse ^a   Ho, Qirong ^a   Ganger, Gregory R ^a   Gibbons, Phillip B ^b   Gibson, Garth A ^a   Xing, Eric P ^a  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b INTEL CORPORATION   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLOUD COMPUTING; ITERATIVE METHODS;

CLOUD COMPUTING ENVIRONMENTS; DYNAMIC CACHE; INPUT DATAS; ITERATIVE ALGORITHM; LARGE-SCALE MACHINE LEARNING; MEMORY BANKS; MODEL PARAMETERS; REPEATING PATTERNS;

MACHINE LEARNING;

EID: 85118315826     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2670979.2670984     Document Type: Conference Paper

References (43)

1. A. Ahmed, M. Aly, J. Gonzalez, S. Narayanamurthy, and A. J. Smola. Scalable inference in latent variable models. In WSDM, 2012.
2. S. Ahn, B. Shahbaba, and M. Welling. Distributed stochastic gradient MCMC. In ICML, 2014.
3. M. Awasthi, D. Nellans, K. Sudan, R. Balasubramonian, and A. Davis. Handling the problems and opportunities posed by multiple on-chip memory controllers. In PACT, 2010.
4. W. Bolosky, R. Fitzgerald, and M. Scott. Simple but effective techniques for numa memory management. In SOSP, 1989.
5. S. Brin and L. Page. The anatomy of a large-scale hypertextual Web search engine. Computer Networks, 1998.
6. A. Brown, T. Mowry, and O. Krieger. Compiler-based I/O prefetching for out-of-core applications. In ACM Transactions on Computer Systems (TOCS), 2001.
7. Y. Bu, B. Howe, M. Balazinska, and M. D. Ernst. Haloop: Efficient iterative data processing on large clusters. In Proc. VLDB Endow, 2010.
8. S. Byna, Y. Chen, X. Sun, R. Thakur, and W. Gropp. Parallel I/O prefetching using MPI file caching and I/O signatures. In ACM/IEEE Supercomputing, 2008.
9. P. Cao, E. Felton, and K. Li. Implementation and performance of application-controlled file caching. In OSDI, 1994.
10. R. Chandra, S. Devine, B. Verghese, A. Gupta, and M. Rosenblum. Scheduling and page migration for multiprocessor compute servers. InASPLOS, 1994.
11. F. Chang and G. A. Gibson. Automatic I/O hint generation through speculative execution. In OSDI, 1999.
12. C.-T. Chu, S. K. Kim, Y. A. Lin, Y. Yu, G. Bradski, A. Ng, and K. Olukotun. Map-reduce for machine learning on multicore. In NIPS, 2006.
13. J. Cipar, Q. Ho, J. K. Kim, S. Lee, G. R. Ganger, G. Gibson, K. Keeton, and E. Xing. Solving the straggler problem with bounded staleness. InHotOS, 2013.
14. A. Coates, B. Huval, T. Wang, D. Wu, B. Catanzaro, and N. Andrew. Deep learning with COTS HPC systems. In ICML, 2013.
15. H. Cui, J. Cipar, Q. Ho, J. K. Kim, S. Lee, A. Kumar, J. Wei, W. Dai, G. R. Ganger, P. B. Gibbons, G. A. Gibson, and E. P. Xing. Exploiting bounded staleness to speed up big data analytics. In USENIX ATC, 2014.
16. J. Dean, G. Corrado, R. Monga, K. Chen, M. Devin, Q. Le, M. Mao, M. Ranzato, A. Senior, P. Tucker, K. Yang, and A. Ng. Large scale distributed deep networks. In NIPS, 2012.
17. K. Fraser and F. Chang. Operating system I/O speculation: How two invocations are faster than one. In USENIX Annual Technical Conference, 2003.
18. R. Gemulla, E. Nijkamp, P. J. Haas, and Y. Sismanis. Large-scale matrix factorization with distributed stochastic gradient descent. InKDD, 2011.
19. A. Gerbessiotis and L. Valiant. Direct bulk-synchronous parallel algorithms. In Scandinavian Workshop on Algorithm Theory, 1992.
20. G. Gibson, G. Grider, A. Jacobson, and W Lloyd. PRObE: A thousand-node experimental cluster for computer systems research. USENIX;login:, 2013.
21. J. Gonzalez, Y Low, H. Gu, D. Bickson, and C. Guestrin. PowerGraph: Distributed graph-parallel computation on natural graphs. In OSDI, 2012.
22. J. Griffioen and R. Appleton. Reducing file system latency using a predictive approach. In Summer USENIX, 1994.
23. T. L. Griffiths and M. Steyvers. Finding scientific topics. Proceedings of the National Academy of Sciences of the United States of America, 2004.
24. Q. Ho, J. Cipar, H. Cui, S. Lee, J. K. Kim, P. B. Gibbons, G. A. Gibson, G. R. Ganger, and E. P. Xing. More effective distributed ML via a stale synchronous parallel parameter server. In NIPS, 2013.
25. Intel. Intel® Threading Building Blocks. https://www.threadingbuildingblocks.org.
26. H. Kwak, C. Lee, H. Park, and S. Moon. What is twitter, a social network or a news media? In WWW, 2010.
27. A. Kyrola, G. Blelloch, and C. Guestrin. GraphChi: Large-scale graph computation on just a PC. In OSDI, 2012.
28. J. Langford, A. J. Smola, and M. Zinkevich. Slow learners are fast. In NIPS, 2009.
29. H. Lei and D. Duchamp. An analytical approach to file prefetching. In USENIX Annual Technical Conference, 1997.
30. Y. Low, J. Gonzalez, A. Kyrola, D. Bickson, C. Guestrin, and J. M. Hellerstein. GraphLab: A new parallel framework for machine learning. In UAI, 2010.
31. G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: A system for large-scale graph processing. In SIGMOD, 2010.
32. D. Murray, F. McSherry, R. Isaacs, M. Isard, P. Barham, and M. Abadi. Naiad: A timely dataflow system. In SOSP, 2013.
33. R. H. Patterson, G. A. Gibson, E. Ginting, D. Stodolsky, and J. Zelenka. Informed prefetching and caching. In SOSP, 1995.
34. D. Peng and F. Dabek. Large-scale incremental processing using distributed transactions and notifications. In OSDI, 2010.
35. R. Power and J. Li. Piccolo: Building fast, distributed programs with partitioned tables. In OSDI, 2010.
36. A. Roy, I. Mihailovic, and W. Zwaenepoel. X-Stream: Edge-centric graph processing using streaming partitions. In SOSP, 2013.
37. TILEPro. TILEPro processor family: TILEPro64 overview. http://www.tilera.com/products/processors/TILEPro-Family, 2013.
38. A. Tumanov, J. Wise, O. Mutlu, and G. R. Ganger. Asymmetry-aware execution placement on manycore chips. In Workshop on Systems for Future Multicore Architectures (SFMA), 2013.
39. UCI. UCI Machine Learning Repository. http://archive.ics.uci.edu/ml/datasets/Bag+of+Words.
40. Y. Wang, X. Zhao, Z. Sun, H. Yan, L. Wang, Z. Jin, L. Wang, Y. Gao, J. Zeng, Q. Yang, et al. Towards topic modeling for big data. arXiv preprint arXiv:1405.4402, 2014.
41. M. Zaharia, T. Das, H. Li, S. Shenker, and I. Stoica. Discretized streams: An efficient and fault-tolerant model for stream processing on large clusters. In SOSP, 2013.
42. R. Zhang and J. Kwok. Asynchronous distributed ADMM algorithm for global variable consensus optimization. In ICML, 2014.
43. Y. Zhang, Q. Gao, L. Gao, and C. Wang. PrIter: A distributed framework for prioritized iterative computations. In SoCC, 2011.