Twister: A runtime for iterative MapReduce

HPDC 2010 - Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

Volumn , Issue , 2010, Pages 810-818

  Ekanayake, Jaliya ^a,b   Li, Hui ^a,b   Zhang, Bingjing ^a,b   Gunarathne, Thilina ^a,b   Bae, Seung Hee ^a,b   Qiu, Judy ^b   Fox, Geoffrey ^a,b  

^a INDIANA UNIVERSITY   (United States)

^b INDIANA UNIVERSITY   (United States)

Author keywords

Cloud technologies; Iterative algorithms; MapReduce

Indexed keywords

DATA PARALLEL; DISTRIBUTED COMPUTING; DRYADLINQ; ITERATIVE ALGORITHM; ITS ARCHITECTURE; LARGE SCALE DATA; MAP-REDUCE; PERFORMANCE COMPARISON; PROGRAMMING MODELS; RUNTIMES; SCIENTIFIC APPLICATIONS;

ALGORITHMS; ITERATIVE METHODS;

MULTIPROCESSING SYSTEMS;

EID: 78650003594     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1851476.1851593     Document Type: Conference Paper

References (38)

1. J. Dean and S. Ghemawat, "MapReduce: simplified data processing on large clusters," Commun. ACM, vol. 51, pp. 107-113, 2008.
2. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: distributed data-parallel programs from sequential building blocks," presented at the Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, Lisbon, Portugal, 2007.
3. J. Ekanayake, A. Balkir, T. Gunarathne, G. Fox, C. Poulain, N. Araujo, and R. Barga, "DryadLINQ for Scientific Analyses," presented at the 5th IEEE International Conference on e-Science, Oxford UK, 2009.
4. J. Ekanayake, S. Pallickara, and G. Fox, "MapReduce for Data Intensive Scientific Analyses," presented at the Proceedings of the 2008 Fourth IEEE International Conference on eScience, 2008.
5. J. Ekanayake, X. Qiu, T. Gunarathne, S. Beason, and G. Fox, "High Performance Parallel Computing with Clouds and Cloud Technologies," in Cloud Computing and Software Services: Theory and Techniques, ed: CRC Press (Taylor and Francis).
6. G. Fox, S.-H. Bae, J. Ekanayake, X. Qiu, and H. Yuan, "Parallel Data Mining from Multicore to Cloudy Grids," presented at the International Advanced Research Workshop on High Performance Computing and Grids (HPC2008), Cetraro, Italy, 2008.
7. MPI (Message Passing Interface). Available: http://www-unix.mcs.anl.gov/ mpi/
8. PVM (Parallel Virtual Machine). Available: http://www.csm.ornl.gov/pvm/
9. C.-T. Chu, S. K. Kim, Y.-A. Lin, Y. Yu, G. R. Bradski, A. Y. Ng, and K. Olukotun, "Map-Reduce for Machine Learning on Multicore," in NIPS, ed: MIT Press, 2006, pp. 281-288.
10. Apache Hadoop. Available: http://hadoop.apache.org/
11. Y. Gu and R. L. Grossman, "Sector and Sphere: the design and implementation of a high-performance data cloud," Philosophical transactions. Series A, Mathematical, physical, and engineering sciences, vol. 367, pp. 2429-2445, 2009.
12. Twister:A Runtime for Iterative MapReduce. Available: http://www.iterativemapreduce.org/
13. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and C. J., "DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language," in Symposium on Operating System Design and Implementation (OSDI), 2008.
14. Disco project. Available: http://discoproject.org/
15. S. Ghemawat, H. Gobioff, and S.-T. Leung, "The Google file system," SIGOPS Oper. Syst. Rev., vol. 37, pp. 29-43, 2003.
16. J. B. MacQueen, "Some Methods for Classification and Analysis of MultiVariate Observations," in Proc. of the fifth Berkeley Symposium on Mathematical Statistics and Probability. vol. 1, L. M. L. Cam and J. Neyman, Eds., ed: University of California Press, 1967.
17. K. Rose, E. Gurewwitz, and G. Fox, "A deterministic annealing approach to clustering," Pattern Recogn. Lett., vol. 11, pp. 589-594, 1990.
18. S. Brin and L. Page. The Anatomy of a Large-Scale Hypertextual Web Search Engine. Available: http://infolab.stanford.edu/~backrub/google.html
19. J. de Leeuw, "Applications of convex analysis to multidimensional scaling," Recent Developments in Statistics, pp. 133-145, 1977.
20. S. Pallickara and G. Fox, "NaradaBrokering: A Distributed Middleware Framework and Architecture for Enabling Durable Peer-to-Peer Grids," presented at the Middleware 2003, 2003.
21. ActiveMQ. Available: http://activemq.apache.org/
22. C. Moretti, H. Bui, K. Hollingsworth, B. Rich, P. Flynn, and D. Thain, "All-Pairs: An Abstraction for Data Intensive Computing on Campus Grids," in IEEE Transactions on Parallel and Distributed Systems, 2010, pp. 33-46.
23. O. Gotoh, "An improved algorithm for matching biological sequences," Journal of Molecular Biology vol. 162, pp. 705-708, 1982.
24. Source Code. Smith Waterman Software. Available: http://jaligner. sourceforge.net/
25. J. Qiu, J. Ekanayake, T. Gunarathne, J. Y. Choi, S.-H. Bae, Y. Ruan, S. Ekanayake, S. Wu, S. Beason, G. Fox, M. Rho, and H. Tang, "Data Intensive Computing for Bioinformatics," in Data Intensive Distributed Computing, ed: IGI Publishers, 2010.
26. A. F. A. Smit, R. Hubley, and P. Green. (2004, Repeatmasker. Available: http://www.repeatmasker.org
27. J. Jurka, "Repbase Update:a database and an electronic journal of repetitive elements," Trends in Genetics, vol. 6, pp. 418-420, 2000.
28. J. Kruskal, "Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis," Psychometrika, vol. 29, pp. 1-27, 1964.
29. Y. Takane, Young, F. W., & de Leeuw, J., "Nonmetric individual differences multidimensional scaling: an alternating least squares method with optimal scaling features," Psychometrika, vol. 42, pp. 7-67, 1977.
30. I. Borg, & Groenen, P. J., Modern Multidimensional Scaling: Theory and Applications: Springer, 2005.
31. Y. Zhu, S. Ye, and X. Li, "Distributed PageRank computation based on iterative aggregation-disaggregation methods," presented at the Proceedings of the 14th ACM international conference on Information and knowledge management, Bremen, Germany, 2005.
32. S. Kamvar, T. Haveliwala, C. Manning, and G. Golub, "Exploiting the Block Structure of the Web for Computing PageRank," Stanford InfoLab, Technical Report2003.
33. The Power Method. Available: http://en.wikipedia.org/wiki/Pagerank#Power- Method
34. (2009, The ClueWeb09 Dataset. Available: http://boston.lti.cs.cmu.edu/ Data/clueweb09/
35. C. Ranger, R. Raghuraman, A. Penmetsa, G. Bradski, and C. Kozyrakis, "Evaluating MapReduce for multi-core and multiprocessor systems," in 13th International Symposium on High-Performance Computer Architecture, 2007, pp. 13-24.
36. C. Team. (2009, Condor DAGMan. Available: http://www.cs.wisc.edu/condor/ dagman/.
37. LINQ Language-Integrated Query. Available: http://msdn.microsoft.com/en- us/netframework/aa904594.aspx
38. Y. Zhao, M. Hategan, B. Clifford, I. Foster, G. v. Laszewski, V. Nefedova, I. Raicu, T. Stef-Praun, and M. Wilde, "Swift: Fast, Reliable, Loosely Coupled Parallel Computation," in IEEE Congress on Services, 2007, pp. 199-206.