Exploiting dynamic resource allocation for efficient parallel data processing in the cloud

IEEE Transactions on Parallel and Distributed Systems

Volumn 22, Issue 6, 2011, Pages 985-997

  Warneke, Daniel ^a   Kao, Odej ^a  

^a TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

Author keywords

cloud computing.; high throughput computing; loosely coupled applications; Many task computing

Indexed keywords

CLOUD SYSTEMS; COMPUTE RESOURCES; DYNAMIC RESOURCE ALLOCATIONS; HIGH-THROUGHPUT COMPUTING; HOMOGENEOUS CLUSTER; JOB EXECUTION; KILLER-APPLICATION; LOOSELY COUPLED APPLICATIONS; MAP-REDUCE; PARALLEL DATA PROCESSING; PROCESSING JOBS; PROCESSING TIME; PRODUCT PORTFOLIOS; TASK COMPUTING; TASK-SCHEDULING; VIRTUAL MACHINES;

CLOUDS; COMPUTER RESOURCE MANAGEMENT; COMPUTER SYSTEMS; DATA FUSION; DATA HANDLING; INDUSTRIAL RESEARCH; RESOURCE ALLOCATION;

CLOUD COMPUTING;

EID: 79955523308     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPDS.2011.65     Document Type: Article

References (30)

1. Amazon Web Services LLC, "Amazon Elastic Compute Cloud (Amazon EC2)," http://aws.amazon.com/ec2/, 2009.
2. Amazon Web Services LLC, "Amazon Elastic MapReduce," http://aws.amazon.com/elasticmapreduce/, 2009.
3. Amazon Web Services LLC, "Amazon Simple Storage Service," http://aws.amazon.com/s3/, 2009.
4. D. Battré, S. Ewen, F. Hueske, O. Kao, V. Markl, and D. Warneke, "Nephele/PACTs: A Programming Model and Execution Framework for Web-Scale Analytical Processing," Proc. ACM Symp. Cloud Computing (SoCC '10), pp. 119-130, 2010.
5. R. Chaiken, B. Jenkins, P.-A. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou, "SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets," Proc. Very Large Database Endowment, vol. 1, no. 2, pp. 1265-1276, 2008.
6. H. chih Yang, A. Dasdan, R.-L. Hsiao, and D.S. Parker, "Map- Reduce-Merge: Simplified Relational Data Processing on Large Clusters," Proc. ACM SIGMOD Int'l Conf. Management of Data, 2007.
7. M. Coates, R. Castro, R. Nowak, M. Gadhiok, R. King, and Y. Tsang, "Maximum Likelihood Network Topology Identification from Edge-Based Unicast Measurements," SIGMETRICS Performance Evaluation Rev., vol. 30, no. 1, pp. 11-20, 2002. (Pubitemid 35009505)
8. R. Davoli, "VDE: Virtual Distributed Ethernet," Proc. Testbeds and Research Infrastructures for the Development of Networks and Communities, Int'l Conf., pp. 213-220, 2005. (Pubitemid 43727712)
9. J. Dean and S. Ghemawat, "MapReduce: Simplified Data Processing on Large Clusters," Proc. Sixth Conf. Symp. Opearting Systems Design and Implementation (OSDI '04), p. 10, 2004.
10. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G.B. Berriman, J. Good, A. Laity, J.C. Jacob, and D.S. Katz, "Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems," Scientific Programming, vol. 13, no. 3, pp. 219-237, 2005. (Pubitemid 43022937)
11. T. Dornemann, E. Juhnke, and B. Freisleben, "On-Demand Resource Provisioning for BPEL Workflows Using Amazon's Elastic Compute Cloud," Proc. Ninth IEEE/ACM Int'l Symp. Cluster Computing and the Grid (CCGRID '09), pp. 140-147, 2009.
12. I. Foster and C. Kesselman, "Globus: A Metacomputing Infrastructure Toolkit," Int'l J. Supercomputer Applications, vol. 11, no. 2, pp. 115-128, 1997. (Pubitemid 127673968)
13. J. Frey, T. Tannenbaum, M. Livny, I. Foster, and S. Tuecke, "Condor-G: A Computation Management Agent for Multi- Institutional Grids," Cluster Computing, vol. 5, no. 3, pp. 237-246, 2002.
14. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks," Proc. Second ACM SIGOPS/EuroSys European Conf. Computer Systems (EuroSys '07), pp. 59-72, 2007. (Pubitemid 47281575)
15. A. Kivity, "Kvm: The Linux Virtual Machine Monitor," Proc. Ottawa Linux Symp. (OLS '07), pp. 225-230, July 2007.
16. D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youseff, and D. Zagorodnov, "Eucalyptus: A Technical Report on an Elastic Utility Computing Architecture Linking Your Programs to Useful Systems," technical report, Univ. of California, Santa Barbara, 2008.
17. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins, "Pig Latin: A Not-So-Foreign Language for Data Processing," Proc. ACM SIGMOD Int'l Conf. Management of Data, pp. 1099-1110, 2008.
18. O. O'Malley and A.C. Murthy, "Winning a 60 Second Dash with a Yellow Elephant," technical report, Yahoo!, 2009.
19. R. Pike, S. Dorward, R. Griesemer, and S. Quinlan, "Interpreting the Data: Parallel Analysis with Sawzall," Scientific Programming, vol. 13, no. 4, pp. 277-298, 2005. (Pubitemid 43063583)
20. I. Raicu, I. Foster, and Y. Zhao, "Many-Task Computing for Grids and Supercomputers," Proc. Workshop Many-Task Computing on Grids and Supercomputers, pp. 1-11, Nov. 2008.
21. I. Raicu, Y. Zhao, C. Dumitrescu, I. Foster, and M. Wilde, "Falkon: A Fast and Light-Weight TasK ExecutiON Framework," Proc. ACM/IEEE Conf. Supercomputing (SC '07), pp. 1-12, 2007.
22. L. Ramakrishnan, C. Koelbel, Y.-S. Kee, R. Wolski, D. Nurmi, D. Gannon, G. Obertelli, A. YarKhan, A. Mandal, T.M. Huang, K. Thyagaraja, and D. Zagorodnov, "VGrADS: Enabling e-Science Workflows on Grids and Clouds with Fault Tolerance," Proc. Conf. High Performance Computing Networking, Storage and Analysis (SC '09), pp. 1-12, 2009.
23. R. Russell, "Virtio: Towards a De-Facto Standard for Virtual I/O Devices," SIGOPS Operating System Rev., vol. 42, no. 5, pp. 95-103, 2008.
24. M. Stillger, G.M. Lohman, V. Markl, and M. Kandil, "LEO-DB2's LEarning Optimizer," Proc. 27th Int'l Conf. Very Large Data Bases (VLDB '01), pp. 19-28, 2001.
25. The Apache Software Foundation "Welcome to Hadoop!" http:// hadoop.apache.org/, 2009.
26. G. von Laszewski, M. Hategan, and D. Kodeboyina, Workflows for e-Science Scientific Workflows for Grids. Springer, 2007.
27. D. Warneke and O. Kao, "Nephele: Efficient Parallel Data Processing in the Cloud," Proc. Second Workshop Many-Task Computing on Grids and Supercomputers (MTAGS '09), pp. 1-10, 2009.
28. D. Wentzlaff, C. Gruenwald III, N. Beckmann, K. Modzelewski, A. Belay, L. Youseff, J. Miller, and A. Agarwal, "An Operating System for Multicore and Clouds: Mechanisms and Implementation," Proc. ACM Symp. Cloud Computing (SoCC '10), pp. 3-14, 2010.
29. T. White, Hadoop: The Definitive Guide. O'Reilly Media, 2009.
30. Y. Zhao, M. Hategan, B. Clifford, I. Foster, G. von Laszewski, V. Nefedova, I. Raicu, T. Stef-Praun, and M. Wilde, "Swift: Fast, Reliable, Loosely Coupled Parallel Computation," Proc. Services, '07 IEEE Congress on, pp. 199-206, July 2007.