MapReduce workload modeling with statistical approach

Journal of Grid Computing

Volumn 10, Issue 2, 2012, Pages 279-310

  Yang, Hailong ^a   Luan, Zhongzhi ^a   Li, Wenjun ^a   Qian, Depei ^a  

^a BEIHANG UNIVERSITY   (China)

Author keywords

Cloud computing; Data intensive computing; MapReduce; Performance prediction; Statistical analysis; Workload characterization

Indexed keywords

APPLICATION DEVELOPMENT; COMPREHENSIVE ANALYSIS; CONFIGURATION PARAMETERS; CONFIGURATION SYSTEM; CORE BUSINESS; DATA-INTENSIVE COMPUTATION; DATA-INTENSIVE COMPUTING; INDUSTRIAL ORGANIZATION; MAP-REDUCE; OPEN SOURCE PROJECTS; PERFORMANCE MODELING; PERFORMANCE OPTIMIZATIONS; PERFORMANCE PREDICTION; POOR PERFORMANCE; REGRESSION MODEL; STATISTICAL APPROACH; WORKLOAD CHARACTERISTICS; WORKLOAD CHARACTERIZATION; WORKLOAD MODELING;

CLOUD COMPUTING; CLUSTER ANALYSIS; PRINCIPAL COMPONENT ANALYSIS; REGRESSION ANALYSIS; STATISTICAL METHODS;

FORECASTING;

EID: 84860890249     PISSN: 15707873     EISSN: None     Source Type: Journal    
DOI: 10.1007/s10723-011-9201-4     Document Type: Article

References (37)

1. Armbrust, M., Fox, A., Griffith, R., Joseph, A.D., Katz, R.H., Konwinski, A., Lee, G., Patterson, D.A., Rabkin, A., Stoica, I., Zaharia, M.: Above the clouds: a Berkeley view of cloud computing. Technical Report No. UCB/EECS-2009-28, Electrical Engineering and Computer Sciences, University of California at Berkeley (2009)
2. Rimal, B., Jukan, A., Katsaros, D., Goeleven, Y.: Architectural requirements for cloud computing systems: an enterprise cloud approach. Journal ofGrid Computing 9(1), 3-26 (2011)
3. Thusoo, A., Sarma, J.S., Jain, N., Zheng, S., Chakka, P., Ning, Z., Antony, S., Hao, L., Murthy, R.: Hive-a petabyte scale data warehouse using Hadoop. In: Proceedings of IEEE 26th International Conference on Data Engineering (ICDE) (2010)
4. Yahoo Developer Network: Yahoo! launches world's largest Hadoop Production Application. Available online at http://developer.yahoo.com/blogs/hadoop/ posts/2008/02/yahoo-worlds-largest-production-hadoop/ (2008). Accessed on Nov. 2011
5. Pallis, G., Katsifodimos, A., Dikaiakos, M.: Searching for software on the EGEE infrastructure. Journal of Grid Computing 8(2), 281-304 (2010)
6. Thain, D., Moretti, C., Hemmes, J.: Chirp: a practical global filesystem for cluster and Grid computing. Journal of Grid Computing 7(1), 51-72 (2009)
7. McClatchey, R., Anjum, A., Stockinger, H., Ali, A., Willers, I., Thomas, M.: Data Intensive and Network Aware (DIANA) Grid scheduling. Journal of Grid Computing 5(1), 43-64 (2007)
8. Yu, C., Marinescu, D.: Algorithms for divisible load scheduling of data-intensive applications. Journal of Grid Computing 8(1), 133-155 (2010)
9. Cai, Z., Kumar, V., Schwan, K.: IQ-Paths: predictably high performance data streams across dynamic network overlays. Journal of Grid Computing 5(2), 129-150 (2007)
10. Zaharia, M., Konwinski, A., Joseph, A.D., Randy, H., Katz, I.S.: Improving MapReduce performance in heterogeneous environments. In: Proceedings of the USENIX Conference on Operating Systems Design and Implementation (OSDI) (2008)
11. Zaharia, M., Borthakur, D., Sarma, J.S., Elmeleegy, K., Shenker, S., Stoica, I.: Job scheduling for multiuser MapReduce clusters. Technical Report No. UCB/ EECS-2009-55, Electrical Engineering and Computer Sciences, University of California at Berkeley (2009)
12. Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., Goldberg, A.: Quincy: fair scheduling for distributed computing clusters. In: Proceedings of the ACM SIGOPS 22nd Symposium on Operating systems Principles (SOSP) (2009)
13. Condie, T., Conway, N., Alvaro, P., Hellerstein, J.M., Elmeleegy, K., Sears, R.: MapReduce online. In: Proceedings of the USENIX Conference on Networked Systems Design and Implementation (NSDI) (2010)
14. Jiang, D., Ooi, B.C., Shi, L., Wu, S.: The performance of MapReduce: an in-depth study. Proc. VLDB Endow. 3(1-2), 472-483 (2010)
15. Chen, Y., Ganapathi, A.S., Fox, A., Katz, R.H., Patterson, D.A.: Statistical workloads for energy efficient MapReduce. Technical Report No. UCB/EECS-2010-6, Electrical Engineering and Computer Sciences, University of California at Berkeley (2010)
16. Apache Hadoop: Gridmix. Available online at http:// hadoop.apache.org/mapreduce/docs/current/gridmix. html (2010). Accessed on Nov. 2011
17. Apache Hive: Hive performance benchmarks. Available online at https://issues.apache.org/jira/browse/ HIVE-396 (2010). Accessed on Nov. 2011
18. Shengsheng, H., Jie, H., Jinquan, D., Tao, X., Bo, H.: The HiBench benchmark suite: characterization of the MapReduce-based data analysis. In: Proceedings of IEEE 26th International Conference on Data EngineeringWorkshops (ICDEW) (2010)
19. Apache Hadoop MapReduce: Mumak: Map-Reduce simulator. Available online at https://issues.apache.org/ jira/browse/MAPREDUCE-728 (2009). Accessed on Nov. 2011
20. Wang, G., Butt, A.R., Pandey, P., Gupta, K.: Using realistic simulation for performance analysis of mapreduce setups. In: Proceedings of the ACM Workshop on Large-Scale System and Application Performance (2009)
21. Hammoud, S., Maozhen, L., Yang, L., Alham, N.K., Zelong, L.: MRSim: a discrete event based MapReduce simulator. In: Proceedings of International Conference on Fuzzy Systems and Knowledge Discovery (FSKD) (2010)
22. Babu, S.: Towards automatic optimization of MapReduce programs. In: Proceedings of the ACM Symposium on Cloud computing (SoCC) (2010)
23. Koehler, M., Kaniovskyi, Y., Benkner, S.: An adaptive framework for the execution of data-intensiveMapReduce applications in the cloud. In: Proceedings of IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum (IPDPSW) (2011)
24. Jahani, E., Cafarella, M.J., Ré, C.: Automatic optimization for MapReduce programs. Proc. VLDB Endow. 4(6), 385-396 (2011)
25. Rizvandi, N.B., Zomaya, A.Y., Boloori, A.J., Taheri, J.: Preliminary results: modeling relation between total execution time of MapReduce applications and number of mappers/reducers. Technical Report No. 679, Center for Distributed and High Performance Computing, School of Information Technologies, University of Sydney (2011)
26. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107-113 (2008)
27. Shvachko, K., Hairong, K., Radia, S., Chansler, R.: The Hadoop distributed file system. In: Proceedings of IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST) (2010)
28. Apache Nutch: Nutch homepage. Available online at http://nutch.apache.org/ (2010). Accessed on Nov. 2011
29. SmartFrog Organization: SmartFrog homepage. Available online at http://wiki.smartfrog.org/wiki/display/ sf/SmartFrog+Home (2007). Accessed on Nov. 2011
30. Apache Mahout: Mahout homepage. Available online at http://mahout.apache.org/ (2010). Accessed on Nov. 2011
31. Apache Hadoop: Hadoop Wiki Power-By. Available online at http://wiki.apache.org/hadoop/PoweredBy (2010). Accessed on Nov. 2011
32. Farnham, I.M., Johannesson, K.H., Singh, A.K., Hodge, V.F., Stetzenbach, K.J.: Factor analytical approaches for evaluating groundwater trace element chemistry data. Anal. Chim. Acta 490(1-2), 123-138 (2003)
33. Manly, B.F.: Multivariate Statistical methods: A Primer. Chapman & Hall, Ltd., London (1986)
34. Vapnik, V., Golowich, S.E., Smola, A.J.: Support vector method for function approximation, regression estimation and signal processing. In: Proceedings of Conference on Neural Information Processing Systems (NIPS) (1996)
35. Smola, A.J., Schölkopf, B.:Atutorial on support vector regression. Stat. Comput. 14(3), 199-222 (2004)
36. Inspur Company: Inspur homepage. Available online at http://en.inspur.com/ (2006). Accessed on Nov. 2011
37. Beihang University NICC: The network information and computing center. Available online at http://nic. buaa.edu.cn/ (2010). Accessed on Nov. 2011