Workload characterization on a production Hadoop cluster: A case study on Taobao

Proceedings - 2012 IEEE International Symposium on Workload Characterization, IISWC 2012

Volumn , Issue , 2012, Pages 3-13

  Ren, Zujie ^a   Xu, Xianghua ^a   Wan, Jian ^a   Shi, Weisong ^b   Zhou, Min ^c  

^a HANGZHOU DIANZI UNIVERSITY   (China)

^b Wayne State University   (United States)

^c Taobao Inc   (China)

Author keywords

Hadoop; MapReduce; Workload characterization

Indexed keywords

COMPUTING PARADIGM; DATA PLATFORM; GAIN INSIGHT; HADOOP; JOB CHARACTERISTICS; LARGE CLUSTERS; LARGE-SCALE DATASETS; LARGE-SCALE PRODUCTION; MAP-REDUCE; PERFORMANCE OPTIMIZATIONS; PRODUCTION ENVIRONMENTS; SCIENTIFIC COMPUTATION; SOCIAL NETWORKS; SYSTEM THROUGHPUT; WEB SEARCHES; WORKLOAD CHARACTERIZATION;

ELECTRONIC COMMERCE; ENGINEERS; JOB ANALYSIS; WEBSITES;

CHARACTERIZATION;

EID: 84873482265     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IISWC.2012.6402895     Document Type: Conference Paper

References (33)

1. J. Dean and S. Ghemawat, "Mapreduce: Simplified data processing on large clusters," in OSDI, 2004, pp. 137-150.
2. T. White, Hadoop - The Definitive Guide. O'Reilly, 2009.
3. S. Ghemawat, H. Gobioff, and S.-T. Leung, "The google file system," in ACM SOSP, vol. 37, no. 5, 2003, pp. 29-43.
4. Ganglia. [Online]. Available: ganglia.sourceforge.net
5. Y. Chen, S. Alspaugh, D. Borthakur, and R. H. Katz, "Energy efficiency for large-scale mapreduce workloads with significant interactive analysis," in EuroSys. ACM, 2012, pp. 43-56.
6. J. D. C. Little, "A prof for the queuing formula L=hW," Operations Research, vol. 9, 1961.
7. X. Liu, J. Han, Y. Zhong, C. Han, and X. He, "Implementing webGIS on hadoop: A case study of improving small file I/O performance on HDFS," in CLUSTER, 2009, pp. 1-8.
8. G. Mackey, S. Sehrish, and J. Wang, "Improving metadata management for small files in HDFS," in CLUSTER, 2009, pp. 1-4.
9. S. Kavulya, J. Tan, R. Gandhi, and P. Narasimhan, "An analysis of traces from a production mapreduce cluster," in CCGRID, 2010, pp. 94-103.
10. M. Zaharia, D. Borthakur, J. S. Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling," in EuroSys, 2010, pp. 265-278.
11. C. He, Y. Lu, and D. Swanson, "Matchmaking: A new mapreduce scheduling technique," in CloudCom, 2011, pp. 40-47.
12. E. Andersen, The Statistical Analysis of Categorical Data. Springer, Berlin, 1992.
13. S. Krishnamoorthy and A. Choudhary, "A scalable distributed shared memory architecture," JPDC, vol. 22, no. 3, pp. 547-554, 1994.
14. B. Schroeder and G. A. Gibson, "A large-scale study of failures in highperformance computing systems," in DSN, 2006, pp. 249-258.
15. R. K. Sahoo, A. Sivasubramaniam, M. S. Squillante, and Y. Zhang, "Failure data analysis of a large-scale heterogeneous server environment," in DSN, 2004, p. 772.
16. D. Liu and Y.-H. Lee, "Pfair scheduling of periodic tasks with allocation constraints on multiple processors," in IPDPS, 2004.
17. J. Lee, A. Easwaran, and I. Shin, "LLF schedulability analysis on multiprocessor platforms," in IEEE Real-Time Systems Symposium, 2010, pp. 25-36.
18. D. Ersoz, M. S. Yousif, and C. R. Das, "Characterizing network traffic in a cluster-based, multi-tier data center," in ICDCS, 2007, p. 59.
19. V. Paxson, "Empirically derived analytic models of wide-area TCP connections," IEEE/ACM Trans. Netw, vol. 2, no. 4, pp. 316-336, 1994.
20. L. N. Bairavasundaram, A. C. Arpaci-Dusseau, R. H. Arpaci-Dusseau, G. R. Goodson, and B. Schroeder, "An analysis of data corruption in the storage stack," ACM Transactions on Storage, vol. 4, no. 3, pp. 821-834, 2008.
21. E. Thereska, A. Donnelly, and D. Narayanan, "Sierra: practical powerproportionality for data center storage," in EuroSys, 2011, pp. 169-182.
22. F. Wang, Q. Xin, B. Hong, S. A. Brandt, E. L. Miller, D. D. E. Long, and T. T. Mclarty, "File system workload analysis for large scale scientific computing applications," in MSST, 2004, pp. 139-152.
23. E. Hernández-Orallo and J. Vila-Carbó, "Web server performance analysis using histogram workload models," Computer Networks, vol. 53, no. 15, pp. 2727-2739, 2009.
24. W. Shi, Y. Wright, E. Collins, and V. Karamcheti, "Workload characterization of a personalized web site and its implications for dynamic content caching," in WCW, 2002, pp. 1-16.
25. A. Iamnitchi, S. Doraimani, and G. Garzoglio, "Workload characterization in a high-energy data grid and impact on resource management," Cluster Computing, vol. 12, no. 2, pp. 153-173, 2009.
26. K. Christodoulopoulos, V. Gkamas, and E. A. Varvarigos, "Statistical analysis and modeling of jobs in a grid environment," J. Grid Comput, vol. 6, no. 1, 2008.
27. E. Medernach, "Workload analysis of a cluster in a grid environment," in Job Scheduling Strategies for Parallel Processing, 2005, pp. 36-61.
28. B. Song, C. Ernemann, and R. Yahyapour, "User group-based workload analysis and modelling," in CCGRID, 2005, pp. 953-961.
29. M. Zaharia, D. Borthakur, J. S. Sarma, S. Shenker, and I. Stoica, "Job scheduling for multi-user mapreduce clusters," Univ. of Calif., Berkeley, CA, Technical Report No. UCB/EECS-2009-55, Apr. 2009.
30. A. K. Mishra, J. L. Hellerstein, W. Cirne, and C. R. Das, "Towards characterizing cloud backend workloads: insights from google compute clusters," SIGMETRICS Performance Evaluation Review, vol. 37, no. 4, pp. 34-41, 2010.
31. B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears, "Benchmarking cloud serving systems with YCSB," in SoCC, J. M. Hellerstein, S. Chaudhuri, and M. Rosenblum, Eds., 2010, pp. 143-154.
32. Y. Chen, A. Ganapathi, R. Griffith, and R. H. Katz, "The case for evaluating mapreduce performance using workload suites," in MASCOTS, 2011, pp. 390-399.
33. Y. Chen, S. Alspaugh, and R. H. Katz, "Design insights for mapreduce from diverse production workloads," University of California, Berkeley, Tech. Rep. UCB/EECS-2012-17, 2012.