An analysis of traces from a production MapReduce cluster

CCGrid 2010 - 10th IEEE/ACM International Conference on Cluster, Cloud, and Grid Computing

Volumn , Issue , 2010, Pages 94-103

  Kavulya, Soila ^a   Tany, Jiaqi ^b   Gandhi, Rajeev ^a   Narasimhan, Priya ^a  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b DSO NATIONAL LABORATORIES   (Singapore)

Author keywords

[No Author keywords available]

Indexed keywords

ACADEMIC RESEARCH; CLOUD COMPUTING; DATA SETS; DATA-INTENSIVE APPLICATION; HISTORICAL DATA; IMPACT PERFORMANCE; INSTANCE-BASED LEARNING TECHNIQUES; JOB COMPLETION; PARALLEL PROCESSING; PERFORMANCE PROBLEMS; PROGRAMMING PARADIGMS; RESOURCE UTILIZATIONS; RUNNING-IN; SCHEDULING DECISIONS; SERVICE PROVIDER; TEMPORAL LOCALITY;

GRID COMPUTING; LEARNING ALGORITHMS;

CLUSTER COMPUTING;

EID: 77954901315     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CCGRID.2010.112     Document Type: Conference Paper

References (30)

1. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters." Communications of the ACM, vol. 51, pp. 107-113, 2008.
2. Hadoop, "Powered by Hadoop, " http://wiki.apache.org/hadoop/ PoweredBy.
3. Yahoo!, "M45 supercomputing project, " 2009, http://research.yahoo.com/node/1884.
4. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg, "Quincy: fair scheduling for distributed computing clusters, " in ACM Symposium on Operating Systems Principles, Big Sky, Montana, Oct. 2009, pp. 261-276.
5. Yahoo!, "Hadoop capacity scheduler, " 2008, https://issues.apache.org/jira/browse/HADOOP-3445.
6. J. Tan, X. Pan, S. Kavulya, R. Gandhi, and P. Narasimhan, "SALSA: Analyzing Logs as State Machines, " in USENIX Workshop on Analysis of System Logs, San Diego, CA, Dec. 2008.
7. X. Pan, J. Tan, S. Kavulya, R. Gandhi, and P. Narasimhan, "Ganesha: Black-Box Diagnosis of MapReduce Systems, " in Workshop on Hot Topics in Measurement and Modeling of Computer Systems (HotMetrics), Seattle, WA, Jun. 2009.
8. -, "Blind Men and the Elephant: Piecing together Hadoop for diagnosis, " in International Symposium on Software Reliability Engineering (ISSRE), Mysuru, India, Nov. 2009.
9. T. A. S. Foundation, "Hadoop, " 2007, http://hadoop.apache.org/ core.
10. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters, " in USENIX Symposium on Operating Systems Design and Implementation, San Francisco, CA, Dec. 2004, pp. 137-150.
11. S. Ghemawat, H. Gobioff, and S. Leung, "The Google File System." in ACM Symposium on Operating Systems Principles, Lake George, NY, Oct 2003, pp. 29-43.
12. B. Ripley, "R's MASS statistical package, " 2009, http://cran.r-project.org/web/packages/VR/index.html.
13. J. Tan, X. Pan, S. Kavulya, R. Gandhi, and P. Narasimhan, "Mochi: Visual Log-Analysis Based Tools for Debugging Hadoop, " in USENIX Workshop on Hot Topics in Cloud Computing (HotCloud), San Diego, CA, Jun. 2009.
14. T. A. S. Foundation, "The Map/Reduce Tutorial, " 2008, http://hadoop.apache.org/common/docs/current/mapredtutorial.html.
15. C. W. Gini, "Variability and mutability, contribution to the study of statistical distributions and relations, " Studi Economico-Giuridici della R. Universita de Cagliari, 1912, reviewed in: Light, R.J., Margolin, B.H.: An Analysis of Variance for Categorical Data. J. American Statistical Association, Vol. 66 pp. 534-544 (1971).
16. W. Smith, "Prediction services for distributed computing, " in International Parallel and Distributed Processing Symposium, Long Beach, CA, March 2007, pp. 1-10.
17. N. H. Kapadia, J. A. Fortes, and C. E. Brodley, "Predictive applicationperformance modeling in a computational grid environment, " in International Symposium on High-Performance Distributed Computing. Redondo Beach, CA: IEEE Computer Society, Aug. 1999, p. 6.
18. A. B. Downey and D. G. Feitelson, "The elusive goal of workload characterization, " SIGMETRICS Conference on Measurement and Modeling of Computer Systems, vol. 26, no. 4, pp. 14-29, 1999.
19. R. D. Wilson and T. R. Martinez, "Improved heterogeneous distance functions, " Journal of Artificial Intelligence Research, vol. 6, pp. 1-34, 1997.
20. S. S. Chris Atkeson, Andrew Moore, "Locally weighted learning, " Artificial Intelligence Review, vol. 11, pp. 11-73, Apr. 1997.
21. G. W. Stewart, "Collinearity and least squares regression, " Statistical Science, vol. 2, no. 1, pp. 68-84, 1987. [Online]. Available: http://www.jstor.org/stable/2245615.
22. S. Krishnaswamy, S. W. Loke, and S. W. Loke, "Estimating computation times of data-intensive applications, " IEEE Distributed Systems Online, vol. 5, no. 4, 2004.
23. U. Lublin and D. G. Feitelson, "The workload on parallel supercomputers: modeling the characteristics of rigid jobs, " Journal of Parallel and Distributed Computing, vol. 63, no. 11, pp. 1105-1122, Nov. 2003.
24. H. Li, D. L. Groep, and L. Wolters, "Workload characteristics of a multi-cluster supercomputer, " in Job Scheduling Strategies for Parallel Processing, New York, NY, Jun. 2004, pp. 176-193.
25. B. Schroeder and G. Gibson, "A large-scale study of failures in highperformance computing systems, " in Dependendable Systems and Networks, Philadelphia, PA, Jun. 2006.
26. J. Gray, "A census of tandem system availability between 1985 and 1990, " in IEEE Transactions on Reliability, Oct. 1990.
27. D. Oppenheimer, A. Ganapathi, and D. Patterson, "Why do Internet services fail, and what can be done about it?" in USENIX Symposium on Internet Technologies and Systems, Mar. 2003.
28. R. Iyer, D. Rossetti, and M. Hsueh, "Measurement and modeling of computer reliability as affected by system activity, " in ACM Transactions on Computer Systems, Aug. 1986.
29. R. Sahoo, M. Squillante, A. Sivasubramaniam, and Y. Zhang, "Failure data analysis of a large-scale heterogeneous server environment, " in Dependendable Systems and Networks, Florence, Italy, Jun. 2004.
30. G. Wang, A. R. Butt, P. Pandey, and K. Gupta, "A simulation approach to evaluating design decisions in MapReduce setup, " in International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems, London, UK, Sep. 2009.