Visual, log-based causal tracing for performance debugging of MapReduce systems

Proceedings - International Conference on Distributed Computing Systems

Volumn , Issue , 2010, Pages 795-806

  Tan, Jiaqi ^a   Kavulya, Soila ^b   Gandhi, Rajeev ^b   Narasimhan, Priya ^b  

^a DSO NATIONAL LABORATORIES   (Singapore)

^b CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ANALYSIS TECHNIQUES; DEBUGGING TOOLS; MAP-REDUCE; NON-INTRUSIVE; PERFORMANCE DEBUGGING; PERFORMANCE PROBLEMS; PROGRAM BEHAVIOR; REAL-WORLD; REAL-WORLD PROBLEM; STATE-MACHINE; STRUCTURE AND PERFORMANCE;

DISTRIBUTED COMPUTER SYSTEMS; VISUALIZATION;

PROGRAM DEBUGGING;

EID: 77955941295     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICDCS.2010.63     Document Type: Conference Paper

References (41)

1. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," in USENIX Symposium on Operating Systems Design and Implementation, San Francisco, CA, Dec 2004, pp. 137-150.
2. Apache Software Foundation, "Hadoop," 2007, http://hadoop.apache.org/core.
3. -, "Powered by Hadoop," 2009, http://wiki.apache.org/hadoop/ PoweredBy.
4. Yahoo! Developer Network, "Yahoo! launches world's largest hadoop production application (hadoop and distributed computing at yahoo!)," Feb 2008, http://developer.yahoo.net/blogs/hadoop/2008/02/yahoo-worlds-largest- production-hadoop.html.
5. Facebook, "Engineering @ facebook's notes: Hadoop," Jun 2008, http://www.facebook.com/note.php?note id=16121578919.
6. A. Murthy, "Hadoop MapReduce - Tuning and Debugging," 2008, http://tinyurl.com/c9eau2.
7. M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen, "Performance debugging for distributed system of black boxes," in ACM Symposium on Operating Systems Principles, Bolton Landing, NY, Oct 2003, pp. 74-89.
8. R. Fonseca, G. Porter, R. Katz, S. Shenker, and I. Stoica, "X-Trace: A pervasive network tracing framework," in USENIX Symposium on Networked Systems Design and Implementation, Cambridge, MA, Apr 2007.
9. E. Thereska, B. Salmon, J. Strunk, M. Wachs, M. Abd-El-Malek, J. Lopez, and G. Ganger, "Stardust: tracking activity in a distributed storage system," SIGMETRICS Perform. Eval. Rev., vol. 34, no. 1, pp. 3-14, 2006.
10. J. Tan, X. Pan, S. Kavulya, R. Gandhi, and P. Narasimhan, "Mochi: Visual Log-Analysis Based Tools for Debugging Hadoop," in USENIX Workshop on Hot Topics in Cloud Computing (HotCloud), San Diego, CA, May 2009.
11. J. Tan, X. Pan, S. Kavulya, R. Gandhi, and P. Narasimhan, "SALSA: Analyzing Logs as State Machines," in USENIX Workshop on Analysis of System Logs (WASL), San Diego, CA, Dec 2008.
12. Apache Software Foundation, "Pig," 2007, http://hadoop.apache. org/pig.
13. S. Ghemawat, H. Gobioff, and S. Leung, "The Google file system." in ACM Symposium on Operating Systems Principles, Lake George, NY, Oct 2003, pp. 29-43.
14. Apache Software Foundation, "Hadoop Users' Mailing List," 2008, http://mail-archives.apache.org/mod-mbox/hadoop-core-user.
15. J. Tan, "Log-based Approaches to Characterizing and Diagnosing MapReduce Systems," Carnegie Mellon University Master's Thesis, Tech. Rep. CMU-CS-09-143, Jul 2009.
16. J. Tan and C. Douglas, "Add ReduceID to Shuffle ClientTrace," 2009, http://issues.apache.org/jira/browse/MAPREDUCE-479.
17. J. Boulon, A. Konwinski, R. Qi, A. Rabkin, E. Yang, and M. Yang, "Chukwa: A Large-scale Monitoring System," in Cloud Computing and Its Applications, Chicago, IL, Oct 2008.
18. J. Tan, X. Pan, S. Kavulya, E. Marinelli, R. Gandhi, and P. Narasimhan, "Kahuna: Problem Diagnosis for MapReduce-based Cloud Computing Environments," in 12th IEEE/IFIP Network Operations and Management Symposium (NOMS), Osaka, Japan, Apr 2010.
19. Amazon Web Services LLC, "Amazon Elastic Compute Cloud," 2009, http://aws.amazon.com/ec2/.
20. U. Kang, C. Tsourakakis, A. P. Appel, C. Faloutsos, and J. Leskovec., "Radius plots for mining tera-byte scale graphs: Algorithms, patterns, and observations," SIAM International Conference on Data Mining, 2010.
21. M. Y. Chen, E. Kiciman, E. Fratkin, A. Fox, and E. Brewer, "Pinpoint: Problem determination in large, dynamic internet services," in IEEE Conference on Dependable Systems and Networks, Bethesda, MD, Jun 2002.
22. P. Barham, A. Donnelly, R. Isaacs, and R. Mortier, "Using Magpie for request extraction and workload modelling," in USENIX Symposium on Operating Systems Design and Implementation, San Francisco, CA, Dec 2004.
23. E. Kiciman and A. Fox, "Detecting application-level failures in component-based internet services," IEEE Trans. on Neural Networks: Special Issue on Adaptive Learning Systems in Communication Networks, vol. 16, no. 5, pp. 1027-1041, Sep 2005.
24. E. Koskinen and J. Jannotti, "Borderpatrol: isolating events for black-box tracing," in Eurosys '08: Proceedings of the 3rd ACM SIGOPS/EuroSys European Conference on Computer Systems 2008. New York, NY, USA: ACM, 2008, pp. 191-203.
25. I. Cohen, S. Zhang, M. Goldszmidt, J. Symons, T. Kelly, and A. Fox, "Capturing, indexing, clustering, and retrieving system history," in ACM Symposium on Operating Systems Principles, Brighton, United Kingdom, Oct 2005, pp. 105-118.
26. S. Shende and A. Malony, "The Tau Parallel Performance System," International Journal of High Performance Computing Applications, vol. 20, no. 2, pp. 287-331, 2006.
27. B. Miller, M. Callaghan, J. Cargille, J. Hollingsworth, R. Bruce Irvin, K. Karavanic, K. Kunchithapadam, and T. Newhall, "The Paradyn Parallel Performance Measurement Tool," IEEE Computer, vol. 28, pp. 37-46, 1995.
28. L. Mariani and F. Pastore, "Automated Identification of Failure Causes in System Logs," in International Symposium on Software Reliability Engineering (ISSRE), Seattle, WA, Nov 2008.
29. G. Jiang, H. Chen, U. Ungureanu, and K. Yoshihira, "Multi-resolution Abnormal Trace Detection Using Varied-length N-grams and Automata," in International Conference on Autonomic Computing (ICAC), Seattle, WA, Jun 2005.
30. D. Lorenzoli, L. Mariani, and M. Pezze, "Inferring State-based Behavior Models," in Workshop on Dynamic Analysis (WODA), Shanghai, China, May 2006.
31. W. Xu, L. Huang, A. Fox, D. Patterson, and M. Jordan, "Detecting large-scale system problems by mining console logs," in Symposium on Operating Systems Principles (SOSP), Big Sky, MT, Oct 2009.
32. J. Lou, Q. Fu, Y. Wang, and J. Li, "Mining dependency in distributed systems through unstructured log analysis," in 2nd USENIX Workshop on Analysis of System Logs (WASL), Big Sky, MT, Oct 2009.
33. X. Pan, J. Tan, S. Kavulya, R. Gandhi, and P. Narasimhan, "Ganesha: Black-Box Diagnosis of MapReduce Systems," in Workshop on Hot Topics in Measurement & Modeling of Computer Systems (HotMetrics), Seattle, WA, Jun 2009.
34. -, "The Blind Men and the Elephant: Piecing Together Hadoop for Diagnosis," in IEEE International Symposium on Software Reliability Engineering (ISSRE), Industrial Track, Mysuru, India, Nov 2009.
35. S. Kavulya, J. Tan, R. Gandhi, and P. Narasimhan, "An Analysis of Traces from a Production MapReduce Cluster," in IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing (CCGrid), Melbourne, Victoria, Australia, May 2010.
36. Yahoo! Inc., "Yahoo! reaches for the stars with M45 supercomputing project," 2007, http://research.yahoo.com/node/1884.
37. G. Cretu-Ciocarlie, M. Budiu, and M. Goldszmidt, "Hunting for problems with artemis," in USENIX Workshop on Analysis of System Logs, 2008.
38. P. MaLachlan, T. Munzner, E. Koutsofios, and S. North, "LiveRAC - Interactive Visual Exploration of System Management Time-Series Data," in SIGCHI Conference on Human Factors in Computing Systems (CHI'08), Florence, Italy, Apr 2008.
39. Q. Liao, A. Blaich, A. Striegel, and D. Thain, "ENaVis: Enterprise Network Activities Visualization," in Large Installation System Administration Conference (LISA), San Diego, CA, Nov 2008.
40. J. Tan, "SALSA state-machine extraction from Hadoop logs," 2009, http://issues.apache.org/jira/browse/CHUKWA-94.
41. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks," in European Conference on Computer Systems (EuroSys), Lisbon, Portugal, Mar 2007.