Kahuna: Problem diagnosis for MapReduce-based cloud computing environments

Proceedings of the 2010 IEEE/IFIP Network Operations and Management Symposium, NOMS 2010

Volumn , Issue , 2010, Pages 112-119

  Tan, Jiaqi ^a   Pan, Xinghao ^a   Marinelli, Eugene ^b   Kavulya, Soila ^b   Gandhi, Rajeev ^b   Narasimhan, Priya ^b  

^a DSO NATIONAL LABORATORIES   (Singapore)

^b CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BLACK BOXES; CLOUD COMPUTING ENVIRONMENTS; EXPERIMENTAL EVALUATION; FAULTY NODE; MAP-REDUCE; MAPREDUCE CLUSTERS; PERFORMANCE METRICS; PERFORMANCE PROBLEMS;

EID: 77957761115     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/NOMS.2010.5488446     Document Type: Conference Paper

References (29)

1. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," in OSDI, San Francisco, CA, Dec 2004, pp. 137-150.
2. Apache Software Foundation, "Hadoop," 2007, http://hadoop.apache. org/core.
3. -, "Powered by Hadoop," 2009, http://wiki.apache.org/hadoop/ PoweredBy.
4. I. Cohen, S. Zhang, M. Goldszmidt, J. Symons, T. Kelly, and A. Fox, "Capturing, indexing, clustering, and retrieving system history," in SOSP, Brighton, United Kingdom, Oct 2005, pp. 105-118.
5. M. Y. Chen, E. Kiciman, E. Fratkin, A. Fox, and E. Brewer, "Pinpoint: Problem determination in large, dynamic internet services," in DSN, Bethesda, MD, Jun 2002.
6. M. K. Aguilera, J. C. Mogul, J. L. Wiener, P. Reynolds, and A. Muthitacharoen, "Performance debugging for distributed system of black boxes," in SOSP, Bolton Landing, NY, Oct 2003, pp. 74-89.
7. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters," Communcations of the ACM, vol. 51, no. 1, pp. 107-113, Jan 2008.
8. X. Pan, J. Tan, S. Kavulya, R. Gandhi, and P. Narasimhan, "Ganesha: Black-Box Diagnosis of MapReduce Systems," in Workshop on Hot Topics in Measurement & Modeling of Computer Systems (HotMetrics), Seattle, WA, Jun 2009.
9. J. Tan, X. Pan, S. Kavulya, R. Gandhi, and P. Narasimhan, "SALSA: Analyzing Logs as State Machines," in USENIX WASL, San Diego, CA, Dec 2008.
10. S. Ghemawat, H. Gobioff, and S. Leung, "The Google file system." in SOSP, Lake George, NY, Oct 2003, pp. 29 - 43.
11. Apache Software Foundation, "Apache's JIRA issue tracker," 2006, https://issues.apache.org/jira.
12. Amazon Web Services LLC, "Amazon Elastic Compute Cloud," 2009, http://aws.amazon.com/ec2/.
13. Yahoo! Inc., "Yahoo! reaches for the stars with M45 supercomputing project," 2007, http://research.yahoo.com/node/1884.
14. S. Godard, "SYSSTAT," 2008, http://pagesperso-orange.fr/ sebastien. godard.
15. D. M. Endres and J. E. Schindelin, "A new metric for probability distributions," Information Theory, IEEE Transactions on, vol. 49, no. 7, pp. 1858-1860, 2003.
16. D. R. A. Dempster, N. Laird, "Maximum likelihood from incomplete data via the em algorithm," Journal of the Royal Statistical Society, vol. 39, pp. 1,38, 1977.
17. C. Olston and B. Reed and U. Srivastava and R. Kumar and A. Tomkins, "Pig Latin: A Not-So-Foreign Language for Data Processing," in SIGMOD, VancoBuver, C, Canada, Jun 2008.
18. P. Barham, A. Donnelly, R. Isaacs, and R. Mortier, "Using Magpie for request extraction and workload modelling," in OSDI, San Francisco, CA, Dec 2004.
19. M. Chen, A. X. Zheng, J. Lloyd, M. I. Jordan, and E. Brewer, "Failure diagnosis using decision trees," in ICAC, New York, NY, May 2004, pp. 36-43.
20. E. Kiciman and A. Fox, "Detecting application-level failures in component-based internet services," IEEE Trans. on Neural Networks: Special Issue on Adaptive Learning Systems in Communication Networks, vol. 16, no. 5, pp. 1027- 1041, Sep 2005.
21. P. Reynolds, C. Killian, J. Wiener, J. Mogul, M. Shah, and A. Vahdat, "Pip: Detecting the unexpected in distributed systems," in NSDI, San Jose, CA, May 2006.
22. R. Fonseca, G. Porter, R. Katz, S. Shenker, and I. Stoica, "X-Trace: A pervasive network tracing framework," in NSDI, Cambridge, MA, Apr 2007.
23. S. Pertet, R. Gandhi, and P. Narasimhan, "Fingerpointing Correlated Failures in Replicated Systems," in SysML, Cambridge, MA, Apr 2007.
24. M. Munawar, M. Jiang, and P. Ward, "Monitoring Multi-tier Clustered Systems with Invariant Metrics Relationships," in International Workshop on Software Engineering for Adaptive and Self-Managing Systems, Leipzig, Germany, May 2008.
25. G. Cretu-Ciocarlie, M. Budiu, and M. Goldszmidt, "Hunting for Problems with Artemis," in USENIX WASL, San Diego, CA, Dec 2008.
26. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks," in EuroSys, Lisbon, Portugal, Mar 2007.
27. H. Wang, J. Platt, Y. Chen, R. Zhang, and Y. Wang, "Automatic Misconfiguration Troubleshooting with PeerPressure," in OSDI, San Francisco, CA, Dec 2004.
28. W. Xu, L. Huang, A. Fox, D. Patterson, and M. Jordan, "Detecting large-scale system problems by mining console logs," in SOSP, Big Sky, MT, Oct 2009.
29. J. Tan, X. Pan, S. Kavulya, R. Gandhi, and P. Narasimhan, "Mochi: Visual Log-Analysis Based Tools for Debugging Hadoop," in USENIX HotCloud, San Diego, CA, May 2009.