Assisting developers of big data analytics applications when deploying on Hadoop clouds

Proceedings - International Conference on Software Engineering

Volumn , Issue , 2013, Pages 402-411

  Shang, Weiyi ^a   Jiang, Zhen Ming ^a   Hemmati, Hadi ^a   Adams, Brain ^b   Hassan, Ahmed E ^a   Martin, Patrick ^a  

^a QUEEN S UNIVERSITY   (Canada)

^b ÉCOLE POLYTECHNIQUE DE MONTRÉAL   (Canada)

Author keywords

Big Data Analytics Application; Cloud Computing; Hadoop; Log Analysis; Monitoring and Debugging

Indexed keywords

CLOUD ENVIRONMENTS; EXECUTION SEQUENCES; HADOOP; LARGE AMOUNTS OF DATA; LOG ANALYSIS; PARALLEL PROCESSING; RUN-TIME ANALYSIS; SOFTWARE APPLICATIONS;

CLOUD COMPUTING; SOFTWARE ENGINEERING;

APPLICATION PROGRAMS;

EID: 84886412175     PISSN: 02705257     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICSE.2013.6606586     Document Type: Conference Paper

References (26)

1. D. Fisher, R. DeLine, M. Czerwinski, and S. Drucker, "Interactions with big data analytics," interactions, vol. 19, no. 3, pp. 50-59, May 2012.
2. N. Wingfield, "Virtual product, real profits: Players spend on zynga's games, but quality turns some off," Wall Street Journal.
3. "Ebay is powered by hadoop," http://wiki.apache.org/hadoop/ PoweredBy.
4. T. White, Hadoop: The Definitive Guide. Oreilly & Associates Inc, 2009.
5. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: distributed data-parallel programs from sequential building blocks," SIGOPS Oper. Syst. Rev., vol. 41, no. 3, pp. 59-72, Mar. 2007.
6. S. Sorkin, "Large-scale, unstructured data retreival and analysis using splunk," Technical paper, Splunk Inc, 2009.
7. P. Mundkur, V. Tuulos, and J. Flatow, "Disco: a computing platform for large-scale data analytics," in Erlang '11: Proc. of the 10th ACM SIGPLAN workshop on Erlang. New York, NY, USA: ACM, 2011, pp. 84-89.
8. J. Dean and S. Ghemawat, "Mapreduce: simplified data processing on large clusters," Commun. ACM, vol. 51, no. 1, pp. 107-113, January 2008.
9. W. Shang, Z. M. Jiang, B. Adams, A. E. Hassan, M. Godfrey, M. Nasser, and P. Flora, "An Exploratory Study of the Evolution of Communicated Information about the Execution of Large Software Systems," in WCRE '11: Proceedings of the 18th Working Conference on Reverse Engineering, Lero, Limerick, Ireland, October 2011.
10. Z. M. Jiang, A. E. Hassan, G. Hamann, and P. Flora, "An automated approach for abstracting execution logs to execution events," J. Softw. Maint. Evol., vol. 20, no. 4, pp. 249-267, 2008.
11. L. Page, S. Brin, R. Motwani, and T. Winograd, "The pagerank citation ranking: Bringing order to the web." Stanford InfoLab, Technical Report 1999-66, November 1999.
12. Z. M. Jiang, A. E. Hassan, G. Hamann, and P. Flora, "Automatic identification of load testing problems," in ICSM '08: Proc. of 24th IEEE International Conference on Software Maintenance. Beijing, China: IEEE, 2008, pp. 307-316.
13. "America's most wanted-a metric to detect persistently faulty machines in hadoop," http://hadoopblog.blogspot.com/2010/06/americasmost- wanted-metric-to-detect.html.
14. D. Cotroneo, S. Orlando, and S. Russo, "Failure classification and analysis of the java virtual machine," in Proceedings of the 26th IEEE International Conference on Distributed Computing Systems, ser. ICDCS '06. Washington, DC, USA: IEEE Computer Society, 2006, pp. 17-.
15. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins, "Pig latin: a not-so-foreign language for data processing," in SIGMOD '08: Proc. of the 2008 ACM SIGMOD international conference on Management of data. New York, NY, USA: ACM, 2008, pp. 1099-1110.
16. A. F. Gates, O. Natkovich, S. Chopra, P. Kamath, S. M. Narayanamurthy, C. Olston, B. Reed, S. Srinivasan, and U. Srivastava, "Building a highlevel dataflow system on top of Map-Reduce: the Pig experience," Proc. VLDB Endow., vol. 2, no. 2, pp. 1414-1425, 2009.
17. "Amazon ec2," https://aws.amazon.com/ec2/.
18. M. Fischer, J. Oberleitner, H. Gall, and T. Gschwind, "System evolution tracking through execution trace analysis," in IWPC '05: Proceedings of the 13th International Workshop on Program Comprehension. Washington, DC, USA: IEEE Computer Society, 2005, pp. 237-246.
19. J. Kothari, D. Bespalov, S. Mancoridis, and A. Shokoufandeh, "On evaluating the efficiency of software feature development using algebraic manifolds," in ICSM '08: International Conference on Software Maintenance, 2008, pp. 7-16.
20. D. R̈othlisberger, O. Greevy, and O. Nierstrasz, "Exploiting Runtime Information in the IDE," in ICPC '08: Proceedings of the 2008 The 16th IEEE International Conference on Program Comprehension. Washington, DC, USA: IEEE Computer Society, 2008, pp. 63-72.
21. I. Beschastnikh, Y. Brun, S. Schneider, M. Sloan, and M. D. Ernst, "Leveraging existing instrumentation to automatically infer invariantconstrained models," in ESEC/FSE '11: Proc. of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering. New York, NY, USA: ACM, 2011, pp. 267-277.
22. B. Cornelissen, A. Zaidman, A. van Deursen, L. Moonen, and R. Koschke, "A systematic survey of program comprehension through dynamic analysis," IEEE Trans. Softw. Eng., vol. 35, pp. 684-702, September 2009.
23. S. Reiss, The Field programming environment: A friendly integrated environment for learning and development. Springer, 1995, vol. 298.
24. J. Boulon, A. Konwinski, R. Qi, A. Rabkin, E. Yang, and M. Yang, "Chukwa, a large-scale monitoring system," in CCA '08: Proc. of the first workshop on Cloud Computing and its Applications, 2008, pp. 1-5.
25. J. Tan, X. Pan, S. Kavulya, R. Gandhi, and P. Narasimhan, "Salsa: analyzing logs as state machines," in WASL'08: Proceedings of the First USENIX conference on Analysis of system logs. Berkeley, CA, USA: USENIX Association, 2008, pp. 6-6.
26. W. Xu, L. Huang, A. Fox, D. Patterson, and M. I. Jordan, "Detecting large-scale system problems by mining console logs," in SOSP '09: Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles. New York, NY, USA: ACM, 2009, pp. 117-132.