An Industrial Case Study on the Automated Detection of Performance Regressions in Heterogeneous Environments

Proceedings - International Conference on Software Engineering

Volumn 2, Issue , 2015, Pages 159-168

  Foo, King Chun ^a   Jiang, Zhen Ming ^b   Adams, Bram ^c   Hassan, Ahmed E ^d   Zou, Ying ^d   Flora, Parminder ^a  

^a BlackBerry ^*  (Canada)

^b YORK UNIVERSITY   (Canada)

^c ÉCOLE POLYTECHNIQUE DE MONTRÉAL   (Canada)

^d QUEEN S UNIVERSITY   (Canada)

Author keywords

Load testing; Performance analysis; Performance regression; Performance regression testing; Performance testing

Indexed keywords

AUTOMATION; LOAD TESTING; OPEN SOURCE SOFTWARE; OPEN SYSTEMS; REGRESSION ANALYSIS; SCALES (WEIGHING INSTRUMENTS); SOFTWARE ENGINEERING; TESTING;

HETEROGENEOUS ENVIRONMENTS; HISTORICAL PERFORMANCE DATA; INDUSTRIAL PERFORMANCE; PERFORMANCE ANALYSIS; PERFORMANCE DEGRADATION; PERFORMANCE REGRESSION; PERFORMANCE TESTING; REGRESSION TESTING;

SOFTWARE TESTING;

EID: 84951813851     PISSN: 02705257     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICSE.2015.144     Document Type: Conference Paper

References (37)

1. Puppet labs: It automation software for system administrators. http: //puppetlabs. com/.
2. A. Avritzer and B. Larson. Load testing software using deterministic state testing. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), 1993.
3. A. Avritzer and E. J. Weyuker. The automatic generation of load test suites and the assessment of the resulting software. IEEE Transactions on Software Engineering, 1995.
4. A. Avritzer and E. J. Weyuker. The role of modeling in the performance testing of e-commerce applications. IEEE Transactions on Software Engineering, 2004.
5. G. K. Baah, A. Podgurski, and M. J. Harrold. Causal inference for statistical fault localization. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA), 2010.
6. E. Bauer and R. Kohavi. An empirical comparison of voting classification algorithms: Bagging, boosting, and variants. Machine Learning, 1999.
7. C. Begin. ibatis jpetstore. http: //sourceforge. net/projects/ibatisjpetstore/.
8. P. Bodík, M. Goldszmidt, and A. Fox. Hilighter: Automatically building robust signatures of performance behavior for small-and large-scale systems. In Proceedings of the SYSML workshop, 2008.
9. L. Breiman. Bagging predictors. Machine Learning, 1996.
10. D. Breitgand, M. Goldstein, E. Henis, and O. Shehory. Performance management via adaptive thresholds with separate control of false positive and false negative errors. In Integrated Network Management, 2009.
11. C. Buchta and M. Hahsler. arules: Mining association rules and frequent itemsets. http: //r-forge. r-project. org/projects/arules/.
12. L. Bulej, T. Kalibera, and P. Tma. Repeated results analysis for middleware regression benchmarking. Performance Evaluation, 2005.
13. I. Cohen, M. Goldszmidt, T. Kelly, J. Symons, and J. S. Chase. Correlating instrumentation data to system states: A building block for automated diagnosis and control. In Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation, 2004.
14. I. Cohen, S. Zhang, M. Goldszmidt, J. Symons, T. Kelly, and A. Fox. Capturing, indexing, clustering, and retrieving system history. In Proceedings of the 20th ACM Symposium on Operating Systems Principles, 2005.
15. J. Dougherty, R. Kohavi, and M. Sahami. Supervised and unsupervised discretization of continuous features. In Proceedings of ICML, 1995.
16. Dell dvd store. http: //linux. dell. com/dvdstore/.
17. K. C. Foo, Z. M. Jiang, B. Adams, A. E. Hassan, Y. Zou, and P. Flora. Mining performance regression testing repositories for automated performance analysis. In Proceedings of the 10th International Conference on Quality Software, 2010.
18. M. J. Harrold. Reduce, reuse, recycle, recover: Techniques for improved regression testing. In Proceedings of the IEEE International Conference on Software Maintenance, 2009.
19. J. Herrington. Five common php database problems. http: //www. ibm. com/developerworks/library/os-php-dbmistake/index. html, 2006.
20. M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. Experiments of the effectiveness of dataflow-and controlflow-based test adequacy criteria. In Proceedings of the 16th International Conference on Software Engineering (ICSE), 1994.
21. M. Jiang, M. A. Munawar, T. Reidemeister, and P. A. Ward. System monitoring with metric-correlation models: Problems and solutions. In Proceedings of the 6th International Conference on Autonomic Computing (ICAC), 2009.
22. M. Jiang, M. A. Munawar, T. Reidemeister, and P. A. S. Ward. Automatic fault detection and diagnosis in complex software systems by information-theoretic monitoring. In Proceedings of the International Conference on Dependable Systems and Networks (DSN), 2009.
23. Z. M. Jiang, A. E. Hassan, G. Hamann, and P. Flora. Automatic identification of load testing problems. In Proceedings of the International Conference on Software Maintenance (ICSM), 2008.
24. Z. M. Jiang, A. E. Hassan, G. Hamann, and P. Flora. Automated performance analysis of load tests. In Proceedings of the International Conference on Software Maintenance (ICSM), 2009.
25. H. Malik, B. Adams, A. E. Hassan, P. Flora, and G. Hamann. Using load tests to automatically compare the subsystems of a large enterprise system. In Proceedings of the Annual Computer Software and Applications Conference, 2010.
26. H. Malik, H. Hemmati, and A. E. Hassan. Automatic detection of performance deviations in the load testing of large scale systems. In Proceedings of the 2013 International Conference on Software Engineering (ICSE), 2013.
27. Neoload. http: //www. neotys. com.
28. L. Northrop. Ultra-large-scale systems- The software challenge of the future. Technical report, SEI, Carnegie Mellon, June 2006.
29. D. Oppenheimer and D. A. Patterson. Architecture and dependability of large-scale internet services. IEEE Internet Computing, 2002.
30. A. Orso, N. Shi, and M. J. Harrold. Scaling regression testing to large software systems. In Proceedings of the 12th ACM SIGSOFT Twelfth International Symposium on Foundations of Software Engineering (FSE), 2004.
31. R. Quinlan. Bagging, boosting, and c4. 5. In Proceedings of the Thirteenth National Conference on Artificial Intelligence-Volume 1, volume 5, 1996.
32. Replication package. https: //dl. dropboxusercontent. com/u/4803154/ FooRep. zip.
33. Rpe2 relative server performance. http: //www. ideasinternational. com/ IT-Buyers/Server-Performance.
34. W. Shang, Z. M. Jiang, H. Hemmati, B. Adams, A. E. Hassan, and P. Martin. Assisting developers of big data analytics applications when deploying on hadoop clouds. In Proceedings of the 2013 International Conference on Software Engineering (ICSE), 2013.
35. M. Woodside, G. Franks, and D. C. Petriu. The future of software performance engineering. In Proceedings of the Future of Software Engineering track (FOSE), 2007.
36. C. Yilmaz, A. S. Krishna, A. Memon, A. Porter, D. C. Schmidt, A. Gokhale, and B. Natarajan. Main effects screening: A distributed continuous quality assurance process for monitoring performance degradation in evolving software systems. In Proceedings of the International Conference on Software Engineering (ICSE), 2005.
37. S. Zhang, I. Cohen, J. Symons, and A. Fox. Ensembles of models for automated diagnosis of system performance problems. In Proceedings of the International Conference on Dependable Systems and Networks (DSN), 2005.