Empirical evaluations of regression test selection techniques: A systematic review

ESEM'08: Proceedings of the 2008 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

Volumn , Issue , 2008, Pages 22-31

  Engström, Emelie ^a   Skoglund, Mats ^a   Runeson, Per ^a  

^a LUND UNIVERSITY   (Sweden)

Author keywords

Regression testing; Systematic review; Test selection

Indexed keywords

COMPARATIVE EVALUATIONS; EMPIRICAL EVALUATIONS; EMPIRICAL EVIDENCES; EMPIRICAL STUDIES; REGRESSION TEST SELECTIONS; REGRESSION TESTING; SMALL GROUPS; SMALL VARIATIONS; SOFTWARE ENGINEERING JOURNALS; SYSTEMATIC REVIEW; TEST SELECTION;

REGRESSION ANALYSIS; SOFTWARE TESTING; TESTING; VERIFICATION;

COMPUTER SOFTWARE SELECTION AND EVALUATION;

EID: 62949161375     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1414004.1414011     Document Type: Conference Paper

References (61)

1. Agrawal, H., Horgan, J.R., Krauser, E.W., and London, S.A. 1993. Incremental regression testing. In Proceedings. Conference on Software Maintenance 1993. CSM-93 (Cat. NO.93CH3360-5). IEEE Comput. Soc. Press, 348-57.
2. Anjaneyulu, P. and Animesh, B. 2005. An approach for test suite selection to validate applications on deployment of COTS upgrades. In Proceedings. 12th Asia-Pacific Software Engineering Conference. IEEE Computer Society, 7 pp.
3. Bible, J., Rothermel, G., and Rosenblum, D.S. 2001. A comparative study of coarse- and fine-grained safe regression test-selection techniques. ACM Transactions on Software Engineering and Methodology. 10(2), 149-183.
4. Binkley, D. 1998. The application of program slicing to regression testing. Information and Software Technology. 40(11-12), 583-94.
5. Brereton, P., Kitchenham, BA., Budgen, D., Turner, M., and Khalil, M. 2007. Lessons from applying the systematic literature review process within the software engineering domain. Journal of Systems and Software. 80(4), 571-83.
6. Chen, Y.-F., Rosenblum, D.S., and Vo, K.-P. 1994. Test tube: A system for selective regression testing. In Proceedings - International Conference on Software Engineering. IEEE, Los Alamitos, CA, USA, 211-220.
7. Chengying, M. and Yansheng, L. 2005. Regression testing for component-based software systems by enhancing change information. In Proceedings. 12th Asia-Pacific Software Engineering Conference. IEEE Computer Society, 8 pp.
8. Dybâ, T., Dingsöyr, T., and Hanssen, G.K. 2007. Applying Systematic Reviews to Diverse Study Types: An Experience Report. In First International Symposium on Empirical Software Engineering and Measurement, 2007, ESEM 2007. 225-234.
9. Elbaum, S., Kallakuri, P., Malishevsky, A., Rothermel, G., and Kanduri, S. 2003. Understanding the effects of changes on the cost-effectiveness of regression testing techniques. Software Testing, Verification and Reliability. 13(2), 65-83.
10. Fischer, K., Raji, F., and Chruscicki, A. 1981. A methodology for retesting modified software. In NTC '81. IEEE 1981 National Telecommunications Conference. Innovative Telecommunications - Key to the Future. IEEE, 6-3.
11. Graves, T.L., Harrold, M.J., Kim, J.M., Porter, A., and Rothermel, G. 2001. An empirical study of regression test selection techniques. ACM Transactions on Software Engineering and Methodology. 10(2), 184-208.
12. Gregg, R. and Mary Jean, H. 2000. A Safe, Efficient Regression Test Selection Technique.
13. Gupta, R., Harrold, M.J., and Sofia, M.L. 1992. An approach to regression testing using slicing. In Conference on Software Maintenance 1992 (Cat.No.92CH3206-0). IEEE Comput. Soc. Press, 299-308.
14. Gupta, R., Harrold, M.J., and Soffa, M.L. 1996. Program slicing-based regression testing techniques. Software Testing, Verification and Reliability. 6(2), 83-111.
15. Haftmann, F., Kossmann, D., and Lo, E. 2007. A framework for efficient regression tests on database applications. VLDB Journal. 16(1), 145-64.
16. Harrold, M.J., Jones, J.A., Tongyu, L., Donglin, L., Orso, A., Pennings, M., Sinha, S., Spoon, S.A., and Gujarathi, A. 2001. Regression test selection for Java software. In SIGPLAN Not. (USA). ACM, 312-26.
17. Harrold, M.J. and Souffa, M.L. 1988. An incremental approach to unit testing during maintenance. In Proceedings of the Conference on Software Maintenance -1988 (IEEE Cat. N0.88CH2615-3). IEEE Comput. Soc. Press, 362-7.
18. Hartmann, J. and Robson, D.J. 1988. Approaches to regression testing. In Proceedings of the Conference on Software Mmntenance - 1988 (IEEE Cat. No.88CH2615-3). IEEE Comput. Soc. Press, 368-72.
19. Hartmann, J. and Robson, D.J. 1990. Techniques for selective revalidation. IEEE Software. 7(1), 31-6.
20. Hutchins, M., Foster, H., Goradia, T., and Ostrand, T. 1994. Experiments on the effectiveness of dataflow- and controlflow-based test adequacy criteria. In ICSE-16. 16th International Conference on Software Engineering (Cat. No.94CH3409-0). IEEE Comput. Soc. Press, 191-200.
21. Hyunsook, D., Elbaum, S., and Rothermel, G. 2004. Infrastructure support for controlled experimentation with software testing and regression testing techniques. In 2004 International Symposium on Empirical Software Engineering. IEEE Comput. Soc, 60-70.
22. Jiang, Z., Robinson, B., Williams, L., and Smiley, K. 2006. Applying regression test selection for COTS-based applications. In 28th International Conference on Software Engineering Proceedings. ACM, 512-21.
23. Jiang, Z., Robinson, B., Williams, L., and Smiley, K. 2006. A lightweight process for change identification and regression test selection in using COTS components. In Fifth International Conference on Commercial-off-the-Shelf (COTS)-Based Software Systems. IEEE Computer Society, 7 pp.
24. Juristo, N., Moreno, A.M., Vegas, S., and Solari, M. 2006. In search of what we experimentally know about unit testing [software testing]. IEEE Software. 23(6), 72-80.
25. Kampenes Vigdis, B., Dybå, T., Hannay Jo, E., and Sjöberg Dag, I.K. 2007. A systematic review of effect size in software engineering experiments. Information and Software Technology. 49(11-12), 1073-1073.
26. Kim, J.-M., Porter, A., and Rothermel, G. 2005. An empirical study of regression test application frequency. Software Testing, Verification and Reliability. 15(4), 257-279.
27. Kitchenham, B.A. 2007. Guidelines for performing Systematic Literature reviews in Software Engineering Version 2.3. Technical Report S.o.C.S.a.M. Software Engineering Group, Keele University and Department of Computer Science University of Durham.
28. Kitchenham, B.A., Mendes, E., and Travassos, G.H. 2007. Cross versus within-company cost estimation studies: a systematic review. IEEE Transactions on Software Engineering. 33(5), 316-29.
29. Klosch, RR., Glaser, P.W., and Truschnegg, R.J. 2002. A testing approach for large system portfolios in industrial environments. Journal of Systems and Software. 62(1), 11-20.
30. Koju, T., Takada, S., and Doi, N. 2003. Regression Test Selection based on Intermediate Code for Virtual Machines. In Conference on Software Maintenance. Institute of Electrical and Electronics Engineers Inc., 420-429.
31. Leung, H.K.N, and White, L. 1989. Insights into regression testing. In Conference on Software Maintenance. Publ by IEEE, Piscataway, NJ, USA, 60-69.
32. Leung, H.K.N, and White, L. 1990. A study of integration testing and software regression at the integration level. In Proceedings. Conference on Software Maintenance 1990 (Cat. No.90CH2921-5). IEEE Comput. Soc. Press, 290-301.
33. Mansour, N., Bahsoon, R., and Baradhi, G. 2001. Empirical comparison of regression test selection algorithms. Journal of Systems and Software. 57(1), 79-90.
34. Mansour, N. and El-Fakih, K. 1997. Natural optimization algorithms for optimal regression testing. In Proceedings - IEEE Computer Society's International Computer Software & Applications Conference. IEEE, Los Alamitos, CA, USA, 511-514.
35. Memon, A.M. 2004. Using tasks to automate regression testing of GUIs. In IASTED International Conference on Artificial Intelligence and Applications - AIA 2004. ACTA Press, 477-82.
36. Orso, A., Harrold, MJ., Rosenblum, D., Rothermel, G., Soffa, M.L., and Do, H. 2001. Using component metacontent to support the regression testing of component-based software. In Proceedings IEEE International Conference on Software Maintenance. ICSM 2001. IEEE Comput. Soc, 716-25.
37. Orso, A., Nanjuan, S., and Harrold, M.J. 2004. Scaling regression testing to large software systems. In Softw. Eng. Notes (USA). ACM, 241-51.
38. Pasala, A. and Bhowmick, A. 2005. An approach for test suite selection to validate applications on deployment of COTS upgrades. In Proceedings - Asia-Pacific Software Engineering Conference, APSEC. IEEE Computer Society, Los Alamitos, CA 90720-1314, United States, 401-407.
39. Pei, H., Xiaolin, L., Kung, D.C., Chih-Tung, H., Liang, L., Toyoshima, Y., and Chen, C. 1997. A technique for the selective revalidation of OO software. Journal of Software Maintenance: Research and Practice. 9(4), 217-33.
40. Ren, X., Shah, F., Tip, F., Ryder, B.G., and Chesley, O. 2004. Chianti: A tool for change impact analysis of java programs. In 19th Annual ACM Conference on Object-Oriented Programming, Systems, Languages, and Applications, OOPSLA'04. Association for Computing Machinery, New York, NY 10036-5701, United States, 432-448.
41. Rothermel, G. and Harrold, M.J. 1994. A framework for evaluating regression test selection techniques. In ICSE-16. 16th International Conference on Software Engineering (Cat. No.94CH3409-0). IEEE Comput. Soc. Press, 201-10.
42. Rothermel, G. and Harrold, M.J. 1996. Analyzing regression test selection techniques. IEEE Transactions on Software Engineering. 22(8), 529-51.
43. Rothermel, G. and Harrold, M.J. 1997. A safe, efficient regression test selection technique. ACM Transactions on Software Engineering and Methodology. 6(2), 173-210.
44. Rothermel, G., Harrold, M.J., and Dedhia, J. 2000. Regression test selection for C++ software. Journal of Software Testing Verification and Reliability. 10(2), 77-109.
45. Sajeev, A.S.M. and Wibowo, B. 2003. Regression test selection based on version changes of components. In Tenth Asia-Pacific Software Engineering Conference. IEEE Comput. Soc, 78-85.
46. Skoglund, M. and Runeson, P. 2005. A case study of the class firewall regression test selection technique on a large scale distributed software system. In 2005 International Symposium on Empirical Software Engineering (IEEE Cat. No. 05EX1213). IEEE, 10 pp.
47. Staples, M. and Niazi, M. 2007. Experiences using systematic review guidelines. The Journal of Systems & Software. 80(9), 1425-37.
48. Toshihiko, K., Shingo, T., and Norihisa, D. 2003. Regression test selection based on intermediate code for virtual machines. In Proceedings International Conference on Software Mmntenance ICSM 2003. IEEE Comput. Soc, 420-9.
49. White, L. and Abdullah, K. 1997. A firewall approach for the regression testing of object-oriented software. Software Quality Week
50. White, L., Jaber, K., and Robinson, B. 2005. Utilization of extended firewall for object-oriented regression testing. In IEEE International Conference on Software Maintenance, ICSM. IEEE Computer Society, Los Alamitos, CA 90720-1314, United States, 695-698.
51. White, L.J. and Leung, H.K.N. 1992. A firewall concept for both control-flow and data-flow in regression integration testing. In Conference on Software Maintenance 1992 (Cat.No.92CH3206-0). IEEE Comput. Soc. Press, 262-71.
52. Willmor, D. and Embury, S.M. 2005. A safe regression test selection technique for database-driven applications. In Proceedings of the 21st IEEE International Conference on Software Mmntenance. IEEE Comput. Soc, 421-30.
53. Vokolos, F.I. and Frankl, P.G. 1997. Pythia: a regression test selection tool based on textual differencing. In Reliability, Quality and Safety of Software-Intensive Systems. IFIP TC5 WG5.4 3rd International Conference. Chapman & Hall, 3-21.
54. Vokolos, F.J. and Frankl, P.G. 1998. Empirical evaluation of the textual differencing regression testing technique. In Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272). IEEE Comput. Soc, 44-53.
55. Wong, W.E., Horgan, J.R., London, S., and Agrawal, H. 1997. A study of effective regression testing in practice. In Proceedings. The Eighth International Symposium on Software Reliability Engineering (Cat. No.97TB100170). IEEE Comput. Soc, 264-74.
56. Wong, W.E., Horgan, LR., London, S., and Agrawal, H. 1997. Study of effective regression testing in practice. In Proceedings of the International Symposium on Software Reliability Engineering, ISSRE. IEEE Comp Soc, Los Alamitos, CA, USA, 264-274.
57. Wu, Y., Chen, M.-H., and Kao, H.M. 1999. Regression testing on object-oriented programs. In Proceedings 10th International Symposium on Software Reliability Engineering (Cat. NO.PR00443). IEEE Comput. Soc, 270-9.
58. Yanping, C., Robert, L.P., and Sims, D.P. 2002. Specificationbased regression test selection with risk analysis. Proceedings of the 2002 conference of the Centre for Advanced Studies on Collaborative research. IBM Press.
59. Yih-Farn, C., Rosenblum, D.S., and Kiem-Phong, V. 1994. TESTTUBE: a system for selective regression testing. In ICSE-16.16th International Conference on Software Engineering (Cat. No.94CH3409-0). IEEE Comput. Soc. Press, 211-20.
60. Zheng, J., Robinson, B., Williams, L., and Smiley, K. 2005. An initial study of a lightweight process for change identification and regression test selection when source code is not available. In Proceedings - International Symposium on Software Reliability Engineering, ISSRE. IEEE Computer Society, Los Alamitos, CA 90720-1314, United States, 225234.
61. Zheng, J., Robinson, B., Williams, L., and Smiley, K. 2006. A lightweight process for change identification and regression test selection in using COTS components. In Proceedings - Fifth International Conference on Commercial-off-the-Shelf (COTS)-Based Software Systems. Institute of Electrical and Electronics Engineers Computer Society, Piscataway, NJ 08855-1331, United States, 137-143.