Experimental study to compare the use of metamorphic testing and assertion checking

Ruan Jian Xue Bao/Journal of Software

Volumn 20, Issue 10, 2009, Pages 2637-2654

  Zhang, Zhen Yu ^a   Chan, W K ^b   Tse, T H ^a   Hu, Pei Feng ^c  

^a UNIVERSITY OF HONG KONG   (Hong Kong)

^b CITY UNIVERSITY OF HONG KONG   (Hong Kong)

^c China Merchants Bank   (Hong Kong)

Author keywords

Assertion checking; Controlled experiment; Empirical evaluation; Metamorphic testing; Test oracle

Indexed keywords

ASSERTION-CHECKING; CONTROLLED EXPERIMENT; EMPIRICAL EVALUATIONS; METAMORPHIC TESTING; TEST ORACLES;

COMPUTER SOFTWARE SELECTION AND EVALUATION; ECONOMIC AND SOCIAL EFFECTS; FAULT DETECTION; SOFTWARE TESTING; TELLURIC PROSPECTING;

EXPERIMENTS;

EID: 70449725267     PISSN: 10009825     EISSN: None     Source Type: Journal    
DOI: 10.3724/SP.J.1001.2009.00578     Document Type: Article

References (44)

1. Hu P, Zhang Z, Chan WK, Tse TH. An empirical comparison between direct and indirect test result checking approaches. In: Proc. of the 3rd Int'l Workshop on Software Quality Assurance (SOQUA 2006), in conjunction with the 14th ACM SIGSOFT Symp. on Foundations of Software Engineering (SIGSOFT 2006/FSE-14). New York: ACM Press, 2006. 6-13.
2. Beizer B. Software Testing Techniques. New York: Van Nostrand Reinhold, 1990.
3. Weyuker EJ. On testing non-testable programs. The Computer Journal, 1982, 25 (4): 465-470.
4. Chen TY, Feng J, Tse TH. Metamorphic testing of programs on partial differential equations: A case study. In: Proc. of the 26th Annual Int'l Computer Software and Applications Conf. (COMPSAC 2002). Los Alamitos: IEEE Computer Society Press, 2002. 327-333.
5. Chan WK, Cheung SC, Ho JCF, Tse TH. PAT: A pattern classification approach to automatic reference oracles for the testing of mesh simplification programs. Journal of Systems and Software, 2008. doi: 10.1016/j.jss.2008.07.019
6. Chan WK, Ho JCF, Tse TH. Piping classification to metamorphic testing: An empirical study towards better effectiveness for the identification of failures in mesh simplification programs. In: Proc. of the 31st Annual Int'l Computer Software and Applications Conf. (COMPSAC 2007), Vol. 1. Los Alamitos: IEEE Computer Society Press, 2007. 397-404.
7. Tse TH, Yau SS, Chan WK, Lu H, Chen TY. Testing context-sensitive middleware-based software applications. In: Proc. of the 28th Annual Int'l Computer Software and Applications Conf. (COMPSAC 2004), Vol. 1. Los Alamitos: IEEE Computer Society Press, 2004. 458-465.
8. Lu H, Chan WK, Tse TH. Testing context-aware middleware-centric programs: A data flow approach and an RFID-based experimentation. In: Proc. of the 14th ACM SIGSOFT Int'l Symp. on Foundations of Software Engineering (SIGSOFT 2006/FSE-14). New York: ACM Press, 2006. 242-252.
9. Chan WK, Chen TY, Lu H, Tse TH, Yau SS. A metamorphic approach to integration testing of context-sensitive middleware-based applications. In: Proc. of the 5th Int'l Conf. on Quality Software (QSIC 2005). Los Alamitos: IEEE Computer Society Press, 2005. 241-249.
10. Chan WK, Chen TY, Lu H, Tse TH, Yau SS. Integration testing of context-sensitive middleware-based applications: A metamorphic approach. Int'l Journal of Software Engineering and Knowledge Engineering, 2006, 16 (5): 677-703.
11. Chan WK, Cheung SC, Leung KRPH. A metamorphic testing approach for online testing of service-oriented software applications. Int'l Journal of Web Services Research, 2007, 4 (2): 60-80.
12. Chen TY, Tse TH, Zhou ZQ. Fault-Based testing without the need of oracles. Information and Software Technology, 2003, 45 (1): 1-9.
13. Binder RV. Testing Object-Oriented Systems: Models, Patterns, and Tools. Reading: Addison Wesley, 2000.
14. Taylor RN. Assertions in programming languages. ACM SIGPLAN Notices, 1980, 15 (1): 105-114.
15. Chen TY, Tse TH, Zhou ZQ. Semi-Proving: An integrated method based on global symbolic evaluation and metamorphic testing. In: Proc. of the 2002 ACM SIGSOFT Int'l Symp. on Software Testing and Analysis (ISSTA 2002). New York: ACM Press, 2002. 191-195.
16. Gotlieb A, Botella B. Automated metamorphic testing. In: Proc. of the 27th Annual Int'l Computer Software and Applications Conf. (COMPSAC 2003). Los Alamitos: IEEE Computer Society Press, 2003. 34-40.
17. Sun Y, Jones EL. Specification-Driven automated testing of GUI-based Java programs. In: Proc. of the 42nd Annual Southeast Regional Conf. (ACM-SE 42). New York: ACM Press, 2004. 140-145.
18. Chan WK, Cheung SC, Leung KRPH. Towards a metamorphic testing methodology for service-oriented software applications. The 1st Int'l Conf. on Services Engineering (SEIW 2005). In: Proc. of the 5th Int'l Conf. on Quality Software (QSIC 2005). Los Alamitos: IEEE Computer Society Press, 2005. 470-476.
19. Cobleigh RL, Avrunin GS, Clarke LA. User guidance for creating precise and accessible property specifications. In: Proc. of the 14th ACM SIGSOFT Int'l Symp. on Foundations of Software Engineering (SIGSOFT 2006/FSE-14). New York: ACM Press, 2006. 208-218.
20. Andrews JH, Briand LC, Labiche Y. Is mutation an appropriate tool for testing experiments? In: Proc. of the 27th Int'l Conf. on Software Engineering (ICSE 2005). New York: ACM Press, 2005. 402-411.
21. Chapman D. A program testing assistant. Communications of the ACM, 1982, 25 (9): 625-634.
22. Ar S, Blum M, Codenotti B, Gemmell P. Checking approximate computations over the reals. In: Proc. of the 25th Annual ACM Symp. on Theory of Computing (STOC'93). New York: ACM Press, 1993. 786-795.
23. Blum M, Kannan S. Designing programs that check their work. Journal of the ACM, 1995, 42 (1): 269-291.
24. Blum M, Luby M, Rubinfeld R. Self-Testing/correcting with applications to numerical problems. Journal of Computer and System Sciences, 1993, 47 (3): 549-595.
25. Xie Q, Memon AM. Designing and comparing automated test oracles for GUI-based software applications. ACM Trans. on Software Engineering and Methodology, 2007, 16 (1): Article No. 4.
26. Offutt AJ, Lee A, Rothermel G, Untch RH, Zapf C. An experimental determination of sufficient mutant operators. ACM Trans. on Software Engineering and Methodology, 1996, 5 (2): 99-118.
27. Helm R, Holland IM, Gangopadhyay D. Contracts: Specifying behavioral compositions in object-oriented systems. In: Proc. of the 5th Annual Conf. on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA'90). ACM SIGPLAN Notices, 1990, 25 (10): 169-180.
28. Briand LC, Di Penta M, Labiche Y. Assessing and improving state-based class testing: A series of experiments. IEEE Trans. on Software Engineering, 2004, 30 (11): 770-783.
29. Meyer B. Applying 'design by contract'. IEEE Computer, 1992, 25 (10): 40-51.
30. Last M, Friedman M, Kandel A. The data mining approach to automated software testing. In: Proc. of the 9th ACM SIGKDD Int'l Conf. on Knowledge Discovery and Data Mining (KDD 2003). New York: ACM Press, 2003. 388-396.
31. Vanmali M, Last M, Kandel A. Using a neural network in the software testing process. Int'l Journal of Intelligent Systems, 2002, 17 (1): 45-62.
32. Francis P, Leon D, Minch M, Podgurski A. Tree-Based methods for classifying software failures. In: Proc. of the 15th Int'l Symp. on Software Reliability Engineering (ISSRE 2004). Los Alamitos: IEEE Computer Society Press, 2004. 451-462.
33. Podgurski A, Leon D, Francis P, Masri W, Minch M, Sun J, Wang B. Automated support for classifying software failure reports. In: Proc. of the 25th Int'l Conf. on Software Engineering (ICSE 2003). Los Alamitos: IEEE Computer Society Press, 2003. 465-475.
34. Bowring JF, Rehg JM, Harrold MJ. Active learning for automatic classification of software behavior. In: Proc. of the 2004 ACM SIGSOFT Int'l Symp. on Software Testing and Analysis (ISSTA 2004). New York: ACM Press, 2004. 195-205.
35. Chan WK, Cheng MY, Cheung SC, Tse TH. Automatic goal-oriented classification of failure behaviors for testing XML-based multimedia software applications: An experimental case study. Journal of Systems and Software, 2006, 79 (5): 602-612.
36. Beydeda S. Self-Metamorphic-Testing components. In: Proc. of the 30th Annual Int'l Computer Software and Applications Conf. (COMPSAC 2006), Vol. 1. Los Alamitos: IEEE Computer Society Press, 2006. 265-272.
37. Wu P. Iterative metamorphic testing. In: Proc. of the 29th Annual Int'l Computer Software and Applications Conf. (COMPSAC 2005), Vol. 1. Los Alamitos: IEEE Computer Society Press, 2005. 19-24.
38. Murphy C. Using runtime testing to detect defects in applications without test oracles. In: Companion to Proc. of the 16th ACM SIGSOFT Int'l Symp. on Foundations of Software Engineering (SIGSOFT 2008/FSE-16). New York: ACM Press, 2008.
39. Chan FT, Chen TY, Cheung SC, Lau MF, Yiu SM. Application of metamorphic testing in numerical analysis. In: Proc. of the IASTED Int'l Conf. on Software Engineering (SE'98). Calgary: ACTA Press, 1998. 191-197.
40. Prechelt L, Unger B, Tichy WF, Brössler P, Votta LG. A controlled experiment in maintenance comparing design patterns to simpler solutions. IEEE Trans. on Software Engineering, 2001, 27 (12): 1134-1144.
41. Vokáč M, Tichy W, Sjoberg DIK, Arisholm E, Aldrin M. A controlled experiment comparing the maintainability of program designed with and without design patterns: A replication in a real programming environment. Empirical Software Engineering, 2004, 9 (3): 149-195.
42. Howden WE. Weak mutation testing and completeness of test sets. IEEE Trans. on Software Engineering, 1982, SE-8 (4): 371-379.
43. Ma YS, Offutt AJ, Kwon YR. MuJava: An automated class mutation system. Software Testing, Verification and Reliability, 2005, 15 (2): 97-133.
44. Namin AS, Andrews JH, Murdoch DJ. Sufficient mutation operators for measuring test effectiveness. In: Proc. of the 30th Int'l Conf. on Software Engineering (ICSE 2008). New York: ACM Press, 2008. 351-360.