How effectively does metamorphic testing alleviate the oracle problem?

IEEE Transactions on Software Engineering

Volumn 40, Issue 1, 2014, Pages 4-22

  Liu, Huai ^a   Kuo, Fei Ching ^b   Towey, Dave ^c   Chen, Tsong Yueh ^b  

^a RMIT UNIVERSITY   (Australia)

^b SWINBURNE UNIVERSITY OF TECHNOLOGY   (Australia)

^c UNIVERSITY OF NOTTINGHAM NINGBO CHINA   (China)

Author keywords

metamorphic relation; metamorphic testing; oracle problem; Software testing; test oracle

Indexed keywords

APPLICATION PROGRAMS; COST EFFECTIVENESS; FAULT DETECTION; TESTING;

DETECTION CAPABILITY; EMPIRICAL STUDIES; METAMORPHIC RELATIONS; METAMORPHIC TESTING; ORACLE PROBLEM; SOFTWARE ANALYSIS; TEST CASE; TEST ORACLES;

SOFTWARE TESTING;

EID: 84898462811     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2013.46     Document Type: Article

References (42)

1. P.E. Ammann and J.C. Knight, "Data Diversity: An Approach to Software Fault Tolerance," IEEE Trans. Computers, vol. 37, no. 4, pp. 418-425, Apr. 1988.
2. A. Avizienis, "The N-Version Approach to Fault-Tolerant Software," IEEE Trans. Software Engineering, vol. 11, no. 12, pp. 1491-1501, Dec. 1985.
3. A.C. Barus, T.Y. Chen, D. Grant, F.-C. Kuo, and M.-F. Lau, "Testing of Heuristic Methods: A Case Study of Greedy Algorithm," Proc. Third IFIP TC 2 Central and Eastern European Conf. Software Eng. Techniques (CEE-SET '08), Lecture Notes in Computer Science, vol. 4980, pp. 246-260, 2011.
4. S. Beydeda, "Self-Metamorphic-Testing Components," Proc. 30th Ann. Int'l Computer Software and Applications Conf. (COMPSAC '06), pp. 265-272, 2006.
5. T.Y. Chen, S.C. Cheung, and S.M. Yiu, "Metamorphic Testing: A New Approach for Generating Next Test Cases," Technical Report HKsUST-CS98-01, Dept. of Computer Science, Hong Kong Univ. of Science and Technology, 1998.
6. T.Y. Chen, J.W.K. Ho, H. Liu, and X. Xie, "An Innovative Approach for Testing Bioinformatics Programs Using Metamorphic Testing," BMC Bioinformatics, vol. 10, pp. 24:1-24:12, 2009.
7. T.Y. Chen, D.H. Huang, T.H. Tse, and Z.Q. Zhou, "Case Studies on the Selection of Useful Relations in Metamorphic Testing," Proc. Fourth Ibero-American Symp. Software Eng. and Knowledge Eng. (JIISIC '04), pp. 569-583, 2004.
8. T.Y. Chen, F.-C. Kuo, H. Liu, and S. Wang, "Conformance Testing of Network Simulators Based on Metamorphic Testing Technique," Proc. 29th IFIP Int'l Conf. Formal Techniques for Networked and Distributed Systems (FORTE '09), vol. 5522, pp. 243-248, 2009.
9. T.Y. Chen, F.-C. Kuo, R.G. Merkel, and T.H. Tse, "Adaptive Random Testing: The ART of Test Case Diversity," J. Systems and Software, vol. 83, no. 1, pp. 60-66, 2010.
10. T.Y. Chen, T.H. Tse, and Z. Zhou, "Semi-Proving: An Integrated Method for Program Proving, Testing, and Debugging," IEEE Trans. Software Eng., vol. 37, no. 1, pp. 109-125, Jan./Feb. 2011.
11. T.Y. Chen and Y.T. Yu, "On the Relationship Between Partition and Random Testing," IEEE Trans. Software Eng., vol. 20, no. 12, pp. 977-980, Dec. 1994.
12. R.A. DeMillo, R.J. Lipton, and F.G. Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," Computer, vol. 11, no. 4, pp. 34-41, Apr. 1978.
13. H. Do, S. Elbaum, and G. Rothermel, "Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and Its Potential Impact," Empirical Software Eng., vol. 10, no. 4, pp. 405-435, 2005. (Pubitemid 41408554)
14. X. Feng, D.L. Parnas, T.H. Tse, and T. O'Callaghan, "A Comparison of Tabular Expression-Based Testing Strategies," IEEE Trans. Software Eng., vol. 37, no. 5, pp. 616-634, Sept./Oct. 2011.
15. J.E. Forrester and B.P. Miller, "An Empirical Study of the Robustness of Windows NT Applications Using Random Testing," Proc. Fourth USENIX Windows Systems Symp. (USENIX-WIN '00), pp. 59-68, 2000.
16. G. Fraser and A. Zeller, "Mutation-Driven Generation of Unit Tests and Oracles," Proc. 19th Int'l Symp. Software Testing and Analysis (ISSTA '10), pp. 147-158, 2010.
17. A. Gotlieb and B. Botella, "Automated Metamorphic Testing," Proc. 27th Ann. Int'l Computer Software and Applications Conf. (COMPSAC '03), pp. 34-40, 2003.
18. B. Hailpern and P. Santhanam, "Software Debugging, Testing, and Verification," IBM Systems J., vol. 41, no. 1, pp. 4-12, 2002. (Pubitemid 34811649)
19. H. Hemmati, A. Arcuri, and L. Briand, "Achieving Scalable Model-Based Testing Through Test Case Diversity," ACM Trans. Software Eng. and Methodology, vol. 22, no. 1, pp. 6:1-6:42, 2013.
20. R.M. Hierons, "Oracles for Distributed Testing," IEEE Trans. Software Eng., vol. 38, no. 3, pp. 629-641, May/June 2012.
21. P. Hu, Z. Zhang, W.K. Chan, and T.H. Tse, "An Empirical Comparison between Direct and Indirect Test Result Checking Approaches," Proc. Third Int'l Workshop Software Quality Assurance (SOQUA '06), pp. 6-13, 2006. (Pubitemid 47209721)
22. Y.F. Hu, R.J. Allan, and K.C.F. Maguire, "Comparing the Performance of JAVA with Fortran and C for Numerical Computing," technical report, Daresbury Laboratory, CCLRC, http://www.dl.ac.uk/TCSC/UKHEC/JASPA/bench.pdf, 2000.
23. JFlex, The fast scanner generator for java. http://jflex.de/, 2009.
24. J.C. Knight and N.G. Leveson, "An Experimental Evaluation of the Assumption of Independence in Multi-Version Programming," IEEE Trans. Software Eng., vol. 12, no. 1, pp. 96-109, Jan. 1986.
25. F.-C. Kuo, T.Y. Chen, and W.K. Tam, "Testing Embedded Software by Metamorphic Testing: A Wireless Metering System Case Study," Proc. 36th IEEE Conf. Local Computer Networks (LCN '11), pp. 295-298, 2011.
26. H.T. Lau, A Java Library of Graph Algorithms and Optimisation. Taylor & Francis, 2007.
27. Y.-S. Ma, J. Offutt, and Y.-R. Kwon, "MuJava: An Automated Class Mutation System," Software Testing, Verification and Reliability, vol. 15, no. 2, pp. 97-133, 2005. (Pubitemid 40858060)
28. L.I. Manolache and D.G. Kourie, "Software Testing Using Model Programs," Software: Practice and Experience, vol. 31, no. 13, pp. 1211-1236, 2001. (Pubitemid 33080539)
29. J. Mayer and R. Guderlei, "An Empirical Study on the Selection of Good Metamorphic Relations," Proc. 30th Ann. Int'l Computer Software and Applications Conf. (COMPSAC '06), pp. 475-484, 2006. (Pubitemid 46661681)
30. B.P. Miller, G. Cooksey, and F. Moore, "An Empirical Study of the Robustness of MacOS Applications Using Random Testing," ACM SIGOPS Operating Systems Rev., vol. 41, no. 1, pp. 78-86, 2007.
31. B.P. Miller, L. Fredriksen, and B. So, "An Empirical Study of the Reliability of UNIX Utilities," Comm. ACM, vol. 33, no. 12, pp. 32-44, 1990. (Pubitemid 21731525)
32. C. Murphy, K. Shen, and G.E. Kaiser, "Automatic System Testing of Programs without Test Oracles," Proc. 18th Int'l Symp. Software Testing and Analysis (ISSTA '09), pp. 189-200, 2009.
33. G.J. Myers, The Art of Software Testing, second ed. John Wiley & Sons, Revised and updated by T. Badgett and T. M. Thomas with C. Sandler, 2004.
34. W.H. Press, S.A. Teukolsky, W.T. Vetterling, and B.P. Flannery, Numerical Recipes in C: The Art of Scientific Computing. Cambridge Univ. Press, 1992.
35. P. Rao, Z. Zheng, T.Y. Chen, N. Wang, and K. Cai, "Impacts of Test Suites Class Imbalance on Spectrum-Based Fault Localization Techniques," Proc. 13th Int'l Conf. Quality Software (QSIC '13), pp. 260-267, 2013.
36. D.S. Rosenblum, "A Practical Approach to Programming with Assertions," IEEE Trans. Software Eng., vol. 21, no. 1, pp. 19-31, Jan. 1995.
37. S. Segura, R.M. Hierons, D. Benavides, and A. Ruiz-Cort-es, "Automated Metamorphic Testing on the Analyses of Feature Models," Information and Software Technology, vol. 53, no. 3, pp. 245-258, 2011.
38. M. Staats, G. Gay, and M.P.E. Heimdahl, "Automated Oracle Creation Support, or: How I Learned to Stop Worrying about Fault Propagation and Love Mutation Testing," Proc. 34th Int'l Conf. Software Eng. (ICSE '12), pp. 870-880, 2012.
39. C. Sun, G. Wang, B. Mu, H. Liu, Z. Wang, and T. Y. Chen, "Metamorphic Testing for Web Services: Framework and a Case Study," Proc. Ninth Int'l Conf. Web Services (ICWS '10), pp. 283-290, 2011.
40. Q. Tao, W. Wu, C. Zhao, and W. Shen, "An Automatic Testing Approach for Compiler Based on Metamorphic Testing Technique," Proc. 17th Asia Pacific Software Eng. Conf. (APSEC '10), 2010.
41. X. Xie, W.E. Wong, T.Y. Chen, and B. Xu, "Metamorphic Slice: An Application in Spectrum-Based Fault Localization," Information and Software Technology, vol. 55, no. 5, pp. 866-879, 2013.
42. Z. Zhang, W.K. Chan, T.H. Tse, and P. Hu, "Experimental Study to Compare the Use of Metamorphic Testing and Assertion Checking," J. Software, vol. 20, no. 10, pp. 2637-2654, 2009.