Achievements and challenges of metamorphic testing

Proceedings - 2013 4th World Congress on Software Engineering, WCSE 2013

Volumn , Issue , 2013, Pages 73-77

  Hui, Zhan Wei ^a   Huang, Song ^a  

^a PLA UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

Author keywords

Metamorphic relation; Metamorphic testing; Test oracle problem

Indexed keywords

SOFTWARE ENGINEERING; TESTING;

METAMORPHIC RELATIONS; METAMORPHIC TESTING; ORACLE PROBLEM; SOFTWARE TESTING TECHNIQUES; TEST ORACLES;

SOFTWARE TESTING;

EID: 84899097022     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/WCSE.2013.16     Document Type: Conference Paper

References (43)

1. Ammann, P. E. and Knight, J. C. Data diversity: an approach to software fault tolerance, IEEE Transactions on Computers 1988, 37, 4, 418-425.
2. Andrews, J. H., Briand, L. C., and Labiche, Y. Is mutation an appropriate tool for testing experiments? ICSE 2005, ACM, New York, NY, USA, 402-411.
3. Asrafi, M., Liu, H., and Kuo, F. C. On Testing Effectiveness of Metamorphic Relations: A Case Study. 2011 Fifth International Conference on Secure Software Integration and Reliability Improvement. 2011, IEEE Computer Society Washington, DC, USA, 147-156.
4. Murphy, C., Shen, K., and Kaiser, G. Automatic system testing of programs without test oracles. In Proceedings of the Eighteenth international Symposium on Software Testing and Analysis (ISSTA 2009), Chicago, Illinois, USA, 2009, 189-199.
5. Beizer, B. 1990. Software Testing Techniques. New York, NY: Van Nostrand Reinhold.
6. Murphy, C., Kaiser, G., Vo, I., and Chu, M. Quality assurance of software applications using the in vivo testing approach. In Proc. of the Second IEEE International Conference on Software Testing, Verification and Validation (ICST'2009), Denver, Colorado, USA, 2009, 111-120.
7. Baresi, L. and Young, M. Test oracles, Technical Report CIS-TR01-02, Department of Computer and Information Science 2001, University of Oregon, Eugene, OR.
8. Dong, G. W., Nie, C. H., Xu, B. W., AND WANG, L. L. An effective iterative metamorphic testing algorithm based on program path analysis. Proceeding of 7th Annual International Conference on Quality Software (QsIc 2007), 2007, 11-October 12, Portland, Oregon, USA, 292-297.
9. Murphy, C., Kaiser, G., Hu, L., and Wu, L. Properties of machine learning applications for use in metamorphic testing. In Proc. of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE'2008), San Francisco, USA, 2008, 867-872.
10. Hui, Z. W., Huang, S., and Ren Z. P. Metamorphic Testing Integer Overflow Faults of Mission Critical Programs: A Case Study. Mathematical Problems in Engineering. Volume 2013, Article ID 381389, 2013, http://dx. doi. org/10. 1155/2013/381389.
11. Xie, X. Y., W. K. Ho, J., Murphy, C., Kaiser, G., Xu, B. W., and Chen, T. Y. Testing and Validating Machine Learning Classifiers by Metamorphic Testing. Journal of Systems and Software, 84 4, April, 2011, 544-558.
12. Blum, M. and Kannan, S. 1995. Designing programs that check their work, Proceedings of the 31st Annual ACM Symposium on Theory of Computing (STOC '89), 86-97. New York, NY: ACM Press, 1989
13. Also Journal of the ACM, 1995, vol. 42, no. 1, 269-291.
14. Zhang, Z. Y., Chen, W. K., and Tse, T. H., Hu, P. F. Experimental Study to Compare the Use of Metamorphic Testing and Assertion Checking. Journal of Software, 20, 10, 2009, 2637-2654.
15. Murphy, C. and Kaiser, G. Metamorphic Runtime Checking of Non-Testable Programs. Columbia University Dept of Computer Science Tech Report cucs-042-09, 2009, http://hdl. handle. net/10022/AC:p: 9293.
16. Mayer, J., and Guderlei, R. An Empirical Study on the Selection of Good Metamorphic Relations. In Proceedings of the 30th Annual international Computer Software and Applications Conference (COMPSAC 2006), Chicago, IL, 2006, 475-484.
17. Zhou, Z. Q., Tse, T. H., Kuo, F. C., and Chen, T. Y. Automated functional testing of Web search engines in the absence of an oracle, Technical Report TR-2007-06, Department of Computer Science, The University of Hong Kong, Hong Kong, 2007.
18. Chan, W. K., C. F. Ho, J., and Tse, T. H. Piping Classification to Metamorphic Testing: An Empirical Study towards Better Effectiveness for the Identification of Failures in Mesh Simplification Programs. COMPSAC, City University of Hong Kong, Kowloon, China, 2007, 1, 397-404.
19. Wu, P., Shi, X. C., Tang, J. J., Lin, H. M., and Chen, T. Y. Metamorphic testing and special case testing A case study, Journal of Software, 16, 7, 2005, 1210-1220.
20. Xie, X. Y., W. K. Ho, J., Murphy, C., Kaiser, G., Xu, B. W., and Chen, T,Y. Application of metamorphic testing to supervised classifiers. In Proc. of the 9th International Conference on Quality Software (QSIC 2009), Jeju, South Korea, 2009, 135-144.
21. Chan, W. K., Chen, T. Y., Lu, H., Tse, T. H., and Yau, S. S. A metamorphic approach to integration testing of context-sensitive middleware-based applications. In Proceedings of the Fifth International Conference on Quality Software (QSIC 2005), Los Alamitos, California. IEEE Computer Society Press 2005, 241-249.
22. Chan, W. K., Chen, T. Y., Lu, H., Tse, T. H., and Yau, S. S. Integration testing of context-sensitive middleware-based applications: a metamorphic approach, International Journal of Software Engineering and Knowledge Engineering, 2006, 16, 5, 677-703.
23. Hui, Z. W. and Huang, S. 2012. Research on Fundamental of Metamorphic Testing. 2012 PLA Graduate of Engineering Innovation Forum (PLAGEIF) (Best Paper Award), Wuhan, China 2012. 11, 217-223 (In Chinese).
24. Chan, W. K., Cheung, S. C., and Leung, K. R. P. H. A metamorphic testing approach for online testing of service-oriented software applications, International Journal of Web Services Research, 2007, 4, 2, 60-80.
25. Sun, C. A., Wang. G., Mu, B. H., Liu, H., Wang, Z. S., and Chen, T. Y. A Metamorphic Relation-Based Approach to Testing Web Services without Oracles. International Journal of Web Services Research (IJWSR), 9, 1, 2012, 51-73.
26. Dong, G. W., Nie, C. H., and Xu, B. W. Effective metamorphic testing based on program path analysis. Chinese Journal of Computers, 2009, 32 5. 1002-1013. (In Chinese)
27. Zhou, Z. Q., Zhang, S., Hagenbuchner, M., Tse, T. H., Kuo, F.-C., and Chen, T. Y. Automated Functional Testing of Online Search Services, Software Testing, Verification and Reliability, 2010, doi:10. 1002/stvr. 437.
28. Chen, T. Y., Feng, J., and Tse, T. H. Metamorphic testing of programs on partial differential equations: a case study, Proceedings of the 26th Annual International Computer Software and Applications Conference (COMPSAC 2002). IEEE Computer Society Press, Los Alamitos, 2002, 327-333.
29. Chen, T. Y., Cheung, S. C., and Yiu, S. M. Metamorphic testing: a new approach for generating next test cases, Technical Report HKUSTCS98-01, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong, 1998.
30. Chen, T. Y., Huang, D. H., Tse, T. H., and Zhou, Z. Q. Case studies on the selection of useful relations in metamorphic testing. In Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering, Madrid, Spain. Polytechnic University of Madrid, 2004, 569-583.
31. Lu, X. L, and Dong, Y. W. Testing of service-oriented software: a metamorphic testing approach. Journal of Computer Applications, 2011,31. 7, 1756-1758.
32. Chen, T. Y., Kuo, F. C., Liu, Y., and Tang, A. Metamorphic testing and testing with special values. Proceeding of the 5th International Conference on Software Engineering, Artificial Intelligence, Networking, and Parallel/Distributed Computing(SNPD 2004), Mt. Pleasant, Michigan, 2004, 128-134
33. Chen, T. Y., Kuo, F. C., Tse, T. H., Zhou, Z. Q. Metamorphic Testing and Beyond. Proceedings of the International Workshop on Software Technology and Engineering Practice (STEP 2003), Los Alamitos: IEEE Computer Society Press, 2003, 94-100.
34. Chen, T. Y., Kuo, F.-C., Tam, W. K., and Merkel, R. Testing a software based PID controller using metamorphic testing, Proceedings of the International Conference on Pervasive and Embedded Computing and Communication Systems. PECCS2011, 2011, 387-396.
35. Murphy, C., Kaiser, G., and Arias, M. An approach to software testing of machine learning applications. In Proc. of the 19th international conference on software engineering and knowledge engineering (SEKE'2007), Boston, Massachusetts, USA, 167-172, 2007.
36. Gotlieb, A. and Botella, B. Automated metamorphic testing. In Proceedings of the 27th IEEE Annual International Computer Software and Applications Conference (COMPSAC2003), November, 2003, 34-40.
37. Sun, C. A., Wang. G., Mu, B. H., Liu, H., Wang, Z. S., and Chen, T. Y. Metamorphic Testing for Web Services: Framework and a Case Study. 2011 IEEE International Conference on Web Services (ICWS), Washington, DC, 4-9 July 2011, 283-290.
38. Lu, X. L., Dong, Y. W., and Luo, C. Testing of Component-based Software: a Metamorphic Testing Methodology. 2010 7th International Conference on Ubiquitous Intelligence & Computing and 7th International Conference on Autonomic & Trusted Computing (UIC/ATC), 26-29 Oct, 2010, Xian, Shaanxi, 272-276.
39. Mayer, J. and Guderlei, R. An empirical study on the selection of good metamorphic relations, Proceeding of the 30th Annual International Computer Software and Applications Conference (COMPSAC2006), 2006, 475-484.
40. Weyuker, E. J. On testing non-testable programs, The Computer Journal, 25, 4, 1982, 465-470.
41. Murphy, C., Vaughan, M., Ilahi, W., and Kaiser, G. Automatic detection of previously-unseen application states for deployment environment testing and analysis. In Proc. of the 5th International Workshop on Automation of Software Test (AST), Cape Town, South Africa, 2010, 16-23.
42. Dong, G. W., Xu, B. W., Chen, L, Nie, C. H., and Wang, L. L. Case Studies on Testing with Compositional Metamorphic Relations. Journal of Southeast University (English Edition), 2008, 24, 4, 437-443.
43. Murphy, C. and Kaiser, G. Improving the dependability of machine learning applications. Technical Report CUCS-49-08, Dept. of Computer Science, Columbia University, 2008.