Using machine learning techniques to detect metamorphic relations for programs without test oracles

2013 IEEE 24th International Symposium on Software Reliability Engineering, ISSRE 2013

Volumn , Issue , 2013, Pages 1-10

  Kanewala, Upulee ^a   Bieman, James M ^a  

^a Veterinary Dentistry and Oral Surgery   (United States)

Author keywords

Decision trees; Machine learning; Metamorphic relation; Metamorphic testing; Mutation analysis; Scientific software testing; Software testing; Support vector machines; Test oracles

Indexed keywords

METAMORPHIC RELATIONS; METAMORPHIC TESTING; MUTATION ANALYSIS; SCIENTIFIC SOFTWARES; TEST ORACLES;

DATA FLOW ANALYSIS; DECISION TREES; LEARNING SYSTEMS; SOFTWARE RELIABILITY; SUPPORT VECTOR MACHINES;

SOFTWARE TESTING;

EID: 84893326644     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ISSRE.2013.6698899     Document Type: Conference Paper

References (43)

1. E. J. Weyuker, "On testing non-testable programs", Comput. J., vol. 25, no. 4, pp. 465-470, 1982.
2. R. Sanders and D. Kelly, "The challenge of testing scientific software", in Proc. Conf. for the Association for Software Testing (CAST), Toronto, July 2008, pp. 30-36.
3. L. N. Joppa, G. McInerny, R. Harper, L. Salido, K. Takeda, K. O'Hara, D. Gavaghan, and S. Emmott, "Troubling trends in scientific software use", Science, vol. 340, no. 6134, pp. 814-815, 2013.
4. T. Y. Chen, S. C. Cheung, and S. M. Yiu, "Metamorphic testing: a new approach for generating next test cases", Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong, Tech. Rep. HKUST-CS98-01, 1998.
5. T. Y. Chen, T. H. Tse, and Z. Q. Zhou, "Fault-based testing without the need of oracles", Information and Software Technology, vol. 45, no. 1, pp. 1-9, 2003.
6. Z. Q. Zhou, D. H. Huang, T. H. Tse, Z. Yang, H. Huang, and T. Y. Chen, "Metamorphic testing and its applications", in Proc. of the 8th Int. Symp. on Future Software Technology (ISFST 2004). Software Engineers Association, 2004.
7. X. Xie, J. W. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y. Chen, "Testing and validating machine learning classifiers by metamorphic testing", Journal of Systems and Software, vol. 84, no. 4, pp. 544-558, 2011.
8. T. Y. Chen, D. H. Huang, T. H. Tse, and Z. Q. Zhou, "Case studies on the selection of useful relations in metamorphic testing", in Proc. of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC 2004), 2004, pp. 569-583.
9. C. Murphy, G. E. Kaiser, L. Hu, and L. Wu, "Properties of machine learning applications for use in metamorphic testing", in Proc. of the 20th Int. Conf. on Software Engineering & Knowledge Engineering (SEKE'2008), San Francisco, CA, USA, 2008, pp. 867-872.
10. D. Zhang and J. J. Tsai, Advances in Machine Learning Applications in Software engineering. Idea Group Publishing, 2007.
11. J. R. Quinlan, "Induction of decision trees", Mach. Learn., vol. 1, no. 1, pp. 81-106, Mar. 1986.
12. C. Cortes and V. Vapnik, "Support-vector networks", Machine Learning, vol. 20, pp. 273-297, 1995.
13. J. R. Quinlan, C4.5: programs for machine learning. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1993.
14. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, and I. H. Witten, "The weka data mining software: an update", SIGKDD Explor. Newsl., vol. 11, no. 1, pp. 10-18, Nov. 2009.
15. H. Zhu, "On information and sufficiency", Annals of Statistics, 1997.
16. R. Vallee-Rai and L. J. Hendren, "Jimple: Simplifying Java bytecode for analyses and transformations", 1998.
17. J. Huang and C. Ling, "Using AUC and accuracy in evaluating learning algorithms", Knowledge and Data Engineering, IEEE Transactions on, vol. 17, no. 3, pp. 299-310, march 2005.
18. F. Provost, T. Fawcett, and R. Kohavi, "The case against accuracy estimation for comparing induction algorithms", in Proc. of the 15th Int. Conf. on Machine Learning, 1997, pp. 445-453.
19. C. X. Ling, J. Huang, and H. Zhang, "AUC: a better measure than accuracy in comparing learning algorithms", in Proc. of IJCAI03, 2003, pp. 329-341.
20. J. H. Andrews, L. C. Briand, and Y. Labiche, "Is mutation an appropriate tool for testing experiments?" in Proc. of the 27th Int. Conf. on Software Engineering, ser. ICSE'05. New York, NY, USA: ACM, 2005, pp. 402-411.
21. Y. Jia and M. Harman, "An analysis and survey of the development of mutation testing", IEEE Transactions on Software Engineering, vol. 37, pp. 649-678, 2011.
22. Y.-S. Ma and J. Offutt, "Description of method-level mutation operators for java", November 2005.
23. G. M. Weiss, "Mining with rarity: a unifying framework", SIGKDD Explor. Newsl., vol. 6, no. 1, pp. 7-19, Jun. 2004.
24. R. Burbidge and B. Buxton, "An introduction to support vector machines for data mining", in Keynote Papers, Young OR12, University of Nottingham, Operational Research Society, 2001, pp. 3-15.
25. S. Beiden, M. Maloof, and R. Wagner, "A general model for finitesample effects in training and testing of competing classifiers", Pattern Analysis and Machine Intelligence, IEEE Trans on, vol. 25, no. 12, pp. 1561-1569, Dec. 2003.
26. R. De Millo, R. Lipton, and F. Sayward, "Hints on test data selection: Help for the practicing programmer", Computer, vol. 11, no. 4, pp. 34-41, april 1978.
27. C. Murphy, "Metamorphic testing techniques to detect defects in applications without test oracles", PhD dissertation, Columbia University, 2010.
28. S. Hasan, "Automatic detection of metamorphic properties of software", Columbia University, Tech. Rep. CUCS-003-12, 2009.
29. C. Murphy, M. S. Raunak, A. King, S. Chen, C. Imbriano, G. Kaiser, I. Lee, O. Sokolsky, L. Clarke, and L. Osterweil, "On effective testing of health care simulation software", in Proc. of the 3rd Workshop on Software Engineering in Health Care, ser. SEHC'11. New York, NY, USA: ACM, 2011, pp. 40-47.
30. J. Ding, T. Wu, D. Wu, J. Q. Lu, and X.-H. Hu, "Metamorphic testing of a Monte Carlo modeling program", in Proc. of the 6th Int. Workshop on Automation of Software Test, ser. AST'11. New York, NY, USA: ACM, 2011, pp. 1-7.
31. T. Y. Chen, J. W. K. Ho, H. Liu, and X. Xie, "An innovative approach for testing bioinformatics programs using metamorphic testing." BMC Bioinformatics, vol. 10, 2009.
32. R. Guderlei and J. Mayer, "Statistical metamorphic testing testing programs with random output by means of statistical hypothesis tests and metamorphic testing", in Proc. of the 7th Int. Conf. on Quality Software, ser. QSIC'07, 2007, pp. 404-409.
33. T. Y. Chen, J. Feng, and T. H. Tse, "Metamorphic testing of programs on partial differential equations: A case study", in Proc. of the 26th Int. Computer Software and Applications Conf. on Prolonging Software Life: Development and Redevelopment, ser. COMPSAC'02. Washington, DC, USA: IEEE Computer Society, 2002, pp. 327-333.
34. C. Murphy, K. Shen, and G. Kaiser, "Using JML runtime assertion checking to automate metamorphic testing in applications without test oracles", in Proc. of the 2009 Int. Conf. on Software Testing Verification and Validation, ser. ICST'09. Washington, DC, USA: IEEE Computer Society, 2009, pp. 436-445.
35. -, "Automatic system testing of programs without test oracles", in Proc. of the 18th Int. symp. on Software testing and analysis, ser. ISSTA'09. New York, NY, USA: ACM, 2009, pp. 189-200.
36. L. C. Briand, "Novel applications of machine learning in software testing", in Proc. of the 2008 The 8th Int. Conf. on Quality Software. Washington, DC, USA: IEEE Computer Society, 2008, pp. 3-10.
37. J. F. Bowring, J. M. Rehg, and M. J. Harrold, "Active learning for automatic classification of software behavior", in Proc. of the 2004 ACM SIGSOFT Int. Symposium on Software Testing and Analysis, ser. ISSTA'04. New York, NY, USA: ACM, 2004, pp. 195-205.
38. L. C. Briand, Y. Labiche, and Z. Bawar, "Using machine learning to refine black-box test specifications and test suites", in Proc. of the 8th Int. Conf. on Quality Software, ser. QSIC'08. Washington, DC, USA: IEEE Computer Society, 2008, pp. 135-144.
39. L. C. Briand, Y. Labiche, and X. Liu, "Using machine learning to support debugging with tarantula", in Software Reliability, 2007. ISSRE'07. The 18th IEEE Int. Symposium on, 2007, pp. 137-146.
40. K. Frounchi, L. C. Briand, L. Grady, Y. Labiche, and R. Subramanyan, "Automating image segmentation verification and validation by learning test oracles", Inf. Softw. Technol., vol. 53, no. 12, pp. 1337-1348, Dec. 2011.
41. J.-H. Lo, "Predicting software reliability with support vector machines", in Computer Research and Development, 2010 2nd Int. Conf. on, may 2010, pp. 765-769.
42. F. Wang, L.-W. Yao, and J.-H. Wu, "Intelligent test oracle construction for reactive systems without explicit specifications", in Dependable, Autonomic and Secure Computing (DASC), 2011 IEEE 9th Int. Conf. on, dec. 2011, pp. 89-96.
43. T. Gärtner, P. Flach, and S. Wrobel, "On graph kernels: Hardness results and efficient alternatives", in Learning Theory and Kernel Machines, ser. Lecture Notes in Computer Science. Springer Berlin/Heidelberg, 2003, vol. 2777, pp. 129-143.