Testing and validating machine learning classifiers by metamorphic testing

Journal of Systems and Software

Volumn 84, Issue 4, 2011, Pages 544-558

  Xie, Xiaoyuan ^a,d,e   Ho, Joshua W K ^b   Murphy, Christian ^c,f   Kaiser, Gail ^c   Xu, Baowen ^e   Chen, Tsong Yueh ^a  

^a SWINBURNE UNIVERSITY OF TECHNOLOGY   (Australia)

^b BRIGHAM AND WOMEN S HOSPITAL   (United States)

^c Columbia University ^*  (United States)

^d SOUTHEAST UNIVERSITY   (China)

^e NANJING UNIVERSITY   (China)

^f UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

Machine learning; Metamorphic testing; Oracle problem; Test oracle; Validation; Verification

Indexed keywords

MACHINE-LEARNING; METAMORPHIC TESTING; ORACLE PROBLEM; TEST ORACLE; VALIDATION;

BIOINFORMATICS; COMPUTATIONAL LINGUISTICS; COMPUTER SOFTWARE SELECTION AND EVALUATION; LEARNING SYSTEMS;

LEARNING ALGORITHMS;

EID: 79751532765     PISSN: 01641212     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jss.2010.11.920     Document Type: Conference Paper

References (40)

1. E. Alpaydin Introduction to Machine Learning 2004 The MIT Press
2. J.H. Andrews, L.C. Briand, and Y. Labiche Is mutation an appropriate tool for testing experiments? Proceedings of the 27th International Conference on Software Engineering (ICSE) 2005 402 411
3. D. Balzarotti, M. Cova, V. Felmetsger, N. Jovanovic, E. Kirda, C. Kruegel, and G. Vigna Saner: composing static and dynamic analysis to validate sanitization in web applications Proceedings of the 29th IEEE Symposium on Security and Privacy (S&P) 2008 387 401
4. L. Briand Novel applications of machine learning in software testing Proceedings of the 8th International Conference on Quality Software(QSIC) 2008 3 10
5. W.K. Chan, S.C. Cheung, and K.R.P.H. Leung A metamorphic testing approach for online testing of service-oriented software applications International Journal of Web Services Research 4 1 2007 60 80
6. T.J. Cheatham, J.P. Yoo, and N.J. Wahl Software testing: a machine learning experiment Proceedings of the ACM 23rd Annual Conference on Computer Science 1995 135 141
7. Chen, T.Y., Cheung, S.C., Yiu, S., 1998. Metamorphic testing: a new approach for generating next test cases. Tech. Rep. HKUST-CS98-01, Dept. of Computer Science, Hong Kong University of Science and Technology.
8. T.Y. Chen, J.W.K. Ho, H. Liu, and X. Xie An innovative approach for testing bioinformatics programs using metamorphic testing BMC Bioinformatics 10 2009 24 36
9. T.Y. Chen, D.H. Huang, T.H. Tse, and Z.Q. Zhou Case studies on the selection of useful relations in metamorphic testing Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC) 2004 569 583
10. Chen, T.Y., Tse, T.H., Zhou, Z., in press. Semi-proving: an integrated method for program proving, testing, and debugging. IEEE Transactions on Software Engineering.
11. T.Y. Chen, T.H. Tse, and Z.Q. Zhou Fault-based testing without the need of oracles Information and Software Technology 44 15 2002 923 931
12. M.D. Davis, and E.J. Weyuker Pseudo-oracles for non-testable programs Proceedings of the ACM Annual Conference 1981 254 257
13. J. Demsar, B. Zupan, G. Leban, and T. Curk Orange: from experimental machine learning to interactive data mining Lecture Notes in Computer Science 2004 537 539 (Pubitemid 39740441)
14. J. Duran, and S. Ntafos An evaluation of random testing IEEE Transactions on Software Engineering 10 4 1984 438 444
15. J.E. Gewehr, M. Szugat, and R. Zimmer BioWeka - extending the Weka framework for bioinformatics Bioinformatics 23 5 2007 651 653 (Pubitemid 46522607)
16. A. Gotlieb, and B. Botella Automated metamorphic testing Proceedings of the 27th Annual International Conference on Computer Software and Applications (COMPSAC) 2003 34 40
17. R. Guderlei, and J. Mayer Statistical metamorphic testing testing programs with random output by means of statistical hypothesis tests and metamorphic testing Proceedings of the 7th International Conference on Quality Software (QSIC) 2007 404 409
18. J.W.K. Ho, M.W. Lin, S. Adelstein, and C.G. dos Remedios Customising an antibodyleukocyte capture microarray for Systemic Lupus Erythematosus: Beyond biomarker discovery Proteomics - Clinical Applications 4 2 2009 179 189
19. J.W.K. Ho, M. Stefani, C.G. dos Remedios, and M.A. Charleston Differential variability analysis of gene expression and its application to human diseases Bioinformatics 24 13 2008 390 398
20. J. Knight, and N. Leveson An experimental evaluation of the assumption of independence in multi-version programming IEEE Transactions on Software Engineering 12 1 1986 96 109 (Pubitemid 16575808)
21. Y.-S. Ma, J. Offutt, and Y.R. Kwon MuJava: an automated class mutation system Journal of Software Testing, Verification and Reliability 15 2 2005 97 133 (Pubitemid 40858060)
22. B. Madan, K. Goeva-Popstojanova, K. Vaidyanathan, and K. Trivedi A method for modeling and quantifying the security attributes of intrusion tolerant systems Performance Evaluation 56 1-4 2004 167 186
23. G.J. McLachlan, K.-A. Do, and C. Ambroise Analyzing Microarray Gene Expression Data 2004 Wiley
24. Mell, P., Hu, V., Lippmann, R., Haines, J., Zissman, M., 2003. An overview of issues in testing intrusion detection systems. Tech. Rep. NIST IR 7007, National Institute of Standard and Technology.
25. M. Middendorf, A. Kundaje, M. Shah, Y. Freund, C.H. Wiggins, and C. Leslie Motif discovery through predictive modeling of gene regulation Research in Computational Molecular Biology 3500 2005 538 552 (Pubitemid 41420690)
26. T. Mitchell Machine Learning: An Artificial Intelligence Approach, vol. III 1983 Morgan Kaufmann
27. Murphy, C., Kaiser, G., 2010. Empirical evaluation of approaches to testing applications without test oracles. Tech. Rep. CUCS-039-09, Dept. of Computer Science, Columbia University.
28. C. Murphy, G. Kaiser, L. Hu, and L. Wu Properties of machine learning applications for use in metamorphic testing Proceedings of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE) 2008 867 872
29. C. Murphy, K. Shen, and G. Kaiser Using JML runtime assertion checking to automate metamorphic testing in applications without test oracles Proceedings of 2nd International Conference on Software Testing Verification and Validation (ICST) 2009 436 445
30. Newman, D.J., Hettich, S., Blake, C.L., Merz, C.J., 1998. UCI Repository of Machine Learning Databases. Dept. of Information and Computer Science, University of California.
31. A. Offutt, A. Lee, G. Rothermel, R. Untch, and C. Zapf An experimental determination of sufficient mutant operators ACM Transactions on Software Engineering and Methodology 5 2 1996 99 118 (Pubitemid 126418670)
32. N. Puketza, K. Zhang, M. Chung, B. Mukherjee, and R. Olsson A methodology for testing intrusion detection systems IEEE Transactions on Software Engineering 22 10 2002 719 729
33. S. Segura, R. Hierons, D. Benavides, and A. Ruiz-Cortés Automated test data generation on the analyses of feature models: a metamorphic testing approach Proceedings of 3rd International Conference on Software Testing Verification and Validation (ICST) 2010 35 44
34. T.H. Tse, S.S. Yau, W.K. Chan, H. Lu, and T.Y. Chen Testing context-sensitive middleware-based software applications Proceedings of the 28th Annual International Conference on Computer Software and Applications (COMPSAC) 2004 458 466 (Pubitemid 40673893)
35. E.J. Weyuker On testing non-testable programs Computer Journal 25 4 1982 465 470
36. I.H. Witten, and E. Frank Data Mining: Practical Machine Learning Tools and Techniques 2nd edition 2005 Morgan Kaufmann
37. X. Xie, J.W.K. Ho, C. Murphy, G. Kaiser, B.W. Xu, and T.Y. Chen Application of metamorphic testing to supervised classifiers Proceedings of the 9th International Conference on Quality Software (QSIC) 2009 135 144
38. Xie, X., Wong, W.E., Chen, T.Y., Xu, B., 2010. Spectrum-based fault localization without test oracles. Tech. Rep. UTDCS-07-10, Dept. of Computer Science, University of Texas at Dallas.
39. S. Yoo Metamorphic testing of stochastic optimisation Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation Workshops (ICSTW) 2010 192 201
40. D. Zhang, and J. Tsai Machine learning and software engineering Software Quality Journal 11 2 2003 87 119 (Pubitemid 39019685)