Automatic system testing of programs without test oracles

Proceedings of the 18th International Symposium on Software Testing and Analysis, ISSTA 2009

Volumn , Issue , 2009, Pages 189-199

  Murphy, Christian ^a   Shen, Kuang ^a   Kaiser, Gail ^a  

^a Columbia University ^*  (United States)

Author keywords

Metamorphic testing; Oracle problem; Software testing

Indexed keywords

METADATA; QUALITY ASSURANCE; SYSTEM THEORY;

ARBITRARY INPUTS; AUTOMATIC SYSTEMS; EMPIRICAL STUDIES; MANUAL INTERVENTION; METAMORPHIC TESTING; MULTIPLE OUTPUTS; ORACLE PROBLEM; REAL WORLD PROJECTS;

SOFTWARE TESTING;

EID: 85008244135     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1572272.1572295     Document Type: Conference Paper

References (39)

1. SVM application list. http://www.clopinet.com/isabelle/Projects/SVM/applist.html.
2. J. R. Abrial. Specification Language Z. Oxford Univ Press, 1980.
3. J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments? In Proc of the 27th International Conference on Software Engineering (ICSE), pages 402-411, 2005.
4. M. Blum and S. Kannan. Designing programs that check their work. Journal of the ACM, 42(1):269-291, Jan. 1995.
5. C. Boyapati, S. Khurshid, and D. Marinov. Korat: automated testing based on Java predicates. In Proc of the 2002 ACM SIGSOFT international symposium on software testing and analysis, pages 123-133, 2002.
6. T. Y. Chen, S. C. Cheung, and S. Yiu. Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01, Dept. of Computer Science, Hong Kong Univ. of Science and Technology, 1998.
7. T. Y. Chen, T. H. Tse, and Z. Q. Zhou. Fault-based testing without the need of oracles. Information and Software Technology, 44(15):923-931, 2002.
8. T. Y. Chen, T. H. Tse, and Z. Q. Zhou. Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing. In Proc. of the ACM SIGSOFT international symposium on software testing and analysis (ISSTA), pages 191-195, 2002.
9. W. J. Cody Jr. and W. Waite. Software Manual for the Elementary Functions. Prentice Hall, 1980.
10. M. D. Davis and E. J. Weyuker. Pseudo-oracles for non-testable programs. In Proc. of the ACM '81 Conference, pages 254-257, 1981.
11. M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin. Dynamically discovering likely programming invariants to support program evolution. In Proc. of the 21st International Conference on Software Engineering (ICSE), pages 213-224, 1999.
12. A. Gotleib and B. Botella. Automated metamorphic testing. In Proc. of 27th annual international computer software and applications conference (COMPSAC), pages 34-40, 2003.
13. P. Gross et al. Predicting electricity distribution feeder failures using machine learning susceptibility analysis. In Proc. of the 18th Conference on Innovative Applications in Artificial Intelligence, 2006.
14. R. Guderlei and J. Mayer. Statistical metamorphic testing-testing programs with random output by means of statistical hypothesis tests and metamorphic testing. In Proc of the Seventh International Conference on Quality Software, pages 404-409, 2007.
15. S. Hangal and M. S. Lam. Tracking down software bugs using automatic anomaly detection. In Proc. of the 24th International Conference on Software Engineering (ICSE), pages 291-301, 2002.
16. J. A. Hanley and B. J. McNeil. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143:29-36, 1982.
17. J. Henkel and A. Diwan. Discovering algebraic specifications from Java classes. In Proc. of the 17th European Conference on Object-Oriented Programming ECOOP, 2003.
18. D. Hoffman. Heuristic test oracles. Software Testing and Quality Engineering, pages 29-32, 1999.
19. D. Jackson. Alloy: a lightweight object modelling notation. ACM Transactions on Software Engineering and Methodology, 11(2):256-290, 2002.
20. J. Knight and N. Leveson. An experimental evaluation of the assumption of independence in multi-version programming. IEEE Transactions on Software Engineering, 12(1):96-109, 1986.
21. T. Mitchell. Machine Learning: An Artificial Intelligence Approach. Morgan Kaufmann, 1983.
22. C. Murphy and G. Kaiser. Improving the dependability of machine learning applications. Technical Report CUCS-49-08, Dept. of Computer Science, Columbia University, 2008.
23. C. Murphy, G. Kaiser, and M. Arias. Parameterizing random test data according to equivalence classes. In Proc of the 2nd international workshop on random testing, pages 38-41, 2007.
24. C. Murphy, G. Kaiser, M. Chu, and I. Vo. Quality assurance of software applications using the in vivo testing approach. In Proc of the Second IEEE International Conference on Software Testing, Verification and Validation (ICST), 2009.
25. C. Murphy, G. Kaiser, L. Hu, and L. Wu. Properties of machine learning applications for use in metamorphic testing. In Proc. of the 20th international conference on software engineering and knowledge engineering (SEKE), pages 867-872, 2008.
26. C. Murphy, K. Shen, and G. Kaiser. Using JML runtime assertion checking to automate metamorphic testing in applications without test oracles. In Proc of the Second IEEE International Conference on Software Testing, Verification and Validation (ICST), 2009.
27. D. J. Newman, S. Hettich, C. L. Blake, and C. J. Merz. UCI repository of machine learning databases. University of California, Dept of Information and Computer Science, 1998.
28. I. Nunes, A. Lopes, V. Vasconcelos, J. Abreu, and L. S. Reis. Checking the conformance of Java classes against algebraic specifications. In In Proceedings of ICFEM'06, volume 4260 of LNCS, pages 494-513. Springer-Verlag, 2006.
29. S. Osman, D. Subhraveti, G. Su, and J. Nieh. The design and implementation of Zap: A system for migrating computing environments. In Proc of the Fifth Symposium on Operating Systems Design and Implementation (OSDI), pages 361-376, 2002.
30. J. Platt. Fast training of support vector machines using sequential minimal optimization. Advances in Kernel Methods, Support Vector Learning, 1999.
31. J. R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufman, 1993.
32. S. Sankar. Run-time consistency checking of algebraic specifications. In Proceedings of the 1991 international symposium on software testing, analysis, and verification, pages 123-129, 1991.
33. S. Sankar, A. Goyal, and P. Sikchi. Software testing using algebraic specification based test oracles. Technical Report CSL-TR-93-566, Dept. of Computer Science, Stanford Univ., 2003.
34. C. Spearman. Footrule for measuring correlation. British Journal of Psychology, 2:89-108, June 1906.
35. V. N. Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
36. K. Wang and S. Stolfo. Anomalous payload-based network intrusion detection. In Proc. of the Seventh International Symposium on Recent Advances in Intrusion Detection (RAID), Sept. 2004.
37. E. J. Weyuker. On testing non-testable programs. Computer Journal, 25(4):465-470, November 1982.
38. I. H. Witten and E. Frank. Data Mining: Practical Machine Learning Tools and Techniques, 2nd Edition. Morgan Kaufmann, 2005.
39. X. Xie, J. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y. Chen. Improving the quality of computational science software by using metamorphic relations to test machine learning applications. Technical Report CUCS-004-09, Dept. of Computer Science, Columbia University, January 2009.