An empirical comparison between direct and indirect test result checking approaches

Proceedings of the Third International Workshop on Software Quality Assurance, SOQUA 2006

Volumn , Issue , 2006, Pages 6-13

  Hu, Peifeng ^a   Zhang, Zhenyu ^a   Chan, W K ^b   Tse, T H ^a  

^a UNIVERSITY OF HONG KONG   (Hong Kong)

^b CITY UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Controlled experiment; Empirical evaluation; Metamorphic testing; Test oracle

Indexed keywords

COMPUTER PROGRAM LISTINGS; COST EFFECTIVENESS; FAULT TOLERANT COMPUTER SYSTEMS; PROBLEM SOLVING;

ASSERTION CHECKING; EMPIRICAL EVALUATION; METAMORPHIC TESTING; OPEN SOURCE PROGRAMS;

SOFTWARE TESTING;

EID: 34547662248     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1188895.1188901     Document Type: Conference Paper

References (36)

1. J. H. Andrews, L. C. Briand, and Y. Labiche. Is mutation an appropriate tool for testing experiments'?. In Proceedings of the 27th International Conference on Software Engineering (ICSE 2005), pages 402-411. ACM Press, New York, 2005.
2. S. Ar, M. Blum, B. Codenotti, and P. Gemmell. Checking approximate computations over the reals. In Proceedings of the 25th Annual ACM Symposium on Theory of Computing (STOC '93), pages 786-795. ACM Press, New York, 1993.
3. B. Beizer. Software Testing Techniques. Van Nostrand Reinhold, New York, 1990.
4. R. V. Binder. Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison Wesley, Reading, Massachusetts, 2000.
5. M. Blum, and S. Kannan. Designing programs that check their work. Journal of the ACM, 42 (1): 269-291, 1995.
6. M. Blum, M. Luby, and R. Rubinfeld. Self-testing/correcting with applications to numerical problems. Journal of Computer and System Sciences, 47 (3): 549-595, 1993.
7. J. F. Bowring, J. M. Rehg, and M. J. Harrold. Active learning for automatic classification of software behavior. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2004), pages 195-205. ACM Press, New York, 2004.
8. L. C. Briand, M. Di Penta, and Y. Labiche. Assessing and improving state-based class testing: a series of experiments. IEEE Transactions on Software. Engineering, 30 (11): 770-783, 2004.
9. F.T. Chan, T. Y. Chen, S. C. Cheung, M. F. Lau, and S. M. Yiu. Application of metamorphic testing in numerical analysis. In Proceedings of the IASTED International Conference on Software Engineering (SE '98), pages 191-197. ACTA Press, Calgary, Canada, 1998.
10. W K. Chan, T. Y. Chen, H. Lu, T. H. Tse, and S. S. Yau. A metamorphic approach to integration testing of context-sensitive middleware-based applications. In Proceedings of the 5th International Conference, on Quality Software (QSIC 2005), pages 241-249. IEEE Computer Society Press, Los Alamitos, California, 2005.
11. W K. Chan, T. Y. Chen, H. Lu, T. H. Tse, and S. S. Yau. Integration testing of context-sensitive middleware-based applications: a metamorphic approach. International Journal of Software Engineering and Knowledge Engineering, to appear.
12. W K. Chan, M. Y. Cheng, S. C. Cheung, and T. H. Tse. Automatic goal-oriented classification of failure behaviors for testing XML-based multimedia software applications: an experimental case study. Journal, of Systems and Software, 79 (5): 602-612, 2006.
13. W K. Chan, S. C. Cheung, and K.R. P. H. Leung. Towards a metamorphic testing methodology for service-oriented software applications. The 1st International Conference on Services Engineering (SEIW 2005), in Proceedings of the 5th International Conference on Quality Software (QSIC 2005), pages 470-476. IEEE Computer Society Press, Los Alamitos, California, 2005.
14. W K. Chan, S. C. Cheung, and K.R. P. H. Leung. A metamorphic testing approach for online testing of service-oriented software applications. International Journal of Web Services Research, to appear.
15. D. Chapman. A program testing assistant. Communications of the ACM, 25 (9): 625-634, 1982.
16. T. Y. Chen, J. Feng, and T. H. Tse. Metamorphic testing of programs on partial differential equations: a case study. In Proceedings of the 26th Annual. International Computer Software and Applications Conference (COMPSAC 2002), pages 327-333. IEEE Computer Society Press, Los Alamitos, California, 2002.
17. T. Y. Chen, T. H. Tse, and Z. Q. Zhou. Semi-proving: an integrated method based on global symbolic evaluation and metamorphic testing. In Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2002), pages 191-195. ACM Press, New York, 2002.
18. T. Y. Chen, T. H. Tse, and Z. Q. Zhou. Fault-based testing without the need of oracles. Information and Software Technology, 45 (1): 1-9, 2003.
19. R.L. Cobleigh, G. S. Avrunin, and L. A. Clarke. User guidance for creating precise and accessible property specifications. In Proceedings of the 14th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT2006/FSE-14). ACM Press, New York, 2006.
20. P. Francis, D. Leon, M. Minch, and A. Podgurski. Tree-based methods for classifying software failures. In Proceedings of the 15th International. Symposium on Software Reliability Engineering (ISSRE 2004), pages 451-462. IEEE Computer Society Press, Los Alamitos, California, 2004.
21. P. G. Frankl and S.N. Weiss. An experimental comparison of the effectiveness of branch testing and data flow testing. IEEE Transactions on Software Engineering, 19 (8): 774-787, 1993.
22. A. Gotlieb and B. Botella. Automated metamorphic testing. In Proceedings of the 27th Annual International Computer Software and Applications Conference (COMPSAC 2003), pages 34-40. IEEE Computer Society Press, Los Alamitos, California, 2003.
23. R. Helm, I. M. Holland, and D. Gangopadhyay. Contracts: specifying behavioral compositions in object-oriented systems. In Proceedings of the 5th Annual Conference on Object-Oriented Programming Systems, Languages, and Applications (OOPSLA '90), ACM SIGPLANNotices, 25 (10): 169-180, 1990.
24. M. Last, M. Friedman, and A. Kandel. The data mining approach to automated software testing. In Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2003), pages 388-396. ACM Press, New York, 2003.
25. H. Lu, W. K. Chan, and T. H. Tse. Testing context-aware middleware-centric programs: a data flow approach and an RFID-based experimentation. In Proceedings of the 14th ACMSIGSOFTInternational. Symposium on Foundations of Software. Engineering (SIGSOFT 2006/FSE-14). ACM Press, New York, 2006.
26. Y-S. Ma, A. J. Offutt, and Y-R. Kwon. MuJava: an automated class mutation system. Software Testing, Verification and Reliability, 15 (2): 97-133, 2005.
27. L. I. Manolache and D. G. Kourie. Software testing using model programs. Software: Practice and Experience, 31 (13): 1211-1236, 2001.
28. A. M. Memon, M. E. Pollack, and M. L. Soffa. Automated test oracles for GUIs. In Proceedings of the 8th ACM SIGSOFT International Symposium on Foundations of Software Engineering (SIGSOFT2000/FSE-8), pages 30-39. ACM Press, New York, 2000.
29. A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf. An experimental determination of sufficient mutant operators. ACM Transactions on Software Engineering and Methodology, 5 (2): 99-118, 1996.
30. A. Podgurski, D. Leon, P. Francis, W. Masri, M. Minch, J. Sun, and B. Wang. Automated support for classifying software failure reports. In Proceedings of the 25th International Conference on Software Engineering (ICSE 2003), pages 465-475. IEEE Computer Society Press, Los Alamitos, California, 2003.
31. Y. Sun and E. L. Jones. Specification-driven automated testing of GUI-based Java programs. In Proceedings of the 42nd Annual Southeast Regional Conference (ACM-SE 42), pages 140-145. ACM Press, New York, 2004.
32. R. N. Taylor. Assertions in programming languages. ACM SIGPLAN Notices, 15(1): 105-114, 1980.
33. T.H. Tse, S.S. Yau, W.K. Chan, H. Lu, andT. Y. Chen. Testing context-sensitive middleware-based software applications. In Proceedings of the 28th Annual. International Computer Software and Applications Conference (COMPSAC 2004), volume 1, pages 458-165. IEEE Computer Society Press, Los Alamitos, California, 2004.
34. M. Vanmali, M. Last, and A. Kandel. Using a neural network in the software testing process. International Journal of Intelligent Systems, 17 (1): 45-62, 2002.
35. E.J. Weyuker. On testing non-testable programs. The Computer Journal, 25 (4): 465-470, 1982.
36. Q. Xie and A. Memon. Designing and comparing automated test oracles for GUI-based software applications. ACM Transactions on Software Engineering and Methodology, to appear.