Programs, tests, and oracles: The foundations of testing revisited

Proceedings - International Conference on Software Engineering

Volumn , Issue , 2011, Pages 391-400

  Staats, Matt ^a   Whalen, Michael W ^a   Heimdahl, Mats P E ^a  

^a UNIVERSITY OF MINNESOTA   (United States)

Author keywords

testing formalism; theory of testing

Indexed keywords

EMPIRICAL RESEARCH; EMPIRICAL STUDIES; FORMAL FOUNDATION; PROGRAM TESTING; TEST ORACLES; TESTING PROCESS; THEORY OF TESTING;

COMPUTER SOFTWARE SELECTION AND EVALUATION; FOUNDATIONS; RESEARCH; SOFTWARE ENGINEERING;

SOFTWARE TESTING;

EID: 79959901246     PISSN: 02705257     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1985793.1985847     Document Type: Conference Paper

References (47)

1. Z. Al-Khanjari, M. Woodward, and H. Ramadhan. Critical analysis of the pie testability technique. Software Quality Journal, 10(4):331-354, 2002.
2. A. Avizienis, J.-C. Laprie, B. Randell, and C. Landwehr. Basic Concepts and Taxonomy of Dependable and Secure Computing. IEEE Trans. Dependable and Secure Computing, 1(1):11-33, 2004.
3. B. Beizer. Software Testing Techniques, 2nd Edition. Van Nostrand Reinhold, New York, 1990.
4. G. Bernot. Testing against formal specifications: A theoretical view. In TAPSOFT'91: Colloquium on Trees in Algebra and Programming (CAAP'91), page 99. Springer, 1991.
5. G. Bernot, M. Gaudel, B. Marre, and U. Liens. Software testing based on formal specifications: a theory and a tool. Software Engineering Journal, 6(6):387-405, 1991.
6. A. Bertolino. Software testing research: Achievements, challenges, dreams. In L. Briand and A. Wolf, editors, Future of Software Engineering 2007. IEEE-CS Press, 2007.
7. L. Briand. A critical analysis of empirical research in software testing. In Empirical Software Engineering and Measurement, 2007. ESEM 2007. First Int'l Symposium on, pages 1-8, 2007.
8. L. Briand, M. DiPenta, and Y. Labiche. Assessing and improving state-based class testing: A series of experiments. IEEE Trans. on Software Engineering, 30 (11), 2004.
9. T. Budd and D. Angluin. Two notions of correctness and their relation to testing. Acta Informatica, 18(1):31-45, 1982. (Pubitemid 13474053)
10. W. Chen, R. Untch, G. Rothermel, S. Elbaum, and J. Von Ronne. Can fault-exposure-potential estimates improve the fault detection abilities of test suites? Software Testing Verification and Reliability, 12(4):197-218, 2002.
11. R. DeMillo, R. Lipton, and F. Sayward. Hints on test data selection: Help for the practicing programmer. IEEE computer, 11(4):34-41, 1978. (Pubitemid 8631038)
12. P. Frankl and E. Weyuker. A formal analysis of the fault-detecting ability of testing methods. In IEEE Trans. on Software Engineering, 1993.
13. A. Gargantini and C. Heitmeyer. Using model checking to generate tests from requirements specifications. Software Engineering Notes, 24(6):146-162, November 1999.
14. M. Gaudel. Testing can be formal, too. Lecture Notes in Computer Science, 915:82-96, 1995.
15. M. Geller. Test data as an aid in proving program correctness. In Proc. of the 3rd ACM SIGACT-SIGPLAN Symp. on Principles on Programming Languages, pages 209-218. ACM New York, NY, USA, 1976.
16. J. B. Goodenough and S. L. Gerhart. Toward a theory of testing: Data selection criteria. In R. T. Yeh, editor, Current trends in programming methodology. Prentice Hall, 1979.
17. J. Gourlay. A mathematical framework for the investigation of testing. IEEE Trans. on Software Engineering, pages 686-709, 1983.
18. D. Hamlet. Foundations of software testing: dependability theory. ACM SIGSOFT Software Engineering Notes, 19(5):128-139, 1994.
19. R. Hierons. Comparing test sets and criteria in the presence of test hypotheses and fault domains. ACM Transactions on Software Engineering and Methodology (TOSEM), 11(4):448, 2002.
20. W. Howden. Reliability of the path analysis testing strategy. IEEE Transactions on Software Engineering, 2(3), 1976.
21. W. Howden. Weak mutation testing and completeness of test sets. IEEE Trans. on Software Engineering, pages 371-379, 1982.
22. M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. Experiments of the effectiveness of dataflow-and controlflow-based test adequacy criteria. In Proc. of the 16th Int'l Conference on Software Engineering, pages 191-200. IEEE Computer Society Press Los Alamitos, CA, USA, 1994.
23. J. Mayer and R. Guderlei. Test oracles using statistical methods. In Proc. of the First Int'l Workshop on Software Quality, pages 179-189. Citeseer, 2004.
24. A. Memon, I. Banerjee, and A. Nagarajan. What test oracle should I use for effective GUI testing? Automated Software Engineering, 2003. Proc. 18th IEEE Int'l Conf. on, pages 164-173, 2003.
25. L. Morell. A theory of fault-based testing. IEEE Transactions on Software Engineering, 16(8):844-857, 1990.
26. A. Parrish and S. Zweben. Analysis and refinement of software test data adequacy properties. IEEE Trans. on Software Engineering, 17(6):565-581, 1991. (Pubitemid 21675164)
27. A. Parrish and S. Zweben. Clarifying some fundamental concepts in software testing. IEEE Trans. on Software Engineering, 19(7):742-746, 1993.
28. A. Rajan, M. Whalen, and M. Heimdahl. The effect of program and model structure on MC/DC test adequacy coverage. In Proc. of the 30th Int'l Conference on Software engineering, pages 161-170. ACM New York, NY, USA, 2008.
29. S. Rayadurgam and M. P. Heimdahl. Coverage based test-case generation using model checkers. In Proc. of the 8th IEEE Int'l. Conf. and Workshop on the Engineering of Computer Based Systems, pages 83-91. IEEE Computer Society, April 2001.
30. D. J. Richardson, S. L. Aha, and T. O'Malley. Specification-based test oracles for reactive systems. In Proc. of the 14th Int'l Conference on Software Engineering, pages 105-118. Springer, May 1992.
31. G. Rothermel, M. J. Harrold, J. Ostrin, and C. Hong. An empirical study of the effects of minimization on the fault detection capabilities of test suites. In Proceedings of the International Conference on Software Maintenance, pages 34-43, November 1998.
32. M. Staats, M. Whalen, and M. Heimdahl. Better testing through oracle selection (NIER track). In Proc. of the Int'l Conf. on Software Engineering 2011, 2011.
33. J. Voas. Dynamic testing complexity metric. Software Quality Journal, 1(2):101-114, 1992.
34. J. Voas. PIE: A dynamic failure-based technique. IEEE Trans. on Software Engineering, 18(8):717-727, 1992.
35. J. Voas and K. Miller. Putting assertions in their place. In Software Reliability Engineering, 1994., 5th Int'l Symposium on, pages 152-157, 1994.
36. S. Weiss. Comparing test data adequacy criteria. ACM SIGSOFT Software Engineering Notes, 14(6):42-49, 1989.
37. E. Weyuker. On testing non-testable programs. The Computer Journal, 25(4):465, 1982.
38. E. Weyuker. Axiomatizing software test data adequacy. IEEE Trans. on Software Engineering, 12(12):1128-1138, 1986. (Pubitemid 17500453)
39. E. Weyuker. The evaluation of program-based software test data adequacy criteria. Communications of the ACM, 31(6):668-675, 1988. (Pubitemid 18626361)
40. E. Weyuker and T. Ostrand. Theories of program testing and the application of revealing subdomains. IEEE Trans. on Software Engineering, pages 236-246, 1980.
41. E. Weyuker, S. Weiss, and D. Hamlet. Comparison of program testing strategies. In Proc. of the Symposium on Testing, Analysis, and Verification, page 10. ACM, 1991.
42. W. Wong, J. Horgan, S. London, and A. Mathur. Effect of test set minimization on fault detection effectiveness. In Proc. of the 17th Int'l Conf. on Software Engineering, pages 41-50. ACM, 1995.
43. M. Woodward and K. Halewood. From weak to strong, dead or alive? an analysis of some mutation testing issues. In Software Testing, Verification, and Analysis, 1988., Proc. of the 2nd Workshop on, pages 152-158, 1988.
44. H. Zhu. Axiomatic assessment of control flow-based software test adequacy criteria. Software Engineering Journal, 10(5):194-204, 1995.
45. H. Zhu and P. Hall. Test data adequacy measurement. Software Engineering Journal, 8(1):21-29, 1993. (Pubitemid 23613235)
46. H. Zhu, P. Hall, and J. R. May. Software unit test coverage and adequacy. ACM Computing Surveys, 29(4):366-427, December 1997.
47. H. Zhu and X. He. A theory of behaviour observation in software testing. Technical report, 1999.