An approach for experimentally evaluating effectiveness and efficiency of coverage criteria for software testing

International Journal on Software Tools for Technology Transfer

Volumn 10, Issue 2, 2008, Pages 145-160

  Gupta, Atul ^a   Jalote, Pankaj ^a,b  

^a INDIAN INSTITUTE OF TECHNOLOGY KANPUR   (India)

^b INDIAN INSTITUTE OF TECHNOLOGY DELHI   (India)

Author keywords

Coverage based testing; Coverage criteria; Experimental evaluation; Mutation operators; Statistical analysis; Test case generation

Indexed keywords

BLOCK CODES; SOFTWARE TESTING; STATISTICAL METHODS;

COVERAGE BASED TESTING; MUTATION OPERATORS; TEST CASE GENERATION;

COMPUTER SOFTWARE SELECTION AND EVALUATION;

EID: 40349090174     PISSN: 14332779     EISSN: 14332787     Source Type: Journal    
DOI: 10.1007/s10009-007-0059-5     Document Type: Conference Paper

References (53)

1. Andrews, J.H., Briand, L.C., Labiche, Y.: Is mutation an appropriate tool for testing experiments? In: ICSE '05: Proceedings of the 27th International Conference on Software Engineering, pp. 402-411. ACM Press, New York (2005)
2. Antoniol, G., Briand, L.C., Penta, M.D., Labiche, Y.: A case study using the round-trip strategy for state-based class testing. In: ISSRE '02: Proceedings of the 13th International Symposium on Software Reliability Engineering, pp. 269-279. IEEE Computer Society, Washington (2002)
3. Basili, V.R.: The role of experimentation in soPtware engineering: past, current, and future. In: ICSE '96: Proceedings of the 18th International Conference on Software Engineering, pp. 442-449. IEEE Computer Society, Washington (1996)
4. Basili, V.R., Caldeira, G., Rombach, H.D.: The goal question metric approach. In: Encyclopedia of Software Engineering (1994)
5. Basili V.R. and Selby R.W. (1987). Comparing the effectiveness of software testing strategies. IEEE Trans. Softw. Eng. 13(12): 1278-1296
6. Basili V.R., Selby R.W. and Hutchens D.H. (1986). Experimentation in software engineering. IEEE Trans. Softw. Eng. 12(7): 733-743
7. BCEL Homepage. http://jakarta.apache.org/bcel/
8. Briand L.C. and Labiche Y. (2004). Empirical studies of software testing techniques: Challenges, practical strategies and future research. SIGSOFT Softw. Eng. Notes 29(5): 1-3
9. Briand, L.C., Labiche, Y., Wang, Y.: Using simulation to empirically investigate test coverage criteria based on statechart. In: ICSE '04: Proceedings of the 26th International Conference on Software Engineering, pp. 86-95. IEEE Computer Society, Washington (2004)
10. Chang, J., Richardson, D.J., Sankar, S.: Structural specification-based testing with adl. In: ISSTA, pp. 62-70 (1996)
11. CruiseControl. http://www-dse.doc.ic.ac.uk/concurrency/book
12. DeMillo R.A., Lipton R.J. and Sayward F.G. (1978). Hints on test data selection: Help for the practicing programmer. IEEE Computer 11(4): 34-41
13. Dick, J., Faivre, A.: Automating the generation and sequencing of test cases from model-based specifications. In: FME, pp. 268-284 (1993)
14. Do, H., Elbaum, S., Rothermel, G.: Infrastructure support for controlled experimentation with software testing and regression testing techniques. In: ISESE '04: Proceedings of the 2004 International Symposium on Empirical Software Engineering, pp. 60-70. IEEE Computer Society, Washington (2004)
15. Fowler M. (1999). Refactoring: Improving the Design of Existing Code. Addison-Wesley, Reading
16. Frankl, P.G., Deng, Y.: Comparison of delivered reliability of branch, data flow and operational testing: A case study. In: ISSTA '00: Proceedings of the 2000 ACM SIGSOFT International Symposium on Software Testing and Analysis, pp. 124-134. ACM Press, New York (2000)
17. Frankl P.G. and Weiss S.N. (1993). An experimental comparison of the effectiveness of branch testing and data flow testing. IEEE Trans Softw. Eng. 19(8): 774-787
18. Frankl P.G. and Weyuker E.J. (1993). A formal analysis of the fault-detecting ability of testing methods. IEEE Trans Softw. Eng. 19(3): 202-213
19. Frankl P.G. and Weyuker E.J. (1993). Provable improvements on branch testing. IEEE Trans. Softw. Eng. 19(10): 962-975
20. Goodenough J.B. and Gerhart S.L. (1975). Toward a theory of test data selection. IEEE Trans. Software Eng. 1(2): 156-173
21. Grieskamp, W., Gurevich, Y., Schulte, W., Veanes, M.: Generating finite state machines from abstract state machines. In: ISSTA '02: Proceedings of the 2002 ACM SIGSOFT International Symposium on software Testing and Analysis, pp. 112-122. ACM Press, New York (2002)
22. Hamlet, R.: Theoretical comparison of testing methods. In: TAV3: Proceedings of the ACM SIGSOFT '89 3rd Symposium on Software Testing, Analysis, and Verification, pp. 28-37. ACM Press, New York (1989)
23. Höst, M., Wohlin, C., Thelin, T.: Experimental context classification: Incentives and experience of subjects. In: ICSE '05: Proceedings of the 27th International Conference on Software Engineering, pp. 470-478 (2005)
24. Hutchins, M., Foster, H., Goradia, T., Ostrand T.: Experiments of the effectiveness of dataflow- and controlflow-based test adequacy criteria. In ICSE '94: Proceedings of the 16th International Conference on Software Engineering, pp. 191-200. IEEE Computer Society Press, Los Alamitos (1994)
25. JavaCodeCoverage. http://atulkg.googlepages.com/JavaCodeCoverage/
26. JUnit Home Page. http://www.junit.org
27. Juristo N. and Moreno A.M. (2001). Basics of Software Engineering Experimentation. Springer, Heidelberg
28. Juristo N., Moreno A.M. and Vegas S. (2004). Reviewing 25 years of testing technique experiments. Empirical Softw. Engg. 9(1-2): 7-44
29. Juristo N., Moreno A.M. and Vegas S. (2004). Towards building a solid empirical body of knowledge in testing techniques. SIGSOFT Softw. Eng. Notes 29(5): 1-4
30. Juristo N., Moreno A.M., Vegas S. and Solari M. (2006). In search of what we experimentally know about unit testing. IEEE Softw. 23(6): 72-80
31. Kamsties, E., Lott, C.M.: An empirical evaluation of three defect-detection techniques. In: ESEC '95: Proceedings of the 5th European Software Engineering Conference, pp. 362-383. Springer, London (1995)
32. Kapoor K. and Bowen J.P. (2004). Experimental evaluation of the tolerance for control-flow test criteria: Research articles. Softw. Test. Verif. Reliab. 14(3): 167-187
33. Kim, S., Clark, J., McDermid, J.: The rigorous generation of java mutation operators using hazop. In: Proceedings of ICSSEA, pp. 9-10 (1999)
34. Kitchenham B.A., Pfleeger S.L., Pickard L.M., Jones P.W., Hoaglin D.C., Emam K.E. and Rosenberg J. (2002). Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. 28(8): 721-734
35. Lingampally, R., Gupta, A., Jalote, P.: A multipurpose code coverage tool for java. In: HICSS '07: Proceedings of 40th Annual Hawaii International Conference on System Sciences; minitrack on Automated Software Testing and Analysis: Techniques, Practices and Tools, pp. 261-270. IEEE Computer Society, Los Alamitos (2007)
36. MaA Y.-S., Offutt, J., Kwon, Y.-R.: Mujava: a mutation system for java. In: ICSE '06: Proceeding of the 28th International Conference on Software Engineering, pp. 827-830. ACM Press, New York (2006)
37. Miller, J.: Can results from software engineering experiments be safely combined? In: METRICS '99: Proceedings of the 6th International Symposium on Software Metrics, pp. 152-158. IEEE Computer Society, Washington (1999)
38. Monopoly - RealEstate. http://open.ncsu.edu/se/monopoly/
39. Montgomery D.C. (2001). Design and Analysis of Experiments. Wiley, London
40. MuJava Home Page. http://www.ise.gmu.edu/~ofut/mujava/
41. Murray, L., Carrington, D.A., MacColl, I., McDonald, J., Strooper, P.A.: Formal derivation of finite state machines for class testing. In: ZUM, pp. 42-59 (1998)
42. MySQL Home Page. http://www.mysql.com/
43. Ntafos, S.C.: An evaluation of required element testing strategies. In: ICSE '84: Proceedings of the 7th International Conference on Software Engineering, pp. 250-256. IEEE Press, Piscataway (1984)
44. Offutt A.J. (1992). Investigations of the software testing coupling effect. ACM Trans. Softw. Eng. Methodol. 1(1): 5-20
45. Offutt, A.J., Xiong, Y., Liu, S.: Criteria for generating specification-based tests. In: ICECCS '99: 5th International Conference on Engineering of Complex Computer Systems, pp. 119-129. IEEE Computer Society, Washington (1999)
46. Shull, F., Basili, V., Carver, J., Maldonado, J.C., Travassos, G.H., Mendonça, M., Fabbri, S.: Replicating software engineering experiments: Addressing the tacit knowledge problem. In: ISESE '02: Proceedings of the 2002 International Symposium on Empirical Software Engineering, pp. 7-16. IEEE Computer Society, Washington (2002)
47. Sjoberg D.I.K., Hannay J.E., Hansen O., Kampenes V.B., Karahasanovic A., Liborg N.-K. and Rekdal A.C. (2005). A survey of controlled experiments in software engineering. IEEE Trans. Softw. Eng. 31(9): 733-753
48. Stocks P. and Carrington D.A. (1996). A framework for specification-based testing. IEEE Trans. Softw. Eng. 22(11): 777-793
49. Weyuker, E.: Can we measure software testing effectiveness? In: Metrics '93: Proceedings of 1st International Symposium on Software Metrics, pp. 100-107. IEEE Computer Society, Washington (1993)
50. Weyuker, E.J., Weiss, S.N., Hamlet, D.: Comparison of program testing strategies. In: TAV4: Proceedings of the Symposium on Testing, Analysis, and Verification, pp. 1-10. ACM Press, New York (1991)
51. Wohlin C., Runeson P., Host M., Ohlsson M.C., Regnell B. and Wesslen A. (2000). Experimentation in Software Engineering - An Introduction. Kluwer, Dordrecht
52. Wood, M., Roper, M., Brooks, A., Miller, J.: Comparing and combining software defect detection techniques: A replicated empirical study. In: ESEC '97/FSE-5: Proceedings of the 6th European Conference/5th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 262-277. Springer, New York (1997)
53. Zhu H., Hall P.A.V. and May J.H.R. (1997). Software unit test coverage and adequacy. ACM Comput. Surv. 29(4): 366-427