Finding faults: Manual testing vs. random+ testing vs. user reports

Proceedings - International Symposium on Software Reliability Engineering, ISSRE

Volumn , Issue , 2008, Pages 157-166

  Ciupa, Ilinca ^a   Meyer, Bertrand ^a   Oriol, Manuel ^a   Pretschner, Alexander ^a  

^a ETH ZURICH   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

EMPIRICAL STUDIES; INCIDENT REPORTS; MANUAL TESTS; RANDOM TESTS; TESTING STRATEGIES; THREE DIFFERENT TECHNIQUES;

COMPUTER SOFTWARE SELECTION AND EVALUATION;

SOFTWARE RELIABILITY;

EID: 67249134092     PISSN: 10719458     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ISSRE.2008.18     Document Type: Conference Paper

References (41)

1. IEEE standard classification for software anomalies. IEEE Std 1044-1993, 2 Jun 1994.
2. E. Allen. Bug Patterns in Java. APress L. P., 2002.
3. S. Artzi, M. Ernst, K. A, C. Pacheco, and J. Perkins. Finding the needles in the haystack: Generating legal test inputs for object-oriented programs. In Proc. 1st Workshop on Model- Based Testing and Object-Oriented Systems, 2006.
4. M. Barnett, K. R. M. Leino, and W. Schulte. The spec# programming system: An overview. In Proc. CASSIS, 2004.Springer LNCS 3362.
5. V. Basili and R. Selby. Comparing the effectiveness of software testing strategies. IEEE TSE, 13(12):1278-1296, 1987.
6. B. Beizer. Software Testing Techniques. John Wiley & Sons, Inc., New York, NY, USA, 1990.
7. E. Bernard, B. Legeard, X. Luck, and F. Peureux. Generation of test sequences from formal specifications: GSM 11-11 standard case study. Software-Practice & Experience, 34(10):915-948, August 2004.
8. C. Boyapati, S. Khurshid, and D. Marinov. Korat: automated testing based on Java predicates. In Proc. Intl. Symp. on Software Testing and Analysis, pages 123-133, 2002.
9. P. Chalin. Are practitioners writing contracts? In Springer LNCS 4157, pages 100-113, 2006.
10. R. Chillarege, I. Bhandari, J. Chaar, M. Halliday, D. Moebus, B. Ray, and M.-Y. Wong. Orthogonal defect classification concept for in-process measurements. IEEE TSE, 18(11):943-956, 1992. (Pubitemid 23608410)
11. I. Ciupa, A. Leitner, M. Oriol, and B. Meyer. Experimental assessment of random testing for object-oriented software. In Proc. ISSTA, pages 84-94, 2007. (Pubitemid 47317582)
12. I. Ciupa, A. Pretschner, A. Leitner, M. Oriol, and B. Meyer. On the predictability of random tests for object-oriented software. In Proc. ICST, pages 72-81, 2008.
13. C. Csallner and Y. Smaragdakis. JCrasher: an automatic robustness tester for Java. Software: Practice and Experience, 34(11):1025-1050, 2004.
14. C. Csallner and Y. Smaragdakis. DSD-Crasher: A Hybrid Analysis Tool for Bug Finding. In Proc. Intl. Symp. on Software Testing and Analysis, pages 245-254, July 2006. (Pubitemid 46645535)
15. J. Duran and S. Ntafos. An Evaluation of Random Testing. IEEE TSE, SE-10(4):438-444, July 1984. (Pubitemid 15421446)
16. M. D. Ernst. Dynamically Discovering Likely Program Invariants. PhD thesis, University of Washington Department of Computer Science and Engineering, August 2000.
17. P. Frankl and S. Weiss. An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing. IEEE TSE, 19(8):774-787, 1993.
18. P. Frankl, S. Weiss, and C. Hu. All-uses versus mutation testing: An experimental comparison of effectiveness, 1994.
19. P. Frankl and E. Weyuker. An Applicable Family of Data Flow Testing Criteria. IEEE TSE, 14(10):1483-1498, 1998.
20. M. Girgis and M. Woodward. An experimental comparison of the error exposing ability of program testing criteria. In Proc. IEEE/ACM workshop on software testing, pages 64- 73, July 1986.
21. J. Gray. Why do computers stop and what can be done about it? In Symposium on Reliability in Distributed Software and Database Systems, pages 3-12, 1986.
22. W. Gutjahr. Partition testing versus random testing: the influence of uncertainty. IEEE TSE, 25(5):661-674, 1999. (Pubitemid 30541839)
23. D. Hamlet and R. Taylor. Partition Testing Does Not Inspire Confidence. IEEE TSE, 16(12):1402-1411, Dec. 1990.
24. S. Hangal and M. S. Lam. Tracking down software bugs using automatic anomaly detection. In ICSE 02: Proceedings of the 24th International Conference on Software Engineering, pages 291-301, New York, NY, USA, 2002. ACM. (Pubitemid 35009118)
25. M. Heimdahl, D. George, and R. Weber. Specification Test Coverage Adequacy Criteria = Specification Test Generation Inadequacy Criteria? In Proc. 8th IEEE High Assurance in Systems Engineering Workshop, February 2004.
26. K. Henningsson and C.Wohlin. Assuring fault classification agreement - an empirical evaluation. In Proc. Intl. Symp. On Empirical Software Engineering, pages 95-104, 2004. (Pubitemid 40062822)
27. D. Hovemeyer andW. Pugh. Finding bugs is easy. SIGPLAN Not., 39(12):92-106, 2004. (Pubitemid 40303302)
28. M. Hutchins, H. Foster, T. Goradia, and T. Ostrand. Experiments of the effectiveness of dataflow- and controlflowbased test adequacy criteria. In Proc. ICSE, pages 191-200, 1994.
29. E. Kamsties and C. Lott. An empirical evaluation of three defect-detection techniques. In Proc. ESEC, pages 362-383, 1995.
30. A. Leitner and I. Ciupa. AutoTest. se.inf.ethz.ch/ people/leitner/auto- test/, 2005 - 2007.
31. R. Lutz. Targeting Safety-Related Errors During Software Requirements Analysis. In Proc. ACM SIGSOFT FSE, pages 99-106, 1993.
32. Y. Ma, Y. Kwon, and J. Offutt. Inter-Class Mutation Operators for Java. In Proc. ISSRE, pages 352-366, 2002.
33. B. Meyer. Attached types and their application to three open problems of object-oriented programming. In Proc. ECOOP, pages 1-32, 2005. Springer LNCS 3586. (Pubitemid 41435974)
34. C. Oriat. Jartege: a tool for random generation of unit tests for java classes. Technical Report RR-1069-I, CNRS, Universite Joseph Fourier Grenoble I, June 2004.
35. C. Pacheco and M. D. Ernst. Eclat: Automatic generation and classification of test inputs. In Proc. ECOOP, pages 504-527, 2005. (Pubitemid 41435995)
36. C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedbackdirected random test generation. In Proc. ICSE, pages 75-84, 2007.
37. J. Ploski, M. Rohr, P. Schwenkenberg, and W. Hasselbring. Research issues in software fault categorization. SIGSOFT Softw. Eng. Notes, 32(6):6, 2007.
38. A. Pretschner, W. Prenninger, S. Wagner, C. Kühnel, M. Baumgartner, B. Sostawa, R. Zölch, and T. Stauner. One evaluation of model-based testing and its automation. In Proc. ICSE, pages 392-401, 2005.
39. M. Utting, A. Pretschner, and B. Legeard. A taxonomy of model-based testing. Technical Report 04/2006, Department of Computer Science, The University ofWaikato (New Zealand), April 2006.
40. E. Weyuker and B. Jeng. Analyzing Partition Testing Strategies. IEEE TSE, 17(7):703-711, 1991. (Pubitemid 21674681)
41. M. Wood, M. Roper, A. Brooks, and J. Miller. Comparing and combining software defect detection techniques: a replicated empirical study. SIGSOFT Softw. Eng. Notes, 22(6):262-277, 1997. (Pubitemid 127132032)