Assessing, comparing, and combining state machine-based testing and structural testing: A series of experiments

IEEE Transactions on Software Engineering

Volumn 37, Issue 2, 2011, Pages 161-187

  Mouchawrab, Samar ^a   Briand, Lionel C ^b   Labiche, Yvan ^a   Di Penta, Massimiliano ^c  

^a CARLETON UNIVERSITY   (Canada)

^b SIMULA RESEARCH LABORATORY   (Norway)

^c UNIVERSITY OF SANNIO   (Italy)

Author keywords

controlled experiments; state machines; State based software testing; structural testing

Indexed keywords

CONTROLLED EXPERIMENT; COVERAGE CRITERIA; FAULT DETECTION EFFECTIVENESS; MODEL BASED TESTING; MODEL-BASED; QUALITATIVE ANALYSIS; SOFTWARE ORGANIZATION; SPECIFIC STATE; STATE MACHINE; STATE MACHINE MODELS; STATE MACHINES; STATE-BASED SOFTWARE TESTING; STRUCTURAL TESTING; TEST STRATEGIES; TEST TECHNIQUES;

COMPUTER SOFTWARE SELECTION AND EVALUATION; CONTOUR FOLLOWERS; COST BENEFIT ANALYSIS; COSTS; FAULT DETECTION; MODELS; QUALITY CONTROL; SOFTWARE ENGINEERING; SOFTWARE TESTING;

EXPERIMENTS;

EID: 79953189406     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2010.32     Document Type: Article

References (69)

1. Eclipse, http://www.eclipse.org, 2009.
2. Eclipse Metrics Plugin, http://sourceforge.net/projects/metrics, 2009.
3. Eclipse Test and Performance Tools Platform Project (TPTP), http://www.eclipse.org/tptp, 2009.
4. Software-Artifact Infrastructure Repository (SIR), http://sir.unl. edu/portal/index.html, 2009.
5. J.H. Andrews, L.C. Briand, and Y. Labiche, "Is Mutation an Appropriate Tool for Testing Experiments?" Proc. 27th Int'l Conf. Software Eng., pp. 402-411, 2005.
6. J.H. Andrews, L.C. Briand, Y. Labiche, and A.S. Namin, "Using Mutation Analysis for Assessing and Comparing Testing Coverage Criteria," IEEE Trans. Software Eng., vol. 32, no. 8, pp. 608-624, Aug. 2006. (Pubitemid 46405276)
7. E. Arisholm and D.I. Sjøberg, "Evaluating the Effect of a Delegated versus Centralized Control Style on the Maintainability of Object-Oriented Software," IEEE Trans. Software Eng., vol. 30, no. 8, pp. 521-534, Aug. 2004.
8. D. Baldwin and F. Sayward, "Heuristics for Determining Equivalence of Program Mutations," Dept. of Computer Science 276, Yale Univ., 1979.
9. V.R. Basili, F. Shull, and F. Lanubile, "Building Knowledge through Families of Experiments," IEEE Trans. Software Eng., vol. 25, no. 4, pp. 456-473, July/Aug. 1999. (Pubitemid 129304112)
10. B. Beizer, Software Testing Techniques, second ed. Van Nostrand Reinhold Co., 1990.
11. J.M. Bieman and J.L. Schultz, "An Empirical Evaluation (and Specification) of the All-du-Paths Testing Criterion," ACM Software Eng. J., vol. 7, no. 1, pp. 43-51, 1992.
12. R.V. Binder, Testing Object-Oriented Systems-Models, Patterns, and Tools. Addison-Wesley, 1999.
13. K. Bogdanov and M. Holcombe, "Statechart Testing Method for Aircraft Control Systems," Software Testing, Verification and Reliability, vol. 11, no. 1, pp. 39-54, 2001. (Pubitemid 32325739)
14. L.C. Briand, "A Critical Analysis of Empirical Research in Software Testing," Proc. Int'l Symp. Empirical Software Eng. and Measurement, Sept. 2007.
15. L.C. Briand, M. Di Penta, and Y. Labiche, "Assessing and Improving State-Based Class Testing: A Series of Experiments," IEEE Trans. Software Eng., vol. 30, no. 11, pp. 770-783, Nov. 2004.
16. L.C. Briand and Y. Labiche, "A UML-Based Approach to System Testing," Software and Systems Modeling, vol. 1, no. 1, pp. 10-42, 2002.
17. L.C. Briand, Y. Labiche, and J. Cui, "Automated Support for Deriving Test Requirements from UML Statecharts," J. Software and Systems Modeling, special issue, vol. 4, no. 4, pp. 399-423, 2005. (Pubitemid 43118835)
18. L.C. Briand, Y. Labiche, and Q. Lin, "Improving State-Based Coverage Criteria Using Data Flow Information," Proc. IEEE Int'l Conf. Software Reliability Eng., pp. 95-104, 2005. (Pubitemid 44733893)
19. L.C. Briand, Y. Labiche, and H. Sun, "Investigating the Use of Analysis Contracts to Improve the Testability of Object-Oriented Code," Software-Practice and Experience, vol. 33, no. 7, pp. 637-672, 2003.
20. L.C. Briand, Y. Labiche, and Y. Wang, "Using Simulation to Empirically Investigate State Coverage Criteria Based on Statecharts," Proc. ACM Int'l Conf. Software Eng., pp. 86-95, May 2004.
21. B. Bruegge and A.H. Dutoit, Object-Oriented Software Engineering: Using UML, Patterns and Java. Prentice Hall, 2004.
22. P. Chevalley and P. Thevenod-Fosse, "Automated Generation of Statistical Test Cases from UML State Diagrams," Proc. 25th Ann. Int'l Computer Software and Applications Conf., pp. 205-214, 2001. (Pubitemid 33068617)
23. P. Chevalley and P. Thevenod-Fosse, "An Empirical Evaluation of Statistical Testing Designed from UML State Diagrams: The Flight Guidance System Case Study," Proc. 12th Int'l Symp. Software Reliability Eng., pp. 254-263, 2001. (Pubitemid 34080714)
24. T.S. Chow, "Testing Software Design Modeled by Finite-State Machines," IEEE Trans. Software Eng., vol. 4, no. 3, pp. 178-187, May 1978.
25. J. Cohen, Statistical Power Analysis for the Behavioral Sciences, second ed. Lawrence Erlbaum Assoc. Inc., 1988.
26. J. Daly, A. Brooks, J. Miller, M. Roper, and M. Wood, "Verification of Results in Software Maintenance through External Replication," Proc. Int'l Conf. Software Maintenance, pp. 50-57, 1994.
27. R.A. DeMillo, R.J. Lipton, and F.G. Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," Computer, vol. 11, no. 4, pp. 34-41, Apr. 1978.
28. J.L. Devore and N. Farnum, Applied Statistics for Engineers and Scientists, 1999.
29. T. Dybå, V.B. Kampenes, and D.I.K. Sjøberg, "A Systematic Review of Statistical Power in Software Engineering Experiments," Information and Software Technology, vol. 48, no. 8, pp. 745-755, 2006. (Pubitemid 44218930)
30. G.A. Ferguson and Y. Takane, Statistical Analysis in Psychology and Education, sixth ed. McGraw-Hill Ryerson Limited, 2005.
31. P.G. Frankl and S.N. Weiss, "An Experimental Comparison of the Effectiveness of the All-Uses and All-Edges Adequacy Criteria," Proc. Fourth Symp. Software Testing, Analysis and Verification, pp. 154-164, Oct. 1991.
32. P.G. Frankl, S.N. Weiss, and C. Hu, "All-Uses versus Mutation Testing: An Experimental Comparison of Effectiveness," Systems and Software, vol. 38, no. 3, pp. 235-253, 1997. (Pubitemid 127384432)
33. H. Gomaa, Designing Concurrent, Distributed, and Real-Time Applications with UML. Addison-Wesley Professional, 2000.
34. R.J. Harris, A Primer of Multivariate Statistics. Lawrence Erlbaum Assoc., 2001.
35. S. Holm, "A Simple Sequentially Rejective Multiple Test Procedure," Scandinavian J. Statistics, vol. 6, pp. 65-70, 1979.
36. N. Holt, B. Anda, K. Asskildt, L.C. Briand, J. Endresen, and S. Frøystein, "Experiences with Precise State Modeling in an Industrial Safety Critical System," Proc. Critical Systems Development Using Modeling Languages, Workshop in Conjunction with UML '06, pp. 68-77, 2006.
37. R. Holt, W.D. Boehm-Davis, and A.C. Shultz, "Mental Representations of Programs for Student and Professional Programmers," Empirical Studies of Programmers: Second Workshop, pp. 33-46, Ablex Publishing Corp., 1987.
38. H.S. Hong, Y.G. Kim, S.D. Cha, D.H. Bae, and H. Ural, "A Test Sequence Selection Method for Statecharts," Software Testing, Verification and Reliability, vol. 10, no. 4, pp. 203-227, 2000. (Pubitemid 32137043)
39. M. Höst, B. Regnell, and C. Wohlin, "Using Students as Subjects-A Comparative Study of Students and Professionals in Lead-Time Impact Assessment," Empirical Software Eng., vol. 5, no. 3, pp. 201-214, 2000. (Pubitemid 32085481)
40. M. Hutchins, H. Foster, T. Goradia, and T. Ostrand, "Experiments of the Effectiveness of Dataflow-and Controlflow-Based Test Adequacy Criteria," Proc. 16th Int'l Conf. Software Eng., 1994.
41. A. Kleppe, J. Warmer, and W. Bast, MDA Explained: The Model Driven Architecture: Practice and Promise. Addison-Wesley Professional, 2003.
42. C. Larman, Applying UML and Patterns: An Introduction to Object-Oriented Analysis and Design and Iterative Development. Prentice Hall, 2004.
43. D. Lee and M. Yannakakis, "Principles and Methods of Testing Finite State Machines-A Survey," Proc. IEEE, vol. 84, no. 8, pp. 1090-1123, Aug. 1996. (Pubitemid 126752579)
44. P.S. Levy and S. Lemeshow, Sampling of Populations: Methods and Applications, third ed. Wiley, 1999.
45. Y.-S. Ma, Y.-R. Kwon, and J. Offutt, "Inter-Class Mutation Operators for Java," Proc. 13th Int'l Symp. Software Reliability Eng., Nov. 2002.
46. Y.-S. Ma and J. Offutt, "Description of Method-Level Mutation Operators for Java," http://cs.gmu.edu/~offutt/mujava/mutopsMethod.pdf, 2010.
47. Y.-S. Ma, J. Offutt, and Y.R. Kwon, "MuJava: An Automated Class Mutation System," Software Testing, Verification and Reliability, vol. 15, no. 2, pp. 97-133, 2005. (Pubitemid 40858060)
48. B. Marick, Craft of Software Testing: Subsystems Testing Including Object-Based and Object-Oriented Testing. Prentice-Hall, 1985.
49. P. McMinn and M. Holcombe, "Evolutionary Testing Using an Extended Chaining Approach," Evolutionary Computation, vol. 14, no. 1, pp. 41-64, 2006. (Pubitemid 43955272)
50. B. Meyer, "Applying Design by Contract," Computer, vol. 25, no. 10, pp. 40-51, Oct. 1992.
51. S. Mouchawrab, L.C. Briand, Y. Labiche, and M. Di Penta, "Assessing, Comparing, and Combining State Machine-Based Testing and Structural Testing: A Series of Experiments," Carleton Univ. TR SCE-08-09, 2009.
52. K.R. Murphy and B. Myors, Statistical Power Analysis: A Simple and General Model for Traditional and Modern Hypothesis Tests. Lawrence Erlbaum, 1998.
53. C. Nebut, F. Fleurey, Y. Le Traon, and J.-M. Jezequel, "Automatic Test Generation: A Use Case Driven Approach," IEEE Trans. Software Eng., vol. 32, no. 3, pp. 140-155, Mar. 2006.
54. A.J. Offutt and A. Abdurazik, "Generating Tests from UML Specifications," Proc. Second Int'l Conf. Unified Modeling Language, pp. 416-429, 1999.
55. A.J. Offutt and W.M. Craft, "Using Compiler Optimization Techniques to Detect Equivalent Mutants," J. Software Testing, Verification and Reliability, vol. 4, no. 3, pp. 131-154, 1994.
56. J. Offutt, S. Liu, A. Abdurazik, and P. Ammann, "Generating Test Data from State-Based Specifications," J. Software Testing, Verification and Reliability, vol. 13, no. 1, pp. 25-53, 2003.
57. J. Offutt and J. Pan, "Automatically Detecting Equivalent Mutants and Infeasible Paths," Software Testing, Verification and Reliability, vol. 7, no. 3, pp. 165-192, 1997. (Pubitemid 127661584)
58. T.J. Ostrand and M.J. Balcer, "The Category-Partition Method for Specifying and Generating Fuctional Tests," Comm. ACM, vol. 31, no. 6, pp. 676-686, 1988.
59. P. Piwowarski, M. Ohba, and J. Caruso, "Coverage Measurement Experience during Function Test," Proc. 15th Int'l Conf. Software Eng., 1998.
60. A. Pretschner, W. Prenninger, S. Wagner, C. Kü hnel, M. Baumgartner, B. Sostawa, R. Zölch, and T. Stauner, "One Evaluation of Model-Based Testing and Its Automation," Proc. Int'l Conf. Software Eng., pp. 392-401, 2005.
61. F. Shull, J. Singer, and D.I.K. Sjøberg, Guide to Advanced Empirical Software Engineering. Springer, 2008.
62. S. Siegel, Non-Parametric Statistics for the Behavioral Sciences. McGraw-Hill, 1956.
63. S. Sinha and M.J. Harrold, "Analysis and Testing of Programs with Exception Handling Constructs," IEEE Trans. Software Eng., vol. 26, no. 9, pp. 849-871, Sept. 2000.
64. L.E. Toothaker, Introductory Statistics for the Behavioral Sciences, second ed. McGraw-Hill College, 1986.
65. M. Utting and B. Legeard, Practical Model-Based Testing: A Tools Approach. Morgan-Kaufmann, 2006.
66. J. Warmer and A. Kleppe, The Object Constraint Language: Getting Your Models Ready for MDA. Addison-Wesley Professional, 2004.
67. E. Weyuker, "The Cost of Data Flow Testing: An Empirical Study," IEEE Trans. Software Eng., vol. 16, no. 2, pp. 121-128, Feb. 1990. (Pubitemid 20666024)
68. E. Weyuker, T. Goradia, and A. Singh, "Automatically Generating Test Data from a Boolean Specification," IEEE Trans. Software Eng., vol. 20, no. 5, pp. 353-363, May 1994.
69. C. Wohlin, P. Runeson, M. Host, M.C. Ohlsson, B. Regnell, and A. Wesslen, Experimentation in Software Engineering: An Introduction. Kluwer, 2000.