Efficient software verification: Statistical testing using automated search

IEEE Transactions on Software Engineering

Volumn 36, Issue 6, 2010, Pages 763-777

  Poulding, Simon ^a   Clark, John A ^a  

^a UNIVERSITY OF YORK   (United Kingdom)

Author keywords

optimization; Software program verification; test coverage of code; testing strategies

Indexed keywords

AUTOMATED SEARCHES; DYNAMIC TESTING; HIGH PROBABILITY; PRACTICAL METHOD; REAL-WORLD; SOFTWARE VERIFICATION; SOFTWARE/PROGRAM VERIFICATION; STATISTICAL TESTING; STRUCTURAL TESTING; TEST COVERAGE; TEST DATA; TEST SETS; TESTING STRATEGIES;

AUTOMATIC TEST PATTERN GENERATION; COMPUTER SOFTWARE SELECTION AND EVALUATION; OPTIMIZATION; TEST FACILITIES; TESTING; VERIFICATION;

PROBABILITY DISTRIBUTIONS;

EID: 78649786792     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2010.24     Document Type: Article

References (48)

1. P. Thévenod-Fosse and H. Waeselynck, "An Investigation of Statistical Software Testing," J. Software Testing, Verification and Reliability, vol. 1, no. 2 pp. 5-26, 1991.
2. P. Thévenod-Fosse, "Software Validation by Means of Statistical Testing: Retrospect and Future Direction," Dependable Computing for Critical Applications, A. Avizienis and J. C. Laprie, eds., pp. 23-50, Springer, 1991.
3. P. Thévenod-Fosse, H. Waeselynck, and Y. Crouzet, "Software Statistical Testing," Technical Report 95178, Laboratoire d'Analyse et d'Architecture des Systèmes du CNRS (LAAS), 1995.
4. P. Thévenod-Fosse and H. Waeselynck, "Statemate Applied to Statistical Testing," Proc. Int'l Symp. Software Testing and Analysis, pp. 99-109, 1993.
5. M. Harman and B.F. Jones, "Search-Based Software Engineering," Information and Software Technology, vol. 43, pp. 833-839, 2001.
6. J. Clark, J.J. Dolado, M. Harman, R. Hierons, B. Jones, M. Lumkin, B. Mitchell, S. Mancoridis, K. Rees, M. Roper, and M. Shepperd, "Reformulating Software Engineering as a Search Problem," IEE Proc. Software, vol. 150, no. 3, pp. 161-175, 2003.
7. Y. Zhang, M. Harman, and S.A. Mansouri, "The Multi-Objective Next Release Problem," Proc. Genetic and Evolutionary Computation Conf., pp. 1129-1137, 2007.
8. E. Alba and F. Chicano, "Software Project Management with GAs," Information Sciences, vol. 177, no. 11, pp. 2380-2401, 2007.
9. P. Emberson and I. Bate, "Minimising Task Migrations and Priority Changes in Mode Transitions," Proc. 13th IEEE Real-Time and Embedded Technology and Applications Symp., pp. 158-167, 2007.
10. M. Harman and L. Tratt, "Pareto Optimal Search Based Refactoring at the Design Level," Proc. Genetic and Evolutionary Computation Conf., pp. 1106-1113, 2007.
11. C. Hao, J.A. Clark, and J.L. Jacob, "Synthesising Efficient and Effective Security Protocols," Proc. Workshop Automated Reasoning for Security Protocol Analysis, pp. 25-40, 2004.
12. P. McMinn, "Search-Based Software Test Data Generation: A Survey," Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105-156, 2004.
13. D. White, J.A. Clark, J. Jacob, and S. Poulding, "Evolving Software in the Presence of Resource Constraints," Proc. Genetic and Evolutionary Computation Conf., pp. 1775-1782, 2008.
14. M. Harman, "The Current State and Future of Search Based Software Engineering," Proc. 29th Int'l Conf. Software Eng. Future of Software Eng., pp. 342-357, 2007. (Pubitemid 47484936)
15. J. Duran and S. Ntafos, "An Evaluation of Random Testing," IEEE Trans. Software Eng., vol. 10, no. 4, pp. 438-444, July 1984.
16. D. Hamlet, "When Only Random Testing Will Do," Proc. First Int'l Workshop Random Testing, pp. 1-9, 2006.
17. H. Zhu, P.A.V. Hall, and J.H.R. May, "Software Unit Test Coverage and Adequacy," ACM Computing Surveys, vol. 29, no. 4, pp. 366-427, 1997.
18. M.J. Harrold, R. Gupta, and M.L. Soffa, "A Methodology for Controlling the Size of a Test Suite," ACM Trans. Software Eng. and Methodology, vol. 2, no. 3, pp. 270-285, 1993.
19. J.A. Jones and M.J. Harrold, "Test-Suite Reduction and Prioritization for Modified Condition/Decision Coverage," IEEE Trans. Software Eng., vol. 29, no. 3, pp. 195-209, Mar. 2003.
20. S. Ntafos, "On Random and Partition Testing," ACM SIGSOFT Software Eng. Notes, vol. 23, no. 2, pp. 42-48, 1998.
21. DO-178B, Software Considerations in Airborne Systems and Equipment Certification, Radio Technical Commission for Aeronautics, 1992.
22. Defence Standard 00-55(PART2)/Issue 2, Requirements for Safety Related Software in Defence Equipment, Part 2: Guidance, Ministry of Defence, 1997.
23. P. Thévenod-Fosse and H. Waeselynck, "Towards a Statistical Approach to Testing Object-Oriented Programs," Proc. IEEE Ann. Int'l Symp. Fault Tolerant Computing, pp. 99-108, 1997.
24. A. Denise, M.-C. Gaudel, and S.-D. Gouraud, "A Generic Method for Statistical Testing," Technical Report 1386, Laboratoire de Recherche en Informatique (LRI), CNRS - Univ. de Paris Sud, Apr. 2004.
25. N. Baskiotis, M. Sebag, M.-C. Gaudel, and S. Gouraud, "A Machine Learning Approach for Statistical Software Testing," Proc. Int'l Joint Conf. Artificial Intelligence, pp. 2274-2279, 2007.
26. R.P. Pargas, M.J. Harrold, and R.R. Peck, "Test-Data Generation Using Genetic Algorithms," Software Testing, Verification and Reliability, vol. 9, pp. 263-282, 1999.
27. B. Korel, "Automatic Software Test Data Generation," IEEE Trans. Software Eng., vol. 16, no. 8, pp. 870-879, Aug. 1990. (Pubitemid 20738724)
28. N. Tracey, J. Clark, and K. Mander, "Automated Program Flaw Finding Using Simulated Annealing," ACM SIGSOFT Software Eng. Notes, vol. 23, no. 2, pp. 73-81, 1998.
29. N.J. Tracey, "A Search-Based Automated Test-Data Generation Framework for Safety-Critical Software," PhD dissertation, Dept. of Computer Science, Univ. of York, 2000.
30. J. Wegener, A. Baresel, and H. Sthamer, "Evolutionary Test Environment for Automatic Structural Testing," Information and Software Technology, vol. 43, pp. 841-854, 2001.
31. S. Forrest and M. Mitchell "Relative Building-Block Fitness and the Building Block Hypothesis," Foundations of Genetic Algorithms 2, D.L. Whitley, ed., pp. 109-126, Morgan Kaufmann, 1993.
32. M. Galassi et al., GNU Scientific Library Reference Manual, second ed. Network Theory, 2006.
33. "JDK 1.4 Demo Applets." http://java.sun.com/applets/jdk/1.4/ index.html, 2010.
34. "WCET Analysis Project." http://www.mrtc.mdh.se/projects/wcet/ benchmarks.html, 2010.
35. M.E. Delamaro and J.C. Maldonado, "Proteum/IM 2.0: An Integrated Mutation Testing Environment," Mutation Testing for the New Century, pp. 91-101, Kluwer Academic Publishers, 2001.
36. T.J. DiCiccio and B. Efron, "Bootstrap Confidence Intervals," Statistical Science, vol. 11, no. 3, pp. 189-212, 1996.
37. N.L. Leech and A.J. Onwuegbuzie, "A Call for Greater Use of Nonparametric Statistics," technical report, US Dept. of Education, Educational Resources Information Center, 2002.
38. F. Wilcoxon, "Individual Comparisons by Ranking Methods," Biometrics Bull., vol. 1, no. 6, pp. 80-83, 1945.
39. A. Vargha and H. Delaney, "A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong," J. Educational and Behavioral Statistics, vol. 25, no. 2, pp. 101-132, 2000.
40. E.W. Weisstein, "'Bernoulli Distribution.' From MathWorld - A Wolfram Web Resource." http://mathworld.wolfram.com/BernoulliDistribution. html, 2010.
41. E.W. Weisstein, "'Geometric Distribution.' From MathWorld - A Wolfram Web Resource." http://mathworld.wolfram.com/GeometricDistribution. html, 2010.
42. A.E. Eiben, Z. Michalewicz, M. Schoenauer, and J.E. Smith, "Parameter Control in Evolutionary Algorithms," Parameter Setting in Evolutionary Algorithms, F.G. Lobo, C.F. Lima, and Z. Michalewicz, eds., vol. 54, pp. 19-46, Springer, 2007.
43. W.J. Gutjahr and G.C. Pflug, "Simulated Annealing for Noisy Cost Functions," J. Global Optimization, vol. 8, no. 1, pp. 1-13, 1996.
44. S. Rana, D.L. Whitley, and R. Cogswell, "Searching in the Presence of Noise," Parallel Problem Solving from Nature, pp. 198-207, Springer, 1996.
45. H.-G. Beyer, "Evolutionary Algorithms in Noisy Environments: Theoretical Issues and Guidelines for Practice," Computer Methods in Applied Mechanics and Eng., vol. 186, pp. 239-267, 2000.
46. S. Markon, D.V. Arnold, T. Bäck, T. Beielstein, and H.-G. Beyer, "Thresholding - A Selection Operator for Noisy ES," Proc. 2001 IEEE Congress on Evolutionary Computation, pp. 465-472, 2001.
47. S. Yoo and M. Harman, "Pareto Efficient Multi-Objective Test Case Selection," Proc. 2007 Int'l Symp. Software Testing and Analysis, pp. 140-150, 2007.
48. K. Lakhotia, M. Harman, and P. McMinn, "Handling Dynamic Data Structures in Search Based Testing," Proc. Genetic and Evolutionary Computation Conf., pp. 1759-1766, 2008.