A theoretical and empirical study of search-based testing: Local, global, and hybrid search

IEEE Transactions on Software Engineering

Volumn 36, Issue 2, 2010, Pages 226-247

  Harman, Mark ^a   McMinn, Phil ^b  

^a KING'S COLLEGE LONDON   (United Kingdom)

^b UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Artificial intelligence; Automated test data generation; Evolutionary testing; Genetic algorithms; Hill climbing; Problem solving; Royal road; Schema theory; Search based software engineering; Search based testing; Testing and debugging; Testing tools

Indexed keywords

AUTOMATED TEST DATA GENERATION; EVOLUTIONARY TESTING; HILL CLIMBING; SCHEMA THEORY; SEARCH-BASED SOFTWARE ENGINEERING; TESTING AND DEBUGGING; TESTING TOOLS;

COMPUTER SOFTWARE; DATA COMMUNICATION SYSTEMS; GENETIC ALGORITHMS; OPTIMIZATION; PROGRAM DEBUGGING; ROADS AND STREETS; TEST FACILITIES;

PROBLEM SOLVING;

EID: 77950626862     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2009.71     Document Type: Article

References (61)

1. The Software-Artifact Infrastructure Repository, http://sir.unl. edu/portal/index.html, 2009.
2. J. Aguilar-Ruiz, I. Ramos, J.C. Riquelme, and M. Toro, "An Evolutionary Approach to Estimating Software Development Projects," Information and Software Technology, vol.43, no.14, pp. 875-882, 2001. (Pubitemid 33050967)
3. G. Antoniol, M.D. Penta, and M. Harman, "Search-Based Techniques Applied to Optimization of Project Planning for a Massive Maintenance Project," Proc. IEEE Int'l Conf. Software Maintenance, pp. 240-249, 2005.
4. A.J. Bagnall, V.J. Rayward-Smith, and I.M. Whittley, "The Next Release Problem," Information and Software Technology, vol.43, no.14, pp. 883-890, 2001.
5. J.E. Baker, "Reducing Bias and Inefficiency in the Selection Algorithm," Proc. Second Int'l Conf. Genetic Algorithms and Their Application, 1987.
6. A. Baresel and H. Sthamer, "Evolutionary Testing of Flag Conditions," Proc. Genetic and Evolutionary Computation Conf., pp. 2442-2454, 2003. (Pubitemid 36861156)
7. A. Baresel, H. Sthamer, and M. Schmidt, "Fitness Function Design to Improve Evolutionary Structural Testing," Proc. Genetic and Evolutionary Computation Conf., pp. 1329-1336, 2002.
8. A. Bertolino, "Software Testing Research: Achievements, Challenges, Dreams," Proc. Int'l Conf. Future of Software Eng. 2007, pp. 85-103, 2007. (Pubitemid 47484919)
9. L. Bottaci, "Instrumenting Programs with Flag Variables for Test Data Search by Genetic Algorithm," Proc. Genetic and Evolutionary Computation Conf., pp. 1337-1342, 2002.
10. S. Bouktif, G. Antoniol, E. Merlo, and M. Neteler, "A Novel Approach to Optimize Clone Refactoring Activity," Proc. Genetic and Evolutionary Computation Conf., pp. 1885-1892, 2006. (Pubitemid 44611571)
11. S. Bouktif, H. Sahraoui, and G. Antoniol, "Simulated Annealing for Improving Software Quality Prediction," Proc. Genetic and Evolutionary Computation Conf., pp. 1893-1900, 2006. (Pubitemid 44611572)
12. L.C. Briand, Y. Labiche, and M. Shousha, "Stress Testing Real- Time Systems with Genetic Algorithms," Proc. Genetic and Evolutionary Computation Conf., pp. 1021-1028, 2005. (Pubitemid 43226410)
13. British Standards Inst., "BS 7925-1 Vocabulary of Terms in Software Testing," 1998.
14. G. Canfora, M.D. Penta, R. Esposito, and M.L. Villani, "An Approach for QoS-Aware Service Composition Based on Genetic Algorithms," Proc. Genetic and Evolutionary Computation Conf., pp. 1069-1075, 2005. (Pubitemid 43226416)
15. K. Derderian, R. Hierons, M. Harman, and Q. Guo, "Automated Unique Input Output Sequence Generation for Conformance Testing of FSMs," The Computer J., vol.39, pp. 331-344, 2006.
16. H. Do, S. Elbaum, and G. Rothermel, "Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and Its Potential Impact," Empirical Software Eng., vol.10, no.4, pp. 405-435, 2005. (Pubitemid 41408554)
17. J.J. Dolado, "A Validation of the Component-Based Method for Software Size Estimation," IEEE Trans. Software Eng., vol.26, no.10, pp. 1006-1021, Oct. 2000. (Pubitemid 33776325)
18. R. Ferguson and B. Korel, "The Chaining Approach for Software Test Data Generation," ACM Trans. Software Eng. and Methodology, vol.5, no.1, pp. 63-86, 1996. (Pubitemid 126669196)
19. A. Finkelstein, M. Harman, A. Mansouri, J. Ren, and Y. Zhang, "Fairness Analysis in Requirements Assignments," Proc. IEEE Int'l Requirements Eng. Conf., 2008.
20. R.L. Glass, Facts and Fallacies of Software Engineering. Addison Wesley, 2002.
21. D. Greer and G. Ruhe, "Software Release Planning: An Evolutionary and Iterative Approach," Information and Software Technology, vol.46, no.4, pp. 243-253, 2004. (Pubitemid 38484387)
22. M. Harman, "The Current State and Future of Search Based Software Engineering," Proc. Int'l Conf. Future of Software Eng. 2007, pp. 342-357, 2007. (Pubitemid 47484936)
23. M. Harman, Y. Hassoun, K. Lakhotia, P. McMinn, and J. Wegener, "The Impact of Input Domain Reduction on Search-Based Test Data Generation," Proc. ACM SIGSOFT Symp. Foundations of Software Eng., pp. 155-164, 2007.
24. M. Harman, L. Hu, R. Hierons, J.Wegener, H. Sthamer, A. Baresel, and M. Roper, "Testability Transformation," IEEE Trans. Software Eng., vol.30, no.1, pp. 3-16, Jan. 2004.
25. M. Harman and B. Jones, "Search-Based Software Engineering," Information and Software Technology, vol.43, no.14, pp. 833-839, 2001. (Pubitemid 33050963)
26. M. Harman, K. Lakhotia, and P. McMinn, "A Multi-Objective Approach to Search-Based Test Data Generation," Proc. Genetic and Evolutionary Computation Conf., pp. 1098-1105, 2007. (Pubitemid 47291668)
27. M. Harman and P. McMinn, "A Theoretical & Empirical Analysis of Evolutionary Testing and Hill Climbing for Structural Test Data Generation," Proc. Int'l Symp. Software Testing and Analysis, pp. 73-83, 2007. (Pubitemid 47317581)
28. M. Harman and L. Tratt, "Pareto Optimal Search-Based Refactoring at the Design Level," Proc. Genetic and Evolutionary Computation Conf., pp. 1106-1113, 2007. (Pubitemid 47291669)
29. J.H. Holland, Adaptation in Natural and Artificial Systems. Univ. of Michigan Press, 1975.
30. Y. Jia and M. Harman, "Constructing Subtle Faults Using Higher Order Mutation Testing," Proc. Eighth Int'l Working Conf. Source Code Analysis and Manipulation, 2008.
31. T. Jones, "Evolutionary Algorithms, Fitness Landscapes and Search," PhD thesis, Univ. of New Mexico, 1995.
32. T.M. Khoshgoftaar, L. Yi, and N. Seliya, "A Multi-Objective Module-Order Model for Software Quality Enhancement," IEEE Trans. Evolutionary Computation, vol.8, no.6, pp. 593-608, Dec. 2004. (Pubitemid 40085397)
33. B. Korel, "Automated Software Test Data Generation," IEEE Trans. Software Eng., vol.16, no.8, pp. 870-879, Aug. 1990. (Pubitemid 20738724)
34. B. Korel, "Dynamic Method for Software Test Data Generation," Software Testing, Verification and Reliability, vol.2, no.4, pp. 203- 213, 1992.
35. K. Lakhotia, M. Harman, and P. McMinn, "Handling Dynamic Data Structures in Search-Based Testing," Proc. Genetic and Evolutionary Computation Conf., pp. 1759-1766, 2008.
36. D. Leffingwell and D. Widrig, Managing Software Requirements: A Use Case Approach. Addison Wesley, 2003.
37. K. Mahdavi, M. Harman, and R. Hierons, "A Multiple Hill Climbing Approach to Software Module Clustering," Proc. IEEE Int'l Conf. Software Maintenance, pp. 315-324, 2003.
38. N. Mansour and M. Salame, "Data Generation for Path Testing," Software Quality J., vol.12, no.2, pp. 121-134, 2004. (Pubitemid 39019670)
39. G. McGraw, C. Michael, and M. Schatz, "Generating Software Test Data by Evolution," IEEE Trans. Software Eng., vol.27, no.12, pp. 1085-1110, Dec. 2001. (Pubitemid 34035564)
40. P. McMinn, "Search-Based Software Test Data Generation: A Survey," Software Testing, Verification and Reliability, vol.14, no.2, pp. 105-156, 2004.
41. P. McMinn, "Search-Based Failure Discovery Using Testability Transformations to Generate Pseudo Oracles," Proc. Genetic and Evolutionary Computation Conf., pp. 1689-1696, 2009.
42. P. McMinn, D. Binkley, and M. Harman, "Empirical Evaluation of a Nesting Testability Transformation for Evolutionary Testing," ACM Trans. Software Eng. Methodology, vol.18, no.3, 2009.
43. P. McMinn, M. Harman, D. Binkley, and P. Tonella, "The Species Per Path Approach to Search-Based Test Data Generation," Proc. Int'l Symp. Software Testing and Analysis, pp. 13-24, 2006. (Pubitemid 46645515)
44. P. McMinn and M. Holcombe, "Evolutionary Testing Using an Extended Chaining Approach," Evolutionary Computation, vol.14, pp. 41-64, 2006. (Pubitemid 43955272)
45. W. Miller and D. Spooner, "Automatic Generation of Floating- Point Test Data," IEEE Trans. Software Eng., vol.2, no.3, pp. 223-226, Sept. 1976.
46. B.S. Mitchell and S. Mancoridis, "On the Automatic Modularization of Software Systems Using the Bunch Tool," IEEE Trans. Software Eng., vol.32, no.3, pp. 193-208, Mar. 2006.
47. M. Mitchell, S. Forrest, and J.H. Holland, "The Royal Road for Genetic Algorithms: Fitness Landscapes and GA Performance," Proc. First European Conf. Artificial Life, pp. 245-254, 1992.
48. H. Mühlenbein and D. Schlierkamp-Voosen, "Predictive Models for the Breeder Genetic Algorithm: I. Continuous Parameter Optimization," Evolutionary Computation, vol.1, no.1, pp. 25-49, 1993.
49. National Institute of Standards and Technology, "The Economic Impacts of Inadequate Infrastructure for Software Testing," Planning Report 02-3 May 2002.
50. M. O'Keeffe and M. O'Cinneide, "Search-Based Software Maintenance," Proc. Conf. Software Maintenance and ReEng., pp. 249-260, 2006.
51. R. Pargas, M. Harrold, and R. Peck, "Test-Data Generation Using Genetic Algorithms," Software Testing, Verification and Reliability, vol.9, no.4, pp. 263-282, 1999. (Pubitemid 30546633)
52. Radio Technical Commission for Aeronautics, "RTCA DO178-B Software Considerations in Airborne Systems and Equipment Certification," 1992.
53. C.R. Reeves and J.E. Rowe, Genetic Algorithms-Principles and Perspectives: A Guide to GA Theory. Springer, 2002.
54. O. Seng, J. Stammel, and D. Burkhart, "Search-Based Determination of Refactorings for Improving the Class Structure of Object- Oriented Systems," Proc. Genetic and Evolutionary Computation Conf., pp. 1909-1916, 2006. (Pubitemid 44611574)
55. N. Tracey, J. Clark, K. Mander, and J. McDermid, "Automated Test Data Generation for Exception Conditions," Software- Practice and Experience, vol.30, no.1, pp. 61-79, 2000. (Pubitemid 30555918)
56. H.-C. Wang and B. Jeng, "Structural Testing Using Memetic Algorithm," Proc. Second Taiwan Conf. Software Eng., 2006.
57. J. Wegener, A. Baresel, and H. Sthamer, "Evolutionary Test Environment for Automatic Structural Testing," Information and Software Technology, vol.43, no.14, pp. 841-854, 2001. (Pubitemid 33050964)
58. J. Wegener, H. Sthamer, B.F. Jones, and D.E. Eyres, "Testing Real- Time Systems Using Genetic Algorithms," Software Quality J., vol.6, no.2, pp. 127-135, 1997. (Pubitemid 127479282)
59. D. Whitley, "The GENITOR Algorithm and Selection Pressure: Why Rank-Based Allocation of Reproductive Trials Is Best," Proc. Int'l Conf. Genetic Algorithms, J.D. Schaffer, ed., pp. 116- 121, 1989.
60. S. Xanthakis, C. Ellis, C. Skourlas, A.L. Gall, S. Katsikas, and K. Karapoulios, "Application of Genetic Algorithms to Software Testing (Application Des Algorithms Génétiques Au Test Des Logiciels)," Proc. Fifth Int'l Conf. Software Eng. and Its Applications, pp. 625-636, 1992.
61. M. Xiao, M. El-Attar, M. Reformat, and J. Miller, "Empirical Evaluation of Optimization Algorithms When Used in Goal- Oriented Automated Test Data Generation Techniques," Empirical Software Eng., vol.12, no.2, pp. 183-239, 2007. (Pubitemid 46375363)