Reducing test effort: A systematic mapping study on existing approaches

Information and Software Technology

Volumn 54, Issue 10, 2012, Pages 1092-1106

  Elberzhager, Frank ^a   Rosbach, Alla ^a   Münch, Jürgen ^b   Eschbach, Robert ^c  

^a FRAUNHOFER IESE   (Germany)

^b ITK Engineering AG ^*  (Finland)

^c Model Based and Control Systems Design   (Germany)

Author keywords

Efficiency improvement; Mapping study; Quality assurance; Software testing; Test effort reduction

Indexed keywords

DEFECTS; MAPPING; QUALITY ASSURANCE; SOFTWARE DESIGN; TESTING;

DEFECT CONTENTS; EFFICIENCY IMPROVEMENT; FUTURE RESEARCH DIRECTIONS; MAPPING STUDIES; SYSTEMATIC MAPPING STUDIES; TEST EFFORTS; TEST STRATEGIES; TESTING EFFORT;

SOFTWARE TESTING;

EID: 84863453536     PISSN: 09505849     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.infsof.2012.04.007     Document Type: Article

References (161)

1. A. Bertolino, R. Mirandola, E. Peciola, A case study in branch testing automation, in: Third International Conference on Achieving Quality in Software, 1997, pp. 47-59. (Pubitemid 127416791)
2. K. Young, and H. Soo A case study on the improvement of software test effectiveness through static testing Journal of KISS: Software and Applications 34 March 2007 212 218
3. W. Tsai, Xinyu Zhou, R. Paul, Yinong Chen, Xiaoying Bai, A coverage relationship model for test case selection and ranking for multi-version software, in: IEEE International Symposium on High Assurance, Systems Engineering, 2007, pp. 105-112.
4. H. Kim, and H. Han A defect prevention model based on SW-FMEA Journal of KISS: Software and Applications 33 July 2006 605 614
5. T. Xie and J. Zhao, A framework and tool supports for generating test inputs of AspectJ programs, in: 5th International Conference on Aspect-oriented Software, Development, 2006, pp. 190-201. (Pubitemid 46644582)
6. O. Shatnawi, and P. Kapur A generalized software fault classification model WSEAS Transactions on Computers 7 2008 1375 1384
7. F. Belli, A. Hollmann, M. Kleinselbeck, A graph-model-based testing method compared with the classification tree method for test case generation, in: 3rd IEEE International Conference on Secure Software Integration Reliability Improvement, 2009, pp. 193-200.
8. M. Penaloza, A. Logar, J. Johnson, M. Boucher, A Java unit and integration testing tool, in: Proceedings of the ISCA 16th International Conference Computers and their Applications, 2001, pp. 358-61.
9. A. Boklund, C. Selvefors, A low budget approach to distributed automated black-box testing, in: International Conference on Software Engineering Research and, Practice, 2005, pp. 302-308.
10. P. Neto, R. Resende, C. Padua, A method for information systems testing automation, in: 17th International Conference on Advanced Information Systems Engineering, 2005, pp. 504-18. (Pubitemid 41336124)
11. E. Diaz, J. Tuya, R. Blanco, A modular tool for automated coverage in software testing, in: Proceedings. Eleventh Annual International Workshop on Software Technology and Engineering, Practice, 2004, pp. 241-246.
12. H. Miao, P. Liu, J. Mei, H. Zeng, A new approach to automated redundancy reduction for test sequences, in: 15th IEEE Pacific Rim International Symposium on Dependable, Computing, 2009, pp. 93-98.
13. Y. Cui, L. Li, and S. Yao, A new strategy for pairwise test case generation, in: 3rd International Symposium on Intelligent Information Technology Application, 2009, pp. 303-306.
14. B. Biswal, P. Nanda, and D. Mohapatra A novel approach for scenario-based test case generation International Conference on Information Technology 2008 2008 244 247
15. M. Catelani, L. Ciani, V. Scarano, A. Bacioccola, A novel approach to automated testing to increase software reliability, in: IEEE Instrumentation and Measurement Technology Conference, 2008, pp. 1499-502.
16. L. Briones, E. Brinksma, M. Stoelinga, A semantic framework for test coverage, in: 4th International, Symposium on Automated Technology for Verification and Analysis, 2006, pp. 399-414. (Pubitemid 44850351)
17. X. Teng, and H. Pham A software-reliability growth model for N-version programming systems IEEE Transactions on Reliability 51 2002 311 321
18. E. Arisholm, L. Briand, and E. Johannessen A systematic and comprehensive investigation of methods to build and evaluate fault prediction models Journal of Systems and Software 83 January 2010 2 17
19. S. Ali, L. Briand, H. Hemmati, and R. Panesar-Walawege A systematic review of the application and empirical investigation of search-based test case generation IEEE Transactions on Software Engineering 99 2010 1
20. E. Martins, M. de Fatima Mattiello-Francisco, A tool for fault injection and conformance testing of distributed systems, in: First Latin-American, Symposium on Dependable Computing, 2003, pp. 282-302.
21. M. Gallagher, and V. Lakshmi Narasimhan ADTEST: a test data generation suite for Ada software systems IEEE Transactions on Software Engineering 23 1997 473 484 (Pubitemid 127828308)
22. F. Wang, S. Wang, Y. Ji, An automatic generation method of executable test case using model-driven architecture, in: 4th International Conference on Innovative Computing, Information and Control, 2009, pp. 389-393.
23. A.G. Koru, and J. Tian An empirical comparison and characterization of high defect and high complexity modules Journal of Systems and Software 67 2003 153 163
24. Y. Shin, L. Williams, An empirical model to predict security vulnerabilities using code complexity metrics, in: 2nd International Symposium on Empirical Software Engineering and, Measurement, 2008, pp. 315-317.
25. A.M. Smith, G.M. Kapfhammer, An empirical study of incorporating cost into test suite reduction and prioritization, in: 24th Annual ACM Symposium on Applied, Computing, 2009, pp. 461-467.
26. T. Graves, M. Harrold, J. Kim, A. Porter, and G. Rothermel An empirical study of regression test selection techniques ACM Transactions on Software Engineering and Methodology 10 2001 184 208 (Pubitemid 33602334)
27. M. Tang, M. Kao, M. Chen, An empirical study on object-oriented metrics, in: Proceedings of METRICS '99: Sixth International Symposium on Software Metrics, 1999, pp. 242-249.
28. H.M. Olague, L.H. Etzkorn, S.L. Messimer, and H.S. Delugach An empirical validation of object-oriented class complexity metrics and their ability to predict error-prone classes in highly iterative, or agile, software: a case study Journal of Software Maintenance and Evolution 20 2008 171 197
29. S. Wagner, F. Deissenboeck, M. Aichner, J. Wimmer, M. Schwalb, An evaluation of two bug pattern tools for Java, in: 1st International Conference on Software Testing, Verification and Validation, 2008, pp. 248-257.
30. J. Sant, A. Souter, L. Greenwald, An exploration of statistical models for automated test case generation, in: Proceedings of the Third International Workshop on Dynamic, Analysis, 2005, pp. 1-7.
31. G. Woo, H. Seok Chae, H. Jang, An intermediate representation approach to reducing test suites for retargeted compilers, in: 12th Ada-Europe International Conference on Reliable Software Technologies, 2007, pp. 100-113.
32. M.A. Wojcicki, P. Strooper, An iterative empirical strategy for the systematic selection of a combination of verification and validation technologies, in: 5th International Workshop on Software, Quality, 2007, p. 9.
33. L. Kawakami, A. Knabben, D. Rechia, D. Bastos, O. Pereira, R. Pereira e Silva, L.C. Dos Santos, An object-oriented framework for improving software reuse on automated testing of mobile phones, in: 19th IFIP TC6/WG6.1 International Conference on Testing of Communicating Systems and 7th International Workshop on Formal Approaches to Testing Software, 2007, pp. 199-211. (Pubitemid 47290161)
34. W. Ding, J. Kou, K. Li, Z. Yang, An optimization method of test suite in regression test model, in: WRI World Congress on Software Engineering, 2009, pp. 180-183.
35. R. Selby, and V. Basili Analyzing error-prone system structure IEEE Transactions on Software Engineering 17 1991 141 152
36. K.H.S. Hla, Y.S. Choi, J.S. Park, Applying particle swarm optimization to prioritizing test cases for embedded real time software retesting, in: 8th IEEE International Conference on Computer and Information Technology Workshops, 2008, pp. 527-532.
37. N. Oster, Automated generation and evaluation of dataflow-based test data for object-oriented software, in: 1st International Conference on the Quality of Software Architectures and 2nd International Workshop on Software, Quality, 2005, pp. 212-226.
38. D. Mohapatra, P. Bhuyan, D.P. Mohapatra, Automated test case generation and its optimization for path testing using genetic algorithm and sampling, in: WASE International Conference on Information, Engineering, 2009, pp. 643-646.
39. M. Harman, F. Islam, T. Xie, S. Wappler, Automated test data generation for aspect-oriented programs, in: 8th ACM International Conference on Aspect-Oriented Software, Development, 2009, pp. 185-196.
40. R. Gupta, M. Soffa, Automatic generation of a compact test suite, Algorithms, Software, Architecture. Information Processing, in: IFIP 12th World Computer Congress, 1992, pp. 237-243.
41. H.K. Seung, S.K. Hyeon, Automatic generation of testing environments for web applications, in: International Conference on Computer Science and, Software Engineering, 2008, pp. 694-697.
42. R. Romli, S. Sulaiman, K. Zamli, Automatic programming assessment and test data generation a review on its approaches, in: International Symposium in Information Technology, 2010, pp. 1186-1192.
43. A. Khamis, M. Girgis, and A. Ghiduk Automatic software test data generation for spanning sets coverage using genetic algorithms Computing and Informatics 26 2007 383 401
44. B. Baudry, F. Fleurey, J. Jezequel, and Y. Le Traon Automatic test case optimization: a bacteriologic algorithm IEEE Software 22 2005 76 82 (Pubitemid 40556024)
45. N. Oster, F. Saglietti, Automatic test data generation by multi-objective optimisation, in: 25th International Conference on Computer Safety Reliability, and Security, 2006, pp. 426-38. (Pubitemid 44744064)
46. P. Bokil, P. Darke, U. Shrotri, R. Venkatesh, Automatic test data generation for C programs, in: Third IEEE International Conference on Secure Software Integration and Reliability Improvement, 2009, pp. 359-368.
47. D.L. Barbosa, H.S. Lima, P.D.L. Machado, J.C.A. Figueiredo, M.A. Juca, and W.L. Andrade Automating functional testing of components from UML specifications International Journal of Software Engineering and Knowledge Engineering 17 2007 339 358 (Pubitemid 47172752)
48. K. Im, T. Im, J.D. McGregor, Automating test case definition using a domain specific language, in: 46th Annual Southeast Regional Conference on XX, 2008, pp. 180-185.
49. M. Vieira, J. Leduc, B. Hasling, R. Subramanyan, and J. Kazmeier, Automation of GUI testing using a model-driven approach, in Proceedings of the 2006 international workshop on Automation of software test, 2006, p. 9.
50. L. Pan, B. Zou, J. Li, H. Chen, Bi-objective model for test-suite reduction based on modified condition/decision coverage, in: Proceedings. 11th Pacific Rim International Symposium on Dependable, Computing, 2005, p. 7.
51. R. Zhao, Z. Li, Boundary value testing using integrated circuit fault detection rule, in: Testing: Academic and Industrial Conference - Practice and Research, Techniques, 2009, pp. 3-12.
52. E. Moritz, Case study: how analysis of customer found defects can be used by system test to improve quality, in: 31st International Conference on, Software Engineering, 2009, pp. 123-129.
53. J.J. Li, D. Weiss, and H. Yee Code-coverage guided prioritized test generation Information and Software Technology 48 2006 1187 1198 (Pubitemid 44693352)
54. R.S. Sisodia, V. Channakeshava, Combinatorial approach for automated platform diversity testing, in: 4th International Conference on Software, Engineering Advances, 2009, pp. 134-139.
55. R. Kuhn, R. Kacker, Yu Lei, and J. Hunter Combinatorial software testing Computer 42 2009 94 96
56. M. Hamill, and K. Goseva-Popstojanova Common trends in software fault and failure data IEEE Transactions on Software Engineering 35 2009 484 496
57. C. Huang Cost-reliability-optimal release policy for software reliability models incorporating improvements in testing efficiency Journal of Systems and Software 77 2005 139 155 (Pubitemid 40612683)
58. Z. Li, S. Lu, S. Myagmar, Y. Zhou, CP-Miner: a tool for finding copy-paste and related bugs in operating system code, in: Proceedings of the Sixth Symposium on Operating Systems Design and Implementation, 2004, pp. 289-302.
59. G. Wassermann, D. Yu, A. Chander, D. Dhurjati, H. Inamura, Z. Su, Dynamic test input generation for web applications, in: Proceedings of the 2008 International Symposium on Software Testing and, Analysis, 2008, p. 249.
60. M. Sherriff, N. Nagappan, L. Williams, M. Vouk, Early estimation of defect density using an in-process Haskell metrics model, in: 1st International Workshop on Advances in Model-Based Testing, 2005.
61. N. Ohlsson, A.C. Eriksson, and M. Helander Early risk-management by identification of fault-prone modules Empirical Software Engineering 2 1997 166 173 (Pubitemid 127506165)
62. W.E. Wong, Y. Lei, X. Ma, Effective generation of test sequences for structural testing of concurrent programs, in: 10th IEEE International Conference on Engineering of Complex Computer Systems, 2005, pp. 539-548. (Pubitemid 41495782)
63. T.L. Graves, M.J. Harrold, J. Kim, A. Porter, and G. Rothermel Empirical study of regression test selection techniques, Proceedings of the International Conference on Software Engineering 1998 1998 188 197
64. R. Gupta, and M. Soffa Employing static information in the generation of test cases Software Testing, Verification and Reliability 3 1993 29 48
65. O. Laitenberger, and J. Debaud Encompassing life cycle centric survey of software inspection Journal of Systems and Software 50 2000 5 31 (Pubitemid 32212306)
66. C. Lin, and C. Huang Enhancing and measuring the predictive capabilities of testing-effort dependent software reliability models Journal of Systems and Software 81 2008 1025 1038 (Pubitemid 351522336)
67. T. Ummu Salima, A. Askarunisha, N. Ramaraj, Enhancing the efficiency of regression testing through intelligent agents, in: International Conference on Computational Intelligence and Multimedia Applications, 2008, pp. 103-108. (Pubitemid 351424404)
68. S. Kpodjedo, F. Ricca, G. Antoniol, P. Galinier, Evolution and search based metrics to improve defects prediction, in: 1st International Symposium on Search Based, Software Engineering, 2009, pp. 23-32.
69. P.L. Li, J. Herbsleb, M. Shaw, B. Robinson, Experiences and results from initiating field defect prediction and product test prioritization efforts at ABB Inc., in: 28th International Conference on, Software Engineering, 2006, pp. 413-422. (Pubitemid 46604733)
70. T. Illes-Seifert, and B. Paech Exploring the relationship of a file's history and its fault-proneness: an empirical method and its application to open source programs Information and Software Technology 52 2010 539 558
71. T. Illes-Seifert, B. Paech, Exploring the relationship of history characteristics and defect count: an empirical study, in: Workshop on Defects in Large Software Systems, 2008, pp. 11-15.
72. S. Kuo, C. Huang, and M. Lyu Framework for modeling software reliability, using various testing-efforts and fault-detection rates IEEE Transactions on Reliability 50 2001 310 320 (Pubitemid 34099647)
73. O. Shatnawi, Generalized software fault classification model, in: 12th WSEAS International Conference on Computers, 2008, pp. 993-998.
74. Y. Yu, S. Ng, E. Chan, Generating, selecting and prioritizing test cases from specifications with tool support, in: Proceedings. Third International Conference on Quality Software, 2003, pp. 83-90.
75. B. Kitchenham, Guidelines for Performing Systematic Literature Reviews in Software Engineering, Version 2.3. Technical Report S.o.C.S.a.M. Software Engineering Group, Keele University and Department of Computer Science University of Durham, 2007.
76. G. Fraser, B.K. Aichernig, and F. Wotawa Handling model changes: regression testing and test-suite update with model-checkers Electronic Notes in Theoretical Computer Science 190 2007 33 46 (Pubitemid 47302939)
77. M.N. Ngo, and H.B.K. Tan Heuristics-based infeasible path detection for dynamic test data generation Information and Software Technology 50 2008 641 655 (Pubitemid 351601999)
78. L. Huang, and B. Boehm How much software quality investment is enough: a value-based approach IEEE Software 23 2006 88 95 (Pubitemid 46405207)
79. A. Beer, S. Mohacsi, C. Stary, IDATG: an open tool for automated testing of interactive software, in: Proceedings of the 1998 IEEE 22nd Annual International Computer Software & Applications Conference, 1998, pp. 470-475.
80. IEEE Xplore, .
81. R. Gerlich, R. Gerlich, T. Boll, P. Chevalley, Improving test automation by deterministic methods in statistical testing, in: DASIA: Data Systems In Aerospace, 2006, p. Eurospace.
82. B. Baudry, F. Fleurey, Y. Le Traon, Improving test suites for efficient fault localization, in: 28th International Conference on, Software Engineering, 2006, pp. 82-91. (Pubitemid 46600905)
83. J. Chaar, M. Halliday, I. Bhandari, and R. Chillarege In-process evaluation for software inspection and test IEEE Transactions on Software Engineering 19 11 1993 1055 1070
84. D. Winkler, S. Biffl, K. Faderl, Investigating the temporal behavior of defect detection in software inspection and inspection-based testing, in: 11th International Conference on Product-Focused Software Process Improvement, 2010, pp. 17-31.
85. A. Calvagna, A. Gargantini, IPO-s: incremental generation of combinatorial interaction test data based on symmetries of covering arrays, in: IEEE International Conference on Software Testing, Verification, and Validation Workshops, 2009, pp. 10-18.
86. R. Paul, Metrics based classification trees for software test monitoring and management, in: Proceedings Sixth International Conference on Tools with, Artificial Intelligence, 1994, pp. 534-540.
87. J. Michura, M. Capretz, Metrics suite for class complexity, in: Proceedings. ITCC International Conference on Information Technology: Coding and, Computing, 2005, pp. 404-409. (Pubitemid 41290260)
88. C.Y. Htoon, N.L. Thein, Model-based testing considering cost, reliability and software quality, in: 6th Asia-Pacific Symposium on Information and Telecommunication Technologies, 2005, pp. 160-164. (Pubitemid 46289997)
89. I. Bluemke, Object oriented metrics useful in the prediction of class testing complexity, in: Proceedings of the 27th EUROMICRO Conference, 2001, pp. 130-136.
90. P.R.F. Nunes, S. Hanazumi, A.C.V. De Melo, OConGraX - Automatically generating data-flow test cases for fault-tolerant systems, in: 21st IFIP International Conference on Testing of Communicating Systems and 9th International Workshop on Formal Approaches to Testing of Software, 2009, pp. 229-234.
91. R. Lea, S. Chen, and C. Chung, On generating test data from prototypes, in: Proceedings of the Fifteenth Annual International Computer Software and Applications Conference, 1991, pp. 345-350.
92. C. Huang, J. Lo, S. Kuo, M. Lyu, Optimal allocation of testing resources for modular software systems, in: Proceedings 13th International Symposium on Software, Reliability Engineering, 2002, pp. 129-138.
93. C. Huang, J. Lo, J. Lin, C. Sue, C. Lin, Optimal resource allocation and sensitivity analysis for modular software testing, in: Proceedings. IEEE Fifth International Symposium on Multimedia, Software Engineering, 2003, pp. 231-238.
94. P. Jha, D. Gupta, Bo Yang, and P. Kapur Optimal testing resource allocation during module testing considering cost, testing effort and reliability Computers & Industrial Engineering 57 2009 1122 1130
95. C. Huang, and M.R. Lyu Optimal testing resource allocation, and sensitivity analysis in software development IEEE Transactions on Reliability 54 4 2005 592 603 (Pubitemid 43000693)
96. F. Elberzhager, J. Muench, D. Rombach, B. Freimut, Optimizing cost and quality by integrating inspection and test processes, in: Proceedings of the International Conference on Software and Systems Process, 2011, pp. 3-12.
97. F. Vokolos, P. Frankl, Pythia: a regression test selection tool based on textual differencing, in: Proceedings of 3rd International Conference on Reliability, Quality and Safety of Software-Intensive, System, 1997, pp. 3-21.
98. M. Marre, A. Bertolino, Reducing and estimating the cost of test coverage criteria, in: Proceedings of IEEE 18th International Conference on, Software Engineering, 1996, pp. 486-94.
99. V. Jagannath, Yun Young Lee, B. Daniel, D. Marinov, Reducing the costs of bounded-exhaustive testing, in: 12th International Conference on Fundamental Approaches to Software Engineering, 2009, pp. 171-185.
100. D. Brenner, C. Atkinson, R. Malaka, M. Merdes, B. Paech, and D. Suliman Reducing verification effort in component-based software engineering through built-in testing Information Systems Frontiers 9 2 2007 151 162 (Pubitemid 46897537)
101. G. Fraser, F. Wotawa, Redundancy based test-suite reduction, in: 10th International Conference on Fundamental Approaches to, Software Engineering, 2007, pp. 291-305. (Pubitemid 350259357)
102. N. Li, T. Xie, N. Tillmann, J. de Halleux, W. Schulte, Reggae: Automated test generation for programs using complex regular expressions, in: 2009 24th IEEE/ACM International Conference on Automated Software Engineering, 2009, pp. 515-519.
103. E. Martins, V. G. Vieira, Regression test selection for testable classes, in: 5th European Dependable Computing Conference, 2005, pp. 453-470. (Pubitemid 41312777)
104. Q. Xie, M. Grechanik, C. Fu, REST: a tool for reducing effort in script-based testing, in: 24th IEEE International Conference on Software, Maintenance, 2008, pp. 468-469.
105. W. Tsai, L. Yu, X. Liu, A. Saimi, Y. Xiao, Scenario-based test case generation for state-based embedded systems, in: Conference Proceedings of the IEEE International Performance, Computing, and Communications Conference, 2003, pp. 335-342.
106. D. Binkley Semantics guided regression test cost reduction IEEE Transactions on Software Engineering 23 8 1997 498 516 (Pubitemid 127828310)
107. C. Huang, J. Lo, S. Kuo, M.R. Lyu, Software reliability modeling and cost estimation incorporating testing-effort and efficiency, in: Proceedings of the 1999 10th International Symposium on Software Reliability Engineering, 1999, pp. 62-72. (Pubitemid 30516652)
108. M.J. Gallagher, V. Narasimhan, Software test data generation using program instrumentation, in: Proceedings of the IEEE 1st International Conference on Algorithms and Architectures for Parallel Processing, Part 1, 1995, pp. 575-584.
109. J. Prabhu, N. Malmurugan, G. Gunasekaran, R. Gowtham, Study of ERP test-suite reduction: based on modified condition/decision coverage, in: Second International Conference on Computer Research and, Development, 2010, pp. 373-378.
110. K. Petersen, R. Feldt, S. Mujtaba, M. Mattsson, Systematic mapping studies in software engineering, in: Proceedings of the 12th International Conference on Evaluation and Assessment in, Software Engineering, 2008, pp. 1-10.
111. F. Pinte, N. Oster, F. Saglietti, Techniques and tools for the automatic generation of optimal test data at code, model and interface level, in: 30th International Conference on, Software Engineering, 2008, pp. 927-928.
112. L. Lazic, and N. Mastorakis Techniques to reduce a set of test cases WSEAS Transactions on Computers 5 11 2006 2813 2826 (Pubitemid 44193674)
113. P. Saraph, M. Last, A. Kandel, Test case generation and reduction by automated input-output analysis, in: Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics, 2003, pp. 768-773.
114. Y. Yongfeng, L. Bin, L. Minyan, L. Zhen, Test cases generation for embedded real-time software based on extended UML, in: International Conference on Information Technology and Computer Science, 2009, pp. 69-74.
115. A. Spillner Test criteria and coverage measures for software integration testing Software Quality Journal 4 4 1995 275 286
116. A. Gupta, P. Jalote, Test inspected unit or inspect unit tested code? in: First International Symposium on Empirical Software Engineering and Measurement, 2007. ESEM 2007, 2007, pp. 51-60.
117. D. Travison, G. Staneff, Test instrumentation and pattern matching for automatic failure identification, in: 1st International Conference on Software Testing, Verification and Validation, 2008, pp. 377-386.
118. M. Hirayama, O. Mizuno, and T. Kikuno Test item prioritizing metrics for selective software testing IEICE Transactions on Information and Systems 87 12 2004 2733 2743 (Pubitemid 40021345)
119. R. Hewett, P. Kijsanayothin, D. Smavatkul, Test order generation for efficient object-oriented class integration testing, in: 20th International Conference on Software Engineering & Knowledge, Engineering, 2008, pp. 703-708.
120. C. Baek, S. Park, and K. Choi TEST: an effective automation tool for testing embedded software WSEAS Transactions on Information Science and Applications 2 2005 1214 1219 (Pubitemid 41046401)
121. Health, Social, and Economic Research, The Economic Impacts of Inadequate Infrastructure for Software Testing, National Institute of Standards and Technology, 2002.
122. D. Talby, The perceived value of authoring and automating acceptance tests using a model driven development toolset, in: Workshop on Automation of Software, Test, 2009, pp. 154-157.
123. A. Perez, S. Kaiser, Top-down reuse for multi-level testing, in: 17th IEEE International Conference and Workshops on Engineering of Computer-Based Systems, 2010, pp. 150-159.
124. M. Gegick, L. Williams, Toward the use of automated static analysis alerts for early identification of vulnerability- and attack-prone components, in: 2nd International Conference on Internet Monitoring and Protection, 2007.
125. Z. Li, J. Zhu, L. Zhang, N. Mitsumori, Towards a practical and effective method for Web services test case generation, in: Workshop on Automation of Software, Test, 2009, pp. 106-114.
126. J. Zhao, T. Xie, N. Li, Towards regression test selection for AspectJ programs, in: 2nd Workshop on Testing Aspect-oriented Programs, 2006, pp. 21-26. (Pubitemid 46645432)
127. T. Khoshgoftaar, Xiaojing Yuan, E. Allen, W. Jones, and J. Hudepohl Uncertain classification of fault-prone software modules Empirical Software Engineering 7 4 2002 297 318 (Pubitemid 35377447)
128. P. Kapur, O. Shatnawi, A. Aggarwal, and R. Kumar Unified framework for developing testing effort dependent software reliability growth models WSEAS Transactions on Systems 8 4 2009 521 531
129. K. Whitmill, Usage based test case generation, in: Proceedings Software Testing Analysis and, Review, 1996, pp. 123-152.
130. N. Nagappan, L. Williams, M. Vouk, J. Osborne, Using in-process testing metrics to estimate post-release field quality, in: 18th IEEE International Symposium on Software, Reliability Engineering, 2007, pp. 209-214.
131. F. Elberzhager, R. Eschbach, J. Muench, Using inspection results for prioritizing test activities, in: Proceedings of the 21st International Symposium on Software Reliability Engineering, Supplemental Proceedings, 2010, pp. 263-272.
132. P. Pocatilu Using open source software testing tools for automated unit testing Open Source Science Journal 1 1 2009 163 172
133. T. Ostrand, E. Weyuker, R. Bell, Using static analysis to determine where to focus dynamic testing effort, in: Second International Workshop on Dynamic Analysis and 26th International Conference on, Software Engineering, 2004, pp. 1-8.
134. M. Nita, D. Notkin, White-box approaches for improved testing and analysis of configurable software systems, in: 31st International Conference on, Software Engineering, 2009, pp. 307-310.
135. Zotero, .
136. D. Jackson, M. Thomas, L.I. Millett (Eds.). Software for Dependable Systems: Sufficient Evidence? Committee on Certifiably Dependable Software Systems, National Research Council, National Academy of Sciences, 2007.
137. M. Ivarsson, and T. Gorschek A method for evaluating rigor and industrial relevance of technology evaluations Empirical Software Engineering 16 3 2011 365 395
138. IEEE Standard 610.12-1990. IEEE Standard Glossary of Software Engineering Terminology, 1990.
139. A.K. Amanpreet, A.S. Brar, P.S. Sandhu, An empirical approach for software fault prediction, in: Fifth International Conference on Industrial and, Information Systems, 2010, pp. 261-265.
140. S. Nachiyappan, A.Vimaladevi, C.B. SelvaLakshmi, An evolutionary algorithm for regression test suite reduction, in: International Conference on Communication and Computational Intelligence.
141. M. D'Ambros, M. Lanza, R. Robbes, An extensive comparison of bug prediction approaches, in: 7th IEEE Working Conference on Mining Software Repositories, 2010, pp. 31-41.
142. D. Chen, X. Li, S. Zhao, Auto-generation and redundancy reduction of test cases for reactive systems, in: 2nd International Conference on Software Technology and, Engineering, 2010, pp. 125-130.
143. W. Jin, A. Orso, T. Xie, Automated behavioral regression testing, in: Third International Conference on Software Testing, Verification and Validation, 2010, pp. 137-146.
144. W. Xu, D. Huang, Automated testing for database system, in: International Conference on Biomedical Engineering and Computer Science, 2010, pp. 1-4.
145. H. Cichos, T.S. Heinze, Efficient reduction of model-based generated test suites through test case pair prioritization, in: Workshop on Model-Driven Engineering, Verification, and Validation, 2010, pp. 37-42.
146. L. Yunfeng, B. Kerong, Metrics selection for fault-proneness prediction of software modules, in: International Conference on Computer Design And Appliations, 2010, pp. 191-195.
147. M.P. Usaola, and P.R. Mateo Mutation testing cost reduction techniques: a survey IEEE Software 27 3 2010 80 86
148. Q. Gu1, B. Tang, D.X. Chen, Optimal regression testing based on selective coverage of test requirements, in: International Symposium on Parallel and Distributed Processing with Applications, 2010, pp. 419-426.
149. B. Jiang, Y. Mu, Z. Zhang, Research of optimization algorithm for path-based regression testing suit, in: Second International Workshop on Education Technology and Computer Science, 2010, pp. 303-306.
150. W. Afzal, R. Torkar R. Feldt, G. Wikstrand, Search-based prediction of fault-slip-through in large software projects, in: 2nd International Symposium on Search Based, Software Engineering, pp. 79-88.
151. L.T. Giang, D. Kang, D.H. Bae, Software fault prediction models for web applications, in: 34th Annual IEEE Computer Software and Applications Conference Workshops, pp. 51-56.
152. R. Lincke, T. Gutzmann, W. Löwe, Software quality prediction models compared, in: 10th International Conference on Quality Software, 2010, pp. 82-91.
153. N. Pan, F. Zeng, Y.H. Huang, Test case reduction based on program invariant and genetic algorithm, in: 6th International Conference on Wireless Communications Networking and Mobile, Computing, 2010, pp. 1-5.
154. A.K. Pandey, N.K. Goyal, Test effort optimization by prediction and ranking of fault-prone software modules, in: 2nd International Conference on Reliability, Safety & Hazard, 2010, pp. 136-142.
155. F.Q.B. da Silva, A.L.M. Santos, S. Soares, A.C.C. Franca, C.V.F. Monteiro, and F.F. Maciel Six years of systematic literature reviews in software engineering: an updated tertiary study Information and Software Technology 53 2011 899 913
156. B. Kitchenham, O.P. Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkmann Systematic literature reviews in software engineering - a systematic literature review Information and Software Technology 51 2009 7 15
157. N. Juristo, A.M. Moreno, and S. Vegas Reviewing 25 years of testing technique experiments Empirical Software Engineering Journal 1-2 2004 7 44
158. P. Runeson, C. Andersson, T. Thelin, A. Andrews, and T. Berling What do we know about defect detection methods? IEEE Software 23 3 2006 82 86
159. Z. Zakaria, R. Atan, A.A.A. Ghani, N.F.M. Sani, Unit testing approaches for BPEL: a systematic review, in: 16th Asia Pacific, Software Engineering Conference, 2009, pp. 316-322.
160. C. Catal, and B. Diri A systematic review of software fault prediction studies Expert Systems with Applications 36 4 2009 7346 7354
161. B. Haugset, G.K. Hanssen, Automated acceptance testing: a literature review and an industrial case study, Agile, 2008, pp. 27-38.