A survey of combinatorial testing

ACM Computing Surveys

Volumn 43, Issue 2, 2011, Pages

  Nie, Changhai ^a   Leung, Hareton ^b  

^a NANJING UNIVERSITY   (China)

^b HONG KONG POLYTECHNIC UNIVERSITY   (Hong Kong)

Author keywords

Combinatorial testing (CT); Covering array; Software testing; Test case generation

Indexed keywords

ACTIVE FIELD; BASIC CONCEPTS; COMBINATORIAL TESTING; COVERING ARRAYS; EMPIRICAL STUDIES; FAILURE DIAGNOSIS; GENERATION ALGORITHM; KEY ISSUES; PRIORITIZATION; RESEARCH GROUPS; SAMPLING MECHANISMS; STATE OF RESEARCH; TEST CASE GENERATION; TESTING PROCEDURE; TESTING RESULTS; TESTING TECHNIQUE;

COMPUTER SOFTWARE SELECTION AND EVALUATION; SOFTWARE TESTING; SURVEYS; TESTING;

RESEARCH;

EID: 79551702481     PISSN: 03600300     EISSN: 15577341     Source Type: Journal    
DOI: 10.1145/1883612.1883618     Document Type: Article

References (101)

1. BACH, J. AND SCHROEDER, P. J. 2004. Pairwise testing - A best practice that isn't. In Proceedings of the 22nd Pacific Northwest Software Quality Conference. 180-196.
2. BERLING, T. AND RUNESON, P. 2003. Efficient evaluation of multifactor dependent system performance using fractional factorial design. IEEE Trans. Softw. Engin. 29, 9, 769-781.
3. BLASS, A. AND GUREVICH, Y. 2002. Pairwise testing. Bull. Euro. Assoc. Theor. Comput. Sci. 78, 100-132.
4. BROWNLIE, R., PROWSE, J., AND PHADKE, M. S. 1992. Robust testing of at and t pmx/starmail using oats. AT and T Techn. 3, 71, 41-47.
5. BRYCE, R. C. AND COLBOURN, C. J. 2005. Constructing interaction test suites with greedy algorithms. In Proceedings of the 20th IEEE/ACMInternational Conference on Automated Software Engineering (ASE'05). ACM, New York, 440-443.
6. BRYCE, R. C. AND COLBOURN, C. J. 2006. Prioritized interaction testing for pair-wise coverage with seeding and constraints. Inform. Softw. Technol. 48, 10, 960-970. (Pubitemid 44354089)
7. BRYCE, R. C. AND COLBOURN, C. J. 2007a. The density algorithm for pairwise interaction testing. Softw. Test. Verif. Reliab. 17, 3, 159-182. (Pubitemid 47354558)
8. BRYCE, R. C. AND COLBOURN, C. J. 2007b. One-Test-at-a-Time heuristic search for interaction test suites. In Proceedings of the 9th Annual Conference on Genetic and Evolutionary Computation (GECCO'07). ACM, New York, 1082-1089. (Pubitemid 47291666)
9. BRYCE, R. C. AND COLBOURN, C. J. 2008. A density-based greedy algorithm for higher strength covering arrays. Softw. Test. Verif. Reliab. 17, 3, 1-17.
10. BRYCE, R. C., COLBOURN, C. J., AND COHEN, M. B. 2005. A framework of greedy methods for constructing interaction test suites. In Proceedings of the 27th International Conference on Software Engineering (ICSE'05). ACM, New York, 146-155.
11. BRYCE, R. C. 2005. Test prioritization for pairwise interaction coverage. In Proceedings of the 1st International Workshop on Advances in Model-Based Testing (A-MOST'05). ACM, New York, 1-7.
12. BRYCE, R. C. AND MEMON, A.M. 2007. Test suite prioritization by interaction coverage. InWorkshop on Domain Specific Approaches to Software Test Automation (DOSTA'07). ACM, New York, 1-7. (Pubitemid 351438669)
13. BURR, K. AND YOUNG, W. 1998. Combinatorial test techniques: Table-Based automation, test generation, and code coverage. In Proceedings of the International Conference on Software Testing Analysis and Review. 503-513.
14. BURROUGHS, K., JAIN, A., AND ERICKSON, R. 1994. Improved quality of protocol testing through techniques of experimental design. In Proceedings of the IEEE International Conference on Record, 'Serving Humanity Through Communications.' Vol. 2. 745-752.
15. CHENG, C.,DUMITRESCU, A., AND SCHROEDER, P. 2003. Generating small combinatorial test suites to cover inputoutput relationships. In Proceedings of the 3rd International Conference on Quality Software (QSIC'03). IEEE Computer Society, Los Alamitos, CA, 76-82.
16. COHEN, D. M., DALAL, S. R., FREDMAN, M. L., AND PATTON, G. C. 1997. The aetg system: An approach to testing based on combinatorial design. IEEE Trans. Softw. Engin. 23, 7, 437-444. (Pubitemid 127828305)
17. COHEN, D. M., DALAL, S. R., KAJLA, A., AND PATTON, G. C. 1994. The automatic efficient tests generator. In Proceedings of the 5th IEEE International Symposium Software Reliability Engineering. IEEE Press, 303-309.
18. COHEN, D. M., DALAL, S. R., PARELIUS, J., AND PATTON, G. C. 1996. The combinatorial design approach to automatic test generation. IEEE Softw. 13, 5, 83-88.
19. COHEN, D. M. AND FREDMAN, M. L. 1998. New techniques for designing qualitatively independent systems. J. Combin. Des. 6, 6, 411-416. (Pubitemid 128702863)
20. COHEN, M., GIBBONS, P.,MUGRIDGE, W., COLBOURN, C., AND COLLOFELLO, J. 2003a. Variable strength interaction testing of components. In Proceedings of the 27th Annual International Conference on Computer Software and Applications. 413-418.
21. COHEN,M. B. 2004. Designing test suites for software interaction testing. Ph.D. thesis,University of Auckland, New Zealand.
22. COHEN, M. B.,COLBOURN, C. J., AND LING, A. C. H. 2003b. Augmenting simulated annealing to build interaction test suites. In Proceedings of the 14th International Symposium on Software Reliability Engineering (ISSRE'03). IEEE Computer Society, Los Alamitos, CA, 394.
23. COHEN, M. B., DWYER, M. B., AND SHI, J. 2007a. Exploiting constraint solving history to construct interaction test suites. In Proceedings of the Testing: Academic and Industrial Conference Practice and Research Techniques (TAICPART-MUTATION'07). IEEE Computer Society, 121-132.
24. COHEN, M. B., DWYER, M. B., AND SHI, J. 2007b. Interaction testing of highly-configurable systems in the presence of constraints. In Proceedings of the International Symposium on Software Testing and Analysis (ISSTA'07). ACM, New York, 129-139. (Pubitemid 47317587)
25. COHEN, M. B., DWYER, M. B., AND SHI, J. 2008. Constructing interaction test suites for highly-configurable systems in the presence of constraints: A greedy approach. IEEE Trans. Softw. Engin. 34, 5, 633-650.
26. COHEN, M. B., GIBBONS, P. B.,MUGRIDGE, W. B., AND COLBOURN, C. J. 2003c. Constructing test suites for interaction testing. In Proceedings of the 25th International Conference on Software Engineering (ICSE'03). IEEE Computer Society, Los Alamitos, CA, 38-48.
27. COHEN, M. B., SNYDER, J., AND ROTHERMEL, G. 2006. Testing across configurations: Implications for combinatorial testing. SIGSOFT Softw. Engin. Notes 31, 6, 1-9.
28. COLBOURN, C. J. 2006. Detecting and locating interaction faults. Electron. Not. Discr. Math. 27, 14, 17-18. (Pubitemid 44688659)
29. COLBOURN, C. J.,MARTIROSYAN, S. S.,MULLEN, G. L., SHASHA, D., SHERWOOD, G. B., AND YUCAS, J. L. 2005. Products of mixed covering arrays of strength two. J. Combin. Des. 2, 14, 124-138.
30. COLBOURN, C. J. AND MCCLARY, D. W. 2008. Locating and detecting arrays for interaction faults. J. Combin. Optimiz. 15, 1, 17-48. (Pubitemid 350276194)
31. CZERWONKA, J. 2006. Pairwise testing in the real world: Practical extensions to test-case scenarios. In Proceedings of the 24th Pacific Northwest Software Quality Conference.
32. DALAL, S., JAIN, A., KARUNANITHI, N., LEATON, J., AND LOTT, C. 1998a. Model-based testing of a highly programmable system. In Proceedings of the 9th International Symposium on Software Reliability Engineering (ISSRE'98). IEEE Computer Society, Los Alamitos, CA, 174.
33. DALAL, S. R., JAIN, A., KARUNANITHI, N., LEATON, J. M., LOTT, C. M., PATTON, G. C., AND HOROWITZ, B. M. 1999. Model-based testing in practice. In Proceedings of the 21st International Conference on Software Engineering (ICSE'99). IEEE Computer Society Press, Los Alamitos, CA, 285-294.
34. DALAL, S. R., JAIN, A., PATTON, G., RATHI, M., AND SEYMOUR, P. 1998b. Aetgsm web: A web based service for automatic efficient test generation from functional requirements. In Proceedings of the 2nd Workshop on Industrial-Strength Formal Specification Techniques (WIFT'98). IEEE Computer Society, Los Alamitos, CA, 84-85.
35. DALAL, S. R. AND MALLOWS, C. L. 1998. Factor-Covering designs for testing software. Technometrics 40, 3, 234-243. (Pubitemid 128605632)
36. DENNY, P. C. AND GIBBONS, P. B. 2000. Case studies and new results in combinatorial enumeration. J. Combin. Des. 8, 239-260.
37. DONG, G., NIE, C., XU, B., AND WANG, L. 2007. An effective iterative metamorphic testing algorithm based on program path analysis. In Proceedings of the 7th International Conference on Quality Software (QSIC'07). IEEE Computer Society, Los Alamitos, CA, 292-297.
38. DUMITRESCU, A. 2003. Efficient algorithms for generation of combinatorial covering suites. In Proceedings of the 14th Annual International Symposium Algorithms and Computation (ISAAC'03). Springer, 300-308.
39. DUNIETZ, I. S., EHRLICH, W. K., SZABLAK, B. D., MALLOWS, C. L., AND IANNINO, A. 1997. Applying design of experiments to software testing: experience report. In Proceedings of the 19th International Conference on Software Engineering (ICSE'97). ACM, New York, 205-215.
40. FISHER, R. 1926. The arrangement of field experiments. J. Ministry Agricult. Great Britain 33, 503-513.
41. FOUCHÉ, S.,COHEN, M. B., AND PORTER, A. 2007. Towards incremental adaptive covering arrays. In Proceedings of the the 6th Joint Meeting on European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC-FSE Companion'07). ACM, New York, 557-560. (Pubitemid 351438720)
42. GHAZI, S.A. AHMED, M. 2003. Pair-wise test coverage using genetic algorithms. In Proceedings of the Congress on Evolutionary Computation (CEC'03). IEEE Computer Society, Los Alamitos, CA, 1420-1424.
43. HARMAN, M. 2007. The current state and future of search based software engineering. In Proceedings of the Conference on Future of Software Engineering FOSE'07, 342-357. (Pubitemid 47484936)
44. HARTMAN, A. 2002. Software and hardware testing using combinatorial covering suitess. Graph Theory, Combin. Algor. 6, 3, 237-266.
45. HARTMAN, A. AND RASKIN, L. 2004. Problems and algorithms for covering arrays. Discr.Math. 284, 1-3, 149-156.
46. HNICH, B., PRESTWICH, S., AND SELENSKY, E. 2005. Constraint-Based approaches to the covering test problem. In Proceedings of the Joint ERCIM/Co-LogNET International Workship on Constraint Solving and Constraint Logic Programming. Lecture Notes in Computer Science. Springer Berlin, 172-186.
47. HNICH, B., PRESTWICH, S. D., SELENSKY, E., AND SMITH, B. M. 2006. Constraint models for the covering test problem. Constraints 11, 2-3, 199-219.
48. HULLER, J. 2000. Reducing time to market with combinatorial design method testing. In Proceedings of the the International Council on Systems Engineering (INCOSE) Conference.
49. KAHNG, A. B. AND REDA, S. 2004. Combinatorial group testing methods for the bist diagnosis problem. In Proceedings of the Asia and South Pacific Design Automation Conference (ASP-DAC'04). IEEE Press, 113-116.
50. KOBAYASHI, N. 2002. Design and evaluation of automatic test generation strategies for functional testing of software. Ph.D. thesis, Osaka University, Osaka, Japan.
51. KOBAYASHI, N., TSUCHIYA, T., AND KIKUNO, T. 2002. A new method for constructing pair-wise covering designs for software testing. Inform. Process. Lett. 81, 2, 85-91. (Pubitemid 33050953)
52. KRISHNAN, R.,KRISHNA, S. M., AND NANDHAN, P. S. 2007. Combinatorial testing: Learnings from our experience. SIGSOFT Softw. Engin. Notes 32, 3, 1-8.
53. KUHN, D. R. AND REILLY, M. J. 2002. An investigation of the applicability of design of experiments to software testing. In Proceedings of the 27th Annual NASA Goddard Software Engineering Workshop (SEW'02). IEEE Computer Society, Los Alamitos, CA, 91.
54. KUHN, D. R. AND ANDWALLACE, D. R. 2004. Software fault interactions and implications for software testing. IEEE Trans. Softw. Engin. 30, 6, 418-421.
55. LEI, Y., CARVER, R. H., KACKER, R., AND KUNG, D. C. 2007a. A combinatorial testing strategy for concurrent programs. Softw. Test., Verif. Reliab. 17, 4, 207-225. (Pubitemid 350268689)
56. LEI, Y., KACKER, R., KUHN, D. R., OKUN, V., AND LAWRENCE, J. 2007b. Ipog: A general strategy for t-way software testing. In Proceedings of the 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07). IEEE Computer Society, Los Alamitos, CA, 549-556. (Pubitemid 46900712)
57. LEI, Y., KACKER, R., KUHN, D. R., OKUN, V., AND LAWRENCE, J. 2008. Ipog/ipog-d: Efficient test generation for multi-way combinatorial testing. Softw. Test., Verif. Reliab. 18, 3, 125-148.
58. LEI, Y. AND TAI, K. C. 2001. In-parameter-oder: A test generation strategy for pairwise testing. Tech. rep. TR 2001-03 5, 109-111.
59. LI, Z., HARMAN, M., AND HIERONS, R. M. 2007. Search algorithms for regression test case prioritization. IEEE Trans. Softw. Engin. 33, 4, 225-237. (Pubitemid 46523213)
60. LOTT, C., JAIN, A., AND DALAL, S. 2005. Modeling requirements for combinatorial software testing. SIGSOFT Softw. Engin. Notes 30, 4, 1-7.
61. MAITY, S. AND NAYAK, A. 2005. Improved test generation algorithms for pair-wise testing. In Proceedings of the 16th IEEE International Symposium on Software Reliability Engineering (ISSRE'05). IEEE Computer Society, Los Alamitos, CA, 235-244. (Pubitemid 44733906)
62. MANDL, R. 1985. Orthogonal latin squares: An application of experiment design to compiler testing. Commu. ACM 28, 10, 1054-1058. (Pubitemid 15587965)
63. MARRE, M. AND BERTOLINO, A. 2003. Using spanning sets for coverage testing. IEEE Trans. Softw. Engin. 29, 11, 974-984.
64. MATS, G., LINDSTR, B., OFFUTT, J., AND ANDLER, S. F. 2006a. An evaluation of combination strategies for test case selection. Empir. Softw. Engin. 11, 4, 583-611.
65. MATS, G. AND OFFUTT, J. 2007. Input parameter modeling for combination strategies. In Proceedings of the 25th Conference on IASTED International Multi-Conference (SE'07). ACTA Press, 255-260.
66. MATS, G., OFFUTT, J., AND ANDLER, S. F. 2005. Combination testing strategies: A survey. Softw. Test. Verif. Reliab. 15, 3, 167-199. (Pubitemid 41320808)
67. MATS, G., OFFUTT, J., AND MELLIN, J. 2006b. Handling constraints in the input space when using combination strategies for software testing. Tech. rep. HS-IKI-TR-06-001 1, 1-39.
68. NIE, C., XU, B., SHI, L., AND DONG, G. 2005. Automatic test generation for n-way combinatorial testing. In Proceedings of the 2nd InternationalWorkshop on Software Quality (SOQUA). Springer Berlin, 203-211.
69. NIE, C., XU, B., SHI, L., AND WANG, Z. 2006a. A new heuristic for test suite generation for pair-wise testing. In Proceedings of the International Conference on Software Engineering and Knowledge Engineering, (SEKE'06), 517-521.
70. NIE, C., XU, B., WANG, Z., AND SHI, L. 2006b. Generating optimal test set for neighbor factors combinatorial testing. In Proceedings of the 6th International Conference on Quality Software (QSIC'06). IEEE Computer Society, Los Alamitos, CA, 259-265. (Pubitemid 46948635)
71. NURMELA, K. J. 2004. Upper bounds for covering arrays by tabu search. Discr. Appl. Math. 138, 1-2, 143-152.
72. OFFUTT, A. J. 1994. A practical system formutation testing: Help for the common programmer. In Proceedings of the IEEE International Test Conference on TEST: The Next 25 Years. IEEE Computer Society, Los Alamitos, CA, 824-830.
73. QU, X., COHEN, M., AND WOOLF, K. 2007. Combinatorial interaction regression testing: A study of test case generation and prioritization. In Proceedings of the IEEE International Conferance on Software Maintenance (ICSM). IEEE Computer Society, 413-418.
74. SALEM, A.M. 2001. A software testing model: Using design of experiments (doe) and logistic regression. Ph.D. thesis, Florida Institute of Technology, Melbourne, Florida.
75. SALEM, A. M.,REKAB, K., AND WHITTAKER, J. A. 2004. Prediction of software failures through logistic regression. Inform. Softw. Technol. 46, 12, 781-789.
76. SAMPATH, S., BRYCE, R., VISWANATH, G., KANDIMALLA, V., AND KORU, A. 2008. Prioritizing user-session-based test cases for web applications testing. In Proceedings of the 1st International Conference on Software Testing, Verification, and Validation, 141-150.
77. SCHROEDER, P. J. 2002. Black-Box test reduction using input-output analysis. Ph.D. thesis, Illinois Institute of Technology, Chicago, IL.
78. SCHROEDER, P. J., BOLAKI, P., AND GOPU, V. 2004. Comparing the fault detection effectiveness of n-way and random test suites. In Proceedings of the International Symposium on Empirical Software Engineering (ISESE). IEEE Computer Society, 49-59. (Pubitemid 40062818)
79. SCHROEDER, P. J.,FAHERTY, P., AND KOREL, B. 2002. Generating expected results for automated black-box testing. In Proceedings of the IEEE International Conference on Automated Software Engineering (ASE). IEEE Computer Society, 139-148.
80. SCHROEDER, P. J. AND KOREL, B. 2000a. Black-box test reduction using input-output analysis. SIGSOFT Softw. Engin. Notes 25, 5, 173-177.
81. SCHROEDER, P. J. AND KOREL, B. 2000b. Constraint of model-based test generation using input-output analysis. In Proceedings of the Automated Software Engineering Doctoral Symposium. ACM, 35-43.
82. SEROUSSI, G. AND BSHOUTY, N. 1988. Vector sets for exhaustive testing of logic circuits. IEEE Trans. Inf. Theory 34, 3, 513-522.
83. SHERWOOD, G. 1994. Effective testing of factor combinations. In Proceedings of the 3rd International Conference on Software Testing, Analysis, and Review(STAR94).
84. SHERWOOD, G. B., MARTIROSYAN, S. S., AND COLBOURN, C. J. 2005. Covering arrays of higher strength from permutation vectors. J. Combin. Des. 3, 14, 202-213. (Pubitemid 43670004)
85. SHI, L., NIE, C., AND XU, B. 2005. A software debugging method based on pairwise testing. In Proceedings of the International Conferences on Computational Science (ICCS'05). Springer, 1088-1091. (Pubitemid 41342629)
86. SHIBA, T., TSUCHIYA, T., AND KIKUNO, T. 2004. Using artificial life techniques to generate test cases for combinatorial testing. In Proceedings of the 28th Annual International Computer Software and Applications Conference (COMPSAC'04). IEEE Computer Society, Los Alamitos, CA, 72-77. (Pubitemid 40673832)
87. TAI, K. C. AND LEI, Y. 2002. A test generation strategy for pairwise testing. IEEE Trans. Softw. Engin. 28, 1, 109-111. (Pubitemid 34129838)
88. TUNG, Y.-W. AND ALDIWAN, W. 2000. Automating test case generation for the new generation mission software system. In Proceedings of the IEEE Aerospace Conference. 431-437. (Pubitemid 32838334)
89. TURBAN, R. C. 2006. Algorithms for covering arrays. Ph.D. thesis, University of Ahzam Tempe, AZ.
90. WANG, Z., NIE, C., AND XU, B. 2007. Generating combinatorial test suite for interaction relationship. In Proceedings of the 4th International Workshop on Software Quality Assurance (SOQUA'07). ACM, New York, 55-61.
91. WANG, Z., XU, B., AND NIE, C. 2008. Greedy heuristic algorithms to generate variable strength combinatorial test suite. In Proceedings of the International Conference on Quality Software (QSIC). 155-160.
92. WHITE, L. AND ALMEZEN, H. 2000. Generating test cases for gui responsibilities using complete interaction sequences. In Proceedings of the 11th International Symposium on Software Reliability Engineering (ISSRE'00). IEEE Computer Society, Los Alamtos, CA, 110.
93. WILLIAMS, A. W. 2000. Determination of test configurations for pair-wise interaction coverage. In Proceedings of the IFIP TC6/WG6.1 13th International Conference on Testing Communicating Systems (TestCom'00). Kluwer, 59-74.
94. WILLIAMS, A. W. 2002. Software component interaction testing: coverage measurement and generation of configurations. Ph.D. thesis, University of Ottawa, Canada.
95. WILLIAMS, A. W. AND PROBERT, R. L. 2002. Formulation of the interaction test coverage problem as an integer program. In Proceedings of the IFIP 14th International Conference on Testing Communicating Systems XIV (TestCom'02). Kluwer, 283.
96. WILLIAMS, A. W. AND PROBERT, R. L. 1996. A practical strategy for testing pair-wise coverage of network interfaces. In Proceedings of the 7th International Symposium on Software Reliability Engineering (ISSRE'96). IEEE Computer Society, Los Alamtos, CA, 246.
97. WILLIAMS, A. W. AND PROBERT, R. L. 2001. A measure for component interaction test coverage. In Proceedings of the ACS/IEEE International Conference on Computer Systems and Applications (AICCSA'01). IEEE Computer Society, Los Alamitos, CA, 304.
98. XU, B., NIE, C., SHI, L., CHU, W. C., YANG, H., AND CHEN, H. 2003a. Test plan design for software configuration testing. In Software Engineering Research and Practice, CSREA Press, 686-692.
99. XU, L., XU, B., NIE, C., CHEN, H., AND YANG, H. 2003b. A browser compatibility testing method based on combinatorial testing. In Proceedings of the International Conference on Web Engineering (ICWE. Springer, Berlin, 310-313.
100. YAN, J. AND ZHANG, J. 2006. Backtracking algorithms and search heuristics to generate test suites for combinatorial testing. In Proceedings of the 30th Annual International Conference on Computer Software and Applications (COMPSAC'06). 385-394.
101. YILMAZ, C., COHEN, M. B., AND PORTER, A. A. 2006. Covering arrays for efficient fault characterization in complex configuration spaces. IEEE Trans. Softw. Engin. 32, 1, 20-34. (Pubitemid 43193063)