Combination testing strategies: A survey

Software Testing Verification and Reliability

Volumn 15, Issue 3, 2005, Pages 167-199

  Grindal, Mats ^a   Offutt, Jeff ^a   Andler, Sten F ^b  

^a UNIVERSITY OF SKÖVDE   (Sweden)

^b George Mason University   (United States)

Author keywords

AETG; Category partition; Combination strategies; Orthogonal arrays; Test case selection

Indexed keywords

GENETIC ALGORITHMS; HEURISTIC METHODS; ITERATIVE METHODS; TESTING;

CATEGORY PARTITION; COMBINATION STRATEGIES; ORTHOGONAL ARRAYS; TEST CASE SELECTION;

SOFTWARE ENGINEERING;

EID: 24944512941     PISSN: 09600833     EISSN: None     Source Type: Journal    
DOI: 10.1002/stvr.319     Document Type: Article

References (53)

1. Ostrand TJ, Balcer MJ. The Category-Partition method for specifying and generating functional tests. Communications of the ACM 1988; 31 (6):676-686.
2. Myers GJ. The Art of Software Testing. Wiley: New York, 1979.
3. Beizer B. Software Testing Techniques. Van Nostrand Reinhold: New York, 1990.
4. Myers GJ. A controlled experiment in program testing and code walkthroughs/inspection. Communications of the ACM 1978; 21 (9):760-768.
5. Basili VR, Selby RW. Comparing the effectiveness of software testing strategies. IEEE Transactions on Software Engineering 1987; 13 (12):1278-1296.
6. Reid S. An empirical analysis of equivalence partitioning, boundary value analysis and random testing. Proceedings of the 4th International Software Metrics Symposium (METRICS'97), Albuquerque, NM, 5-7 November 1997. IEEE Computer Society Press: Los Alamitos, CA, 1997; 64-73.
7. Wood M, Roper M, Brooks A, Miller J. Comparing and combining software defect detection techniques: A replicated study. Proceedings of the Sixth European Software Engineering Conference (ESEC/FSE 97) (Lecture Notes in Computer Science, vol. 1013). Springer: Bedin, 1997; 262-277.
8. So SS, Cha SD, Shimeall TJ, Kwon YR. An empirical evaluation of six methods to detect faults in software. Software Testing, Verification and Reliability 2002; 12(3):155-171.
9. Duran JW, Ntafos SC. An evaluation of random testing. IEEE Transactions on Software Engineering 1984; 10(4):438-444.
10. Hamlet D, Taylor R. Partition testing does not inspite confidence. IEEE Transactions on Software Engineering 1990; 16 (12):1402-1411.
11. Gutjahr WJ. Partition testing vs. random testing: The influence of uncertainty. IEEE Transactions on Software Engineering 1999; 25 (5):661-674.
12. Yin H, Lebne-Dengel Z, Malaiya YK. Automatic test generation using checkpoint encoding and antirandom testing. Technical Report CS-97-116, Colorado State University, 1997.
13. Grochtmann M, Grimm K. Classification trees for partition testing. Software Testing, Verification and Reliability 1993; 3(2):63-82.
14. Grochtmann M, Grimm K, Wegener J. Tool-supported test case design for black-box testing by means of the classificationtree editor. Proceedings of the 1st European International Conference on Software Testing, Analysis & Review (EuroSTAR 1993), London, U.K., October 1993. Qualtech Conferences, 1993; 169-176.
15. Lehmann E, Wegener J. Test case design by means of the CTE XL. Proceedings of the 8th European International Conference on Software Testing, Analysis & Review (EuroSTAR 2000), Copenhagen, Denmark, December 2000. Qualtech Conferences, 2000; 1-10.
16. Williams AW, Probert RL. A measure for component interaction test coverage. Proceedings of the ACSI/IEEE International Conference on Computer Systems and Applications (AICCSA 2001), Beirut, Lebanon, June 2001. IEEE Computer Society Press: Los Alamitos, CA, 2001; 304-311.
17. Cohen DM, Dalal SR, Fredman ML, Patton GC. The AETG system: An approach to testing based on combinatorial design. IEEE Transactions on Software Engineering 1997; 23(7):437-444.
18. Grindal M, Lindström B, Offutt AJ, Andler SF. An evaluation of combination strategies for test case selection. Technical Report HS-IDA-TR-03-001, Department of Computer Science, University of Skövde, 2003.
19. Cohen MB, Gibbons PB, Mugridge WB, Colburn CJ. Constructing test cases for interaction testing. Proceedings of the 25th International Conference on Software Engineering (ICSE'03), May 2003. IEEE Computer Society Press: Los Alamitos, CA, 2003; 38-48.
20. Ammann PE, Offutt AJ. Using formal methods to derive test frames in category-partition testing. Proceedings of the Ninth Annual Conference on Computer Assurance (COMPASS'94), Gaithersburg, MD, June 1994. IEEE Computer Society Press: Los Alamitos, CA, 1994; 69-80.
21. Burroughs K, Jain A, Erickson RL. Improved quality of protocol testing through techniques of experimental design. Proceedings of the IEEE International Conference on Communications (Supercomm/ICC'94), New Odeans, LA, 1-5 May 1994. IEEE Computer Society Press: Los Alamitos, CA, 1994; 745-752.
22. Cohen DM, Dalal SR, Kajla A, Patton GC. The automatic efficient test generator (AETG) system. Proceedings of Fifth International Symposium on Software Reliability Engineering (ISSRE'94), Monterey, CA, 6-9 November 1994. IEEE Computer Society Press: Los Alamitos, CA, 1994; 303-309.
23. Cohen DM, Dalal SR, Parelius J, Patton GC. The combinatorial design approach to automatic test generation. IEEE Software 1996, 13 (5):83-89.
24. Shiba T, Tsuchiya T, Kikuno T. Using artificial life techniques to generate test cases for combinatorial testing. Proceedings of 28th Annual International Computer Software and Applications Conference (COMPSAC'04), Hong Kong, China, 28-30 September 2004. IEEE Computer Society Press: Los Alamitos, CA, 2004; 72-77.
25. Mandl R. Orthogonal Latin Squares: An application of experiment design to compiler testing. Communications of the ACM 1985; 28 (10):1054-1058.
26. Burr K, Young W. Combinatorial test techniques: Table-based automation, test generation and code coverage. Proceedings of the International Conference on Software Testing, Analysis, and Review (STAR'98), San Diego, CA, 26-28 October 1998. Software Quality Engineering, 1998.
27. Malaiya YK. Antirandom testing: Getting the most out of black-box testing. Proceedings of the International Symposium on Software Reliability Engineering (ISSRE'95), Toulouse, France, October 1995. IEEE Computer Society Press: Los Alamitos, CA, 1995; 86-95.
28. Hoffman DM, Strooper PA, White L. Boundary values and automated component testing. Software Testing, Verification and Reliability 1999; 9(1):3-26.
29. Sherwood G. Effective testing of factor combinations. Proceedings of the Third International Conference on Software Testing, Analysis, and Review (STAR94), Washington, DC, Software Quality Engineering, May 1994. Software Quality Engineering, 1994.
30. Lei Y, Tai KC. In-parameter-order: A test generation strategy for pair-wise testing. Proceedings of the Third IEEE High Assurance Systems Engineering Symposium, November 1998. IEEE Computer Society Press: Los Alamitos, CA, 1998; 254-261.
31. Lei Y, Tai KC. A test generation strategy for pairwise testing. Technical Report TR-2001-03, Department of Computer Science, North Carolina State University, Raleigh, 2001.
32. Tai KC, Lei Y. A test generation strategy for pairwise testing. IEEE Transactions on Software Engineering 2002; 28(1):109-111.
33. Huller J. Reducing time to market with combinatorial design method testing. Proceedings of 10th Annual International Council on Systems Engineering (INCOSE'00), Minneapolis, MN, 16-20 July 2000. INCOSE, 2000.
34. Williams AW, Probert RL. A practical strategy for testing pair-wise coverage of network interfaces. Proceedings of the 7th International Symposium on Software Reliability Engineering (ISSRE'96), White Plains, New York, 30 October-2 November 1996. IEEE Computer Society Press: Los Alamitos, CA, 1996; 246-254.
35. Williams AW. Determination of test configurations for pair-wise interaction coverage. Proceedings of the 13th International Conference on the Testing of Communicating Systems (TestCom 2000), Ottawa, Canada, August 2000. Kluwer Academic: Norwell, MA, 2000; 59-74.
36. Williams AW, Probert RL. Formulation of the interaction test coverage problem as an integer problem. Proceedings of the 14th International Conference on the Testing of Communicating Systems (TestCom 2002), Berlin, Germany, March 2002. European Telecommunications Standards Institute, ETSI, 2002; 283-298.
37. Kobayashi N, Tsuchiya T, Kikuno T. A new method for constructing pair-wise covering designs for software testing. Information Processing Letters 2002; 81(2):85-91.
38. Rapps S, Weyuker EJ. Selecting software test data using data flow information. IEEE Transactions on Software Engineering 1985; 11 (4):367-375.
39. Heller E. Using design of experiment structures to generate software test cases. Proceedings of the 12th International Conference on Testing Computer Software, New York, 1995. ACM Press: New York, 1995; 33-11.
40. Dalal S, Mallows CL. Factor-covering designs for testing software. Technometrics 1998; 50(3):234-243.
41. Kuhn DR, Reilly MJ. An investigation of the applicability of design of experiments to software testing. Proceedings of the 27th NASA/IEEE Software Engineering Workshop, NASA Goddard Space Flight Center, MD, 4-6 December 2002. NASA/IEEE Computer Society Press: Los Alamitos, CA, 2002; 91-95.
42. Kuhn DR, Wallace DR, Gallo AM Jr. Software fault interactions and implications for software testing. IEEE Transactions on Software Engineering 2004; 30(6):418-421.
43. Dunietz IS, Ehrlich WK, Szablak BD, Mallows CL, Iannino A. Applying design of experiments to software testing. Proceedings of 19th International Conference on Software Engineering (ICSE'97), Boston, MA, May 1997. ACM Press: New York, 1997; 205-215.
44. Piwowarski P, Ohba Mn Caruso J. Coverage measurement experience during function test. Proceedings of 15th International Conference on Software Engineering (ICSE'93), Baltimore, MD, May 1993. IEEE Computer Society Pass: Los Alamitos, CA, 1993; 287-301.
45. Dalal SR, Jain A, Karunanithi N, Leaton JM, Lott CM, Patton GC, Horowitz BM. Model-based testing in practice. Proceedings of 21st International Conference on Software Engineering (ICSE'99), Los Angeles, CA, 16-22 May 1999. ACM Press: New York, 1999; 285-294.
46. Toczki J, Kocsis F, Gyimóthy T, Dányi G, Kókai G. Sys/3-a software development tool. Proceedings of the Third International Workshop on Compiler Compilers (CC'90), Schwerin, Germany, 22-24 October 1990 (Lecture Notes in Computer Science, vol. 477). Springer: Berlin, 1990; 193-207.
47. Brownlie R, Prowse J, Phadke MS. Robust testing of AT&T PMX/StarMAIL using OATS. AT&T Technical Journal 1992; 71(3):41-47.
48. Dalal SR, Jain A, Patton GC, Rathi M, Seymour P. AETG web: A web based service for automatic efficient test generation from functional requirements. Proceedings of 2nd IEEE Workshop on Industrial-Strength Formal Specification Techniques, Boca Raton, FL, 21-23 October 1999. IEEE Computer Society Press: Los Alamitos, CA, 1999; 84-85.
49. Daley N, Hoffman D, Strooper P. A framework for table driven testing of Java classes. Software - Practice and Experience 2002; 32 (5):465-493.
50. Dalal SR, Jain A, Karunanithi N, Leaton JM, Lott CM. Model-based testing of a highly programmable system. Proceedings of the 9th International Symposium in Software Reliability Engineering (ISSRE'98), Paderborn, Germany, 4-7 November 1998. IEEE Computer Society Press: Los Alamitos, CA, 1998; 174-178.
51. Yilmaz C, Cohen MB, Porter A. Covering arrays for efficient fault characterization in complex configuration spaces. Proceedings of the ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2004), Boston, MA. ACM Software Engineering Notes 2004; 29(4):45-54.
52. Kropp NP, Koopman PJ, Siewiorek DP. Automated robustness testing of off-the-shelf software components. Proceedings of FTCS'98: Fault Tolerant Computing Symposium, Munich, Germany, 23-25 June 1998. IEEE Computer Society Press: Los Alamitos, CA, 1998; 230-239.
53. Berling T, Runesson P. Efficient evaluation of multi-factor dependent system performance using fractional factorial design. IEEE Transactions on Software Engineering 2003; 29(9):769-781.