Combinatorial Testing of Software with Binary Inputs: A State-of-the-Art Review

Proceedings - 2016 IEEE International Conference on Software Quality, Reliability and Security-Companion, QRS-C 2016

Volumn , Issue , 2016, Pages 55-60

  Vilkomir, Sergiy ^a  

^a EAST CAROLINA UNIVERSITY   (United States)

Author keywords

binary; combinatorial testing; pairwise; t way

Indexed keywords

APPLICATION PROGRAMS; BINS; C (PROGRAMMING LANGUAGE); COMPUTER SOFTWARE SELECTION AND EVALUATION; NETWORK SECURITY; NUCLEAR POWER PLANTS; SAFETY ENGINEERING; SAFETY TESTING; SOFTWARE RELIABILITY;

BINARY; COMBINATORIAL METHOD; COMBINATORIAL TESTING; EFFECTIVE APPROACHES; PAIRWISE; SAFETY CRITICAL APPLICATIONS; STATE-OF-THE ART REVIEWS; T-WAY;

SOFTWARE TESTING;

EID: 84991800788     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/QRS-C.2016.12     Document Type: Conference Paper

References (47)

1. Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence, "IPOG: A general strategy for t-way software testing," Proceedings of the 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS'07), Tucson, AZ, USA, 26-29 March 2007, pp. 549-556.
2. J. Czerwonka, "Pairwise Testing in the Real World: Practical Extensions to Test Case Generators," Proceedings of 24th Pacific Northwest Software Quality Conference, Portland, Oregon, USA, 9-11October 2006, pp. 419-430.
3. D. R. Kuhn, R. Kacker, and Y. Lei, Introduction to Combinatorial Testing, Chapman and Hall/CRC, 2013, 341 pages.
4. M. Grindal, J. Offutt, and S.F.,ler, "Combination testing strategies: a survey," Software Testing, Verification and Reliability, vol. 15, no. 3, Mar. 2005, pp. 167-199.
5. C. Nie and H. Leung, "A survey of combinatorial testing," ACM Computing Surveys (CSUR) 43, no. 2, 2011.
6. J. Chilenski and S. Miller, "Applicability of Modified Condition/Decision Coverage to Software Testing," Software Engineering Journal, September 1994, pp. 193-200.
7. RTCA, DO-178B Standard "Software Considerations in Airborne Systems and Equipment Certification," 1982.
8. N. Kobayashi, T. Tsuchiya, and T. Kikuno, "Non-specification-based approaches to logic testing for software," Information and Software Technology, vol. 44, no. 2, Feb. 2002, pp. 113-121.
9. N.G. Leveson, M.P.E. Heimdahl, H. Hildreth, and J.D. Reese, "Requirements specification for process-control systems," IEEE Trans. Software Engineering, vol. 20, no. 9, Sep. 1994, pp. 684-707.
10. E. Weyuker, T. Goradia, and A. Singh, "Automatically generating test data from a Boolean specification," IEEE Trans. Software Engineering, vol. 20, no. 5, May 1994, pp. 353-363.
11. T. Y. Chen and M. F. Lau, "Two test data selection strategies towards testing of Boolean specifications," Proceedings of the Twenty-First Annual International Computer Software and Applications Conference (COMPSAC'97), Washington DC, 13-15 Aug. 1997, pp. 608-611.
12. D. R. Kuhn, "Fault classes and error detection capability of specification-based testing," ACM Trans. Software Engineering and Methodology (TOSEM), vol. 8, no. 4, Oct. 1999, pp. 411-424.
13. Y. Malaiya, "Antirandom testing: getting the most out of black-box testing," Proceedings of the Sixth International Symposium on Software Reliability Engineering, Toulouse, France, 24-27 Oct. 1995, pp. 86-95.
14. W. Ballance, W. Jenkins, and S. Vilkomir, "Probabilistic assessment of effectiveness of software testing for safety-critical systems," Proceedings of 10th Int. Probabilistic Safety Assessment and Management Conference (PSAM 10), 7-11 June 2010, Seattle, Washington, USA.
15. W. Ballance, S. Vilkomir, and W. Jenkins, "Effectiveness of Pairwise Testing for Software with Boolean Inputs," Proceedings of the Fifth International Conference on Software Testing, Verification and Validation (ICST 2012), April 17-21, 2012, Montreal, Canada, pp. 580-586.
16. S. Vilkomir, O. Starov, and R. Bhambroo, "Evaluation of t-wise Approach for Testing Logical Expressions in Software," Proceedings of the IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops (ICSTW 2013), 18-20 March 2013, Luxembourg, Luxembourg, pp. 249-256.
17. W. Jenkins, S. Vilkomir, and W. Ballance, "Fault Evaluator: a tool for experimental investigation of effectiveness in software testing," Proceedings of the 2010 IEEE Int. Conference on Progress in Informatics and Computing (PIC 2010), 10-12 December 2010, Shanghai, China, pp. 1077-1083.
18. NIST, "ACTS tool," http://csrc.nist.gov/groups/SNS/acts/Accessed on June 2016.
19. Z. Wang and Y. Qi, "Why combinatorial testing works: Analyzing minimal failure-causing schemas in logic expressions," Proceedings of the IEEE 8th International Conference on Software Testing, Verification and Validation Workshops (ICSTW 2015), 13-17 April 2015, Graz, Austria.
20. Z. Chen, T. Y. Chen, and B. Xu, "A revisit of fault class hierarchies in general Boolean specifications." ACM Transactions on Software Engineering and Methodology (TOSEM) 20, no. 3, 2011.
21. V. Okun, "Specification mutation for test generation and analysis," PhD Diss., University of Maryland Baltimore County, 2004.
22. S. Vilkomir, K. Kapoor, and J. Bowen. "Tolerance of control-flow testing criteria," Proceedings of 27th IEEE Annual International Computer Software and Applications Conference (COMPSAC 2003), Dallas, Texas, USA, 3-6 November 2003, pp. 182-187.
23. S. Vilkomir and G. Pennell, "Should We Care about "Don't Care" Testing Inputs Empirical Investigation of Pair-Wise Testing," Proceedings of the IEEE 9th International Conference on Software Testing, Verification and Validation Workshops (ICSTW 2016), 10-15 April 2016, Chicago, USA, pp. 39-45.
24. I. Segall, R. Tzoref-Brill, and E. Farchi, "Using binary decision diagrams for combinatorial test design," Proceedings of the 2011 International Symposium on Software Testing and Analysis, 17-21 July 2011, Toronto, ON, Canada, pp. 254-264.
25. E. Salecker, R. Reicherdt, and S. Glesner, "Calculating prioritized interaction test sets with constraints using binary decision diagrams," Proceedings of the IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops, Berlin, Germany, 21-25 March 2011, pp. 278-285.
26. J. Lawrence, R. Kacker, Y. Lei, D. R. Kuhn, and M. Forbes, "A survey of binary covering arrays," The Electronic Journal of Combinatorics, 18, no. 1, 2011.
27. A. Calvagna and A. Gargantini, "A logic-based approach to combinatorial testing with constraints," Proceedings of the Second International Conference on Tests and Proofs (TAP 2008), Prato, Italy, 9-11 April 2008, LNCS 4966, pp. 66-83.
28. A. Gargantini and P. Vavassori, "Efficient combinatorial test generation based on multivalued decision diagrams," Proceedings of the 10th International Haifa Verification Conference "Hardware and Software: Verification and Testing" (HVC 2014), Haifa, Israel, 18-20 November 2014, Springer, LNCS 8855, pp. 220-235.
29. A. Gargantini and P. Vavassori, "Using decision trees to aid algorithm selection in combinatorial interaction tests generation," Proceedings of the IEEE 8th International Conference on Software Testing, Verification and Validation Workshops (ICSTW 2015), 13-17 April 2015, Graz, Austria.
30. L. Yu, W. Tsai, W. Zhao, J. Zhu, and Q. Wang, "Towards Selecting Test Data Using Topological Structure of Boolean Expressions," Proceedings of the Ninth International Conference on Quality Software (QSIC 2009), 24-25 August 2009, Jeju, Korea, pp. 31-40.
31. O. Mondragon, A. Q. Gates, and S. Roach, "Prospec: Support for Elicitation and Formal Specification of Software Properties," Electronic Notes in Theoretical Computer Science 89, no. 2, 2003, pp. 67-88.
32. S. Salamah, O. Ochoa, and Y. Jacquez, "Using Pairwise Testing to Verify Automatically-Generated Formal Specifications," Proceedings of the IEEE 16th International Symposium on High Assurance Systems Engineering (HASE 2015), Daytona Beach Shores, FL, USA, 8-10 Jan. 2015, pp. 279-280.
33. I. Chung, "Modeling Pairwise Test Generation from Cause-Effect Graphs as a Boolean Satisfiability Problem," International Journal of Contents, Jul, 10(3), 2014, pp. 41-46.
34. D. Jackson, Software Abstractions, MIT Press, 2006.
35. J. Bracho-Rios, J. Torres-Jimenez, and E. Rodriguez-Tello, "A new backtracking algorithm for constructing binary covering arrays of variable strength," Proceedings of the 8th Mexican International Conference on Artificial Intelligence (MICAI-2009), 9-13 November 2009, Guanajuato, Mexico, Springer, LNCS 5845, pp. 397-407.
36. E. Rodriguez-Tello and J. Torres-Jimenez, "Memetic algorithms for constructing binary covering arrays of strength three," 9th international conference on Artificial Evolution (EA'09), 26-28 October 2009, Strasbourg, France, Revised Selected Papers, Springer, LNCS 5975, pp. 86-97.
37. J. Torres-Jimenez and E. Rodriguez-Tello, "New bounds for binary covering arrays using simulated annealing," Information Sciences, 185, no. 1, 2012, pp. 137-152.
38. M. B. Cohen, C. J. Colbourn, and A. C. Ling, "Constructing strength three covering arrays with augmented annealing," Discrete Mathematics, 308 (13), 2008, pp. 2709-2722.
39. C. Yilmaz, "Test case-aware combinatorial interaction testing," IEEE Transactions on Software Engineering, 39, no. 5, 2013, pp. 684-706.
40. C. Yilmaz, E. Dumlu, M. B. Cohen, and A. Porter, "Reducing Masking Effects in CombinatorialInteraction Testing: A Feedback Driven Adaptive Approach," IEEE Transactions on Software Engineering, 40, no. 1, 2014, pp. 43-66.
41. R. Bryce, A. Rajan, and M. P. E. Heimdahl, "Interaction testing in model-based development: Effect on model-coverage," Proceedings of the 13th Asia Pacific Software Engineering Conference (APSEC'06), 6-8 Dec. 2006, Bangalore, India, pp. 259-268.
42. S. Miller, A. Tribble, T. Carlson, and E. J. Danielson, "Flight guidance system requirements specification," Technical Report CR-2003-212426, NASA, June 2003.
43. S. Vilkomir and D.,erson, "Relationship between pair-wise and MC/DC testing: Initial experimental results," Proceedings of the IEEE 8th International Conference on Software Testing, Verification and Validation Workshops (ICSTW 2015), 13-17 April 2015, Graz, Austria.
44. CodeCover, http://codecover.org Accessed on June 2016.
45. R. Bartholomew, "An industry proof-of-concept demonstration of automated combinatorial test," Proceedings of the IEEE 8th International Workshop on Automation of Software Test (AST), San Francisco, CA, USA, 18-19 May 2013, pp. 118-124.
46. R. Bartholomew, "Using Combinatorial Testing to Reduce Software Rework," CrossTalk, 23, 2014, pp. 23-26.
47. J. Czerwonka, "On Use of Coverage Metrics in Assessing Effectiveness of Combinatorial Test Designs," Proceedings of the IEEE Sixth International Conference on Software Testing, Verification and Validation Workshops (ICSTW 2013), 18-20 March 2013, Luxembourg, Luxembourg, pp. 257-266.