Generating structurally complex test cases by data mutation: A case study of testing an automated modelling tool

Computer Journal

Volumn 52, Issue 5, 2009, Pages 571-588

  Shan, Lijun ^a   Zhu, Hong ^b  

^a NATIONAL UNIVERSITY OF DEFENSE TECHNOLOGY   (China)

^b OXFORD BROOKES UNIVERSITY   (United Kingdom)

Author keywords

Data mutation method; Modelling tools; Software testing; Test case generation

Indexed keywords

AUTOMATED MODELLING; COST EFFECTIVE; DATA MUTATION; DATA MUTATION METHOD; EXPERIMENT DATA; INPUT DATA; MODELLING TOOLS; MUTATION OPERATORS; MUTATION TESTING; TEST CASE; TEST CASE GENERATION; TEST DATA; TESTING SOFTWARE;

COMPUTER SOFTWARE SELECTION AND EVALUATION; SOFTWARE TESTING; TESTING;

AUTOMATIC TEST PATTERN GENERATION;

EID: 68049104013     PISSN: 00104620     EISSN: 14602067     Source Type: Journal    
DOI: 10.1093/comjnl/bxm043     Document Type: Conference Paper

References (83)

1. Shan, L. and Zhu, H. (2006) Testing software modelling tools using data mutation. Proc. Int. Workshop on Automation of Software Test (AST'06) at ICSE'06, Shanghai, China, May 23, pp. 43-49. ACM, New York, NY, USA.
2. Howden, W. E. (1975) Methodology for the generation of program test data. IEEE Trans. Comput., 24, 554-560.
3. Ramamoorthy, C., Ho, S. and Chen, W. (1976) On the automated generation of program test data. IEEE Trans. Softw. Eng., SE-2, 293-300.
4. King, J. (1975) A new approach to program testing. Proc. Int. Conf. Reliable Software, Los Angeles, CA, USA, April 21-23, pp. 228-233. ACM, New York, NY, USA.
5. Clarke, L. (1976) A system to generate test data and symbolically execute programs. IEEE Trans. Softw. Eng., SE-2, 215-222.
6. Howden, W. E. (1977) Symbolic testing and the DISSECT symbolic evaluation system. IEEE Trans. Softw. Eng., SE-3, 266-278.
7. Howden, W. E. (1978) An evaluation of the effectiveness of symbolic testing. Softw. Pract. Exp., 8, 381-397.
8. Korel, B. (1990) Automated software test data generation. IEEE Trans. Softw. Eng., SE-16, 870-879.
9. Beydeda, S. and Gruhn, V. (2003) BINTEST - binary search-based test case generation. Proc. 27th Int. Computer Software and Applications Conf. (COMPSAC'03), Dallas, TX, USA, November 3-6, pp. 28-33. IEEE Computer Society, Los Alamitos, CA, USA.
10. Gupta, N., Mathur, A. P. and Soffa, M. L. (2000) Generating Test Data for Branch Coverage. Proc. 15th IEEE Int. Conf. Automated Software Engineering (ASE'00), Grenoble, France, September 11-15, pp. 219-228. IEEE Computer Society Press, Los Alamitos, CA, USA.
11. Pargas, R. P., Harrold, M. J. and Peck, R. R. (1999) Test-data generation using genetic algorithms. Softw. Test. Verif. Reliab., 9, 263-282.
12. DeMillo, R. A., Guindi, D. S., McCracken, W. M., Offutt, A. J. and King, K. N. (1988) An extended overview of the Mothra software testing environment. Proc. 2nd Workshop on Software Testing, Verification and Analysis, Banff, Canada, July 19-21, pp. 142-151. IEEE Computer Society, Washington, DC, USA.
13. DeMillo, R. A. and Offutt, A. J. (1991) Constraint-based automatic test data generation. IEEE Trans. Softw. Eng., SE-17, 900-909.
14. DeMillo, R. A. and Offutt, A. J. (1993) Experimental results from an automatic test case generator. ACM Trans. Softw. Eng. Methodol., 2, 109-127.
15. McMinn, P. (2004) Search-based software test data generation: A survey. Softw. Test. Verif. Reliab., 14, 105-156.
16. McMinn, P. and Holcombe, M. (2003) The state problem for evolutionary testing. Proc. Genetic and Evolutionary Computation Conf. (GECCO'03 Chicago, IL, USA, July 12-16, pp. 2488-2497. Springer-Verlag, Berlin.
17. Zhu, H., Hall, P. and May, J. (1997) Software unit test coverage and adequacy. ACM Comput. Surv., 29, 366-427.
18. Tai, K.-C. (1993) Predicate-based test generation for computer programs. Proc. 15th Int. Conf. Software Engineering (ICSE'93), Baltimore, MD, USA, May 17-21, pp. 267-276. IEEE Computer Society, Los Alamitos, CA, USA.
19. Stocks, P. A. and Carrington, D. A. (1993) Test templates: A specification-based testing framework. Proc. 15th Int. Conf. Software Engineering (ICSE'93), Baltimore, MD, USA, May 17-21, pp. 405-414. IEEE Computer Society, Los Alamitos, CA, USA.
20. Ammann, P. and Offutt, J. (1994) Using formal methods to derive test frames in category-partition testing. Proc. 9th IEEE Annual Conf. Computer Assurance (COMPASS'94), Gaithersburg, MD, USA, 27 June-1 July, pp. 69-80. IEEE Computer Society, Los Alamitos, CA, USA.
21. Denney, R. (1991) Test-case generation from Prolog-based specifications. IEEE Softw., 8, 49-57.
22. Bouge, L., Choquet, N., Fribourg, L. and Gaudel, M.-C. (1986) Test set generation from algebraic specifications using logic programming. J. Syst. Softw., 6, 343-360.
23. Doong, R. K. and Frankl, P. G. (1994) The ASTOOT approach to testing object-oriented programs. ACM Trans. Softw. Eng. Methodol., 3 101-130.
24. Chen, H. Y., Tse, T. H. and Chen, T. Y. (2001) TACCLE: A methodology for object-oriented software testing at the class and cluster levels. ACM Trans. Softw. Eng. Methodol., 10, 56-109.
25. Fujiwara, S., Bochmann, G., Khendek, F., Amalou, M. and Ghedamsi, A. (1991) Test selection based on finite state models. IEEE Trans. Softw. Eng., SE-17, 591-603.
26. Lee, D. and Yannakakis, M. (1996) Principles and methods of testing finite state machines - a survey. Proc. IEEE, 84, 1090-1123.
27. Hierons, R. M. (2001) Checking states and transitions of a set of communicating finite state machines. Microprocess. Microsyst., 24, 443-452.
28. Morasca, S. and Pezze, M. (eds) (1990) Using High-Level Petri Nets for Testing Concurrent and Real-Time Systems. North Holland, Amsterdam.
29. Zhu, H. and He, X. (2002) A methodology of testing high-level Petri nets. Inf. Softw. Technol., 44, 473-489.
30. Zhu, H., Jin, L. and Diaper, D. (1999) Application of task analysis to the validation of software requirements. Proc. 11th Int. Conf. Software Engineering and Knowledge Engineering (SEKE'99), Kaiserslautern, Germany, June 16-19, pp. 239-245. Knowledge Systems Institute, Skokie, IL, USA.
31. Zhu, H., Jin, L. and Diaper, D. (2002) Software requirements validation via task analysis. J. Syst. Softw., 61, 145-169.
32. Hartman, A. and Nagin, K. (2004) The AGEDIS tools for model based testing. Proc. ACM SIGSOFT Int. Symp. Software Testing and Analysis (ISSTA'04), Boston, MA, USA, July 11-14, pp. 129-132. ACM, New York, NY, USA.
33. Offutt, J. and Abdurazik, A. (2000) Using UML collaboration diagrams for static checking and test generation. Proc. 3rd Int. Conf. Unified Modelling Language (UML'00), York, UK, October 2-6, pp. 383-395. Springer-Verlag, Berlin, Heidelberg.
34. Tahat, L. H., Bader, A. J., Vaysburg, B. and Korel, B. (2001) Requirement-based automated black-box test generation. Proc. 25th Int. Computer Software and Applications Conf. (COMPSAC'01), Chicago, IL, USA, October 8-12, pp. 489-495. IEEE Computer Society, Los Alamitos, CA, USA.
35. Vaysburg, B., Tahat, L. H. and Korel, B. (2002) Dependence analysis in reduction of requirement based test suites. Proc. ACM SIGSOFT International Symposium Software Testing and Analysis (ISSTA'02), Roma, Italy, July 22-24, pp. 107-111. ACM New York, NY, USA.
36. Li, S., Wang, J. and Qi, Z.-C. (2004) Property-oriented test generation from UML statecharts. Proc. 19th Int. Conf. Automated Software Engineering (ASE'04), Linz, Austria, September 20-25, pp. 122-131. IEEE Computer Society, Los Alamitos, CA, USA.
37. Li, S., Wang, J., Wang, X. and Qi, Z.-C. (2005) Configuration-oriented symbolic test sequence construction method for EFSM. Proc. 29th Annual International Computer Software and Applications Conf. (COMPSAC'05), Edinburgh, Scotland, UK, July 25-28, pp. 13-18. IEEE Computer Society, Los Alamitos, CA, USA.
38. Pretschner, A. (2005) Model-based testing (Tutorial Abstract). Proc. 27th Int Conf. Software Engineering (ICSE'05), St. Louis, MO, USA, May 15-21, pp. 722-723. ACM, New York, NY, USA.
39. Pretschner, A., Prenninger, W., Wagner, S., Kühnel, C., Baumgartner, M., Sostawa, B., Zölch, R. and Stauner, T. (2005) One evaluation of model-based testing and its automation. Proc 27th Int Conf. Software Engineering (ICSE'05), St. Louis, Mo, USA, May 15-21, pp. 392-401, New York, NY, USA.
40. Heimdahl, M. P. E. (2005) Model-based testing: Challenges ahead (Panel Discussion Position paper). Proc. 29th Annual Int. Computer Software and Applications Conf. (COMPSAC'05), Edinburgh, Scotlands, UK, July 25-28, pp. 330. IEEE Computer Society, Los Alamitos, CA, USA.
41. Avritzer, A. and Larson, B. (1993) Load testing software using deterministic state testing. Proc. ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA'93), June 28-30, pp. 82-88. ACM, New York, NY, USA.
42. Avritzer, A. and Weyuker, E. J. (1994) Generating test suites for software load testing. Proc. ACM SIGSOFT Int. Symp. Software Testing and Analysis (ISSTA'94), International Symposium August 17-19, pp. 44-57. ACM, New York, NY, USA.
43. Whittaker, J.A. and Poore, J.H. (1993) Markov analysis of software specifications. ACM Trans. Softw. Eng. Methodol., 2, 93-106.
44. Guen, H.L., Marie, R. and Thelin, T. (2004) Reliability estimation for statistical usage testing using Markov chains. Proc. 15th Int. Symp. Software Reliability Engineering (ISSRE'04), Rennes and Saint-Malo, France, November 02-05, pp. 54-65. IEEE Computer Society, Los Alamitos, CA, USA.
45. Prowell, S. J. (2005) Using Markov chain usage models to test complex systems. Proc. 38th Annual Hawaii International Conf. System Sciences (HICSS'05), Big Island, HI, January 3-6, pp. 318. IEEE Computer Society Los Alamitos, CA, USA.
46. Bertolini, C., Farina, A. G., Fernandes, P. and Oliveira, F. M. (2004) Test case generation using stochastic automata networks: Quantitative analysis. Proc. 2nd Int. Conf. Software Engineering and Formal Methods (SEFM'04), Beijing, China, September 28-30, pp. 251-260. IEEE Computer Society, Los Alamitos, CA, USA.
47. Farina, A. G., Fernandes, P. and Oliveira, F. M. (2002) Measurement and empirical software engineering: Representing software usage models with stochastic automata networks. Proc. 14th Int. Conf. Software Engineering and Knowledge Engineering (SEKE'02), Ischia, Italy, July 15-19, pp. 401-406. ACM, New York, NY, USA.
48. Fine, S. and Ziv, A. (2003) Coverage directed test generation for functional verification using Bayesian networks. Proc. 40th ACM IEEE Annual Conf. Design Automation (DAC'03), Anaheim, CA, USA, June 2-6, pp. 286-291. ACM, New York, NY, USA.
49. Yang, C. S. and Pollock, L. (1996) Towards a structural load testing tool. Proc. ACM SIGSOFT Int. Symp. Software Testing and Analysis (ISSTA'96), San Diego, CA, USA, January 8-10, pp. 201-208. ACM, New York, NY, USA.
50. Zhang, J., Xu, C. and Cheung, S. C. (2001) Automatic generation of database instances for white-box testing. Proc. 25th Int. Computer Software and Applications Conf. (COMPSAC'01), Chicago, IL, USA, October 8-12, pp. 161-165. IEEE Computer Society.
51. Fisher, M., Cao, M., Rothermel, G., Cook, C. and Burnett, M. (2002) Automated test case generation for spreadsheets. Proc. 22rd Int. Conf. Software Engineering (ICSE'02), Orlando, FL, USA, May 9-25, pp. 141-151. ACM, New York, NY, USA.
52. Li, J. B. and Miller, J. (2005) Testing the semantics of W3C XML schema. Proc. 29th Annual Int. Computer Software and Applications Conf. (COMPSAC'05), Edinburgh, Scotland, UK, July 25-28, pp. 443-448. IEEE Computer Society, Los Alamitos, CA, USA.
53. Lee, S. C. and Offutt, J. (2001) Generating test cases for XML-based web component interactions using mutation analysis. Proc. 12th Int. Symp. Software Reliability Engineering (ISSRE'01), Hong Kong, China, November 27-30, pp. 200-209. IEEE Computer Society, Los Alamitos, CA, USA.
54. Boujarwah, A. S. and Saleh, K. (1997) Compiler test case generation methods: A survey and assessment. Inf. Softw. Technol., 39, 617-625.
55. DeMillo, R. A., Lipton, R. J. and Sayward, F. G. (1978) Hints on test data selection: Help for the practising programmer. Computer, 11, 34-41.
56. Hamlet, R. G. (1977) Testing programs with the aid of a compiler. IEEE Trans. Softw. Eng., SE-3, 279-290.
57. King, K. N. and Offutt, A. J. (1991) A FORTRAN language system for mutation-based software testing. Softw.-Pract. Exp., 21, 685-718.
58. Budd, T. A. (1981) Mutation analysis: Ideas, examples, problems and prospects. In Chandrasekaran, B. and Radicchi, S. (eds), Computer Program Testing. North-Holland, Amsterdam.
59. Howden, W. E. (1982) Weak mutation testing and completeness of test sets. IEEE Trans. Softw. Eng., SE-8, 371-379.
60. Woodward, M. R. and Halewood, K. (1988) From weak to strong - dead or alive? An analysis of some mutation testing issues. Proc. 2nd Workshop Software Testing, Verification and Analysis, Banff, Canada, July 19-21, pp. 152-158. IEEE Computer Society, Washington, DC, USA.
61. Mathur, A. P. (1991) Performance, effectiveness, and reliability issues in software testing. Proc. 15th Annual Int. Computer Software and Applications Conf., Tokyo, Japan, September 11-13, pp. 604-605. IEEE Computer Society, Los Alamitos, CA, USA.
62. Gopal, A. and Budd, T. (1983) Program testing by specification mutation. TR 83-17. University of Arizona, Tucson, Arizona, USA.
63. Woodward, M. R. (1993) Errors in algebraic specifications and an experimental mutation testing tool. Softw. Eng. J., 8, 211-224.
64. Murnane, T. and Reed, K. (2001) On the effectiveness of mutation analysis as a black box testing technique. Proc. 13th Australian Software Engineering Conf. (ASWEC'01), Canberra, Australia, August 27-28, p. 12. IEEE Computer Society, Los Alamitos, CA, USA.
65. Offutt, J. and Xu, W. (2004) Generating test cases for Web Services using data perturbation. Proc. Workshop on Testing, Analysis and Verification of Web Services (TAV-WEB), Boston, MA, USA, July, pp. 1-10. ACM, New York, NY, USA.
66. Xu, W., Offutt, J. and Luo, J. (2005) Testing Web Services by XML perturbation. Proc. 16th IEEE Int. Symp. Software Reliability Engineering. (ISSRE'05), Chicago, IL, USA, November 8-11, pp. 257-266. IEEE Computer Society, Los Alamitos, CA, USA.
67. Meek, B. and Siu, K. K. (1989) The effective of error seeding. SIGPLAN Not., 24, 81-89.
68. Chen, T. Y., Leung, H. and Mak, I. K. (2004) Adaptive random testing. Proc. 9th Asian Computing Science Conf. (ASIAN'04), Lecture Notes Computer Science, Vol. 3321 Chaing Mai, Thailand, December 8-10, pp. 320-329. Springer-Verlag, Berlin, Heidelberg.
69. Chen, T. Y., Kuo, F. C., Merkel, R. G. and Ng, S. P. (2003) Mirror adaptive random testing. Proc. 3rd Int. Conf. Quality Software (QSIC'03), Dallas, TX, USA, November 6-7, pp. 4-11. IEEE Computer Society, Los Alamitos, CA, USA.
70. Chan, K. P., Chen, T. Y. and Towey, D. (2006) Restricted random testing: adaptive random testing by exclusion. Int. J. Softw. Eng. Knowl. Eng. 16, 553-584.
71. Chan, K. P., Chen, T. Y. and Towey, D. (2006) Probabilistic adaptive random testing. Proc. 6th Int. Conf. Quality Software (QSIC'06), Beijing, China, October 26-28, pp. 274-280. IEEE Computer Society, Los Alamitos, CA, USA.
72. Chan, F. T., Chen, T. Y., Mak, I. K. and Yu, Y. T. (1996) Proportional sampling strategy: Guidelines for software testing practitioners. Inf. Softw. Technol., 38, 775-782.
73. Chen, T. Y., Kuo, F.-C., Tse, T. H. and Zhou, Z. Q. (2003) Metamorphic testing and beyond. Proc. 11th Annual International Workshop on Software Technology and Engineering Practice (STEP'03), Amsterdam, The Netherlands, September 19-21, pp. 94-100. IEEE Computer Society, Los Alamitos, CA, USA.
74. Zhu,H. and Shan, L. (2005) Caste-centric modelling of multi-agent systems: The CAMLE modelling language and automated tools. In Beydeda, S. and Gruhn, V. (eds), Model-driven Software Development, Research and Practice in Software Engineering, Vol. II. Springer-Verlag, Berlin, Heidelberg.
75. Fowler, M. and Scott, K. (2003) UML Distilled: A Brief Guide to the Standard Object Modeling Language (3rd edn). Addison Wesley, Boston, MA.
76. Yourdon, E. (1988) Modern Structured Analysis. Prentice Hall, Englewood Cliffs, NJ.
77. Shan, L. and Zhu, H. (2004) Consistency check in modeling multi-agent systems. Proc. 26th Int. Computer Software and Applications Conf. (COMPSAC'04), Hong Kong, China, September 28-30, pp. 114-121. IEEE Computer Society, Los Alamitos, CA, USA.
78. Moukas, A. (1997) Amalthaea: Information discovery and filtering using a multi-agent evolving ecosystem. J. Appl. AI, 11, 437-457.
79. Zhu, H. (2002) Formal specification of evolutionary software agents. Proceedings of 4th Int. Conf. Formal Engineering Methods (ICFEM'02 , Shanghai, China, October 21-25, pp. 249-261. Springer-Verlag, Berlin, Heidelberg.
80. Shan, L. and Zhu, H. (2003) Modelling and specification of scenarios and agent behaviour. Proc. IEEE/WIC Conf. Intelligent Agent Technology (IAT'03), Halifax, Canada, October 13-16, pp. 32-38. IEEE Computer Society, Los Alamitos, CA, USA.
81. Zhu, H. and Shan, L. (2005) Agent-oriented modelling and specification of web services. Proc. 10th IEEE Int. Workshop on Object-oriented Real-time Dependable Systems (WORDS 2005), Sedona, AZ, USA, February 2-4, pp. 152-159. IEEE Computer Society, Los Alamitos, CA, USA.
82. Harrold, M. J., Offutt, J. A. and Tewary, K. (1997) An approach to fault modelling and fault seeding using the program dependence graph. J. Syst. Softw., 36, 273-296.
83. Zhou, Z. Q., Huang, D. H., Tse, T. H., Yang, Z., Huang, H. and Chen, T. Y. (2004) Metamorphic testing and its applications. Proceedings of 8th International Symposium on Future Software Technology (ISFST 2004 Xian, China, October 20-22, pp. 346-351. Software Engineers Association, Tokyo, Japan.