A manifesto for higher order mutation testing

ICSTW 2010 - 3rd International Conference on Software Testing, Verification, and Validation Workshops

Volumn , Issue , 2010, Pages 80-89

  Harman, Mark ^a   Jia, Yue ^a   Langdon, William B ^a  

^a KING'S COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

FIRST ORDER; HIGHER ORDER; MUTATION TESTING; OPTIMIZATION TECHNIQUES; RESEARCH AGENDA; SEARCH-BASED SOFTWARE ENGINEERING;

COMPUTER SOFTWARE SELECTION AND EVALUATION; VERIFICATION;

SOFTWARE TESTING;

EID: 77953759620     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICSTW.2010.13     Document Type: Conference Paper

References (62)

1. A. T. Acree. On Mutation. Phd thesis, Georgia Institute of Technology, Atlanta, Georgia, 1980.
2. K. Adamopoulos, M. Harman, and R. M. Hierons. How to Overcome the Equivalent Mutant Problem and Achieve Tailored Selective Mutation Using Co-evolution. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO'04), volume 3103 of LNCS, pages 1338-1349, Seattle, Washington, USA, 26th-30th, June 2004. Springer.
3. W. Afzal, R. Torkar, and R. Feldt. A systematic review of search-based testing for non-functional system properties. Information and Software Technology, 51(6):957-976, 2009.
4. H. Agrawal, R. A. DeMillo, B. Hathaway, W. Hsu, W. Hsu, E. W. Krauser, R. J. Martin, A. P. Mathur, and E. Spafford. Design of Mutant Operators for the C Programming Language. Technique Report SERCTR- 41-P, Purdue University, West Lafayette, Indiana, March 1989.
5. J. H. Andrews, L. C. Briand, and Y. Labiche. Is Mutation an Appropriate Tool for Testing Experiments? In Proceedings of the 27th International Conference on Software Engineering (ICSE'05), pages 402 - 411, St Louis, Missouri, 15-21 May 2005.
6. D. Baldwin and F. G. Sayward. Heuristics for Determining Equivalence of Program Mutations. Research Report 276, Yale University, New Haven, Connecticut, 1979.
7. E. F. Barbosa, J. C. Maldonado, and A. M. R. Vincenzi. Toward the determination of sufficient mutant operators for C. Software Testing, Verification and Reliability, 11(2):113-136, May 2001.
8. B. Baudry, F. Fleurey, J.-M. Jezequel, and Y. Le Traon. Genes and Bacteria for Automatic Test Cases Optimization in the .NET Environment. In Proceedings of the 13th International Symposium on Software Reliability Engineering (ISSRE'02), pages 195-206, Annapolis, Maryland, 12-15 November 2002.
9. T. A. Budd. Mutation Analysis of Program Test Data. Phd thesis, Yale University, New Haven, Connecticut, 1980.
10. B. Cheng and J. Atlee. From state of the art to the future of requirements engineering. In L. Briand and A. Wolf, editors, Future of Software Engineering 2007, Los Alamitos, California, USA, 2007. IEEE Computer Society Press. This volume.
11. S. Y. Chong, P. Tino, and X. Yao. Measuring generalization performance in co-evolutionary learning. IEEE Transactions on Evolutionary Computation, 12(4):479-505, August 2008.
12. M. Daran and P. Thévenod-Fosse. Software Error Analysis: A Real Case Study Involving Real Faults and Mutations. ACM SIGSOFT Software Engineering Notes, 21(3):158-177, May 1996.
13. K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6:182-197, Apr. 2002.
14. R. A. DeMillo, D. S. Guindi, K. N. King, W. M. McCracken, and A. J. Offutt. An Extended Overview of the Mothra Software Testing Environment. In Proceedings of the 2nd Workshop on Software Testing, Verification, and Analysis (TVA'88), pages 142-151, Banff Alberta, Canada, July 1988. IEEE Computer society.
15. R. A. DeMillo, R. J. Lipton, and F. G. Sayward. Hints on Test Data Selection: Help for the Practicing Programmer. Computer, 11(4):34-41, April 1978.
16. R. A. DeMillo and A. P. Mathur. On the Use of Software Artifacts to Evaluate the Effectiveness of Mutation Analysis in Detecting Errors in Production Software. Technique Report SERC-TR-92-P, Purdue University, West Lafayette, Indiana, 1992.
17. R. A. DeMillo and A. J. Offutt. Constraint-Based Automatic Test Data Generation. IEEE Transactions on Software Engineering, 17(9):900- 910, September 1991.
18. R. A. DeMillo and A. J. Offutt. Experimental Results From an Automatic Test Case Generator. ACM Transactions on Software Engineering and Methodology, 2(2):109-127, April 1993.
19. H. Do and G. Rothermel. A Controlled Experiment Assessing Test Case Prioritization Techniques via Mutation Faults. In Proceedings of the 21st IEEE International Conference on Software Maintenance (ICSM'05), pages 411-420, Budapest, Hungary, 25-30 September 2005.
20. S. Eldh, S. Punnekkat, H. Hansson, and P. J̈onsson. Component Testing Is Not Enough - A Study of Software Faults in Telecom Middleware. In Proceedings of the 19th IFIP International Conference on Testing of Communicating Systems and 7th International Workshop on Formal Approaches to Testing of Software (TestCom'07) and the 7th International Workshop (FATES'07), Tallinn, Estonia, 26-29 June 2007.
21. B. J. M. Grün, D. Schuler, and A. Zeller. The Impact of Equivalent Mutants. In Proceedings of the 4th International Workshop on Mutation Analysis (MUTATION'09), pages 192-199, Denver, Colorado, 1-4 April IEEE Computer Society. published with Proceedings of the 2nd International Conference on Software Testing, Verification, and Validation Workshops.
22. R. G. Hamlet. Testing Programs with the Aid of a Compiler. IEEE Transactions on Software Engineering, 3(4):279-290, July 1977.
23. M. Harman. The current state and future of search based software engineering. In L. Briand and A. Wolf, editors, Future of Software Engineering 2007, pages 342-357, Los Alamitos, California, USA, 2007. IEEE Computer Society Press. (Pubitemid 47484936)
24. M. Harman, R. Hierons, and S. Danicic. The Relationship Between Program Dependence and Mutation Analysis. In Proceedings of the 1st Workshop on Mutation Analysis (MUTATION'00), pages 5-13, San Jose, California, 6-7 October 2001. published in book form, as Mutation Testing for the New Century.
25. M. Harman and R. M. Hierons. An overview of program slicing. Software Focus, 2(3):85-92, 2001.
26. M. Harman, A. Mansouri, and Y. Zhang. Search based software engineering: A comprehensive analysis and review of trends techniques and applications. Technical Report TR-09-03, Department of Computer Science, King's College London, April 2009.
27. R. M. Hierons, M. Harman, and S. Danicic. Using Program Slicing to Assist in the Detection of Equivalent Mutants. Software Testing, Verification and Reliability, 9(4):233-262, December 1999.
28. Y. Jia and M. Harman. Constructing Subtle Faults Using Higher Order Mutation Testing. In Proceedings of the 8th International Working Conference on Source Code Analysis and Manipulation (SCAM'08), pages 249-258, Beijing, China, 28-29 September 2008.
29. Y. Jia and M. Harman. MILU: A Customizable, Runtime-Optimized Higher Order Mutation Testing Tool for the Full C Language. In Proceedings of the 3rd Testing: Academic and Industrial Conference Practice and Research Techniques (TAIC PART'08), pages 94-98, Windsor, UK, 29-31 August 2008. IEEE Computer Society.
30. Y. Jia and M. Harman. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions of Software Engineering, To appear, 2009.
31. Y. Jia and M. Harman. Higher Order Mutation Testing. Journal of Information and Software Technology, 51(10):1379-1393, October 2009.
32. K. N. King and A. J. Offutt. A Fortran Language System for Mutation- Based Software Testing. Software:Practice and Experience, 21(7):685- 718, October 1991.
33. W. B. Langdon. Genetic Programming and Data Structures: Genetic Programming + Data Structures = Automatic Programming!, volume 1 of Genetic Programming. Kluwer, Boston, 24 Apr. 1998.
34. W. B. Langdon and M. Harman. Evolving gzip matches kernel from an nvidia CUDA template. Technical Report TR-10-02, Department of Computer Science, King's College London, London, WC2R 2LS, UK, 5 Feb. 2010.
35. W. B. Langdon, M. Harman, and Y. Jia. Efficient multi objective MC, GA, GP search and higher order mutation testing. Submitted. Extended version of TAICPART conference paper.
36. W. B. Langdon, M. Harman, and Y. Jia. Multi objective mutation testing with genetic programming. In 4th Testing Academia and Industry Conference - Practice And Research Techniques (TAIC PART'09), pages 21-29, Windsor, UK, 4th-6th September 2009.
37. W. B. Langdon and A. P. Harrison. Evolving DNA motifs to predict GeneChip probe performance. Algorithms in Molecular Biology, 4(6), 19 Mar. 2009.
38. A. P. Mathur. Performance, Effectiveness, and Reliability Issues in Software Testing. In Proceedings of the 5th International Computer Software and Applications Conference (COMPSAC'79), pages 604-605, Tokyo, Japan, 11-13 September 1991.
39. A. P. Mathur and W. E. Wong. An Empirical Comparison of Mutation and Data Flow Based Test Adequacy Criteria. Technique report, Purdue University, West Lafayette, Indiana, 1993.
40. E. S. Mresa and L. Bottaci. Efficiency of Mutation Operators and Selective Mutation Strategies: An Empirical Study. Software Testing, Verification and Reliability, 9(4):205-232, December 1999.
41. A. S. Namin and J. H. Andrews. Finding Sufficient Mutation Operators via Variable Reduction. In Proceedings of the 2nd Workshop on Mutation Analysis (MUTATION'06), page 5, Raleigh, North Carolina, November 2006. IEEE Computer Society.
42. A. S. Namin and J. H. Andrews. On Sufficiency of Mutants. In Proceedings of the 29th International Conference on Software Engineering (ICSE COMPANION'07), pages 73-74, Minneapolis, Minnesota, 20-26 May 2007.
43. A. J. Offutt. The Coupling Effect: Fact or Fiction. ACM SIGSOFT Software Engineering Notes, 14(8):131-140, December 1989.
44. A. J. Offutt. Investigations of the Software Testing Coupling Effect. ACM Transactions on Software Engineering and Methodology, 1(1):5- 20, January 1992.
45. A. J. Offutt and W. M. Craft. Using Compiler Optimization Techniques to Detect Equivalent Mutants. Software Testing, Verification and Reliability, 4(3):131-154, September 1994.
46. A. J. Offutt, Z. Jin, and J. Pan. The Dynamic Domain Reduction Approach for Test Data Generation: Design and Algorithms. Technical Report ISSE-TR-94-110, George Mason University, Fairfax, Virginia, 1994.
47. A. J. Offutt, Z. Jin, and J. Pan. The Dynamic Domain Reduction Procedure for Test Data Generation. Software:Practice and Experience, 29(2):167-193, February 1999.
48. A. J. Offutt, A. Lee, G. Rothermel, R. H. Untch, and C. Zapf. An Experimental Determination of Sufficient Mutant Operators. ACM Transactions on Software Engineering and Methodology, 5(2):99-118, April 1996.
49. A. J. Offutt and J. Pan. Detecting Equivalent Mutants and the Feasible Path Problem. In Proceedings of the 1996 Annual Conference on Computer Assurance, pages 224-236, Gaithersburg, Maryland, June 1996. IEEE Computer Society Press.
50. A. J. Offutt, G. Rothermel, and C. Zapf. An Experimental Evaluation of Selective Mutation. In Proceedings of the 15th International Conference on Software Engineering (ICSE'93), pages 100-107, Baltimore, Maryland, May 1993. IEEE Computer Society Press.
51. R. Poli, W. B. Langdon, and N. F. McPhee. A field guide to genetic programming. Published via http://lulu.com and freely available at http://www.gp-field-guide.org.uk, 2008. (With contributions by J. R. Koza).
52. R. Pressman. Software Engineering: A Practitioner's Approach. McGraw-Hill Book Company Europe, Maidenhead, Berkshire, England, UK., 3rd edition, 1992. European adaptation (1994). Adapted by Darrel Ince. ISBN 0-07-707936-1.
53. R. Purushothaman and D. E. Perry. Toward Understanding the Rhetoric of Small Source Code Changes. IEEE Transactions on Software Engineering, 31(6):511-526, 2005.
54. O. Räihä. A survey on search based software design. Technical Report Technical Report D-2009-1, Department of Computer Sciences, University of Tampere, 2009.
55. M. Sahinoglu and E. H. Spafford. A Bayes Sequential Statistical Procedure for Approving Software Products. In Proceedings of the IFIP Conference on Approving Software Products (ASP'90), pages 43-56, Garmisch Partenkirchen, Germany, September 1990. Elsevier Science.
56. B. H. Smith and L. Williams. On Guiding the Augmentation of an Automated Test Suite via Mutation Analysis. Empirical Software Engineering, 14(3):341-369, 2009.
57. I. Sommerville. Software Engineering. Addison-Wesley, 6th edition, 2001.
58. F. Tip. A survey of program slicing techniques. Journal of Programming Languages, 3(3):121-189, Sept. 1995.
59. J. M. Voas. PIE: A Dynamic Failure-Based Technique. IEEE Transactions of Software Engineering, 18(8):717-726, August 1992.
60. Wikipedia. Père David's Deer-Wikipedia. http://en.wikipedia.org, 2008.
61. W. E.Wong. On Mutation and Data Flow. Phd thesis, Purdue University, West Lafayette, Indiana, 1993.
62. W. E. Wong and A. P. Mathur. Reducing the Cost of Mutation Testing: An Empirical Study. Journal of Systems and Software, 31(3):185-196, December 1995.