Testing scientific software: A systematic literature review

Information and Software Technology

Volumn 56, Issue 10, 2014, Pages 1219-1232

  Kanewala, Upulee ^a   Bieman, James M ^a  

^a Veterinary Dentistry and Oral Surgery   (United States)

Author keywords

Scientific software; Software quality; Software testing; Systematic literature review

Indexed keywords

CODES (SYMBOLS); COMPUTER SOFTWARE SELECTION AND EVALUATION; SOFTWARE ENGINEERING; TESTING;

CODE CLONE DETECTION; CULTURAL DIFFERENCE; ENGINEERING COMMUNITY; SCIENTIFIC SOFTWARES; SOFTWARE QUALITY; SYSTEMATIC LITERATURE REVIEW; SYSTEMATIC TESTING; WEATHER PREDICTION;

SOFTWARE TESTING;

EID: 84905104495     PISSN: 09505849     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.infsof.2014.05.006     Document Type: Review

References (81)

1. A.J. Abackerli, P.H. Pereira, and N. Calônego Jr. A case study on testing CMM uncertainty simulation software (VCMM) J. Brazilian Soc. Mech. Sci. Eng. 32 2010 8 14
2. K. Ackroyd, S. Kinder, G. Mant, M. Miller, C. Ramsdale, and P. Stephenson Scientific software development at a research facility IEEE Softw. 25 4 2008 44 51
3. W. Afzal, R. Torkar, and R. Feldt A systematic review of search-based testing for non-functional system properties Inform. Softw. Technol. 51 6 2009 957 976
4. P. Ammann, and J. Offutt Introduction to Software Testing first ed. 2008 Cambridge University Press New York, NY, USA
5. R. Bagnara, M. Carlier, R. Gori, A. Gotlieb, Symbolic path-oriented test data generation for floating-point programs, in: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation (ICST), 2013, pp. 1-10.
6. S. Brilliant, J. Knight, and N. Leveson Analysis of faults in an n-version software experiment IEEE Trans. Softw. Eng. 16 2 1990 238 247
7. J. Carver, R.P. Kendall, S.E. Squires, and D.E. Post Software development environments for scientific and engineering software: a series of case studies Proceedings of the 29th International Conference on Software Engineering (ICSE '07) 2007 IEEE Computer Society Washington, DC, USA 550 559
8. Jeffrey Carver, R. Bartlett, D. Heaton, L. Hochstein, What scientists and engineers think they know about software engineering: a survey, Tech. Rep. SAND2011-2196, Sandia National Laboratories, 2011.
9. T. Chen, J. Feng, T.H. Tse, Metamorphic testing of programs on partial differential equations: a case study, in: Proceedings of the 26th Annual International Computer Software and Applications Conference, 2002 (COMPSAC 2002), 2002, pp. 327-333.
10. T.Y. Chen, S.C. Cheung, S.M. Yiu, Metamorphic testing: a new approach for generating next test cases, Tech. Rep. HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong, 1998.
11. T.Y. Chen, J.W.K. Ho, H. Liu, and X. Xie An innovative approach for testing bioinformatics programs using metamorphic testing BMC Bioinform. 10 2009
12. T. Clune, and R. Rood Software testing and verification in climate model development IEEE Softw. 28 6 2011 49 55
13. M. Cox, and P. Harris Design and use of reference data sets for testing scientific software Anal. Chim. Acta 380 23 1999 339 351
14. T. Dahlgren Performance-driven interface contract enforcement for scientific components H. Schmidt, I. Crnkovic, G. Heineman, J. Stafford, Component-Based Software Engineering Lecture Notes in Computer Science vol. 4608 2007 Springer Berlin, Heidelberg 157 172
15. T.L. Dahlgren, and P.T. Devanbu Improving scientific software component quality through assertions Proceedings of the Second International Workshop on Software Engineering for High Performance Computing System Applications (SE-HPCS '05) 2005 ACM New York, NY, USA 73 77
16. M.D. Davis, and E.J. Weyuker Pseudo-oracles for non-testable programs Proceedings of the ACM '81 conference (ACM '81) 1981 ACM New York, NY, USA 254 257
17. J.B. Drake, P.W. Jones, and G.R. Carr Jr. Overview of the software design of the community climate system model Int. J. High Perform. Comput. Appl. 19 3 2005 177 186
18. P. Dubois Testing scientific programs Comput. Sci. Eng. 14 4 2012 69 73
19. T. Dyba, T. Dingsoyr, G. Hanssen, Applying systematic reviews to diverse study types: an experience report, in: First International Symposium on Empirical Software Engineering and Measurement, 2007 (ESEM 2007), September 2007, pp. 225-234.
20. S.M. Easterbrook Climate change a grand software challenge Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research (FoSER '10) 2010 ACM New York, NY, USA 99 104
21. S.M. Easterbrook, and T.C. Johns Engineering the software for understanding climate change Comput. Sci. Eng. 11 6 2009 65 74
22. S.L. Eddins Automated software testing for matlab Comput. Sci. Eng. 11 6 2009 48 55
23. E. Engström, P. Runeson, and M. Skoglund A systematic review on regression test selection techniques Inform. Softw. Technol. 52 1 2010 14 30
24. P.E. Farrell, M.D. Piggott, G.J. Gorman, D.A. Ham, C.R. Wilson, and T.M. Bond Automated continuous verification for numerical simulation Geosci. Model Develop. 4 2 2011 435 449
25. R. Guderlei, J. Mayer, Statistical metamorphic testing programs with random output by means of statistical hypothesis tests and metamorphic testing, in: Seventh International Conference on Quality Software, 2007 (QSIC '07), October 2007, pp. 404-409.
26. J.E. Hannay, C. MacLeod, J. Singer, H.P. Langtangen, D. Pfahl, and G. Wilson How do scientists develop and use scientific software? Proceedings of the 2009 ICSE Workshop on Software Engineering for Computational Science and Engineering (SECSE '09) 2009 IEEE Computer Society Washington, DC, USA 1 8
27. L. Hatton The T experiments: errors in scientific software IEEE Comput. Sci. Eng. 4 2 1997 27 38
28. L. Hatton, and A. Roberts How accurate is scientific software? IEEE Trans. Softw. Eng. 20 10 1994 785 797
29. M. Heroux, J. Willenbring, Barely sufficient software engineering: 10 practices to improve your CSE software, in: ICSE Workshop on Software Engineering for Computational Science and Engineering, 2009 (SECSE '09), 2009, pp. 15-21.
30. M.A. Heroux, J.M. Willenbring, M.N. Phenow, Improving the development process for CSE software, in: 15th EUROMICRO International Conference on Parallel, Distributed and Network-Based Processing, 2007 (PDP '07), February 2007, pp. 11-17.
31. L. Hochstein, and V. Basili The asc-alliance projects: a case study of large-scale parallel scientific code development Computer 41 3 2008 50 58
32. D. Hook, D. Kelly, Testing for trustworthiness in scientific software, in: ICSE Workshop on Software Engineering for Computational Science and Engineering, 2009 (SECSE '09), May 2009, pp. 59-64.
33. D.W. Kane, M.M. Hohman, E.G. Cerami, M.W. McCormick, K.F. Kuhlmman, and J.A. Byrd Agile methods in biomedical software development: a multi-site experience report BMC Bioinform. 7 2006 273
34. U. Kanewala, J. Bieman, Using machine learning techniques to detect metamorphic relations for programs without test oracles, in: 2013 IEEE 24th International Symposium on Software Reliability Engineering (ISSRE), November 2013, pp. 1-10.
35. U. Kanewala, and J.M. Bieman Techniques for testing scientific programs without an oracle Proc. 5th International Workshop on Software Engineering for Computational Science and Engineering 2013 IEEE 48 57
36. D. Kelly, R. Gray, and Y. Shao Examining random and designed tests to detect code mistakes in scientific software J. Comput. Sci. 2 1 2011 47 56
37. D. Kelly, D. Hook, and R. Sanders Five recommended practices for computational scientists who write software Comput. Sci. Eng. 11 5 2009 48 53
38. D. Kelly, R. Sanders, Assessing the quality of scientific software, in: First International Workshop on Software Engineering for Computational Science and Engineering, 2008.
39. D. Kelly, S. Smith, and N. Meng Software engineering for scientists Comput. Sci. Eng. 13 5 2011 7 11
40. D. Kelly, S. Thorsteinson, and D. Hook Scientific software testing: analysis with four dimensions IEEE Softw. 28 3 2011 84 90
41. B. Kitchenham, Procedures for performing systematic reviews, Technical report, Keele University and NICTA, 2004.
42. B. Kitchenham, O.P. Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman Systematic literature reviews in software engineering a systematic literature review Inform. Softw. Technol. 51 1 2009 7 15
43. B. Kitchenham, S. Charters, Guidelines for performing systematic literature reviews in software engineering (version 2.3), Technical report, Keele University and University of Durham, 2007.
44. K. Kreyman, D.L. Parnas, S. Qiao, Inspection procedures for critical programs that model physical phenomena, CRL Report no. 368, McMaster University, 1999.
45. P.C. Lane, and F. Gobet A theory-driven testing methodology for developing scientific software J. Exp. Theor. Artif. Intell. 24 4 2012 421 456
46. G. Lemos, E. Martins, Specification-guided golden run for analysis of robustness testing results, in: 2012 IEEE Sixth International Conference on Software Security and Reliability (SERE), June 2012, pp. 157-166.
47. L.S. Chin, D. Worth, C. Greenough, A survey of software testing tools for computational science, Tech. Rep. RAL-TR-2007-010, Rutherford Appleton Laboratory, 2007.
48. J. Mayer, R. Guderlei, On random testing of image processing applications, in: Sixth International Conference on Quality Software, 2006 (QSIC 2006), October 2006, pp. 85-92.
49. J. Mayer, A.A. Informationsverarbeitung, U. Ulm, On testing image processing applications with statistical methods, in: Software Engineering (SE 2005), Lecture Notes in Informatics, 2005, pp. 69-78.
50. K. Meinke, and F. Niu A learning-based approach to unit testing of numerical software A. Petrenko, A. Simo, J. Maldonado, Testing Software and Systems Lecture Notes in Computer Science vol. 6435 2010 Springer Berlin, Heidelberg 221 235
51. G. Miller A scientist's nightmare: software problem leads to five retractions Science 314 5807 2006 1856 1857
52. C. Morris, Some lessons learned reviewing scientific code, in: Proc. First International Workshop on Software Engineering for Computational Science and Engineering, 2008.
53. C. Morris, J. Segal, Some challenges facing scientific software developers: the case of molecular biology, in: Fifth IEEE International Conference on e-Science, 2009 (e-Science '09), December 2009, pp. 216-222.
54. C. Murphy, G. Kaiser, M. Arias, An approach to software testing of machine learning applications, in: 19th International Conference on Software Engineering and Knowledge Engineering (SEKE), 2007, pp. 167-172.
55. C. Murphy, G. Kaiser, and M. Arias Parameterizing random test data according to equivalence classes Proceedings of the 2nd International Workshop on Random Testing: Co-located with the 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2007), RT '07 2007 ACM New York, NY, USA 38 41
56. C. Murphy, G. Kaiser, L. Hu, L. Wu, Properties of machine learning applications for use in metamorphic testing, in: Proc. of the 20th International Conference on Software Engineering and Knowledge Engineering (SEKE), July 2008, pp. 867-872.
57. C. Murphy, M.S. Raunak, A. King, S. Chen, C. Imbriano, G. Kaiser, I. Lee, O. Sokolsky, L. Clarke, and L. Osterweil On effective testing of health care simulation software Proceedings of the 3rd Workshop on Software Engineering in Health Care (SEHC '11) 2011 ACM New York, NY, USA 40 47
58. C. Murphy, K. Shen, and G. Kaiser Using JML runtime assertion checking to automate metamorphic testing in applications without test oracles Proceedings of the 2009 International Conference on Software Testing Verification and Validation (ICST '09) 2009 IEEE Computer Society Washington, DC, USA 436 445
59. L. Nguyen-Hoan, S. Flint, and R. Sankaranarayana A survey of scientific software development Proceedings of the 2010 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM '10) 2010 ACM New York, NY, USA 12:1 12:10
60. J. Pipitone, and S. Easterbrook Assessing climate model software quality: a defect density analysis of three models Geosci. Model Develop. 5 4 2012 1009 1022
61. J. Pitt-Francis, M.O. Bernabeu, J. Cooper, A. Garny, L. Momtahan, J. Osborne, P. Pathmanathan, B. Rodriguez, J.P. Whiteley, and D.J. Gavaghan Chaste: using agile programming techniques to develop computational biology software Philos. Trans. Roy. Soc. A: Math. Phys. Eng. Sci. 366 1878 2008 3111 3136
62. D.E. Post, and R.P. Kendall Software project management and quality engineering practices for complex, coupled multiphysics, massively parallel computational simulations: lessons learned from ASCI Int. J. High Perform. Comput. Appl. 18 4 2004 399 416
63. H. Remmel, B. Paech, P. Bastian, and C. Engwer System testing a scientific framework using a regression-test environment Comput. Sci. Eng. 14 2 2012 38 45
64. W. Reupke, E. Srinivasan, P. Rigterink, D. Card, The need for a rigorous development and testing methodology for medical software, in: Proceedings of the Symposium on the Engineering of Computer-Based Medical Systems, 1988, June 1988, pp. 15-20.
65. R. Sanders, D. Kelly, The challenge of testing scientific software, in: Proceedings Conference for the Association for Software Testing (CAST), Toronto, July 2008, pp. 30-36.
66. R. Sanders, and D. Kelly Dealing with risk in scientific software development IEEE Softw. 25 4 2008 21 28
67. J. Segal When software engineers met research scientists: a case study Emp. Softw. Eng. 10 2005 517 536
68. J. Segal, Some problems of professional end user developers, in: IEEE Symposium on Visual Languages and Human-Centric Computing, 2007 (VL/HCC 2007), 2007, pp. 111-118.
69. J. Segal, Models of scientific software development, in: 2008 Workshop Software Eng. in Computational Science and Eng. (SECSE 08), 2008.
70. J. Segal, Scientists and software engineers: a tale of two cultures, in: J. Buckley, J. Rooksby, R. Bednarik (Eds.), PPIG 2008: Proceedings of the 20th Annual Meeting of the Psychology of Programming Interest Group, Lancaster University, Lancaster, UK, Proceedings: 20th Annual Meeting of the Psychology of Programming Interest Group; Lancaster, United Kingdom; September 10-12, 2008, 2008.
71. J. Segal Software development cultures and cooperation problems: a field study of the early stages of development of software for a scientific community Comput. Support. Cooper. Work 18 5-6 2009 581 606
72. J. Segal, Some challenges facing software engineers developing software for scientists, in: ICSE Workshop on Software Engineering for Computational Science and Engineering, 2009 (SECSE '09) May 2009, pp. 9-14.
73. M. Sletholt, J. Hannay, D. Pfahl, and H. Langtangen What do we know about scientific software development's agile practices? Comput. Sci. Eng. 14 2 2012 24 37
74. B. Smith A test harness th for numerical applications and libraries P. Gaffney, J. Pool, Grid-Based Problem Solving Environments IFIP The International Federation for Information Processing vol. 239 2007 Springer US 227 241
75. M.C. Smith, R.L. Kelsey, J.M. Riese, G.A. Young, Creating a flexible environment for testing scientific software, in: D.A. Trevisani, A.F. Sisti (Eds.), Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, vol. 5423, August 2004, pp. 288-296.
76. S.A. Vilkomir, W.T. Swain, J.H. Poore, and K.T. Clarno Modeling input space for testing scientific computational software: a case study Proceedings of the 8th International Conference on Computational Science, Part III (ICCS '08) 2008 Springer-Verlag Berlin, Heidelberg 291 300
77. G.S. Walia, and J.C. Carver A systematic literature review to identify and classify software requirement errors Inform. Softw. Technol. 51 7 2009 1087 1109
78. E.J. Weyuker On testing non-testable programs Comput. J. 25 4 1982 465 470
79. W. Wood, and W. Kleb Exploring xp for scientific research IEEE Softw. 20 3 2003 30 36
80. X. Xie, J.W. Ho, C. Murphy, G. Kaiser, B. Xu, and T.Y. Chen Testing and validating machine learning classifiers by metamorphic testing J. Syst. Softw. 84 4 2011 544 558
81. S. Yoo, Metamorphic testing of stochastic optimisation, in: 2010 Third International Conference on Software Testing, Verification, and Validation Workshops (ICSTW), 2010, pp. 192-201.