Predicting metamorphic relations for testing scientific software: A machine learning approach using graph kernels

Software Testing Verification and Reliability

Volumn 26, Issue 3, 2016, Pages 245-269

  Kanewala, Upulee ^a   Bieman, James M ^b   Ben Hur, Asa ^b  

^a Idaho National Laboratory   (United States)

^b Department of Computer Science   (United States)

Author keywords

graph kernels; metamorphic relations; metamorphic testing; support vector machines

Indexed keywords

ABILITY TESTING; ARTIFICIAL INTELLIGENCE; FLOW GRAPHS; FORECASTING; GRAPHIC METHODS; LEARNING SYSTEMS; SOFTWARE TESTING; SUPPORT VECTOR MACHINES;

AUTOMATED SOFTWARE TESTING; GRAPH KERNELS; GRAPH-BASED REPRESENTATIONS; MACHINE LEARNING APPROACHES; MEASURING SIMILARITIES; METAMORPHIC RELATIONS; METAMORPHIC TESTING; SCIENTIFIC SOFTWARES;

DATA FLOW GRAPHS;

EID: 84963615021     PISSN: 09600833     EISSN: 10991689     Source Type: Journal    
DOI: 10.1002/stvr.1594     Document Type: Article

References (55)

1. Sanders R, Kelly D,. The challenge of testing scientific software. In Proceedings of the Conference for the Association for Software Testing (CAST): Toronto 2008; 30-36.
2. Sanders R, Kelly D,. Dealing with risk in scientific software development. IEEE Software 2008; 25 (4): 21-28.
3. Hatton L,. The T experiments: errors in scientific software. IEEE Computational Science Engineering 1997; 4 (2): 27-38.
4. Abackerli AJ, Pereira PH, Calônego Jr. N,. A case study on testing CMM uncertainty simulation software (VCMM). Journal of the Brazilian Society of Mechanical Sciences and Engineering 2010; 32: 8-14.
5. Miller G,. A scientist's nightmare: software problem leads to five retractions. Science 2006; 314 (5807): 1856-1857.
6. Hatton L, Roberts A,. How accurate is scientific software? IEEE Transactions on Software Engineering 1994; 20 (10): 785-797.
7. Dubois PF,. Testing scientific programs. Computing in Science Engineering 2012; 14 (4): 69-73.
8. Weyuker EJ,. On testing non-testable programs. The Computer Journal 1982; 25 (4): 465-470.
9. Joppa LN, McInerny G, Harper R, Salido L, Takeda K, O'Hara K, Gavaghan D, Emmott S,. Troubling trends in scientific software use. Science 2013; 340 (6134): 814-815.
10. Kanewala U, Bieman JM,. Testing scientific software: a systematic literature review. Information and Software Technology 2014; 56 (10): 1219-1232.
11. Chen TY, Cheung SC, Yiu SM,. Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong, 1998.
12. Chen TY, Tse TH, Zhou ZQ,. Fault-based testing without the need of oracles. Information and Software Technology 2003; 45 (1): 1-9.
13. Zhou ZQ, Huang DH, Tse TH, Yang Z, Huang H, Chen TY,. Metamorphic testing and its applications. In Proceedings of the 8th International Symposium on Future Software Technology (ISFST 2004). Software Engineers Association: Xian, China 2004; 346-351.
14. Xie X, Ho JWK, Murphy C, Kaiser G, Xu B, Chen TY,. Testing and validating machine learning classifiers by metamorphic testing. Journal of Systems and Software 2011; 84 (4): 544-558.
15. Chen TY, Huang DH, Tse TH, Zhou ZQ,. Case studies on the selection of useful relations in metamorphic testing. In Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC 2004), Madrid, Spain 2004; 569-583.
16. Kanewala U, Bieman JM,. Using machine learning techniques to detect metamorphic relations for programs without test oracles. In Proceedings of the 24th IEEE International Symposium on Software Reliability Engineering (ISSRE), Pasadena, CA, USA 2013; 1-10.
17. Liu H, Liu X, Chen TY,. A new method for constructing metamorphic relations. In Proceedings of the 12th International Conference on Quality Software (QSIC), Xian, Shaanxi, China 2012; 59-68.
18. Zhang J, Chen J, Hao D, Xiong Y, Xie B, Zhang L, Mei H,. Search-based inference of polynomial metamorphic relations. In Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineering, ASE '14 ACM: New York, NY, USA, 2014.
19. Murphy C, Raunak MS, King A, Chen S, Imbriano C, Kaiser G, Lee I, Sokolsky O, Clarke L, Osterweil L,. On effective testing of health care simulation software. In proceedings 3rd Workshop on Software Engineering in Health Care, SEHC '11. ACM: New York, NY, USA 2011; 40-47.
20. Ding J, Wu T, Wu D, Lu JQ, Hu X-H,. Metamorphic testing of a Monte Carlo modeling program. In Proceedings of the 6th International Workshop on Automation of Software Test, AST '11. ACM: New York, NY, USA 2011; 1-7.
21. Chen TY, Ho JWK, Liu H, Xie X,. An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics 2009; 10: 24-36.
22. Guderlei R, Mayer J,. Statistical metamorphic testing-testing programs with random output by means of statistical hypothesis tests and metamorphic testing. In Proceedings 7th of the International Conference on Quality Software (QSIC), Portland, Oregon, USA 2007; 404-409.
23. Chen TY, Feng J, Tse TH,. Metamorphic testing of programs on partial differential equations: a case study. In Proceedings of the 26th International Computer Software and Applications Conference on Prolonging Software Life: Development and Redevelopment, COMPSAC '02. IEEE Computer Society: Washington, DC, USA 2002; 327-333.
24. Segura S, Durán A, Sánchez AB, Berre DL, Lonca E, Ruiz-Cortés A,. Automated metamorphic testing of variability analysis tools. Software Testing, Verification and Reliability 2015; 25 (2): 138-163.
25. Le V, Afshari M, Su Z,. Compiler validation via equivalence modulo inputs. In Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI '14. ACM: New York, NY, USA 2014; 216-226.
26. Núñez A, Hierons RM,. A methodology for validating cloud models using metamorphic testing. Annals of Telecommunications 2015; 70 (3-4): 127-135.
27. Lindvall M, Ganesan D, Ardal R, Wiegand RE,. Metamorphic model-based testing applied on NASA DAT-an experience report. In Proceedings of the 37th IEEE/ACM International Conference on Software Engineering (ICSE), Vol. 2, Florence, Italy 2015; 129-138.
28. Allen FE,. Control flow analysis. SIGPLAN Not. 1970; 5 (7): 1-19.
29. Vallee-Rai R, Hendren LJ,. Jimple: simplifying Java bytecode for analyses and transformations, 1998.
30. Kondor RI, Lafferty J,. Diffusion kernels on graphs and other discrete structures. In Proceedings 19th International Conference on Machine Learning, Sydney, Australia 2002; 315-322.
31. Gärtner T, Flach P, Wrobel S,. On graph kernels: hardness results and efficient alternatives. In Learning theory and kernel machines, Schölkopf B, Warmuth M, (eds)., Lecture Notes in Computer Science, Vol. 2777. Springer Berlin Heidelberg: Heidelberg, Germany 2003; 129-143.
32. Shervashidze N, Vishwanathan SVN, Petri T, Mehlhorn K, Borgwardt K,. Efficient graphlet kernels for large graph comparison. In Proceedings of the International Workshop on Artificial Intelligence and Statistics, Welling M, van Dyk D, (eds). Society for Artificial Intelligence and Statistics: Clearwater Beach, Florida, USA, 2009.
33. Murphy C, Kaiser GE, Hu L, Wu L,. Properties of machine learning applications for use in metamorphic testing. In Proceedings of the 20th International Conference on Software Engineering & Knowledge Engineering (SEKE'2008): San Francisco, CA, USA 2008; 867-872.
34. Hu P, Zhang Z, Chan WK, Tse TH,. An empirical comparison between direct and indirect test result checking approaches, Proceedings of the 3rd International Workshop on Software Quality Assurance 2006; 6-13.
35. Liu H, Kuo F-C, Towey D, Chen TY,. How effectively does metamorphic testing alleviate the oracle problem? IEEE Transactions on Software Engineering 2014; 40 (1): 4-22.
36. Mishra KS, Kaiser G,. Effectiveness of teaching metamorphic testing. In CUCS-020-12, Department of Computer Science, Columbia University, 2012.
37. Ben-Hur A, Weston J,. A user's guide to support vector machines. In Data mining techniques for the life sciences, Carugo O, Eisenhaber F, (eds)., Methods in Molecular Biology, vol. 609, Humana Press: Totowa, NJ 2010; 223-239.
38. Huang J, Ling CX,. Using AUC and accuracy in evaluating learning algorithms. IEEE Transactions on Knowledge and Data Engineering 2005; 17 (3): 299-310.
39. Murphy C,. Metamorphic testing techniques to detect defects in applications without test oracles. PhD dissertation, 2010.
40. DeMillo RA, Lipton RJ, Sayward FG,. Hints on test data selection: help for the practicing programmer. Computer 1978; 11 (4): 34-41.
41. Murphy C, Shen K, Kaiser G,. Using JML runtime assertion checking to automate metamorphic testing in applications without test oracles. In Proceedings of the 2009 International Conference on Software Testing Verification and Validation, ICST '09 IEEE Computer Society: Washington, DC, USA 2009; 436-445.
42. Murphy C, Shen K, Kaiser G,. Automatic system testing of programs without test oracles. In Proceedings of the Eighteenth International Symposium on Software Testing and Analysis, ISSTA '09. ACM: New York, NY, USA 2009; 189-200.
43. Briand LC,. Novel applications of machine learning in software testing. In Proceedings of the 8th International Conference on Quality Software. IEEE Computer Society: Washington, DC, USA 2008; 3-10.
44. Bowring JF, Rehg JM, Harrold MJ,. Active learning for automatic classification of software behavior. In Proceedings of the 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA '04 ACM: New York, NY, USA 2004; 195-205.
45. Briand LC, Labiche Y, Bawar Z,. Using machine learning to refine black-box test specifications and test suites. In Proceedings 8th International Conference on Quality Software, QSIC '08. IEEE Computer Society: Washington, DC, USA 2008; 135-144.
46. Briand LC, Labiche Y, Liu X,. Using machine learning to support debugging with tarantula. In Proceedings 18th IEEE International Symposium on Software Reliability (ISSRE), Trollhattan, Sweden 2007; 137-146.
47. Frounchi K, Briand LC, Grady L, Labiche Y, Subramanyan R,. Automating image segmentation verification and validation by learning test oracles. Information and Software Technology 2011; 53 (12): 1337-1348.
48. Lo J-H,. Predicting software reliability with support vector machines. In Proceedings of the 2nd International Conference on Computer Research and Development 2010; 765-769.
49. Wang F, Yao L-W, Wu J-H,. Intelligent test oracle construction for reactive systems without explicit specifications. In Proceedings of the 9th International Conference on Dependable, Autonomic and Secure Computing (DASC), Sydney, Australia 2011; 89-96.
50. Evgeniou T, Pontil M,. Regularized multi-task learning. In Proceedings of the tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM: Seattle, Washington, USA 2004; 109-117.
51. Madjarov G, Kocev D, Gjorgjevikj D, Džeroski S,. An extensive experimental comparison of methods for multi-label learning. Pattern Recognition 2012; 45 (9): 3084-3104.
52. Shawe-Taylor J, Cristianini N,. Kernel Methods for Pattern Analysis. Cambridge University Press: New York, NY, USA, 2004.
53. Borgwardt KM, Kriegel H-P,. Shortest-path kernels on graphs. In Proceedings of the 5th IEEE International Conference on Data Mining, Houston, Texas, USA 2005; 74-81.
54. Ramon J, Gärtner T,. Expressivity versus efficiency of graph kernels. In Proceedings of the 1st International Workshop on Mining Graphs, Trees and Sequences, Cavtat-Dubrovnik, Croatia 2003; 65-74.
55. Borgwardt KM, Ong CS, Schönauer S, Vishwanathan SVN, Smola AJ, Kriegel HP,. Protein function prediction via graph kernels. Bioinformatics 2005; 21 (suppl 1): i47-i56.