A Survey on Metamorphic Testing

IEEE Transactions on Software Engineering

Volumn 42, Issue 9, 2016, Pages 805-824

  Segura, Sergio ^a   Fraser, Gordon ^b   Sanchez, Ana B ^a   Ruiz Cortes, Antonio ^a  

^a UNIVERSITY OF SEVILLE   (Spain)

^b UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Metamorphic testing; oracle problem; survey

Indexed keywords

COMPUTER GRAPHICS; CONCRETE TESTING; SURVEYING; SURVEYS; WEB SERVICES;

APPLICATION AREA; EMPIRICAL STUDIES; INPUT-OUTPUT RELATIONS; METAMORPHIC RELATIONS; METAMORPHIC TESTING; ORACLE PROBLEM; RESEARCH RESULTS; TEST EXECUTION;

SOFTWARE TESTING;

EID: 84988891109     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2016.2532875     Document Type: Article

References (148)

1. E. J. Weyuker, "On testing non-testable programs," Comput. J., vol. 25, no. 4, pp. 465-470, 1982.
2. S. Anand, E. K. Burke, T. Y. Chen, J. Clark, M. B. Cohen, W. Grieskamp, M. Harman, M. J. Harrold, and P. Mcminn. (Aug. 2013). An orchestrated survey of methodologies for automated software test case generation. J. Syst. Softw. [Online] 86(8): 1978-2001. Available: http://dx.doi.org/10.1016/j.jss.2013.02.061.
3. E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, "The oracle problem in software testing: A survey," IEEE Trans. Softw. Eng., vol. 41, no. 5, pp. 507-525, May 2015.
4. H. Liu, F.-C. Kuo, D. Towey, and T. Y. Chen, "How effectively does metamorphic testing alleviate the oracle problem?" IEEE Trans. Softw. Eng., vol. 40, no. 1, pp. 4-22, Jan. 2014.
5. T. Y. Chen, S. C. Cheung, and S. M. Yiu, "Metamorphic testing: A new approach for generating next test cases," Dept. Comput. Sci., The Hong Kong Univ. Sci. Technol., Hong Kong, Tech. Rep. HKUST-CS98-01, 1998.
6. W. J. Cody, Software Manual for the Elementary Functions. Upper Saddle River, NJ, USA: Prentice-Hall, 1980.
7. M. Blum, M. Luby, and R. Rubinfeld. (1993). Self-testing/correcting with applications to numerical problems. J. Comput. Syst. Sci. [Online]. 47(3), pp. 549-595. Available: http://www.sciencedirect. com/science/article/pii/002200009390044W.
8. P. E. Ammann and J. C. Knight. (Apr. 1988). Data diversity: An approach to software fault tolerance. IEEE Trans. Comput. [Online]. 37(4), pp. 418-425. Available: http://dx.doi.org/10.1109/12.2185.
9. T. Y. Chen, F.-C. Kuo, T. H. Tse, and Z. Q. Zhou, "Metamorphic testing and beyond," in Proc. 11th Annu. Int. Workshop Softw. Technol. Eng. Practice, Sep. 2003, pp. 94-100.
10. T. H. Tse, "Research directions on model-based metamorphic testing and verification," in Proc. 29th Annu. Int. Comput. Softw. Appl. Conf., vol. 1, Jul. 2005, p. 332.
11. T. Y. Chen, "Metamorphic testing: A simple approach to alleviate the oracle problem," in Proc. 5th IEEE Int. Symp. Service Oriented Syst. Eng., Jun. 2010, pp. 1-2.
12. W. K. Chan and T. H. Tse, "Oracles are hardly attain'd, and hardly understood: Confessions of software testing researchers," in Proc. 13th Int. Conf. Quality Softw., Jul. 2013, pp. 245-252.
13. T. Y. Chen, "Metamorphic testing: A simple method for alleviating the test oracle problem," in Proc. 10th Int. Workshop Autom. Softw. Test, 2015, pp. 53-54. [Online]. Available: http://dl.acm. org/citation.cfm?id=2819261.2819278.
14. T. Y. Chen, F.-C. Kuo, D. Towey, and Z. Q. Zhou, "Metamorphic testing: Applications and integration with other methods: Tutorial synopsis," in Proc. 12th Int. Conf. Quality Softw., Aug. 2012, pp. 285-288.
15. Z. Hui and S. Huang, "Achievements and challenges of metamorphic testing," in Proc. 4th World Congr. Softw. Eng., Dec. 2013, pp. 73-77.
16. U. Kanewala and J. M. Bieman, "Techniques for testing scientific programs without an oracle," in Proc. 5th Int. Workshop Softw. Eng. Comput. Sci. Eng., May 2013, pp. 48-57.
17. G. Dong, B. Xu, L. Chen, C. Nie, and L. Wang, "Survey of metamorphic testing," J. Frontiers Comput. Sci. Technol., vol. 3, no. 2, pp. 130-143, 2009.
18. T. Y. Chen, T. H. Tse, and Z. Q. Zhou. (2003). Fault-based testing without the need of oracles. Inform. Softw. Technol. [Online]. 45 (1), pp. 1-9. Available: http://dx.doi.org/10.1016/S0950-5849 (02)00129-5.
19. M. D. Ernst, J. H. Perkins, P. J. Guo, S. McCamant, C. Pacheco, M. S. Tschantz, and C. Xiao. (2007, Dec.). The Daikon system for dynamic detection of likely invariants. Sci. Comput. Program. [Online]. 69(1-3), pp. 35-45. Available: http://dx.doi.org/10.1016/j.scico.2007.01.015.
20. Z. Q. Zhou, S. Zhang, M. Hagenbuchner, T. H. Tse, F.-C. Kuo, and T. Y. Chen. (Jun. 2012). Automated functional testing of online search services. Softw. Testing, Verification Rel. J. [Online]. 22(4), pp. 221-243. Available: http://dx.doi.org/10.1002/stvr.437.
21. B. Kitchenham, "Procedures for performing systematic reviews," Keele Univ. and NICTA, Staffordshire, U.K., Tech. Rep. TR/SE-0401, 2004.
22. J. Webster and R. Watson. (2002). Analyzing the past to prepare for the future: Writing a literature review. MIS Quart. [Online]. 26(2), pp. xiii-xxiii. Available: http://www.misq.org/archivist/vol/no26/issue2/GuestEd.pdf.
23. D. Benavides, S. Segura, and A. Ruiz-Cort-es, "Automated analysis of feature models 20 years later: A literature review," Inform. Syst., vol. 35, no. 6, pp. 615-636, 2010.
24. Y. J. M.Harman and Y. Zhang, "Achievements, open problems and challenges for search based software testing," in Proc. 8th IEEE Int. Conf. Softw. Testing, Verification Validation, Apr. 2015, pp. 1-12.
25. Y. Jia and M. Harman. (2011, Sep.). An analysis and survey of the development of mutation testing. IEEE Trans. Softw. Eng., vol. 37, no. 5, pp. 649-678, Sep. 2011. [Online]. Available: http://dx.doi. org/10.1109/TSE.2010.62.
26. S. Segura, G. Fraser, A. B. S-anchez, and A. Ruiz-Cortes. (2016, Feb.). Metamorphic testing: A literature review. Appl. Softw. Eng. Res. Group, Univ. Seville, Seville, Spain. Tech. Rep. ISA-16-TR-02. [Online]. Available: http://www.isa.us.es/sites/default/files/isa-16-tr-02-0.pdf.
27. T. Y. Chen, D. H. Huang, T. H. Tse, and Z. Q. Zhou, "Case studies on the selection of useful relations in metamorphic testing," in Proc. 4th Ibero-Amer. Symp. Softw. Eng. Knowl. Eng., 2004, pp. 569-583.
28. T. Y. Chen, F.-C. Kuo, Y. Liu, and A. Tang, "Metamorphic testing and testing with special values," in Proc. 5th ACIS Int. Conf. Softw. Eng., Artif. Intell., Netw. Parallel/Distrib. Comput., 2004, pp. 128-134.
29. Z. Zhang, W. K. Chan, T. H. Tse, and P. Hu, "Experimental study to compare the use of metamorphic testing and assertion checking," J. Softw., vol. 20, no. 10, pp. 2637-2654, 2009.
30. T. Y. Chen, T. H. Tse, and Z. Q. Zhou, "Semi-proving: An integrated method for program proving, testing, and debugging," IEEE Trans. Softw. Eng., vol. 37, no. 1, pp. 109-125, Jan. 2011.
31. X. Xie, J. W. K. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y. Chen. (2011, Apr.). Testing and validating machine learning classifiers by metamorphic testing. J. Syst. Softw. [Online]. 84 (4), pp. 544-558. Available: http://dx.doi.org/10.1016/j. jss.2010.11.920.
32. A. C. Barus, T. Y. Chen, D. Grant, F.-C. Kuo, and M. F. Lau, " Testing of heuristic methods: A case study of greedy algorithm," in Software Engineering Techniques, Z. Huzar, R. Koci, B. Meyer, B. Walter, and J. Zendulka, Eds. Berlin, Germany: Springer, 2011, vol. 4980, pp. 246-260. [Online]. Available: http://dx.doi.org/10.1007/978-3-642-22386-0-19.
33. F.-C. Kuo, S. Liu, and T. Y. Chen, "Testing a binary space partitioning algorithm with metamorphic testing," in Proc. ACM Symp. Appl. Comput., 2011, pp. 1482-1489. [Online]. Available: http://doi.acm.org/10.1145/1982185.1982502.
34. C. Sun, G. Wang, B. Mu, H. Liu, Z. Wang, and T. Y. Chen, "Metamorphic testing for web services: Framework and a case study," in Proc. IEEE Int. Conf. Web Services, Jul. 2011, pp. 283-290.
35. Z.-W. Hui, S. Huang, H. Li, J.-H. Liu, and L.-P. Rao, "Measurable metrics for qualitative guidelines of metamorphic relation," in Proc. IEEE 39th Annu. Comput. Softw. Appl. Conf., vol. 3, Jul. 2015, pp. 417-422.
36. P. Hu, Z. Zhang, W. K. Chan, and T. H. Tse, "An empirical comparison between direct and indirect test result checking approaches," in Proc. 3rd Int. Workshop Softw. Quality Assurance, 2006, pp. 6-13. [Online]. Available: http://doi.acm.org/10.1145/1188895.1188901.
37. T. Y. Chen, F.-C. Kuo, R. Merkel, and W. K. Tam, "Testing an open source suite for open queuing network modelling using metamorphic testing technique," in Proc. IEEE 14th Int. Conf. Eng. Complex Comput. Syst., Jun. 2009, pp. 23-29.
38. J. Mayer and R. Guderlei, "An empirical study on the selection of good metamorphic relations," in Proc. 30th Annu. Int. Comput. Softw. Appl. Conf., vol. 1, Sep. 2006, pp. 475-484.
39. Z. Q. Zhou, D. H. Huang, T. H. Tse, Z. Yang, H. Huang, and T. Y. Chen, "Metamorphic testing and its applications," in presented at the 8th Int. Symp. Future Software Technology, Xian, China, 2004.
40. T. Y. Chen, J. W. K. Ho, H. Liu, and X. Xie. (2009). An innovative approach for testing bioinformatics programs using metamorphic testing. BioMed. Central Bioinform. J. [Online]. 10(1), p. 24. Available: http://www.biomedcentral.com/1471-2105/10/24.
41. G. Batra and J. Sengupta, "An efficient metamorphic testing technique using genetic algorithm," in Information Intelligence, Systems, Technology and Management (ser. Communications in Computer and Information Science), S. Dua, S. Sahni, and D. Goyal, Eds., Berlin, Germany: Springer, 2011, vol. 141, pp. 180-188. [Online]. Available: http://dx.doi.org/10.1007/978-3-642-19423-8-19.
42. L. Chen, L. Cai, J. Liu, Z. Liu, S. Wei, and P. Liu, "An optimized method for generating cases of metamorphic testing," in Proc. 6th Int. Conf. New Trends Inform. Sci. Service Sci. Data Mining, Oct. 2012, pp. 439-443.
43. J. Ding, T. Wu, J. Q. Lu, and X. Hu, "Self-checked metamorphic testing of an image processing program," in Proc. 4th Int. Conf. Secure Softw. Integr. Rel. Improvement, Jun. 2010, pp. 190-197.
44. G. Dong, T. Guo, and P. Zhang, "Security assurance with program path analysis and metamorphic testing," in Proc. 4th IEEE Int. Conf. Softw. Eng. Service Sci., May 2013, pp. 193-197.
45. F.-C. Kuo, Z. Q. Zhou, J. Ma, and G. Zhang, "Metamorphic testing of decision support systems: A case study," IET Softw., vol. 4, no. 4, pp. 294-301, Aug. 2010.
46. M. Asrafi, H. Liu, and F.-C. Kuo, "On testing effectiveness of metamorphic relations: A case study," in Proc. 5th Int. Conf. Secure Softw. Integr. Rel. Improvement, Jun. 2011, pp. 147-156.
47. Y. Cao, Z. Q. Zhou, and T. Y. Chen, "On the correlation between the effectiveness of metamorphic relations and dissimilarities of test case executions," in Proc. 13th Int. Conf. Quality Softw., Jul. 2013, pp. 153-162.
48. Z. Q. Zhou, "Using coverage information to guide test case selection in adaptive random testing," in Proc. Comput. Softw. Appl. Conf. Workshops, Jul. 2010, pp. 208-213.
49. R. Just and F. Schweiggert, "Automating software tests with partial oracles in integrated environments," in Proc. 5th Workshop Autom. Softw. Test, 2010, pp. 91-94. [Online]. Available: http://doi.acm.org/10.1145/1808266.1808280.
50. R. Just and F. Schweiggert. (2011). Automating unit and integration testing with partial oracles. Softw. Quality J. [Online]. 19(4), pp. 753-769. Available: http://dx.doi.org/10.1007/s11219-011-9151-x.
51. X. Xie, J. Tu, T. Y. Chen, and B. Xu, "Bottom-up integration testing with the technique of metamorphic testing," in Proc. 14th Int. Conf. Quality Softw., Oct. 2014, pp. 73-78.
52. W. K. Chan, T. Y. Chen, H. Lu, T. H. Tse, and S. S. Yau. (2006). Integration testing of context-sensitive middleware-based applications: A metamorphic approach. Int. J. Softw. Eng. Knowl. Eng. [Online]. 16(5), pp. 677-704. Available: http://dblp.uni-trier.de/db/journals/ijseke/ijseke16.html#ChanCLTY06.
53. Z. Hui and S. Huang, "A formal model for metamorphic relation decomposition," in Proc. 4th World Congr. Softw. Eng., Dec. 2013, pp. 64-68.
54. H. Liu, X. Liu, and T. Y. Chen, "A new method for constructing metamorphic relations," in Proc. 12th Int. Conf. Quality Softw., Aug. 2012, pp. 59-68.
55. G. Dong, B. Xu, L. Chen, C. Nie, and L. Wang, "Case studies on testing with compositional metamorphic relations," J. Southeast Univ. (English Edition), vol. 24, no. 4, pp. 437-443, 2008.
56. U. Kanewala and J. M. Bieman, "Using machine learning techniques to detect metamorphic relations for programs without test oracles," in Proc. IEEE 24th Int. Symp. Softw. Rel. Eng., Nov. 2013, pp. 1-10.
57. U. Kanewala, "Techniques for automatic detection of metamorphic relations," in Proc. IEEE 7th Int. Conf. Softw. Testing, Verification Validation Workshops, Mar. 2014, pp. 237-238.
58. C. Murphy, G. Kaiser, and L. Hu, "Properties of machine learning applications for use in metamorphic testing," Dept. Comput. Sci., Columbia Univ., New York NY, USA, Tech. Rep. CUCS-011-08, 2008.
59. U. Kanewala, J. M. Bieman, and A. Ben-Hur. (2015). Predicting metamorphic relations for testing scientific software: A machine learning approach using graph kernels. Softw. Testing, Verification Rel. [Online]. Available: http://dx.doi. org/10.1002/stvr.1594.
60. J. Zhang, J. Chen, D. Hao, Y. Xiong, B. Xie, L. Zhang, and H. Mei, "Search-based inference of polynomial metamorphic relations," in Proc. 29th ACM/IEEE Int. Conf. Automat. Softw. Eng., 2014, pp. 701-712. [Online]. Available: http://doi.acm.org/10.1145/2642937.2642994.
61. A. Carzaniga, A. Goffi, A. Gorla, A. Mattavelli, and M. Pezz-e, "Cross-checking oracles from intrinsic software redundancy," in Proc. 36th Int. Conf. Softw. Eng., 2014, pp. 931-942. [Online]. Available: http://doi.acm.org/10.1145/2568225.2568287.
62. A. Goffi, A. Gorla, A. Mattavelli, M. Pezze, and P. Tonella, "Search-based synthesis of equivalent method sequences," in Proc. 22nd ACM SIGSOFT Int. Symp. Found.Softw. Eng., 2014, pp. 366-376. [Online]. Available: http://doi.acm.org/10.1145/2635868.2635888.
63. A. Goffi, "Automatic generation of cost-effective test oracles," in Proc. 36th Int. Conf. Softw. Eng., 2014, pp. 678-681. [Online]. Available: http://doi.acm.org/10.1145/2591062.2591078.
64. F. Su, J. Bell, C. Murphy, and G. Kaiser, "Dynamic inference of likely metamorphic properties to support differential testing," in Proc. 10th Int.Workshop Autom. Softw. Test, 2015, pp. 55-59. [Online]. Available: http://dl.acm.org/citation.cfm?id=2819261.2819279.
65. T. Y. Chen, P. Poon, and X. Xie. (2015). METRIC: METamorphic Relation Identification based on the category-choice framework. J. Syst. Softw. [Online]. Available: http://www.sciencedirect. com/science/article/pii/S0164121215001624.
66. T. Y. Chen, P.-L. Poon, S.-F. Tang, and T. H. Tse, "Dessert: a divide-and-conquer methodology for identifying categories, choices, and choice relations for test case generation," IEEE Trans. Softw. Eng., vol. 38, no. 4, pp. 794-809, Jul./Aug. 2012.
67. A. Gotlieb and B. Botella, "Automated metamorphic testing," in Proc. 27th Annu. Int. Conf. Comput. Softw. Appl., 2003, pp. 34-40. [Online]. Available: http://dl.acm.org/citation. cfm?id=950785.950794.
68. P. Wu, X. Shi, J. Tang, H. Lin, and T. Y. Chen, "Metamorphic testing and special case testing: A case study," J. Softw., vol. 16, pp. 1210-1220, 2005.
69. S. Segura, R. M. Hierons, D. Benavides, and A. Ruiz-Cort-es. Automated metamorphic testing on the analyses of feature models. Inform. Softw. Technol. [Online]. 53(3), pp. 245-258. Available: http://www.sciencedirect.com/science/article/pii/S0950584910001904.
70. P. Wu, "Iterative metamorphic testing," in Proc. 29th Annu. Int. Comput. Softw. Appl. Conf., vol. 1, pp. 19-24, Jul. 2005.
71. G. Dong, C. Nie, B. Xu, and L. Wang, "An effective iterative metamorphic testing algorithm based on program path analysis," in Proc. 7th Int. Conf. Quality Softw., Oct. 2007, pp. 292-297.
72. S. Segura, R. M. Hierons, D. Benavides, and A. Ruiz-Cortes, "Automated test data generation on the analyses of feature models: A metamorphic testing approach," in Proc. 3rd Int. Conf. Softw. Testing, Verification Validation, Apr. 2010, pp. 35-44.
73. S. Segura, A. Dur-an, A. B. S-anchez, D. L. Berre, E. Lonca, and A. Ruiz-Cort-es. (2015). Automated metamorphic testing of variability analysis tools. Softw. Testing, Verification Rel. [Online]. 25(2), pp. 138-163. Available: http://dx.doi.org/10.1002/stvr.1566.
74. R. Guderlei and J. Mayer, "Statistical metamorphic testing: Testing programs with random output by means of statistical hypothesis tests and metamorphic testing," in Proc. 7th Int. Conf. Quality Softw., Oct. 2007, pp. 404-409.
75. C. Murphy and G. Kaiser. (2010). Empirical evaluation of approaches to testing applications without test oracles. Columbia Univ. Comput. Sci., New York, NY, Tech. Rep. CUCS-039-10. [Online]. Available: http://hdl.handle.net/10022/AC:P:10525.
76. C. Murphy, M. S. Raunak, A. King, S. Chen, C. Imbriano, G. Kaiser, I. Lee, O. Sokolsky, L. Clarke, and L. Osterweil, "On effective testing of health care simulation software," in Proc. 3rd Workshop Softw. Eng. Health Care, 2011, pp. 40-47. [Online]. Available: http://doi.acm.org/10.1145/1987993.1988003.
77. C. Murphy, "Using runtime testing to detect defects in applications without test oracles," in Proc. Found.Softw. Eng. Doctoral Symp., 2008, pp. 21-24. [Online]. Available: http://doi.acm.org/10.1145/1496653.1496659.
78. C. Murphy, K. Shen, and G. Kaiser, "Using JML runtime assertion checking to automate metamorphic testing in applications without test oracles," in Proc. Second Int. Conf. Softw. Testing Verification Validation, 2009, pp. 436-445.
79. (2016). "Java Modeling Language (JML). [Online]. Available: http://www.jmlspecs.org.
80. C. Murphy, G. Kaiser, L. Hu, and L. Wu, "Properties of machine learning applications for use in metamorphic testing," in Proc. Int. Conf. Softw. Eng. Knowl. Eng., 2008, pp. 867-872.
81. C. Murphy, K. Shen, and G. Kaiser, "Automatic system testing of programs without test oracles," in Proc. 18th Int. Symp. Softw. Testing Anal., 2009, pp. 189-200. [Online]. Available: http://doi. acm.org/10.1145/1572272.1572295.
82. H. Zhu, "Jfuzz: A tool for automated Java unit testing based on data mutation and metamorphic testing methods," in Proc. 2nd Int. Conf. Trustworthy Syst. Appl., Jul. 2015, pp. 8-15.
83. W. K. Chan, S. C. Cheung, and K. R. P. Leung, "Towards a metamorphic testing methodology for service-oriented software applications," in Proc. 5th Int. Conf. Quality Softw., Sep. 2005, pp. 470-476.
84. W. K. Chan, S. C. Cheung, and K. R. P. H. Leung. (2007). A metamorphic testing approach for online testing of service-oriented software applications. Int. J. Web Serv. Res. [Online]. 4(2), pp. 61-81. Available: http://dblp.uni-trier.de/db/journals/jwsr/jwsr4. html#ChanCL07.
85. C. Sun, G. Wang, B. Mu, H. Liu, Z. Wang, and T. Y. Chen. (2012, Jan.). A metamorphic relation-based approach to testing web services without oracles. Int. J. Web Serv. Res. [Online]. 9(1), pp. 51-73. Available: http://dx.doi.org/10.4018/jwsr.2012010103.
86. C. Castro-Cabrera and I. Medina-Bulo, "An approach to metamorphic testing for WS-BPEL compositions," in Proc. Int. Conf. e-Bus., Jul. 2011, pp. 1-6.
87. C. Castro-Cabrera and I. Medina-Bulo, "Application of metamorphic testing to a case study in web services compositions," in E-Business and Telecommunications (ser. Communications in Computer and Information Science), M. Obaidat, J. Sevillano, and J. Filipe, Eds., Berlin, Germany: Springer, 2012, vol. 314, pp. 168-181. [Online]. Available: http://dx.doi.org/10.1007/978-3-642-35755-8-13.
88. OASIS: Web Services Business Process Execution Language. (2007, Apr.). [Online]. Available: 2.0. http://docs.oasis-open. org/wsbpel/2.0/OS/wsbpel-v2.0-OS.html.
89. Z. Q. Zhou, T. H. Tse, F.-C. Kuo, and T. Y. Chen, "Automated functional testing of web search engines in the absence of an oracle," Dept. Comput. Sci., The Univ. Hong Kong, Tech. Rep. TR-2007-06, 2007.
90. Z. Q. Zhou, S. Xiang, and T. Y. Chen, "Metamorphic testing for software quality assessment: A study of search engines," IEEE Trans. Softw. Eng., 2015, Doi: 10.1109/TSE.2015.2478001.
91. J. Mayer and R. Guderlei, "On random testing of image processing applications," in Proc. 6th Int. Conf. Quality Softw., Oct. 2006, pp. 85-92.
92. R. Guderlei and J. Mayer. (2007). Towards automatic testing of imaging software by means of random and metamorphic testing. Int. J. Softw. Eng. Knowl. Eng. [Online]. 17(06), pp. 757-781. Available: http://www.worldscientific.com/doi/abs/10.1142/S0218194007003471.
93. W. K. Chan, J. C. F. Ho, and T. H. Tse, "Piping classification to metamorphic testing: An empirical study towards better effectiveness for the identification of failures in mesh simplification programs," in Proc. 31st Annu. Int. Comput. Softw. Appl. Conf., vol. 1, pp. 397-404, Jul. 2007.
94. W. K. Chan, J. C. F. Ho, and T. H. Tse. (2010, Jun.). Finding failures from passed test cases: Improving the pattern classification approach to the testing of mesh simplification programs. Softw. Testing, Verification Rel. J. [Online]. 20(2), pp. 89-120. Available: http://dx.doi.org/10.1002/stvr.v20:2.
95. R. Just and F. Schweiggert, "Evaluating testing strategies for imaging software by means of mutation analysis," in Proc. Int. Conf. Softw. Testing, Verification Validation Workshops, Apr. 2009, pp. 205-209.
96. T. Jameel, L. Mengxiang, and C. Liu, "Test oracles based on metamorphic relations for image processing applications," in Proc. 16th IEEE/ACIS Int. Conf. Softw. Eng., Artif. Intell., Netw. Parallel/Distrib. Comput., Jun. 2015, pp. 1-6.
97. T. H. Tse, S. S. Yau, W. K. Chan, H. Lu, and T. Y. Chen, "Testing context-sensitive middleware-based software applications," in Proc. 28th Annu. Int. Comput. Softw. Appl. Conf., vol. 1, pp. 458-466, Sep. 2004.
98. W. K. Chan, T. Y. Chen, H. Lu, T. H. Tse, and S. S. Yau, "A metamorphic approach to integration testing of context-sensitive middleware-based applications," in Proc. 5th Int. Conf. Quality Softw., Sep. 2005, pp. 241-249.
99. W. K. Chan, T. Y. Chen, S. C. Cheung, T. H. Tse, and Z. Zhang, " Towards the testing of power-aware software applications for wireless sensor networks," in Ada Europe 2007-Reliable Software Technologies, N. Abdennadher and F. Kordon, Eds., Berlin, Germany: Springer, 2007, vol. 4498, pp. 84-99. [Online]. Available: http://dx.doi.org/10.1007/978-3-540-73230-3-7.
100. F.-C. Kuo, T. Y. Chen, and W. K. Tam, "Testing embedded software by metamorphic testing: A wireless metering system case study," in Proc. IEEE 36th Conf. Local Comput. Netw., Oct. 2011, pp. 291-294.
101. M. Jiang, T. Y. Chen, F.-C. Kuo, and Z. Ding, "Testing central processing unit scheduling algorithms using metamorphic testing," in Proc. 4th IEEE Int. Conf. Softw. Eng. Service Sci., May 2013, pp. 530-536.
102. K. Y. Sim, W. K. S. Pao, and C. Lin, "Metamorphic testing using geometric interrogation technique and its application," in Proc. 2nd Int. Conf. Elect. Eng./Electron., Comput., Telecommun., Inform. Technol., 2005, pp. 91-95.
103. T. Y. Chen, F.-C. Kuo, H. Liu, and S. Wang, " Conformance testing of network simulators based on metamorphic testing technique," in Formal Techniques for Distributed Systems, D. Lee, A. Lopes, and A. Poetzsch-Heffter, Eds., Berlin, Germany: Springer, vol. 5522, pp. 243-248, 2009. [Online]. Available: http://dx.doi.org/10.1007/978-3-642-02138-1-19.
104. OMNeT++ system. (2015, Dec.). [Online]. Available: http://www.omnetpp.org.
105. J. Ding, T. Wu, D. Xu, J. Q. Lu, and X. Hu, "Metamorphic testing of a Monte Carlo modeling program," in Proc. 6th Int. Workshop Autom. Softw. Test, 2011, pp. 1-7. [Online]. Available: http://doi. acm.org/10.1145/1982595.1982597.
106. A. Nuñez and R. M. Hierons. (2014). A methodology for validating cloud models using metamorphic testing. Ann. Telecommun.-annales des t-el-ecommunications [Online]. pp. 1-9. Available: http://dx.doi.org/10.1007/s12243-014-0442-7.
107. A. Nuñez, J. L. Vazquez-Poletti, A. C. Caminero, G. G. Castañe, J. Carretero, and I. M. Llorente. (2012). icancloud: A flexible and scalable cloud infrastructure simulator. J. Grid Comput., 10(1), pp. 185-209. [Online]. Available: http://dx.doi.org/10.1007/s10723-012-9208-5.
108. P. C. Cañizares, A. N-uñez, M. N-uñez, and J. J. Pardo, "A methodology for designing energy-aware systems for computational science," Procedia Comput. Sci., vol. 51, pp. 2804-2808, 2015, (Int. Conf. Comput. Sci.) [Online]. Available: http://www.sciencedirect. com/science/article/pii/S1877050915012466.
109. X. Xie, J. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y. Chen, "Application of metamorphic testing to supervised classifiers," in Proc. 9th Int. Conf. Quality Softw., Aug. 2009, pp. 135-144.
110. J. E. Gewehr, M. Szugat, and R. Zimmer. (2007, Feb.). Bioweka-extending the weka framework for bioinformatics. Bioinformatics. [Online]. 23(5), pp. 651-653. Available: http://dx.doi.org/10.1093/bioinformatics/btl671.
111. Z. Jing, H. Xuegang, and Z. Bin. (2011). An evaluation approach for the program of association rules algorithm based on metamorphic relations. J. Electron. (China) [Online]. 28(4-6), pp. 623-631. Available: http://dx.doi.org/10.1007/s11767-012-0743-9.
112. L. L. Pullum and O. Ozmen, "Early results from metamorphic testing of epidemiological models," in Proc. ASE/IEEE Int. Conf. BioMedical Comput., Dec. 2012, pp. 62-67.
113. A. Ramanathan, C. A. Steed, and L. L. Pullum, "Verification of compartmental epidemiological models using metamorphic testing, model checking and visual analytics," in Proc. ASE/IEEE Int. Conf. BioMed. Comput., Dec. 2012, pp. 68-73.
114. S. Beydeda, "Self-metamorphic-testing components," in Proc. 30th Annu. Int. Comput. Softw. Appl. Conf., vol. 2, pp. 265-272, Sep. 2006.
115. X. Lu, Y. Dong, and C. Luo, "Testing of component-based software: A metamorphic testing methodology," in Proc. Int. Conf. Ubiquitous Intell. Comput. Int. Conf. Auton. Trusted Comput., Oct. 2010, pp. 272-276.
116. T. Y. Chen, J. Feng, and T. H. Tse, "Metamorphic testing of programs on partial differential equations: A case study," in Proc. 26th Int. Comput. Softw. Appl. Conf. Prolonging Softw. Life: Develop. Redevelop., 2002, pp. 327-333. [Online]. Available: http://dl.acm. org/citation.cfm?id=645984.675903.
117. C. Aruna and R. S. R. Prasad, "Metamorphic relations to improve the test accuracy of multi precision arithmetic software applications," in Proc. Int. Conf. Adv. Comput., Commun. Informat., Sep. 2014, pp. 2244-2248.
118. Q. Tao, W. Wu, C. Zhao, and W. Shen, "An automatic testing approach for compiler based on metamorphic testing technique," in Proc. 17th Asia Pacific Softw. Eng. Conf., Nov. 2010, pp. 270-279.
119. V. Le, M. Afshari, and Z. Su, "Compiler validation via equivalence modulo inputs," in Proc. 35th ACM SIGPLAN Conf. Program. Language Des. Implementation, 2014, pp. 216-226. [Online]. Available: http://doi.acm.org/10.1145/2594291.2594334.
120. T. Y. Chen, F.-C. Kuo, and Z. Q. Zhou, "An effective testing method for end-user programmers," in Proc. 1st Workshop End-User Softw. Eng., 2005, pp. 1-5. [Online]. Available: http://doi. acm.org/10.1145/1082983.1083236.
121. K. Y. Sim, C. S. Low, and F.-C. Kuo, "Detecting faults in technical indicator computations for financial market analysis," in Proc. 2nd Int. Conf. Inform. Sci. Eng., Dec. 2010, pp. 2749-2754.
122. (2016). MetaTrader 4 Trading Terminal. [Online]. Available: http://www.metaquotes.net/en/metatrader4/trading-terminal.
123. S. Yoo, "Metamorphic testing of stochastic optimisation," in Proc. 3rd Int. Conf. Softw. Testing, Verification, Validation Workshops, Apr. 2010, pp. 192-201.
124. Y. Yao, S. Huang, and M. Ji, "Research on metamorphic testing for oracle problem of integer bugs," in Proc. 4th Int. Conf. Adv. Comput. Sci. Inf. Eng., ser. Advances in Intelligent and Soft Computing, 2012, vol. 168, pp. 95-100. [Online]. Available: http://dx. doi.org/10.1007/978-3-642-30126-1-16.
125. Y. Yao, C. Zheng, S. Huang, and Z. Ren, "Research on metamorphic testing: A case study in integer bugs detection," in Proc. 4th Int. Conf. Intell. Syst. Design Eng. Appl., 2013, Nov. 2013, pp. 488-493.
126. Z. Hui, S. Huang, Z. Ren, and Y. Yao, "Metamorphic testing integer overflow faults of mission critical program: A case study," Math. Problems Eng., vol. 2013, pp. 1-6, 2013.
127. G. Batra and G. Singh, "An automated metamorphic testing technique for designing effective metamorphic relations," Contemporary Computing (ser. Communications in Computer and Information Science), M. Parashar, D. Kaushik, O. Rana, R. Samtaney, Y. Yang, and A. Zomaya, Eds., Berlin, Germany: Springer, 2012, vol. 306, pp. 152-163. [Online]. Available: http://dx.doi. org/10.1007/978-3-642-32129-0-20.
128. C. Sun, Z. Wang, and G. Wang. (2014). A property-based testing framework for encryption programs. Frontiers Comput. Sci. [Online]. 8(3), pp. 478-489. Available: http://dx.doi.org/10.1007/s11704-014-3040-y.
129. C. Aruna and R. S. R. Prasad, "Adopting metamorphic relations to verify non-testable graph theory algorithms," in Proc. 2nd Int. Conf. Adv. Comput. Commun. Eng., May 2015, pp. 673-678.
130. M. Lindvall, D. Ganesan, R. Ardal, and R. Wiegand, "Metamorphic model-based testing applied on NASA DAT-An experience report," in Proc. 37th IEEE Int. Conf. Softw. Eng., vol. 2, pp. 129-138, May 2015.
131. T. Y. Chen, T. H. Tse, and Z. Q. Zhou, "Fault-based testing in the absence of an oracle," in Proc. 25th Annu. Int. Comput. Softw. Appl. Conf., 2001, pp. 172-178.
132. C. Cadar and K. Sen. (Feb 2013). Symbolic execution for software testing: Three decades later. Commun. ACM[Online]. 56(2), pp. 82-90. Available: http://doi.acm.org/10.1145/2408776.2408795.
133. I. Erete and A. Orso, "Optimizing constraint solving to better support symbolic execution," in Proc. Workshop Constraints Softw. Testing, Verification, Anal., 2011, pp. 310-315.
134. T. Y. Chen, T. H. Tse, and Z. Q. Zhou, "Semi-proving: An integrated method based on global symbolic evaluation and metamorphic testing," in Proc. ACM SIGSOFT Int. Symp. Softw. Testing Anal., 2002, pp. 191-195. [Online]. Available: http://doi. acm.org/10.1145/566172.566202.
135. G. Dong, S. Wu, G.Wang, T. Guo, and Y. Huang, "Security assurance with metamorphic testing and genetic algorithm," in Proc. IEEE/WIC/ACM Int. Conf. Web Intell. Intell. Agent Technol., vol. 3, pp. 397-401, Aug. 2010.
136. X. Xie, W. E. Wong, T. Y. Chen, and B. Xu, "Spectrum-based fault localization: Testing oracles are no longer mandatory," in Proc. 11th Int. Conf. Quality Softw., Jul. 2011, pp. 1-10.
137. X. Xie, W. E. Wong, T. Y. Chen, and B. Xu. (2013). Metamorphic slice: An application in spectrum-based fault localization," Inform. Softw. Technol. [Online]. 55(5), pp. 866-879. Available: http://www.sciencedirect.com/science/article/pii/S0950584912001759.
138. (2016). Siemens Suite. [Online]. Available: http://sir.unl.edu/portal/bios/tcas.php#siemens.
139. Y. Lei, X. Mao, and T. Y. Chen, "Backward-slice-based statistical fault localization without test oracles," in Proc. 13th Int. Conf. Quality Softw., Jul. 2013, pp. 212-221.
140. Y. Lei, X. Mao, Z. Dai, and C. Wang, "Effective statistical fault localization using program slices," in Proc. Comput. Softw. Appl. Conf., Jul. 2012, pp. 1-10.
141. P. Rao, Z. Zheng, T. Y. Chen, N. Wang, and K. Cai, "Impacts of test suite's class imbalance on spectrum-based fault localization techniques," in Proc. 13th Int. Conf. Quality Softw., Jul. 2013, pp. 260-267.
142. C. Aruna and R. S. R. Prasad, "Testing approach for dynamic web applications based on automated test strategies," in ICT and Critical Infrastructure: Proc. 48th Annu. Convention of Comput. Soc. India-Vol II, ser. Advances in Intelligent Systems and Computing, vol. 249, pp. 399-410, 2014. [Online]. Available: http://dx. doi.org/10.1007/978-3-319-03095-1-43.
143. S. Artzi, A. Kiezun, J. Dolby, F. Tip, D. Dig, A. Paradkar, and M. D. Ernst, "Finding bugs in dynamic web applications," in Proc. Int. Symp. Softw. Testing Anal., 2008, pp. 261-272. [Online]. Available: http://doi.acm.org/10.1145/1390630.1390662.
144. H. Liu, I. I. Yusuf, H. W. Schmidt, and T. Y. Chen, "Metamorphic fault tolerance: An automated and systematic methodology for fault tolerance in the absence of test oracle," in Proc. 36th Int. Conf. Softw. Eng., 2014, pp. 420-423. [Online]. Available: http://doi.acm.org/10.1145/2591062.2591109.
145. H. Jin, Y. Jiang, N. Liu, C. Xu, X. Ma, and J. Lu, "Concolic metamorphic debugging," in Proc. IEEE 39th Annu. Comput. Softw. Appl. Conf., vol. 2, pp. 232-241, Jul. 2015.
146. H. Liu, F.-C. Kuo, and T. Y. Chen, "Teaching an end-user testing methodology," in Proc. 23rd IEEE Conf. Softw. Eng. Educ. Training, Mar. 2010, pp. 81-88.
147. S. Yoo and M. Harman. (2012). Regression testing minimization, selection and prioritization: A survey. Softw. Testing, Verification Rel. [Online]. 22(2), pp. 67-120. Available: http://dx.doi.org/10.1002/stvr.430.
148. M. D. Ernst, J. Cockrell, W. G. Griswold, and D. Notkin, "Dynamically discovering likely program invariants to support program evolution," in Proc. 21st Int. Conf. Softw. Eng., 1999, pp. 213-224. [Online]. Available: http://doi.acm.org/10.1145/302405.302467.