Metamorphic Testing for Software Quality Assessment: A Study of Search Engines

IEEE Transactions on Software Engineering

Volumn 42, Issue 3, 2016, Pages 260-280

  Zhou, Zhi Quan ^a   Xiang, Shaowen ^a   Chen, Tsong Yueh ^b  

^a UNIVERSITY OF WOLLONGONG   (Australia)

^b SWINBURNE UNIVERSITY OF TECHNOLOGY   (Australia)

Author keywords

lack of system specification; metamorphic testing; oracle problem; quality assessment; search engine; Software quality; user oriented testing; validation; verification

Indexed keywords

COMPUTER SOFTWARE SELECTION AND EVALUATION; SEARCH ENGINES; SPECIFICATIONS; TESTING; VERIFICATION;

METAMORPHIC TESTING; ORACLE PROBLEM; QUALITY ASSESSMENT; SOFTWARE QUALITY; SYSTEM SPECIFICATION; USER ORIENTED; VALIDATION;

SOFTWARE TESTING;

EID: 84963808208     PISSN: 00985589     EISSN: 19393520     Source Type: Journal    
DOI: 10.1109/TSE.2015.2478001     Document Type: Article

References (53)

1. C. Ghezzi, M. Jazayeri, and D. Mandrioli, Fundamentals of Software Engineering, 2nd ed. Englewood Cliffs, NJ, USA: Prentice-Hall, 2002.
2. R. M. Hierons, "Oracles for distributed testing," IEEE Trans. Softw. Eng., vol. 38, no. 3, pp. 629-641, May/Jun. 2012.
3. T. Y. Chen, S. C. Cheung, and S. M. Yiu, "Metamorphic testing: A new approach for generating next test cases," Dept. Comput. Sci., Hong Kong Univ. Sci. Technol., Hong Kong, Tech. Rep. HKUSTCS98-01, 1998.
4. T. Y. Chen, T. H. Tse, and Z. Q. Zhou, "Semi-proving: An integrated method based on global symbolic evaluation and metamorphic testing," in Proc. ACM SIGSOFT Int. Symp. Softw. Testing Anal., 2002, pp. 191-195.
5. T. Y. Chen, T. H. Tse, and Z. Q. Zhou, "Fault-based testing without the need of oracles," Inf. Softw. Technol., vol. 45, no. 1, pp. 1-9, 2003.
6. T. Y. Chen, J. W. K. Ho, H. Liu, and X. Xie, "An innovative approach for testing bioinformatics programs using metamorphic testing," BMC Bioinformat., vol. 10, no. 24, p. 24, 2009.
7. C. Murphy, K. Shen, and G. E. Kaiser, "Automatic system testing of programs without test oracles," in Proc. 18th ACM SIGSOFT Int. Symp. Softw. Testing Anal., 2009, pp. 189-200.
8. X. Xie, J. W. K. Ho, C. Murphy, G. Kaiser, B. Xu, and T. Y. Chen, "Testing and validating machine learning classifiers by metamorphic testing," J. Syst. Softw., vol. 84, pp. 544-558, 2011.
9. J. Zhang, J. Chen, D. Hao, Y. Xiong, B. Xie, L. Zhang, and H. Mei, "Search-based inference of polynomial metamorphic relations," in Proc. 29th IEEE/ACM Int. Conf. Autom. Softw. Eng., 2014, pp. 701-712.
10. V. Le, M. Afshari, and Z. Su, "Compiler validation via equivalence modulo inputs," in Proc. 35th ACM SIGPLAN Conf. Program. Lang. Des. Implementation, 2014, pp. 216-226.
11. J. Regehr. (2014, Jun. 20). Finding compiler bugs by removing dead code [Online]. Available: http://blog.regehr.org/archives/1161
12. A. Núñez and R. M. Hierons, "A methodology for validating cloud models using metamorphic testing," Ann. Telecommun., vol. 70, no. 3-4, pp. 127-135, 2015.
13. S. Segura, A. Durán, A. B. Sánchez, D. L. Berre, E. Lonca, and A. Ruiz-Cortés, "Automated metamorphic testing of variability analysis tools," Softw. Testing, Verification Rel., vol. 25, pp. 138-163, 2015.
14. M. Lindvall, D. Ganesan, R. Årdal, and R. E. Wiegand, "Metamorphic model-based testing applied on NASA DAT - an experience report," in Proc. 37th Int. Conf. Softw. Eng., 2015, pp. 129-138.
15. T. Y. Chen, T. H. Tse, and Z. Q. Zhou, "Semi-proving: An integrated method for program proving, testing, and debugging," IEEE Trans. Softw. Eng., vol. 37, no. 1, pp. 109-125, Jan./Feb. 2011.
16. X. Xie, W. E. Wong, T. Y. Chen, and B. Xu, "Metamorphic slice: An application in spectrum-based fault localization," Inf. Softw. Technol., vol. 55, no. 5, pp. 866-879, 2013.
17. Y. Cao, Z. Q. Zhou, and T. Y. Chen, "On the correlation between the effectiveness of metamorphic relations and dissimilarities of test case executions," in Proc. 13th Int. Conf. Quality Softw., 2013, pp. 153-162.
18. H. Liu, F.-C. Kuo, D. Towey, and T. Y. Chen, "How effectively does metamorphic testing alleviate the oracle problem?" IEEE Trans. Softw. Eng., vol. 40, no. 1, pp. 4-22, Jan. 2014.
19. I. Soboroff, "Dynamic test collections: Measuring search effectiveness on the live Web," in Proc. 29th Annu. Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2006, pp. 276-283.
20. NetMarketshare. (2015, Mar.). Market share reports - desktop search engine market share [Online]. Available: https://www. netmarketshare.com
21. M. Pezzé and M. Young, Software Testing and Analysis: Process, Principles, and Techniques. New York, NY, USA: Wiley, 2008.
22. D. Lo, J. Li, L. Wong, and S.-C. Khoo, "Mining iterative generators and representative rules for software specification discovery," IEEE Trans. Knowl. Data Eng., vol. 23, no. 2, pp. 282-296, Feb. 2011.
23. T. Y. Chen, F.-C. Kuo, and Z. Q. Zhou, "An effective testing method for end-user programmers," in Proc. Workshop End-User Softw. Eng., 2005, pp. 21-25.
24. A. J. Ko, R. Abraham, L. Beckwith, A. Blackwell, M. Burnett, M. Erwig, C. Scaffidi, J. Lawrance, H. Lieberman, B. Myers, M. B. Rosson, G. Rothermel, M. Shaw, and S. Wiedenbeck, "The state of the art in end-user software engineering," ACM Comput. Surveys, vol. 43, no. 3, pp. 21:1-21:44, 2011.
25. T. Y. Chen, F.-C. Kuo, D. Towey, and Z. Q. Zhou, "Metamorphic testing: Applications and integration with other methods," in Proc. 12th Int. Conf. Quality Softw., 2012, pp. 285-288.
26. T. Y. Chen, D. H. Huang, T. H. Tse, and Z. Q. Zhou, "Case studies on the selection of useful relations in metamorphic testing," in Proc. 4th Ibero-Am. Symp. Softw. Eng. Knowl. Eng., 2004, pp. 569-583.
27. Z. Q. Zhou, T. H. Tse, F.-C. Kuo, and T. Y. Chen, "Automated functional testing of web search engines in the absence of an oracle," Dept. Comput. Sci., The Univ of Hong Kong, Hong Kong, Tech. Rep. TR-2007-06, 2007.
28. Z. Q. Zhou, S. Zhang, M. Hagenbuchner, T. H. Tse, F.-C. Kuo, and T. Y. Chen, "Automated functional testing of online search services," Softw. Testing, Verification Rel., vol. 22, no. 4, pp. 221-243, 2012.
29. T. Imielinski and A. Signorini, "If you ask nicely, I will answer: Semantic search and today's search engines," in Proc. IEEE Int. Conf. Semantic Comput., 2009, pp. 184-191.
30. N. Kristof, "Boycott Microsoft Bing," The New York Times, Nov. 20, 2009.
31. S. Denyer, "Microsoft denies censoring results in the 'uncensored' version of Chinese-language Bing," The Washington Post, Feb. 12, 2014.
32. P. Carsten, "Microsoft denies global censorship of China-related searches," Reuters, Feb. 12, 2014.
33. A. Sohn. (2009, Nov. 20). Committed to comprehensive results [Online]. Available: http://bit.ly/6CD49e
34. S. Morasca, D. Taibi, and D. Tosi, " T-DOC: Atool for the automatic generation of testing documentation for OSS products," in Open Source Software: New Horizons (series. IFIP Advances in Information and Communication Technology), P. Agerfalk, C. Boldyreff, J. M. González-Barahona, G. R. Madey, and J. Noll, Eds. Berlin, Germany: Springer-Verlag, 2010, vol. 319, pp. 200-213.
35. Quora. Will.es domain hurt my search results [Online]. Available: http://www.quora.com/Will-es-domain-hurt-my-search-results, 2012.
36. Y. Freund, R. Iyer, R. E. Schapire, and Y. Singer, "An efficient boosting algorithm for combining preferences," J. Mach. Learn. Res., vol. 4, pp. 933-969, 2003.
37. M. A. Qader, S. Rahman, and M. Hasan, "Structured web search in small domain," in Proc. 16th Int. Conf. Comput. Inf. Technol., 2013, pp. 426-431.
38. S. Ieong, N. Mishra, E. Sadikov, and L. Zhang, "Domain bias in web search," in Proc. 5th ACM Int. Conf. Web Search Data Mining, 2012, pp. 413-422.
39. J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, MI, USA: Hillsdale, 1988.
40. T. R. Levine and C. R. Hullett, "Eta squared, partial eta squared, and misreporting of effect size in communication research," Human Commun. Res., vol. 28, no. 4, pp. 612-625, 2002.
41. A. Field, Discovering Statistics Using IBM SPSS Statistics, 4th ed. Newbury Park, CA, USA: Sage, 2013.
42. J. Cohen, "Statistical power analysis," Current Directions Psychol. Sci., vol. 1, no. 3, pp. 98-101, 1992.
43. A. Alasiry, M. Levene, and A. Poulovassilis, "Extraction and evaluation of candidate named entities in search engine queries," in Proc. 13th Int. Conf. Web Inf. Syst. Eng., 2012, pp. 483-496.
44. J. R. Smarr, "Categorization by character-level models: Exploiting the sound symbolism of proper names," Master's thesis, The Symbolic Systems Program, Stanford Univ., Stanford, CA, USA, 2003
45. "Systems and Software Engineering - Systems and Software Quality Requirements and Evaluation (SQuaRE) - System and Software Quality Models," ISO/IEC 25010:2011, 2011.
46. T. Y. Chen and R. Merkel, "An upper bound on software testing effectiveness," ACM Trans. Softw. Eng. Meth., vol. 17, no. 3, pp. 16:1-16:27, 2008.
47. A. Arcuri, M. Z. Iqbal, and L. Briand, "Random testing: Theoretical results and practical implications," IEEE Trans. Softw Eng., vol. 38, no. 2, pp. 258-277, Mar./Apr. 2012.
48. J. Guo, G. Xu, X. Cheng, and H. Li, "Named entity recognition in query," in Proc. 32nd Int. ACM SIGIR Conf. Res. Develop. Inf. Retrieval, 2009, pp. 267-274.
49. G. Rossi, O. Pastor, D. Schwabe, and L. Olsina, Web Engineering: Modelling and Implementing Web Applications. New York, NY, USA: Springer-Verlag, 2008.
50. D. Carmel and E. Yom-Tov, Estimating the Query Difficulty for Information Retrieval. San Mateo, CA, USA: Morgan, 2010.
51. S. Wagner, Software Product Quality Control. New York, NY USA: Springer-Verlag, 2013.
52. IEEE Standards Association, IEEE Standard - Adoption of ISO/IEC 15026-2:2011, Systems and Software Engineering - Systems and Software Assurance - Part 2: Assurance Case, IEEE Std 15026-2TM2011, 2011.
53. IEEE Project Number: P25010. Standard - Adoption of ISO/IEC 25010-2010, Systems and Software Engineering - Systems and Software Quality Requirements and Evaluation, (SQuaRE) - System and Software Quality Models, 2011.