Automated functional testing of online search services

Software Testing Verification and Reliability

Volumn 22, Issue 4, 2012, Pages 221-243

  Zhou, Zhi Quan ^a   Zhang, Shujia ^a   Hagenbuchner, Markus ^a   Tse, T H ^b   Kuo, Fei Ching ^c   Chen, T Y ^c  

^a UNIVERSITY OF WOLLONGONG   (Australia)

^b UNIVERSITY OF HONG KONG   (Hong Kong)

^c SWINBURNE UNIVERSITY OF TECHNOLOGY   (Australia)

Author keywords

Internet search; metamorphic testing; software testing; verification and validation

Indexed keywords

AUTOMATED FUNCTIONAL TESTING; INTERNET SEARCHES; METAMORPHIC TESTING; ONLINE SEARCH; SEARCH SERVICES; SERVICE QUALITY; VERIFICATION AND VALIDATION;

SEARCH ENGINES; SOFTWARE TESTING;

INTERNET;

EID: 84862066626     PISSN: 09600833     EISSN: 10991689     Source Type: Journal    
DOI: 10.1002/stvr.437     Document Type: Conference Paper

References (32)

1. Patterson AL,. Multiple index based information retrieval system. United States Patent Application 20060106792, 2006. Available at: [22 May 2010 ].
2. Kc M,. Building a prototype for quality information retrieval from the World Wide Web. PhD Thesis, Faculty of Informatics, University of Wollongong, Australia, 2009. Available at: [22 May 2010 ].
3. Landoni M, Bell S,. Information retrieval techniques for evaluating search engines: A critical overview. Aslib Proceedings: New Information Perspectives 2000; 52 (3): 124-129.
4. Zhao H, Meng W, Yu C,. Automatic extraction of dynamic record sections from search engine result pages. Proceedings of the 32nd International Conference on Very Large Data Bases (VLDB 2006), VLDB Endowment, 2006; 989-1000.
5. Chu H, Rosenthal M,. Search engines for the World Wide Web: A comparative study and evaluation methodology. Proceedings of the ASIS 1996 Annual Conference, 1996. Available at: [22 May 2010 ].
6. Su LT,. A comprehensive and systematic model of user evaluation of Web search engines: I. Theory and background. Journal of the American Society for Information Science and Technology 2003; 54 (13): 1175-1192.
7. Clarke SJ, Willett P,. Estimating the recall performance of Web search engines. Aslib Proceedings: New Information Perspectives 1997; 49 (7): 184-189. (Pubitemid 127712809)
8. Weyuker EJ,. On testing non-testable programs. The Computer Journal 1982; 25 (4): 465-470.
9. Buckley C, Dimmick D, Soboroff I, Voorhees E,. Bias and the limits of pooling. Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006). ACM Press: New York, NY, 2006; 619-620.
10. Carterette B, Allan J, Sitaraman R,. Minimal test collections for retrieval evaluation. Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006). ACM Press: New York, NY, 2006; 268-275.
11. Järvelin K, Kekäläinen J,. IR evaluation methods for retrieving highly relevant documents. Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2000). ACM Press: New York, NY, 2000; 41-48.
12. Järvelin K, Kekäläinen J,. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems 2002; 20 (4): 422-446. (Pubitemid 44642296)
13. Soboroff I,. Dynamic test collections: Measuring search effectiveness on the live Web. Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2006). ACM Press: New York, NY, 2006; 276-283.
14. Soboroff I, Nicholas C, Cahan P,. Ranking retrieval systems without relevance judgments. Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2001). ACM Press: New York, NY, 2001; 66-73. (Pubitemid 32951057)
15. Chen TY, Tse TH, Zhou ZQ,. Semi-proving: An integrated method for program proving, testing, and debugging. IEEE Transactions on Software Engineering 2010; DOI: 10.1109/TSE.2010.23.
16. Kilgarriff A,. Googleology is bad science. Computational Linguistics 2007; 33 (1): 147-151.
17. Grefenstette G,. The WWW as a resource for example-based MT tasks. Proceedings of the 21st International Conference on Translating and the Computer. Aslib Publications: London, U.K., 1999.
18. Keller F, Lapata M,. Using the Web to obtain frequencies for unseen bigrams. Computational Linguistics 2003; 29 (3): 459-484. (Pubitemid 37445781)
19. Lapata M, Keller F,. Web-based models for natural language processing. ACM Transactions on Speech and Language Processing 2005; 2 (1):Article No. 3.
20. Nakov P, Hearst M,. Search engine statistics beyond the n-gram: Application to noun compound bracketing. Proceedings of the 9th Conference on Computational Natural Language Learning (CONLL 2005). Association for Computational Linguistics: Morristown, NJ, 2005; 17-24.
21. Turney PD,. Mining the Web for synonyms: PMI-IR versus LSA on TOEFL. Proceedings of the 12th European Conference on Machine Learning (EMCL 2001). Springer: London, U.K., 2001; 491-502. (Pubitemid 33331094)
22. Pezzè M, Young M,. Software Testing and Analysis: Process, Principles, and Techniques. Wiley: New York, NY, 2008.
23. Ammann P, Offutt J,. Introduction to Software Testing. Cambridge University Press: New York, NY, 2008.
24. Zhou ZQ, Tse TH, Kuo F-C, Chen TY,. Automated functional testing of Web search engines in the absence of an oracle. Technical Report TR-2007-06, Department of Computer Science, The University of Hong Kong: Pokfulam, Hong Kong, 2007.
25. Available at: [18 April 2009 ].
26. Page L, Brin S, Motwani R, Winograd T,. The PageRank citation ranking: Bringing order to the Web. Stanford InfoLab Publication 1999-1966. Stanford University: Palo Alto, CA, 1999. Available at: [22 May 2010].
27. Tsoi AC, Morini G, Scarselli F, Hagenbuchner M, Maggini M,. Adaptive ranking of Web pages. Proceedings of the 12th International Conference on World Wide Web (WWW 2003). ACM Press: New York, NY, 2003; 356-365.
28. Kc M, Hagenbuchner M, Tsoi AC,. Quality information retrieval for the World Wide Web. 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT 2008). IEEE Computer Society Press: Los Alamitos, CA, 2008; 655-661.
29. Yu H, Vahdat A,. The costs and limits of availability for replicated services. Proceedings of the 18th ACM Symposium on Operating Systems Principles (SOSP'01). ACM Press: New York, NY, 2001; 29-42. (Pubitemid 33612603)
30. Yu H, Vahdat A,. Minimal replication cost for availability. Proceedings of the 21st Annual Symposium on Principles of Distributed Computing (PODC'02). ACM Press: New York, NY, 2002; 98-107.
31. Tsoi AC, Hagenbuchner M, Chau R, Lee V,. Unsupervised and supervised learning of graph domains. Innovations in Neural Information Paradigms and Applications (Studies in Computational Intelligence, vol. 247). Springer: Berlin, Germany, 2009; 43-65.
32. Yong SL, Hagenbuchner M, Tsoi AC,. Ranking Web pages using machine learning approaches. Proceedings of the 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT 2008) IEEE Computer Society Press: Los Alamitos, CA, 2008; 677-680.