Web search solved?: All result rankings the same?

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2010, Pages 529-538

  Zaragoza, Hugo ^a   Cambazoglu, B Barla ^a   Baeza Yates, Ricardo ^a  

^a YAHOO RESEARCH   (Spain)

Author keywords

Editorial judgments; Relevance; Web search evaluation

Indexed keywords

EDITORIAL JUDGMENTS; EMPIRICAL RESULTS; LOWER AND UPPER BOUNDS; RELEVANCE; RELEVANCE MEASURE; US MARKETS; WEB SEARCH QUERIES; WEB SEARCHES;

INFORMATION RETRIEVAL; KNOWLEDGE MANAGEMENT; SEARCH ENGINES;

WORLD WIDE WEB;

EID: 78651277510     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1871437.1871507     Document Type: Conference Paper

References (54)

1. A. Al-Maskari, M. Sanderson, and P. Clough. The relationship between IR effectiveness measures and user satisfaction. In Proc. 30th Annual Int'l ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 773-774, 2007.
2. J. Allan, B. Carterette, and J. Lewis. When will information retrieval be "good enough"? In Proc. 28th Annual Int'l ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 433-440, 2005.
3. P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A. de Vries, and E. Yilmaz. Relevance assessment: are judges exchangeable and does it matter. In Proc. 31st Annual Int'l ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 667-674, 2008.
4. J. Bar-Ilan. Comparing rankings of search results on the Web. Information Processing & Management, 41(6):1511-1519, 2005.
5. J. Bar-Ilan, K. Keenoy, E. Yaari, and M. Levene. User rankings of search engine results. Journal of the American Society for Information Science and Technology, 58(9): 1254-1266, 2007.
6. J. Bar-Ilan, M. Mat-Hassan, and M. Levene. Methods for comparing rankings of search engine results. Computer Networks, 50(10):1448-1463, 2006.
7. M. M. Beg. A subjective measure of web search quality. Information Sciences, 169(3-4):365-381, 2005.
8. K. Bharat and A. Broder. A technique for measuring the relative size and overlap of public web search engines. Computer Networks and ISDN Systems, 30(1-7):379-388, 1998.
9. P. Borlund and P. Ingwersen. Measures of relative relevance and ranked half-life: performance indicators for interactive IR. In Proc. 21st Annual Int'l ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 324-331, 1998.
10. A. Broder. A taxonomy of web search. ACM SIGIR Forum, 36(2):3-10, 2002.
11. C. Buckley and E. Voorhees. Retrieval system evaluation. In TREC: Experiment and Evaluation in Information Retrieval, chapter 3, pages 53-75. The MIT Press, 2005.
12. F. Can, R. Nuray, and A. B. Sevdik. Automatic performance evaluation of web search engines. Information Processing & Management, 40(3):495-514, 2004.
13. B. Carterette. On rank correlation and the distance between rankings. In Proc. 32nd Int'l ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 436-443, 2009.
14. O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In Proc. 18th ACM Conf. on Information and Knowledge Management, pages 621-630, 2009.
15. A. Chowdhury and I. Soboroff. Automatic evaluation of World Wide Web search services. In Proc. 25th Annual Int'l ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 421-422, 2002.
16. H. Chu and M. Rosenthal. Search engines for the World Wide Web: a comparative study and evaluation methodology. In Proc. 59th Annual Meeting of the American Society for Information Science, pages 127-135, 1996.
17. W. Cooper. Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. American Documentation, 19(1):30-41, 1968.
18. W. Ding and G. Marchionini. A comparative study of web search service performance. In Proc. 59th Annual Meeting of the American Society for Information Science, pages 136-142, 1996.
19. R. Fagin, R. Kumar, M. Mahdian, D. Sivakumar, and E. Vee. Comparing partial rankings. SIAM Journal on Discrete Mathematics, 20(3):628-648, 2006.
20. R. Fagin, R. Kumar, and D. Sivakumar. Comparing top k lists. In Proc. 14th Annual AGM-SIAM Symp. on Discrete Algorithms, pages 28-36, 2003.
21. D. Gayo-Avello. A survey on session detection methods in query logs and a proposal for future evaluation. Information Sciences, 179(12): 1822-1843, 2009.
22. M. Gordon and P. Pathak. Finding information on the World Wide Web: the retrieval effectiveness of search engines. Information Processing & Management, 35(2):141-180, 1999.
23. D. Hawking, N. Craswell, P. Bailey, and K. Griffihs. Measuring search engine quality. Information Retrieval, 4(1):33-59, 2001.
24. N. Hochstotter and D. Lewandowski. What users see - structures in search engine results pages. Information Sciences, 179(12): 1796-1812, 2009.
25. B. J. Jansen, A. Spink, C. Blakely, and S. Koshman. Defining a session on web search engines. Journal of the American Society for Information Science and Technology, 58(6):862-871, 2007.
26. K. Jarvelin and J. Kekalainen. IR evaluation methods for retrieving highly relevant documents. In Proc. 23rd Annual Int'l ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 41-48, 2000.
27. K. Jarvelin and J. Kekalainen. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):422-446, 2002.
28. E. C. Jensen, S. M. Beitzel, O. Frieder, and A. Chowdhury. A framework for determining necessary query set sizes to evaluate web search effectiveness. In Special Interest Tracks and Posters of the 14th Int'l Conf. on World Wide Web, pages 1176-1177, 2005.
29. M. Kendall. A new measure of rank correlation. Biometrika, 30(1-2):81-93, 1938.
30. Y. S. Kim and B. H. Kang. Coverage and timeliness analysis of search engines with webpage monitoring results. In Proc. 8th Int'l Conf. on Web Information Systems Engineering, pages 361-372, 2007.
31. R. Kumar and S. Vassilvitskii. Generalized distances between rankings. In Proc. 19th Int'l Conf. on World Wide Web, pages 571-580, 2010.
32. D. Lewandowski. The retrieval effectiveness of web search engines: considering results descriptions. Journal of Documentation, 64(6):915-937, 2008.
33. D. Lewandowski. A three-year study on the freshness of web search engine databases. Journal of Information Science, 34(6):817-831, 2008.
34. D. Lewandowski. The retrieval effectiveness of search engines on navigational queries. ASLIB Proceedings, 62, 2010. In press.
35. L. Li and Y. Shang. A new method for automatic performance comparison of search engines. World Wide Web, 3(4):241-247, 2000.
36. B. Liu. Personal evaluations of search engines: Google, Yahoo! and Live (MSN), http://cs.uic.edu/~liub/searchEval/2006-2007.html, 2007.
37. R. Losee. Text Retrieval and Filtering: Analytic Models of Performance. Kluwer Academic, 1998.
38. Q. Mei and K. Church. Entropy of search logs: how hard is search? With personalization? With backoff? In Proc. Int'l Conf. on Web Search and Data Mining, pages 45-54, 2008.
39. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems, 27(1): 1-27, 2008.
40. A. Mowshowitz and A. Kawaguchi. Measuring search engine bias. Information Processing & Management, 41(5):1193-1205, 2005.
41. R. Nuray and F. Can. Automatic ranking of information retrieval systems using data fusion. Information Processing & Management, 42(3):595-614, 2006.
42. S. Pollack. Measures for the comparison of information retrieval systems. American Documentation, 19(4):387-397, 1968.
43. H. Qin and P. P. Rau. Relevance measurement on Chinese search results. In Proc. 12th Int'l Conf. on Human-Computer Interaction, pages 981-988, 2007.
44. C. Saraiva, E. de Moura, N. Ziviani, W. Meira, R. Fonseca, and B. Ribeiro-Neto. Rank-preserving two-level caching for scalable search engines. In Proc. 24th Annual Int'l ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 51-58, 2001.
45. Y. Shang and L. Li. Precision evaluation of search engines. World Wide Web, 5(2):159-173, 2002.
46. C. Smith and P. Kantor. User adaptation: good results from poor systems. In Proc. 31th Annual Int'l ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 147-154, 2008.
47. C. Spearman. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72-101, 1904.
48. A. Spink, B. J. Jansen, C. Blakely, and S. Koshman. A study of results overlap and uniqueness among major web search engines. Information Processing & Management, 42(5):1379-1391, 2006.
49. M. Taylor, H. Zaragoza, N. Craswell, S. Robertson, and C. Burges. Optimisation methods for ranking functions with multiple parameters. In Proc. 15th ACM Conf. on Information and Knowledge Management, pages 585-593, 2006.
50. M. Thelwall. Quantitative comparisons of search engine results. Journal of the American Society for Information Science and Technology, 59(11): 1702-1710, 2008.
51. S. Tongchim, V. Sornlertlamvanich, and H. Isahara. Improving search performance: a lesson learned from evaluating search engines using Thai queries. IEICE Transactions on Information and Systems, E90-D(10):1557-1564, 2007.
52. C. J. van Rijsbergen. Information Retrieval, 2nd ed. Butterworths, 1979.
53. L. Vaughan and M. Thelwall. Search engine coverage bias: evidence and possible causes. Information Processing & Management, 40(4):693-707, 2004.
54. E. Yilmaz, J. A. Aslam, and S. Robertson. A new rank correlation coefficient for information retrieval. In Proc. 31st Annual Int'l ACM SIGIR Conf. on Research and Development in Information Retrieval, pages 587-594, 2008.