Evaluating search systems using result page context

IIiX 2010 - Proceedings of the 2010 Information Interaction in Context Symposium

Volumn , Issue , 2010, Pages 105-114

  Bailey, Peter ^a   Craswell, Nick ^a   White, Ryen W ^a   Chen, Liwei ^a   Satyanarayana, Ashwin ^a   Tahaghoghi, S M M ^a  

^a MICROSOFT   (United States)

Author keywords

Evaluation; Measurement; Web search relevance

Indexed keywords

CRANFIELD; EVALUATION; EVALUATION METHOD; INFORMATION REDUNDANCIES; MULTIPLE SEARCH; RELEVANCE ASSESSMENTS; RELEVANCE JUDGMENT; SEARCH SYSTEM; USER EXPERIENCE; WEB SEARCHES;

INFORMATION RETRIEVAL; SAFETY DEVICES; SEARCH ENGINES;

WORLD WIDE WEB;

EID: 77957955135     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1840784.1840801     Document Type: Conference Paper

References (45)

1. A. Al-Maskari, M. Sanderson, P. Clough, and E. Airio. (2008). The good and the bad system: does the test collec-tion predict users' effectiveness? Proc. SIGIR.
2. J. Allan, B. Carterette, and J. Lewis. (2005). When will in-formation retrieval be "good enough"? Proc. SIGIR.
3. P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A.P. de Vries, and E. Yilmaz. (2008). Relevance assessment: are judges exchangeable and does it matter. Proc. SIGIR.
4. P. Bailey, N. Craswell, R.W. White, L. Chen, A. Satyana-rayana, and S.M.M. Tahaghoghi. (2010). Evaluating whole-page relevance. Proc. SIGIR, (poster paper).
5. P. Bailey, P. Thomas, and D. Hawking. (2007). Does brand-name influence perceived search result quality? Yahoo!, Google, and WebKumara. Proc. ADCS.
6. P. Borlund. (2000). Experimental components for the evalu-ation of interactive information retrieval systems. Journal of Documentation, 53(3): 71-90.
7. G. Buscher, S. Dumais, and E. Cutrell. (2010). The good, the bad, and the random: An eye-tracking study of ad quality in web search. Proc. SIGIR.
8. B. Carterette, P.N. Bennett, D.M. Chickering, and S. Du-mais. (2008). Here or there: preference judgments for rele-vance. Proc. ECIR, 16-27.
9. R. Chandrasekar, M.R. Scott, D. Slawson, A.R.D. Rajan, and D. Makoski. (2008). Measuring search experience satis-faction using explicit context-aware feedback. Proc. Work-shop on Human-Computer Interaction and Information Re-trieval.
10. C.L.A. Clarke, E. Agichtein, S. Dumais, and R.W. White. (2007). The influence of caption features on clickthrough patterns in web search. Proc. SIGIR.
11. C.L.A. Clarke, N. Craswell, and I. Soboroff. (2009). Over-view of the TREC 2009 web track. Proc. TREC.
12. C.W. Cleverdon. (1960). ASLIB Cranfield research project on the comparative efficiency of indexing systems. ASLIB Proceedings, XII, 421-431.
13. J. Cohen. (1960). A coefficient for agreement for nominal scales. Education and Psych. Measurement, 20: 37-46.
14. N. Craswell, O. Zoeter, M. Taylor, and B. Ramsey. (2008). An experimental comparison of click position-bias models. Proc. WSDM, 87-94.
15. E. Cutrell and Z. Guan. (2007). What are you looking for?: an eye-tracking study of information usage in web search. Proc. CHI, 407-416.
16. D. Downey, S. Dumais, D. Liebling, and E. Horvitz. (2008). Understanding the relationship between searchers? queries and information goals. Proc. CIKM, 449-458.
17. S. Dumais and N. Belkin. (2005). The TREC interactive tracks: putting the user into search. TREC: Experimentation and Evaluation in Information Retrieval. MIT Press.
18. J. Fleiss. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5): 378-382.
19. Z. Guan and E. Cutrell. (2007). An eye tracking study of the effect of target rank on web search. Proc. CHI, 417-420.
20. D. Hawking and N. Craswell. (2005). The very large collec-tion and web tracks. TREC: Experimentation and Evaluation in Information Retrieval, 199-232. MIT Press.
21. D. Hawking, N. Craswell, P. Bailey, and K. Griffiths. (2001). Measuring search engine quality. Information Retrieval, 4(1): 33-59.
22. D. Hawking, N. Craswell, P. Thistlewaite, and D. Harman. (1999). Results and challenges in Web search evaluation. Proc. WWW, 1321-1330.
23. W. Hersh. (2002). TREC 2002 interactive track report. Proc. TREC.
24. P. Ingwersen and K. Järvelin. (2005). The Turn: Integration of Information Seeking and Retrieval in Context. Springer.
25. B.J. Jansen and U. Pooch. (2001). Web user studies: a re-view and framework for future work. JASIST, 52(3): 235-246.
26. B.J. Jansen, M. Zhang, and C.D. Schultz. (2009). Brand and its effect on user perception of search engine performance. JASIST, 60(8): 1572-1595.
27. K. Järvelin and J. Kekäläinen. (2000). IR evaluation meth-ods for retrieving highly relevant documents. Proc. SIGIR.
28. D. Kelly, K. Gyllstrom, and E.W. Bailey. (2009). A compar-ison of query and term suggestion features for interactive searching. Proc SIGIR.
29. R. Kohavi, T. Crook, and R. Longbotham. (2009). Online experimentation at Microsoft. Proc. Workshop on Data Min-ing Case Studies and Practice Prize.
30. J. R. Landis and G. G. Koch (1977). The measurement of observer agreement for categorical data. Biometrics, 33: 159-174.
31. J. Li, S. Huffman, and A. Tokuda. (2009). Good abandon-ment in mobile and PC internet search. Proc. SIGIR.
32. M.B. Meyer and J.M. Booker. (2001). Eliciting and Analyz-ing Expert Judgment: A Practical Guide. ASA-SIAM.
33. J. Nielsen and K. Pernice. (2009). Eyetracking Web Usabil-ity. New Riders Press.
34. F. Radlinski, M. Kurup, and T. Joachims. (2008). Learning diverse rankings with multi-armed bandits? Proc. ICML.
35. F. Radlinski, R. Kleinberg, and T. Joachims. (2008). How does clickthrough data reflect retrieval quality? Proc. CIKM, 43-52.
36. F. Radlinski and T. Joachims. (2006). Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. Proc. AAAI, 1406-1412.
37. F. Radlinski and N. Craswell. (2010). Comparing the sensi-tivity of information retrieval metrics. Proc. SIGIR.
38. P.E. Shrout and J.L. Fliess (1979). Intraclass correlations: uses in assessing rater reliability. Psychological Bulletin, 86(2): 420-428.
39. P. Thomas and D. Hawking. (2006). Evaluation by compar-ing result sets in context. Proc. CIKM, 94-101.
40. A. Turpin, F. Scholer, K. Järvelin, M. Wu, and J.S. Culpepper. (2009). Including summaries in system evaluation. Proc. SIGIR, 508-515.
41. A. Turpin and F. Scholer. (2006). User performance versus precision measures for simple search tasks. Proc. SIGIR.
42. E. M. Voorhees. (2008). On test collections for adaptive information retrieval. Information Processing and Manage-ment, 44(6): 1879-1885.
43. E. M. Voorhees and D. Harman. (2005). TREC: Experiment and Evaluation in Information Retrieval. MIT Press.
44. K. Wang, T. Walker, and Z. Zheng. (2009). PSkip: estimat-ing relevance ranking quality from web search clickthrough data. Proc. KDD, 1355-1364.
45. R.W. White, M. Bilenko, and S. Cucerzan. (2007). Studying the use of popular destinations to enhance web search inter-action. Proc. SIGIR.