Relevance and effort: An analysis of document utility

CIKM 2014 - Proceedings of the 2014 ACM International Conference on Information and Knowledge Management

Volumn , Issue , 2014, Pages 91-100

  Yilmaz, Emine ^a   Verma, Manisha ^a   Craswell, Nick ^b   Radlinski, Filip ^b   Bailey, Peter ^b  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

^b MICROSOFT   (United States)

Author keywords

Effort; Relevance assessments; User behavior

Indexed keywords

KNOWLEDGE MANAGEMENT;

COMPLEX DOCUMENTS; DOCUMENT UTILITY; DWELL TIME; EFFORT; RELEVANCE ASSESSMENTS; RELEVANCE JUDGMENT; TEST COLLECTION; USER BEHAVIORS;

BEHAVIORAL RESEARCH;

EID: 84928709333     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2661829.2661953     Document Type: Conference Paper

References (43)

1. A. Al-Maskari, M. Sanderson, P. Clough, and E. Airio. The good and the bad system: Does the test collection predict users' effectiveness? In Proc. SIGIR, Singapore, 2008.
2. J. Allan, B. Carterette, and J. Lewis. When will information retrieval be "good enough"? In Proc. SIGIR, Salvador, Brazil, 2005.
3. L. Azzopardi. The economics in interactive information retrieval. In Proc. SIGIR, pages 15-24, New York, NY, USA, 2011. ACM.
4. L. Azzopardi. Modelling interaction with economic models of search. In Proc. SIGIR, pages 3-12, New York, NY, USA, 2014. ACM.
5. P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A. P. de Vries, and E. Yilmaz. Relevance assessment: Are judges exchangeable and does it matter. In Proc. SIGIR, pages 667-674, Singapore, 2008.
6. J. C. Brown and M. Eskenazi. Student, text and curriculum modeling for reader-specific document retrieval.
7. G. Buscher, A. Dengel, and L. van Elst. Query expansion using gaze-based feedback on the subdocument level. In Proc. SIGIR, pages 387-394, Singapore, 2008. ACM.
8. G. Buscher, L. van Elst, and A. Dengel. Segment-level display time as implicit feedback: A comparison to eye tracking. In Proc. SIGIR, Boston, USA, 2009.
9. B. Carterette, P. D. Clough, E. Kanoulas, and M. Sanderson. Report on the ecir 2011 workshop on information retrieval over query sessions. SIGIR Forum, 45(2): 76-80, Jan. 2012.
10. B. Carterette, E. Kanoulas, and E. Yilmaz. Simulating simple user behavior for system effectiveness evaluation. In Proc. CIKM, Glasgow, UK, 2011.
11. B. Carterette, E. Kanoulas, and E. Yilmaz. Incorporating variability in user behavior into systems based evaluation. In Proc. CIKM, Maui, USA, 2012.
12. B. Carterette and I. Soboroff. The effect of assessor error on ir system evaluation. In Proc. SIGIR, Geneva, Switzerland, 2010.
13. P. Chandar, W. Webber, and B. Carterette. Document features predicting assessor disagreement. In Proc. SIGIR, Dublin, Ireland, 2013.
14. M. Claypool, P. Le, M. Wased, and D. Brown. Implicit interest indicators. In Proceedings of the 6th international conference on Intelligent user interfaces, pages 33-40. ACM, 2001.
15. W. S. Cooper. On selecting a measure of retrieval effectiveness, part i. JASIST, 24(2): 87-100, 1973.
16. A. P. De Vries, G. Kazai, and M. Lalmas. Tolerance to irrelevance: A user-effort oriented evaluation of retrieval systems without predefined retrieval unit. RIAO Conference Proceedings, pages 463-473, 2004.
17. G. Dupret and C. Liao. A model to estimate intrinsic document relevance from the clickthrough logs of a web search engine. In Proc. WSDM, New York, USA, 2010.
18. G. Dupret and B. Piwowarski. A user browsing model to predict search engine click data from past observations. In Proc. SIGIR, Singapore, 2008.
19. S. Fox, K. Karnawat, M. Mydland, S. Dumais, and T. White. Evaluating implicit measures to improve web search. ACM Trans. Inf. Syst., 23(2): 147-168, Apr. 2005.
20. S. Fox, K. Karnawat, M. Mydland, S. Dumais, and T. White. Evaluating implicit measures to improve web search. ACM Trans. Inf. Syst., 23(2): 147-168, Apr. 2005.
21. Q. Guo and E. Agichtein. Beyond dwell time: Estimating document relevance from cursor movements and other post-click searcher behavior. In Proc. WWW, Lyon, France, 2012.
22. W. Hersh, A. Turpin, S. Price, B. Chan, D. Kramer, L. Sacherek, and D. Olson. Do batch and user evaluations give the same results? In Proc. SIGIR, Athens, Greece, 2000.
23. T. Joachims, L. A. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Trans. Inf. Syst., 25(2), 2007.
24. G. Kazai, N. Craswell, E. Yilmaz, and S. Tahaghoghi. An analysis of systematic judging errors in information retrieval. In Proc. CIKM, Maui, USA, 2012.
25. D. Kelly and N. J. Belkin. Display time as implicit feedback: Understanding task effects. In Proc. SIGIR, Sheffield, UK, 2004.
26. D. Kelly and J. Teevan. Implicit feedback for inferring user preference: a bibliography. In ACM SIGIR Forum, Volume 37: 2, pages 18-28. ACM, 2003.
27. D. Kelly and J. Teevan. Implicit feedback for inferring user preference: A bibliography. SIGIR Forum, 37(2): 18-28, Sept. 2003.
28. Y. Kim, A. Hassan, R. W. White, and I. Zitouni. Modeling dwell time to predict click-level satisfaction. In WSDM, 2014.
29. J. P. Kincaid, R. P. Fishburne Jr, R. L. Rogers, and B. S. Chissom. Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. Technical report, DTIC Document, 1975.
30. M. Morita and Y. Shinoda. Information filtering based on user behavior analysis and best match text retrieval. In Proc. SIGIR, Dublin, Ireland, 1994.
31. F. Radlinski and T. Joachims. Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. In Proc. AAAI, 2006.
32. T. Saracevic. Relevance: A review of the literature and a framework for thinking on the notion in information science. JASIST, pages 321-343, 1975.
33. T. Saracevic. Relevance: A review of the literature and a framework for thinking on the notion in information science. part ii: nature and manifestations of relevance. JASIST, 58(13): 1915-1933, 2007.
34. F. Scholer, A. Turpin, and M. Sanderson. Quantifying test collection quality based on the consistency of relevance judgements. In Proc. SIGIR, Beijing, China, 2011.
35. D. Sculley, R. G. Malkin, S. Basu, and R. J. Bayardo. Predicting bounce rates in sponsored search advertisements. In Proc. SIGKDD, Paris, France, 2009.
36. M. D. Smucker and C. L. Clarke. Time-based calibration of effectiveness measures. In Proc. SIGIR, Portland, USA, 2012.
37. A. H. Turpin and W. Hersh. Why batch and user evaluations do not give the same results? In Proc. SIGIR, New Orleans, USA, 2001.
38. R. Villa and M. Halvey. Is relevance hard work?: Evaluating the effort of making relevant assessments. In Proc. SIGIR, pages 765-768, 2013.
39. E. M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Information processing & management, 36(5): 697-716, 2000.
40. K. Wang, T. Walker, and Z. Zheng. Pskip: Estimating relevance ranking quality from web search clickthrough data. In Proc. SIGKDD, pages 1355-1364, Paris, France, 2009.
41. W. Webber. Re-examining the effectiveness of manual review. In Proc. SIGIR Information Retrieval for E-Discovery Workshop, page 2, 2011.
42. W. Webber, B. Toth, and M. Desamito. Effect of written instructions on assessor agreement. In Proc. SIGIR, Portland, USA, 2012.
43. R. W. White and D. Kelly. A study on the effects of personalization and task information on implicit feedback performance. In Proc. CIKM, Arlington, USA, 2006.