Do user preferences and evaluation measures line up?

SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2010, Pages 555-562

  Sanderson, Mark ^a   Paramita, Monica Lestari ^a   Clough, Paul ^a   Kanoulas, Evangelos ^a  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Evaluation measures; Mechanical turk; User experiment

Indexed keywords

EFFECTIVENESS MEASURE; EVALUATION MEASURES; LARGE POPULATION; MEASURES OF EFFECTIVENESS; MECHANICAL TURKS; NEW RESULTS; RETRIEVAL SYSTEMS; SEARCH TASKS; TEST COLLECTION; USER EXPERIMENT; WEB RETRIEVAL;

EXPERIMENTS; SEARCH ENGINES;

INFORMATION RETRIEVAL;

EID: 77956037058     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1835449.1835542     Document Type: Conference Paper

References (29)

1. Agrawal, R., Gollapudi, S., Halverson, A., & Ieong, S. 2009. Diversifying search results. ACM WSDM, 5-14.
2. Al-Maskari, A., Sanderson, M. & Clough, P., 2007. The relationship between IR effectiveness measures and user satisfaction. ACM SIGIR, 773-774.
3. Al-Maskari, A., Sanderson, M., Clough, P., & Airio, E. 2008. The good and the bad system: does the test collection predict users' effectiveness? ACM SIGIR, 59-66.
4. Allan, J., Carterette, B., & Lewis, J. 2005. When will information retrieval be "good enough"? ACM SIGIR, 433-440.
5. Alonso, O., Rose, D. E., & Stewart, B. 2008. Crowdsourcing for relevance evaluation. SIGIR Forum 42, 2, 9-15.
6. Alonso, O. & Mizzaro, S., 2009. Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation. 15-16.
7. Arni, T., Tang, J., Sanderson, M., Clough, P. 2008. Creating a test collection to evaluate diversity in image retrieval. Workshop on Beyond Binary Relevance, SIGIR 2008.
8. Barry, C. L. 1994. User-defined relevance criteria: an exploratory study. J. Am. Soc. Inf. Sci. 45, 3, 149-159.
9. Broder, A. 2002. A taxonomy of web search. SIGIR Forum 36(2) 3-10.
10. th WWW Conf, 1-10
11. Chapelle, O., Metzler, D., Zhang, Y., Grinspan, P., 2009. Expected reciprocal rank for graded relevance, ACM CIKM, 621-630
12. Clarke, C., Kolla, M., Cormack, G., Vechtomova, O., Ashkan, A., Büttcher, S., MacKinnon, I. 2008. Novelty & diversity in information retrieval evaluation. ACM SIGIR, 659-666.
13. Clarke, C., Kolla, M., & Vechtomova, O. 2009. An Effectiveness Measure for Ambiguous and Underspecified Queries. Advances in Information Retrieval Theory, 188-199.
14. Clarke, C., Craswell, N., and Soboroff, I. 2009. Preliminary Report on the TREC 2009 Web Track. TREC 2009 Notebook.
15. Hersh, W.R. et al., 2002. Factors associated with success in searching MEDLINE and applying evidence to answer clinical questions, Am Med Inform Assoc.
16. Hersh, W., Turpin, A., Price, S., Chan, B., Kramer, D., Sacherek, L., & Olson, D. 2000. Do batch and user evaluations give the same results? ACM SIGIR, 17-24.
17. Huuskonen, S. & Vakkari, P. 2008. Students' search process and outcome in Medline in writing an essay for a class on evidence-based medicine. Journal of Documentation, 64(2), 287-303.
18. Joachims, T., 2002. Evaluating retrieval performance using click through data. Workshop on Mathematical/Formal Methods in IR, 12-15.
19. Radlinski, F., Kurup, M., Joachims, T. 2008. How does click through data reflect retrieval quality? ACM CIKM, 43-52.
20. Robertson, S. 2006. On GMAP: and other transformations, ACM CIKM, 78-83
21. Smith, C.L. & Kantor, P.B., 2008. User adaptation: good results from poor systems. ACM SIGIR, 147-154.
22. Su, L.T., 1992. Evaluation measures for interactive information retrieval. IP&M, 28(4), 503-516.
23. Tagliacozzo, R., 1977. Estimating the satisfaction of information users. Bulletin of the Medical Library Association, 65(2), 243-249.
24. Thomas, D.R. 2006. A General Inductive Approach for Analyzing Qualitative Evaluation Data. American Journal of Evaluation, 27(2), 237-246
25. Thomas, P. & Hawking, D., 2006. Evaluation by comparing result sets in context. ACM CIKM, 94-101.
26. Turpin, A.H., Hersh, W. 2001. Why batch and user evaluations do not give the same results. ACM SIGIR, 225-231.
27. Turpin, A. & Scholer, F. 2006. User performance versus precision measures for simple search tasks. ACM SIGIR, 11-18.
28. Yilmaz, E., Aslam, J., Robertson, S. 2008. A new rank correlation coefficient for information retrieval. ACM SIGIR, 587-594.
29. Zhai, C.X., Cohen, W.W. & Lafferty, J., 2003. Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. ACM SIGIR, 10-17.