Evaluating the effort involved in relevance assessments for images

SIGIR 2014 - Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2014, Pages 887-890

  Halvey, Martin ^a   Villa, Robert ^b  

^a GLASGOW CALEDONIAN UNIVERSITY   (United Kingdom)

^b UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Assessment; Effort; Images; Judgment; Relevance

Indexed keywords

ASSESSMENT; EFFORT; IMAGES; JUDGMENT; RELEVANCE;

INFORMATION RETRIEVAL;

EID: 84904564284     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2600428.2600735     Document Type: Conference Paper

References (15)

1. Al-Harbi, A. L. and Smucker, M. D. User Expressions of Relevance Judgment Certainty. HCIR 2013.
2. Azzopardi, L., Kelly, D. and Brennan, K. How query cost affects search behavior. SIGIR 2013, 23-32.
3. Burford, B., Briggs, P. and Eakins, J. P. A taxonomy of the image: on the classification of content for image retrieval. Visual Communication, 2, 2 (2003), 123-161.
4. Grubinger, M. Analysis and evaluation of visual information systems performance. PhD Thesis 2007.
5. Grubinger, M., Clough, P., Hanbury, A. and Müller, H. Overview of the ImageCLEFphoto 2007 photographic retrieval task. CLEF 2007.
6. Hart, S. G. and Stavenland, L. E. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. .Human Mental Workload. Elsevier, 1988, 139-183.
7. Hauptmann, A. G., Lin, W., Yan, R., Yang, J. and Chen, M. Extreme video retrieval: joint maximization of human and computer performance. ACM Multimedia 2006, 385-394.
8. Hürst, W., Snoek, C. G., Spoel, W. and Tomin, M. Size matters! how thumbnail number, size, and motion influence mobile video retrieval. MMM 2011, 230-240.
9. Mizzaro, S. Relevance: The whole history. JASIS, 48, 9 (1997), 810-832.
10. Saracevic, T. RELEVANCE: A review of and a framework for the thinking on the notion in information science. JASIS, 26, 6 (1975), 321-343.
11. Smucker, M. D. and Clarke, C. L. A. Time-based calibration of effectiveness measures. SIGIR 2012, 95-104.
12. Sormunen, E. Liberal relevance criteria of TREC -: counting on negligible documents? SIGIR 2002, 324-330.
13. Spink, A., Greisdorf, H. and Bateman, J. From highly relevant to not relevant: examining different regions of relevance. Information Processing & Management, 34, 5 (1998), 599.
14. Torralba, A., Fergus, R. and Freeman, W. T. 80 million tiny images: A large data set for nonparametric object and scene recognition. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 30, 11 (2008), 1958-1970.
15. Villa, R. and Halvey, M. Is relevance hard work?: evaluating the effort of making relevant assessments. SIGIR 2013, 765-768.