Is relevance hard work? Evaluating the effort of making relevant assessments

SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2013, Pages 765-768

  Villa, Robert ^a   Halvey, Martin ^b  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

^b GLASGOW CALEDONIAN UNIVERSITY   (United Kingdom)

Author keywords

Relevance; User models; User studies

Indexed keywords

EMPIRICAL EVALUATIONS; HARD WORK; RELEVANCE; RELEVANCE JUDGMENT; RELEVANT DOCUMENTS; TEST COLLECTION; USER MODELS; USER STUDY;

INFORMATION RETRIEVAL;

EID: 84883081310     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2484028.2484150     Document Type: Conference Paper

References (16)

1. Mizzaro. S., 1997. Relevance: The whole history. Journal of the American Society for Info. Science, 48(9), 810-832.
2. Spink, A., Greisdorf, H., and Bateman, J. 1998. From highly relevant to not relevant: examining different regions of relevance. Inf. Process. Manage. 34, 5 (Sept 1998), 599-621.
3. Carterette, B., Allan, J. and Sitaraman, R. 2006. Minimal test collections for retrieval evaluation. SIGIR 2006, 268-275.
4. Tang R., Solomon P. 1998. Toward an understanding of the dynamics of relevance judgment: An analysis of one person's search behavior, Inf. Process. Manage, 34 (2-3), 237-256.
5. Sormunen, E. 2002. Liberal relevance criteria of TREC: counting on negligible documents? SIGIR 2002, 324-330.
6. Taylor, A. 2011. User relevance criteria choices and the information search process. IP & M. 48, 136-153.
7. Wang, J. 2011. Accuracy, agreement, speed, and perceived difficulty of users' relevance judgments for e-discovery. SIGIR 2011 Information Retrieval for E-Discovery (SIRE) Workshop, Beijing, China, July 28, 2011.
8. Carterette, B. 2011. System effectiveness, user models, and user utility: a conceptual framework for investigation. ACM SIGIR '11. ACM, New York, NY, USA, 903-912.
9. Vakkari, P. 2000. Relevance and contributing information types of searched documents in task performance. SIGIR 2000, 2-9.
10. Jacek Gwizdka. 2010. Distribution of cognitive load in Web search. J. Am. Soc. Inf. Sci. Technol. 61, 11 (Nov 10), 2167-2187.
11. Gwizdka, J. (2009). Assessing Cognitive Load on Web Search Tasks. The Ergonomics Open Journal. Bentham Open Access.
12. Iizuka, J., Okamoto, A., Horiuchi, Y., Ichikawa, A. 2009. Considerations of Efficiency and Mental Stress of Search Tasks on Websites by Blind Persons. UAHCI '09, 693-700.
13. Hart, S.G., Staveland, L.E. 1988. Development of a NASA-TLX (Task load index): Results of empirical and theoretical research. In: Hancock, P.A., Meshkati, N. (eds.) Human Mental Workload, 139-183.
14. Louise T. Su. 1992. Evaluation measures for interactive information retrieval. IP & M 28, 4 (March 1992), 503-516.
15. Allan, J. 2005. HARD Track Overview, TREC 2005
16. Smucker M.D., Jethani, C.P. 2011. Measuring assessor accuracy: a comparison of nist assessors and user study participants. SIGIR 2011, 1231-1232.