Factors affecting relevance judgment: A report from TREC Legal track

Journal of Documentation

Volumn 67, Issue 2, 2011, Pages 264-278

  Chu, Heting ^a  

^a LONG ISLAND UNIVERSITY   (United States)

Author keywords

Assessment; Information management; Information retrieval

Indexed keywords

EID: 79952294580     PISSN: 00220418     EISSN: None     Source Type: Journal    
DOI: 10.1108/00220411111109467     Document Type: Article

References (24)

1. Barry, C.L. (1994), User-defined relevance criteria: an exploratory study in Journal of the American Society for Information Science and Technology, Vol. 45, No. 3, pp. 149-59.
2. Barry, C.L. and Schamber, L. (1998), Users' criteria for relevance evaluation: a cross-situational comparison in Information Processing and Management, Vol. 34, Nos 2/3, pp. 219-36.
3. Bateman, J. (1999), Modeling the importance of end-user relevance criteria in Proceedings of the 62nd Annual Meeting of the American Society for Information Science and Technology, Vol. 36, pp. 396-406.
4. Borlund, P. (2003), The concept of relevance in Journal of the American Society for Information Science and Technology, Vol. 54, No. 10, pp. 913-25.
5. Choi, Y. and Rasumussen, E.M. (2002), Users' relevance criteria in image retrieval in American history in Information Processing and Management, Vol. 38, No. 5, pp. 695-726.
6. Chu, H. (2007), TREC 2007 Legal track interactive task: a report from the LIU team, The 16th Text REtrieval Conference (TREC 2007) Proceedings, National Institute of Standards and Technology, Gaithersburg, MD, NIST Special Publication: SP 500-274.
7. Crystal, A. and Greenberg, J. (2006), Relevance criteria identified by health information users during web searches in Journal of the American Society for Information Science and Technology, Vol. 57, No. 10, pp. 1368-82.
8. Eisenberg, M.B. and Barry, C.L. (1988), Order effects: a study of the possible influence of presentation order on user judgments of document relevance in Journal of the American Society of Information Science, Vol. 39, No. 5, pp. 293-300.
9. Huang, M.H. and Wang, H.Y. (2004), The influence of document presentation order and number of documents judged on users' judgments of relevance in Journal of the American Society for Information Science and Technology, Vol. 55, No. 11, pp. 970-9.
10. Maglaughlin, K.L. and Sonnenwald, D.H. (2002), User perspectives on relevance criteria: a comparison among relevant, partially relevant, and not-relevant judgments in Journal of the American Society for Information Science and Technology, Vol. 53, No. 5, pp. 327-42.
11. Miller, G.A. (1956), The magical number seven plus or minus two: some limits on our capacity for processing information in Psychological Review, Vol. 63, No. 2, pp. 81-97.
12. Ruthven, I., Baillie, M. and Elsweiler, D. (2007), The relative effects of knowledge, interest and confidence in assessing relevance in Journal of Documentation, Vol. 63, No. 4, pp. 482-504.
13. Saracevic, T. (2007a), Relevance: a review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance in Journal of the American Society for Information Science and Technology, Vol. 58, No. 13, pp. 1915-33.
14. Saracevic, T. (2007b), Relevance: a review of the literature and a framework for thinking on the notion in information science. Part III: behavior and effects of relevance in Journal of the American Society for Information Science and Technology, Vol. 58, No. 13, pp. 2126-44.
15. Schamber, L. (1991), Users' criteria for evaluation in a multimedia environment in Proceedings of the 54th Annual Meeting of the American Society for formation Science, Vol. 28, pp. 126-33.
16. Schamber, L. (1994), Relevance and information behavior in Annual Review of Information Science and Technology, Vol. 29, pp. 3-48.
17. Schamber, L., Eisenberg, M.B. and Nilan, M.S. (1990), A re-examination of relevance: toward a dynamic, situational definition in Information Processing and Management, Vol. 26, No. 6, pp. 755-76.
18. Tang, R. and Solomon, P. (2001), Use of relevance criteria across stages of document evaluation: on the complementarity of experimental and naturalistic studies in Journal of the American Society for Information Science and Technology, Vol. 52, No. 8, pp. 676-85.
19. Taylor, A.R., Cool, C., Belkin, N.J. and Amadio, W.J. (2007), Relationships between categories of relevance criteria and stage in task completion in Information Processing and Management, Vol. 43, No. 4, pp. 1071-84.
20. Tomlinson, S., Oard, D.W., Baron, J.R. and Thompson, P. (2007), Overview of the TREC 2007 Legal track in The 16th Text Retrieval Conference (TREC 2007) Proceedings, National Institute of Standards and Technology, Gaithersburg, MD, pp. 51-84, NIST Special Publication: 500-274.
21. Twait, M. (2005), Undergraduate students' source selection criteria: a qualitative study in The Journal of Academic Librarianship, Vol. 31, No. 6, pp. 567-73.
22. Vakkari, P. and Hakala, N. (2000), Changes in relevance criteria and problem stages in task performance in Journal of Documentation, Vol. 56, No. 5, pp. 540-62.
23. Xu, Y. and Chen, Z. (2006), Relevance judgment: what do information users consider beyond topicality? in Journal of the American Society for Information Science and Technology, Vol. 57, No. 7, pp. 961-73.
24. Xu, Y. and Wang, D. (2008), Order effect in relevance judgment in Journal of the American Society for Information Science and Technology, Vol. 59, No. 8, pp. 1264-75.