The effect of threshold priming and need for cognition on relevance calibration and assessment

SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2013, Pages 623-632

  Scholer, Falk ^a   Kelly, Diane ^b   Wu, Wan Ching ^b   Lee, Hanseul S ^b   Webber, William ^c  

^a RMIT UNIVERSITY   (Australia)

^b UNIVERSITY OF NORTH CAROLINA   (United States)

^c UNIVERSITY OF MARYLAND   (United States)

Author keywords

Assessors; Calibration; Evaluation; Need for cognition; Order effects; Relevance assessments; Relevance behavior; Threshold priming

Indexed keywords

ASSESSORS; EVALUATION; NEED FOR COGNITIONS; ORDERING EFFECTS; RELEVANCE ASSESSMENTS; RELEVANCE BEHAVIOR; THRESHOLD PRIMING;

INFORMATION RETRIEVAL;

CALIBRATION;

EID: 84883103658     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2484028.2484090     Document Type: Conference Paper

References (32)

1. Carol Barry. User-defined relevance criteria: an exploratory study. Journal of the American Society for Information Science, 45: 149-159, 1994.
2. Pia Borlund. The concept of relevance in IR. Journal of the American Society for Information Science and Technology, 54(10): 913-925, 2003.
3. John T. Cacioppo and Richard E. Petty. The need for cognition. Journal of Personality and Social Psychology, 42(1):116-131, 1982.
4. John T. Cacioppo, Richard E. Petty, Jeffery A. Feinstein, and W. Blaire G. Jarvis. Dispositional differences in cognitive motivation: The life and times of individuals varying in need for cognition. Psychological Bulletin, 119(2):197-253, 1996.
5. Ben Carterette and Ian Soboroff. The effect of assessor errors on IR system evaluation. In Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval, pages 539-546, Geneva, Switzerland, 2010.
6. Carlos A. Cuadra and Robert V. Katter. Opening the black box of relevance. Journal of Documentation, 23(4):291-303, 1967.
7. David Davidson. The effect of individual differences of cognitive style on judgments of document relevance. Journal of the American Society for Information Science, 28(5):273-284, 1977.
8. Efthimis N. Efthimiadis and Mary A. Hotchkiss. Legal discovery: does domain expertise matter? Proceedings of the American Society for Information Science and Technology, 45:1-2, 2008.
9. Michael Eisenberg and Carol Barry. Order effects: a study of the possible influence of presentation order on user judgments of document relevance. Journal of the American Society for Information Science, 39:293-300, 1988.
10. Nigel Ford, David Miller, and Nicola Moss. Web search strategies and human individual differences: cognitive and demographic factors, internet attitudes and approaches. Journal of the American Society for Information Science and Technology, 56:741-756, 2005.
11. Maura Grossman and Gordon V. Cormack. Inconsistent assessment of responsiveness in e-discovery: difference of opinion or human error? In DESI IV: The ICAIL Workshop on Setting Standards for Searching Electronically Stored Information in Discovery Proceedings, pages 1-11, June 2011.
12. Stephen P. Harter. Psychological relevance and information science. Journal of the American Society for Information Science, 43: 602-615, 1992.
13. Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2):65-70, 1979.
14. Mu-hsuan Huang and Hui-yu Wang. The influence of document presentation order and number of documents judged on users' judgments of relevance. Journal of the American Society for Information Science and Technology, 55(11):970-979, 2004.
15. Gabriella Kazai, Jaap Kamps, Marijn Koolen, and Natasa Milic-Frayling. Crowdsourcing for book search evaluation: impact of hit design on comparative system ranking. In Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval, pages 205-214, Beijing, China, 2011. ACM.
16. Gabriella Kazai, Nick Craswell, Emine Yilmaz, and S. M. M. Tahaghoghi. An analysis of systematic judging errors in information retrieval. In Proceedings of the ACM International Conference on Information and Knowledge Management, pages 105-114, Maui, Hawaii, USA, 2012.
17. John Le, Andy Edmonds, Vaughn Hester, and Lukas Biewald. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 workshop on crowdsourcing for search evaluation, pages 21-26, 2010.
18. Douglas W. Oard and William Webber. Information retrieval for e-discovery. 2013. In submission. http://ediscovery.umiacs.umd.edu/pub/ ow12fntir.pdf.
19. Lorraine M. Purgaillis Parker and Robert E. Johnson. Does order of presentation affect users' judgment of documents? Journal of the American Society for Information Science, 41(7):493-494, 1990.
20. Tefko Saracevic. Relevance: a review of and a framework for the thinking on the notion of information science. Journal of the American Society for Information Science, 26(6):321-343, 1975.
21. Tefko Saracevic. Relevance: a review of the literature and a framework for thinking on the notion in information science. part II: Nature and manifestations of relevance. Journal of the American Society for Information Science and Technology, 58:1915-1933, 2007.
22. Tefko Saracevic. Relevance: a review of the literature and a framework for thinking on the notion in information science. part III: Behavior and effects of relevance. Journal of the American Society for Information Science and Technology, 58:2126-2144, 2007.
23. Falk Scholer, Andrew Turpin, and Mark Sanderson. Quantifying test collection quality based on the consistency of relevance judgements. In Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval, pages 1063-1072, Beijing, China, 2011.
24. Eero Sormunen. Liberal relevance criteria of TREC - counting on negligible documents? In Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval, pages 324-330, Tampere, Finland, 2002.
25. Anastasios Tombros, Ian Ruthven, and Joemon M. Jose. How users assess web pages for information seeking. Journal of the American Society for Information Science and Technology, 56(4):327-344, 2005.
26. Christian Unkelbach, Vanessa Ostheimer, Frowin Fasold, and Daniel Memmert. A calibration explanation of serial position effects in evaluative judgments. Organizational Behavior and Human Decision Processes, 119:103-113, 2012.
27. Pertti Vakkari. Changes in search tactics and relevance judgments when preparing a research proposal: a summary of the findings of a longitudinal study. Information Retrieval, 4(3):295-310, 2001.
28. Bas Verplanken. Need for cognition and external information search: Responses to time pressure during decision-making. Journal of Research in Personality, 27(3):238-252, 1993.
29. Ellen M. Voorhees. Variations in relevance judgements and the measurement of retrieval effectiveness. Information Processing and Management, 36(5):697-716, 2000.
30. Jianqiang Wang and Dagobert Soergel. A user study of relevance judgments for e-discovery. Proceedings of the American Society for Information Science and Technology, 47:1-10, 2010.
31. William Webber. Re-examining the effectiveness of manual review. In Proc. SIGIR Information Retrieval for E-Discovery Workshop, pages 2:1-8, Beijing, China, July 2011.
32. William Webber, Bryan Toth, and Marjorie Desamito. Effect of written instructions on assessor agreement. In Proceedings of the ACM SIGIR International Conference on Research and Development in Information Retrieval, pages 1053-1054, Portland, Oregon, USA, 2012.