Multidimensional relevance modeling via psychometrics and crowdsourcing

SIGIR 2014 - Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2014, Pages 435-444

  Zhang, Yinglong ^a   Zhang, Jin ^a   Lease, Matthew ^a   Gwizdka, Jacek ^a  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

Crowdsourcing; Psychometrics; Relevance judgment

Indexed keywords

CROWDSOURCING; MEASURE MODEL; METHODOLOGICAL FRAMEWORKS; MULTI-DIMENSIONAL MODEL; MULTI-DIMENSIONAL RELEVANCE; OBSERVED DATA; PSYCHOMETRICS; RELEVANCE JUDGMENT;

INFORMATION RETRIEVAL;

EID: 84904570709     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2600428.2609577     Document Type: Conference Paper

References (77)

1. Alonso, O. 2013. Implementing crowdsourcing-based relevance experimentation: an industrial perspective. Information Retrieval. 16, 2, 101-120.
2. Anderson, J.C. and Gerbing, D.W. 1988. Structural equation modeling in practice: A review and recommended two-step approach. Psychological Bulletin. 103, 3, 411-423.
3. Bailey, P. et al. 2008. Relevance assessment: are judges exchangeable and does it matter. SIGIR'08, 667-674.
4. Balatsoukas, P. and Ruthven, I. 2012. An eye-tracking approach to the analysis of relevance judgments on the Web: The case of Google search engine. JASIST. 63, 9, 1728-1746.
5. Baokstein, A. 1979. Relevance. JASIS. 30, 5, 269-273.
6. Barry, C.L. 1994. User-defined relevance criteria: An exploratory study. JASIS. 45, 3, 149-159.
7. Barry, C.L. and Schamber, L. 1998. Users? criteria for relevance evaluation: A cross-situational comparison. IP&M. 34, 2-3, 219-236.
8. Bateman, J. 1998. Changes in Relevance Criteria: A Longitudinal Study. Proceedings of the ASIS Annual Meeting. 35, 23-32.
9. Behrend, T.S. et al. 2011. The viability of crowdsourcing for survey research. Behavior research methods. 43, 3, 800-813.
10. Blanco, R. et al. 2011. Repeatable and Reliable Search System Evaluation Using Crowdsourcing. Proceedings of SIGIR'2011 New York, NY, USA, 923-932.
11. Borlund, P. 2003. The concept of relevance in IR. JASIST. 54, 10, 913-925.
12. Boyce, B. 1982. Beyond topicality: A two stage view of relevance and the retrieval process. IP&M 18, 3, 105-109.
13. Bradford, S.C. 1934. Sources of information on specific subjects. Engineering: An Illustrated Weekly Journal (London). 137, 26, 85-86.
14. Browne, M.W. 2000. Psychometrics. Journal of the American Statistical Association. 95, 450, 661-665.
15. Cacioppo, J.T. and Petty, R.E. 1984. The Elaboration Likelihood Model of Persuasion. Advances in Consumer Research. 11, 1 673-675.
16. Chouldechova, A. and Mease, D. 2013. Differences in Search Engine Evaluations Between Query Owners and Non-owners. Proceedings of the Sixth ACM International Conference on Web Search and Data Mining, 103-112.
17. Cognitive Interviewing: http://www.uk.sagepub.com/textbooks/Book225856? prodId=Bo ok225856. Accessed: 2014-01-24.
18. Cohen, J. 1988. Statistical power analysis for the behavioral sciences. L. Erlbaum Associates.
19. Cool, C. et al. 1993. Characteristics of Texts affecting relevance judgements. Proceedings of the 14th National Online Meeting, 77-84.
20. Da Costa Pereira, C. et al. 2012. Multidimensional relevance: Prioritized aggregation in a personalized Information Retrieval setting. IP&M. 48, 2, 340-357.
21. Cuadra, C.A. and Katter, R.V. 1967. Opening the Black Box of "Relevance." Journal of Documentation. 23, 4, 291-303.
22. Dwyer, J. 2002. Communication in Business: Strategies and Skills. Prentice Hall.
23. Eickhoff, C. et al. 2013. Copulas for Information Retrieval. Proceedings of SIGIR'2013 (New York, NY, USA), 663-672.
24. Eickhoff, C. and Vries, A.P. de 2013. Increasing cheat robustness of crowdsourcing tasks. Information Retrieval. 16, 2, 121-137.
25. Franklin, S.B. et al. 1995. Parallel Analysis: a method for determining significant principal components. Journal of Vegetation Science. 6, 1, 99-106.
26. Furr, M. 2011. Scale Construction and Psychometrics for Social and Personality Psychology. SAGE.
27. Goldberg, L.R. and Kilkowski, J.M. 1985. The prediction of semantic consistency in self-descriptions: characteristics of persons and of terms that affect the consistency of responses to synonym and antonym pairs. Journal of personality and social psychology. 48, 1, 82-98.
28. Green, R. 1995. Topical relevance relationships. I. Why topic matching fails. JASIS. 46, 9, 646-653.
29. Greisdorf, H. 2003. Relevance thresholds: a multi-stage predictive model of how users evaluate information. IP&M, 403-423.
30. Grice, H.P. 1989. Studies in the way of words. Harvard University Press.
31. Gwizdka, J. 2014. News Stories Relevance Effects on Eyemovements. Proceedings of the Symposium on Eye Tracking Research and Applications, 283-286.
32. Harter, S.P. 1992. Psychological relevance and information science. JASIS. 43, 9, 602-615.
33. Hatcher, L. 2013. Advanced statistics in research: reading, understanding, and writing up data analysis results. ShadowFinch Media, LLC.
34. Hjorland, B. and Christensen, F.S. 2002. Work tasks and sociocognitive relevance: A specific example. JASIST. 53, 11, 960-965.
35. Hosseini, M. et al. 2012. On Aggregating Labels from Multiple Crowd Workers to Infer Relevance of Documents. Advances in Information Retrieval. R. Baeza-Yates et al., eds. Springer Berlin Heidelberg. 182-194.
36. Hox, J.J. and Bechger, T.M. 2007. An introduction to structural equation modeling.
37. Hu, L. and Bentler, P.M. 1999. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal. 6, 1, 1-55.
38. Huang, X. and Soergel, D. 2013. Relevance: An improved framework for explicating the notion. JASIST. 64, 1, 18-35.
39. Johnson, J.R. et al. 1981. Characteristics of Errors in Accounts Receivable and Inventory Audits. The Accounting Review. 56, 2, 270-293.
40. Kazai, G. et al. 2012. An Analysis of Systematic Judging Errors in Information Retrieval. Proceedings of CIKM'2012 (New York, NY, USA), 105-114.
41. Kazai, G. et al. 2011. Crowdsourcing for Book Search Evaluation: Impact of Hit Design on Comparative System Ranking. Proceedings of SIGIR'2011 (New York, NY, USA), 205-214.
42. Kittur, A. et al. 2008. Crowdsourcing User Studies with Mechanical Turk. Proceedings of SIGCHI'2008 (New York, NY, USA), 453-456.
43. Lancaster, F.W. 1968. Information retrieval systems: characteristics, testing, and evaluation. Wiley.
44. Lesk, M.E. and Salton, G. 1968. Relevance assessments and retrieval system evaluation. Information Storage and Retrieval. 4, 4, 343-359.
45. Levitin, A. and Redman, T. 1995. Quality dimensions of a conceptual view. IP&M. 31, 1, 81-88.
46. Little, G. 2009. TurKit: Tools for iterative tasks on mechanical turk. IEEE Symposium on Visual Languages and Human- Centric Computing, 252-253.
47. Liu, T.-Y. 2009. Learning to Rank for Information Retrieval. Found. Trends Inf. Retr. 3, 3, 225-331.
48. M, P. and Bonett, D.G. 1980. Significance tests and goodness of fit in the analysis of covariance structures. Psychological Bulletin. 88, 3, 588-606.
49. Maron, M.E. 1977. On indexing, retrieval and the meaning of about. JASIS. 28, 1, 38-43.
50. Marshall, C.C. and Shipman, F.M. 2013. Experiences Surveying the Crowd: Reflections on Methods, Participation, and Reliability. Proceedings of the 5th Annual ACM Web Science Conference, 234-243.
51. Mizzaro, S. 1997. Relevance: The whole history. JASIS. 48, 9, 810-832.
52. Moshfeghi, Y. et al. 2013. Understanding Relevance: An fMRI Study. Advances in Information Retrieval. P. Serdyukov et al., eds. Springer Berlin Heidelberg. 14-25.
53. Mueller, R.O. and Hancock, G.R. 2008. Best practices in structural equation modeling. Best practices in quantitative methods. 488-508.
54. Murphy, K.P. 2012. Machine Learning: A Probabilistic Perspective. Mit Press.
55. Pearson - Modern Measurement: Theory, Principles, and Applications of Mental Appraisal, 2/E - Steven J. Osterlind: http://www.pearsonhighered.com/ educator/product/Modern- Measurement-Theory-Principles-and-Applications-of- Mental- Appraisal/9780137010257.page. Accessed: 2014-01-24.
56. Principles and Practice of Structural Equation Modeling: Third Edition: http://www.guilford.com/cgibin/cartscript.cgi?page=pr/kline.htm&dir= research/res-quant. Accessed: 2014-01-24.
57. Proceedings of the International Conference on Scientific Information - Two Volumes: http://books.nap.edu/openbook.php?record-id=10866&page=6 87. Accessed: 2014-01-26.
58. Rees, A.M. and Schultz, D.G. 1967. A Field Experimental Approach to the Study of Relevance Assessments in Relation to Document Searching. Final Report to the National Science Foundation. Volume I.
59. Relevance as process: judgements in the context of scholarly research: http://www.informationr.net/ir/10-2/paper226. Accessed: 2014-01-24.
60. Sanderson, M. 2010. Test Collection Based Evaluation of Information Retrieval Systems. Foundations and Trends in Information Retrieval. 4, 4, 247-375.
61. Saracevic, T. 2007. Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: nature and manifestations of relevance. JASIST. 58, 13, 1915-1933.
62. Saracevic, T. 2007. Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance. JASIST. 58, 13, 2126-2144.
63. Schamber, L. 1994. Relevance and Information Behavior. Annual Review of Information Science and Technology (ARIST). 29, 3-48.
64. Scheines, R. et al. 1999. Bayesian estimation and testing of structural equation models. Psychometrika. 64, 1, 37-52.
65. Tabachnick, B.G. and Fidell, L.S. 2012. Using Multivariate Statistics. Pearson Education, Limited.
66. Tang, R. and Solomon, P. 1998. Toward an understanding of the dynamics of relevance judgment: An analysis of one person?s search behavior. IP&M. 34, 2-3, 237-256.
67. Taylor, A.R. et al. 2007. Relationships between categories of relevance criteria and stage in task completion. IP&M. 43, 4, 1071-1084.
68. The Social Construction of Meaning: An Alternative Perspective on Information Sharing: 2003. http://pubsonline.informs.org/doi/abs/10.1287/isre. 14.1.87.1476 5. Accessed: 2014-01-24.
69. Tsikrika, T. and Lalmas, M. 2007. Combining Evidence for Relevance Criteria: A Framework and Experiments in Web Retrieval. Advances in Information Retrieval. G. Amati et al., eds. Springer Berlin Heidelberg. 481-493.
70. Vakkari, P. and Hakala, N. 2000. Changes in relevance criteria and problem stages in task performance. Journal of Documentation. 56, 5, 540-562.
71. Voorhees, E.M. 1998. Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness. Proceedings of SIGIR'1998 (New York, NY, USA), 315-323.
72. Wilson, D. and Sperber, D. 2002. Relevance Theory. Handbook of Pragmatics. G. Ward and L. Horn, eds. Blackwell.
73. De Winter, J.C.F. and Dodou, D. 2012. Factor recovery by principal axis factoring and maximum likelihood factor analysis as a function of factor pattern and sample size. Journal of Applied Statistics. 39, 4, 695-710.
74. Worthington, R.L. and Whittaker, T.A. 2006. Scale Development Research A Content Analysis and Recommendations for Best Practices. The Counseling Psychologist. 34, 6, 806-838.
75. Wright, S. Correlation and causation.
76. Xu, Y. (Calvin) and Chen, Z. 2006. Relevance judgment: What do information users consider beyond topicality? JASIST. 57, 7, 961-973.
77. Zuccon, G. et al. 2013. Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems. Information Retrieval. 16, 2, 267-305.