The beauty contest revisited: Measuring consensus rankings of relevance using a game

ACM International Conference Proceeding Series

Volumn , Issue , 2014, Pages 17-21

  Harris, Christopher G ^a  

^a STATE UNIVERSITY OF NEW YORK AT OSWEGO   (United States)

Author keywords

Consensus Prediction; Economic Theory; Gamification; Information Retrieval; Rank Biased Overlap; Retrieval Effectiveness

Indexed keywords

ECONOMICS;

CONSENSUS RANKING; ECONOMIC THEORIES; GAMIFICATION; MULTIPLE CONTEXTS; RANK BIASED OVERLAP; RECALL AND PRECISION; RETRIEVAL EFFECTIVENESS; TEXTUAL DOCUMENTS;

INFORMATION RETRIEVAL;

EID: 84899873138     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2594776.2594780     Document Type: Conference Paper

References (17)

1. Alonso, O., &Mizzaro, S. (2009, July). Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. In Proceedings of the SIGIR 2009 Workshop on the Future of IR Evaluation (pp. 15-16).
2. Alonso, O., &Mizzaro, S. (2012). Using crowdsourcing for TREC relevance assessment. Information Processing &Management, 48(6), 1053-1066.
3. Argawal, R. Comparing Ranked List. 2013. Retrieved January 19, 2014 from http://ragrawal.wordpress.com /2013/01/18/comparing-ranked-list
4. Eickhoff, C., Harris, C. G., de Vries, A. P., &Srinivasan, P. (2012, August). Quality through flow and immersion: gamifying crowdsourced relevance assessments. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval (pp. 871-880). ACM.
5. Grubinger, M., Clough, P., Hanbury, A., &M?ller, H. (2008). Overview of the ImageCLEFphoto 2007 photographic retrieval task. In Advances in Multilingual and Multimodal Information Retrieval (pp. 433-444). Springer Berlin Heidelberg.
6. Grubinger, M., Clough, P., M?ller, H., &Deselaers, T. (2006). The iapr tc-12 benchmark: A new evaluation resource for visual information systems. In International Workshop OntoImage (pp. 13-23).
7. Harris, C. G., &Srinivasan, P. (2013). Comparing crowdbased, game-based, and machine-based approaches in initial query and query refinement tasks. In Advances in Information Retrieval (pp. 495-506). Springer Berlin Heidelberg.
8. Hogarth, R. M. (1978). A note on aggregating opinions. Organizational Behavior and Human Performance, 21(1), 40-46.
9. Keynes, J. M. (1936). General theory of employment, interest and money. Atlantic Books.
10. Kittur, A., Chi, E. H., &Suh, B. (2008, April). Crowdsourcing user studies with Mechanical Turk. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 453-456). ACM.
11. Liu, T. Y. (2009). Learning to rank for information retrieval. Foundations and Trends in Information Retrieval, 3(3), 225-331.
12. Makridakis, S., &Winkler, R. L. (1983). Averages of forecasts: Some empirical results. Management Science, 29(9), 987-996.
13. Prelec, D. (2004). A Bayesian truth serum for subjective data. Science, 306(5695), 462-466.
14. Shaw, A. D., Horton, J. J., &Chen, D. L. (2011, March). Designing incentives for inexpert human raters. In Proceedings of the ACM 2011 conference on Computer supported cooperative work (pp. 275-284). ACM.
15. Surowiecki, J. (2005). The wisdom of crowds. Random House Digital, Inc.
16. Vorhees, V., &Harman, D. (1999). Overview of the eighth Text REtrieval Conference (TREC-8). In Proc. TREC.
17. Webber, W., Moffat, A., &Zobel, J. (2010). A similarity measure for indefinite rankings. ACM Transactions on Information Systems (TOIS), 28(4), 20.