Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments

Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2016

Volumn , Issue , 2016, Pages 139-148

  McDonnell, Tyler ^a   Lease, Matthew ^b   Kutlu, Mucahid ^c   Elsayed, Tamer ^c  

^a University of Texas at Austin   (United States)

^b UNIVERSITY OF TEXAS AT AUSTIN   (United States)

^c QATAR UNIVERSITY   (Qatar)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE;

ADDED VALUES; COST-BENEFITS ANALYSIS; DATA QUALITY; MECHANICAL TURKS; RELEVANCE JUDGEMENT; SEARCH QUERIES; TASK COMPLETION TIME; WEB-PAGE; WIN-WIN; WORKERS';

COST BENEFIT ANALYSIS;

EID: 85107998643     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (28)

1. Adar, E. 2011. Why i hate mechanical turk research (and workshops). In CHI Workshop on Crowdsourcing and Human Comp.
2. Alonso, O.; Rose, D. E.; and Stewart, B. 2008. Crowdsourcing for relevance evaluation. In ACM SigIR Forum, volume 42, 9-15.
3. Alonso, O. 2009. Guidelines for designing crowdsourcingbased relevance experiments. CiteSeerX DOI 10.1.1.149.6649.
4. Artstein, R., and Poesio, M. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics 34(4):555-596.
5. Bailey, P.; Craswell, N.; Soboroff, I.; Thomas, P.; de Vries, A. P.; and Yilmaz, E. 2008. Relevance assessment: are judges exchangeable and does it matter. In SIGIR, 667-674. ACM.
6. Bernstein, M. S.; Little, G.; Miller, R. C.; Hartmann, B.; Ackerman, M. S.; Karger, D. R.; Crowell, D.; and Panovich, K. 2010. Soylent: a word processor with a crowd inside. In UIST, 313-322. ACM.
7. Blanco, R.; Halpin, H.; Herzig, D. M.; Mika, P.; Pound, J.; Thompson, H. S.; and Tran Duc, T. 2011. Repeatable and reliable search system evaluation using crowdsourcing. In SIGIR, 923-932. ACM.
8. Carletta, J. 1996. Assessing agreement on classification tasks: the kappa statistic. Computational linguistics 22(2):249-254.
9. Carterette, B.; Pavlu, V.; Fang, H.; and Kanoulas, E. 2010. Million query track 2009 overview. In Proceedings of NIST TREC.
10. Clarke, C. L.; Craswell, N.; and Soboroff, I. 2010. Overview of the TREC 2009 Web Track. In Proceedings of NIST TREC.
11. Dawid, A. P., and Skene, A. M. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied statistics 28(1):20-28.
12. Donahue, J., and Grauman, K. 2011. Annotator rationales for visual recognition. In ICCV, 1395-1402. IEEE.
13. Google. 2016. Search quality rating guidelines. Inside Search: How Search Works. www.google.com/insidesearch/.
14. Hosseini, M.; Cox, I. J.; Milić-Frayling, N.; Kazai, G.; and Vinay, V. 2012. On aggregating labels from multiple crowd workers to infer relevance of documents. In ECIR. Springer. 182-194.
15. Ipeirotis, P. 2010. A plea to amazon: Fix mechanical turk! Blog: Behind Enemy Lines. October 21, 2010. www.behindthe-enemy-lines.com.
16. Kittur, A.; Nickerson, J. V.; Bernstein, M.; Gerber, E.; Shaw, A.; Zimmerman, J.; Lease, M.; and Horton, J. 2013. The Future of Crowd Work. In CSCW, 1301-1318. ACM.
17. Kittur, A.; Chi, E. H.; and Suh, B. 2008. Crowdsourcing user studies with mechanical turk. In Proceedings of the SIGCHI conference on human factors in computing systems, 453-456. ACM.
18. Marshall, C. C., and Shipman, F. M. 2013. Experiences surveying the crowd: Reflections on methods, participation, and reliability. In 5th Annual Web Science Conference, 234-243. ACM.
19. Ratcliff, J.W., and Metzener, D. E. 1988. Pattern-matching-the gestalt approach. Dr Dobbs Journal 13(7):46.
20. Sanderson, M. 2010. Test collection based evaluation of information retrieval systems. Now Publishers Inc.
21. Saracevic, T. 2007. Relevance: A review of the literature and a framework for thinking on the notion in information science. part iii: Behavior and effects of relevance. Journal of the American Society for Information Science and Technology 58(13):2126-2144.
22. Sheshadri, A., and Lease, M. 2013. SQUARE: A Benchmark for Research on Computing Crowd Consensus. In Proceedings of the AAAI Conference on Human Computation (HCOMP), 156-164.
23. Tang, R.; Shaw Jr, W. M.; and Vevea, J. L. 1999. Towards the identification of the optimal number of relevance categories. Journal of the Association for Information Science and Technology 50(3):254.
24. Trotman, A.; Pharo, N.; and Jenkinson, D. 2007. Can we at least agree on something. In Proceedings of the SIGIR 2007 Workshop on Focused Retrieval, 49-56.
25. Voorhees, E. M.; Harman, D. K.; et al. 2005. TREC: Experiment and evaluation in information retrieval. The MIT Press.
26. Voorhees, E. M. 2000. Variations in relevance judgments and the measurement of retrieval effectiveness. Information processing & management 36(5):697-716.
27. Voorhees, E. M. 2001. The philosophy of information retrieval evaluation. In Evaluation of cross-language information retrieval systems, 355-370. Springer.
28. Zaidan, O. F.; Eisner, J.; and Piatko, C. D. 2007. Using annotator rationales to improve machine learning for text categorization. In HLT-NAACL, 260-267.