Measuring crowd truth for medical relation extraction

AAAI Fall Symposium - Technical Report

Volumn FS-13-04, Issue , 2013, Pages 2-9

  Aroyo, Lora ^a   Welty, Chris ^b  

^a VU UNIVERSITY AMSTERDAM   (Netherlands)

^b IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

SEMANTICS;

COST-EFFECTIVE APPROACH; CRITICAL STEPS; CROWDSOURCING; GOLD STANDARDS; GROUND TRUTH DATA; LARGE AMOUNTS; LOW QUALITIES; RELATION EXTRACTION;

EXTRACTION;

EID: 84898881443     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (20)

1. Alonso, O., and Baeza-Yates, R. 2011. Design and implementation of relevance assessments using crowdsourcing. In In Proc. ECAIR, 153-164. Springer-Verlag.
2. Ang, J.; Dhillon, R.; Krupski, A.; Shriberg, E.; and Stolcke, A. 2002. Prosody-based automatic detection of annoyance and frustration in human-computer dialog. In in Proc. ICSLP 2002, 2037-2040.
3. Aroyo, L., and Welty, C. 2013a. Crowd truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. In Web Science 2013. ACM.
4. Aroyo, L., and Welty, C. 2013b. Harnessing disagreement in crowdsourcing a relation extraction gold standard. Technical Report No.203386, IBM Research.
5. Bunescu, R., and Mooney, R. 2006. Subsequence kernels for relation extraction. In Weiss, Y.; Scholkopf, B.; and Platt, J., eds., Advances in Neural Information Processing Systems 18. Cambridge, MA: MIT Press. 171-178.
6. Carlson, A.; Betteridge, J.; Hruschka, Jr., E. R.; and Mitchell, T. M. 2009. Coupling semi-supervised learning of categories and relations. In Proceedings of the NAACL HLT 2009 Workshop on Semi-Supervised Learning for Natural Language Processing, SemiSupLearn '09, 1-9. Stroudsburg, PA, USA: Association for Computational Linguistics.
7. Chen, D., and Dolan, W. 2011. Building a persistent workforce on mechanical turk for multilingual data collection.
8. Chklovski, T., and Mihalcea, R. 2003. Exploiting agreement and disagreement of human annotators for word sense disambiguation. In UNT Scholarly Works. UNT Digital Library.
9. Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20:37-46.
10. Finin, T.; Murnane, W.; Karandikar, A.; Keller, N.; Martineau, J.; and Dredze, M. 2010. Annotating named entities in twitter data with crowdsourcing. In In Proc. NAACL HLT, CSLDAMT '10, 80-88. Association for Computational Linguistics.
11. Gligorov, R.; Hildebrand, M.; van Ossenbruggen, J.; Schreiber, G.; and Aroyo, L. 2011. On the role of user-generated metadata in audio visual collections. In K-CAP, 145-152.
12. Hovy, E.; Mitamura, T.; and Verdejo, F. 2012. Event coreference annotation manual. Technical report, Information Sciences Institute (ISI).
13. Litman, D. J. 2004. Annotating student emotional states in spoken tutoring dialogues. In In Proc. 5th SIGdial Workshop on Discourse and Dialogue, 144-153.
14. Mintz, M.; Bills, S.; Snow, R.; and Jurafsky, D. 2009. Distant supervision for relation extraction without labeled data. In In Proc. ACL and Natural Language Processing of the AFNLP: Vol2, 1003-1011. Association for Computational Linguistics.
15. Raykar, V. C., and Yu, S. 2012. Eliminating spammers and ranking annotators for crowdsourced labeling tasks. J. Mach. Learn. Res. 13:491-518.
16. Sarasua, C.; Simperl, E.; and Noy, N. F. 2012. Crowdmap: Crowdsourcing ontology alignment with microtasks. In International Semantic Web Conference (1), 525-541.
17. van Zwol, R.; Garcia, L.; Ramirez, G.; Sigurbjornsson, B.; and Labad, M. 2008. Video tag game. In WWW Conference, developer track). ACM.
18. Viera, A. J., and Garrett, J. M. 2005. Understanding interobserver agreement: the kappa statistic. Family Medicine 37(5):360-363. (Pubitemid 40656185)
19. Wang, C.; Fan, J.; Kalyanpur, A.; and Gondek, D. 2011. Relation extraction with relation topics. In EMNLP, 1426-1436.
20. Zhou, Z.-H., and Li, M. 2010. Semi-supervised learning by disagreement. Knowl. Inf. Syst. 24(3):415-439.