"Dr. Detective": Combining gamification techniques and crowdsourcing to create a gold standard in medical text

CEUR Workshop Proceedings

Volumn 1030, Issue , 2013, Pages 16-31

  Dumitrache, Anca ^a,c   Aroyo, Lora ^a   Welty, Chris ^b   Sips, Robert Jan ^c   Levas, Anthony ^b  

^a VU UNIVERSITY AMSTERDAM   (Netherlands)

^b IBM T J WATSON RESEARCH CENTER   (United States)

^c IBM   (Netherlands)

Author keywords

Crowdsourcing; Games with a purpose; Gold standard; Information extraction; Natural language processing

Indexed keywords

DIAGNOSIS; GOLD; INFORMATION RETRIEVAL; NATURAL LANGUAGE PROCESSING SYSTEMS; SOCIAL NETWORKING (ONLINE);

CROWDSOURCING; CROWDSOURCING PLATFORMS; GAME WITH A PURPOSE; GAMES WITH A PURPOSE; GOLD STANDARDS; MEDICAL EXPERTS; NATURAL LANGUAGE PROCESSING; PILOT STUDIES;

SEMANTIC WEB;

EID: 84923882840     PISSN: 16130073     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. Ferrucci, D., Brown, E., Chu-Carroll, J., Fan, J., Gondek, D., Kalyanpur, A. A., Lally, A., Murdock, J. W., Nyberg, E., Prager, J., Schlaefer, N., Welty, C.: Building Watson: An Overview of the DeepQA Project. AI Magazine 31 (2010) 59-79
2. Kalyanpur, A., Boguraev, B., Patwardhan, S., Murdock, J. W., Lally, A., Welty, C., Prager, J. M., Coppola, B., Fokoue-Nkoutche, A., Zhang, L., (et al).: Structured data and inference in DeepQA. IBM Journal of Research and Development 56 (3. 4) (2012) 10-1
3. Aroyo, L., Welty, C.: Harnessing disagreement for event semantics. Detection, Representation, and Exploitation of Events in the Semantic Web 31
4. Aroyo, L., Welty, C.: Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard. WebSci2013. ACM (2013)
5. Inel, O., Aroyo, L., Welty, C., Sips, R. J.: Exploiting Crowdsourcing Disagreement with Various Domain-Independent Quality Measures. Technical report, VU University Amsterdam (July 2013). http://crowd-watson. nl/tech-reports/20130702.pdf
6. Von Ahn, L., Dabbish, L.: Labeling images with a computer game. In: Proceedings of the SIGCHI conference on Human factors in computing systems, ACM (2004) 319-326
7. Von Ahn, L., Maurer, B., McMillen, C., Abraham, D., Blum, M.: recaptcha: Human-based character recognition via web security measures. Science 321 (5895) (2008) 1465-1468
8. Lintott, C. J., Schawinski, K., Slosar, A., Land, K., Bamford, S., Thomas, D., Raddick, M. J., Nichol, R. C., Szalay, A., Andreescu, D., (et al).: Galaxy zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey. Monthly Notices of the Royal Astronomical Society 389 (3) (2008) 1179-1189
9. Siorpaes, K., Hepp, M.: Games with a Purpose for the Semantic Web. Intelligent Systems, IEEE 23 (3) (2008) 50-60
10. Chamberlain, J., Poesio, M., Kruschwitz, U.: Phrase Detectives - A Web-based Collaborative Annotation Game. In: Proceedings of I-Semantics. (2008)
11. Scharl, A., Sabou, M., Gindl, S., Rafelsberger, W., Weichselbraun, A.: Leveraging the Wisdom of the Crowds for the Acquisition of Multilingual Language Resources. In: Proc. 8th LREC - International Conference on Language Resources and Evaluation. (2012)
12. Lin, H.: Crowd Watson: Crowdsourced Text Annotations. Technical report, VU University Amsterdam (July 2013). http://crowd-watson. nl/tech-reports/20130704.pdf
13. Bodenreider, O.: The unified medical language system (UMLS): integrating biomedical terminology. Nucleic acids research 32(suppl 1) (2004) D267-D270
14. Goldberg, H. S., Hsu, C., Law, V., Safran, C.: Validation of clinical problems using a UMLS-based semantic parser. In: Proceedings of the AMIA Symposium, American Medical Informatics Association (1998) 805
15. Aroyo, L., Welty, C.: Measuring crowd truth for medical relation extraction. In: AAAI 2013 Fall Symposium on Semantics for Big Data (in print). (2013)
16. Soberón, G., Aroyo, L., Welty, C., Inel, O., Lin, H., Overmeen, M.: Crowd Truth Metrics. Technical report, VU University Amsterdam (July 2013). http://crowd-watson. nl/tech-reports/20130703.pdf
17. Tokarchuk, O., Cuel, R., Zamarian, M.: Analyzing crowd labor and designing incentives for humans in the loop. IEEE Internet Computing 16 (5) (2012) 0045-51
18. McLaughlin, G. H.: SMOG grading: A new readability formula. Journal of reading 12 (8) (1969) 639-646
19. Doak, C. C., Doak, L. G., Root, J. H.: Teaching patients with low literacy skills. AJN The American Journal of Nursing 96 (12) (1996) 16M
20. Von Ahn, L., Dabbish, L.: Designing games with a purpose. Communications of the ACM 51 (8) (2008) 58-67
21. Sherry, J. L.: Flow and media enjoyment. Communication Theory 14 (4) (2004) 328-347