Crowdsourcing in biomedicine: Challenges and opportunities

Briefings in Bioinformatics

Volumn 17, Issue 1, 2016, Pages 23-32

  Khare, Ritu ^a   Good, Benjamin M ^b   Leaman, Robert ^c   Su, Andrew I ^b   Lu, Zhiyong ^b  

^a CHILDREN S HOSPITAL OF PHILADELPHIA   (United States)

^b SCRIPPS RESEARCH INSTITUTE   (United States)

^c NATIONAL INSTITUTES OF HEALTH   (United States)

Author keywords

Amazon Mechanical Turk; Big data mining; Biomedicine; Community challenges; Crowdsourcing; Games

Indexed keywords

BIOMEDICINE; CLINICAL RESEARCH; CROWDSOURCING; GENOMICS; MARKET; MINING; BIOLOGY; DATA MINING; HUMAN; INTERNET; SEARCH ENGINE; SMARTPHONE; SOCIAL MEDIA; TRENDS; VIDEO GAME;

COMPUTATIONAL BIOLOGY; CROWDSOURCING; DATA MINING; HUMANS; INTERNET; SEARCH ENGINE; SMARTPHONE; SOCIAL MEDIA; VIDEO GAMES;

EID: 84960090889     PISSN: 14675463     EISSN: 14774054     Source Type: Journal    
DOI: 10.1093/bib/bbv021     Document Type: Article

References (95)

1. Estellés-Arolas E, González-Ladrón-de-Guevara F. Towards an integrated crowdsourcing definition. J Inf Sci 2012;38: 189-200.
2. Islamaj Dogan R, Murray GC, Neveol A, et al. Understanding PubMed user search behavior through log analysis. Database 2009;2009:bap018.
3. Khare R, Leaman R, Lu Z. Accessing biomedical literature in the current information landscape. Methods Mol Biol 2014; 1159:11-31.
4. Khare R, Burger J, Aberdeen J, et al. Scaling Drug Indication Curation through Crowdsourcing. Database 2015; 2015:bav016.
5. Burger J, Doughty EK, Khare R, et al. Hybrid curation of genemutation relations combining automated extraction and crowdsourcing. Database 2014;2014:bau094.
6. Leiter A, Sablinski T, Diefenbach M, et al. Use of crowdsourcing for cancer clinical trial development. J Natl Cancer Inst 2014;106:dju258.
7. Good BM, Su AI. Games with a scientific purpose. Genome Biol 2011;12:135.
8. McCoy AB, Wright A, Laxmisan A, et al. Development and evaluation of a crowdsourcingmethodology for knowledge base construction: identifying relationships between clinical problems andmedications. J AmMed Inf Assoc 2012;19:713-18.
9. Bow HC, Dattilo JR, Jonas AM, et al. A crowdsourcing model for creating preclinical medical education study tools. Acad Med 2013;88:766-70.
10. Good BM, Su AI. Crowdsourcing for bioinformatics. Bioinformatics 2013;29:1925-33.
11. Snow R, O'Connor B, Jurafsky D, et al. Cheap and Fast-But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks. In: Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Honolulu, Hawaii, 2008, pp. 254-63.
12. Yetisgen-Yildiz M, Solti I, Xia F, et al. Preliminary experiments with Amazon's mechanical turk for annotating medical named entities. In: NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon's Mechanical Turk. Association for Computational Linguistics, Los Angeles, CA, 2010, pp. 180-3.
13. Lu Z, Kao HY, Wei CH, et al. The gene normalization task in BioCreative III. BMC Bioinformatics 2011;12(Suppl 8): S2.
14. Van Landeghem S, Bjorne J, Wei CH, et al. Large-scale event extraction from literature with multi-level gene normalization. PloS One 2013;8:e55814.
15. Lakhani KR, Boudreau KJ, Loh PR, et al. Prize-based contests can provide solutions to computational biology problems. Nat Biotechnol 2013;31:108-11.
16. Cooper S, Khatib F, Treuille A, et al. Predicting protein structures with a multiplayer online game. Nature 2010;466: 756-60.
17. Leaman R, Good BM, Su AI, et al. Session Introduction. In: Pacific Symposium on Biocomputing, Vol. 20, Hawaii, 2015, pp. 267-9.
18. White RW, Tatonetti NP, Shah NH, et al. Web-scale pharmacovigilance: listening to signals from the crowd. J Am Med Inf Assoc 2013;20:404-8.
19. Odgers DJ, Harpaz R, Callahan A, et al. Analyzing search behavior of healthcare professionals for drug safety surveillance. In: Pacific Symposium on Biocomputing, Vol. 20, Hawaii, 2015, pp. 306-17.
20. Freifeld CC, Brownstein JS, Menone CM, et al. Digital drug safety surveillance: monitoring pharmaceutical products in twitter. Drug Safety 2014;37:343-50.
21. Leaman R, Wojtulewicz L, Sullivan R, et al. Towards internetage pharmacovigilance: extracting adverse drug reactions from user posts to health-related social networks. In: Proceedings of the 2010 Workshop on Biomedical Natural Language Processing, Association for Computational Linguistics, 2010, pp. 117-25.
22. Tatonetti NP, Ye PP, Daneshjou R, et al. Data-driven prediction of drug effects and interactions. Sci Trans Med 2012;4: 125ra131.
23. Yang CC, Yang H, Jiang L. Postmarketing drug safety surveillance using publicly available health-consumer-contributed content in social media. ACM Trans Manag Inf Syst 2014;5.
24. Ryan PB, Madigan D, Stang PE, et al. Medication-wide association studies. CPT Pharmacometr Syst Pharmacol, 2, e76.
25. Wang W, Haerian K, Salmasian H, et al. A drug-adverse event extraction algorithm to support pharmacovigilance knowledge mining from PubMed citations. AMIA 2011;2011: 1464-70.
26. Ingram DG, Matthews CK, Plante DT. Seasonal trends in sleep-disordered breathing: evidence from Internet search engine query data. Sleep Breath 2015;19:79-84.
27. Araz OM, Bentley D, Muelleman RL. Using Google Flu Trends data in forecasting influenza-like-illness related ED visits in Omaha, Nebraska. Am J Emerg Med 2014;32:1016-23.
28. Bichero S. Global Smartphone Installed Base Forecast by Operating System for 88 Countries: 2007 to 2017.WWWdocument, http://www.strategyanalytics.com/default.aspx?mod = reportabstractviewer&a0=7834.
29. Pickard KT, Swan M. Big Desire to Share Big Health Data: A Shift in Consumer Attitudes toward Personal Health Information. In: AAAI 2014 Spring Symposia: Big Data Becomes Personal: Knowledge into Meaning, Association for the Advancement of Artificial Intelligence, 2014.
30. Morton CC. Innovating openly: researchers and patients turn to crowdsourcing to collaborate on clinical trials, drug discovery, and more. IEEE Pulse 2014;5:63-7.
31. Turner-McGrievy GM, Helander EE, Kaipainen K, et al. The use of crowdsourcing for dietary self-monitoring: crowdsourced ratings of food pictures are comparable to ratings by trained observers. J Am Med Inf Assoc 2014, doi: 10.1136/amiajnl-2014- 002636.
32. Huang M, Neveol A, Lu Z. Recommending MeSH terms for annotating biomedical articles. J Am Med Inf Assoc 2011;18: 660-7.
33. Lu Z, Kim W, Wilbur WJ. Evaluation of Query Expansion using MeSH in PubMed. Inf Retr Boston 2009;12:69-80.
34. Lu Z, Kim W, Wilbur WJ. Evaluating relevance ranking strategies for MEDLINE retrieval. J Am Med Inf Assoc 2009;16:32-6.
35. Van Auken K, Schaeffer ML, McQuilton P, et al. BC4GO: a fulltext corpus for the BioCreative IV GO task. Database 2014; 2014:bau074.
36. Islamaj Dogan R, Neveol A, Lu Z. A context-blocks model for identifying clinical relationships in patient records. BMC Bioinformatics 2011;12(Suppl 3), S3.
37. Leaman R, Khare R, Lu Z. NCBI at 2013 ShARe/CLEF eHealth Shared Task: Disorder Normalization in Clinical Notes with DNorm. In: CLEF 2013 Evaluation Labs and Workshop. The CLEF Initiative, Valencia - Spain, pp. 23-26, 2013.
38. Neveol A, Islamaj Dogan R, Lu Z. Semi-automatic semantic annotation of PubMed queries: a study on quality, efficiency, satisfaction. J Biomed Informatics, 44, 310-18.
39. Zhai H, Lingren T, Deleger L, et al. Web 2.0-based crowdsourcing for high-quality gold standard development in clinical natural language processing. J Med Internet Res 2013;15:e73.
40. Leaman R, Islamaj Dogan R, Lu Z. DNorm: disease name normalization with pairwise learning to rank. Bioinformatics 2013; 29:2909-17.
41. Leaman R, Wei CH, Lu Z. tmChem: a high performance tool for chemical named entity recognition and normalization. J Cheminform 2015;7(Suppl 1):S3.
42. Baumgartner W, Lu Z, Johnson HL, et al. An integrated approach to concept recognition in biomedical text. In: Proceedings of the Second BioCreative Challenge Evaluation Workshop, 2007, Vol. 23, pp. 257-71.
43. MacLean DL, Heer J. Identifying medical terms in patient-authored text: a crowdsourcing-based approach. J Am Med Inform Assoc 2013;20:1120-7.
44. Lafferty JD, McCallum A, Pereira FCN. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In: Proceedings of ICML '01 Proceedings of the Eighteenth International Conference on Machine Learning, 2001, pp. 282-9.
45. Good BM, Nanis M, Wu C, et al. Microtask Crowdsourcing for Disease Mention Annotation in PubMed Abstracts. In: Pacific Symposium on Biocomputing, Vol. 20, Hawaii, 2015, pp. 282-93.
46. Dogan RI, Leaman R, Lu Z. NCBI disease corpus: a resource for disease name recognition and concept normalization. J Biomed Inform 2014;47:1-10.
47. Dogan RI, Lu Z. An improved corpus of disease mentions in PubMed citations. In: Workshop on Biomedical Natural Language Processing. Association for Computational Linguistics, Montreal, Canada, 2012, pp. 91-9.
48. Wei CH, Kao HY, Lu Z. PubTator: a web-based text mining tool for assisting biocuration. Nucleic Acids Res 2013;41:W518-22.
49. Wei CH, Harris BR, Li D, et al. Accelerating literature curation with text-mining tools: a case study of using PubTator to curate genes in PubMed abstracts. Database 2012;2012:bas041.
50. Arighi CN, Carterette B, Cohen KB, et al. An overview of the BioCreative 2012 Workshop Track III: interactive text mining task. Database 2013;2013:bas056.
51. Lu Z, Hirschman L. Biocuration workflows and text mining: overview of the BioCreative 2012 Workshop Track II. Database 2012;2012:bas043.
52. Khare R, Li J, Lu Z. LabeledIn: cataloging labeled indications for human drugs. J Biomed Inform 2014;52:448-56.
53. Burger JD, Doughty E, Bayer S, et al. Validating candidate gene-mutation relations in MEDLINE abstracts via crowdsourcing. Data Integr Life Sci Lect Notes Comput Sci 2012;7348: 83-91.
54. Wei CH, Kao HY, Lu Z. SR4GN: a species recognition software tool for gene normalization. PloS One 2012;7:e38460.
55. Doughty E, Kertesz-Farkas A, Bodenreider O, et al. Toward an automatic method for extracting cancer- and other diseaserelated point mutations from the biomedical literature. Bioinformatics 2011;27:408-15.
56. Mortensen JM, Minty EP, Januszyk M, et al. Using the wisdom of the crowds to find critical errors in biomedical ontologies: a study of SNOMED CT. J Am Med Inform Assoc 2014, doi: 10.1136/amiajnl-2014-002901.
57. Brown AW, Allison DB. Using crowdsourcing to evaluate published scientific literature: methods and example. PloS One 2014;9:e100647.
58. Finn RD, Gardner PP, Bateman A. Making your database available through Wikipedia: the pros and cons. Nucleic Acids Res 2012;40:D9-12.
59. Loguercio S, Good BM, Su AI. Dizeez: an online game for human gene-disease annotation. PloS One 2013;8:e71171.
60. Good BM, Howe DG, Lin SM, et al. Mining the Gene Wiki for functional genomic knowledge. BMC Genomics 2011;12:603.
61. Waldispühl J, Kam A, Gardner P. Crowdsourcing RNA Structural Alignments with an Online Computer Game. In: Pacific Symposiumon Biocomputing, Vol. 20, Hawaii, 2015, pp. 330-41.
62. Ahn LV, Dabbish L. Designing games with a purpose. Commun ACM 2008;51:58-67.
63. Gottl F. Crowdsourcing with Gaminification. Advances in Embedded Interactive Systems, Vol. 2, Passau, Germany, 2014.
64. Stegman M. Immune Attack players perform better on a test of cellular immunology and self confidence than their classmates who play a control video game. Faraday Discuss 2014; 169:403-23.
65. Perry D, Aragon C, Cruz S, et al. Human centered game design for bioinformatics and cyberinfrastructure learning. In: Proceedings of the ACM Conference on Extreme Science and Engineering Discovery Environment: Gateway to Discovery. Association for Computing Machinery (ACM), 2013.
66. Kawrykow A, Roumanis G, Kam A, et al. Phylo: a citizen science approach for improving multiple sequence alignment. PloS One 2012;7:e31362.
67. Khatib F, DiMaio F, Foldit Contenders G, et al. Crystal structure of a monomeric retroviral protease solved by protein folding game players. Nat Struct Mol Biol 2011;18:1175-7.
68. Lee J, Kladwang W, Lee M, et al. RNA design rules froma massive open laboratory. Proc Natl Acad Sci USA 2014;111:2122-7.
69. Treuille A, Das R. Scientific rigor through videogames. Trends Biochem Sci 2014;39:507-9.
70. Riedel-Kruse IH, Chung AM, Dura B, et al. Design, engineering and utility of biotic games. Lab Chip 2011;11:14-22.
71. Riedel-Kruse I, Blikstein P. Biotic games and cloud experimentation as novel media for biophysics education. Bulletin of the American Physical Society 2014;59.
72. Moult J, Pedersen JT, Judson R, et al. A large-scale experiment to assess protein structure prediction methods. Proteins 1995; 23:ii-v.
73. Arighi CN, Wu CH, Cohen KB, et al. BioCreative-IV virtual issue. Database 2014;2014:bau039.
74. Wu CH, Arighi CN, Cohen KB, et al. BioCreative-2012 virtual issue. Database 2012;2012:bas049.
75. Kim JD, Pyysalo S, Ohta T, et al. Overview of BioNLP Shared Task 2011. In: Proceedings of BioNLP Shared Task 2011 Workshop. Association for Computational Linguistics, Portland, OR, 2011, pp. 1-6.
76. Hersh W, Voorhees E. TREC genomics special issue overview. Inf Retr 2009;12:1-15.
77. Uzuner O, South BR, Shen S, et al. 2010 i2b2/VA challenge on concepts, assertions, and relations in clinical text. J Am Med Inform Assoc 2011;18:552-6.
78. Huang M, Lu Z. Community Challenges in Biomedical Text Mining over 10 Years: Success, Failure, and the Future. Brief Bioinformatics 2016;17(1):132-44.
79. Marbach D, Costello JC, Kuffner R, et al. Wisdom of crowds for robust gene network inference. Nat Methods 2012;9: 796-804.
80. Margolin AA, Bilal E, Huang E, et al. Systematic analysis of challenge-driven improvements in molecular prognostic models for breast cancer. Sci Trans Med 2013;5:181re181.
81. Khoury GA, Liwo A, Khatib F, et al. WeFold: a coopetition for protein structure prediction. Proteins 2014;82:1850-68.
82. Ross J, Irani I, Silberman MS, et al. Who are the Crowdworkers?: Shifting Demographics in Amazon Mechanical Turk. In: CHI '10 Extended Abstracts on Human Factors in Computing Systems. ACM, Atlanta, Georgia, 2010, pp. 2863-72.
83. Butler D. When Google got flu wrong. Nature 2013;494: 155-6.
84. Fort K, Adda G, Cohen KB. Amazon Mechanical Turk: Gold Mine or Coal Mine?. In: Computational Linguistics, 37(2) (2011), pp. 413-20.
85. Machine D, Ophoff J. Understanding What Motivates Participation on Crowdsourcing Platforms. In: e-Skills for Knowledge Production and Innovation Conference, Cape Town, South Africa, 2014, pp. 191-200.
86. Parvanta C, Roth Y, Keller H. Crowdsourcing 101: a few basics to make you the leader of the pack. Health Promot Pract 2013; 14:163-7.
87. Khare R, An Y, Song IY, et al. Can clinicians create high-quality databases: a study on a flexible electronic health record (fEHR) system. In: International Health Informatics Symposium. ACM, Washington, DC, 2010, pp. 8-17.
88. Wagholikar KB, MacLaughlin KL, Kastner TM, et al. Formative evaluation of the accuracy of a clinical decision support systemfor cervical cancer screening. J Am Med Inform Assoc 2013; 20:749-57.
89. Khare R, An Y, Wolf S, et al. Understanding the EMR error control practices among gynecologic physicians. In: iConference 2013, iSchools, Fort Worth, Texas, 2013, pp. 289-301.
90. Blair DR, Wang K, Nestorov S, et al. Quantifying the impact and extent of undocumented biomedical synonymy. PLoS Comput Biol 2014;10:e1003799.
91. Yu B, Willis M, Sun P, et al. Crowdsourcing participatory evaluation of medical pictograms using Amazon Mechanical Turk. J Med Int Res 2013;15:e108.
92. Nevin CR, Westfall AO, Rodriguez JM, et al. Gamification as a tool for enhancing graduate medical education. Postgrad Med J 2014;90:685-93.
93. Brown HR, Zeidman P, Smittenaar P, et al. Crowdsourcing for cognitive science-the utility of smartphones. PloS One 2014;9: e100662.
94. Harpaz R, DuMouchel W, LePendu P, et al. Empirical Bayes Model to Combine Signals of Adverse Drug Reactions. In: Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, 2013, pp. 1339-47.
95. Neilson J. Participation inequality: lurkers vs. contributors in internet communities. Jakob Nielsen's Alert-box, 2006.