Using the wisdom of the crowds to find critical errors in biomedical ontologies: A study of SNOMED CT

Journal of the American Medical Informatics Association

Volumn 22, Issue 3, 2015, Pages 640-648

  Mortensen, Jonathan M ^a   Minty, Evan P ^a,b   Januszyk, Michael ^a   Sweeney, Timothy E ^a   Rector, Alan L ^c   Noy, Natalya F ^a,d   Musen, Mark A ^a  

^a STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b UNIVERSITY OF CALGARY   (Canada)

^c UNIVERSITY OF MANCHESTER   (United Kingdom)

^d GOOGLE INC   (United States)

Author keywords

Biomedical ontology; Crowdsourcing; Ontology engineering; SNOMED CT

Indexed keywords

ARTICLE; BAYESIAN LEARNING; CROWDSOURCING; ERROR; MEDICAL ONTOLOGY; METHODOLOGY; PROSPECTIVE STUDY; QUALITY CONTROL; SYSTEMATIZED NOMENCLATURE OF MEDICINE; TAXONOMY; BAYES THEOREM; BIOLOGICAL ONTOLOGY; CLASSIFICATION; DISEASES; HUMAN;

BAYES THEOREM; BIOLOGICAL ONTOLOGIES; CROWDSOURCING; DISEASE; HUMANS; SYSTEMATIZED NOMENCLATURE OF MEDICINE;

EID: 84964697298     PISSN: 10675027     EISSN: 1527974X     Source Type: Journal    
DOI: 10.1136/amiajnl-2014-002901     Document Type: Article

References (27)

1. Howe J. The rise of crowdsourcing. Wired Mag. 2006;14:1-4.
2. Quinn AJ, Bederson BB. Human computation: a survey and taxonomy of a growing field. Proceedings of the 2011 annual conference on Human factors in computing systems-CHI'11. Vancouver, BC: ACM, 2011:1403-1412.
3. Lintott CJ, Schawinski K, Slosar A, et al. Galaxy Zoo: morphologies derived from visual inspection of galaxies from the Sloan Digital Sky Survey. Mon Not R Astron Soc. 2008; 389:1179-1189.
4. Cooper S, Khatib F, Treuille A, et al. Predicting protein structures with a multiplayer online game. Nature. 2010;466: 756-760.
5. Staab S, Studer R. Handbook on ontologies. 2nd edn. Springer-Verlag New York Inc, 2009.
6. Bodenreider O, Stevens R. Bio-ontologies: current trends and future directions. Brief Bioinform. 2006;7:256-274.
7. Rubin DL, Shah NH, Noy NF. Biomedical ontologies: a functional perspective. Brief Bioinform. 2008;9:75-90.
8. Hunter L, Lu Z, Firby J, et al. OpenDMAP: an open source, ontology-driven concept analysis engine, with applications to capturing knowledge regarding protein transport, protein interactions and cell-type-specific gene expression. BMC Bioinformatics. 2008;9:78.
9. Hoehndorf R, Dumontier M, Gennari JH, et al. Integrating systems biology models and biomedical ontologies. BMC Syst Biol. 2011;5:124.
10. Segal E, Shapira M, Regev A, et al. Module networks: identifying regulatory modules and their condition-specific regulators from gene expression data. Nat Genet. 2003;34:166-176.
11. LePendu P, Iyer S V, Bauer-Mehren A, et al. Pharmacovigilance using clinical notes. Clin Pharmacol Ther. 2013;93: 547-555.
12. Whetzel PL, Noy NF, Shah NH, et al. BioPortal: enhanced functionality via new web services from the National Center for Biomedical Ontology to access and use ontologies in software applications. Nucleic Acids Res. 2011;39:W541-545.
13. Blumenthal D, Tavenner M. The "meaningful use" regulation for electronic health records. N Engl J Med. 2010;363: 501-504.
14. Ceusters W, Smith B, Goldberg L. A terminological and ontological analysis of the NCI Thesaurus. Methods Inf Med. 2005;44:498.
15. Rector AL, Brandt S, Schneider T. Getting the foot out of the pelvis: modeling problems affecting use of SNOMED CT hierarchies in practical applications. J Am Med Informatics Assoc. 2011;18:432-440.
16. Zhu X, Fan JW, Baorto DM, et al. A review of auditing methods applied to the content of controlled biomedical terminologies. J Biomed Inform. 2009;42:413-425.
17. Ochs C, Perl Y, Geller J, et al. Scalability of abstraction-network-based quality assurance to large SNOMED hierarchies. AMIA Annu Symp Proc. 2013;2013:1071-1080.
18. Noy NF, Mortensen JM, Alexander PR, et al. Mechanical Turk as an ontology engineer? Using microtasks as a component of an ontology engineering workflow. Web Science, 2013.
19. Lawley MJ, Bousquet C. Fast classification in Protégé: Snorocket as an OWL 2 EL reasoner. Proceedings of the 6th Australasian Ontology Workshop (IAOA'10). Conferences in Research and Practice in Information Technology 2010; 45-49.
20. Horridge M, Bechhofer S. The OWL API: a Java API for working with OWL 2 ontologies. OWLED 2009;11-21.
21. Mortensen JM, Noy NF, Musen MA, et al. Crowdsourcing ontology verification. International Conference on Biomedical Ontologies. 2013.
22. Simpson E, Roberts S, Psorakis I, et al. Dynamic Bayesian combination of multiple imperfect classifiers, 2012. http://arxiv.org/abs/1206.1831
23. Linstone HA, Turoff M. The Delphi method: techniques and applications. Addison-Wesley, 1975.
24. Randolph JJ. Free-marginal multirater kappa (multirater jfree): an alternative to Fleiss' fixed-marginal multirater kappa. Joensuu learning and instruction symposium, 2005.
25. Efron B. The Jackknife, the Bootstrap and other resampling plans. SIAM, 1982.
26. Khatri P, Druaghici S. Ontological analysis of gene expression data: current tools, limitations, and open problems. Bioinformatics. 2005;21:3587-3595.
27. Murphy SN, Weber G, Mendis M, et al. Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2). J Am Med Informatics Assoc. 2010; 17:124-130.